[00:01:37] !log catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master' [00:01:43] Logged the message, Master [00:01:56] !log catrope synchronized php-1.22wmf13/extensions/VisualEditor 'Update VE to master' [00:02:01] Logged the message, Master [00:04:30] !log catrope synchronized php-1.22wmf12/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch' [00:04:36] Logged the message, Master [00:04:47] !log catrope synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch' [00:04:52] Logged the message, Master [00:18:58] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:07] Okay [00:19:25] So did you guys upgrade from Ubuntu 8.04 to 12.04? [00:20:37] I thought they were on 10.04 [00:20:56] There's this ticket to track updating to 12.04: https://bugzilla.wikimedia.org/show_bug.cgi?id=36623 [00:23:04] Oh okay [00:26:35] (03PS1) 10Danny B.: skwiktionary: Set site logo to local file [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80321 [00:28:44] (03PS2) 10Bsitu: Enable job queue to process web and email notifs on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/61647 [00:29:53] Techman, Krenair: I believe we're mostly on precise (12.04) now [00:30:13] Okay. Is the rest 10.04 or 8.04? [00:30:43] k [00:31:14] I don't know. Probably some mix of the two biased towards 10.04, but I have no data on this [00:31:26] I just know that over the past year I've seen emails like "all machines in cluster XYZ are now on 12.04, yay!" [01:14:48] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:35] (03PS1) 10Tim Landscheidt: Tools: Add python-oursql to exec_environ [operations/puppet] - 10https://gerrit.wikimedia.org/r/80327 [01:44:11] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:01] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 95449 bytes in 3.214 second response time [02:15:08] !log LocalisationUpdate completed (1.22wmf13) at Thu Aug 22 02:15:07 UTC 2013 [02:15:14] Logged the message, Master [02:22:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [02:28:28] !log LocalisationUpdate completed (1.22wmf12) at Thu Aug 22 02:28:28 UTC 2013 [02:28:34] Logged the message, Master [02:38:17] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Aug 22 02:38:17 UTC 2013 [02:38:23] Logged the message, Master [02:44:27] PROBLEM - Host mw16 is DOWN: PING CRITICAL - Packet loss = 100% [02:45:47] RECOVERY - Host mw16 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [02:52:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [03:23:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [03:30:04] (03PS1) 10Tim Landscheidt: WIP: Tools: Add infrastructure for AWStats [operations/puppet] - 10https://gerrit.wikimedia.org/r/80332 [03:33:07] (03CR) 10Tim Landscheidt: [C: 04-1] "Not even tested properly." [operations/puppet] - 10https://gerrit.wikimedia.org/r/80332 (owner: 10Tim Landscheidt) [03:38:36] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:36] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [03:51:42] (03CR) 10Dzahn: [C: 031] "+1, dunno about Leslie's question, just (minor: single quotes around user and file modes would be nice)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/53145 (owner: 10Petrb) [03:53:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [03:55:22] (03PS5) 10Dzahn: replicate Gerrit repos to Jenkins slave gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/75500 (owner: 10Hashar) [03:55:29] (03CR) 10jenkins-bot: [V: 04-1] replicate Gerrit repos to Jenkins slave gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/75500 (owner: 10Hashar) [03:58:48] (03CR) 10Dzahn: [C: 031] "path conflict / rebase button fails" [operations/puppet] - 10https://gerrit.wikimedia.org/r/75499 (owner: 10Hashar) [04:04:18] (03CR) 10Dzahn: [C: 032] beta: phase out shell autoupdater [operations/puppet] - 10https://gerrit.wikimedia.org/r/76905 (owner: 10Hashar) [04:08:13] (03CR) 10Dzahn: [C: 032] fix system_role for role::protoproxy::ssl::beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/75074 (owner: 10Hashar) [04:11:53] (03CR) 10Dzahn: [C: 032] mediawiki_singlenode: exec[] -> Exec[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77125 (owner: 10Hashar) [04:47:41] (03CR) 10Greg Grossmeier: "Not an Apache conf expert, so not commenting on that, but, I just want to be explicit here:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/80314 (owner: 10Dzahn) [04:48:33] Augeas[iptables udp2log_drop_udp_udp purge] is broken [04:48:49] it's set repeatedly by every puppet run on fluorine [04:48:53] haven't investigated [04:52:55] Tim, did you merge my auth log Puppet change from yesterday? I don't see it on fluorine, and I don't see failure in the Puppet log either. Not pinging since it's not urgent. [05:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [05:56:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [06:14:45] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [06:34:37] (03PS1) 10Faidon: Cleanup ns0/1-old [operations/dns] - 10https://gerrit.wikimedia.org/r/80340 [06:34:37] (03PS1) 10Faidon: Remove comment next to ns0/ns1/ns2 NS records [operations/dns] - 10https://gerrit.wikimedia.org/r/80341 [06:34:45] :D [06:38:20] (03CR) 10Faidon: [C: 032 V: 032] Cleanup ns0/1-old [operations/dns] - 10https://gerrit.wikimedia.org/r/80340 (owner: 10Faidon) [06:39:45] (03CR) 10Faidon: [C: 032 V: 032] Remove comment next to ns0/ns1/ns2 NS records [operations/dns] - 10https://gerrit.wikimedia.org/r/80341 (owner: 10Faidon) [06:56:34] PROBLEM - NTP peers on dobson is CRITICAL: NTP CRITICAL: No response from NTP server [07:01:10] (03PS1) 10Faidon: authdns: switch final migration bits [operations/puppet] - 10https://gerrit.wikimedia.org/r/80343 [07:01:11] (03PS1) 10Faidon: Cleanup dns::auth-server [operations/puppet] - 10https://gerrit.wikimedia.org/r/80344 [07:01:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:44] (03CR) 10Faidon: [C: 032] authdns: switch final migration bits [operations/puppet] - 10https://gerrit.wikimedia.org/r/80343 (owner: 10Faidon) [07:02:09] (03CR) 10Faidon: [C: 032] Cleanup dns::auth-server [operations/puppet] - 10https://gerrit.wikimedia.org/r/80344 (owner: 10Faidon) [07:02:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.156 second response time [07:02:34] RECOVERY - NTP peers on dobson is OK: NTP OK: Offset -0.000389 secs [07:13:14] (03PS1) 10Faidon: authdns: add IPv6 [operations/puppet] - 10https://gerrit.wikimedia.org/r/80345 [07:14:06] (03CR) 10Faidon: [C: 032] authdns: add IPv6 [operations/puppet] - 10https://gerrit.wikimedia.org/r/80345 (owner: 10Faidon) [07:23:06] (03PS1) 10Faidon: Add IPv6 mapped address for rubidium/mexia/eeden [operations/puppet] - 10https://gerrit.wikimedia.org/r/80346 [07:23:21] (03CR) 10Faidon: [C: 032 V: 032] Add IPv6 mapped address for rubidium/mexia/eeden [operations/puppet] - 10https://gerrit.wikimedia.org/r/80346 (owner: 10Faidon) [08:27:18] (03PS2) 10TTO: Deny reupload permission to users and autoconfirmed on ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78502 [08:34:39] (03CR) 10Matthias Mullie: [C: 031] Enable job queue to process web and email notifs on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/61647 (owner: 10Bsitu) [08:57:35] PROBLEM - Puppet freshness on lvs1001 is CRITICAL: No successful Puppet run in the last 10 hours [09:18:56] i thought apergos fixed that issue [09:43:29] which issue [09:48:24] (03PS2) 10TTO: Add autoreview protection level for arwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76277 [09:48:56] the lvs1001 puppet freshness issue with snmp traps [09:49:22] that's not an snmptrap issue [09:49:37] the remaining whiners are either hosts half-decommed or hosts where [09:49:49] Skipping run of Puppet configuration client; administratively disabled; use 'puppet Puppet configuration client --enable' to re-enable. [09:50:52] if no one actually locked it I'm happy to re-enable puppet over there, I presumed you might be looking at it after my mishap of the other day [09:51:52] special pages update still doesn't run, is it on purpose or known but yet not solved bug? [09:53:50] i didn't lock puppet on that box [09:53:55] don't think paravoid did either? [09:54:00] in any case, that lock gets removed after a day [09:54:24] which box? [09:54:37] lvs1001 [09:54:41] (I have /ignored the freshness checks) [09:54:51] no I haven't [09:55:02] so that's probably the snmptrap issue then [09:55:11] it's not the snmptrap issue [09:55:28] yes it is [09:55:29] the syslog on lvs1001 says it's been locked [09:55:34] your change from yesterday [09:55:51] ipaddress_eth0 will be null on the lvs box [09:55:52] the syslog from right now says it's been locked [09:56:10] okay, but the trap would be broken too [09:56:19] cat snmp.conf [09:56:31] it has nothign in it right now cause puppet isn't running there [09:56:40] I'm running it. [09:56:45] on lvs1002 it's fine [09:56:51] the conf fille [09:57:18] because I put that manually there perhaps [09:57:22] yesterday [09:57:35] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Thu Aug 22 09:57:32 UTC 2013 [09:58:02] facter | grep ipaddr [09:58:09] ipaddress_eth0 => 208.80.154.55 [09:58:22] and the template checks for existence first [09:58:24] done [09:58:25] The last Puppet run was at Tue Aug 20 06:47:52 UTC 2013 (3067 minutes ago). [09:58:25] before using that var [09:58:52] so two days + 3h [09:59:13] it now has a nice snmp.conf file (it had one without any line) [09:59:17] before the snmp.conf change :) [09:59:19] that has the canonical ip in it [09:59:32] due to your puppet run [10:00:17] and nte that icinga says it is happy now [10:00:51] 1041 packets transmitted, 962 received, 7% packet loss, time 1041541ms [10:00:54] blergh [10:00:56] eqiad -> esams [10:01:17] bast1001 -> hooft specifically [10:01:40] also bast1001 -> home, so it's on the eqiad side, not esams [10:07:30] seems better [10:10:05] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:55] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [10:19:35] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:20:56] (03PS1) 10Faidon: Switch ns2 to new /32 service IP [operations/dns] - 10https://gerrit.wikimedia.org/r/80352 [10:20:57] (03PS1) 10Faidon: Add AAAA for ns0/1/2 [operations/dns] - 10https://gerrit.wikimedia.org/r/80353 [10:20:58] (03PS5) 10TTO: Continuing to clean up InitialiseSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78637 [10:21:52] (03CR) 10TTO: "Please merge it soon! The manual rebase took me about half an hour. (Is there a better way?)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78637 (owner: 10TTO) [10:22:45] (03CR) 10Faidon: [C: 032] Switch ns2 to new /32 service IP [operations/dns] - 10https://gerrit.wikimedia.org/r/80352 (owner: 10Faidon) [10:23:25] (03CR) 10TTO: "And already this conflicts with Ibc5e3c89eadde079c12a29559f822fa25a146bae. Should I make a separate parallel changeset for these conflicti" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78637 (owner: 10TTO) [10:23:46] (03PS2) 10Faidon: Add AAAA for ns0/1/2 [operations/dns] - 10https://gerrit.wikimedia.org/r/80353 [10:24:20] (03CR) 10Faidon: [C: 032] Add AAAA for ns0/1/2 [operations/dns] - 10https://gerrit.wikimedia.org/r/80353 (owner: 10Faidon) [10:28:35] (03PS1) 10Faidon: authdns: switch eeden to new ns2 service IP [operations/puppet] - 10https://gerrit.wikimedia.org/r/80354 [10:30:21] (03CR) 10Faidon: [C: 032] authdns: switch eeden to new ns2 service IP [operations/puppet] - 10https://gerrit.wikimedia.org/r/80354 (owner: 10Faidon) [10:38:55] (03PS1) 10Faidon: authdns: fix interface for service IP [operations/puppet] - 10https://gerrit.wikimedia.org/r/80355 [10:39:38] (03CR) 10Faidon: [C: 032] authdns: fix interface for service IP [operations/puppet] - 10https://gerrit.wikimedia.org/r/80355 (owner: 10Faidon) [10:41:36] (03PS1) 10ArielGlenn: ref to misc::statistics::rsync::jobs removed [operations/puppet] - 10https://gerrit.wikimedia.org/r/80356 [10:42:16] (03CR) 10ArielGlenn: [C: 032] ref to misc::statistics::rsync::jobs removed [operations/puppet] - 10https://gerrit.wikimedia.org/r/80356 (owner: 10ArielGlenn) [10:46:35] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Thu Aug 22 10:46:26 UTC 2013 [11:03:14] !log disabling puppet on eeden for the next couple of days [11:03:18] Logged the message, Master [11:15:35] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [11:28:15] RECOVERY - search indices - check lucene status page on search32 is OK: HTTP OK: HTTP/1.1 200 OK - 504 bytes in 0.055 second response time [11:28:36] wtf nimsoft [11:28:42] nice [11:32:28] 8004: No A records found for ns0.wikimedia.org [11:33:22] it sent a recovery, but I haven't changed anything [11:33:35] and I don't see anything wrong [11:34:45] even nicer [11:34:52] I got the recovery page just now [11:36:17] !log switching off alerts for watchmouse's DNS alert while investigating [11:36:22] Logged the message, Master [11:38:09] I don't see it [11:38:12] the problem [11:39:38] it started when I added the AAAAs [11:39:41] might just be buggy [11:50:44] not seeing any issues [11:54:12] okay, I switched the check to wikimedia.org NS [11:54:29] hm, I wonder if that would be satisfied by glues... [11:55:54] No NS records found for wikimedia.org [11:56:01] duh? [11:59:02] (03CR) 10Akosiaris: [C: 04-1] "The patch does not do what the commit message implies. The commit message states http->https on etherpad.wikimedia.org which for me means " [operations/puppet] - 10https://gerrit.wikimedia.org/r/80314 (owner: 10Dzahn) [12:29:21] stupid wathcmouse [12:29:41] (03PS1) 10Akosiaris: Making the process of defining host backups easier [operations/puppet] - 10https://gerrit.wikimedia.org/r/80363 [13:04:50] !log changed watchmouse alert and re-enabled [13:04:55] Logged the message, Master [13:07:21] (03PS1) 10Anomie: Enable CodeEditor for CSS/JS on testing wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80366 [13:51:19] (03PS1) 10Faidon: authdns: fix puppet evaluation order [operations/puppet] - 10https://gerrit.wikimedia.org/r/80369 [13:51:49] (03CR) 10Faidon: [C: 032] authdns: fix puppet evaluation order [operations/puppet] - 10https://gerrit.wikimedia.org/r/80369 (owner: 10Faidon) [14:30:41] !log maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug' [14:30:47] Logged the message, Master [14:33:20] !log reedy synchronized php-1.22wmf14/ 'Initial file sync' [14:33:25] Logged the message, Master [14:34:24] !log reedy synchronized php-1.22wmf14/extensions/OAuth/ [14:34:30] Logged the message, Master [14:34:46] !log maxsem synchronized php-1.22wmf13/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php 'debug over' [14:34:51] Logged the message, Master [14:36:10] (03PS1) 10Reedy: Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80385 [14:36:36] !log reedy synchronized docroot and w [14:36:41] Logged the message, Master [14:54:56] !log reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache [14:55:03] Logged the message, Master [15:03:24] On srv281 can someone remove /usr/local/apache/common/php-1.22wmf2 as root please? [15:07:00] Reedy, /me wonders if sync-common should just bail out if run as root [15:07:30] !log reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf14, build l10n cache [15:07:35] Logged the message, Master [15:12:17] (03CR) 10Reedy: [C: 032] Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80385 (owner: 10Reedy) [15:12:25] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf13 for now [15:12:26] (03Merged) 10jenkins-bot: Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80385 (owner: 10Reedy) [15:12:30] Logged the message, Master [15:18:32] srv281 is to be decommed most likely [15:18:41] there is a ticket ( Reedy ) [15:23:57] {{sokillit}} [15:29:25] (03PS1) 10Faidon: Remove AAAA for ns0/1/2 [operations/dns] - 10https://gerrit.wikimedia.org/r/80392 [15:29:48] (03CR) 10Faidon: [C: 032] Remove AAAA for ns0/1/2 [operations/dns] - 10https://gerrit.wikimedia.org/r/80392 (owner: 10Faidon) [15:33:18] apergos and reedy there is a ticket..i have it and will be taking it offline shortly [15:33:37] w00t [15:34:42] reedy removed /usr/local/apache/common/php-1.22wmf2 on srv281 [15:34:56] (03PS1) 10MaxSem: Decommission srv281, RT#5647 [operations/puppet] - 10https://gerrit.wikimedia.org/r/80393 [15:35:00] cmjohnson1, ^^ [15:36:32] haha [15:36:33] maxsem can you add files/dhcpd/linux-host-entries.ttyS1-115200:host srv281 [15:37:02] sure, I was worried that it will make the host inaccessible before you wipe it [15:37:13] nah..it's only used for pxe [15:37:54] (03PS2) 10MaxSem: Decommission srv281, RT#5647 [operations/puppet] - 10https://gerrit.wikimedia.org/r/80393 [15:40:18] (03CR) 10Cmjohnson: [C: 032 V: 032] Decommission srv281, RT#5647 [operations/puppet] - 10https://gerrit.wikimedia.org/r/80393 (owner: 10MaxSem) [15:41:27] Is watchmouse/status.wikimedia.org config done from their side/web interface? [15:41:34] maxsem merged...thx!!! [15:41:51] Reedy: yes IIRC [15:48:01] (03CR) 10Tim Landscheidt: "awstats update process must log >> ~/something.log." [operations/puppet] - 10https://gerrit.wikimedia.org/r/80332 (owner: 10Tim Landscheidt) [15:54:00] if anyone wants to watch puppet conf live stream...https://puppetlabs.com/puppetconf-2013-live [15:54:10] starts in 5 mins or so [15:54:25] wee [15:59:19] sure [15:59:30] I signed up but didn't see email from them just yet [16:01:47] hmm says it's enabled and yet I see no presentation [16:05:22] had to reload [16:15:16] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:45] cmjohnson1: apergos: unwatchable with html5 or gnash? [16:20:11] dunno, I have the adobe flash plugin, not sure what it's using though [16:20:21] so far you are missing nothing imho [16:21:17] player.swf? blah blah [16:21:31] and yet the alt text is [16:21:33] "Please upgrade your browser to view HTML 5 content" [16:23:40] haha [16:32:45] (03CR) 10Greg Grossmeier: [C: 031] Enable CodeEditor for CSS/JS on testing wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80366 (owner: 10Anomie) [16:32:46] Jeff_Green: i see 4646 is stalled [16:33:02] we got an OTRS ticket complaining about spam coming from @wikipedia.org [16:33:23] thoughts on copying over SPF from @wikimedia.org ? [16:33:37] that ticket was idiotic [16:33:45] did you read the comments on it? [16:33:55] they were telling us to deploy garbage [16:34:40] Jeff_Green: i did. it's just very hard for me to believe it was really 5 months ago [16:34:59] Jeff_Green: anyway, what about ^^^ ? [16:35:20] there's already work in progress on deploying DKIM which is probably more valuable anyway [16:35:40] ok, is there a ticket? [16:36:02] 5585 [16:36:03] gah, must be sleeping. i just searched for ticket instead of dkim [16:36:24] well that's a request, but paravoid has been working on this [16:36:39] afaik he's working toward covering all wmf mail [16:36:47] well the packing slip seems irrelevant [16:36:52] yes it does [16:36:54] sbernardin: fyi ^ [16:37:13] this double-ticketing stuff is madness [16:37:25] heh [16:37:26] there are like 6 open tickets in two different systems covering spf and dkim [16:37:27] (03PS1) 10Ryan Lane: Use nginx module for protoproxy and disable notify [operations/puppet] - 10https://gerrit.wikimedia.org/r/80401 [16:37:42] my vote is to close all but one of them, and refer to that one [16:38:00] (03CR) 10jenkins-bot: [V: 04-1] Use nginx module for protoproxy and disable notify [operations/puppet] - 10https://gerrit.wikimedia.org/r/80401 (owner: 10Ryan Lane) [16:38:01] mark: ^^ [16:38:04] -_- [16:39:30] puppet has the dumbest syntax [16:40:13] !log mwalker Started syncing Wikimedia installation... : Updating CentralNotice to master [16:40:17] (03PS2) 10Ryan Lane: Use nginx module for protoproxy and disable notify [operations/puppet] - 10https://gerrit.wikimedia.org/r/80401 [16:40:19] Logged the message, Master [16:42:03] heh, /spf/ matches on OSPF :P [16:42:39] jeremyb: what were you referencing up there? [16:43:42] sbernardin: https://rt.wikimedia.org/Ticket/Display.html?id=5585#txn-126586 [16:44:17] Jeff_Green: I'd agree with you (closing all but one), with the caveat to use a public issue tracker when at all possible :) [16:45:15] jeremyb: rt's search feature is sort of a bad joke [16:45:41] Jeff_Green: i don't use the UI usually. i do a search and then go to advanced and manually edit it to match what i want [16:45:52] greg-g: yeah, I understand. otoh working from double task queues is sort of maddening [16:45:54] and then of course i have to click the button to actually get results [16:46:08] jeremyb: yeah me too, but even that is awful [16:46:12] Jeff_Green: one could just be a pointer [16:46:37] Jeff_Green: i'm a big fan of bugzilla's quick search [16:46:51] jeremyb: that's an error on my part [16:47:07] wrong ticket [16:47:11] sbernardin: right but maybe you want to send it to the right place wherever that is :) [16:47:40] yup...thanks [16:47:44] Jeff_Green: anyway, copying spf over *should* be really simple (especially since we don't mail from there); so if we think something may hold up dkim then maybe we should just do spf first. [16:48:04] (to clarify this is copying what jeff did for wikimedia.org onto wikipedia.org and other project domains) [16:48:04] Jeff_Green: shouldn't be "otoh" it should be "right, and for any private stuff we'll X, for public we'll use Y" or, you could use Y with a special non-public queue that can be toggled back/forth as needed :) [16:48:14] i guess we maybe do mail from there for OTRS actually [16:48:24] jeremyb: I don't see why we should rush what is already in progress, rather than continue to work through it as we already are? [16:48:54] jeremyb: I'd chat with faidon about it, see what he's doing. [16:49:12] Jeff_Green: i'm not saying rush. but seems like this should be a 5 min fix. so maybe would be worth it [16:49:20] greg-g: sure :) [16:50:06] (and also shouldn't interfere with dkim? right?) [16:51:08] jeremyb: you can't just roll out SPF or DKIM like that. we have to identify every possible source of legitimate mail first, and make sure they're included [16:52:00] for SPF every IP that originates legit mail has to be included in the DNS record, or we're effectively declaring it spam [16:52:19] and for DKIM every host that originates mail has to sign it on the way out, or we effectively declare it spam [16:52:34] Jeff_Green: well maybe i haven't thought about it enough. AFAIK there's no personal addresses @wikipedia.org (just OTRS aliases for addresses that are primarily on @wikimedia.org) [16:53:52] or maybe really old aliases. but those (like OTRS?) should be inbound only. [16:54:00] anyway, i'm not going to push it :) [16:54:01] could anybody purge the Parsoid varnishes for me? [16:54:03] jeremyb: sure, but I'm sure you can see the issue with rolling out these features without doing the due diligence [16:54:50] !log mwalker Finished syncing Wikimedia installation... : Updating CentralNotice to master [16:54:56] Logged the message, Master [16:55:00] !log updated Parsoid to 437374dc [16:55:06] Logged the message, Master [16:56:51] !log mwalker synchronized php-1.22wmf14/extensions/CentralNotice/special/ 'Updating wmf14 to include CentralNotice bugfix 53032' [16:56:56] Logged the message, Master [17:01:11] (03PS6) 10Demon: replicate Gerrit repos to Jenkins slave lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/75499 (owner: 10Hashar) [17:01:12] (03PS6) 10Demon: replicate Gerrit repos to Jenkins slave gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/75500 (owner: 10Hashar) [17:01:52] <^d> mutante: Rebased those ^ [17:05:24] (03PS1) 10Petr Onderka: Added name and timestamp to dumps [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/80404 [17:20:14] (03PS2) 10Dr0ptp4kt: Instruct robots to not index Wikipedia Zero. No deploy before 25-June-2013. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69420 [17:21:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [17:29:56] (03CR) 10Dr0ptp4kt: [C: 031] "This thing is as ready to go as it's going to get, and I've confirmed with Dan and Tomasz as much. Mark, Faidon, Asher: would you please a" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69420 (owner: 10Dr0ptp4kt) [17:31:06] hah, june [17:31:19] someone of the staff around? [17:31:26] yes [17:31:35] almost always :) [17:31:44] especially during the day on a non-holiday weekday [17:31:51] on nl wiki we don't have don't have auto https yet [17:31:56] hint: you'll find them faster if you say what you need [17:32:07] well it's 8.30 pm here ;) [17:32:19] Larsnl: http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071395.html [17:33:08] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [17:33:10] oh that message hasn't been posted in our pub, but thanks [17:33:38] (03PS2) 10Petr Onderka: Added name and timestamp to dumps [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/80404 [17:34:57] shouldn't this (http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071395.html) notice have been published in our village pump [17:35:29] Larsnl: on which village pump? [17:35:39] any, maybe [17:35:42] Ryan_Lane: the one from nl wiki [17:35:55] nl.wikipedia.org/wiki/Wikipedia:De_kroeg [17:35:56] the problem with village pumps is that there's about 89247293072 of them [17:36:02] ish [17:36:08] +/- 10 [17:36:41] greg-g did post the previous anouncement in hte pub via global delivery system [17:36:45] ah [17:36:47] ok [17:37:00] I always forget about global delivery [17:37:08] we so badly need to fix our communication issues [17:37:11] yeah, the person who actually pushed the buttons on that said he didn't want to spam everyone again just yet [17:37:13] * Ryan_Lane wants flow to handle this [17:37:17] YES [17:37:21] gah, forgot about this channel. so many channels [17:37:29] then I could post something myself and get centralized feedback [17:37:30] cuz now we (#wikipedia-nl) were wondering why it hasn't been set on yet [17:37:30] how could you?! [17:37:54] this channel was the one in the message via GMG [17:37:58] GMS* [17:38:05] Larsnl: yeah, sorry, it was a last minute decision to postpone. We were really trying to get it out on Wed, but it just wasn't going to happen :/ [17:38:43] well it'll come there one day [17:38:49] next week [17:38:53] or is it the week after? [17:38:56] next [17:38:59] 28th [17:39:01] we're on every week deployment now, right [17:39:06] same bat day, same bat channel [17:39:12] could you meaby still post http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071395.html) in all the pubs [17:39:25] define all the pubs? [17:39:51] http://www.wikidata.org/wiki/Q16503#sitelinks-wikipedia [17:39:56] it'll go out in the Tech News this weekend, be a part of my Deployment update I send to wikitech-ambassadors... [17:40:10] oh ok i think that'll be enough [17:40:18] :) [17:40:24] romaine will probably post something about it then [17:40:28] thanks, good to have confirmation :) [17:40:43] "Wikipedia pages linked to this item(184 entries)" [17:40:47] eek [17:40:59] " the problem with village pumps is that there's about 89247293072 of them" [17:41:10] +/- [17:41:11] ;) [17:41:43] but good luck with rolling it out [17:42:03] thanks. hopefully everything will be tested and working well next week :) [17:42:14] he says, to no one in particular [17:42:23] indeed :) [17:43:53] hrmmm, I was hoping those patches from Tim would have been merged today, but Chris is taking his Wikimania day [17:45:50] !log reedy synchronized php-1.22wmf13/includes/api/ApiEditPage.php [17:45:55] Logged the message, Master [17:46:23] greg-g: it already went out to -ambassadors [17:46:45] idk why i didn't think about the postponement when he first said nlwiki... [17:46:48] jeremyb: not my deploy highlights, but yes, robla's message already did [17:46:55] right [17:46:57] :) [17:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [18:01:17] heeaaaay paravoid, any better? [18:01:18] https://gerrit.wikimedia.org/r/#/c/79927/ [18:03:22] (03PS3) 10Petr Onderka: Added name and timestamp to dumps [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/80404 [18:08:19] (03CR) 10Petr Onderka: [C: 032 V: 032] Added name and timestamp to dumps [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/80404 (owner: 10Petr Onderka) [18:22:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.170 second response time [18:35:25] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 1 wikis to 1.22wmf14 [18:35:30] Logged the message, Master [18:37:06] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase 3 wikis to 1.22wmf13 [18:37:11] Logged the message, Master [18:37:38] (03PS1) 10Reedy: Phase 1 wikis to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80418 [18:37:39] (03PS1) 10Reedy: Phase 3 wikis to 1.22wmf13 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80419 [18:48:39] (03CR) 10Reedy: [C: 032] Phase 1 wikis to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80418 (owner: 10Reedy) [18:48:44] (03CR) 10Reedy: [C: 032] Phase 3 wikis to 1.22wmf13 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80419 (owner: 10Reedy) [18:49:40] (03Merged) 10jenkins-bot: Phase 1 wikis to 1.22wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80418 (owner: 10Reedy) [18:49:55] (03Merged) 10jenkins-bot: Phase 3 wikis to 1.22wmf13 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80419 (owner: 10Reedy) [18:55:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:13:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:20]