[00:08:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:07:54 UTC 2013 [00:08:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:08:54 UTC 2013 [00:09:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:09:50 UTC 2013 [00:10:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:42] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:10:37 UTC 2013 [00:11:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:11:54 UTC 2013 [00:12:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 00:14:45 UTC 2013 [00:15:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:12] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:12] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:09] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:39] PROBLEM - Apache HTTP on mw1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:15] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.0003364086151 secs [02:01:33] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001299381256 secs [02:05:05] !log LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013 [02:05:14] Logged the message, Master [02:06:59] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:08] Bah! Why is the mistress of networks, like, not working when she's not at work? [02:12:35] heh [02:12:50] Her touch is all that's left for me to unleash DB replication. :-) [02:13:29] Even if I dared touch her networking gear, I speak fluent Cisco -- Juniper not so much. [02:13:42] !log LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013 [02:13:51] Logged the message, Master [02:13:59] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:25] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013 [02:29:34] Logged the message, Master [02:56:45] LeslieCarr: https://rt.wikimedia.org/Ticket/Display.html?id=5183 when you get a chance [02:57:44] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:04] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 04:08:03 UTC 2013 [04:08:24] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 04:09:11 UTC 2013 [04:09:24] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:36] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 04:10:14 UTC 2013 [04:10:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:31:36] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [04:36:36] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [04:37:56] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.62 ms [05:38:41] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:43:41] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [06:04:11] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:04:11] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:04:11] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:20:11] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:29:10 UTC 2013 [06:29:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:30:30 UTC 2013 [06:31:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:31:45 UTC 2013 [06:32:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:32:51 UTC 2013 [06:33:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:33:52 UTC 2013 [06:34:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:34:48 UTC 2013 [06:35:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 06:35:33 UTC 2013 [06:35:46] New review: Nikerabbit; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64539 [06:36:30] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:38:17] odder got caught in the typical error of asking CR: adding a useless part people will immediately bikeshed on :D [07:02:20] RECOVERY - search indices - check lucene status page on search1007 is OK: HTTP OK: HTTP/1.1 200 OK - 351 bytes in 0.002 second response time [07:31:38] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.007562160492 secs [07:33:08] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.003257513046 secs [07:36:47] hello [07:36:56] apergos: good morning :-D [07:37:11] morning [07:39:44] today, in France, is the 4th holiday day of the month [07:39:50] the city is basically dead :-] [07:40:31] * hashar blames Easter [07:45:40] here it's pretty normal [08:08:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 08:08:09 UTC 2013 [08:08:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 08:09:03 UTC 2013 [08:09:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 08:09:51 UTC 2013 [08:10:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 08:11:07 UTC 2013 [08:11:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 08:15:45 UTC 2013 [08:15:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [10:16:21] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [10:16:21] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [10:38:59] New patchset: Odder; "(bug 48620) Enable Translate extension on Wikimedia Commons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64539 [10:39:42] New patchset: Odder; "(bug 48620) Enable Translate extension on Wikimedia Commons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64539 [10:41:58] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:30] Nemo_bis: very funny. [10:52:17] odder: you forgot ULSEnable false [10:52:29] as per bug comment [11:01:40] there's no such thing in IS or CS, Nemo_bis [11:04:22] so what [11:04:25] * Nemo_bis out [12:07:04] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:04] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:07:55 UTC 2013 [12:08:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:09:04 UTC 2013 [12:09:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:10:08 UTC 2013 [12:10:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:11:04 UTC 2013 [12:11:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:04] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:11:54 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Mon May 20 12:12:25 UTC 2013 [12:12:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:44] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:12:36 UTC 2013 [12:13:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 12:15:05 UTC 2013 [12:15:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:58:00] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [14:03:54] PROBLEM - Host analytics1010 is DOWN: PING CRITICAL - Packet loss = 100% [14:08:21] RECOVERY - Host analytics1010 is UP: PING OK - Packet loss = 0%, RTA = 1.24 ms [14:32:01] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [14:39:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [14:40:51] paravoid, do you have any opinions about/experience with stdeb? [14:49:03] andrewbogott (or other ops people): It looks like when l10nupdate was moved from fenari to tin earlier this week, the ssh keys weren't changed too so it can't actually sync the changes out. See errors in /var/log/l10nupdatelog/l10nupdate.log-20130520.gz (note I fixed the "failed to open stream" error already). Can someone help get that fixed? [14:52:05] anomie: Hm… I think that might be TimStarling's bag, although he's probably gone for the day. Can you open an RT ticket? [14:52:13] New review: Cmcmahon; "This change would be helpful for the next round of automated browser tests." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/62606 [14:52:47] andrewbogott: Sure, let me see if I remember how to do that [14:53:05] !log upgraded all linux machines to linux 3.2.0-43-generic kernel [14:53:07] oops [14:53:13] meant 'analytics machines' [14:53:13] Logged the message, Master [14:54:18] !log rebooting emery to upgrade to 3.2.0-43-generic kernel [14:54:28] Logged the message, Master [14:56:30] PROBLEM - Host emery is DOWN: PING CRITICAL - Packet loss = 100% [14:56:50] RECOVERY - Host emery is UP: PING WARNING - Packet loss = 58%, RTA = 26.66 ms [14:57:30] RECOVERY - udp2log log age for emery on emery is OK: OK: all log files active [14:58:58] andrewbogott: Ok, ticket is 5187. [15:01:39] New patchset: Cmjohnson; "decommissioning db26" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64580 [15:04:16] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64580 [15:10:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [15:28:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [15:41:33] New patchset: Jgreen; "remove pgehres unix user" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64583 [15:42:13] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64583 [15:44:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [15:46:49] @notify LeslieCarr [15:46:49] This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.) [15:48:44] !log rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel [15:48:52] Logged the message, Master [15:49:18] New patchset: Jgreen; "oops, miscommunication. pgehres account rises again like a phoenix." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64584 [15:49:28] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64584 [15:51:23] PROBLEM - Host gadolinium is DOWN: PING CRITICAL - Packet loss = 100% [15:52:13] RECOVERY - Host gadolinium is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [16:02:31] hiii notpeter! you around? [16:05:00] New patchset: Anomie; "Turn on wmgUseCodeEditorForCore in Beta Labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64587 [16:05:03] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:05:03] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:05:03] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:05:44] New review: Anomie; "Simple config change to Beta Labs, already discussed" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/64587 [16:05:54] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64587 [16:06:41] PROBLEM - NTP on gadolinium is CRITICAL: NTP CRITICAL: Offset unknown [16:07:38] ottomata: ^ [16:08:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:07:55 UTC 2013 [16:08:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:09:03 UTC 2013 [16:09:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:10:05 UTC 2013 [16:10:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:11:02 UTC 2013 [16:11:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:41] RECOVERY - NTP on gadolinium is OK: NTP OK: Offset 0.006812691689 secs [16:12:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:11:52 UTC 2013 [16:12:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:12:36 UTC 2013 [16:13:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:37] jeremyb: thanks, looks like it is ok now [16:14:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 20 16:14:48 UTC 2013 [16:15:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:31] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:17] hrmmm, apergos uses gerrit very similarly to how i do. e.g. without git-review [16:39:56] So does Chad among others [16:40:01] I do for some setups [16:40:47] apergos: you left out `git log`. i also tend to use git log -p --decorate --stat --pretty=fuller [16:41:27] New review: ArielGlenn; "so, git commit --amend would have let you fix up your previous patch. this one that you have submit..." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/64095 [16:42:18] I ue git log a lot but [16:42:33] my post was starting to get as long as the original's [16:42:53] so I dithered: put it in? leave it out? in the end I cut it [16:43:23] haha [16:43:35] yeah, that guy is kinda verbose [16:43:48] --quiet [16:51:10] !log shutting down lanthanum to add ssd rt5074 [16:51:16] !log rebooting neon for upgrade [16:51:18] Logged the message, Master [16:51:26] Logged the message, Master [16:54:32] interesting, i reboot and one bot leaves but another shows up [16:58:14] :D [16:59:09] !log ping logmsgbot [16:59:17] Logged the message, Master [16:59:43] RECOVERY - Host lanthanum is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [17:00:10] !log resurrecting icinga-wm and nsca [17:00:18] Logged the message, Master [17:14:19] PROBLEM - NTP on lanthanum is CRITICAL: NTP CRITICAL: Offset unknown [17:18:20] RECOVERY - NTP on lanthanum is OK: NTP OK: Offset -0.002712607384 secs [17:21:53] hiya paravoid! [17:22:05] marktraceur: Ping? [17:22:16] akosiaris has +1ed the puppet hadoop cdh4 stuff. wondering what's next [17:22:46] do you want to review? or would you be comfortable with us getting someone else to review and +1 before we merge? [17:22:47] Coren: Pong [17:23:30] marktraceur: You're the substitute mistress of networks, right? [17:23:35] :-) [17:23:43] (brb) [17:24:52] Coren: I'm...not sure what you mean right now [17:25:29] the ambiguity of English is just lovely. [17:25:30] marktraceur: That'd be because you're the /wrong/ mark. :-) Sorry to have pinged you needlessly. I should learn to stop trusting tab-comletion. :-) [17:26:20] Yuuuup [17:30:36] Reedy: whenever enwiki gets updated, we have update for wikibase https://gerrit.wikimedia.org/r/#/c/64581/ [17:32:03] (back) [17:33:56] New patchset: Catrope; "Enable TemplateData on all wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64595 [17:34:21] Reedy: Would you mind if that ----^^ piggybacked on your wmf5 deploy? [17:34:26] (Or wmf6? I lost count) [17:34:48] !log kernel upgrade cp1043 [17:34:49] You mean from next week? [17:34:50] Or today? [17:34:56] Logged the message, Master [17:34:57] wmf4 [17:35:30] Reedy: Today [17:35:43] Note it's just onto enwiki extra today [17:35:44] Like, whatever it is you're deploying in 25 minutes. That. [17:35:50] Right, that's fine [17:36:00] Other wikis ok with what's in wmf3? [17:36:04] Yes [17:36:13] That extension has been in wmfN for a while [17:36:19] PROBLEM - Host cp1043 is DOWN: CRITICAL - Host Unreachable (208.80.154.53) [17:36:24] We just want it enabled everywhere [17:36:44] And we were too lazy to schedule our own window for it (and it was difficult, as I'm in Europe and your window ate most of the available time) [17:39:10] Reedy , should I +2 E3's two backports for wmf4? greg-g approved https://gerrit.wikimedia.org/r/#/c/64250/ and https://gerrit.wikimedia.org/r/#/c/64254/ [17:39:26] RECOVERY - Host cp1043 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [17:43:58] Sure [17:47:43] !log upgrading/rebooting mobile varnish cache cp1041-cp1044 [17:47:51] Logged the message, Master [17:48:33] !log Reindexing solr [17:48:42] Logged the message, Master [17:49:09] Thanks Reedy, done. [17:52:36] Now that it's not early morning in SF, any ops people want to look at RT #5187? l10nupdate is broken on tin, it looks like it needs ssh keys fixed for the l10nupdate user. [17:55:02] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64595 [17:56:34] Fetching submodule extensions/ProofreadPage [17:56:34] error: The requested URL returned error: 403 while accessing https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ProofreadPage.git/info/refs [17:56:34] fatal: HTTP request failed [17:56:34] FFS [17:56:39] This is starting to piss me off [17:58:26] PROBLEM - Host cp1041 is DOWN: PING CRITICAL - Packet loss = 100% [17:59:08] Looks like I'm updating on fenari and copying across again [17:59:56] RECOVERY - Host cp1041 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [18:00:15] binasher: ping [18:00:23] jdlrobson: ping [18:00:50] * jdlrobson waves [18:03:10] jdlrobson: may I send you a PM? [18:04:26] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4 [18:04:35] Logged the message, Master [18:05:59] PROBLEM - Apache HTTP on mw1023 is CRITICAL: HTTP CRITICAL - No data received from host [18:06:22] New patchset: Reedy; "enwiki to 1.22wmf4" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64600 [18:06:22] New patchset: Hashar; "lanthanum as a jenkins slave" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64601 [18:06:35] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64600 [18:07:59] !log reedy synchronized wmf-config/ 'Enable TemplateData on all wikis' [18:08:07] Logged the message, Master [18:11:47] New patchset: Hashar; "lanthanum as a jenkins slave" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64601 [18:13:38] !log reedy synchronized php-1.22wmf4/extensions/ProofreadPage/ [18:13:47] Logged the message, Master [18:14:58] !log reedy synchronized php-1.22wmf4/extensions/Wikibase/ [18:15:07] Logged the message, Master [18:15:38] New patchset: Dzahn; " require admins::l10nupdate in misc::deployment::l10nupdate for RT-5187" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64603 [18:16:59] !change 64603 | anomie [18:17:00] anomie: https://gerrit.wikimedia.org/r/#q,64603,n,z [18:20:42] !log reedy Started syncing Wikimedia installation... : Rebuild message cache [18:20:50] Logged the message, Master [18:22:20] mutante: Thanks. Will that work, or is everything only trusting l10nupdate@fenari and not l10nupdate@tin? [18:22:34] mutante: Also, it looks like Jenkins -1ed it [18:23:55] that part is just a name of a key, but true at syntax error [18:24:09] New patchset: Reedy; "Remove wgArticleRobotPolicies" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64526 [18:24:30] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64526 [18:25:28] New patchset: Reedy; "Enable VisualEditor on all content namespaces for MW.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63621 [18:25:30] New patchset: Dzahn; " require admins::l10nupdate in misc::deployment::l10nupdate for RT-5187" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64603 [18:25:50] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63621 [18:26:59] New review: Reedy; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/63702 [18:29:13] anomie: ps2 [18:29:18] !log reedy Finished syncing Wikimedia installation... : Rebuild message cache [18:29:26] Logged the message, Master [18:29:40] anomie: just trying to quick fix, doesnt mean i know much about deployment.pp. heh [18:29:53] but that would be included on tin ,yep yep [18:30:31] New patchset: Reedy; "(bug 47749) Categorise {{#babel}} on udmwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64467 [18:30:40] mutante: Well, I can test it easily enough once it's deployed. [18:30:53] 9 minutes to scap [18:30:54] nice [18:31:09] PROBLEM - Host cp1042 is DOWN: PING CRITICAL - Packet loss = 100% [18:31:49] New review: Dzahn; "l10nupdate user on tin as on fenari" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/64603 [18:31:49] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64603 [18:32:19] RECOVERY - Host cp1042 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [18:32:36] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64467 [18:32:36] anomie: ohh.. but .. it's already there.. wth [18:32:39] New patchset: Reedy; "(bug 48578) Enable LQT for all namespaces on ptwikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64460 [18:32:48] anomie: root@tin:/home/l10nupdate/.ssh [18:33:00] it already has the key... ehmm... [18:34:13] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64460 [18:34:15] New patchset: Reedy; "(bug 47574) Change namespace settings for cewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64501 [18:34:54] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64501 [18:35:18] New review: Dzahn; "this already existed on tin (!?)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64603 [18:36:34] New patchset: Reedy; "(bug 48308) Change namespace settings for ukwikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64336 [18:36:59] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64336 [18:38:41] !log reedy synchronized wmf-config/InitialiseSettings.php [18:38:45] New patchset: Reedy; "(bug 48620) Enable Translate extension on Wikimedia Commons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64539 [18:38:50] Logged the message, Master [18:39:38] mutante: For testing, executing "sudo -u l10nupdate dsh -o -oPasswordAuthentication=no -F 30 -cM -m terbium true" seems to illustrate the problem well enough. Works on fenari, "Permission denied (publickey)" on tin. [18:41:41] anomie: ohh.. but that looks like you want that user on terbium [18:41:50] ok [18:42:45] mutante: Well, the actual script (/usr/local/bin/sync-l10nupdate-1) tries to dsh for every host in the mediawiki-installation group [18:43:00] I just picked one to make a simpler test case [18:43:38] !log krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched' [18:43:46] Logged the message, Master [18:44:34] New patchset: Dzahn; "of course this is accounts:: not admins::, just in admins.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64611 [18:45:25] !log Created translate tables on commonswiki [18:45:33] Logged the message, Master [18:45:57] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64611 [18:46:10] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64539 [18:50:12] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on commonswiki' [18:50:22] Logged the message, Master [18:50:55] New patchset: Dzahn; "revert requiring l10nupdate account in misc::deployment::l10nupdate, causes duplicate definition (even though it still seems like it would have made sense to be required where it's actually used) (RT-5187)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64612 [18:53:33] New patchset: Reedy; "Move VHosts config from wgConf to seperate files" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64229 [18:54:48] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64229 [18:59:17] !log reedy synchronized wmf-config/wgConfVHosts.php [18:59:26] Logged the message, Master [19:00:31] !log reedy synchronized wmf-config/wgConf.php [19:00:39] Logged the message, Master [19:03:46] New patchset: Andrew Bogott; "Give analytics access to Yurik." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64616 [19:04:10] !log reedy synchronized w/ [19:04:20] Logged the message, Master [19:04:59] New patchset: Reedy; "SECURITY: fix URI escaping when displaying 404" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64617 [19:05:48] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64617 [19:07:41] !log csteipp synchronized php-1.22wmf3/includes 'Security fix' [19:07:49] Logged the message, Master [19:08:02] PROBLEM - Host cp1044 is DOWN: CRITICAL - Host Unreachable (208.80.154.54) [19:08:32] !log csteipp synchronized php-1.22wmf4/includes/ 'Security fix' [19:08:41] Logged the message, Master [19:08:50] New review: Dzahn; "revert change 64603" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/64612 [19:08:51] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64612 [19:10:34] RECOVERY - Host cp1044 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [19:12:04] !log dist-upgrading cp1(sq51) [19:12:13] Logged the message, Master [19:13:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [19:30:14] PROBLEM - check_apache2 on payments2 is CRITICAL: PROCS CRITICAL: 0 processes with command name apache2 [19:30:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:24] payments hiccups are me--doing some apt updates and reboots [19:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [19:35:14] RECOVERY - check_apache2 on payments2 is OK: PROCS OK: 7 processes with command name apache2 [19:45:52] Change abandoned: Reedy; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60434 [19:47:25] Reedy: that was a lot of e-mail that I just got :) [19:47:42] You'll get less if I just abandon them all ;) [19:48:18] I was just wondering–how long does it generally take for updates in extensions to land on-line/ [19:48:40] There's this guy from ukwikisource bugging me about https://gerrit.wikimedia.org/r/#/c/64331/ :-) [19:50:49] New patchset: Andrew Bogott; "Give analytics access to Yurik." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64616 [19:51:06] +2 anyone? ^ [19:51:43] odder: We can just update it [19:52:01] Only thing atm is it that submodules are fscked on tin [19:52:01] That'd be just awesome. [19:52:11] so i've got to do checkouts on fenari and stuff [19:52:28] New review: coren; "+yurik" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/64616 [19:52:29] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64616 [19:52:31] PROBLEM - Host payments4 is DOWN: PING CRITICAL - Packet loss = 100% [19:53:21] andrewbogott: Merged and pushed. [19:53:27] thanks [19:56:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:41] RECOVERY - Host payments4 is UP: PING OK - Packet loss = 0%, RTA = 26.70 ms [20:00:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time