[00:02:03] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 7.222 second response time [00:04:13] * hoo looks for one of the admins::privatedata people (or a root) [00:12:03] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 207721 bytes in 8.493 second response time [00:18:21] (03PS1) 10Gergő Tisza: Add setting to show a survey for MediaViewer users on some sites [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124036 [00:34:03] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [01:09:34] greg-g: you here? [01:43:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:43:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:43:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:43:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:14:07] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-05 02:14:07+00:00 [02:14:14] Logged the message, Master [02:34:54] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-05 02:34:54+00:00 [02:34:59] Logged the message, Master [02:48:23] bd808|BUFFER, greg-g: wmf21 is broken again on MW.org, possibly by "LocalisationUpdate completed (1.23wmf21) at 2014-04-05 02:34:54+00:00". :-( [03:08:36] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 03:08:33 UTC 2014 (duration 8m 32s) [03:08:41] Logged the message, Master [03:13:47] Hi. I thought you were not going to deploy 1.23wmf21 until the l10n bug was fixed. [03:16:01] https://www.mediawiki.org/wiki/Thread:Project:Support_desk/Missing_interface_messages [03:16:08] ori: [03:16:11] werdna: [03:16:23] ^d: [03:16:26] ? [03:17:54] huh: ? [03:18:13] See MW.org interface, werdna. All the messages are missing. [03:18:21] again? [03:18:22] Or at least a large portion of them. [03:18:22] sigh [03:18:27] Yes, see the thread I linked [03:18:42] You said "Chad fixed this by reverting to 1.23wmf20. Waiting on the localisation team to fix it." [03:18:56] But 1.23wmf21 was redeployed. Why? [03:19:19] hey awjr [03:19:25] ori: poke [03:19:28] ^d: poke [03:20:18] heya werdna [03:20:19] what's up? [03:20:27] awjr: see mediawiki.org [03:20:30] https://www.mediawiki.org/wiki/Thread:Project:Support_desk/Missing_interface_messages [03:21:02] 1.23wmf21 (which has some l10n/i18n bug causing most or all messages not to display) was redeployed [03:21:15] awjr: I'm going to revert back to 1.23wmf20 [03:21:20] eh, im in an airplane; wifi is flaky [03:22:01] werdna: did nobody really notice this? o_O [03:22:19] !log Going to revert deployment of 1.23wmf21 again - still broken [03:22:25] Logged the message, junior [03:22:45] o_O [03:22:57] werdna: localization update didn't fix it? [03:23:32] http://puu.sh/7Wq7d.png [03:23:42] 2 mins ago [03:23:57] !log Actually, going to rerun l10nupdate first just to check. [03:24:02] Revi: they know [03:24:02] Logged the message, junior [03:24:08] I reported it :P [03:24:08] lol ok [03:25:07] werdna: i would think that should fix it; otherwise revert seems sensible [03:26:01] Warning: LU_Updater::readMessages: Unable to parse messages from /var/lib/l10nupdate/mediawiki/extensions/WikimediaIncubator/WikimediaIncubator.i18n.php in /a/common/php-1.23wmf20/extensions/LocalisationUpdate/Updater.php on line 63 [03:26:01] PHP Warning: LU_Updater::readMessages: Unable to parse messages from /var/lib/l10nupdate/mediawiki/extensions/WikimediaIncubator/InfoPage.i18n.php in /a/common/php-1.23wmf20/extensions/LocalisationUpdate/Updater.php on line 63 [03:26:08] looking suspiciously like it won't help [03:26:10] but we'll see [03:27:12] werdna: [03:33:16] keep in mind that most of these errors / warnings were also logged for the 1.20 update, which did succeed, as far as we know [03:27:24] yeah, I saw that yesterday [03:27:35] after you posted the same thing [03:28:35] Roan had an explanation on email, I should read that [03:29:01] huh: bd808|BUFFER and RoanKattouw_away had a big problem diagnosing session today in here, you can look at the logs if you want, they were comfortable with putting us back to where we are... apparently there are demons :/ [03:29:57] werdna: if you have backscroll, look around 15:12 Eastern in here [03:30:14] greg-g: ah you're here [03:30:18] I'm running l10nupdate [03:30:20] cool [03:30:28] and then I was going to run scap, and then l10nupdate again to see if it works [03:30:42] if it doesn't work, revert the revert and get us back to only wmf21 on testwiki :) [03:30:45] 15:12 eastern is when normal time? around 20:12 I suppose [03:30:50] yeah [03:31:33] (to be explicit: no sense in debugging right now on mw.org if the l10nupdate fails the first time, let's just get mw.org in a good state asap) [03:32:17] greg-g: wait, Roan put it the way it is now ? [03:32:46] and bd808 [03:32:57] * huh looks at logs [03:32:57] * greg-g nods [03:35:23] greg-g: nod. Where's mwversions.dat btw [03:35:28] * werdna is rusty [03:36:04] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-05 03:36:04+00:00 [03:36:09] Logged the message, Master [03:36:30] okay let's see [03:36:40] nope, still broken [03:36:59] oh wait, it's not done for wmf21 [03:37:01] * werdna holds horses [03:37:58] :) [03:38:09] werdna: btw, you just need to revert this: https://gerrit.wikimedia.org/r/#/c/124010/ [03:38:12] reverting the revert :) [03:38:52] greg-g: kay [03:39:08] werdna: btw, this is what Niklas said fixed it last night: "This seems to be confirmed by the fact that running "mw-update-l10n", "sync-l10nupdate" and "sync-l10nupdate-1 1.23wmf21" fixed test.wikipedia.org. " [03:39:28] ie: running the l10n parts of scap manaully, basically [03:40:04] ah, I could do that but I figure it would just recur [03:40:11] yeah, no worries anymore [03:41:36] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-05 03:41:36+00:00 [03:41:41] Logged the message, Master [03:41:55] still broken [03:42:24] Will you just revert to 1.23wmf20? [03:42:49] yep [03:43:00] thanks werdna [03:43:09] no worries [03:43:13] still 14:43 on Saturday in Sydney [03:43:18] not a terrible hour like it is wherever you are :p [03:43:33] taking forever to check out the mediawiki-config repo though :) [03:43:50] just 8:43 :) [03:44:10] huh: btw https://wikitech.wikimedia.org/wiki/Incident_documentation/20140403-Deploy [03:44:11] Thank you werdna and greg-g [03:44:24] Thanks for the link [03:44:27] np [03:44:29] just updated it [03:44:30] :) [03:48:40] It's strange, but I find the lack of keyboard shortcuts to be one of the most annoying things about this ;) [03:50:40] waiting for various ssh processes… :p [03:50:58] (03PS1) 10Werdna: mw.org, test2.wp, test.wikidata back to 1.23wmf20 again. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124046 [03:51:01] !log Reverting mw.org, test2 and test.wikidata back to 1.23wmf20 [03:51:06] Logged the message, junior [03:51:30] (03CR) 10Werdna: [C: 032] mw.org, test2.wp, test.wikidata back to 1.23wmf20 again. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124046 (owner: 10Werdna) [03:51:37] (03Merged) 10jenkins-bot: mw.org, test2.wp, test.wikidata back to 1.23wmf20 again. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124046 (owner: 10Werdna) [03:51:53] werdna: testwiki is staying 1.23wmf21 for debugging then? [03:51:58] si [03:52:08] ok [03:56:21] werdna: it's not syncing out? [03:56:30] !log andrew rebuilt wikiversions.cdb and synchronized wikiversions files: Revert mw.org, test2wiki and testwikidatawiki to 1.23wmf20 due to localisation issue [03:56:31] greg-g: I only just hit sync [03:56:46] ok fixed [03:56:57] as a bonus, enwiki is also up [03:57:49] :) [03:58:00] Thanks. [03:58:03] wee, I owe you a beer, werdna :) [03:58:38] greg-g: I didn't do very much, but beers are always welcome [03:58:49] * werdna actually just finished a beer (don't worry, first one of the day :p) [03:58:54] haha [03:58:56] thanks [03:59:10] fixed now :P [04:15:03] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 5 04:15:00 UTC 2014 (duration 50m 39s) [04:15:08] Logged the message, Master [04:44:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:44:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:44:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:44:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:46:53] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:48:53] PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:49:03] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:50:33] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:50:43] RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [04:51:53] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 884355 seconds since restart [04:52:23] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:52:53] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [05:28:31] Another Mikemikev sock - https://en.wikipedia.org/w/index.php?title=Special:Contributions&target=PlasticSpatula5 [05:28:43] Doug_Weller: I'm sure you mean -spi [05:29:00] oops [05:29:06] yes, sorry [05:29:08] bye [06:20:48] (03PS2) 10Hashar: Tools Labs: puppet lint manifests [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [07:45:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [07:45:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [07:45:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [07:45:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:57:14] wtf, test.wikidata is on wmf20 again? :( [10:26:37] (03PS1) 10Hashar: beta: adjust protoproxy for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/124057 [10:46:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [10:46:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [10:46:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [10:46:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:24:14] (03CR) 10Hashar: "Cherry-picked on deployment-salt.eqiad.wmflabs beta cluster puppet master. I have applied the class on deployment-cache-bits01.eqiad.wmfla" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124057 (owner: 10Hashar) [13:47:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [13:47:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [13:47:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [13:47:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [13:51:25] 1970 eh [13:52:36] <_joe|away> yes, I guess what version of ruby are they running [16:03:33] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [16:07:51] springle: Hi! Replication of s1 and s4 to Labs seems to have stopped. [16:48:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:48:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:48:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:48:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:52:28] (03CR) 10Matanya: [C: 031] beta: adjust protoproxy for eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/124057 (owner: 10Hashar) [18:09:09] (03CR) 10Matanya: [C: 031] replace hume with terbium in a comment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123853 (owner: 10Dzahn) [18:52:22] (03CR) 10Ori.livneh: [C: 032] Ensure that status is always defined in deploy.checkout [operations/puppet] - 10https://gerrit.wikimedia.org/r/119232 (owner: 10BryanDavis) [18:52:29] (03PS4) 10Ori.livneh: Ensure that status is always defined in deploy.checkout [operations/puppet] - 10https://gerrit.wikimedia.org/r/119232 (owner: 10BryanDavis) [18:52:35] (03CR) 10Ori.livneh: [C: 032 V: 032] Ensure that status is always defined in deploy.checkout [operations/puppet] - 10https://gerrit.wikimedia.org/r/119232 (owner: 10BryanDavis) [18:53:19] (03PS5) 10Ori.livneh: Support eqiad labs secondary disk [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [19:25:33] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [19:49:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [19:49:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [19:49:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [19:49:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [21:12:07] (03CR) 10coren: [C: 04-1] "Adding configurable mount options to labs_lvm::volume is arguably very useful, but to create and mount the filesystem as xfs you need to c" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [21:13:09] (03CR) 10coren: [C: 031] "Oh, duh, misread the patch. All is well. :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [22:08:06] Coren|Travel: :-] [22:50:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [22:50:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [22:50:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [22:50:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC