[00:03:20] New patchset: Reedy; "Securepoll only enabled on enwiki and metawiki..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47305 [00:03:52] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47305 [00:10:04] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [00:10:06] New patchset: Reedy; "Bug 28276 - Inconsistently named IRC recent changes channels" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [00:11:57] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [00:14:16] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [00:16:21] New patchset: Reedy; "Bug 44315 - tel: protocol should be enabled in $wgUrlProtocols for Wikivoyage sites" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47308 [00:17:51] New review: Reedy; "Doesn't currently work as intended" [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/47308 [00:20:59] New patchset: Reedy; "Bug 44315 - tel: protocol should be enabled in $wgUrlProtocols for Wikivoyage sites" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47308 [00:21:21] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47308 [00:22:13] !log reedy synchronized wmf-config/InitialiseSettings.php [00:22:14] Logged the message, Master [00:36:00] New patchset: Alex Monk; "Bug 28276 - Inconsistently named IRC recent changes channels" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [00:47:29] New patchset: Reedy; "Bug 43185 - Set $wgCategoryCollation to 'uca-default' and rebuild category sort keys on Portuguese Wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47310 [00:47:48] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47310 [00:48:02] !log reedy synchronized wmf-config/InitialiseSettings.php [00:48:03] Logged the message, Master [00:52:27] New review: Krinkle; "Why do we need to repeat this here, aren't the canonical host names known some way?" [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/47307 [00:53:11] New review: Krinkle; "Also, after they are confirmed and agreed on. Need wide announcements to cvn infrastructure and bot ..." [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/47307 [01:10:11] New patchset: Reedy; "Refactor setting of $wgCentralAuthAutoLoginWikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47312 [01:10:49] New review: Reedy; "The notice was the reason I didn't submit and deploy it immediately" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/47307 [01:11:36] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47312 [01:14:14] Krinkle: You should know we like to build hacks on hacks [01:14:15] !log reedy synchronized wmf-config/ [01:14:18] Logged the message, Master [01:14:18] Then apply duct tape [01:17:58] Reedy: Is the mapping of hostname to dbname only one-way (e.g. multiversions/wgConf hackery), or is there a mapping in the other direction as well [01:18:27] Because $lang.$site as default is just plain wrong, has been every since the first hostname outside wikipedia.org got used (wikimedia.org, mediawiki.org etc.) [01:18:55] with a $hostname var, the others can just go away [01:19:09] SiteMatrix does some weird stuff like that [01:19:10] I recall there being a docroot, but that may not be usable [01:19:15] Hm.. right [01:20:06] InitialiseSettings.php [01:20:06] 1091:# wgServer @{ [01:20:06] 1092:'wgServer' => array( [01:20:18] 1171:'wgCanonicalServer' => array( [01:20:30] wgServer and remove // and .org [01:21:39] Right [01:21:48] except that we can't embed those variables [01:22:35] wgConf needs to be able to generate the values for any given dbname [01:22:48] or does it support embedding? [01:22:55] $wgRC2UDPPrefix = str_replace( '\\', '', str_replace( '.org', '', $wgServer ) ); [01:23:01] :D [01:23:04] CommonSettings [01:23:06] RESOLVED WFM [01:23:08] That'd work [01:23:19] Well, + # and \t [01:23:20] but yeah [01:23:43] $wgRC2UDPPrefix = '#' . str_replace( '\\', '', str_replace( '.org', '', $wgServer ) ) . "\t"; [01:24:11] Maybe use a regex though to only trim 1 \\ from start and 1 .org from end [01:24:20] (^\\|.org&) [01:24:24] (^\\|.org$) [01:24:29] Is there much point? [01:24:44] nl.org.vicomania.eu? [01:24:48] o_0 [01:24:57] No, that doesn't exist yet. [01:24:59] But we don't have those [01:25:04] I know, right. [01:25:08] Imagine the consequences [01:25:59] Reedy: or... [01:26:28] Reedy: wmgHostname and have wgServer add .org and //. that would save duplication. We wouldn't change that per wiki anyway, right ? [01:26:52] Every other domain redirects in to the .org [01:26:58] so by that point it should be .org [01:27:15] yep [01:27:25] if it wouldn't then replacing .org wouldn't be enough [01:27:29] either way [01:28:26] anyway, once you're certain it can be done across the board (wgRC2UDPPrefix being always equal to the canonical hostname, sanse tld), announce it. And plenty of time to implement it as clean as can do [01:29:05] pick a date and deployment window < 31 days ahead from next monday and world peace shall naturally follow. [01:29:10] > * [01:29:52] hm.. something caused the Finder to crash. That's a first. [01:32:05] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [01:38:20] New patchset: Reedy; "Set wgCategoryCollation to uca-default on mediawikiwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47314 [01:38:34] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47314 [01:53:14] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [01:53:32] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [01:55:14] Anyone know where the sul images came from? [01:56:55] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [02:00:00] * Reedy makes one.. [02:01:44] New patchset: Reedy; "Login to brwikimedia when logging into any pt project" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47315 [02:02:12] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47315 [02:02:56] !log reedy synchronized images/sul/foundation.png [02:02:56] Logged the message, Master [02:03:42] !log reedy synchronized wmf-config/ [02:03:43] Logged the message, Master [02:08:04] New review: Reedy; "$wgRC2UDPPrefix = '#' . str_replace( '\\', '', str_replace( '.org', '', $wgServer ) ) . "\t";" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/47307 [02:08:24] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [02:09:24] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [02:09:25] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [02:09:25] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [02:09:25] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [02:09:25] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [02:14:24] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [02:18:34] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 189 seconds [02:18:35] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 186 seconds [02:18:54] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 194 seconds [02:19:38] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 202 seconds [02:28:29] !log LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013 [02:28:31] Logged the message, Master [02:37:47] New review: MZMcBride; "I thought Tim and others were opposed to further per-wiki custom thumbnail sizes." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47303 [02:43:55] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 3 seconds [02:44:05] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [02:44:35] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [02:45:09] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [02:51:45] New review: MZMcBride; "Krinkle: I agree that using the canonical hostname ($wgServer, as Reedy suggested) would be best here." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/47307 [03:05:43] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [03:56:05] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [04:12:37] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [04:13:00] New review: Krinkle; "For wikidata it would/should be #www.wikidata." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/47307 [04:16:28] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [04:21:58] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [04:24:08] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [04:30:59] New review: Krinkle; "@Reedy: https://bugzilla.wikimedia.org/show_bug.cgi?id=28276#c14" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/47307 [04:32:08] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [04:32:38] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 185 seconds [04:32:41] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [04:32:50] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 186 seconds [04:36:08] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [04:36:35] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 187 seconds [04:36:38] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 187 seconds [05:14:59] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 17 seconds [05:15:08] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [05:15:09] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [05:15:17] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [05:20:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [06:39:28] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 190 seconds [06:39:31] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 195 seconds [06:40:01] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 205 seconds [06:40:39] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 195 seconds [06:44:31] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [06:44:42] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 1 seconds [06:45:01] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [06:45:54] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [07:04:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [07:37:57] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 193 seconds [07:38:06] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 195 seconds [07:38:17] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 206 seconds [07:39:17] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [07:39:54] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [07:39:57] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [08:13:20] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [08:27:56] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 183 seconds [08:28:00] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 186 seconds [08:28:05] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 186 seconds [08:28:10] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 189 seconds [08:39:38] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [08:39:38] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [08:41:44] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [08:41:45] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [08:41:45] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [08:41:45] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [08:50:01] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [08:50:10] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 1 seconds [08:50:26] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [08:50:36] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [09:07:10] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [09:13:23] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Sun Feb 3 09:13:07 UTC 2013 [09:51:55] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [09:51:55] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [09:51:55] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [09:51:55] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [09:51:55] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [09:54:01] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [10:10:48] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [10:14:39] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [11:06:37] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 191 seconds [11:07:04] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 202 seconds [11:07:10] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 205 seconds [11:07:10] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 206 seconds [11:20:34] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [11:20:53] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [11:21:03] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 1 seconds [11:21:10] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [11:33:27] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [12:08:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 12:07:58 UTC 2013 [12:08:19] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Sun Feb 3 12:08:11 UTC 2013 [12:08:49] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [12:08:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [12:09:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 12:08:50 UTC 2013 [12:09:50] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [12:09:50] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [12:09:50] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [12:09:50] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [12:09:50] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [12:09:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [12:14:50] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [12:29:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 12:29:40 UTC 2013 [12:29:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [13:09:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [13:39:09] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 185 seconds [13:39:10] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 184 seconds [13:39:20] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 192 seconds [13:39:55] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 206 seconds [13:40:49] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 20 seconds [13:41:09] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [13:41:19] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [13:41:34] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [13:56:52] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [14:01:56] 45 [14:02:12] hmpf, bad hand holding coffee [14:13:35] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [14:18:48] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [14:22:07] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [14:24:55] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [14:42:02] New patchset: saper; "Dont add LF to Subject: in gerrit notifications" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47343 [16:02:55] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:07:44] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 16:07:42 UTC 2013 [16:07:56] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:08:14] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 16:08:06 UTC 2013 [16:08:55] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:09:05] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 16:08:57 UTC 2013 [16:09:54] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:17:04] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 16:17:00 UTC 2013 [16:17:54] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:45:14] New patchset: Reedy; "Bug 43812 - Create Sanskrit Wikiquote" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47346 [16:48:04] New patchset: Reedy; "Bug 44413 - Create Wikivoyage Romanian" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47347 [16:52:27] New patchset: Reedy; "Bug 44414 - Create Wikivoyage Polish" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47348 [16:53:20] New patchset: Reedy; "Bug 44413 - Create Wikivoyage Romanian" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47347 [16:56:06] New patchset: Reedy; "Bug 44460 - Create Wikiversity Korean" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47349 [16:58:33] New review: Leinad; "Missing:" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/47348 [16:58:54] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 200 seconds [16:59:01] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 207 seconds [16:59:21] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 216 seconds [16:59:30] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 222 seconds [16:59:31] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 202 seconds [16:59:41] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 213 seconds [17:00:04] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [17:00:06] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 227 seconds [17:00:15] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 234 seconds [17:00:51] New patchset: Reedy; "Bug 44462 - Create Wikipedia Minangkabau" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47350 [17:01:51] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.004 second response time on port 8123 [17:13:31] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 2 seconds [17:13:41] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 2 seconds [17:14:21] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 1 seconds [17:14:21] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 2 seconds [17:23:57] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Feb 3 17:23:54 UTC 2013 [17:24:22] New patchset: Alex Monk; "(bug 44572) Change wgUploadNavigationUrl on enwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47351 [17:24:33] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Sun Feb 3 17:24:18 UTC 2013 [17:25:46] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 186 seconds [17:26:01] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 189 seconds [17:26:21] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 183 seconds [17:26:30] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sun Feb 3 17:26:09 UTC 2013 [17:26:58] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 186 seconds [17:29:03] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sun Feb 3 17:28:46 UTC 2013 [17:29:57] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Sun Feb 3 17:29:31 UTC 2013 [17:32:51] New patchset: Reedy; "Bug 26402 - Labeled section transclusion installation" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47353 [17:38:40] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 186 seconds [17:38:47] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 187 seconds [17:39:00] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 188 seconds [17:39:05] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 188 seconds [17:39:14] New patchset: Reedy; "Bug 37148 - Remove Translate workflow state "updating" for metawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47355 [17:41:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47355 [17:42:03] !log reedy synchronized wmf-config/ [17:42:05] Logged the message, Master [17:47:14] New patchset: Reedy; "Bug 39652 - Add "autoreviewer" to $wgRestrictionLevels on ptwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47357 [17:47:39] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47357 [17:48:18] !log reedy synchronized wmf-config/InitialiseSettings.php [17:48:20] Logged the message, Master [17:49:01] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 3 seconds [17:49:17] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [17:49:35] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [17:49:40] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 1 seconds [17:58:42] New patchset: Alex Monk; "(bug 44048) Create patroller group on enwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47358 [18:05:57] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 187 seconds [18:06:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [18:06:02] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 190 seconds [18:06:12] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 195 seconds [18:06:50] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 208 seconds [18:11:25] New patchset: Alex Monk; "(bug 44412) Enable NewUserMessage extension on ptwiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47359 [18:40:57] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [19:02:09] New patchset: Alex Monk; "(bug 43524) Allow itwikivoyage sysops to upload locally" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47361 [19:07:21] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [19:53:14] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [19:53:14] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [19:53:14] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [19:53:14] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [19:53:14] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [19:55:10] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [19:57:08] New review: Hashar; "I have neeever heard of Velocity Template before. I guess we can test it on labs and then merge it i..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/47343 [20:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 20:07:52 UTC 2013 [20:08:09] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Feb 3 20:08:07 UTC 2013 [20:08:10] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Sun Feb 3 20:08:07 UTC 2013 [20:08:10] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sun Feb 3 20:08:08 UTC 2013 [20:08:20] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sun Feb 3 20:08:10 UTC 2013 [20:08:20] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:08:20] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Sun Feb 3 20:08:11 UTC 2013 [20:08:20] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 20:08:12 UTC 2013 [20:09:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:09:29] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sun Feb 3 20:09:20 UTC 2013 [20:10:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:10:48] New patchset: Hashar; "contint: enable qunit site" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47365 [20:15:19] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [20:17:51] New patchset: Alex Monk; "Only the main namespace should be searched by default on enwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47367 [20:18:46] New patchset: Alex Monk; "(bug 43465) Only the main namespace should be searched by default on enwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47367 [20:24:25] New patchset: Hashar; "contint: enable qunit site" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47365 [20:25:17] New review: Hashar; "This change forgot to enable the site, to make apache listen on port 9412 and to allow IPv6." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44164 [20:37:01] New patchset: Alex Monk; "(bug 44615) Add Index namespace to idwiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47369 [20:45:57] New review: Alex Monk; "Amire80, please remove your -1 from this change." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/29792 [20:57:58] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:58:59] !log jenkins : generating mwext-**-phpcs-HEAD jobs [20:59:01] Logged the message, Master [21:34:10] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [21:48:21] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 23 seconds [21:48:41] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [21:49:01] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [21:49:37] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [22:09:39] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [22:15:40] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [23:49:14] New patchset: Andrew Bogott; "Fix adminbot's postrm" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/47375 [23:49:39] Change merged: Andrew Bogott; [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/47375 [23:57:53] New patchset: Alex Monk; "(bug 43465) Only the main namespace should be searched by default on enwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47367 [23:58:10] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours