[00:33:41] (03PS1) 10Jforrester: SpecialCite is now CiteThisPage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 [00:33:48] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 27 00:33:41 UTC 2014 [00:49:48] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [01:02:03] (03PS2) 10Jforrester: SpecialCite is now CiteThisPage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 [01:02:52] (03CR) 10Jforrester: "PS2 un-removes the import of Cite/SpecialCite.php as it will need to not go out at once." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 (owner: 10Jforrester) [01:08:48] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [01:22:09] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [01:40:08] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:14:47] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-27 02:13:44+00:00 [02:14:56] Logged the message, Master [02:25:13] !log LocalisationUpdate completed (1.24wmf15) at 2014-07-27 02:24:10+00:00 [02:25:20] Logged the message, Master [02:33:48] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 00:33:41 UTC [02:34:08] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 27 02:34:07 UTC 2014 [02:40:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:42:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:44:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:46:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:48:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:50:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:52:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:54:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:56:38] PROBLEM - Puppet freshness on mw1169 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:38:29 UTC [02:58:18] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Sun Jul 27 02:58:11 UTC 2014 [02:59:21] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 27 02:58:15 UTC 2014 (duration 58m 14s) [02:59:27] Logged the message, Master [04:17:29] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 02:17:06 UTC [04:37:09] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jul 27 04:37:02 UTC 2014 [05:16:51] (03PS16) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [05:24:28] !log mysqldump s6 dbstore1002 to dbstore1001 [05:24:33] Logged the message, Master [05:29:15] (03PS17) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [06:28:59] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:09] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:09] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:39] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:39] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:40] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:40] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:49] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:39] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:09] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:09] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:55:09] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 3 failures [07:13:09] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:57:29] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 06:57:07 UTC [09:07:19] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 4 below the confidence bounds [09:17:49] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jul 27 09:17:40 UTC 2014 [10:01:19] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 4 below the confidence bounds [10:26:19] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [11:06:49] PROBLEM - Disk space on db1017 is CRITICAL: DISK CRITICAL - free space: / 285 MB (3% inode=81%): [11:40:29] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 09:39:51 UTC [12:13:29] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 10:12:50 UTC [12:20:29] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Jul 27 12:20:24 UTC 2014 [12:44:34] (03PS1) 10Chmarkine: Wikitech -- use "Header set" instead of "Header append" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149626 [13:33:29] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 27 13:33:25 UTC 2014 [16:19:46] (03PS1) 10Withoutaname: Add "eliminator" group to viwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149637 (https://bugzilla.wikimedia.org/68612) [16:24:59] eliminator ... :P [16:49:25] (03CR) 10Vogone: [C: 04-1] "'suppressrevision' won't work without at least 'deletedhistory'. 'deletedtext' may also make sense in case 'eliminators' should be allowed" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149637 (https://bugzilla.wikimedia.org/68612) (owner: 10Withoutaname) [16:50:09] (03CR) 10Vogone: "Sorry, I obviously meant 'deleterevision', not 'suppressrevision'." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149637 (https://bugzilla.wikimedia.org/68612) (owner: 10Withoutaname) [17:40:29] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 15:40:15 UTC [17:41:27] (03PS1) 10Reedy: Add export-0.9.xsd [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149643 (https://bugzilla.wikimedia.org/68686) [17:43:58] (03PS1) 10Reedy: Close body and html tags [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149644 [19:33:29] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 27 Jul 2014 17:32:41 UTC [19:40:29] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Jul 27 19:40:26 UTC 2014 [20:42:46] (03PS1) 10Ori.livneh: wmflib: add ordered_yaml() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149775 [20:53:29] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 27 20:53:25 UTC 2014 [21:03:44] (03PS1) 10Ori.livneh: wmflib: add ensure_service() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149778 [21:36:09] (03CR) 10JanZerebecki: [C: 031] Wikitech -- use "Header set" instead of "Header append" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149626 (owner: 10Chmarkine) [22:29:29] (03CR) 10Withoutaname: "Shouldn't undelete already convey that? Or is it a problem with core that eventually we'll have to fix later?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149637 (https://bugzilla.wikimedia.org/68612) (owner: 10Withoutaname) [22:57:05] (03PS1) 10Legoktm: Add WikimediaShopLink's SkinBuildSidebar hook directly [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149787 (https://bugzilla.wikimedia.org/55678) [22:58:47] (03CR) 10Legoktm: "Shouldn't be deployed until I7bfca514145828ff32feb4b6eb7c2031edefe364 is live on enwp." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149787 (https://bugzilla.wikimedia.org/55678) (owner: 10Legoktm)