[00:00:13] (03PS1) 10Dzahn: fix syntax for elasticsearch group [operations/puppet] - 10https://gerrit.wikimedia.org/r/135150 [00:00:48] (03PS2) 10Dzahn: fix syntax for elasticsearch group [operations/puppet] - 10https://gerrit.wikimedia.org/r/135150 [00:02:48] (03CR) 10Dzahn: [C: 032] fix syntax for elasticsearch group [operations/puppet] - 10https://gerrit.wikimedia.org/r/135150 (owner: 10Dzahn) [00:06:23] PROBLEM - Puppet freshness on fenari is CRITICAL: Last successful Puppet run was Fri 23 May 2014 09:05:25 PM UTC [00:10:47] (03PS1) 10Dzahn: remove duplicate awjrichards from fenari [operations/puppet] - 10https://gerrit.wikimedia.org/r/135151 [00:11:52] (03PS2) 10Dzahn: remove duplicate awjrichards from fenari [operations/puppet] - 10https://gerrit.wikimedia.org/r/135151 [00:12:37] (03CR) 10Dzahn: [C: 032] "he's stil here. see modules/admin/data/data.yaml: awjrichards:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135151 (owner: 10Dzahn) [00:21:03] (03PS1) 10Dzahn: Revert "fenari to admins yaml" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135152 [00:22:08] (03CR) 10Dzahn: [C: 032] "Duplicate definition: Group[wikidev] due to maintenance classes. gotta fix this after the weekend." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135152 (owner: 10Dzahn) [00:22:23] (03PS2) 10Dzahn: Revert "fenari to admins yaml" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135152 [00:24:32] (03CR) 10Dzahn: [C: 032] Revert "fenari to admins yaml" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135152 (owner: 10Dzahn) [00:27:03] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Sat May 24 00:26:55 UTC 2014 [00:29:02] AaronSchulz: did you recently get your UID fixed? [00:29:08] by andrewbogott_afk maybe [00:33:53] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat May 24 00:33:44 UTC 2014 [00:34:15] !log fixing Aaron's and Ariel's file permissions on fenari [00:34:20] Logged the message, Master [00:46:19] sigh [00:46:24] (03PS1) 10Dzahn: caesium - revert yaml admin include [operations/puppet] - 10https://gerrit.wikimedia.org/r/135155 [00:48:38] (03PS2) 10Dzahn: caesium - revert yaml admin include [operations/puppet] - 10https://gerrit.wikimedia.org/r/135155 [00:49:47] (03CR) 10Dzahn: [C: 032] caesium - revert yaml admin include [operations/puppet] - 10https://gerrit.wikimedia.org/r/135155 (owner: 10Dzahn) [00:53:13] RECOVERY - Puppet freshness on caesium is OK: puppet ran at Sat May 24 00:53:10 UTC 2014 [01:10:51] (03PS1) 10Ori.livneh: Add README [operations/puppet] - 10https://gerrit.wikimedia.org/r/135157 [01:50:30] (03PS3) 10Ricordisamoa: minor changes to InitialiseSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/129464 [01:53:52] (03PS4) 10Ricordisamoa: minor changes to InitialiseSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/129464 [01:57:56] (03CR) 10Ricordisamoa: "I'm not sure about mixing lowercase/capitalized words ("bug" and "Bug") and Python/PHP-style comments." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/129464 (owner: 10Ricordisamoa) [02:13:28] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-24 02:12:25+00:00 [02:13:35] Logged the message, Master [02:21:14] !log LocalisationUpdate completed (1.24wmf6) at 2014-05-24 02:20:11+00:00 [02:21:20] Logged the message, Master [03:02:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat May 24 03:01:50 UTC 2014 (duration 1m 49s) [03:03:01] Logged the message, Master [06:53:22] (03CR) 10Hashar: "Have the job runner on the beta cluster get moved to HHVM as well ?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135136 (owner: 10Aaron Schulz) [06:58:13] PROBLEM - LighttpdHTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:13] RECOVERY - LighttpdHTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5122 bytes in 6.781 second response time [07:28:13] PROBLEM - LighttpdHTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:03] RECOVERY - LighttpdHTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5122 bytes in 0.085 second response time [07:46:31] (03PS1) 10Petrb: inserted iotop to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/135185 [11:32:44] (03CR) 10Odder: [C: 031] Set $wgCategoryCollation to 'uca-cs' on cswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134103 (owner: 10Manybubbles) [16:30:23] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 24 May 2014 01:29:59 PM UTC [16:33:23] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 24 May 2014 01:33:04 PM UTC [17:00:33] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sat May 24 17:00:24 UTC 2014 [18:03:03] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat May 24 18:02:56 UTC 2014 [18:24:13] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 14.29% of data exceeded the critical threshold [500.0] [18:39:13] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0]