[00:12:13] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 1 below the confidence bounds [00:35:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [00:48:32] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sat 31 May 2014 21:47:22 UTC [01:06:12] PROBLEM - RAID on analytics1010 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:08:12] RECOVERY - RAID on analytics1010 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [01:16:52] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jun 1 01:16:44 UTC 2014 [01:17:13] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 5 below the confidence bounds [01:47:12] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [02:14:27] !log LocalisationUpdate completed (1.24wmf6) at 2014-06-01 02:13:23+00:00 [02:14:39] Logged the message, Master [02:25:01] !log LocalisationUpdate completed (1.24wmf7) at 2014-06-01 02:23:58+00:00 [02:25:06] Logged the message, Master [03:10:34] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 1 03:09:28 UTC 2014 (duration 9m 27s) [03:10:39] Logged the message, Master [03:36:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [04:09:07] (03PS1) 10Ori.livneh: Add custom Diamond collector for RCStream [operations/puppet] - 10https://gerrit.wikimedia.org/r/136621 [04:14:15] mutante|away: thanks for the reviews [04:56:01] (03PS1) 10Ori.livneh: rcstream: add 'stream' subcommand to rcstreamctl [operations/puppet] - 10https://gerrit.wikimedia.org/r/136622 [06:03:32] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 01 Jun 2014 03:02:49 UTC [06:37:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [08:03:42] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jun 1 08:03:32 UTC 2014 [09:38:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [11:07:03] (03CR) 10Ori.livneh: [C: 031] mongo: Support newer yaml style configuration (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/135499 (owner: 10Yuvipanda) [11:10:09] _joe_: are you on bugzilla? [11:10:48] (03PS1) 10Nemo bis: [gdash] Add yearly graphs for frontend performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/136631 [11:11:41] (03CR) 10Nemo bis: "I'm curious because of https://meta.wikimedia.org/wiki/Research:The_sudden_decline_of_Italian_Wikipedia" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136631 (owner: 10Nemo bis) [12:39:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [14:03:39] (03PS2) 10Liangent: Set unifont-5.1.20080907.ttf for timeline on ZH projects [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133228 (https://bugzilla.wikimedia.org/20825) [14:50:58] https://www.wikidata.org/wiki/Special:Contributions/10.68.17.64 - is it supposed to be possible to edit with that IP? [14:51:35] Hm, apparently that's tools-exec-09 in labs [14:55:39] Krenair: It's BenBot* [14:55:56] *Bene [15:03:33] after ~48 h still no progress on this issue https://bugzilla.wikimedia.org/show_bug.cgi?id=65978 [15:04:45] It's a weekend [15:05:17] well ... [15:05:31] Reedy: so have time to have a look at https://gerrit.wikimedia.org/r/#/c/133228/ ? :p [15:30:22] PROBLEM - Disk space on analytics1022 is CRITICAL: DISK CRITICAL - free space: / 1068 MB (3% inode=95%): [15:40:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [15:44:23] JohnLewis, yes, I know. [15:44:48] Krenair: and I saw you knew when I looked at #wikidata [16:03:32] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 01 Jun 2014 13:02:49 UTC [16:59:06] (03CR) 10Ori.livneh: [C: 031] [gdash] Add yearly graphs for frontend performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/136631 (owner: 10Nemo bis) [17:27:23] <_joe_> Nemo_bis: I thought the login was unified with gerrit, which is not the case... Why do you need me on bugzilla? [17:27:54] _joe_: oh, just wanted to cc you on a bug report about https://gerrit.wikimedia.org/r/136631 or similar [17:28:36] bugzilla tends to have some stuff for ops too https://bugzilla.wikimedia.org/buglist.cgi?keywords=ops&keywords_type=allwords&list_id=318593&query_format=advanced&resolution=--- [17:31:43] <_joe_> yeah I know... I just assumed I could log in with my labs credentials there :) [17:33:00] <_joe_> Nemo_bis: registered :) [17:34:06] :) thanks [17:34:29] <_joe_> np, I'll be out for most of the day tomorrow anyway [17:34:51] <_joe_> so don't expect me to be promptly responding :) [17:56:02] No emergencies ;) [17:59:45] <_joe_> and tomorrow is bank holiday! we have the joyful military parade in Rome... [18:02:42] Yeah, how lovely [18:31:32] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 01 Jun 2014 15:30:32 UTC [18:41:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [18:50:52] (03PS1) 10Ori.livneh: Make GeoIP lookup code safer [operations/puppet] - 10https://gerrit.wikimedia.org/r/136655 (https://bugzilla.wikimedia.org/64582) [19:03:12] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jun 1 19:03:03 UTC 2014 [19:21:52] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data exceeded the critical threshold [500.0] [19:38:52] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [20:09:12] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 7 below the confidence bounds [20:09:52] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 13.33% of data exceeded the critical threshold [500.0] [20:13:22] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [20:24:52] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [20:30:22] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Jun 1 20:30:16 UTC 2014 [21:27:57] (03PS2) 10Ori.livneh: Make GeoIP lookup code safer [operations/puppet] - 10https://gerrit.wikimedia.org/r/136655 (https://bugzilla.wikimedia.org/64582) [21:42:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [21:44:34] (03CR) 10Ori.livneh: "I applied this on the Beta cluster text varnishes (curl -I http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page to test)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/136655 (https://bugzilla.wikimedia.org/64582) (owner: 10Ori.livneh) [22:27:05] there's a user that sounds like they're having memcached problems on de.wiktionary. FeaturedFeeds won't update the available feed list for them