[00:13:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:34:21] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 00:34:20 UTC 2013 [00:34:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:03:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 01:03:43 UTC 2013 [01:04:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:08:28] (03PS1) 10Reedy: Add test2.m entry pointing at mobile-lb [operations/dns] - 10https://gerrit.wikimedia.org/r/86346 [01:29:42] (03PS1) 10Reedy: Mobile works fine on wikivoyage. Remove specialcase on [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86347 [01:34:03] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 01:33:54 UTC 2013 [01:34:33] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:01:52] !log LocalisationUpdate completed (1.22wmf18) at Sat Sep 28 02:01:52 UTC 2013 [02:02:14] Logged the message, Master [02:02:32] !log LocalisationUpdate completed (1.22wmf19) at Sat Sep 28 02:02:32 UTC 2013 [02:02:52] Logged the message, Master [02:08:41] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 28 02:08:41 UTC 2013 [02:08:58] Logged the message, Master [02:14:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:02] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 02:33:54 UTC 2013 [02:34:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:33] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.113 second response time [05:01:23] PROBLEM - search indices - check lucene status page on search1018 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60648 bytes in 0.017 second response time [06:10:06] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:26] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:29:26] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [06:38:40] (03CR) 10Adamw: [C: 04-1] "This is a great feature to add, thank you!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 (owner: 10Dereckson) [06:39:56] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 06:39:48 UTC 2013 [06:41:10] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [06:42:10] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:43:10] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [06:46:10] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:49:10] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [06:49:10] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [06:52:10] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [06:54:10] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [06:54:10] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:55:10] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [06:56:10] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:58:10] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [07:00:10] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [07:01:10] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:10] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:10] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [07:07:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:34:10] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 07:34:00 UTC 2013 [07:34:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:37] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:33:57] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 08:33:52 UTC 2013 [08:34:37] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:57] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours [08:47:57] PROBLEM - Puppet freshness on hafnium is CRITICAL: No successful Puppet run in the last 10 hours [09:12:16] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:16:11] (03PS1) 10Rillke: Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 [09:18:29] (03PS2) 10Rillke: Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 [09:18:56] (03PS3) 10Rillke: Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 [09:21:46] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [09:34:06] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 09:34:00 UTC 2013 [09:34:16] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:59:49] (03CR) 10Odder: "Please change the way you link to bugs to '12345' instead of pasting an URL to Bugzilla." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [10:08:33] (03CR) 10Rillke: "Would Odder also be so kind telling me why I should do so? Because this way one can't select the URL, open the context menu and press "ope" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [10:10:34] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [10:14:22] (03PS4) 10Rillke: Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 [10:33:54] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 10:33:51 UTC 2013 [10:34:34] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:25:11] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.113 second response time [11:34:41] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 11:34:39 UTC 2013 [11:35:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:22:38] (03CR) 10Odder: [C: 031] Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [12:34:02] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 12:33:55 UTC 2013 [12:34:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:01:56] (03CR) 10Nemo bis: [C: 031] Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [13:04:25] (03CR) 10Steinsplitter: [C: 031] Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [13:12:50] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:34:40] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 13:34:31 UTC 2013 [13:34:50] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [14:26:49] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:27:49] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [15:09:55] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [15:33:45] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [15:33:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 15:33:54 UTC 2013 [15:34:25] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.54 ms [15:34:55] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [15:37:05] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [15:38:02] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.513 second response time [16:10:53] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:28] (03PS1) 10Wizardist: Set Europe/Minsk TZ for bewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 [16:34:33] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 16:34:24 UTC 2013 [16:34:53] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:23] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.129 second response time [16:42:15] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:15] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:15] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:46:14] (03PS2) 10Nemo bis: Set Europe/Minsk TZ for bewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 (owner: 10Wizardist) [16:46:15] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:46:24] (03CR) 10Nemo bis: [C: 031] "Trivial." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 (owner: 10Wizardist) [16:49:15] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [16:49:15] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:15] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:16] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:16] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:55:15] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:15] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:15] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [17:00:15] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:15] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:15] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:15] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [17:15:07] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [17:33:48] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 17:33:46 UTC 2013 [17:34:07] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [18:12:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [18:35:40] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 18:35:39 UTC 2013 [18:35:51] (03CR) 10Ryan Lane: [C: 032] Disable broken Varnish monitoring for professor graphite host [operations/puppet] - 10https://gerrit.wikimedia.org/r/86329 (owner: 10Ori.livneh) [18:36:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [18:37:26] (03CR) 10Ryan Lane: [C: 032] Fix-ups for Gdash module [operations/puppet] - 10https://gerrit.wikimedia.org/r/86330 (owner: 10Ori.livneh) [18:37:48] (03CR) 10Ryan Lane: [C: 032] Re-introduce 'rendering' metric to navtiming.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/86331 (owner: 10Ori.livneh) [18:47:10] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours [18:48:10] PROBLEM - Puppet freshness on hafnium is CRITICAL: No successful Puppet run in the last 10 hours [19:04:50] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 19:04:46 UTC 2013 [19:05:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [19:22:10] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [19:34:30] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 19:34:27 UTC 2013 [19:35:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [19:53:33] RECOVERY - Puppet freshness on hafnium is OK: puppet ran at Sat Sep 28 19:53:30 UTC 2013 [19:55:13] RECOVERY - Puppet freshness on professor is OK: puppet ran at Sat Sep 28 19:55:11 UTC 2013 [20:10:21] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [20:34:01] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 20:33:57 UTC 2013 [20:34:21] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:29] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [21:33:49] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 21:33:43 UTC 2013 [21:34:29] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [21:40:56] Krinkle|detached once you are detached I noticed something you may find interesting [21:41:11] WMUK has aquired or about to acquire qrwp.org [21:41:21] its 8 chars but its just short enough [21:51:02] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [21:52:02] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [22:07:23] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [22:33:53] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sat Sep 28 22:33:48 UTC 2013 [22:34:23] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours