[00:10:06] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:29:26] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.113 second response time [00:34:06] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 00:34:03 UTC 2013 [00:35:06] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:36:26] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 305 seconds [01:14:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:34:11] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 01:34:00 UTC 2013 [01:34:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:01:40] !log LocalisationUpdate completed (1.22wmf18) at Sun Sep 29 02:01:40 UTC 2013 [02:01:59] Logged the message, Master [02:02:22] !log LocalisationUpdate completed (1.22wmf19) at Sun Sep 29 02:02:22 UTC 2013 [02:02:38] Logged the message, Master [02:07:37] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 29 02:07:37 UTC 2013 [02:07:50] Logged the message, Master [02:13:21] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 02:34:50 UTC 2013 [02:35:21] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:21] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:21] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:43:21] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [02:46:21] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [02:49:21] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [02:49:21] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [02:52:21] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [02:54:21] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [02:54:21] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:55:21] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [02:56:21] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:58:21] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [03:00:21] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [03:01:21] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:03:21] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [03:03:21] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [03:03:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 03:03:45 UTC 2013 [03:04:21] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [03:34:23] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 03:34:21 UTC 2013 [03:35:13] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:49] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay 0 seconds [04:47:39] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours [05:11:22] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:12:48] (03CR) 10Ori.livneh: [C: 031] "Yes, please." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86250 (owner: 10Krinkle) [05:21:09] (03PS1) 10TTO: Enable subpages in Programs namespace of metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86414 [05:29:12] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:29:32] PROBLEM - SSH on searchidx2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:31:12] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:32:12] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:33:52] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 05:33:42 UTC 2013 [05:34:22] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:34:32] RECOVERY - SSH on searchidx2 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:35:02] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [05:35:27] (03PS4) 10Physikerwelt: Intial version of puppet script for LaTeXML [operations/puppet] - 10https://gerrit.wikimedia.org/r/61767 [05:42:46] (03PS1) 10TTO: Miscellaneous cleanup of InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 [06:14:59] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:59] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 06:33:56 UTC 2013 [06:34:59] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:09] (03PS1) 10Physikerwelt: Remove customizations from LaTeXML [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/86417 [06:46:23] (03PS1) 10TTO: Clean up wgSiteName in InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86418 [07:12:06] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:17:11] (03CR) 10Physikerwelt: [C: 032 V: 032] Remove customizations from LaTeXML [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/86417 (owner: 10Physikerwelt) [07:34:06] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 07:34:02 UTC 2013 [07:35:06] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:33] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [09:02:33] RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 26.53 ms [09:04:53] PROBLEM - search indices - check lucene status page on search23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:43] RECOVERY - search indices - check lucene status page on search23 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.055 second response time [09:11:53] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:24] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 09:34:22 UTC 2013 [09:34:53] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [10:08:47] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [10:30:57] PROBLEM - search indices - check lucene status page on search1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:47] RECOVERY - search indices - check lucene status page on search1003 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.001 second response time [10:33:57] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 10:33:54 UTC 2013 [10:34:47] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:08:24] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:34:04] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 11:33:54 UTC 2013 [11:34:24] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:53:04] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:04] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [12:06:23] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:13] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 685 seconds [12:15:13] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [12:20:53] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:21:43] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [12:34:43] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 12:34:39 UTC 2013 [12:35:23] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:08] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:08] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [12:44:08] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [12:47:08] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [12:50:08] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [12:50:08] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [12:53:08] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [12:55:08] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [12:55:08] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [12:56:08] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [12:57:08] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: No successful Puppet run in the last 10 hours [12:59:08] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [13:01:08] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [13:02:08] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:04:08] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [13:04:08] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [13:11:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:31:17] (03Abandoned) 10Hashar: Bug 39482 - Rename "chapcomwiki" to "affcomwiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53921 (owner: 10Reedy) [13:34:37] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 13:34:33 UTC 2013 [13:34:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:48:23] (03Abandoned) 10Hashar: comment change only revised comments in file [operations/puppet] - 10https://gerrit.wikimedia.org/r/39342 (owner: 10Dfoy) [13:50:23] (03Abandoned) 10Hashar: Put Parsoid bug messages into #mediawiki-parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/35285 (owner: 10MarkTraceur) [14:08:24] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:44] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:22:34] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [14:27:44] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 8672 seconds [14:30:44] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [14:33:54] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 14:33:49 UTC 2013 [14:34:24] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [14:48:21] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours [14:58:41] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:00:33] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [15:11:41] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:12:41] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [15:38:41] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 12932 seconds [15:41:46] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [15:53:46] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:56:46] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [16:07:11] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 16:33:45 UTC 2013 [16:34:11] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [17:10:00] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [17:34:00] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 17:33:51 UTC 2013 [17:35:00] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [18:55:43] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 24752 seconds [18:56:43] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [19:12:07] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [19:15:47] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 25953 seconds [19:16:47] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [19:20:47] PROBLEM - MySQL Idle Transactions on db1056 is CRITICAL: CRIT longest blocking idle transaction sleeps for 26253 seconds [19:21:47] RECOVERY - MySQL Idle Transactions on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds [19:34:57] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 19:34:47 UTC 2013 [19:35:07] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:29] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [20:22:19] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Sun Sep 29 20:22:15 UTC 2013 [20:27:19] RECOVERY - Disk space on ms-fe1001 is OK: DISK OK [20:27:39] RECOVERY - DPKG on ms-fe1001 is OK: All packages OK [20:27:59] RECOVERY - RAID on ms-fe1001 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [20:33:49] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 20:33:47 UTC 2013 [20:34:29] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [20:39:31] PROBLEM - Memcached on ms-fe1001 is CRITICAL: Connection refused [20:39:41] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: Connection refused [20:43:01] RECOVERY - NTP on ms-fe1001 is OK: NTP OK: Offset -0.032366395 secs [20:51:26] !log swift@pmtpa: pushing rings to set 10.0.6.204/sdm1 10.0.6.204/sdn1 10.0.6.208/sde1 weight from 33 to 66 [20:51:39] Logged the message, Master [20:56:11] PROBLEM - Host ms-fe1001 is DOWN: PING CRITICAL - Packet loss = 100% [20:56:31] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Sun Sep 29 20:56:26 UTC 2013 [20:57:11] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Sun Sep 29 20:57:01 UTC 2013 [20:57:11] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Sun Sep 29 20:57:01 UTC 2013 [20:57:11] RECOVERY - Host ms-fe1001 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [20:57:31] RECOVERY - Memcached on ms-fe1001 is OK: TCP OK - 0.000 second response time on port 11211 [21:01:01] RECOVERY - Disk space on ms-fe1002 is OK: DISK OK [21:01:21] RECOVERY - DPKG on ms-fe1002 is OK: All packages OK [21:01:21] RECOVERY - DPKG on ms-fe1003 is OK: All packages OK [21:01:21] RECOVERY - RAID on ms-fe1002 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [21:01:21] RECOVERY - Disk space on ms-fe1004 is OK: DISK OK [21:01:41] RECOVERY - Disk space on ms-fe1003 is OK: DISK OK [21:01:42] RECOVERY - DPKG on ms-fe1004 is OK: All packages OK [21:01:42] RECOVERY - RAID on ms-fe1004 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [21:01:42] RECOVERY - RAID on ms-fe1003 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [21:03:01] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Sun Sep 29 21:02:54 UTC 2013 [21:06:21] RECOVERY - DPKG on ms-be1001 is OK: All packages OK [21:06:42] RECOVERY - RAID on ms-be1001 is OK: OK: State is Optimal, checked 1 logical device(s) [21:06:42] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [21:06:56] (03PS7) 10Hashar: Linkify references to Analytics Mingle projects. [operations/puppet] - 10https://gerrit.wikimedia.org/r/84338 (owner: 10Diederik) [21:09:24] PROBLEM - Host ms-fe1002 is DOWN: PING CRITICAL - Packet loss = 100% [21:09:34] PROBLEM - Host ms-fe1003 is DOWN: PING CRITICAL - Packet loss = 100% [21:09:44] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:34] RECOVERY - Host ms-fe1003 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [21:11:34] RECOVERY - Host ms-fe1002 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [21:11:44] PROBLEM - Memcached on ms-fe1004 is CRITICAL: Connection refused [21:11:54] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: Connection refused [21:18:34] RECOVERY - NTP on ms-fe1004 is OK: NTP OK: Offset -0.04376995564 secs [21:22:44] RECOVERY - NTP on ms-be1001 is OK: NTP OK: Offset -0.02746844292 secs [21:33:54] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 21:33:53 UTC 2013 [21:34:44] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [21:34:54] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:35:44] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [21:56:05] PROBLEM - Host ms-fe1004 is DOWN: PING CRITICAL - Packet loss = 100% [21:56:46] RECOVERY - Memcached on ms-fe1004 is OK: TCP OK - 0.000 second response time on port 11211 [21:56:55] RECOVERY - Host ms-fe1004 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [22:11:27] PROBLEM - NTP on ms-fe1004 is CRITICAL: NTP CRITICAL: Offset unknown [22:11:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [22:15:27] RECOVERY - NTP on ms-fe1004 is OK: NTP OK: Offset -0.04369580746 secs [22:32:32] (03PS1) 10Physikerwelt: Remove dependencies for SVG code from Makefile [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/86617 [22:32:57] (03CR) 10Physikerwelt: [C: 032 V: 032] Remove dependencies for SVG code from Makefile [operations/debs/latexml] - 10https://gerrit.wikimedia.org/r/86617 (owner: 10Physikerwelt) [22:34:57] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 22:34:53 UTC 2013 [22:35:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [22:43:48] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [22:43:48] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [22:50:48] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [22:50:48] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [22:55:48] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [22:55:48] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [22:56:48] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [22:59:48] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [23:01:48] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [23:02:48] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:04:48] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [23:12:47] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [23:13:36] PROBLEM - MySQL Processlist on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:15:26] PROBLEM - MySQL Slave Running on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:15:26] PROBLEM - MySQL Idle Transactions on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:15:36] PROBLEM - mysqld processes on db1049 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [23:15:56] PROBLEM - MySQL Slave Delay on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:15:56] PROBLEM - MySQL Recent Restart on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:16:06] PROBLEM - MySQL Replication Heartbeat on db1049 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:16:56] PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours [23:20:50] !log powercycling db1049, soft lockup, unresponsive [23:21:02] Logged the message, Master [23:34:56] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Sun Sep 29 23:34:47 UTC 2013 [23:35:46] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [23:43:18] legoktm: ?? [23:43:31] [04:20:51 PM] !log powercycling db1049, soft lockup, unresponsive [23:43:31] Technical_13: you weren't on [23:43:33] ^ [23:44:11] Ahh. That's why there is a 37,225 second db lag... [23:44:56] and why do you ask in this channel? [23:45:20] *s/do/didn't/ [23:45:41] Because I wasn't in this channel? [23:45:45] :p [23:52:11] Huh. Apparently it effects my contributions page as well.. [23:54:55] (03PS1) 10Springle: depool db1049 while repl recovers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86630 [23:55:25] (03CR) 10Springle: [C: 032] depool db1049 while repl recovers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86630 (owner: 10Springle) [23:56:25] !log springle synchronized wmf-config/db-eqiad.php 'depool db1049' [23:56:35] Logged the message, Master