[00:44:51] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [00:45:36] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [00:45:56] ^ Now wasn't that shut down by a sysadmin?! [01:41:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 229 seconds [01:41:57] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 242 seconds [01:46:54] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:48:51] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 656s [01:49:18] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 4 seconds [01:50:21] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 27s [01:53:57] PROBLEM - MySQL Idle Transactions on db38 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:54:24] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL - pattern not found [01:55:54] RECOVERY - LVS HTTP IPv4 on m.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 0.080 second response time [02:16:18] RECOVERY - MySQL Idle Transactions on db38 is OK: OK longest blocking idle transaction sleeps for 0 seconds [02:38:48] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Sat Jun 9 02:38:46 UTC 2012 [02:41:48] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Sat Jun 9 02:41:24 UTC 2012 [02:49:54] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [02:51:24] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Sat Jun 9 02:51:12 UTC 2012 [03:41:57] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:08:48] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [08:13:19] PROBLEM - Host mw1042 is DOWN: PING CRITICAL - Packet loss = 100% [09:41:22] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [09:41:22] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [09:41:22] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [10:45:25] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [12:28:19] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [12:50:22] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [13:42:25] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [16:09:25] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [19:42:25] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [19:42:25] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [19:42:25] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [20:19:52] New review: Krinkle; "(no comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/5289 [20:23:31] New patchset: Krinkle; "Gerrit CSS tweak: word-wrap commit summaries." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5289 [20:23:52] New review: Krinkle; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/5289 [20:23:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5289 [20:46:19] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [21:33:50] New review: SPQRobin; "(no comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/10593 [22:29:22] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [22:51:25] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [23:22:28] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:28:19] RECOVERY - swift-container-auditor on ms-be2 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:43:19] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours