[00:08:27] PROBLEM - swift-container-auditor on ms-be1 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [00:27:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.662 seconds [00:35:27] RECOVERY - swift-container-auditor on ms-be1 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [01:07:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.323 seconds [01:19:56] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [01:41:59] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 215 seconds [01:48:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:44] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 19 seconds [01:56:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 9.081 seconds [02:25:58] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [02:29:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.026 seconds [02:54:28] RECOVERY - Puppet freshness on locke is OK: puppet ran at Sat Apr 21 02:54:28 UTC 2012 [03:27:56] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [03:49:05] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [07:31:33] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [08:40:18] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [08:59:47] RECOVERY - swift-container-auditor on ms-be2 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [11:21:00] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [11:44:24] PROBLEM - MySQL Replication Heartbeat on db24 is CRITICAL: CRIT replication delay 194 seconds [11:44:42] PROBLEM - MySQL Slave Delay on db24 is CRITICAL: CRIT replication delay 206 seconds [11:44:42] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 197 seconds [11:45:09] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 217 seconds [11:54:42] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/5443 [11:57:18] RECOVERY - MySQL Slave Delay on db24 is OK: OK replication delay 2 seconds [11:57:45] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 0 seconds [11:58:03] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [11:58:21] RECOVERY - MySQL Replication Heartbeat on db24 is OK: OK replication delay 0 seconds [13:28:59] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [13:49:59] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [16:05:58] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [16:07:19] RECOVERY - swift-container-auditor on ms-be2 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [17:32:02] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [19:22:57] New patchset: Alex Monk; "Don't log patchsets submitted by L10n-bot or comments/merges made by gerrit2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5547 [19:23:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5547 [20:32:22] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Timeout reading from 10.0.8.23:11000 [20:33:43] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [21:21:26] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [22:44:24] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [22:45:45] RECOVERY - swift-container-auditor on ms-be2 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:30:11] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [23:51:10] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours