[00:02:37] !log applying loopback filter on cr2-pmtpa [00:02:39] Logged the message, Mistress of the network gear. [01:44:24] RECOVERY - MySQL Slave Delay on db1005 is OK: OK replication delay 0 seconds [02:19:34] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1457s [02:30:44] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2127s [02:53:54] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [02:54:44] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 58s [03:16:14] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Sat Feb 4 03:16:10 UTC 2012 [04:18:01] RECOVERY - Disk space on es1004 is OK: DISK OK [04:19:51] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [05:33:47] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [05:33:47] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [05:41:47] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [09:31:33] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 453965 MB (3% inode=99%): [09:41:03] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 384909 MB (3% inode=99%): [15:44:52] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [15:44:52] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [15:53:02] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [17:28:57] New patchset: Bhartshorne; "replaced placeholder ganglia-logtailer module with real one" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2287 [17:30:14] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2287 [17:30:15] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2287 [17:56:08] New patchset: Bhartshorne; "running logtailer every minute instead of every 5 for more detailed stats. fixing typo." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2288 [17:56:29] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2288 [17:56:30] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2288 [18:26:56] New patchset: Diederik; "Pylint support (initial version)" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/2289 [19:45:52] New patchset: Bhartshorne; "fixed inconsistent naming of the metrics" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2290 [19:46:10] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2290 [19:46:10] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2290 [19:46:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2290 [19:47:48] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Sat Feb 4 19:47:27 UTC 2012 [21:15:36] PROBLEM - check_gcsip on payments4 is CRITICAL: Connection timed out [21:15:36] PROBLEM - check_gcsip on payments1 is CRITICAL: Connection timed out [21:15:36] PROBLEM - check_gcsip on payments2 is CRITICAL: Connection timed out [21:15:36] PROBLEM - check_gcsip on payments3 is CRITICAL: Connection timed out [21:21:06] PROBLEM - check_gcsip on payments3 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [21:21:06] PROBLEM - check_gcsip on payments2 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [21:21:06] PROBLEM - check_gcsip on payments4 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [21:21:06] PROBLEM - check_gcsip on payments1 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [21:25:06] RECOVERY - check_gcsip on payments4 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.555 second response time [21:25:06] RECOVERY - check_gcsip on payments1 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.550 second response time [21:25:06] RECOVERY - check_gcsip on payments3 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.558 second response time [21:25:06] RECOVERY - check_gcsip on payments2 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.550 second response time [23:04:49] New patchset: Bhartshorne; "puts are 201, let's split them out." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2291 [23:05:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2291 [23:08:15] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2291 [23:08:16] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2291