[00:00:15] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 181 seconds [00:00:28] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 183 seconds [00:08:39] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 00:08:17 UTC 2013 [00:08:59] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:09:19] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [01:07:05] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 182 seconds [01:07:46] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 184 seconds [01:25:45] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 30 seconds [01:46:05] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 181 seconds [01:46:46] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 184 seconds [01:48:05] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 184 seconds [02:14:50] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [02:25:44] !log LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013 [02:25:52] Logged the message, Master [02:35:29] New review: Apmon; "Trying out PatchSet 10 on labs, there were two things I needed to change to get it working. I don't ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222 [02:37:00] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 181 seconds [02:37:50] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 202 seconds [02:45:50] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [02:46:00] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [02:52:21] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [02:53:21] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [02:58:11] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [03:04:11] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [03:26:21] RECOVERY - MySQL disk space on neon is OK: DISK OK [03:27:21] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 3 processes with args ircecho [04:09:27] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:09:21 UTC 2013 [04:09:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:10:27] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:10:24 UTC 2013 [04:11:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:12:07] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:12:05 UTC 2013 [04:12:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:13:07] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:12:58 UTC 2013 [04:13:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:14:37] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:14:31 UTC 2013 [04:14:48] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:15:27] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:15:23 UTC 2013 [04:16:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:16:57] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:16:46 UTC 2013 [04:17:27] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:17:22 UTC 2013 [04:17:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:18:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:18:37] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:18:29 UTC 2013 [04:18:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:19:17] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:19:07 UTC 2013 [04:19:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:20:07] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:20:06 UTC 2013 [04:20:47] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:20:39 UTC 2013 [04:20:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:21:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:21:48] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:21:40 UTC 2013 [04:21:48] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:21:44 UTC 2013 [04:21:48] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:22:28] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:23:09] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:22:59 UTC 2013 [04:23:09] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:23:02 UTC 2013 [04:23:28] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:23:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:24:17] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:24:09 UTC 2013 [04:24:18] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:24:11 UTC 2013 [04:24:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:24:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:25:20] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:25:11 UTC 2013 [04:25:20] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:25:14 UTC 2013 [04:25:28] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:25:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:26:07] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:26:05 UTC 2013 [04:26:18] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:26:08 UTC 2013 [04:26:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:26:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:26:57] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:26:50 UTC 2013 [04:26:58] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:26:55 UTC 2013 [04:27:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:27:37] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:27:30 UTC 2013 [04:27:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:28:07] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:28:06 UTC 2013 [04:28:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:28:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:50:47] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Sun Mar 3 04:50:37 UTC 2013 [04:50:47] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [04:50:47] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 192 seconds [04:50:57] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 192 seconds [04:57:07] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Mar 3 04:57:03 UTC 2013 [04:57:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:59:47] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [04:59:57] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [21:23:00] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [21:23:20] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [22:34:03] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 187 seconds [22:34:13] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 193 seconds [22:34:33] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 199 seconds [22:34:44] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 202 seconds [22:51:33] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 4 seconds [22:51:43] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 3 seconds [22:59:33] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [23:00:37] PROBLEM - Puppet freshness on db27 is CRITICAL: Puppet has not run in the last 10 hours [23:01:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [23:04:07] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 188 seconds [23:04:17] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 191 seconds [23:05:27] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours