[00:26:34] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [01:01:45] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Dec 30 01:01:28 UTC 2011 [03:51:35] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [04:39:27] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [10:01:14] RECOVERY - MySQL slave status on es1004 is OK: OK: [11:38:33] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [14:01:18] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [14:50:16] !log unbroke replication on fundraisingdb's by skipping a bad query on the slaves [14:50:27] Logged the message, Master [15:31:30] PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 13454s [15:33:19] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 11259s [15:34:20] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 11040s [16:12:20] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [16:15:40] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 1s [16:36:01] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [21:48:11] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours