[01:18:55] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [01:28:58] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:32:52] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [01:32:52] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [01:32:52] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [01:41:25] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 219 seconds [01:42:37] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 288 seconds [01:48:37] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 644s [01:54:28] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [01:57:09] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 19s [01:57:28] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 8 seconds [02:09:54] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [03:44:52] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [04:07:57] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:47:52] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [05:02:52] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [05:17:51] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [05:32:51] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:02:16] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [07:02:16] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [07:55:31] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [07:56:34] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.58 ms [08:00:10] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [08:08:16] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [08:40:39] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [10:52:04] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [10:52:31] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 193 seconds [10:53:16] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 203 seconds [10:53:25] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 208 seconds [11:05:25] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [11:05:25] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [11:06:10] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [11:06:19] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [11:20:16] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [11:30:18] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [11:34:12] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [11:34:12] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [11:34:12] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [12:11:15] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [13:46:13] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [14:09:19] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:49:12] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [15:04:12] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [15:19:12] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [15:34:13] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [17:03:19] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [17:03:19] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [18:03:32] spence is up to 86 hrs since last puppet run... [18:09:18] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [18:17:09] i think i broke something in labs... (but probably just effects this one new user) [18:17:41] got a deadlock timeout try restarting transaction thing on labsconsole during account creation [18:18:02] account exists with the right name and email in LDAP and labsconsole says the account doesn't exist at all [18:18:35] i assume this needs a ryan [18:25:38] here's the broken account: https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=571117&oldid=570852 [21:21:18] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [21:31:12] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [21:35:15] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [21:35:16] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [21:35:16] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [22:12:19] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [22:20:43] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 279 seconds [22:20:43] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 281 seconds [22:30:46] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 2 seconds [22:30:46] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [23:47:16] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours