[00:03:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.928 seconds [00:20:02] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [00:34:17] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [00:37:26] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [00:51:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.086 seconds [01:36:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:38] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 228 seconds [01:41:47] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 238 seconds [01:47:56] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 607s [01:50:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [01:56:29] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:56:29] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [01:56:29] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [01:56:29] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [01:56:47] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [01:58:26] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 4s [01:59:38] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 17 seconds [02:02:29] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [02:22:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [03:25:18] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [03:25:18] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [04:06:56] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [04:28:05] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [07:53:59] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [07:53:59] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [07:54:00] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [07:54:00] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [07:54:01] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [07:54:01] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [07:54:02] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:54:03] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:39:47] PROBLEM - Puppet freshness on lvs5 is CRITICAL: Puppet has not run in the last 10 hours [09:45:24] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 182 seconds [09:45:33] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 188 seconds [09:46:18] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 220 seconds [09:46:27] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 224 seconds [09:49:18] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [09:49:27] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [09:49:54] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [09:50:03] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 1 seconds [10:38:44] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [11:29:21] PROBLEM - Backend Squid HTTP on knsq16 is CRITICAL: Connection refused [11:57:15] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [11:57:15] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [11:57:15] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [11:57:15] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [11:57:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.051 seconds [12:03:15] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:03:15] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [12:04:18] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [12:08:30] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [12:30:38] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [12:41:08] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [13:14:08] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [13:23:17] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [13:25:05] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [13:26:26] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [13:26:26] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [13:29:17] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [14:21:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.572 seconds [14:40:18] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:36] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [15:04:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.091 seconds [15:26:03] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [15:39:51] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [15:51:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.813 seconds [16:07:48] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [16:34:40] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [16:36:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:19] New review: Jeremyb; "Disabled in I30c889c1eeb5a611dcd3446b29." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16941 [16:51:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [16:52:48] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [17:00:18] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [17:18:00] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [17:18:45] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [17:24:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:15] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [17:30:18] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [17:34:48] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.811 seconds [17:47:11] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [17:54:41] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [17:54:41] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [17:54:41] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [17:54:41] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [17:54:41] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [17:54:41] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [17:54:41] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [17:54:42] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [17:54:43] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [17:54:43] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [17:54:43] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [17:54:44] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [17:54:44] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [17:54:45] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [18:09:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.098 seconds [18:40:44] PROBLEM - Puppet freshness on lvs5 is CRITICAL: Puppet has not run in the last 10 hours [18:51:50] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [18:56:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:44] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [19:02:56] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [19:08:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.014 seconds [19:09:50] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [19:31:36] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.044 second response time [19:39:06] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [19:43:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.024 seconds [20:04:54] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [20:29:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:12] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [20:37:18] RECOVERY - Host mw8 is UP: PING WARNING - Packet loss = 80%, RTA = 0.69 ms [20:39:24] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [20:40:36] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [20:42:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.021 seconds [20:42:24] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [20:46:09] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [20:56:39] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [21:07:18] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [21:09:06] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [21:14:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:33] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [21:20:03] RECOVERY - Host mw8 is UP: PING WARNING - Packet loss = 86%, RTA = 0.22 ms [21:26:39] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [21:28:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.878 seconds [21:33:33] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [21:36:06] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [21:40:00] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [21:42:06] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [21:58:09] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [21:58:09] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [21:58:09] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [21:58:09] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [22:02:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:09] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [22:14:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.881 seconds [22:23:08] PROBLEM - NTP on mw8 is CRITICAL: NTP CRITICAL: Offset unknown [22:26:08] RECOVERY - NTP on mw8 is OK: NTP OK: Offset -0.01546311378 secs [22:49:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:59] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [22:51:47] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [23:01:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.524 seconds [23:27:21] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [23:27:21] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [23:28:59] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.047 second response time [23:36:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:29] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [23:50:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [23:51:29] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.045 second response time