[00:06:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [00:15:38] PROBLEM - Puppet freshness on cp1031 is CRITICAL: Puppet has not run in the last 10 hours [00:39:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [01:21:17] New patchset: DamianZaremba; "Update the SNMP traps." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/25758 [01:23:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/25758 [01:26:08] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:38] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [01:36:38] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [01:36:38] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [01:36:38] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [01:36:38] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:40:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [01:41:35] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 254 seconds [01:42:29] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 309 seconds [01:44:59] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [01:46:02] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 1 seconds [02:10:11] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [02:12:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.403 seconds [02:40:29] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Sun Sep 30 02:40:27 UTC 2012 [02:42:08] RECOVERY - Varnish HTTP upload-frontend on cp1031 is OK: HTTP OK HTTP/1.1 200 OK - 641 bytes in 0.053 seconds [02:54:08] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [03:33:26] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Sun Sep 30 03:33:15 UTC 2012 [03:37:29] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [04:26:32] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [05:28:59] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [05:46:59] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [07:10:58] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [07:10:58] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [08:01:06] PROBLEM - Puppet freshness on mw10 is CRITICAL: Puppet has not run in the last 10 hours [08:01:06] PROBLEM - Puppet freshness on mw12 is CRITICAL: Puppet has not run in the last 10 hours [08:01:06] PROBLEM - Puppet freshness on mw13 is CRITICAL: Puppet has not run in the last 10 hours [08:01:06] PROBLEM - Puppet freshness on mw14 is CRITICAL: Puppet has not run in the last 10 hours [08:01:06] PROBLEM - Puppet freshness on mw11 is CRITICAL: Puppet has not run in the last 10 hours [08:01:07] PROBLEM - Puppet freshness on mw15 is CRITICAL: Puppet has not run in the last 10 hours [08:01:07] PROBLEM - Puppet freshness on mw2 is CRITICAL: Puppet has not run in the last 10 hours [08:01:08] PROBLEM - Puppet freshness on mw16 is CRITICAL: Puppet has not run in the last 10 hours [08:01:08] PROBLEM - Puppet freshness on mw5 is CRITICAL: Puppet has not run in the last 10 hours [08:01:09] PROBLEM - Puppet freshness on mw1 is CRITICAL: Puppet has not run in the last 10 hours [08:01:09] PROBLEM - Puppet freshness on mw4 is CRITICAL: Puppet has not run in the last 10 hours [08:01:10] PROBLEM - Puppet freshness on mw3 is CRITICAL: Puppet has not run in the last 10 hours [08:01:10] PROBLEM - Puppet freshness on mw6 is CRITICAL: Puppet has not run in the last 10 hours [08:01:11] PROBLEM - Puppet freshness on mw8 is CRITICAL: Puppet has not run in the last 10 hours [08:01:11] PROBLEM - Puppet freshness on mw7 is CRITICAL: Puppet has not run in the last 10 hours [08:01:12] PROBLEM - Puppet freshness on mw9 is CRITICAL: Puppet has not run in the last 10 hours [08:12:26] New patchset: Dereckson; "(bug 40628) Enable WikiLove extension on it.wikiquote" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25760 [08:53:38] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours [09:07:35] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [11:37:30] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:37:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [11:37:30] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [11:37:30] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [11:37:30] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [12:11:14] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [12:55:20] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [13:28:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.023 seconds [13:38:30] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [14:08:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:36] PROBLEM - Apache HTTP on mw37 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:57] RECOVERY - Apache HTTP on mw37 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.716 second response time [14:24:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.246 seconds [14:27:24] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [14:58:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.290 seconds [15:30:24] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [15:44:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:24] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [15:58:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.020 seconds [16:22:59] PROBLEM - Puppet freshness on db1022 is CRITICAL: Puppet has not run in the last 10 hours [16:22:59] PROBLEM - Puppet freshness on srv237 is CRITICAL: Puppet has not run in the last 10 hours [16:24:02] PROBLEM - Puppet freshness on db1034 is CRITICAL: Puppet has not run in the last 10 hours [16:24:02] PROBLEM - Puppet freshness on cp1015 is CRITICAL: Puppet has not run in the last 10 hours [16:24:02] PROBLEM - Puppet freshness on db51 is CRITICAL: Puppet has not run in the last 10 hours [16:24:02] PROBLEM - Puppet freshness on db26 is CRITICAL: Puppet has not run in the last 10 hours [16:24:02] PROBLEM - Puppet freshness on ms-be9 is CRITICAL: Puppet has not run in the last 10 hours [16:24:03] PROBLEM - Puppet freshness on mw30 is CRITICAL: Puppet has not run in the last 10 hours [16:24:03] PROBLEM - Puppet freshness on srv249 is CRITICAL: Puppet has not run in the last 10 hours [16:24:04] PROBLEM - Puppet freshness on sanger is CRITICAL: Puppet has not run in the last 10 hours [16:25:05] PROBLEM - Puppet freshness on db25 is CRITICAL: Puppet has not run in the last 10 hours [16:25:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: Puppet has not run in the last 10 hours [16:25:05] PROBLEM - Puppet freshness on srv212 is CRITICAL: Puppet has not run in the last 10 hours [16:34:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.032 seconds [16:54:02] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: Puppet has not run in the last 10 hours [16:54:02] PROBLEM - Puppet freshness on db9 is CRITICAL: Puppet has not run in the last 10 hours [16:54:02] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [16:54:02] PROBLEM - Puppet freshness on mw61 is CRITICAL: Puppet has not run in the last 10 hours [16:54:02] PROBLEM - Puppet freshness on sq80 is CRITICAL: Puppet has not run in the last 10 hours [16:55:05] PROBLEM - Puppet freshness on srv240 is CRITICAL: Puppet has not run in the last 10 hours [17:12:02] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [17:12:02] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [17:22:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.208 seconds [18:01:51] PROBLEM - Puppet freshness on mw1 is CRITICAL: Puppet has not run in the last 10 hours [18:01:51] PROBLEM - Puppet freshness on mw10 is CRITICAL: Puppet has not run in the last 10 hours [18:01:51] PROBLEM - Puppet freshness on mw11 is CRITICAL: Puppet has not run in the last 10 hours [18:01:51] PROBLEM - Puppet freshness on mw12 is CRITICAL: Puppet has not run in the last 10 hours [18:01:51] PROBLEM - Puppet freshness on mw14 is CRITICAL: Puppet has not run in the last 10 hours [18:01:52] PROBLEM - Puppet freshness on mw16 is CRITICAL: Puppet has not run in the last 10 hours [18:01:52] PROBLEM - Puppet freshness on mw15 is CRITICAL: Puppet has not run in the last 10 hours [18:01:53] PROBLEM - Puppet freshness on mw13 is CRITICAL: Puppet has not run in the last 10 hours [18:01:53] PROBLEM - Puppet freshness on mw2 is CRITICAL: Puppet has not run in the last 10 hours [18:01:54] PROBLEM - Puppet freshness on mw5 is CRITICAL: Puppet has not run in the last 10 hours [18:01:54] PROBLEM - Puppet freshness on mw3 is CRITICAL: Puppet has not run in the last 10 hours [18:01:55] PROBLEM - Puppet freshness on mw4 is CRITICAL: Puppet has not run in the last 10 hours [18:01:55] PROBLEM - Puppet freshness on mw7 is CRITICAL: Puppet has not run in the last 10 hours [18:01:56] PROBLEM - Puppet freshness on mw6 is CRITICAL: Puppet has not run in the last 10 hours [18:01:56] PROBLEM - Puppet freshness on mw9 is CRITICAL: Puppet has not run in the last 10 hours [18:01:57] PROBLEM - Puppet freshness on mw8 is CRITICAL: Puppet has not run in the last 10 hours [18:11:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.570 seconds [18:47:10] New patchset: MaxSem; "Remove the WLM app banner" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25771 [18:49:35] New patchset: MaxSem; "Remove the WLM app banner" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25771 [18:54:48] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours [18:57:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:45] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [19:12:27] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25771 [19:16:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.016 seconds [19:46:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.790 seconds [20:34:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.669 seconds [21:21:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.375 seconds [21:38:20] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [21:38:20] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [21:38:20] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:38:20] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [21:38:20] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [21:39:22] New patchset: Nemo bis; "(bug 40285) Point Wikipedias logo to more up to date 2.0 version on Commons where available" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23985 [22:11:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:14] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [22:26:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [22:56:44] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [22:59:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.236 seconds [23:39:38] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [23:46:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds