[01:03:02] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:39:20] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 221 seconds [01:39:47] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 249 seconds [01:42:56] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:45:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [02:00:02] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 250 seconds [02:00:29] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 277 seconds [02:24:00] !log LocalisationUpdate completed (1.21wmf4) at Sat Nov 24 02:24:00 UTC 2012 [02:24:08] Logged the message, Master [03:15:02] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [03:50:08] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [03:54:11] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [04:44:32] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [04:44:32] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:44:32] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [05:00:26] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [05:00:26] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [06:01:27] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [06:22:11] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [06:23:32] RECOVERY - Lucene on search13 is OK: TCP OK - 0.003 second response time on port 8123 [08:58:42] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [08:59:54] PROBLEM - Lucene on search14 is CRITICAL: Connection timed out [09:03:03] RECOVERY - Lucene on search14 is OK: TCP OK - 8.995 second response time on port 8123 [09:10:55] New review: Faidon; "I'm generally happy with the concept and the conventions used, some minor comments inline." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/34862 [09:13:06] PROBLEM - Lucene on search14 is CRITICAL: Connection timed out [09:21:48] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [09:51:19] New patchset: Hashar; "java module and class to install OpenJDK" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34862 [09:51:19] New patchset: Hashar; "OpenJDK JRE/JDK on CI host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34863 [09:52:37] New review: Hashar; "Maybe we should start using the default puppet style." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/34863 [10:24:26] New patchset: Raimond Spekking; "Enable AFTv5 for dewiki (WiP - DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [10:27:39] New patchset: Raimond Spekking; "Enable AFTv5 for dewiki (WiP - DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [11:03:59] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [13:15:52] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [14:45:21] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [14:45:21] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:45:21] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [15:01:33] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [15:01:33] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [16:02:25] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [16:29:43] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [16:31:40] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [16:34:58] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [16:52:13] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.061 second response time [19:00:06] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [19:20:30] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [19:23:03] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [19:42:42] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [21:05:23] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [21:59:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.049 seconds [22:41:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.763 seconds [23:16:29] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [23:31:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds