[00:06:29] PROBLEM - DPKG on virt2 is CRITICAL: Timeout while attempting connection [00:21:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.159 second response time [00:51:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [01:26:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.996 second response time [01:51:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [01:54:06] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [01:54:06] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [01:57:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [02:16:50] !log LocalisationUpdate completed (1.22wmf13) at Sat Aug 24 02:16:50 UTC 2013 [02:16:57] Logged the message, Master [02:19:24] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [02:24:24] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:30] !log LocalisationUpdate completed (1.22wmf14) at Sat Aug 24 02:31:29 UTC 2013 [02:31:35] Logged the message, Master [02:42:22] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 24 02:42:22 UTC 2013 [02:42:27] Logged the message, Master [02:54:58] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [02:56:58] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [03:16:12] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [03:17:11] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [03:17:11] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [03:19:11] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [03:28:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:27:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:28:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.663 second response time [04:39:39] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:41:30] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [04:46:39] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:29] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [05:52:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [05:58:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [06:27:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:32:13] PROBLEM - Puppet freshness on eeden is CRITICAL: No successful Puppet run in the last 10 hours [06:32:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [06:35:33] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.109 second response time [07:22:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [07:53:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [08:17:19] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [08:31:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time [08:35:30] PROBLEM - MySQL Slave Delay on db55 is CRITICAL: CRIT replication delay 184 seconds [08:36:33] PROBLEM - MySQL Replication Heartbeat on db55 is CRITICAL: CRIT replication delay 181 seconds [09:35:07] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [09:37:29] RECOVERY - MySQL Replication Heartbeat on db55 is OK: OK replication delay 22 seconds [09:37:38] RECOVERY - MySQL Slave Delay on db55 is OK: OK replication delay 26 seconds [09:55:42] !log optimizing tables on db55. expect lag [09:55:48] Logged the message, Master [09:58:29] PROBLEM - MySQL Replication Heartbeat on db55 is CRITICAL: CRIT replication delay 191 seconds [09:58:38] PROBLEM - MySQL Slave Delay on db55 is CRITICAL: CRIT replication delay 197 seconds [10:13:30] RECOVERY - MySQL Replication Heartbeat on db55 is OK: OK replication delay 14 seconds [10:13:40] RECOVERY - MySQL Slave Delay on db55 is OK: OK replication delay 20 seconds [11:40:26] PROBLEM - Disk space on es1003 is CRITICAL: DISK CRITICAL - free space: /a 381388 MB (3% inode=99%): [11:40:36] PROBLEM - MySQL disk space on es1003 is CRITICAL: DISK CRITICAL - free space: /a 381387 MB (3% inode=99%): [11:55:06] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [11:55:06] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [12:20:03] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [12:25:03] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [12:55:01] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [12:57:01] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:17:00] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [13:18:00] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [13:18:00] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [13:20:00] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [13:27:16] (03PS1) 10Amire80: Don't show the IME in the CodeEditor textarea [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80717 [13:58:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [16:22:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [16:32:23] PROBLEM - Puppet freshness on eeden is CRITICAL: No successful Puppet run in the last 10 hours [16:58:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [16:59:57] PROBLEM - DPKG on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:38:08] PROBLEM - Disk space on cp1046 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 12437 MB (3% inode=99%): /srv/sdb3 12678 MB (4% inode=99%): [18:17:52] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [18:44:28] (03PS1) 10Dereckson: Planet updates. [operations/puppet] - 10https://gerrit.wikimedia.org/r/80732 [19:05:49] (03PS1) 10Dereckson: Planet update [operations/puppet] - 10https://gerrit.wikimedia.org/r/80760 [19:06:07] (03CR) 10Dereckson: "Follow-up: I7cd28ffb81c8f217b399fa021169b53b5ef62163" [operations/puppet] - 10https://gerrit.wikimedia.org/r/73564 (owner: 10Jalexander) [19:26:19] !log updated Parsoid to 8afe5ef [19:26:24] Logged the message, Master [19:36:53] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [21:56:03] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [21:56:03] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [22:20:49] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [22:25:49] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [22:55:54] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [22:57:54] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:15:06] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.109 second response time [23:17:06] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [23:18:06] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [23:18:06] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [23:20:06] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [23:45:33] (03CR) 10Jalexander: [C: 031] Planet updates. [operations/puppet] - 10https://gerrit.wikimedia.org/r/80732 (owner: 10Dereckson) [23:52:07] (03CR) 10Jalexander: [C: 04-1] "I'm at least temporarily -1ing this for now while I think about it (though if Dzahn or another merger wants to override me I won't throw a" [operations/puppet] - 10https://gerrit.wikimedia.org/r/80760 (owner: 10Dereckson)