[00:09:00] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [00:09:00] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [00:09:00] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [00:12:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.466 seconds [00:31:32] New review: Demon; "Commit msg nitpick. Actual change looks ok." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/31670 [00:46:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.951 seconds [01:35:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 263 seconds [01:45:21] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [01:50:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.024 seconds [02:00:15] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 267 seconds [02:19:26] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [02:22:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:44] !log LocalisationUpdate completed (1.21wmf3) at Sun Nov 4 02:28:44 UTC 2012 [02:28:56] Logged the message, Master [02:35:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.207 seconds [02:49:49] !log LocalisationUpdate completed (1.21wmf2) at Sun Nov 4 02:49:49 UTC 2012 [02:49:57] Logged the message, Master [03:20:49] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [03:50:49] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [03:50:49] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [03:50:49] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [03:56:58] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [04:16:28] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 191 seconds [04:16:46] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 198 seconds [04:20:05] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 26 seconds [04:21:25] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [05:02:53] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [05:30:54] !log catrope synchronized php-1.21wmf2/extensions/TimedMediaHandler/ [05:30:58] Logged the message, Master [05:31:30] !log catrope synchronized php-1.21wmf3/extensions/TimedMediaHandler/ [05:31:38] Logged the message, Master [05:53:12] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [06:01:26] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [06:53:10] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [06:54:40] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 9.020 second response time on port 8123 [07:01:28] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [07:02:57] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.027 second response time on port 8123 [07:06:41] PROBLEM - Lucene on search1016 is CRITICAL: Connection timed out [07:09:52] RECOVERY - Lucene on search1016 is OK: TCP OK - 9.020 second response time on port 8123 [07:13:55] New patchset: Dereckson; "(bug 41712) he.wiki images size configuration" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31580 [07:14:34] New review: Dereckson; "PS2: size: 200px x 200px" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/31580 [07:18:04] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [07:19:25] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.027 second response time on port 8123 [07:19:40] !og restarted lucene search on search1016 [08:42:23] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [10:09:42] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [10:09:42] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [10:09:42] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:20:14] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [12:41:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [13:15:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:21] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [13:25:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [13:52:17] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [13:52:17] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:52:17] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [13:57:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.018 seconds [14:45:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [15:03:32] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [15:32:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.020 seconds [16:20:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.027 seconds [17:07:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.046 seconds [17:55:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.324 seconds [18:41:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:41] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [18:44:44] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [18:56:26] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [18:56:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [19:29:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.341 seconds [20:10:36] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [20:10:36] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [20:10:37] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [20:16:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.819 seconds [20:48:07] New review: jan; "Because Faidon does not know where this change is useful I want to explain this now: This change sho..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/29975 [21:05:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:03] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.905 seconds [21:49:58] New review: Multichill; "Do you have any statistics on how many hits you're still getting? Is is possible to redirect the tra..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/31302 [21:52:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:05] New review: Faidon; "The first use case is fine by me and much welcome (and thanks!). The second one is not: labsconsole ..." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/29975 [22:01:05] New review: MaxSem; "Yesterday we had ~3K requests. Redirection to Toolserver would be https://gerrit.wikimedia.org/r/#/c..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/31302 [22:07:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.037 seconds [22:21:43] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [22:40:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.348 seconds [23:23:08] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [23:27:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.260 seconds [23:53:04] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [23:53:04] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [23:53:04] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours