[00:10:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.836 seconds [00:53:51] PROBLEM - Host ms-be3 is DOWN: PING CRITICAL - Packet loss = 100% [00:59:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.031 seconds [01:40:31] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 294 seconds [01:40:49] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 311 seconds [01:43:49] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:45:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 7 seconds [01:49:04] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:28] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 280 seconds [02:00:46] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 298 seconds [02:02:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.263 seconds [02:29:14] !log LocalisationUpdate completed (1.21wmf4) at Sun Nov 18 02:29:14 UTC 2012 [02:29:22] Logged the message, Master [02:36:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [02:41:52] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Sun Nov 18 02:41:31 UTC 2012 [02:52:59] !log LocalisationUpdate completed (1.21wmf3) at Sun Nov 18 02:52:59 UTC 2012 [02:53:07] Logged the message, Master [02:55:49] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Sun Nov 18 02:55:43 UTC 2012 [02:57:19] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.006 second response time on port 11000 [03:46:49] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [03:49:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [04:47:33] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [04:47:33] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [05:36:29] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [05:51:11] RECOVERY - Lucene on search13 is OK: TCP OK - 8.998 second response time on port 8123 [06:07:50] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [06:11:35] PROBLEM - Squid on brewster is CRITICAL: Connection refused [06:17:26] RECOVERY - Lucene on search13 is OK: TCP OK - 0.008 second response time on port 8123 [06:22:32] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [06:27:22] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [06:29:01] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [06:50:10] RECOVERY - Lucene on search13 is OK: TCP OK - 0.002 second response time on port 8123 [06:55:47] RECOVERY - Squid on brewster is OK: TCP OK - 0.008 second response time on port 8080 [06:59:41] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [07:01:20] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [07:22:03] !log aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php [07:22:11] Logged the message, Master [07:38:30] RECOVERY - Lucene on search13 is OK: TCP OK - 8.996 second response time on port 8123 [07:48:33] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [07:53:21] RECOVERY - Lucene on search13 is OK: TCP OK - 8.994 second response time on port 8123 [07:54:55] New review: Jdlrobson; "why is this not deployed?" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/32864 [08:08:21] PROBLEM - Lucene on search13 is CRITICAL: Connection timed out [08:09:19] hrmmm, search13 has been bouncy for the last 2.5 hrs [08:09:42] RECOVERY - Lucene on search13 is OK: TCP OK - 0.002 second response time on port 8123 [08:09:44] i've no idea if that's a box that matters though [08:09:59] yeah I am just seeing that, and restarted it [08:10:14] k [08:10:23] moin moin ;) [08:10:25] it seemed to be out to lunch in the usual way (required shooting) [08:10:31] morning (mostly not here though) [08:10:43] lemme log that [08:10:53] !restarted lucene search on search13 [08:10:58] er [08:11:03] !log restarted lucene search on search13 [08:11:10] Logged the message, Master [08:11:20] * jeremyb pats morebots [08:17:55] huh the things one learns form wikipedia [08:18:04] Molotov cocktails are considered "destructive devices" under the National Firearms Act and regulated by the ATF. [08:18:17] (Bureau of Alcohol, Tobacco, Firearms and Explosives) [08:28:44] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [08:28:44] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [08:28:44] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:28:44] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:33:14] apergos: wouldn't a molotov cocktail be an IED? [08:33:30] dunno [08:33:48] once bureaucracies get hold of something like that who knows what they do to it [09:09:05] !log aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging' [09:09:11] Logged the message, Master [09:16:35] !log aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php 'debug logging' [09:16:42] Logged the message, Master [09:26:06] !log aaron synchronized php-1.21wmf4/includes/upload/UploadFromChunks.php [09:26:12] Logged the message, Master [09:34:11] cute, debugging [09:34:34] Raymond_: we'll soon have more occasions for UploadWizard tests, I hope [09:34:58] Nemo_bis: yeah :) [09:46:27] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [10:31:51] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [10:48:03] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.005 second response time on port 11000 [12:35:18] PROBLEM - Puppet freshness on dobson is CRITICAL: Puppet has not run in the last 10 hours [12:42:21] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [12:48:50] New review: MaxSem; "Because there was no deployment this week, read mobile-tech" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/32864 [13:50:58] New review: Hashar; "That fixed https://bugzilla.wikimedia.org/show_bug.cgi?id=36874" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32004 [14:48:52] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [14:48:52] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:23:49] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [16:30:23] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [17:00:41] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [18:30:09] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [18:30:09] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [18:30:09] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [18:30:09] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [19:43:32] New patchset: Ori.livneh; "$wgVectorCombineUserTalk defaults to true" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33989 [19:46:08] New patchset: Ori.livneh; "$wgVectorCombineUserTalk defaults to true" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33989 [19:47:16] Change merged: Ori.livneh; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33989 [19:47:30] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [20:32:39] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [20:46:44] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.000 second response time on port 11000 [20:58:21] !log maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging' [20:58:28] Logged the message, Master [21:15:55] New patchset: Nikerabbit; "am* beta override no longer needed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34027 [21:25:20] !log maxsem synchronized php-1.21wmf4/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL' [21:25:28] Logged the message, Master [21:26:30] !log maxsem synchronized php-1.21wmf3/extensions/Narayam/ 'Deploying https://gerrit.wikimedia.org/r/#/c/34024/ to unbreak RL' [21:26:37] Logged the message, Master [21:33:56] !log maxsem synchronized php-1.21wmf4/includes/resourceloader/ResourceLoader.php 'Debugging over' [21:34:03] Logged the message, Master [21:44:44] !log maxsem synchronized php-1.21wmf4/extensions/TocTree/ 'https://gerrit.wikimedia.org/r/#/c/33987/ to fix RL warnings' [21:44:51] Logged the message, Master [21:49:52] New patchset: Nemo bis; "(bug 15434) Periodical run of currently disabled special pages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33713 [22:36:15] PROBLEM - Puppet freshness on dobson is CRITICAL: Puppet has not run in the last 10 hours [22:41:11] New patchset: Nemo bis; "(bug 15434) Periodical run of currently disabled special pages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33713 [22:42:25] New patchset: Nemo bis; "(bug 15434) Periodical run of currently disabled special pages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33713 [22:42:53] uh, miracle, it didn't fail now [22:43:27] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [22:48:43] New patchset: MaxSem; "Add the already deployed EventLogging to extension-list" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34034 [22:49:16] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34034 [22:53:49] !log maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/34034/' [22:53:55] Logged the message, Master