[00:07:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 00:07:50 UTC 2013 [00:08:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [00:08:35] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 00:08:25 UTC 2013 [00:09:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [00:11:36] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:26] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [00:13:56] New patchset: Andrew Bogott; "Make sure that adminbot can write to its logfile." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47191 [00:14:24] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47191 [00:15:41] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 207 seconds [00:15:51] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 208 seconds [00:16:08] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 217 seconds [00:16:35] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 229 seconds [00:51:49] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [00:51:58] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [00:51:59] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [00:52:43] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [01:11:10] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Sat Feb 2 01:10:53 UTC 2013 [01:32:15] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:06] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [02:00:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [02:04:20] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:11] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [02:28:18] !log LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013 [02:28:21] Logged the message, Master [02:30:20] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:10] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [02:36:40] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [02:37:43] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [02:37:44] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [02:37:44] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [02:37:44] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [02:38:09] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours [03:05:09] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [03:09:31] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [03:26:08] New patchset: Reedy; "Set wgLanguageCode = nb for nowiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47201 [03:26:35] !log reedy synchronized wmf-config/InitialiseSettings.php [03:26:37] Logged the message, Master [03:26:43] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47201 [03:27:40] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [03:34:53] New patchset: Reedy; "For bug 44570 - Make the parser cache expire at 30 days. Stops us having stale resource links" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47202 [03:35:22] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:12] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.004 second response time [03:41:00] New review: Reedy; "Per https://bugzilla.wikimedia.org/show_bug.cgi?id=44570#c8" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/47202 [03:48:40] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [03:48:40] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [03:48:40] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [03:48:40] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [03:48:40] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [03:50:37] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [03:56:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.648 second response time [04:07:57] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 04:07:53 UTC 2013 [04:08:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [04:08:47] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 04:08:41 UTC 2013 [04:08:57] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [04:09:08] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Sat Feb 2 04:09:04 UTC 2013 [04:09:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [04:09:47] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 04:09:40 UTC 2013 [04:10:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [04:10:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 04:10:30 UTC 2013 [04:11:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [04:12:57] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [04:25:52] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [04:26:10] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 184 seconds [04:26:22] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 189 seconds [04:26:46] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 194 seconds [04:35:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [04:38:32] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: Connection timed out [04:38:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.060 seconds [04:39:22] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [05:30:19] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [05:57:31] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 182 seconds [05:57:46] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [05:57:51] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 188 seconds [05:58:13] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 192 seconds [06:00:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [06:08:02] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [06:08:03] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [06:10:40] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 182 seconds [06:11:12] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 192 seconds [06:11:16] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 193 seconds [06:12:22] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: Connection timed out [06:13:03] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [06:13:12] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 189 seconds [06:13:13] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [06:13:33] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 196 seconds [06:14:16] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 213 seconds [06:20:22] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: Connection timed out [06:21:12] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [06:21:13] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [06:21:16] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [06:21:16] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [06:21:32] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [06:28:22] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 06:28:17 UTC 2013 [06:28:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [06:28:33] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 06:28:31 UTC 2013 [06:29:23] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: Connection timed out [06:29:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [06:30:13] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [06:57:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:04] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [07:10:25] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [07:21:02] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:43] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [07:21:52] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [07:21:53] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [07:22:20] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [07:22:22] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [07:31:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.760 second response time [07:39:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.047 seconds [07:52:40] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [08:10:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:12:04] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [08:12:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:21:04] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [08:21:37] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [08:23:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.791 second response time [08:24:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [08:25:28] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [08:45:18] !log Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000 [08:45:20] Logged the message, Master [08:46:58] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 184 seconds [08:47:18] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 190 seconds [08:48:28] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 221 seconds [08:48:37] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 224 seconds [09:21:12] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 196 seconds [09:21:22] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 200 seconds [09:21:38] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 206 seconds [09:21:47] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 210 seconds [09:23:35] !log maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/' [09:23:36] Logged the message, Master [09:32:08] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [09:32:08] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [09:32:12] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [09:32:22] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [10:00:34] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:25] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [10:10:31] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 15 seconds [10:11:00] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [10:11:11] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [10:11:35] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [10:18:44] grrr [10:27:19] New patchset: Faidon; "Temporarily disable LVS checks for ms-fe.eqiad" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47210 [10:28:05] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47210 [10:29:11] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:02] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [10:34:12] PROBLEM - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:01] RECOVERY - LVS HTTP IPv4 on ms-fe.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 247 bytes in 0.001 second response time [10:35:02] shut up [12:07:48] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 12:07:47 UTC 2013 [12:08:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [12:08:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 12:08:37 UTC 2013 [12:09:38] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [12:37:50] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [12:38:44] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [12:38:44] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [12:38:44] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [12:38:44] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [12:49:14] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [13:06:04] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [13:28:52] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [13:48:49] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 182 seconds [13:49:19] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 185 seconds [13:49:19] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 185 seconds [13:49:52] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [13:49:52] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:49:52] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [13:49:52] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [13:49:52] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [13:50:01] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 191 seconds [13:51:49] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [13:58:13] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [14:09:33] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [14:13:33] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [14:53:13] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [14:53:48] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 194 seconds [14:53:49] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 194 seconds [14:53:58] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 196 seconds [15:07:23] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [15:23:44] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [15:23:44] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [15:23:58] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [15:24:43] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [15:31:11] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [15:47:46] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 188 seconds [15:47:47] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 189 seconds [15:48:23] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 197 seconds [15:48:50] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 202 seconds [15:57:47] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [15:57:47] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [15:57:50] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [15:58:44] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 2 seconds [16:07:46] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 16:07:40 UTC 2013 [16:08:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:08:36] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 16:08:32 UTC 2013 [16:08:46] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [16:08:47] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [16:08:47] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [16:08:47] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [16:08:47] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [16:09:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:09:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 16:09:26 UTC 2013 [16:10:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [16:13:47] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Puppet has not run in the last 10 hours [16:19:44] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 182 seconds [16:19:46] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 182 seconds [16:24:46] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [16:24:59] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [16:33:33] New patchset: Reedy; "Use overriding to muchly simplify wgNamespacesWithSubpages" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/46826 [16:34:12] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/46826 [16:37:10] New patchset: Reedy; "Cleanup wrong overrides to disable subpages in NS 4/5" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47215 [16:37:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47215 [16:38:33] !log reedy synchronized wmf-config/InitialiseSettings.php [16:38:35] Logged the message, Master [16:38:45] New patchset: Reedy; "Use overriding to muchly simplify wgNamespacesToBeSearchedDefault" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/46873 [16:39:05] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/46873 [16:39:34] !log reedy synchronized wmf-config/InitialiseSettings.php [16:39:36] Logged the message, Master [16:45:13] * Nemo_bis pats Reedy  [17:28:48] New patchset: Reedy; "For bug 44570 - Make the parser cache expire at 30 days. Stops us having stale resource links" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47202 [17:31:01] New review: Reedy; "class misc::maintenance::parsercachepurging {" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/47202 [17:33:21] Reedy: https://bugzilla.wikimedia.org/show_bug.cgi?id=44570#c10 – can this purging be broken by the general purging brokeness? [17:33:57] Eh? [17:34:00] I don't follow [17:34:13] purging requests have been broken for some time [17:34:41] manual ?action=purge ? [17:34:53] all [17:35:07] worked fine for me to fix a page on fondationwiki [17:35:15] when? [17:35:20] earlier this week [17:35:21] purging a cached page in mediawiki or in the squids? [17:36:18] The esams squids not recieving purge requests is completely different [17:36:48] oki [17:38:53] I know, it gets confusing [17:39:06] But seemingly that would also mean the mysql parser cache purging is also broken in some way.. [17:39:10] and has been for a while [17:54:25] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [17:58:27] have problem logging in to my labs instance. getting those errors: [17:58:28] debug3: channel 0: close_fds r -1 w -1 e 6 c -1 [17:58:29] debug3: Wrote 32 bytes for a total of 3431 [17:58:29] debug3: Wrote 64 bytes for a total of 3495 [17:58:29] Connection to nagios-dev closed. [17:58:29] Transferred: sent 3136, received 3656 bytes, in 0.3 seconds [17:58:30] Bytes per second: sent 9989.3, received 11645.7 [17:58:32] debug1: Exit status 254 [18:06:21] New patchset: Nemo bis; "(bug 44603) Update planets hourly" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47223 [18:12:36] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [18:19:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [18:21:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [18:22:28] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [18:36:44] notpeter: around? [18:52:04] New patchset: Reedy; "Set $wgNamespacesWithSubpages NS_MEDIAWIKI to true" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47227 [18:53:27] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47227 [18:54:14] !log reedy synchronized wmf-config/InitialiseSettings.php [18:54:15] Logged the message, Master [19:01:00] New patchset: Reedy; "Bug 37374 - Nepali Wikipedia,Wikibooks and Wiktionary has a pdf rendering component which does not function for Devnagari Scripts" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47228 [19:01:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47228 [19:02:34] !log reedy synchronized wmf-config/InitialiseSettings.php [19:02:35] Logged the message, Master [19:05:40] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [19:05:50] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [19:05:59] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [19:06:10] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [20:00:38] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 188 seconds [20:00:39] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 189 seconds [20:01:21] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 204 seconds [20:01:30] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 207 seconds [20:02:29] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:07:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 20:07:44 UTC 2013 [20:08:29] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:08:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 20:08:29 UTC 2013 [20:09:29] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:19:29] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat Feb 2 20:19:25 UTC 2013 [20:19:30] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [20:20:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.766 second response time [21:08:27] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 13 seconds [21:08:51] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [21:09:00] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [21:09:18] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [21:26:12] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 183 seconds [21:26:33] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 188 seconds [21:27:02] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 195 seconds [21:27:18] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 199 seconds [22:33:48] New patchset: Reedy; "Bug 43668 - Re-enable disabled Special pages on small wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47297 [22:35:37] !log reedy synchronized wmf-config/InitialiseSettings.php [22:35:39] Logged the message, Master [22:36:52] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47297 [22:38:41] PROBLEM - Puppet freshness on db1041 is CRITICAL: Puppet has not run in the last 10 hours [22:38:41] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [22:39:24] :o [22:40:40] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [22:40:40] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [22:40:40] PROBLEM - Puppet freshness on niobium is CRITICAL: Puppet has not run in the last 10 hours [22:40:40] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [22:46:13] All of those wikis are in s3 of course. Reedy, so now they'll be picked up by the normal update cronjob, right? [22:46:30] Yup, they should [23:03:02] New patchset: Reedy; "Allow upload.wikimedia.org in $wgCopyUploadsDomains on testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47299 [23:03:21] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [23:03:46] !log reedy synchronized wmf-config/InitialiseSettings.php [23:03:47] Logged the message, Master [23:04:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47299 [23:06:40] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [23:29:52] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [23:30:42] !log reedy synchronized php-1.21wmf8/extensions/SearchExtraNS [23:30:43] Logged the message, Master [23:33:00] New patchset: Reedy; "Add SearchExtraNS" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47301 [23:34:06] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47301 [23:34:39] !log reedy synchronized wmf-config/ [23:34:40] Logged the message, Master [23:35:59] New patchset: Reedy; "Bug 43329 - Enable SearchExtraNs extension on Commons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47302 [23:39:11] Reedy: is upload by url actually enabled on testwiki? [23:40:02] ah, nope, wgCopyUploadsFromSpecialUpload false [23:40:55] It is [23:40:57] New review: Nemo bis; "For those wondering like me: note that $wgCopyUploadsFromSpecialUpload is still false as per default." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47299 [23:41:00] wgAllowCopyUploads [23:41:05] They've got UploadWizard [23:41:18] sure but you can't actually use it [23:41:30] why? [23:41:47] hm do you mean that the flickr uploader fetches any allowed URL? [23:41:58] no [23:42:04] well, maybe [23:42:05] I'm not sure [23:42:11] I dunno if it's flickr specific [23:42:20] I did what was asked :p [23:42:27] The URL entered is not a valid Flickr image or photoset URL. [23:42:39] * Enable copy uploads from Special:Upload. $wgAllowCopyUploads must also be [23:42:39] * true. If $wgAllowCopyUploads is true, but this is false, you will only be [23:42:39] * able to perform copy uploads from the API or extensions (e.g. UploadWizard). [23:43:07] hmm maybe they want to use API [23:43:22] * Reedy shrugs [23:43:24] progress! ;) [23:43:25] +1 [23:47:10] New patchset: Reedy; "Bug 27839 - Offer 460px and 620px as thumbnail size preferences" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47303 [23:47:44] !log reedy synchronized wmf-config/InitialiseSettings.php [23:47:45] Logged the message, Master [23:47:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47303 [23:48:34] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 187 seconds [23:49:03] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 201 seconds [23:49:14] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 210 seconds [23:49:29] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 214 seconds [23:51:08] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [23:51:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [23:51:08] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [23:51:08] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [23:51:09] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [23:53:05] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [23:54:53] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [23:55:15] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [23:55:36] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [23:55:56] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [23:58:47] New patchset: Reedy; "More enabling securepoll configurable" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47304 [23:59:05] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47304 [23:59:37] !log reedy synchronized wmf-config/ [23:59:38] Logged the message, Master