[00:43:18] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [01:15:24] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:27:24] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:42:14] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 265 seconds [01:42:23] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 277 seconds [01:48:32] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 644s [01:51:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:54:14] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 7 seconds [01:54:14] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [01:55:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [02:27:51] !log LocalisationUpdate completed (1.20wmf8) at Sat Aug 4 02:27:51 UTC 2012 [02:28:03] Logged the message, Master [02:31:34] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:46:34] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [03:00:41] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:15:22] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [07:31:49] is one of the bits servers playing up? for past day or so I am having failure of skin being delivered [08:03:05] is one of the bits servers playing up? for past day or so I am having failure of skin being delivered [09:34:11] [[Tech]]; MarcoAurelio; /* User-Agent */ new section; https://meta.wikimedia.org/w/index.php?diff=3989482&oldid=3988667&rcid=3433040 [10:06:02] [[Tech]]; MarcoAurelio; /* $wgSimpleFlaggedRevsUI */ new section; https://meta.wikimedia.org/w/index.php?diff=3989513&oldid=3989482&rcid=3433073 [10:39:27] [[Tech]]; MarcoAurelio; /* Checkboxes at Special:UserRights — Order changed for no reason? */ tracked in bugzilla; https://meta.wikimedia.org/w/index.php?diff=3989593&oldid=3989513&rcid=3433165 [10:44:21] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [11:16:18] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [11:28:18] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:52:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:32:45] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [12:47:09] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [13:01:14] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:16:05] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [14:12:51] http://stats.grok.se/en/latest/User:Earth_Exploding_Live is a bit while - what's going on with that page? [14:13:09] the first spike, of course, was the thing that happened to the Alabama page [14:13:21] but wtf caused #2? [14:13:26] thx [14:45:17] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [14:45:18] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [14:45:18] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:58:33] wow, that jc41/jc33 above is hf25. he really should know better than to expect answers in <1 minute [14:58:39] ... [16:13:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.628 seconds [16:48:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.680 seconds [17:13:41] Reedy: another thing: 5 endless redirects (redirect pages to themselves): http://p.defau.lt/?ySRJ8EiGI1OM3IybnH3EAg [17:14:56] self redirects that is [17:16:06] here's a query which takes a bit longer, but might find current ones on toolserver: http://p.defau.lt/?7G9i7QIywlEs7xgZdQi22g [17:21:04] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:43] jorn: aren't double redirect fixers checking those manually? [17:22:42] Nemo_bis: you mean bots? [17:22:56] jorn: well yes they're mostly bots [17:23:04] some are operated semiautomatically though [17:23:42] well, not sure they find loops... these were just self redirects i found in the dumps... [17:25:00] jorn: why shouldn't they find them? [17:25:11] they're listed by SPecial:DoubleRedirects aren't they [17:25:23] otherwise you wouldn't find so few of them :) [17:27:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [18:12:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [18:57:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.058 seconds [19:42:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.252 seconds [20:27:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:43] [[Tech]]; MZMcBride; /* $wgSimpleFlaggedRevsUI */ +reply; https://meta.wikimedia.org/w/index.php?diff=3990476&oldid=3989593&rcid=3433734 [20:39:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.048 seconds [20:44:51] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [21:11:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [21:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.485 seconds [21:29:22] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [21:53:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:56:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:24] [[Tech]]; MZMcBride; /* User-Agent */ +reply; https://meta.wikimedia.org/w/index.php?diff=3990593&oldid=3990476&rcid=3433836 [22:08:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.568 seconds [22:33:25] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [22:41:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:25] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [22:55:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [23:01:55] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [23:06:33] !replag [23:07:36] @replag [23:07:37] jeremyb: [s1] db32: 1s, db59: 1s, db60: 1s [23:08:14] @replag [23:08:15] jeremyb: No replag currently. See also "replag all". [23:10:27] 1 second [23:10:28] shit. [23:10:58] what's there? [23:12:56] saper: enwiki [23:13:09] @dbinfo enwikip [23:13:13] @dbinfo enwiki [23:13:31] @replag enwiki [23:13:31] jeremyb: [enwiki: s1] db38: 0s, db36: 1s, db32: 1s, db59: 1s, db60: 1s, db12: ? [23:14:09] @replag plwiki [23:14:09] saper: [plwiki: s2] db52: 0s, db53: 0s, db54: 0s, db57: 0s [23:16:55] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [23:26:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.489 seconds