[00:07:00] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.670 seconds [00:07:41] gn8 folks [00:10:55] maplebed: I see fields getting set on req, but where is req used, other than req.path? [00:11:02] * AaronSchulz wishes this code was clearer [00:11:06] so there's req and req.orig. [00:11:08] err. [00:11:18] reqorig. [00:11:41] req is parsed to get into swift, but reqorig is passed back in the thumb handler. [00:11:53] reqorig is a copy of req (made at line 174) [00:11:56] does that help? [00:13:06] maplebed - is there a patch out there? [00:13:23] AaronSchulz: I'm going to drop swift back to 75% and confirm that it's not broken when upload squids talk directly to ms5. [00:13:26] woosters: no, not yet. [00:14:15] maplebed: req.host = '127.0.0.1:%s' % port [00:14:38] so req is passed on implicitly? [00:14:50] yeah, because the proxy is going to ask itself for the object now that the URL has been transformed to a swift-friendly format. [00:15:04] yeah, this is all like a hook handler [00:15:08] makes sense [00:15:13] I think that part hapens at status = int(controller.response_args[0].split()[0]) [00:15:36] req is part of the env and gets thrown into the controller [00:15:44] yes, it is a hook handler. [00:16:01] (the rewrite module is only one of 6 handlers that the query passes through on its way to being answered) [00:24:18] maplebed: so which parts of rewrite.py *do* encode properly? [00:24:50] I'm not really sure... [00:25:03] for the most part, 'proper' never really mattered. [00:25:09] it's just 'the same going out as coming in' that matters. [00:25:10] it writes the file correctly write? [00:25:20] *right [00:25:48] There are plenty of files currently stored in swift with &s in the name, if that's what you mean. [00:26:06] ok, good [00:29:05] AaronSchulz: part of me wants to just change line 129 from upcopy = opener.open(reqorig.url) to upcopy = opener.open(urllib2.quote(reqorig.url)) [00:30:21] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:30:57] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:29] AaronSchulz: I'm going to verify the eqiad test cluster exhibits this bug so we can use it to test fixes. [00:33:43] New patchset: Lcarr; "upping nagios checks to 512 concurrent checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2484 [00:33:49] ok [00:34:38] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2484 [00:34:39] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2484 [00:40:02] maplebed: maybe quote() isn't that bad of an idea [00:41:04] I've got my negative test for eqiad swift done; now doing the positive test. [00:45:26] New patchset: Asher; "http setup for gdash" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2485 [00:46:26] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2485 [00:46:26] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2485 [00:46:28] ok, AaronSchulz I've got two urls that we can use to test on the eqiad cluster. [00:47:27] curl -o /tmp/swift-withamp http://copper.wikimedia.org:8080/wikipedia/commons/thumb/a/a4/Kiwi_%26_snarfle_tooth.png/220px-Kiwi_%26_snarfle_tooth.png [00:47:27] and [00:47:27] curl -o swift-noamp http://copper.wikimedia.org:8080/wikipedia/commons/thumb/0/0e/The_Wiki_Signal.jpg/220px-The_Wiki_Signal.jpg [00:48:13] I'm going to try throwincg the quote in theer and just see what happens. [00:49:15] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.889 seconds [00:49:52] maplebed: instead of 'The resource could not be found', I wish it had the uri :/ [00:50:13] if you tell me where to change the error message in rewrite, I'll throw that in. [00:50:53] in the mean time though, I've got tcpdump showing me what's going in and out of copper. [00:51:23] whee!!! [00:51:33] ValueError: unknown url type: http%3A//ms5.pmtpa.wmnet/wikipedia/commons/thumb/0/0e/The_Wiki_Signal.jpg/222px-The_Wiki_Signal.jpg [00:51:49] ha. it tried to urlencode the whole thing. [00:51:52] ::sigh:: [00:52:06] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.459 seconds [00:55:14] except urllib2.HTTPError,status: [00:55:15] if status == 404: [00:55:22] that seems funky [00:56:22] seems like it should check status.code [01:05:50] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:10] robla: looks like http://docs.python.org/library/urlparse.html is what you were looking for [01:07:28] ah, there we go [01:08:32] yeah, using that should keep you from accidentally escaping parts that shouldn't be escaped [01:08:50] e.g. someone putting a ":80" in the URL [01:08:54] FF crashed :( [01:09:15] dunno about performance though [01:11:14] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.968 seconds [01:11:34] New patchset: Asher; "deleted a character too many" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2486 [01:11:43] maplebed: should I change the "if status == 404:" line? [01:11:55] I'm hesitant [01:11:55] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2486 [01:12:01] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2486 [01:12:02] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2486 [01:12:03] http://docs.python.org/library/urllib2.html#urllib2.HTTPError [01:12:31] it's comparing an obj to a number...no wonder it always goes down the 'else' [01:13:27] what do you mean it always goes down the else logic? [01:13:43] resp = webob.exc.HTTPNotFound('Unexpected error %s' % status) [01:13:45] self.handle404 is called successfully on 404s. [01:13:47] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:57] this is around line 131 [01:14:08] within handle404 [01:14:31] oh, sorry, I was looking at line 239 [01:14:54] sleeping now. good luck [01:15:05] thanks apergos [01:15:09] AaronSchulz: +1 changing it. [01:15:17] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:28] it can also dump the url [01:16:26] reqorig.url specifically [01:18:37] AaronSchulz: http://copper.wikimedia.org:8080/wikipedia/commons/thumb/a/a4/Kiwi_%26_snarfle_tooth.png/222px-Kiwi_%26_snarfle_tooth.png [01:19:01] pnobably not something we want to leave in, but good for now. [01:23:14] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.533 seconds [01:26:23] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [01:27:17] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:17] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [01:29:59] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.961 seconds [01:31:02] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [01:34:02] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:38] maplebed: any luck? [01:35:58] I'm feeling stupid and fighting the tuple returned by urlsplit() [01:37:38] TimStarling: http://www.google.no/patents/US5504900 [01:41:05] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.841 seconds [01:44:50] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.761 seconds [01:49:11] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 193 seconds [01:49:11] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:20] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 202 seconds [01:49:29] PROBLEM - MySQL Slave Delay on db1038 is CRITICAL: CRIT replication delay 200 seconds [01:50:05] PROBLEM - MySQL Replication Heartbeat on db1038 is CRITICAL: CRIT replication delay 235 seconds [01:50:23] PROBLEM - MySQL Replication Heartbeat on db1004 is CRITICAL: CRIT replication delay 253 seconds [01:50:41] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [01:52:02] PROBLEM - DPKG on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:52:20] PROBLEM - MySQL Slave Running on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:53:14] RECOVERY - DPKG on db42 is OK: All packages OK [01:53:32] RECOVERY - MySQL Slave Running on db42 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [01:53:41] PROBLEM - mysqld processes on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:53:41] PROBLEM - Disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:54:17] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:35] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.069 second response time [01:54:53] RECOVERY - Disk space on db42 is OK: DISK OK [01:55:02] RECOVERY - mysqld processes on db42 is OK: PROCS OK: 1 process with command name mysqld [01:55:56] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.535 seconds [01:56:23] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 602s [01:57:26] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 666s [01:58:47] RECOVERY - MySQL Replication Heartbeat on db1034 is OK: OK replication delay 30 seconds [01:59:05] RECOVERY - MySQL Slave Delay on db1034 is OK: OK replication delay 0 seconds [01:59:59] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:27] TimStarling: I feel like I've been experiencing more outdated pages on en.wikipedia.org lately (on http, when I typically use https logged in). The cache doesn't seem to be purging fast enough. [02:11:40] http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard is showing an outdated RFA/RFB table. [02:11:52] when you're logged in? [02:12:01] > Last updated by SoxBot (talk) at 23:30, 8 February 2012 (UTC) [02:12:03] I'm logged out of http. [02:12:22] https://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard gives ... Manually updated by Cyberpower678 (talk) (Online) at 01:45, 10 February 2012 (UTC) [02:12:35] So it's been four days without the cache of that page expiring or being reset? [02:12:42] Err, two. [02:12:46] Subtraction is hard. [02:12:55] so was the page itself edited in that time, or just templates? [02:13:11] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.696 seconds [02:13:18] Page was last edited February 9. [02:13:33] Maybe twelve hours ago? [02:13:48] The template was also edited. [02:14:14] Quite a bit: http://en.wikipedia.org/w/index.php?title=User:X!/RfX_Report&action=history [02:14:32] Anyway, it's just anecdotal, but the frequency seems to be up. [02:16:57] it shouldn't really happen at all for the canonical URL [02:17:14] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:35] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.164 seconds [02:18:38] !log LocalisationUpdate completed (1.18) at Fri Feb 10 02:18:37 UTC 2012 [02:18:39] Logged the message, Master [02:18:44] when did you first notice it? [02:21:40] Double escaping bug when redirecting to commons: http://commons.wikipedia.org/w/api.php?titles=%C5%8C [02:23:26] clearer example: http://commons.wikipedia.org/w/index.php?title=%C5%8C [02:27:32] TimStarling: I think I noticed it within the last few days. Probably not more than a week ago. [02:28:02] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:35] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.168 seconds [02:33:22] sq66:3128 has the old object cached, but its headers have a recent Last-Modified date [02:33:43] it's kind of odd [02:35:42] is what's giving me the old HTML for http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard [02:36:03] Still doing it. I would've thought someone in here would've purged the page by now. ;-) [02:36:57] ah, never mind, I was actually seeing the new object [02:37:16] yeah it's nice when I get a few minutes of debugging time before someone breaks my test case [02:37:29] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:54] can you get the response headers somehow? [02:37:59] maybe you have firebug or something? [02:38:13] Will curl work? [02:38:23] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 59s [02:38:29] possibly, I'll give you a command line [02:38:50] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.560 seconds [02:38:50] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:39:23] curl -vvv-H'Accept-Encoding: gzip, deflate' 'http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard' | zcat | grep footer-info-lastmod [02:39:30] curl -vvv -H'Accept-Encoding: gzip, deflate' 'http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard' | zcat | grep footer-info-lastmod [02:39:46] you have to send the Accept-Encoding header or else you'll get a different cache object from squid [02:40:06] just pastebin the whole output [02:40:20] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.684 seconds [02:40:25] http://p.defau.lt/?qbheLZpNf99fS5PdE2rBfQ [02:40:59] Chrome has "This page was last modified on 9 February 2012 at 15:53." at the bottom of the page as well. [02:41:13] That's the timestamp of the last direct edit to the page. [02:41:23] yes, that's what I get too [02:41:52] what browser are you using to get the incorrect response? [02:42:00] Google Chrome/OS X [02:42:11] 16.0.912.77 [02:42:52] with last modified 8 feb? but you just said chrom gave you 9 feb [02:42:53] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:36] Chrome gives me "This page was last modified on 9 February 2012 at 15:53." at the bottom and ... "Last updated by SoxBot (talk) at 23:30, 8 February 2012 (UTC)" below the table (from the template) at http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard [02:44:28] curl is returning the same "This page was last modified on 9 February 2012 at 15:53." with http://en.wikipedia.org/wiki/Wikipedia:Bureaucrats%27_noticeboard [02:45:52] ok... [02:45:53] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:29] What I've been noticing lately (in general) is that someone links to http://en.wikipedia.org/something, I click it, then the content is outdated. When I go to https://... logged in, the content is up-to-date. I understand the caching infrastructure (in broad terms), but it seems like the cache isn't expiring quickly enough (after template edits, e.g.). [02:49:10] New patchset: Bhartshorne; "fixing bug where images with an ampersand \& in the name fail to load through swift" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2487 [02:50:27] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2487 [02:50:28] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2487 [02:52:21] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.014 seconds [02:52:21] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:52:47] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:54:01] Joan: that template is used on hundreds of pages, so purging will be done via the job queue [02:54:53] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [02:55:20] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [02:55:32] TimStarling: I guess it's only been five hours or so. I'm not sure how long it's supposed to take. I usually just visit pages logged in. [02:56:23] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:34] it's possible that purging from the job queue is broken [02:56:52] bbl [02:57:20] too bad rillke isn't still here. [03:02:25] Hey [03:04:20] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.291 seconds [03:07:29] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.565 seconds [03:11:57] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:03] RECOVERY - MySQL Replication Heartbeat on db1002 is OK: OK replication delay 0 seconds [03:14:30] RECOVERY - MySQL Slave Delay on db1002 is OK: OK replication delay 0 seconds [03:16:09] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:17:03] RECOVERY - MySQL Slave Delay on db1038 is OK: OK replication delay 0 seconds [03:17:30] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.167 seconds [03:18:15] RECOVERY - MySQL Replication Heartbeat on db1038 is OK: OK replication delay 0 seconds [03:19:18] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 211 seconds [03:19:54] PROBLEM - MySQL Slave Delay on db1004 is CRITICAL: CRIT replication delay 247 seconds [03:21:15] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 0 seconds [03:21:15] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [03:21:33] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:48] PROBLEM - MySQL Replication Heartbeat on db1005 is CRITICAL: CRIT replication delay 212 seconds [03:24:15] PROBLEM - MySQL Slave Delay on db1005 is CRITICAL: CRIT replication delay 239 seconds [03:24:33] PROBLEM - MySQL Replication Heartbeat on db1021 is CRITICAL: CRIT replication delay 257 seconds [03:24:33] PROBLEM - MySQL Slave Delay on db1021 is CRITICAL: CRIT replication delay 257 seconds [03:25:54] RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay 0 seconds [03:25:54] RECOVERY - MySQL Slave Delay on db1021 is OK: OK replication delay 0 seconds [03:26:39] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay 0 seconds [03:27:06] RECOVERY - MySQL Slave Delay on db1005 is OK: OK replication delay 0 seconds [03:28:09] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.602 seconds [03:33:33] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:27] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 182 seconds [03:40:27] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 223 seconds [03:43:00] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 217 seconds [03:43:09] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 204 seconds [03:44:12] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.341 seconds [03:46:54] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [03:48:24] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:33] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.855 seconds [03:49:54] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 210 seconds [03:49:54] PROBLEM - MySQL Replication Heartbeat on db1003 is CRITICAL: CRIT replication delay 210 seconds [03:51:51] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:51:51] PROBLEM - Disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:09] PROBLEM - mysqld processes on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:09] PROBLEM - MySQL disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:09] PROBLEM - DPKG on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:18] PROBLEM - MySQL Slave Running on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:18] PROBLEM - MySQL Idle Transactions on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:27] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 210 seconds [03:52:27] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.904 seconds [03:52:27] PROBLEM - RAID on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:52:27] PROBLEM - MySQL Slave Delay on db1003 is CRITICAL: CRIT replication delay 211 seconds [03:52:37] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:03] RECOVERY - Disk space on db42 is OK: DISK OK [03:53:21] RECOVERY - mysqld processes on db42 is OK: PROCS OK: 1 process with command name mysqld [03:53:21] RECOVERY - MySQL disk space on db42 is OK: DISK OK [03:53:21] RECOVERY - DPKG on db42 is OK: All packages OK [03:53:30] RECOVERY - MySQL Slave Running on db42 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [03:53:30] RECOVERY - MySQL Idle Transactions on db42 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:53:39] RECOVERY - RAID on db42 is OK: OK: State is Optimal, checked 2 logical device(s) [03:54:15] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [03:56:30] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:39] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.879 seconds [04:00:41] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:44] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.892 seconds [04:04:44] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.506 seconds [04:05:56] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:50] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:16:35] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.941 seconds [04:16:53] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.832 seconds [04:20:47] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:20:56] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.761 seconds [04:27:32] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.293 seconds [04:31:26] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:35:20] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.519 seconds [04:35:47] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:29] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.118 seconds [04:42:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:53] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.459 seconds [04:46:17] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:39] RECOVERY - MySQL Replication Heartbeat on db1004 is OK: OK replication delay 0 seconds [04:47:39] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.106 seconds [04:47:47] RECOVERY - MySQL Slave Delay on db1004 is OK: OK replication delay 0 seconds [04:47:56] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:29] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 6.039 seconds [04:51:50] PROBLEM - MySQL Idle Transactions on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:52:08] PROBLEM - MySQL Recent Restart on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:52:26] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:52:26] PROBLEM - MySQL Slave Running on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:52:53] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:53:50] RECOVERY - MySQL Slave Running on db42 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [04:54:17] RECOVERY - MySQL Idle Transactions on db42 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:54:26] RECOVERY - MySQL Recent Restart on db42 is OK: OK 4250487 seconds since restart [04:54:44] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [04:55:02] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [04:58:38] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [04:58:47] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [05:17:33] * jeremyb waves apergos [05:43:11] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:44:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:05] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.027 seconds [05:47:05] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.008 seconds [05:51:40] PROBLEM - MySQL Slave Running on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:51:40] PROBLEM - Disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:51:58] PROBLEM - MySQL disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:52:34] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:52:52] PROBLEM - DPKG on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:52:52] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:52:52] PROBLEM - mysqld processes on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:53:01] RECOVERY - Disk space on db42 is OK: DISK OK [05:53:01] RECOVERY - MySQL Slave Running on db42 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [05:53:01] RECOVERY - MySQL disk space on db42 is OK: DISK OK [05:53:46] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 1 seconds [05:54:04] RECOVERY - mysqld processes on db42 is OK: PROCS OK: 1 process with command name mysqld [05:54:04] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [05:54:04] RECOVERY - DPKG on db42 is OK: All packages OK [06:06:49] PROBLEM - MySQL Replication Heartbeat on db1019 is CRITICAL: CRIT replication delay 198 seconds [06:07:52] PROBLEM - MySQL Slave Delay on db1019 is CRITICAL: CRIT replication delay 262 seconds [06:08:10] RECOVERY - MySQL Replication Heartbeat on db1019 is OK: OK replication delay 0 seconds [06:09:13] RECOVERY - MySQL Slave Delay on db1019 is OK: OK replication delay 11 seconds [06:19:52] PROBLEM - MySQL Slave Delay on db1019 is CRITICAL: CRIT replication delay 236 seconds [06:20:01] PROBLEM - MySQL Replication Heartbeat on db1019 is CRITICAL: CRIT replication delay 243 seconds [06:22:34] RECOVERY - MySQL Slave Delay on db1019 is OK: OK replication delay 0 seconds [06:22:34] RECOVERY - MySQL Replication Heartbeat on db1019 is OK: OK replication delay 1 seconds [06:29:47] * apergos waves generally [06:35:00] zzz =__= [06:51:50] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:50] PROBLEM - MySQL Slave Running on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:50] PROBLEM - Disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:52] PROBLEM - MySQL disk space on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:52] PROBLEM - Full LVS Snapshot on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:01] PROBLEM - MySQL Idle Transactions on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:06:00] New patchset: Mark Bergsma; "Decommission sq38, sq46, sq47" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2488 [12:06:20] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2488 [12:07:09] New patchset: Mark Bergsma; "Decommission sq38, sq46, sq47" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2488 [12:07:27] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2488 [12:07:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2488 [12:07:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2488 [12:17:54] New patchset: Mark Bergsma; "Decommission sq31" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2489 [12:18:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2489 [12:18:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2489 [12:18:26] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2489 [12:18:27] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2489 [12:25:35] New patchset: Mark Bergsma; "Decommission sq35" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2490 [12:25:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2490 [12:26:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2490 [12:26:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2490 [13:32:09] New patchset: Mark Bergsma; "Remove old nagios host/service groups for squid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2491 [13:32:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2491 [13:32:41] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2491 [13:32:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2491 [14:13:44] New patchset: Mark Bergsma; "Remove decommissioned servers from api list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2492 [14:14:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2492 [14:23:21] New patchset: Mark Bergsma; "Update more graphs for eqiad" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2493 [14:23:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2493 [14:24:23] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2492 [14:24:23] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2492 [14:25:06] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2493 [14:25:07] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2493 [14:36:20] New patchset: Mark Bergsma; "Corrections" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2494 [14:36:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2494 [14:36:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2494 [14:36:49] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2494 [14:40:04] New patchset: Hashar; "(bug 34141) notify jenkins on ANY merge" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2495 [14:40:27] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2495 [14:47:45] New patchset: Mark Bergsma; "Add mobile graphs, corrections" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2496 [14:48:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2496 [14:48:24] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2496 [14:48:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2496 [14:50:54] New patchset: Mark Bergsma; "Fix color spec" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2497 [14:51:19] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2497 [14:51:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2497 [14:51:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2497 [14:52:53] New review: Demon; "Why are we doing this on merge? Shouldn't this go on push (before the merge happens)?" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/2495 [14:55:15] New patchset: Mark Bergsma; "Automatically clear the cache on Torrus config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2498 [14:55:44] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2498 [14:55:45] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2498 [14:56:37] New patchset: Mark Bergsma; "Automatically clear the cache on Torrus config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2498 [14:57:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2498 [14:57:21] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2498 [14:57:22] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2498 [14:59:27] New patchset: Mark Bergsma; "Fix api colors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2499 [14:59:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2499 [14:59:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2499 [15:01:15] New patchset: Mark Bergsma; "Use LINE1 for service times" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2500 [15:01:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2500 [15:01:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2500 [15:01:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2500 [15:01:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2500 [15:03:15] New patchset: Mark Bergsma; "Really fix API colors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2501 [15:03:44] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2501 [15:03:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2501 [15:03:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2501 [15:07:16] New patchset: Mark Bergsma; "More color fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2502 [15:07:44] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2502 [15:07:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2502 [15:07:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2502 [15:14:51] New patchset: Mark Bergsma; "Turn on KeepAlive on application servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2504 [15:15:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2504 [15:16:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2504 [15:16:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2504 [15:40:27] New patchset: Mark Bergsma; "/etc/init.d/torrus-common restart has proven unreliable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2505 [15:40:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2505 [15:41:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2505 [15:41:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2505 [16:47:17] New review: Hashar; "That basically copy the way subversion works for now." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/2495 [16:54:21] Hey folks, just a heads up that we'll be doing IRC office hours in about 10 minutes with the new "legal and community advocacy" dept. at the WMF -- https://meta.wikimedia.org/wiki/Legal_and_Community_Advocacy [16:59:10]