[00:13:38] There are reports of geoiplookup.wikimedia.org being slow. [00:15:54] don't use geoiplookup.wikimedia.org, then? [00:16:08] doesn't bits support this now? [00:16:23] we moved this to bits to avoid extra DNS lookups [00:16:44] and in general the major slowness with geoiplookup is the DNS lookups [00:16:45] I would be guessing that end users aren't magically choicing to use it [00:16:59] if not then a developer is doing something stupid [00:17:10] according to the bug, it was completely broken for au [00:17:11] could be a gadget [00:17:50] geoiplookup hits varnish, which sends the data back. it's simple and in general it's difficult for it to be broken without other things also being broken [00:17:56] yeap, I get no responce from it in au [00:18:04] tracerout? [00:18:06] *route [00:19:14] Ryan_Lane: https://bugzilla.wikimedia.org/show_bug.cgi?id=55907#c1 [00:19:34] There's a suggestion that https://gerrit.wikimedia.org/r/#/c/90103/ could be related. [00:19:36] https://bugzilla.wikimedia.org/show_bug.cgi?id=55907 [00:19:46] p858snake|l: I win. [00:20:05] i shall let you claim this one Elsie [00:20:10] Thank you, sir. [00:20:29] what code is using geolookup.wikimedia.org? [00:20:34] It's unclear. [00:20:37] I couldn't find anything. [00:20:50] obviously someone is reporting this issue [00:20:54] Several people are. [00:20:55] has anyone asked them why? [00:21:19] realistically, though, this is a routing issue [00:21:23] Right. [00:21:33] i would assume that they just notice that their browser is having the issue [00:21:44] There are two questions here, Ryan is right. [00:21:44] yeah, we need to figure out what's referencing this ;) [00:21:50] and kill those references [00:21:56] but past that, this is a routing issue [00:21:57] and its across projects as well [00:21:57] I think bits.wikimedia.org/geoiplookup is also causing issues for users. [00:22:05] Hmmmmm. [00:22:09] CentralNotice, maybe. [00:24:53] Ryan_Lane: When did it get moved to the ? [00:25:07] It used to be below so that latency or other issues wouldn't cause problems... [00:25:19] geoiplookup has always been in the head [00:25:52] we moved to bits.wm.o/geoiplookup to avoid extra latency from DNS lookups [00:26:05] https://bugzilla.wikimedia.org/show_bug.cgi?id=37727 [00:26:23] ah, I see [00:26:33] I hadn't realized they ever moved it out of head [00:26:51] This seems to be region-specific. [00:26:54] Australia mostly. [00:27:15] which is why I'd imagine this is a routing issue [00:27:17] Ryan_Lane: my trace https://bugzilla.wikimedia.org/show_bug.cgi?id=55907#c5 [00:27:46] Both die at tinet.net. [00:27:54] there's not much I can do about this. LeslieCarr or mark need to look at it [00:28:12] ah, this is to ulsfo [00:28:18] I wonder if ulsfo is failing [00:28:36] https://gerrit.wikimedia.org/r/#/c/90103/ [00:28:41] One of the comments pointed at that. [00:28:47] nope: http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Bits%20caches%20ulsfo&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1382228895&g=network_report&z=large [00:28:47] Though I have no idea if it's related. [00:29:10] cc'ed leslie onto the bug, no idea whats mark bz is [00:29:19] mark@nedworks [00:29:19] I guess I can page leslie [00:29:32] hm. she's at RIPE [00:29:57] all of our networking folk are in europe right now [00:30:15] in roughly the same timezone (2-3AM) [00:30:33] https://en.wikipedia.org/wiki/Wikipedia:VPT#geoiplookup.wikimedia.org [00:30:45] let me take a look at watchmouse [00:31:00] https://en.wikipedia.org/wiki/Wikipedia_talk:Geonotice#Shut_it_off.3F.3F [00:31:46] I could also move the traffic away from ulsfo [00:32:14] It doesn't seem particularly widespread as a problem. [00:32:41] It's high priority, but not urgent, I think. [00:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 00:32:48 UTC 2013 [00:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [00:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 00:34:04 UTC 2013 [00:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [00:35:01] ugh, this will move upload and bits to ulsfo [00:35:03] err [00:35:06] back to eqiad [00:35:15] * Ryan_Lane checks watchmouse first [00:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 00:35:39 UTC 2013 [00:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [00:39:02] uuuuggggh. why is it so fucking hard to find the world map view of access in this tool? [00:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 00:41:46 UTC 2013 [00:42:04] oh. this may be a misconfiguration [00:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:21] I can't ping 198.35.26.116 [00:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 00:42:42 UTC 2013 [00:43:06] indeed. it's not bound anywehre [00:43:08] *anywhere [00:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:15] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 00:44:07 UTC 2013 [00:44:50] in the other datacenters geoiplookup-lb has the same ip as bits [00:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [00:45:05] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 00:44:58 UTC 2013 [00:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] (03PS1) 10Ryan Lane: Point geoiplookup-lb for ulsfo to bits-lb in ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/90732 [00:47:40] (03PS2) 10Ryan Lane: Point geoiplookup-lb for ulsfo to bits-lb in ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/90732 [00:48:50] (03CR) 10Ryan Lane: [C: 032] Point geoiplookup-lb for ulsfo to bits-lb in ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/90732 (owner: 10Ryan Lane) [00:49:39] ok. merged [00:50:07] hm. that's a 1 hour cache [00:50:53] well, so in an hour this will be resolved [00:51:00] for people who already have the issue [00:51:50] (03PS1) 10Physikerwelt: Mathoid service [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 [00:52:46] Ryan_Lane: Thanks! [00:52:58] p858snake|l: Wanna up the bug? [00:53:00] update [00:53:03] Verbs are hard. :-( [00:53:07] I wonder if this is worth failing over to eqiad until then [00:53:08] maybe so [00:53:13] I can do that too. give me a second [00:53:24] It's been happening for the past day or so at least, I think. [00:53:40] I'm not sure another hour will drive anyone crazy. [00:54:41] (03PS1) 10Ryan Lane: Temporarily move OC traffic to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/90734 [00:55:16] just because we've been giving people shitty service for a couple days doesn't mean we should continue to do so for another hour ;) [00:55:39] (03CR) 10Ryan Lane: [C: 032] Temporarily move OC traffic to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/90734 (owner: 10Ryan Lane) [00:56:04] in an hour I'll move the traffic back [00:57:38] p858snake|l: is it working for you now? [00:58:47] should be. my Melbourne based traceroute gives me bits in eqiad [00:59:40] not yet, let me traceroute [00:59:49] telstra may be being stupid again [01:01:34] tracert still dies at the same place, but that might be a dns cache issue on my end somewhere [01:03:07] for geolookup.wikimedia.org? [01:03:08] hm [01:03:26] actually it just errored out differntly [01:03:27] it should be within minutes [01:03:48] which area are you in? [01:04:00] 27 * * wikimedia-foundation-gw.ip4.tinet.net [173.241.128.54] r [01:04:01] eports: Destination net unreachable. [01:04:01] Trace complete. [01:04:03] brisbane [01:04:16] that's a really odd IP [01:05:24] my traceroute works [01:05:29] from brisbane [01:07:16] and Melbourne, and Perth [01:07:19] sydney timed out [01:07:39] ah. didn't that time [01:07:54] none of them are going through tinet, though [01:08:10] do you know what isps they start off with? [01:08:31] Sydney goes through VOCUS.net.au [01:08:56] not sure for brisbane [01:09:24] p858snake|l: can you run this: dig geoiplookup.wikimedia.org [01:09:24] ? [01:09:43] or host, or something else that'll tell me which ip you're getting :) [01:09:45] i'm on windows [01:09:59] nslookup geoiplookup.wikimedia.org [01:10:45] Name: geoiplookup-lb.eqiad.wikimedia.org [01:10:45] Address: 208.80.154.234 [01:10:45] Aliases: geoiplookup.wikimedia.org [01:11:16] ugh, working now [01:11:22] i wonder if chrome was at fault [01:11:56] wait, no i'm being stupid there [01:12:30] and indeed. fucking centralnotice is using geoiplookup.wikimedia.org rather than bits.wikimedia.rog [01:14:05] * Ryan_Lane tracks that down [01:17:31] Is it? [01:17:43] I looked around Meta-Wiki, but didn't see anything incriminating. [01:17:58] Didn't check the extension itself. [01:20:51] it's in the extension [01:20:55] I'm going to push in a patch [01:21:02] though, maybe I see the problem [01:21:13] actually, I almost definitely know why it's this way [01:21:26] bits.wm.o supports ipv6 and geoiplookup.wm.o does not [01:21:33] which forces clients over ipv4 [01:21:40] that's intended [01:25:09] so, yeah, no fixing that for now :( [01:26:35] Oh well. [01:27:20] it's working for me now, thanks ryan [01:27:39] closedmouth: great to hear [01:27:39] yw [01:27:44] Yay. [01:28:51] well, I'm glad I didn't page anyone. would have been crappy to wake someone up for that :D [01:30:26] Ryan_Lane: is there a bug filed about that? [01:30:32] probably [01:30:46] it's actually in the centralnotice code comments [01:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 01:32:40 UTC 2013 [01:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [01:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 01:33:45 UTC 2013 [01:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [01:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 01:35:41 UTC 2013 [01:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [01:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 01:41:47 UTC 2013 [01:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:46] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 01:42:42 UTC 2013 [01:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [01:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 01:43:47 UTC 2013 [01:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 01:44:42 UTC 2013 [01:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [01:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [01:53:20] (03CR) 10Chad: "Why don't we just rename MediaWiki to Wikioid and be done with it? ;-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [02:01:39] !log LocalisationUpdate completed (1.22wmf21) at Sun Oct 20 02:01:39 UTC 2013 [02:02:01] Logged the message, Master [02:02:20] !log LocalisationUpdate completed (1.22wmf22) at Sun Oct 20 02:02:20 UTC 2013 [02:02:33] Logged the message, Master [02:07:08] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 20 02:07:08 UTC 2013 [02:07:31] Logged the message, Master [02:19:05] PROBLEM - Disk space on copper is CRITICAL: DISK CRITICAL - free space: / 356 MB (3% inode=90%): [02:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 02:32:44 UTC 2013 [02:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 02:34:00 UTC 2013 [02:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 02:35:40 UTC 2013 [02:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [02:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 02:41:41 UTC 2013 [02:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 02:42:41 UTC 2013 [02:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [02:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 02:43:41 UTC 2013 [02:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [02:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 02:44:47 UTC 2013 [02:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [02:58:31] (03PS1) 10Ryan Lane: localssl: listen on both ipv6 and ipv4 sockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/90738 [03:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 03:32:46 UTC 2013 [03:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [03:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 03:34:01 UTC 2013 [03:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [03:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 03:35:37 UTC 2013 [03:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [03:42:05] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 03:41:58 UTC 2013 [03:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 03:42:43 UTC 2013 [03:43:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [03:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [03:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 03:43:43 UTC 2013 [03:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [03:45:05] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 03:44:59 UTC 2013 [03:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [04:04:25] (03CR) 10Revi: [C: 031] (bug 55909) Add an alias for NS_USER on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90708 (owner: 10Odder) [04:11:08] ori-l: Don't worry, I found a way to deflect your getSource pastebin. ;-) [04:11:43] hrm? [04:11:51] deflect? [04:11:57] Don't worry about it. [04:12:12] I'm working on mw.inspect right now, actually [04:12:27] like, AS WE SPEAK [04:12:28] I'm not. [04:12:39] I'm upgrading my phone to iOS 7. [04:12:47] And trying to close browser tabs. [04:12:55] And baking brownies. [04:12:59] right-click on the tab [04:13:22] you'll thank me later [04:13:41] Heh. [04:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 04:32:50 UTC 2013 [04:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [04:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 04:34:00 UTC 2013 [04:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [04:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 04:35:40 UTC 2013 [04:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [04:37:55] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [04:40:55] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [04:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 04:41:42 UTC 2013 [04:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 04:42:42 UTC 2013 [04:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 04:43:43 UTC 2013 [04:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:55] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [04:44:46] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 04:44:43 UTC 2013 [04:44:55] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [04:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:57] Elsie: You have iOS 7? [05:01:04] Did it download/install? :) [05:12:45] I just did it. [05:12:50] It was pretty easy. [05:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 05:32:41 UTC 2013 [05:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [05:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 05:33:46 UTC 2013 [05:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [05:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 05:35:37 UTC 2013 [05:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 05:41:45 UTC 2013 [05:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [05:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 05:42:45 UTC 2013 [05:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [05:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 05:43:50 UTC 2013 [05:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 05:44:40 UTC 2013 [05:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [05:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 06:32:46 UTC 2013 [06:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 06:33:46 UTC 2013 [06:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 06:35:41 UTC 2013 [06:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [06:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 06:41:42 UTC 2013 [06:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [06:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 06:42:43 UTC 2013 [06:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [06:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 06:43:53 UTC 2013 [06:44:15] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [06:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 06:44:53 UTC 2013 [06:45:05] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [06:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [06:59:55] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:05] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:55] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:00:55] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:08:05] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:05] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:55] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:55] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:16:55] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:18:55] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [07:19:45] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [07:21:55] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:05] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:05] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:45] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 07:32:46 UTC 2013 [07:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [07:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 07:33:46 UTC 2013 [07:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [07:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 07:35:41 UTC 2013 [07:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [07:37:45] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:37:55] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:37:55] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [07:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 07:41:43 UTC 2013 [07:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [07:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 07:42:43 UTC 2013 [07:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [07:43:45] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 07:43:43 UTC 2013 [07:43:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [07:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 07:44:44 UTC 2013 [07:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 08:32:45 UTC 2013 [08:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [08:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 08:33:45 UTC 2013 [08:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [08:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 08:35:41 UTC 2013 [08:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [08:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 08:41:47 UTC 2013 [08:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [08:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 08:42:42 UTC 2013 [08:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 08:43:47 UTC 2013 [08:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 08:44:52 UTC 2013 [08:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [09:17:28] (03PS2) 10Nemo bis: Simplify misc::maintenance::update_special_pages a bit [operations/puppet] - 10https://gerrit.wikimedia.org/r/90117 [09:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 09:32:43 UTC 2013 [09:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [09:33:45] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 09:33:43 UTC 2013 [09:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 09:35:39 UTC 2013 [09:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [09:42:05] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 09:41:55 UTC 2013 [09:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [09:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 09:42:46 UTC 2013 [09:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [09:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 09:43:51 UTC 2013 [09:44:55] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 09:44:46 UTC 2013 [09:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [09:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [10:32:45] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 10:32:44 UTC 2013 [10:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [10:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 10:34:00 UTC 2013 [10:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [10:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 10:35:40 UTC 2013 [10:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 10:41:52 UTC 2013 [10:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 10:42:42 UTC 2013 [10:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 10:43:47 UTC 2013 [10:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 10:44:42 UTC 2013 [10:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [10:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:45] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [11:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 11:32:45 UTC 2013 [11:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [11:34:05] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 11:34:01 UTC 2013 [11:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [11:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 11:35:41 UTC 2013 [11:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [11:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 11:41:43 UTC 2013 [11:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [11:42:45] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 11:42:43 UTC 2013 [11:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [11:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 11:43:49 UTC 2013 [11:44:45] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 11:44:44 UTC 2013 [11:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [11:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 12:32:50 UTC 2013 [12:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 12:33:45 UTC 2013 [12:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [12:35:05] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [12:35:55] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 12:35:45 UTC 2013 [12:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [12:38:15] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:55] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 12:41:47 UTC 2013 [12:42:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [12:42:55] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Sun Oct 20 12:42:47 UTC 2013 [12:43:45] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:55] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Sun Oct 20 12:43:48 UTC 2013 [12:44:55] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [12:45:05] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Sun Oct 20 12:45:03 UTC 2013 [12:45:15] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [12:57:05] PROBLEM - Disk space on cp1063 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 16427 MB (5% inode=99%): /srv/sdb3 12412 MB (3% inode=99%): [12:57:15] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:05] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [13:32:55] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Sun Oct 20 13:32:51 UTC 2013 [13:33:05] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [13:33:55] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Sun Oct 20 13:33:46 UTC 2013 [13:34:45] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [13:35:45] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Sun Oct 20 13:35:37 UTC 2013 [13:36:25] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [13:41:45] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Sun Oct 20 13:41:43 UTC 2013 [13:42:05]