[00:15:28] AaronSchulz: No idea I assume? [00:19:16] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66026 [00:25:11] dr0ptp4kt: hey [00:25:27] hey there, paravoid. thanks for the review. [00:25:49] dr0ptp4kt: when I visit en.zero I'm getting a sorry banner which references a random IP [00:26:00] cached I'm guessing [00:26:15] paravoid, you able to do a quick google hangout? [00:26:21] not now [00:26:58] paravoid, ok. would you please send me the spoofed user agent and X-CS (and any other spoofed headers) plus the response? [00:27:11] just go to http://en.zero.wikipedia.org/wiki/Main_Page [00:27:28] nothing's spoofed [00:27:31] just my browser :) [00:27:58] paravoid, oh, i see what you mean. try http://en.zero.wikipedia.org/wiki/Main_Page?cachebuster=100 [00:28:09] I know that it works if you bypass caches [00:28:48] paravoid, do you know if there were any changes in cache variance lately? [00:29:32] dr0ptp4kt, come to think of it, that's the expected behaviour - if you visit zero from anywhere, it should tell you that you don't have zero, and tells you your IP addr [00:29:50] but because we cache everything, it will return the previously-cached banner [00:30:10] wasn't that the previous behaviour? [00:31:34] yurik, the "strange" (?) thing is that the IP address that's shown isn't mine. i seem to recall it would usually echo back my own ip address in the past, not something that was cached. [00:31:53] how was that possible? [00:32:09] it should have cached the banner including the ip [00:32:12] yurik, i wonder if it was just lucky happenstance due to x-device [00:32:41] one thing we could do is not to cache anything for zero [00:32:45] …yurik, coupled with the rarity of "normal" users hitting zero on their own. [00:32:48] unless x-cs is set [00:36:39] yurik, i think for the "Sorry" message we could also replace the page render with a 302 to a non-cacheable page. that way we can retain the cached pages for perf. [00:36:56] the non-cacheable page could display the "Sorry" message and the user's current IP address [00:37:02] yurik^ [00:37:22] how do you propose to do that? [00:37:38] not sure what you mean by non-cached page [00:38:18] paravoid, that patch to varnish didn't work for some reason - people at the office can't spoof their IPs :( [00:38:43] mark said that its due to IPv4 vs v6, but dr0ptp4kt couldn't spoof from the office [00:38:49] and was getting IPv4 error message [00:42:15] yurik, my thought is that if the user is redirected, for example, to /wiki/Special:ZeroRatedMobileAccess?notsupported=yes and if we can make //wiki/Special:ZeroRatedMobileAccess?notsupported=yes override the X-Vary-Options header, then we can inform the cache to always hit the origin for that path, meaning it should be possible to return the up-to-date IP address for that path. another less lovely option is to redirect the user to a page [00:42:16] a random value pegged into the redirect. but that results in unnecessary items cached. [00:42:56] er, supported=false :) [00:43:05] not, unsupported=true :) [00:46:36] agree about special page, not hard to make [00:46:47] dr0ptp4kt, but i don't know about the variance [00:46:56] variance on what? [00:47:07] i think we would need to set "don't cache" at all [00:47:43] New review: Dr0ptp4kt; "Going through ZeroRatedMobileAccess, the hook is being called for mobile sites anyway. The differenc..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [00:48:16] yurik, agreed, we'd have to tell it to never cache. [00:49:01] yurik, that's actually an interesting problem, which is similar to the s-maxage card. [00:49:14] yurik, i gotta run to catch a bug. have a good night! [00:49:22] yurik, s/bug/bus/ [00:49:48] dr0ptp4kt, #wikimedia-mobile [00:52:08] New review: Yurik; "X-CS will only be set for the known IPs, so even though we are increasing from zero only to zero and..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [00:55:58] New patchset: Yurik; "Removed carrier-specific filtering" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [02:08:18] !log LocalisationUpdate completed (1.22wmf5) at Thu May 30 02:08:18 UTC 2013 [02:08:30] Logged the message, Master [02:14:49] !log LocalisationUpdate completed (1.22wmf4) at Thu May 30 02:14:49 UTC 2013 [02:14:58] Logged the message, Master [02:37:00] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu May 30 02:37:00 UTC 2013 [02:37:09] Logged the message, Master [05:20:58] New review: ArielGlenn; "The mediawiki and stubs arguments are mandatory so they should not appear in brackets. See http://ww..." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/65706 [06:35:49] Hey all. [06:35:56] Back home! Need sleep! [08:14:59] New review: Mark Bergsma; "This looks like it will double the cache for carriers, because always setting X-CS necessarily also ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [08:37:40] New review: Mark Bergsma; "This doubles the cache size for all carrier views. This can probably be handled in a better way." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/66034 [09:27:37] Change abandoned: Hashar; "might revisit that later" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59626 [09:27:37] Change abandoned: Hashar; "might revisit that later" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59611 [09:30:07] New patchset: Hashar; "beta: syslog-ng on deployment-bastion host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/51668 [09:30:49] New patchset: Hashar; "beta: symlink /a/common" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65254 [09:52:49] New patchset: Mark Bergsma; "Install cp1037-1040 as text Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66074 [09:54:34] New patchset: Mark Bergsma; "Install cp1037-1040 as text Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66074 [09:56:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66074 [10:14:26] mark: I guess I will migrate beta to use varnish as well :-] [10:14:46] yeah [10:14:50] can just ditch squid probably [10:16:11] New patchset: Mark Bergsma; "Add cp1037-1040 as Varnish text" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66077 [10:16:42] Change merged: Mark Bergsma; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66077 [10:18:33] !log mark synchronized wmf-config/squid.php 'Add cp1037-1040 as text Varnish' [10:18:44] Logged the message, Master [10:20:43] mark: did you get deployment-cache-text1.pmtpa.wmflabs working in beta ? [10:20:50] I should have stayed around during the hackaton sorry :( [10:21:37] uhm [10:21:48] yes ryan fixed the problem [10:22:02] but then I had trouble with the manifests and just went to production? [10:22:04] something like that [10:22:04] err: Could not retrieve catalog from remote server: Error 400 on SERVER: $$$lvs::configuration::lvs_service_ips[$::realm]["text"][$::mw_primary] is not an hash or array when accessing it with textsvc at /etc/puppet/manifests/role/cache.pp:393 on node i-00000778.pmtpa.wmflabs [10:22:05] :D [10:22:10] yeah that [10:22:49] I had to make the cache class to use role::cache::configuration instead of lvs::configuration [10:23:03] and make the hash entries to point to lvs::configuration::* if and only if in prod [10:23:04] not ideal :( [10:23:28] not really no [10:23:44] but that is the only way to make them work until we get LVS in labs or a way to emulate it [10:24:15] we can just emulate LVS with a tcp proxy [10:24:23] :-D [10:24:25] (or we could make LVS work...) [10:26:42] perhaps we could just include part of the $lvs:: hash into the role::cache::configuration hash [10:32:36] yup I think I had to do that for the mobile class [10:37:51] New patchset: Mark Bergsma; "Fix the cluster_tier conditionals" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66079 [10:38:47] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66079 [10:45:21] New patchset: Mark Bergsma; "Point esams Varnish at eqiad Varnish instead of Squid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66080 [10:46:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66080 [11:02:41] mark, good morning, any thoughts on how i could debug the varnish issue with xff spoofing from the office? [11:03:12] by reading the code and figuring out where it goes wrong? [11:03:16] or by testing it in labs [11:03:29] I tested it, it worked fine for me [11:03:33] from fenari/bast1001 [11:04:26] New patchset: Mark Bergsma; "Use individual eqiad backends instead of the LVS (Squid) service IP" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66082 [11:05:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66082 [11:11:14] New patchset: Mark Bergsma; "Add IPv6 address to amssq47" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66086 [11:12:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66086 [11:14:29] hello [11:14:35] is Faidon here ? [11:14:42] yes [11:14:44] hello paravoid [11:14:46] hi [11:14:48] what's up? [11:15:01] is git-buildpackage the norm in terms of making debian packages out of git repos ? [11:15:11] or can I use a custom built script I have ? [11:15:30] paravoid: this is in relation to https://mingle.corp.wikimedia.org/projects/analytics/cards/716 [11:17:04] paravoid: so basically my understanding is that Ops need to be able to build the package themselves. I am willing and capable of putting together any and all steps needed to create the package from Ops point of view. [11:17:19] paravoid: Question: does it matter if git-buildpackage is used or not in this process ? [11:17:40] git-buildpackage is the standard way of building a package out of git [11:17:52] what is libdclass-dev? [11:17:58] is it your own code? [11:18:02] someone else's? [11:18:18] I remember an RT ticket a while back where you pointed us to a .deb and nothing else [11:18:18] paravoid: someone else's, but it contains a JNI that drdee and me have worked on [11:18:37] so it's modified? [11:18:49] yes [11:18:54] i think we need two packages [11:19:04] yeah, so let's import it into git/gerrit and work there [11:19:13] paravoid: yes, I agree that RT ticket was somewhat incomplete, so today (with your help) I would like us to check that ticket [11:19:29] sure, no problem [11:19:44] great :) thanks [11:19:52] I'm starting conversion to git-buildpackage [11:20:05] drdee: do you agree with dropping git2deblogs in favor of git-buildpackage ? [11:20:17] yes [11:20:20] cool [11:21:16] New review: Yurik; "Not exactly. All new carriers we bring on board are signed up for both Zero & M, and several large Z..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [11:22:56] New review: Mark Bergsma; "Why not migrate those carriers to just m.wikipedia.org then? A redirect would do it and not have dup..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [11:26:07] yurik: I thought you were going to write to wikitech? [11:26:32] mark about merging zero & m? [11:26:47] I don't know about what, whatever it was you wanted the hangout about [11:26:48] i haven't written it yet because i need to discuss it with Dan [11:27:10] you still need to discuss it with Dan yet you already have a patchset that you want us to approve? :) [11:27:19] mark - unrelated :) [11:27:52] the patch is about marking all traffic with X-CS, so that we make all decisions in backend and don't have two configuration points [11:28:00] that has been decided on our end very long time ago [11:28:13] the hangout / email is about getting rid of ZERO subdomain [11:28:26] and eventually - M subdomain [11:28:36] but that needs to be checked with business first [11:28:37] yes but you're increasing variance for both old and new carriers [11:28:45] while we were just working hard to D [11:28:48] DECREASE variance [11:29:09] mark, the new carriers are always signed up for both M & ZERO -- that's business requirement [11:29:12] so why don't you come up with a grand plan for that first [11:29:31] yes, well, it's a stupid business requirement that's not in line with ops requirements [11:29:34] so about time we fix that then eh [11:29:49] mark, you will have to take that with Kul [11:29:56] if you want to support both domains, then make sure they get cached under a single object [11:30:00] no I don't [11:30:02] you will [11:30:37] mark, ok, i will bring up your objections, but they are not the same -- under ZERO the images are not shown, just links to them [11:30:59] and that's why you want ESI to differentiate? [11:31:01] i do want to get rid of ZERO, but that is not my decision t omake [11:31:29] does m.wikipedia have any carriers that don't allow images there? [11:31:54] ESI -- mostly i wanted it for banners, but they could also be used for imagse [11:32:06] but best of all - if javascript would make those decisions [11:32:19] there will be lots of work converting it to the new system [11:32:40] and its on our top priority right now [11:33:05] keep in mind that on our end we're also working on a system that allows you to configure such stuff based on ip ranges [11:33:11] so I gather that won't be necessary anymore? [11:33:48] we do want that patch in [11:34:17] anyway, it would really be nice if you could put your plans in emails, so we can all discuss and think about it [11:34:23] it's getting a bit confusing otherwise [11:34:35] sure thing, i will spell them out [11:34:38] thank you [11:37:44] New review: Mark Bergsma; "Yurik explained on IRC that the objects are actually different, images vs no images on ZERO." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66034 [11:48:11] yurik: where is the X-Images header set? [11:48:58] mark, in the backend -- pagerendinghooks.ph [11:49:27] why is it varied on then? [11:50:01] does the client send it with e.g. js? [11:51:39] mark, i suspect that header was initially was used to reduce cache variance, before the variance was set to depend on X-CS [11:52:04] right [11:52:08] i'm just wondering how it's supposed to work [11:52:14] it doesn't :) [11:52:20] can we remove it then? :) [11:52:30] i think at this point its a noop [11:52:37] yeah, so then it would be good to remove it [11:52:39] but i would have to check [11:52:45] because it's just confusing now [11:53:00] the additional processing is neglegible I'm sure, but for sanity we should remove it [11:53:44] i think the better way would be to use it properly instead of X-CS variance -- base stuff on X-IMAGE + ESI for the banner [11:54:06] but then you would need to set X-Images in Varnish again [11:54:37] oh, right. bummer. Yes it would [11:54:58] one thing we could do [11:55:21] is use that GEO lookup code to do IP->x-image [11:55:28] yes [11:55:39] we could fairly easily compile a range of ips that don't support images [11:55:39] that's why I don't understand why you want to remove all that stuff now [11:55:57] you can have your backend code generate a database of ip ranges to headers [11:56:04] and varnish can use that to set the headers [11:58:36] i would like to remove it because at this point the logic is scattered around multiple places, and we can't easily change it. For example, varnish checks for allowed languages, even though that list is stored in config page. So if the carrier decides to change that list, two lists have to be updated. Also, I would like to show a warning "this language is not whitelisted by your carrier" [11:59:05] but because varnish filters everything non-zero, we can't do that [11:59:40] I understand, but combined with the libvmapper vmod, any possible plans to merge m and zero and general work to reduce cache variance, it all seems rather confused [11:59:45] so that's why I'm asking to have it an email [11:59:47] then we can discuss it [12:00:24] yep, will write something shortly [12:00:37] (still trying to get IPv4 proxy up :(( [12:00:54] !log Gracefully reloading Zuul to deploy 'Ica1137438a1b9137f01276b4dfe49a82227de518' [12:01:03] Logged the message, Master [12:02:56] New patchset: Odder; "(bug 48308) Change namespace settings for ukwikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66090 [12:06:34] New patchset: Odder; "(bug 48308) Change namespace settings for ukwikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66090 [12:29:36] Who is giving out office VPN tokens? [12:32:31] office it I imagine? [12:32:49] there's a vpn? [12:32:51] who knew [12:34:13] i don't know [12:34:15] there have been [12:34:18] multiple instances [12:34:21] who knows what's there now :) [12:51:55] paravoid ping [12:51:58] pong [12:52:09] paravoid did you know about that problem we have on labs with nfs? [12:52:19] no [12:52:21] https://bugzilla.wikimedia.org/show_bug.cgi?id=48910 [12:52:32] the thing is that nfs server is fucked up and coren is on vacation [12:52:38] petan: I'm back. [12:52:43] And I see what has goine on. [12:52:45] Coren ooooooooooh [12:52:51] Coren what what? [12:53:02] Coren I was looking for it, is that the rpc.mountd thing? [12:53:03] There is a tool, ldaplist, that has stopped working. [12:53:06] or is it something else [12:53:08] ah [12:53:18] Which is what is used to extract all the local groups. [12:53:22] Coren can you write it down somewhere so that we can fix it when you are afk? :P [12:53:29] Coren where it runs? [12:53:31] New patchset: Mark Bergsma; "Split off 503 retry in vcl_error from 5xx retry in vcl_fetch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66095 [12:53:58] petan: That's a tool written by Ryan, I think, or maybe Andrew. It's in python so I'm trying to understand it now. [12:54:04] ryan [12:54:05] ok [12:54:11] what's the issue? [12:54:15] is it running on -login or nfs server? [12:54:22] Rightnow, the only error message is "The search returned an error." [12:54:30] (Which isn't all tha useful) [12:54:35] I suppose there is no chance to get access to that box so that I could eventually fix it when you are not here? :P [12:54:44] It's on the NFS server -- that's part of the whole-labs infrastructure. [12:54:49] mhm [12:54:59] HASHAR [12:55:02] GALLIUM IS SLOW [12:55:16] petan: Possibly at some point, that's something the opsen will have to discuss. Unlike tool labs, it isn't just my call. :-) [12:55:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66095 [12:55:41] ok, can you fix it then? :P [12:55:54] because lot of people are poking me since it broke... [12:56:15] beside that everything works fine [12:56:18] petan: Understandably. That shouldn't ever have broken unless something changed in LDAP, so I'm trying to track down the change. [12:56:29] ok [12:59:05] except Exception: [12:59:06] sys.stderr.write("The search returned an error.\n") [12:59:06] ds.unbind() [12:59:06] sys.exit(1) [12:59:11] How... informative. [12:59:29] talk is cheap [12:59:31] :P [13:00:09] that looks like typical python [13:00:20] but still more than you would get from a c code [13:00:30] (coredump) [13:00:33] Meh. That's little else than: [13:00:55] mark: looking :D [13:01:04] i'm kidding [13:01:05] ldap.INSUFFICIENT_ACCESS: {'info': 'You do not have sufficient privileges to perform an unindexed search', 'desc': 'Insufficient access'} [13:01:16] o.O [13:01:16] Yep, something changed in LDAP [13:01:39] Either an index is gone, or the bind account got its privileges reduced. [13:02:40] Coren can you fix it? :/ [13:02:53] petan: Possibly, I' [13:03:07] I'll have to hunt in Gerrit to try and find a change to LDAP. [13:03:31] If I can't fix the actual problem, I'll hack a workaround somehow. [13:05:26] New patchset: Mark Bergsma; "Tier 2 backends should use the chash director for requesting upstream" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66097 [13:07:45] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66097 [13:10:57] !log gallium: blackholed some web crawler robot. [13:11:06] Logged the message, Master [13:11:42] New patchset: Catrope; "Clean up VisualEditor config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64493 [13:12:25] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64493 [13:14:09] !log catrope synchronized visualeditor.dblist 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' [13:14:18] Logged the message, Master [13:14:43] !log catrope synchronized wmf-config/CommonSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' [13:14:52] Logged the message, Master [13:15:29] !log catrope synchronized wmf-config/InitialiseSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' [13:15:37] Logged the message, Master [13:16:12] New patchset: Mark Bergsma; "Reduce backend timeout to slightly lower than frontend" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66098 [13:16:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66098 [13:19:57] !log gallium: killed stalled Jenkins threads that were producing high I/O. [13:20:06] Logged the message, Master [13:20:18] !log Synced afe912f7a335ae037b8d8b0ba9616dc55848a2f0 for VisualEditor updates (logmsgbot seems to be broken) [13:20:23] * Coren yells unkind things at Gerrit making it so hard to search. [13:20:26] Logged the message, Mr. Obvious [13:20:35] !log (it's not broken, never mind, I'm just incapable of reading things) [13:20:43] Logged the message, Mr. Obvious [13:25:04] New patchset: Mark Bergsma; "The retry503 parameter now specifies the number of restarts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66099 [13:26:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66099 [13:27:26] Aha! I understand what happened. The LDAP grew because of all the new users to bigger than the default index-entry-limit (!) [13:31:38] New patchset: Catrope; "Make VisualEditor work in labs (hopefully)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66100 [13:33:24] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66100 [13:45:08] * Coren tries to figure out the process to make LDAP changes, and fails. [13:48:14] RECOVERY - swift-container-server on ms-be4 is OK: PROCS OK: 13 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server [13:48:14] RECOVERY - swift-account-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-auditor [13:48:24] RECOVERY - DPKG on ms-be4 is OK: All packages OK [13:48:24] RECOVERY - swift-container-updater on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-updater [13:48:24] RECOVERY - swift-account-reaper on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [13:48:24] RECOVERY - Host mw1041 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [13:48:34] RECOVERY - Disk space on ms-be4 is OK: DISK OK [13:48:34] RECOVERY - swift-account-replicator on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-replicator [13:48:34] RECOVERY - swift-object-auditor on ms-be4 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [13:48:44] RECOVERY - NTP on ms-be4 is OK: NTP OK: Offset -0.0001221895218 secs [13:48:44] RECOVERY - swift-object-replicator on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [13:48:44] RECOVERY - swift-account-server on ms-be4 is OK: PROCS OK: 13 processes with regex args ^/usr/bin/python /usr/bin/swift-account-server [13:48:54] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [13:48:54] RECOVERY - swift-object-server on ms-be4 is OK: PROCS OK: 101 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server [13:48:54] RECOVERY - RAID on ms-be4 is OK: OK: State is Optimal, checked 1 logical device(s) [13:49:04] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [13:49:04] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [13:49:04] PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours [13:49:04] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [13:49:04] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours [13:49:25] New patchset: Faidon; "Nagios/Varnish HTCP: fix illegal characters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66102 [13:49:49] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66102 [13:50:14] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Thu May 30 13:50:04 UTC 2013 [13:50:14] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Thu May 30 13:50:04 UTC 2013 [13:50:24] PROBLEM - Disk space on ms-be1012 is CRITICAL: DISK CRITICAL - /var/lib/ceph/osd/ceph-137 is not accessible: Input/output error [13:50:34] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Thu May 30 13:50:24 UTC 2013 [13:50:34] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Thu May 30 13:50:24 UTC 2013 [13:50:34] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Thu May 30 13:50:29 UTC 2013 [13:50:34] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Thu May 30 13:50:30 UTC 2013 [13:50:34] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Thu May 30 13:50:30 UTC 2013 [13:50:35] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Thu May 30 13:50:30 UTC 2013 [13:50:44] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Thu May 30 13:50:35 UTC 2013 [13:50:44] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Thu May 30 13:50:35 UTC 2013 [13:50:44] RECOVERY - Puppet freshness on mc2 is OK: puppet ran at Thu May 30 13:50:35 UTC 2013 [13:50:44] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Thu May 30 13:50:40 UTC 2013 [13:50:44] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Thu May 30 13:50:40 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Thu May 30 13:50:45 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Thu May 30 13:50:45 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Thu May 30 13:50:45 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Thu May 30 13:50:45 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Thu May 30 13:50:50 UTC 2013 [13:50:55] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Thu May 30 13:50:50 UTC 2013 [13:50:55] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Thu May 30 13:50:50 UTC 2013 [13:50:56] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Thu May 30 13:50:50 UTC 2013 [13:51:04] RECOVERY - Puppet freshness on professor is OK: puppet ran at Thu May 30 13:50:55 UTC 2013 [13:51:04] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Thu May 30 13:50:55 UTC 2013 [13:51:04] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Thu May 30 13:50:55 UTC 2013 [13:51:04] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Thu May 30 13:50:55 UTC 2013 [13:51:04] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Thu May 30 13:51:00 UTC 2013 [13:51:05] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Thu May 30 13:51:00 UTC 2013 [13:51:15] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Thu May 30 13:51:05 UTC 2013 [13:51:15] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Thu May 30 13:51:05 UTC 2013 [13:51:15] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Thu May 30 13:51:10 UTC 2013 [13:51:34] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Thu May 30 13:51:25 UTC 2013 [13:52:05] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:05] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:05] RECOVERY - Puppet freshness on mc8 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:05] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:05] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:06] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:06] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:07] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Thu May 30 13:51:55 UTC 2013 [13:52:07] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:08] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:08] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:09] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:09] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:10] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:10] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:11] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Thu May 30 13:52:00 UTC 2013 [13:52:11] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Thu May 30 13:52:01 UTC 2013 [13:52:12] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Thu May 30 13:52:01 UTC 2013 [13:52:14] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Thu May 30 13:52:06 UTC 2013 [13:52:14] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Thu May 30 13:52:06 UTC 2013 [13:52:15] wheeee [13:52:42] heh [13:52:44] Coren any update on fix? :o [13:53:05] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:05] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:05] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:05] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:05] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:06] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:06] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:07] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Thu May 30 13:52:56 UTC 2013 [13:53:07] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Thu May 30 13:53:01 UTC 2013 [13:53:08] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Thu May 30 13:53:01 UTC 2013 [13:53:08] RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Thu May 30 13:53:01 UTC 2013 [13:53:09] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Thu May 30 13:53:01 UTC 2013 [13:53:09] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Thu May 30 13:53:01 UTC 2013 [13:53:15] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:16] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:16] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:16] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:16] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:16] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Thu May 30 13:53:06 UTC 2013 [13:53:24] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Thu May 30 13:53:21 UTC 2013 [13:53:24] RECOVERY - Puppet freshness on linne is OK: puppet ran at Thu May 30 13:53:21 UTC 2013 [13:53:24] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Thu May 30 13:53:21 UTC 2013 [13:53:24] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Thu May 30 13:53:21 UTC 2013 [13:53:24]