[00:00:09] (03PS1) 10Yurik: Replace ZRMA with ZeroBanner+JsonConfig on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136503 [00:12:55] (03PS2) 10Yurik: Replace ZRMA with ZeroBanner+JsonConfig on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136503 [00:15:41] mwalker: I have been highly suspicious that txstatsd is borked for a while. See https://bugzilla.wikimedia.org/show_bug.cgi?id=62667#c11 [00:24:03] (03PS6) 10JanZerebecki: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) [00:27:33] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [00:29:18] (03CR) 10JanZerebecki: "Dropped dhparams as DHE is disabled." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [00:30:56] (03CR) 10Yuvipanda: "We are using a copy of this for the labs proxy as well, so I'll keep an eye on this and copy things over again when this gets merged." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [00:36:19] YuviPanda: a copy? these?: modules/dynamicproxy/templates/*.conf [00:36:30] jzerebecki: yup [01:56:02] (03CR) 10MaxSem: [C: 04-2] "Thanks! This change looks right and I'll deploy it next week but it requires https://gerrit.wikimedia.org/r/#/c/136519/ which I wrote for " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136316 (owner: 10Whym) [02:16:51] !log LocalisationUpdate completed (1.24wmf6) at 2014-05-31 02:15:48+00:00 [02:16:58] Logged the message, Master [02:27:23] !log LocalisationUpdate completed (1.24wmf7) at 2014-05-31 02:26:20+00:00 [02:27:30] Logged the message, Master [03:14:53] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat May 31 03:13:47 UTC 2014 (duration 13m 46s) [03:14:57] Logged the message, Master [03:28:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [04:29:22] PROBLEM - LighttpdHTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:30:12] RECOVERY - LighttpdHTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5122 bytes in 0.629 second response time [04:34:22] PROBLEM - LighttpdHTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:22] RECOVERY - LighttpdHTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5122 bytes in 4.624 second response time [04:37:25] apergos: Pageviews dumps dried out. Last dump provided: pagecounts-20140530-150000 [04:38:02] apergos: could you take a look at Vanadium / dump creation process? [04:54:22] PROBLEM - Host analytics1015 is DOWN: PING CRITICAL - Packet loss = 100% [05:50:12] PROBLEM - Host ms-be1001 is DOWN: PING CRITICAL - Packet loss = 100% [06:29:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [07:12:15] * Nemo_bis pets ms-be1001 [09:30:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [10:26:42] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 14.29% of data exceeded the critical threshold [500.0] [10:33:32] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 31 May 2014 07:33:19 UTC [10:41:42] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [11:03:42] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat May 31 11:03:39 UTC 2014 [11:23:52] (03CR) 10Odder: "Hi Andre, are you back home yet?" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/129671 (owner: 10Odder) [11:28:43] (03CR) 10Odder: "This patch is scheduled for deployment during the morning SWAT window on Monday, 2014-06-02." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136154 (https://bugzilla.wikimedia.org/65905) (owner: 10Jean-Frédéric) [12:31:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [14:10:22] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:13] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.007 second response time [15:32:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [16:03:32] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 31 May 2014 13:02:55 UTC [16:03:52] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat May 31 16:03:42 UTC 2014 [16:52:42] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 14.29% of data exceeded the critical threshold [500.0] [17:00:32] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 31 May 2014 13:59:59 UTC [17:00:53] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sat May 31 17:00:42 UTC 2014 [17:01:15] <_joe__> uhm it's been on for 10 minutes [17:01:19] <_joe__> this is serious [17:02:42] <^demon|away> _joe__: Sup? [17:03:21] <_joe__> ^demon|away: a spike in 5xx responses, most related to commons it seems [17:04:01] <^demon|away> Hmmm [17:04:48] anyone around who can fix this issue: Pageviews dumps dried out. Last dump provided: pagecounts-20140530-150000. Vanadium? apergos? [17:05:09] <_joe__> it stopped now, but most errors seem related to Special:UploadStash [17:08:02] <^demon|away> Hmmm [17:08:11] <^demon|away> Yeah not seeing it now :\ [17:11:36] And bawolff is afk [17:13:26] * ori looks as well [17:16:26] Nemo_bis: what do you do with things like http://commons.wikimedia.org/wiki/Commons:Requests_for_rights/possible_autopatrolled_candidates/sortableTable3000 ? it causes timeouts when trying to load [17:16:42] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [17:17:13] http://pt.wikipedia.org/wiki/Usuário:Gustavotcabral/Auto/Patologia too for that matter [17:18:16] hedonil: first things first, have you filed a bug against the analytics component in bugzilla? [17:19:23] ori: not yet. hope this would be fixed in a minute - but I'll do now, as it's over 24 hours [17:23:49] ori: here it is https://bugzilla.wikimedia.org/show_bug.cgi?id=65978 [17:27:25] ori: That page is useless, as most of those people have already been flagged. I can delete it if it's troublesome. [17:27:43] twkozlowski: would be much appreciated [17:28:22] hedonil: thanks, i'm poking at it on the off-chance that it's something i can fix, but most likely it requires qchris or ottomata [17:28:38] ori: thx [17:30:03] ori: Weird though; people managed to edit it when it was 55k, and it's just 33k now [17:31:18] ori: usually, fix the templates which cause the error :) but the page might be meant for action=edit view only [17:31:37] i really hope hhvm resolves a bunch of these types of issues [17:31:44] I can't get it to load, but it definitely used to. [17:31:55] I remember I hovered over the usernames to see who's been flagged or not. [17:34:11] aww no more viewing [17:35:48] viewing's overrated. editor engagement! [17:39:22] hedonil: ok, the good news is that more recent dumps are available on the host that generates them [17:39:28] so it must be an issue of syncing them to dumps.wm.o [18:04:16] (03PS1) 10Ori.livneh: rcstream: enroll in ganglia; add system role [operations/puppet] - 10https://gerrit.wikimedia.org/r/136549 [18:06:31] (03Abandoned) 10Ori.livneh: start cleaning up role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/133987 (owner: 10Ori.livneh) [18:25:03] (03PS1) 10Ori.livneh: Add resource for server metrics site [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/136550 [18:33:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [18:35:56] haha [18:36:01] somebody is going to be careful with puppet [18:41:50] (03PS1) 10Ori.livneh: diamond: add diamond::collector::nginx resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/136551 [18:43:10] (03CR) 10Ori.livneh: "Companion change: https://gerrit.wikimedia.org/r/#/c/136551/" [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/136550 (owner: 10Ori.livneh) [18:48:22] PROBLEM - Ubuntu mirror in sync with upstream on carbon is CRITICAL: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 12 hours old. [18:50:22] RECOVERY - Ubuntu mirror in sync with upstream on carbon is OK: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 0 hours old. [19:10:37] (03CR) 10Calak: [C: 031] Enable FlaggedRevs for Central Kurdish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136282 (https://bugzilla.wikimedia.org/65809) (owner: 10Reza) [20:02:42] ori: yes, good news that there are more recent dumps.. but if I can't access them it's pretty useless for me [20:11:04] (03PS3) 10John F. Lewis: Redirect usne to us-ne [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133991 (https://bugzilla.wikimedia.org/64557) [20:11:55] jeremyb_ ^ [20:15:46] * twkozlowski mumbles 'Nebraska, Nebraska!' [20:16:58] twkozlowski: meh [20:21:35] :-D [20:32:49] I [20:32:59] I'm moving to Nebraska to block the creation of this domain. [20:33:01] Ha! [20:33:41] twkozlowski: AffCom said tough, you're us-nb :p [20:34:28] or us-neb, depends how much you want to argue :D [20:34:35] I already said that AffCom has no authority to set ISO codes. [20:35:00] Ops wise. [20:35:20] Universe-wise. [20:35:36] If they've assigned US-NE to the chapter, we can't argue as AffCom have been delegated to do that. [20:35:48] or whatever the purpose of AffCom is. [20:36:23] AffCom have no authority to assign ISO codes to anything. Maybe their teddy bears. [20:36:38] Who said they are assigning ISO codes? [20:36:52] 22:35 JohnLewis: If they've assigned US-NE to the chapter, we can't argue as AffCom have been delegated to do that. [20:36:55] They're saying the New England chapter is recognised by US-NE. [20:37:11] They're not saying New England's ISO code is US-NE. [20:38:07] [20:38:28] We're not using us-ne as the ISO code here. We're using it as the subdomain as that is how they are recognised. [20:38:51] Such as, why is Wikimedia UK, uk? When uk is Ukrainian :p Same touchy ground imho [20:38:52] Whatever, I don't really care. [20:39:09] JohnLewis: UA is the ISO code for Ukraine [20:39:56] UK is reserved for the UK, too, next to GB. [20:40:22] meh [20:54:08] JohnLewis: per ISO 3611-1, "MEH" is unreserved [20:54:22] you could still claim it if you played your cards right [20:55:14] Reedy: Reedy: Reedy: Reedy: Reedy: [20:58:19] ori: Thanks for telling me :D [21:34:32] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 30 May 2014 18:25:33 UTC [21:48:43] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data exceeded the critical threshold [500.0] [22:03:42] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0]