[00:09:41] (03PS1) 10Ori.livneh: Consolidate mediawiki::users::* into a single class [operations/puppet] - 10https://gerrit.wikimedia.org/r/133945 [00:17:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [00:59:55] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun May 18 00:59:51 UTC 2014 [01:03:55] Reedy: You should suggest /ignore for bad bots. [01:40:59] bad bots, bad bots [01:56:29] (03PS2) 10Ori.livneh: Consolidate mediawiki::users::* into a single class [operations/puppet] - 10https://gerrit.wikimedia.org/r/133945 [01:58:19] (03CR) 10Ori.livneh: [C: 032] Consolidate mediawiki::users::* into a single class [operations/puppet] - 10https://gerrit.wikimedia.org/r/133945 (owner: 10Ori.livneh) [02:12:25] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3791 MB (3% inode=99%): [02:13:49] !log LocalisationUpdate completed (1.24wmf4) at 2014-05-18 02:12:46+00:00 [02:13:57] Logged the message, Master [02:20:25] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3432 MB (3% inode=99%): [02:24:33] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-18 02:23:29+00:00 [02:24:38] Logged the message, Master [03:00:25] RECOVERY - Disk space on virt0 is OK: DISK OK [03:07:09] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun May 18 03:06:03 UTC 2014 (duration 6m 2s) [03:07:14] Logged the message, Master [03:14:41] Any sysadmins around? A request has come in for a page deletion with 26k revisions. [03:15:17] If that'll cause any problems at all then I'll decline the request. [03:15:30] I'm not convinced by the rationale to delete it myself [03:18:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [04:30:35] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun May 18 01:29:45 2014 [06:10:38] (03PS1) 10Ori.livneh: remove reference to obsoleted configchange script [operations/puppet] - 10https://gerrit.wikimedia.org/r/133952 [06:10:40] (03PS1) 10Ori.livneh: Merge applicationserver::sudo into mediawiki::users [operations/puppet] - 10https://gerrit.wikimedia.org/r/133953 [06:16:19] (03CR) 10Ori.livneh: [C: 032] remove reference to obsoleted configchange script [operations/puppet] - 10https://gerrit.wikimedia.org/r/133952 (owner: 10Ori.livneh) [06:17:27] (03CR) 10Ori.livneh: [C: 032] Merge applicationserver::sudo into mediawiki::users [operations/puppet] - 10https://gerrit.wikimedia.org/r/133953 (owner: 10Ori.livneh) [06:19:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [06:30:15] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun May 18 06:30:11 UTC 2014 [06:33:35] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun May 18 03:33:08 2014 [07:13:44] (03PS1) 10Ori.livneh: Merge applicationserver::packages into mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133956 [07:15:42] (03CR) 10jenkins-bot: [V: 04-1] Merge applicationserver::packages into mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133956 (owner: 10Ori.livneh) [07:16:31] (03PS2) 10Ori.livneh: Merge applicationserver::packages into mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133956 [07:24:06] (03PS1) 10Ori.livneh: move applicationserver::hhvm to beta::hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/133958 [07:26:04] (03CR) 10Ori.livneh: [C: 032] move applicationserver::hhvm to beta::hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/133958 (owner: 10Ori.livneh) [07:32:45] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun May 18 07:32:37 UTC 2014 [07:41:21] (03CR) 10Ori.livneh: [C: 032] "Eep, I missed this. Yes, this will work; no, restarts would not be necessary." [operations/puppet] - 10https://gerrit.wikimedia.org/r/131418 (owner: 10Springle) [09:20:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [10:27:13] Hello Why We don't let archive.org index wikipedia pages anymore? (robots.txt) [11:02:21] (03PS1) 10Andrew Bogott: Mark River Tarnell's account as disabled. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133965 [11:02:59] Does anyone know who River Tarnell (aka 'kate') is? I'm a bit nervous about cleaning up their account even though it seems to be already useless. [11:08:21] andrewbogott: TS admin in the past [11:08:40] liangent: OK -- safe to assume that they don't need access to production cluster? [11:08:48] (03PS1) 10Andrew Bogott: Rename jamesofur to jamesur and sync UID to labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133967 [11:09:00] liangent: or, better yet, do you have an email for them? [11:09:27] andrewbogott: river.tarnell at wikimedia.de , when river was a ts admin [11:09:41] not sure whether it's still valid [11:11:45] liangent: that's helpful, thank you! [11:14:39] * YuviPanda waves a liangent [11:15:48] (03CR) 10Jalexander: [C: 031] Rename jamesofur to jamesur and sync UID to labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133967 (owner: 10Andrew Bogott) [11:16:30] (03CR) 10Andrew Bogott: [C: 032] Mark River Tarnell's account as disabled. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133965 (owner: 10Andrew Bogott) [11:16:36] liangent: Y U NO ON -mobile? [11:17:25] PROBLEM - SSH on hooft is CRITICAL: Server answer: [11:17:45] (03PS2) 10Andrew Bogott: Rename jamesofur to jamesur and sync UID to labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133967 [11:18:11] YuviPanda: ? [11:18:19] liangent: #wikimedia-mobile? [11:18:25] RECOVERY - SSH on hooft is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [11:19:37] (03CR) 10Andrew Bogott: [C: 032] Rename jamesofur to jamesur and sync UID to labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/133967 (owner: 10Andrew Bogott) [11:46:11] (03PS3) 10Ori.livneh: Merge applicationserver::packages into mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133956 [11:48:02] (03CR) 10Ori.livneh: [C: 032] Merge applicationserver::packages into mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133956 (owner: 10Ori.livneh) [12:21:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [12:29:11] (03PS1) 10Andrew Bogott: Renumber many users to match labs: [operations/puppet] - 10https://gerrit.wikimedia.org/r/133968 [12:29:43] (03CR) 10Andrew Bogott: [C: 04-2] "I will merge this during a pre-arranged window on Tuesday morning." [operations/puppet] - 10https://gerrit.wikimedia.org/r/133968 (owner: 10Andrew Bogott) [12:31:01] akosiaris: Have you had any luck escaping from the Trusty dependency hell I left you in last week? [12:39:23] (03PS1) 10Ladsgroup: Enable Echo for en_rtl test wiki and enable flow for a test page [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133969 [12:51:25] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 14.29% of data exceeded the critical threshold [500.0] [13:13:25] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data exceeded the critical threshold [500.0] [13:13:28] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 8 below the confidence bounds [13:17:08] (03PS1) 10Reza: Enable FlaggedRevs for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133970 (https://bugzilla.wikimedia.org/65452) [13:34:25] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [13:58:19] (03CR) 10Calak: [C: 031] Enable FlaggedRevs for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133970 (https://bugzilla.wikimedia.org/65452) (owner: 10Reza) [14:34:25] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [14:41:45] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:35] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.007 second response time [14:57:35] PROBLEM - Puppet freshness on snapshot1001 is CRITICAL: Last successful Puppet run was Sun May 18 11:56:51 2014 [15:00:35] PROBLEM - Puppet freshness on snapshot1004 is CRITICAL: Last successful Puppet run was Sun May 18 11:59:33 2014 [15:03:35] PROBLEM - Puppet freshness on snapshot1003 is CRITICAL: Last successful Puppet run was Sun May 18 12:03:03 2014 [15:18:36] PROBLEM - Puppet freshness on snapshot1002 is CRITICAL: Last successful Puppet run was Sun May 18 12:17:44 2014 [15:22:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [15:24:05] (03PS1) 10Ori.livneh: deduplicate mediawiki / snapshot packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133979 [15:26:17] (03CR) 10Ori.livneh: [C: 032] deduplicate mediawiki / snapshot packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/133979 (owner: 10Ori.livneh) [15:27:55] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Sun May 18 15:27:45 UTC 2014 [15:28:05] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Sun May 18 15:27:56 UTC 2014 [15:28:05] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Sun May 18 15:27:56 UTC 2014 [15:28:05] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Sun May 18 15:28:01 UTC 2014 [15:32:17] (03PS1) 10John F. Lewis: Add DNS for us-ne [operations/dns] - 10https://gerrit.wikimedia.org/r/133980 (https://bugzilla.wikimedia.org/64557) [15:32:23] (03PS1) 10John F. Lewis: Apache set up for us-ne [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133981 (https://bugzilla.wikimedia.org/64557) [15:32:28] (03PS1) 10John F. Lewis: Add us_newikimedia set up configs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) [15:32:46] (03CR) 10jenkins-bot: [V: 04-1] Add us_newikimedia set up configs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [15:34:41] * JohnLewis sighs [15:37:37] (03PS2) 10John F. Lewis: Add us_newikimedia set up configs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) [15:46:55] (03PS2) 10Reza: Enable FlaggedRevs for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133970 (https://bugzilla.wikimedia.org/65452) [15:47:04] (03PS1) 10Ori.livneh: Consolidate applicationserver and mediawiki modules [operations/puppet] - 10https://gerrit.wikimedia.org/r/133983 [15:54:31] (03PS2) 10Ori.livneh: Consolidate applicationserver and mediawiki modules [operations/puppet] - 10https://gerrit.wikimedia.org/r/133983 [15:54:33] (03PS1) 10Ori.livneh: drop unused 'group' param for role::applicationserver::common [operations/puppet] - 10https://gerrit.wikimedia.org/r/133985 [16:16:47] (03PS1) 10Ori.livneh: start cleaning up role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/133987 [16:17:52] (03CR) 10jenkins-bot: [V: 04-1] start cleaning up role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/133987 (owner: 10Ori.livneh) [16:19:03] (03PS2) 10Ori.livneh: start cleaning up role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/133987 [16:20:58] (03CR) 10Ori.livneh: [C: 032] drop unused 'group' param for role::applicationserver::common [operations/puppet] - 10https://gerrit.wikimedia.org/r/133985 (owner: 10Ori.livneh) [16:35:54] (03CR) 10Steinsplitter: [C: 031] Enable FlaggedRevs for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133970 (https://bugzilla.wikimedia.org/65452) (owner: 10Reza) [16:39:34] (03CR) 10Calak: [C: 031] Enable FlaggedRevs for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133970 (https://bugzilla.wikimedia.org/65452) (owner: 10Reza) [16:40:33] Hello Why We don't let archive.org index wikipedia pages anymore? (robots.txt) [16:48:16] ytrezq: where do you see that? [16:48:39] i see '# Don't allow the wayback-maschine to index user-pages ', but the settings that follow it are commented out and restricted to user pages anyway [16:52:17] dat typo :p [17:04:35] JohnLewis: yes, a cute germanism no? [17:04:43] das wayback-maschine [17:06:46] indeed :p [17:07:47] (03CR) 10Ori.livneh: [C: 04-1] "We need this to work, so it's not OK to fail silently if mod_headers is not loaded. So just go ahead and assume that it is." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/111925 (owner: 10BryanDavis) [17:09:03] (03PS1) 10John F. Lewis: Redirect usne to us-ne [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133991 (https://bugzilla.wikimedia.org/64557) [17:09:17] (03PS1) 10John F. Lewis: Add usne DNS [operations/dns] - 10https://gerrit.wikimedia.org/r/133992 (https://bugzilla.wikimedia.org/64557) [17:11:52] * JohnLewis is getting tired of typing 'Bug: 64557' today >.> [17:13:16] JohnLewis: patch looks good, I think. needs ops to merge, tho [17:15:01] patches :p [17:15:14] I think I have 5/6 atm for ops [17:15:27] 2 DNS, 2 Apache and 1 MediaWiki config [17:16:41] idk how to schedule these things anyway :p I assume it is a 'wait for an op to see it' because the process to inform ops is not one I know besides RT. [17:17:00] * hoo facepalms hard [17:17:12] hoo: Just for you :p [17:17:37] I shouldn't try to debug issues that aren't there [17:17:56] hoo: What'ca debugging? [17:18:14] JohnLewis: I thouhgt ULS is not on beta or not the way it is in production [17:18:27] Ah [17:18:27] but actually that was just a mistake in an bug report which made me think that [17:40:55] ori: Just try to access any article since this year. (because I'm talking about articles) [18:23:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [18:31:04] (03CR) 10Ori.livneh: [C: 032] Consolidate applicationserver and mediawiki modules [operations/puppet] - 10https://gerrit.wikimedia.org/r/133983 (owner: 10Ori.livneh) [18:31:41] ori: around ? [18:35:03] Hello, Why We don't let archive.org index wikipedia articles anymore? 03 (robots.txt stylesheets and JavaScript) [18:47:27] Hi ytrezq. [18:47:37] I don't believe archive.org is disallowed. [18:48:19] The line for it is commented out as ori pointed out early. :) [18:48:49] I'm not talking about user pages. [18:49:09] see https://web.archive.org/web/20131102084738/http://fr.wikipedia.org/wiki/HIStory https://web.archive.org/web/20140329032726/http://en.wikipedia.org/wiki/History https://web.archive.org/web/20121223094416/http://fr.wikipedia.org/wiki/Wikip�dia:Accueil_principal [18:49:37] and it worked before : https://web.archive.org/web/20091221192546/http://fr.wikipedia.org/wiki/Wikip�dia:Accueil_principal https://web.archive.org/web/20091221190457/http://en.wikipedia.org/wiki/Main_Page [18:50:29] (03PS1) 10Ori.livneh: Fix-ups for Ie0807c706 [operations/puppet] - 10https://gerrit.wikimedia.org/r/133993 [18:50:41] some stylesheets are disallowed from indexing. [18:52:27] (03CR) 10Ori.livneh: [C: 032] Fix-ups for Ie0807c706 [operations/puppet] - 10https://gerrit.wikimedia.org/r/133993 (owner: 10Ori.livneh) [18:54:30] ytrezq: It is not disallowed in the global robots.txt file therefore this would be an issue for local community not Wikimedia Ops. [18:54:45] *Wikimedia Ops/this channel [18:56:47] (03PS12) 10Giuseppe Lavagetto: puppet-compiler: module for installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/133449 [18:58:08] (03PS13) 10Giuseppe Lavagetto: puppet-compiler: module for installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/133449 [19:01:03] (03PS1) 10Ori.livneh: Qualify mediawiki::jobrunner in a couple of other places [operations/puppet] - 10https://gerrit.wikimedia.org/r/133995 [19:02:47] (03CR) 10Ori.livneh: [C: 032] Qualify mediawiki::jobrunner in a couple of other places [operations/puppet] - 10https://gerrit.wikimedia.org/r/133995 (owner: 10Ori.livneh) [19:08:31] JohnLewis: But it affect all languages... [19:09:04] ytrezq: I can not see an entry which would block it though. [19:13:03] (03PS14) 10Giuseppe Lavagetto: puppet-compiler: module for installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/133449 [19:23:38] (03PS1) 10Ori.livneh: Use role-specific Exec to manage wikimedia-task-appserver's apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/133996 [19:26:50] (03CR) 10Ori.livneh: [C: 032] Use role-specific Exec to manage wikimedia-task-appserver's apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/133996 (owner: 10Ori.livneh) [19:42:32] (03CR) 10Withoutaname: Add us_newikimedia set up configs (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [19:55:12] (03PS1) 10Ori.livneh: Skip jobrunner role def on tmh nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/133998 [19:57:11] (03CR) 10Ori.livneh: [C: 032] Skip jobrunner role def on tmh nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/133998 (owner: 10Ori.livneh) [20:10:34] (03PS3) 10John F. Lewis: Add us_newikimedia set up configs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133982 (https://bugzilla.wikimedia.org/64557) [20:28:14] JohnLewis: http://bits.wikimedia.org/robots.txt probably. [20:29:07] Oh CSS, yes :p [20:29:42] I thought they meant indexing in general though. [20:30:40] I think he's right. [20:31:30] I'll file a bug. [20:36:31] Filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=65464 [20:55:29] Any sysadmins around now? [20:57:56] jgage, it says you're on duty for something. Would you know if I could delete a page with 26k revisions? [21:00:26] ajr: Why do you need one? [21:00:34] big delete [21:00:42] ah [21:00:42] don't want to asplode the servers [21:00:52] aaand have been trying to contact one for ~16 hours now... [21:02:44] ajr: i saw your ticket [21:03:03] want me to have a look and if ok exec ? [21:03:38] yes please! [21:04:16] ajr: How many revs? :p [21:04:32] 26k [21:04:59] yikes [21:05:04] enwiki? :p [21:05:12] ofc [21:05:32] I tried suggesting that they just redirect it because my "guide to deleting big pages" basically says to not whenever possible [21:05:33] ajr: where is the request ? [21:05:36] but they didn't like that idea :( [21:05:47] https://meta.wikimedia.org/wiki/Steward_requests/Miscellaneous#Yet_another_deletion_request [21:06:21] hmm [21:06:28] seems like better with shell [21:06:43] hoo: got a second :)? [21:06:48] maybe :D [21:06:51] what's up? [21:06:53] bug delete [21:06:56] :) [21:07:06] Want me to delete a bug? [21:07:13] big* :P [21:07:17] hoo: ofc that's your job [21:07:17] hoo: shell big delete [21:07:19] oh :D [21:07:31] I can do that [21:07:36] details? [21:07:40] though also if you could delete some mosquitos from my back yard that would always be appreciated [21:07:42] RT 7520 [21:07:50] https://meta.wikimedia.org/wiki/Steward_requests/Miscellaneous#Yet_another_deletion_request <-- the original requesst [21:09:14] looking [21:15:06] I can either 1) shell delete or do it via my steward tools [21:15:10] both is equally bad to the DB [21:15:37] the locals seem to think that deletion is the only possible option [21:15:41] or 3. force the community to redirect it. Jeez. [21:15:54] yeah that works too [21:16:05] they're worried about someone reverting it, but they could just sysop-protect the redirect [21:16:22] ajr: Redirect it going 'deletion declined' :p [21:16:29] lol [21:16:35] then they can argue with hoo. Solved. [21:16:43] I can also delete it [21:16:52] not going to blow the DB [21:16:53] if you are OK with that, then please do [21:17:00] 'But we want it deleted' Well, do you want a deleted page or a working wiki :p [21:17:17] from what I know, 26k isn't too bad [21:17:29] over 50k would be closer to that response :P [21:17:32] It's not something a software should do [21:17:45] but given the current load etc. it will go without to much hassle [21:17:56] ok [21:18:09] there's a storm here right now, so I might lose connectivity - best if someone else deleted it [21:18:26] I'm on it [21:18:30] <3 [21:19:25] I'm watching myself from the database console :P [21:20:26] oooh a 42 second lag from that [21:20:28] nice :D [21:20:35] (according to the rc notice) [21:20:42] If only it could scream 'HELP hoo is make me work too much!' :D [21:21:00] slaves will catch up in a second, write load isn't very high atm [21:21:04] poor kittens [21:21:22] done [21:24:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [21:25:01] solved the ticket, hoo [21:25:07] matanya: Oh, thanks :) [21:25:18] Who opens rt tickets for this? [21:30:17] hoo: Anyone :p [21:31:08] until you do stuff like that on a high load time or so 25k are totally bearable [21:44:25] (03PS5) 10BryanDavis: Send Vary header on http to http redirect [operations/apache-config] - 10https://gerrit.wikimedia.org/r/111925 [23:07:26] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 21.43% of data exceeded the critical threshold [500.0] [23:25:26] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0]