[00:29:08] <grrrit-wm>	 (03PS3) 10BryanDavis: Revert "DNS cleanup" [operations/dns] - 10https://gerrit.wikimedia.org/r/120039 
[01:38:54] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[02:17:32] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf19) at 2014-03-24 02:17:32+00:00
[02:17:44] <morebots>	 Logged the message, Master
[02:41:16] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 24 02:41:13 UTC 2014 (duration 41m 12s)
[02:41:21] <morebots>	 Logged the message, Master
[03:02:02] <grrrit-wm>	 (03PS1) 10Springle: repool db1037 in s5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120484 
[03:02:39] <grrrit-wm>	 (03CR) 10Springle: [C: 032] repool db1037 in s5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120484 (owner: 10Springle)
[03:02:47] <grrrit-wm>	 (03Merged) 10jenkins-bot: repool db1037 in s5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120484 (owner: 10Springle)
[03:03:51] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  's5 repool db1037 warm up'
[03:03:56] <morebots>	 Logged the message, Master
[04:39:54] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[08:29:17] <hashar>	 hello!
[08:29:35] <hashar>	 Openstack has a puppet dashboard: http://puppetdb.openstack.org/ :-D
[08:33:18] <matanya>	 hi hashar 
[08:33:49] <hashar>	 lo
[08:51:04] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[08:53:37] <grrrit-wm>	 (03PS2) 10Hashar: Fix error: timidity service can not be stopped [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 
[08:53:52] <grrrit-wm>	 (03CR) 10Hashar: "Mentioned in commit summary message:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar)
[08:54:29] <grrrit-wm>	 (03CR) 10Hashar: "> looks sane, i would wait with those changes until pmtpa is out." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120013 (owner: 10Hashar)
[08:55:57] <grrrit-wm>	 (03CR) 10Hashar: "> did you see the request at the top of this file? Please abandon." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120005 (owner: 10Hashar)
[08:57:09] <grrrit-wm>	 (03CR) 10Matanya: "fair enough, tim and hashar, i accept your view." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120005 (owner: 10Hashar)
[09:58:41] <grrrit-wm>	 (03PS1) 10ArielGlenn: snapshots: move centralauthdump cron out of role into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120496 
[10:00:39] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] snapshots: move centralauthdump cron out of role into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120496 (owner: 10ArielGlenn)
[10:07:19] <Steinsplitter>	 akosiaris: ping
[10:07:39] <grrrit-wm>	 (03PS1) 10Hashar: contint: install puppet-lint from rubygems on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/120498 
[10:08:28] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] contint: install puppet-lint from rubygems on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/120498 (owner: 10Hashar)
[10:09:33] <grrrit-wm>	 (03PS2) 10Hashar: contint: install puppet-lint from rubygems on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/120498 
[10:11:17] <grrrit-wm>	 (03CR) 10Matanya: "wouldn't it be better to backport a newer version? gem2deb is your friend :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120498 (owner: 10Hashar)
[10:12:23] <grrrit-wm>	 (03PS1) 10Hashar: snapshot: lint list-last-n-good-dumps.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/120499 
[10:13:38] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] snapshot: lint list-last-n-good-dumps.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/120499 (owner: 10Hashar)
[10:13:49] <hashar>	 bahh
[10:15:17] <grrrit-wm>	 (03PS2) 10Hashar: snapshot: lint list-last-n-good-dumps.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/120499 
[10:19:02] <grrrit-wm>	 (03Abandoned) 10Hashar: Move logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/83574 (owner: 10Reedy)
[10:20:42] <grrrit-wm>	 (03Abandoned) 10Hashar: Add csteipp & aaron to admins::jenkins group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119209 (owner: 10Ori.livneh)
[10:37:26] <grrrit-wm>	 (03PS2) 10Hashar: openstack: generic_upstart now use boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118716 
[10:37:29] <grrrit-wm>	 (03PS2) 10Hashar: lvs: generic_upstart now use boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118717 
[10:37:32] <grrrit-wm>	 (03PS2) 10Hashar: twemproxy: generic_upstart now use boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118718 
[10:41:54] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[10:44:37] <grrrit-wm>	 (03PS3) 10Hashar: twemproxy: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118718 
[10:44:47] <grrrit-wm>	 (03PS3) 10Hashar: openstack: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118716 
[10:45:04] <grrrit-wm>	 (03PS3) 10Hashar: lvs: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118717 
[10:47:55] <grrrit-wm>	 (03PS10) 10Hashar: sanity test for refreshWikiversionsCDB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105698 
[10:49:46] <grrrit-wm>	 (03CR) 10Matanya: lvs: generic::upstart_job() now uses boolean values (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118717 (owner: 10Hashar)
[10:58:48] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] openstack: generic::upstart_job() now uses boolean values (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118716 (owner: 10Hashar)
[10:59:51] <grrrit-wm>	 (03CR) 10Matanya: twemproxy: generic::upstart_job() now uses boolean values (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118718 (owner: 10Hashar)
[12:21:43] <grrrit-wm>	 (03PS1) 10Matanya: puppetmaster: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/120512 
[12:30:04] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[12:44:29] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Revert "DNS cleanup" [operations/dns] - 10https://gerrit.wikimedia.org/r/120039 (owner: 10BryanDavis)
[12:46:14] <paravoid>	 the reqerror spike is one user with UA PECL::HTTP/1.7.4 (PHP/5.3.22)
[12:46:21] <paravoid>	 with some API gets
[12:57:04] <paravoid>	 wtf is wrong with ulsfo
[12:57:31] <paravoid>	 icinga was all red for 4xxx for a moment there
[12:58:44] <MaxSem>	 connectivity flapped?
[12:59:12] <paravoid>	 if it did, it was short enough to not be noticed by OSPF/BGP
[13:14:22] <grrrit-wm>	 (03PS1) 10Matanya: mha: fix var scope search [operations/puppet] - 10https://gerrit.wikimedia.org/r/120518 
[13:18:36] <grrrit-wm>	 (03PS2) 10Matanya: mha: fix var scope search [operations/puppet] - 10https://gerrit.wikimedia.org/r/120518 
[13:29:19] * paravoid  grumbles at the icinga labs warnings

[13:35:35] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Fold exim::rt into role::rt [operations/puppet] - 10https://gerrit.wikimedia.org/r/120520 
[13:35:37] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: spamassassin: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120521 
[13:35:39] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: spamassassin: cleanup module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120522 
[13:35:41] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: clamav: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120523 
[13:42:54] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[13:43:17] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: clamav: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120523 
[13:43:19] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: spamassassin: cleanup module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120522 
[13:48:21] <Nemo_bis>	 whoah, spamassassin improvements
[13:48:28] <paravoid>	 not really
[13:48:48] <Nemo_bis>	 (preparation for) improvements
[13:48:55] <Nemo_bis>	 TLC
[13:48:57] <Nemo_bis>	 whatever
[13:49:04] <paravoid>	 do you have something specific in mind?
[13:49:07] <paravoid>	 that you'd like to see fixed I mean
[13:49:11] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Fold exim::rt into role::rt [operations/puppet] - 10https://gerrit.wikimedia.org/r/120520 (owner: 10Faidon Liambotis)
[13:49:31] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Catalog-diffed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120521 (owner: 10Faidon Liambotis)
[13:49:39] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Catalog-diffed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120522 (owner: 10Faidon Liambotis)
[13:49:48] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Catalog-diffed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120523 (owner: 10Faidon Liambotis)
[13:54:28] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: otrs: add missing network::constants include [operations/puppet] - 10https://gerrit.wikimedia.org/r/120526 
[13:54:39] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] otrs: add missing network::constants include [operations/puppet] - 10https://gerrit.wikimedia.org/r/120526 (owner: 10Faidon Liambotis)
[13:54:49] <matanya>	 paravoid: the spamd stuff 
[13:55:00] <paravoid>	 yes?
[13:55:12] <matanya>	 last i asked it was broken
[13:55:19] <paravoid>	 what does that mean?
[13:55:37] <matanya>	 i remember i spoke with akosiaris about it some time ago
[13:55:57] <ottomata>	 akosiaris: https://gerrit.wikimedia.org/r/#/c/117024/ :)
[13:56:16] <matanya>	 he said something about the hoem dir and the usage of systemuser
[13:56:24] <paravoid>	 he's on vacation this week
[13:56:36] <paravoid>	 matanya: do you see those in my changeset? :)
[13:56:46] <matanya>	 see which?
[13:56:58] <ottomata>	 paravoid: akosiaris i?
[13:56:59] <ottomata>	 is?
[13:57:04] <paravoid>	 the broken /var/spamd and systemuser
[13:57:11] <paravoid>	 ottomata: yes, iirc
[13:57:14] <ottomata>	 ah, hmmmm
[13:57:18] <matanya>	 paravoid: no, where is it ?
[13:57:26] * matanya  feels stupid

[13:57:35] <ottomata>	 who is less buys than you and wants to review it?  the main part I feel like needs review is the rsync daemon
[13:57:38] <ottomata>	 on a public IP
[13:57:55] * matanya  found it

[13:58:18] <matanya>	 well, paravoid that :P
[13:59:02] <matanya>	 thanks for fixing it
[13:59:23] <grrrit-wm>	 (03PS13) 10Ottomata: Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 
[14:10:33] <anomie>	 aude: I see you have gerrit change 119311 on for the SWAT deploy in about 50 minutes. But it's not merged to master yet.
[14:13:50] <aude>	 working on it
[14:14:01] <aude>	 i suppose if not in yet, then we'll do later
[14:14:30] <aude>	 i can submit core submodule patch
[14:23:29] <cmjohnson1>	 ottomata: still on for 1030? 
[14:24:22] <ottomata>	 oh yeahhhhhH!
[14:24:32] <ottomata>	 i forgot but now that you rminded me i'm excited!
[14:25:09] <cmjohnson1>	 great...one at a time?  an1015 first
[14:25:22] <ottomata>	 yeah one at a time
[14:26:24] <ottomata>	 wait, 1015?
[14:26:33] <ottomata>	 looking at RT ticket
[14:26:40] <ottomata>	 We should move analytics 1018,1019,1020, and 1025 from Row C into Row D.
[14:27:00] <ottomata>	 cmjohnson1: ^
[14:27:03] <cmjohnson1>	 ok
[14:28:37] <ottomata>	 ok yeah, so there are 7 out of 10 datanodes in row c right now
[14:28:52] <ottomata>	 which means there are 3 in row a
[14:28:56] <ottomata>	 so
[14:29:01] <ottomata>	 we want to balance as best as possible
[14:29:08] <ottomata>	 so we are going to move 3 out of C and into D
[14:29:12] <ottomata>	 that will leave us at
[14:29:18] <ottomata>	 A: 3
[14:29:19] <ottomata>	 C: 4
[14:29:19] <ottomata>	 D: 3
[14:29:34] <ottomata>	 so, 1018,1019,1020 will move to Row D
[14:29:42] <cmjohnson1>	 got it
[14:30:00] <ottomata>	 cool, so let's do 1018!
[14:30:05] <ottomata>	 you ready?  i will go in and shut it down
[14:30:10] <cmjohnson1>	 yes, i am ready
[14:31:51] <ottomata>	 !log stopping hadoop services and shutting down analytics1018
[14:31:57] <morebots>	 Logged the message, Master
[14:32:37] <ottomata>	 cmjohnson1: it should be shutting down now
[14:33:07] <grrrit-wm>	 (03CR) 10Addshore: [C: 031] Remove language subdomains for wikidata.org [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis)
[14:34:24] <icinga-wm>	 PROBLEM - Host analytics1018 is DOWN: PING CRITICAL - Packet loss = 100%  
[14:40:52] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Initial 2.0.0-1 debian release [operations/debs/archiva] (debian) - 10https://gerrit.wikimedia.org/r/115323 (owner: 10Ottomata)
[14:46:45] <anomie>	 aude: Looking at these Wikibase changes for the SWAT, I'm a little lost because I can't actually find where mediawiki/extensions/Wikibase is actually deployed on the WMF servers. I see extensions/Wikidata/extensions/Wikibase seems to have a static copy of it, but that doesn't help me much.
[14:47:27] <hoo>	 anomie: Yep it's in the Wikidata extension
[14:47:45] <hoo>	 We use a copy in there with all of its dependencies
[14:47:56] <aude>	 it's in Wikidata
[14:48:27] <aude>	 i suppose we can wait until the later slot tonight, since we're still organizing these patches
[14:48:37] <hashar>	 ottomata: is the Jenkins debian-glue job any helpful ? :-]
[14:48:48] <ottomata>	 hashar: uhhh
[14:49:09] <ottomata>	 i have never looked at it :-$
[14:49:12] <hashar>	 ohhh
[14:49:26] <hoo>	 yep, it's probably better if we wait
[14:49:44] * aude  might need  nap, but ok

[14:49:59] <hashar>	 I am asking because some init script lintian error got fixed with PS 12 https://gerrit.wikimedia.org/r/#/c/115323/11..12/debian/archiva.init
[14:50:35] <hashar>	 and the job did mention them at https://integration.wikimedia.org/ci/job/operations-debs-archiva-debian-glue/13/testReport/
[14:50:46] <ottomata>	 ahh, i think whatever that was was also mentioned in review
[14:50:51] <ottomata>	 oh and I see the links in the comments now
[14:50:59] <ottomata>	 hmm, cool, I will click on those more often
[14:51:23] <MaxSem>	 hey anomie and manybubbles, looks like we've got nothing to deploy today
[14:51:24] <hashar>	 the whole idea is to report lintian errors automatically
[14:51:37] <ottomata>	 aye, i mean, i see them when I build the package too
[14:51:38] <hashar>	 so the patch author don't waste reviewers time with things that are easily fixable
[14:51:40] <ottomata>	 so I fix many of them then
[14:51:45] <ottomata>	 but some I don't bother fixing
[14:51:47] <hashar>	 :-°
[14:51:55] <hashar>	 now the job does it for ya automatically :]
[14:51:56] <manybubbles>	 MaxSem: sounds good to me
[14:52:01] <manybubbles>	 I'll schedule something for tomorrow
[14:52:13] <aude>	 we might find something for tomorrow
[14:52:13] <hashar>	 ottomata: great to see you got all lintian errors fixed !
[14:53:05] * anomie  is glad for public logs in light of his connection issues this morning

[14:53:45] * matanya  was pinged by lint* :P

[14:54:09] <anomie>	 MaxSem: We almost had some Wikidata patches, but it turned out they weren't quite ready
[14:54:14] <grrrit-wm>	 (03PS1) 10Cmjohnson: Updating dns entry for analytics1018 [operations/dns] - 10https://gerrit.wikimedia.org/r/120533 
[14:56:30] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Updating dns entry for analytics1018 [operations/dns] - 10https://gerrit.wikimedia.org/r/120533 (owner: 10Cmjohnson)
[14:57:19] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Updating dns entry for analytics1018 [operations/dns] - 10https://gerrit.wikimedia.org/r/120533 (owner: 10Cmjohnson)
[14:59:11] <cmjohnson1>	 paravoid: I moved to analytics1018 to a different vlan...had to change ip. Can you double check to make sure we don't have any network issues (will also be moving an1019 and 1020)
[15:00:14] <grrrit-wm>	 (03CR) 10Hashar: [C: 031 V: 032] "deployed on beta cluster via local puppet master" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar)
[15:00:27] <ottomata>	 cmjohnson1: if we boot it back on, i'll log in and check some networking stuff
[15:01:06] <cmjohnson1>	 ottomata: powering on...try and login via mgmt
[15:01:45] <ottomata>	 k...
[15:01:56] <ottomata>	 in console, i see it booting
[15:02:23] <cmjohnson1>	 cool...so are you going to change interface settings?
[15:02:50] <ottomata>	 ja will do
[15:04:10] <cmjohnson1>	 okay..ping me when you wanna move 1019
[15:07:22] <paravoid>	 lemme know if you need anything
[15:11:06] <grrrit-wm>	 (03PS1) 10Hoo man: Run rebuildEntityPerPage.php on Wikidata (once per month) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120535 
[15:15:36] <ottomata>	 cmjohnson1: is 10.64.53.1 the correct gateway?
[15:16:00] <cmjohnson1>	 yes
[15:17:22] <ottomata>	 hm
[15:17:49] <ottomata>	 something isn't working
[15:17:56] <ottomata>	 not yet sure if i've done something wrong
[15:18:01] <ottomata>	 but I can't ping the gateway even
[15:20:39] <ottomata>	 haha, oof, why does this always happen cmjohnson1? :p
[15:21:06] <cmjohnson1>	 did you update /etc/hosts ?
[15:21:24] <grrrit-wm>	 (03Abandoned) 10coren: toollabs: insert sql tool to execnodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/66266 (owner: 10Petrb)
[15:21:27] <ottomata>	 yes
[15:22:29] <cmjohnson1>	 ottomata: is this what you have http://p.defau.lt/?aHr_Tbb_cx_xjZxDT_bt3w
[15:22:40] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: new-style cron via submit host [operations/puppet] - 10https://gerrit.wikimedia.org/r/120538 
[15:23:01] <ottomata>	 yup
[15:23:05] <ottomata>	 cmjohnson1 ^
[15:23:42] <ottomata>	 http://p.defau.lt/?U_f7niay_Jq_daxPAlPkDw
[15:23:44] <cmjohnson1>	 paravoid: is that correct broadcast?  see link 
[15:23:52] <grrrit-wm>	 (03PS1) 10ArielGlenn: remove nonexistent param from content retriever invocation (wikiretriever) [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120539 
[15:24:23] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] remove nonexistent param from content retriever invocation (wikiretriever) [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120539 (owner: 10ArielGlenn)
[15:24:35] <grrrit-wm>	 (03CR) 10coren: [C: 032] Tool Labs: new-style cron via submit host [operations/puppet] - 10https://gerrit.wikimedia.org/r/120538 (owner: 10coren)
[15:24:36] <ottomata>	 i don't htink broadcast would matter in this case, but even so it should be the correct broadcast with that netmask
[15:24:50] <ottomata>	 pretty sure that is all correct
[15:24:53] <hoo>	 Coren: Y u so fast? :P
[15:25:00] <hoo>	 toollabs::execnode also has the sql command
[15:25:01] <cmjohnson1>	 ottomata: the vlan didn't set correctly
[15:25:08] <ottomata>	 oh?
[15:25:11] <hoo>	 and should have been updated
[15:25:20] <Coren>	 hoo: Who put that /there/?
[15:25:37] <Coren>	 finds git blame.
[15:25:40] <hoo>	 Maybe it was me :P
[15:25:43] <hoo>	 It was me
[15:25:49] <Coren>	 Hah-ha!
[15:25:50] <Coren>	 :-)
[15:26:05] <hoo>	 Wasn't aware of exec_environ than I put that up
[15:26:10] <hoo>	 oversight :/
[15:26:35] <Coren>	 It was mostly an accident that I merged the sql bit, I forgot to split the diffs and when I did the git commit I didn't want to bother inmixing them up.  :-)
[15:26:42] <Coren>	 Meh, trivial fix.  :-0
[15:27:19] <yuvipanda>	 Coren: trivial merge? https://gerrit.wikimedia.org/r/#/c/120243/
[15:27:33] <yuvipanda>	 Coren: and https://gerrit.wikimedia.org/r/#/c/120348/
[15:27:33] <cmjohnson1>	 ottomata: something is not right..it won't save to that vlan..maybe mark can offers some advice 
[15:27:59] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Trivial fix is trivial." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120243 (owner: 10Tim Landscheidt)
[15:29:12] <ottomata>	 it won't save?
[15:29:15] <ottomata>	 whatcha mean?
[15:29:25] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Less trivial, but also quite reasonable." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120348 (owner: 10Tim Landscheidt)
[15:29:46] <yuvipanda>	 Coren: ty
[15:30:15] <mark>	 what's up?
[15:30:36] <ottomata>	 mark, cmjohnson1 and I are moving nodes to the new analytics vlan in row d
[15:30:41] <cmjohnson1>	 mark: the interface for analytics1018 won't save to analytics vlan 
[15:30:48] <cmjohnson1>	 it doesn't give me any errors on commit
[15:30:57] <Coren>	 yuvipanda: You may want to look at the apache config on tools-webproxy -- there are a few tricks in place still.  When there isn't a registered webservice, I serve a specific errordocument and when the tools.admin webservice isn't up, I serve a static version.  (And / is handled by the tools.admin webservice)
[15:31:15] <yuvipanda>	 Coren: ah, right. I'll pick that up.
[15:31:17] <mark>	 cmjohnson1: then why do you think so?
[15:31:23] <Coren>	 yuvipanda: You /could/ simply fall back to the apache webproxy, actually, but I'd rather get rid of it entirely.
[15:31:37] <yuvipanda>	 Coren: yeah, I want to replace that thing.
[15:32:28] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: avoid double definition of /usr/bin/sql [operations/puppet] - 10https://gerrit.wikimedia.org/r/120540 
[15:32:54] <hoo>	 ah :)
[15:33:18] <ottomata>	 mark, its possible i've done something wrong when changing network/interfaces and etc/hosts on the box, but i can't ping the gateway, and route takes longer than usual to print routes (not sure what that means)
[15:34:08] <mark>	 you haven't removed those hosts from the other vlan
[15:34:11] <grrrit-wm>	 (03CR) 10BryanDavis: "Nice! I had to comment this bit of code out in my local hacks to test scap under Vagrant. I did not try to do nearly as much research as A" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar)
[15:34:18] <cmjohnson1>	 haha..just about to say that
[15:34:30] <cmjohnson1>	 took a minute 
[15:35:19] <mark>	 [edit interfaces interface-range vlan-public1-d-eqiad]
[15:35:19] <mark>	 +    member "ge-2/0/[3-47]";
[15:35:19] <mark>	 -    member ge-2/0/*;
[15:35:29] <grrrit-wm>	 (03CR) 10coren: [C: 032] Tool Labs: avoid double definition of /usr/bin/sql [operations/puppet] - 10https://gerrit.wikimedia.org/r/120540 (owner: 10coren)
[15:36:37] <cmjohnson1>	 mark: thx 
[15:37:34] <grrrit-wm>	 (03PS3) 10Hashar: beta: sent HTCP purges to eqiad varnishes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116788 
[15:38:50] <grrrit-wm>	 (03CR) 10Hashar: "I have no idea how timidity-daemon ends up being installed on the production application servers though :-(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar)
[15:40:04] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[15:40:59] <grrrit-wm>	 (03PS1) 10ArielGlenn: remove searchlogs rsync conf from datasets manifests, no longer used [operations/puppet] - 10https://gerrit.wikimedia.org/r/120543 
[15:42:03] <cmjohnson1>	 mark: are you going to add to analytics1-d-eqiad ?
[15:42:13] <mark>	 ah I can
[15:42:45] <mark>	 but it was already added?
[15:43:06] <ottomata>	 it is in network.pp
[15:43:08] <ottomata>	 not sure if that is what you mean
[15:43:11] <grrrit-wm>	 (03PS2) 10ArielGlenn: remove searchlogs rsync conf from datasets manifests, no longer used [operations/puppet] - 10https://gerrit.wikimedia.org/r/120543 
[15:43:43] <cmjohnson1>	 oh..still showing under default
[15:44:01] <mark>	 oh
[15:44:05] <mark>	 the interface range is just not setup
[15:44:36] <cmjohnson1>	 yeah, I can get that...there is a msg that you were configuring still so I didn't wanna mess with it
[15:44:47] <cmjohnson1>	 Users currently editing the configuration:
[15:45:09] <mark>	 oh i'm already done
[15:45:11] <mark>	 sorry
[15:45:25] <paravoid>	 ottomata: did you see my response re: stefan?
[15:45:31] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] remove searchlogs rsync conf from datasets manifests, no longer used [operations/puppet] - 10https://gerrit.wikimedia.org/r/120543 (owner: 10ArielGlenn)
[15:46:17] <ottomata>	 yes I asked him to respond
[15:46:29] <ottomata>	 oh he just did
[15:47:14] <cmjohnson1>	 ottomata: i can ping 10.64.53.10 now
[15:47:22] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: add role for the new submit host [operations/puppet] - 10https://gerrit.wikimedia.org/r/120546 
[15:47:52] <ottomata>	 great me too!
[15:49:22] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Simple role addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120546 (owner: 10coren)
[15:50:03] <ottomata>	 great, looks good cmjohnson1
[15:50:30] <ottomata>	 cmjohnson1: i'm going to shut down an19, you ready?  and then do this key revocation for stefan
[15:50:32] <ottomata>	 while you move
[15:50:58] <cmjohnson1>	 give me a few mins...set it to shut down in 10
[15:51:21] <ottomata>	 ok
[15:51:22] * cmjohnson1  needs coffee still

[15:51:26] <ottomata>	 ok no probs
[15:51:29] <ottomata>	 lemm eknow when
[15:53:00] <grrrit-wm>	 (03PS1) 10Ottomata: Removing Stefan Petrea's old ssh keys, installing new [operations/puppet] - 10https://gerrit.wikimedia.org/r/120548 
[15:53:28] <grrrit-wm>	 (03PS2) 10Ottomata: Removing Stefan Petrea's old ssh keys, installing new [operations/puppet] - 10https://gerrit.wikimedia.org/r/120548 
[15:53:42] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Removing Stefan Petrea's old ssh keys, installing new [operations/puppet] - 10https://gerrit.wikimedia.org/r/120548 (owner: 10Ottomata)
[15:53:53] <paravoid>	 jesus how large is this key
[15:54:05] <ottomata>	 hah
[15:54:33] <paravoid>	 seriously, how large is it?
[15:55:16] <ottomata>	 asking
[15:55:19] <MaxSem>	 4k? 8k?
[15:55:30] <paravoid>	 probably 8k?
[15:55:58] <grrrit-wm>	 (03CR) 10Lydia Pintscher: [C: 031] Remove language subdomains for wikidata.org [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis)
[15:56:14] <paravoid>	 mwalker: hey, around?
[15:56:38] <mwalker>	 yep
[15:56:43] <paravoid>	 hi
[15:56:45] <mwalker>	 what can I do for you this morning?
[15:56:55] <paravoid>	 https://wikimediafoundation.org/wiki/Thank_you has a Special:HideBanners link for en.wikidata.org
[15:56:59] <paravoid>	 we are about to deprecate that domain
[15:57:16] <paravoid>	 (wikidata is just "wikidata.org" & "www.wikidata.org")
[15:57:17] <ottomata>	 haha, paravoid 16384
[15:57:23] <paravoid>	 ottomata: yeah, no.
[15:58:12] <mwalker>	 oh; good to know
[15:58:24] <mwalker>	 I shall change the page; and try and track down the other places we probably have that
[15:58:42] <paravoid>	 ottomata: 2K should be enough, 4K if he's feeling really paranoid. revoke the 16K.
[15:58:57] <paravoid>	 (or really, ECDSA if he's feeling paranoid)
[15:59:21] <ottomata>	 sigh, ok...
[15:59:23] <ottomata>	 asking him to join here
[15:59:23] <paravoid>	 mwalker: thanks! I did a quick git grep but couldn't find it anywhere, so it probably was more complicated than that and didn't have the time
[15:59:41] <paravoid>	 mwalker: could you comment on https://gerrit.wikimedia.org/r/119032 when you are done?
[15:59:42] <average>	 joined
[15:59:49] <average>	 hi paravoid , ottomata 
[15:59:53] <paravoid>	 hi average 
[16:00:34] <paravoid>	 mwalker: would you like me to file a bug perhaps? sorry for being an interrupt like that :)
[16:00:55] <ottomata>	 average, paravoid says your key is too long!
[16:01:09] <average>	 is 8192 bits ok ?
[16:01:12] <paravoid>	 no
[16:01:16] <average>	 4096 ?
[16:01:21] <paravoid>	 17:58 < paravoid> ottomata: 2K should be enough, 4K if he's feeling really paranoid. revoke the 16K.
[16:01:24] <paravoid>	 17:58 < paravoid> (or really, ECDSA if he's feeling paranoid)
[16:01:46] <paravoid>	 note that the more bits you add, the longer it will take for you to login
[16:02:07] <average>	 ok, so I guess you'd agree with 4096 
[16:02:14] <average>	 I'll go for that, if that's ok
[16:02:18] <paravoid>	 most of the roots are 2K too, so I don't see the security benefit of you having 4K tbh
[16:02:41] <paravoid>	 but sure, go for 4K if you want
[16:02:43] <paravoid>	 paranoid much? :)
[16:02:48] <average>	 very(lately)
[16:03:11] <icinga-wm>	 RECOVERY - Host analytics1018 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms  
[16:03:11] <icinga-wm>	 PROBLEM - NTP on analytics1018 is CRITICAL: NTP CRITICAL: Offset unknown  
[16:03:36] <ottomata>	 ok average put a new key on office and i'll put it in place
[16:04:06] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Tobias, it's the Special:HideBanners links. I've informed fundraising and they are on it. I'll merge as soon as they're done, hopefully ve" [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis)
[16:04:41] <paravoid>	 ottomata: don't forget to revoke the 16K too now, as it's already being deployed across the infra (unless you didn't get the chance to puppet-merge?)
[16:06:20] <grrrit-wm>	 (03CR) 10Greg Grossmeier: [C: 031] "Also:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120170 (owner: 10Spage)
[16:06:52] <average>	 paravoid , paravoid regenerated, uploaded to office
[16:06:53] <cmjohnson1>	 ottomata  whenever
[16:07:03] <ottomata>	 yeah i'll set it as absent
[16:08:08] <average>	 ottomata: ^^ 
[16:08:50] <ottomata>	 cmjohnson1: ah, i've gotta do this key thing real quick now, and I'm super duper hungry
[16:09:01] <ottomata>	 sorry we are out of sync, could we continue after ops meeting?
[16:09:05] <ottomata>	 or maybe before?
[16:09:12] <cmjohnson1>	 ottomata: sure
[16:09:40] <ottomata>	 average: hm, I htink you need to name your key something different :/
[16:09:40] <James_F>	 greg-g: I was thinking that we should special-case loginwiki and votewiki to get nothing by default, possibly.
[16:09:56] <ottomata>	 or, hm, no, do I?
[16:10:03] <average>	 ottomata: what should I name it ?
[16:10:11] <ottomata>	 paravoid: I can just replace the key, right?  puppet will replace it (effectively revoking the old one)
[16:10:13] <ottomata>	 right?
[16:10:24] <paravoid>	 no
[16:10:25] <paravoid>	 it won't
[16:10:40] <paravoid>	 it doesn't replace, it just adds
[16:10:50] <ottomata>	 really?  even if it is named the same?
[16:10:54] <paravoid>	 I think so
[16:10:56] <ottomata>	 its a define
[16:10:59] <ottomata>	 the name is unique
[16:11:07] <paravoid>	 doesn't matter
[16:11:49] <greg-g>	 James_F: /me nods
[16:12:37] <ottomata>	 ah its not a define, its a resouce
[16:12:37] <ottomata>	 yeah
[16:12:38] <ottomata>	 paravoid
[16:12:39] <ottomata>	 name
[16:12:39] <ottomata>	 (Namevar: If omitted, this attribute’s value defaults to the resource’s title.)
[16:12:39] <ottomata>	 The SSH key comment. This attribute is currently used as a system-wide primary key and therefore has to be unique.
[16:12:51] <paravoid>	 can we figure this out later please?
[16:12:56] <paravoid>	 just ensure => absent it for now
[16:12:58] <paravoid>	 it won't hurt
[16:13:00] <paravoid>	 we can remove it later
[16:13:04] <ottomata>	 ok, average
[16:13:07] <ottomata>	 you need to rename your key then
[16:13:12] <ottomata>	 doesn't matter
[16:13:14] <paravoid>	 it doesn't matter
[16:13:22] <ottomata>	 puppet will error out
[16:13:24] <paravoid>	 the name of the key is arbitrary, you can set it to whatever you want
[16:13:25] <ottomata>	 if they have the same name
[16:13:33] <paravoid>	 it's not encoded in the key itself
[16:13:36] <ottomata>	 the resource is titled by the name
[16:13:41] <ottomata>	 oh no?
[16:13:45] <ottomata>	 the title doesn't matter?
[16:13:47] <ottomata>	 ok
[16:13:49] <paravoid>	 it's just a comment
[16:14:19] <paravoid>	 it doesn't have to be the same on client & server, it's not encoded in the key, it's not part of the exchange at all
[16:16:36] <greg-g>	 just if you were looking at authorized_keys on the server it helps to know which is which :)
[16:17:10] <grrrit-wm>	 (03PS1) 10Ottomata: Revoking Stefan's most recent key and adding a shorter one [operations/puppet] - 10https://gerrit.wikimedia.org/r/120550 
[16:17:20] <grrrit-wm>	 (03PS2) 10Ottomata: Revoking Stefan's most recent key and adding a shorter one [operations/puppet] - 10https://gerrit.wikimedia.org/r/120550 
[16:17:22] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Revoking Stefan's most recent key and adding a shorter one [operations/puppet] - 10https://gerrit.wikimedia.org/r/120550 (owner: 10Ottomata)
[16:17:47] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Revoking Stefan's most recent key and adding a shorter one [operations/puppet] - 10https://gerrit.wikimedia.org/r/120550 (owner: 10Ottomata)
[16:30:53] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: ensure latest for requested packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/120553 
[16:32:24] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Simple tweak." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120553 (owner: 10coren)
[16:38:02] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: tweaks to submit crontab [operations/puppet] - 10https://gerrit.wikimedia.org/r/120556 
[16:41:33] <grrrit-wm>	 (03CR) 10coren: [C: 032] "That works." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120556 (owner: 10coren)
[16:43:07] <andrewbogott>	 mutante: Is the 'planet' labs project migrated?  Or, if not, do you need any support with that?
[16:43:51] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[16:45:14] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: minor typo fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/120558 
[16:47:23] <mutante>	 andrewbogott: i'll likely have it killed before meeting. k?
[16:47:34] <mutante>	 just checking
[16:47:56] <andrewbogott>	 mutante: if you just want me to delete it then I can add it to the list right now, you don't need to do anything.
[16:48:54] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Silly brain." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120558 (owner: 10coren)
[16:52:06] <robh>	 hell week begins now \o/ ;]
[16:54:42] <greg-g>	 oh hey robh, so, since your on RT duty...
[16:56:37] <mutante>	 andrewbogott: Categories:
[16:56:53] <mutante>	 Projects used in production  .. not really
[16:57:14] <andrewbogott>	 ?
[16:57:33] <mutante>	 https://wikitech.wikimedia.org/wiki/Category:Projects_used_in_production
[16:57:49] <mutante>	 it's in that category, and I thought that made you treat it differentely, maybe
[16:58:17] <andrewbogott>	 Nope, it's just on the short list of projects that people have claimed but not finished migrating...
[16:58:33] <andrewbogott>	 all deadlines have long-since passed so I'm on a nagging rampage
[16:59:30] <mutante>	 i don't use it anymore but it seems others do, i'll rebort back within 15 min:)
[17:01:40] <andrewbogott>	 ok
[17:14:35] <matanya>	 andrewbogott: just shut it all down :P
[17:15:15] <andrewbogott>	 matanya: I'm actually pretty surprised at how many projects are still active.  I was betting on 80% attrition, it's turned out to be more like 50-60%.
[17:15:48] <matanya>	 i suspect a lot of them are not used anymore
[17:15:48] <andrewbogott>	 It's really nice that so many people are using labs, and responding quickly and politely to my godzilla-like stomping of pmtpa :)
[17:16:09] <andrewbogott>	 matanya: no, that's what I mean, I've gotten active responses and participation in the migration of about half the projects.
[17:16:38] <andrewbogott>	 well, maybe not quite half, I'll count after the dust settles
[17:17:15] <Coren>	 I'd say the more useful metric is "number of instances in maintained projects" rather than "number of maintained projects"
[17:17:44] <andrewbogott>	 Ah, true, that's definitely more than 50%.
[17:17:47] <mutante>	 andrewbogott: deleted instances, removed hostname, released public IP...
[17:18:00] <mutante>	 re: planet project just has eqiad instance
[17:18:17] <andrewbogott>	 mutante: Thanks, shall I move it to the 'migration finished' column?
[17:18:27] <mutante>	 andrewbogott: yes please
[17:19:35] <mutante>	 if needed i'll request the hostname, but i should never need the public IP again and proxy
[17:22:41] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: further tweaks of submit cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/120563 
[17:46:25] <matanya>	 robh: can you please create a follow up ticket for 7102 ?
[17:48:25] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Guaranteed* bug-free!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120563 (owner: 10coren)
[17:49:58] <andrewbogott>	 ottomata: how goes the migration of labs analytics?
[17:51:55] <ottomata>	  hmm, andrewbogott, i think it is done
[17:52:04] <ottomata>	 we shoudl confirm with dan and christian
[17:52:08] <ottomata>	 but i think anything left can be scrapped
[17:52:13] <ottomata>	 i can delete the nodes I'm responsible for
[17:52:41] <andrewbogott>	 ottomata: can you please confirm and then update https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress appropriately?
[17:55:16] <robh>	 hrmm, damn it
[17:55:32] <robh>	 matanya: if its not decommisioned yet then it doesnt get a wipe ticket
[17:55:39] <robh>	 or someone will mistakenly make it
[17:55:46] <matanya>	 sure
[17:55:47] <ottomata>	 andrewbogott: will do
[17:55:53] <andrewbogott>	 spagewmf: Can you update me about your progress on the 'Editor Engagment' project?  I'm unclear on whether it is done with migration or still needs work...
[17:55:55] <robh>	 matanya: So if these werent decom'd yet what followup ticket do you mean?
[17:56:00] <robh>	 (or did you mean for wipe?)
[17:56:08] <matanya>	 they were deccomed
[17:56:14] <matanya>	 and hence ticket is resolved
[17:56:28] <matanya>	 i requested a wipe ticket
[17:56:33] <robh>	 ahh, ok, then its ok now
[17:56:40] <robh>	 it said 'decom these'
[17:56:44] <robh>	 and then no actual 'this is done'
[17:56:52] <matanya>	 see the change by mutante 
[17:57:06] <matanya>	 he merged his commit there in the ticket
[17:57:06] <robh>	 matanya: yea but there isnt a note in the ticket
[17:57:17] <robh>	 he did?
[17:57:26] <mutante>	 what's up
[17:57:29] <matanya>	 oh, sorry. forgot to mention it, i guess
[17:57:34] <matanya>	 hi mutante 7102
[17:57:34] <robh>	 matanya: yes, you resolved it with no comment
[17:57:36] <robh>	 thats not ok
[17:57:40] <robh>	 ;p
[17:57:49] * matanya  facepalms

[17:57:53] <robh>	 hehe
[17:57:56] <robh>	 caught youuuuu
[17:58:01] * robh  teases in good fun

[17:58:05] <matanya>	 shame on me
[17:58:13] * matanya  is fired!

[17:58:17] <robh>	 hehe, how dare you fuck up one in a thousand tickets, shame.
[17:58:31] <matanya>	 oh, you can't fire a volunteer
[17:58:37] <mutante>	 eh, so that ticket isn't done yet
[17:58:42] <robh>	 indeed
[17:58:44] <matanya>	 yeah, my bad
[17:58:49] <mutante>	 the wiping ticket should exist but be linked to it
[17:59:35] <robh>	 but be stalled or have a shit ton of references to say 'dont wipe thisuntil they are decom'd'
[18:00:25] <andrewbogott>	 marktraceur: I have a bugzilla bug and also a project ('multimedia') that I'm hoping you will close out.  Can you update https://bugzilla.wikimedia.org/show_bug.cgi?id=62616 and https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress please?
[18:00:35] <mutante>	 yea, or use "depends on" when linking.. either way
[18:01:13] <andrewbogott>	 marktraceur: Ah, looks like 'orgcharts' is your thing as well.
[18:01:15] <mutante>	 since we always have those 2 tickets for the same host, one in core and one in pmtpa that way, i started calling them "shutdown" vs. "decom" in some cases, to avoid cnfusio
[18:01:19] <mutante>	 ok, be back after meeting
[18:01:45] <matanya>	 so, robh please revert me :/
[18:02:19] <marktraceur>	 andrewbogott: I cannot verify here - I'm on a plane now  but I will deal with it later today
[18:02:49] <andrewbogott>	 marktraceur: good enough, thank you!  Once projects are migrated please move them into the 'Migration finished' section so I know I can stop nagging :)
[18:03:27] <mutante>	 matanya: robh , moving it back to core-ops, we'll continue after meeting 
[18:03:33] <spagewmf>	 andrewbogott: I haven't started 'Editor Engagement' instance migration, I will do the named hosts this week.
[18:03:57] <andrewbogott>	 spagewmf, can I do anything to help?  This is way behind schedule
[18:04:04] <andrewbogott>	 I can move instances intact if that's useful.
[18:04:11] <andrewbogott>	 Well, possibly, depending on the instance.
[18:07:06] <spagewmf>	 andrewbogott: sorry.  The instances I know about are relatively stock mediawiki-install and labs-vagrant instances. I'll read the instructions now.
[18:08:20] <andrewbogott>	 spagewmf: ok, let me know if you need help with anything
[18:22:43] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  's5 db1037 full steam'
[18:22:48] <morebots>	 Logged the message, Master
[18:26:12] <gi11es>	 varnish question: if I cachebust with a request parameter, does it create duplicates at any varnish level? or does it have no effect at all on the frontend/backend of varnish?
[18:27:12] <gi11es>	 the reason I'm asking this is that we're considering the use of a request parameter for varnish to return an http header to tell the browser this will be a download (for a feature where users will be able click on a link to download a CDN-served image to save it)
[18:27:47] <gi11es>	 I just want to make sure it won't result in any waste of resources
[18:33:21] <gi11es>	 the headers show "miss, miss, miss" when I add a request parameter, which does suggest a level of waste
[18:34:12] <gi11es>	 follow-up question would be whether varnish could be told to pretend that the request parameter isn't there for caching purposes
[18:57:55] <andrewbogott>	 gwicke, I'm going to migrate those instances right now.
[18:58:18] <gwicke>	 andrewbogott: great, thanks!
[18:58:37] <gwicke>	 andrewbogott, wait a moment actually
[18:58:48] <gwicke>	 I can delete some rt test data that we don't need any more
[18:58:52] <gwicke>	 should speed up the migration
[18:59:14] <andrewbogott>	 gwicke: um… the script is running already.  Unless it's tb of data it should be fine.
[18:59:27] <gwicke>	 it's around 50G
[18:59:45] <gwicke>	 don't get a connection anyway, so nm
[18:59:56] <andrewbogott>	 yeah, it's shut down for the sync
[19:00:41] <andrewbogott>	 gwicke: looks like dump-api.wmflabs.org is pointing to nowhere, so I'm going to free it...
[19:01:52] <ottomata>	 cmjohnson1: wanna move an19?
[19:02:08] <cmjohnson1>	 sure
[19:02:15] <ottomata>	 k shutting ti down
[19:02:17] <gwicke>	 andrewbogott, that was a https proxy front-end for what now lives at http://parsoid-tests.wikimedia.org/
[19:02:25] <gwicke>	 IIRC
[19:02:26] <andrewbogott>	 ok
[19:02:35] <gwicke>	 no https at the new location either
[19:02:41] <ottomata>	 !log stopping hadoop services on analytics1019 and shutting it down for move to Row D
[19:02:45] <ottomata>	 cmjohnson1: , you got dns change?
[19:02:46] <morebots>	 Logged the message, Master
[19:02:53] <cmjohnson1>	 umyes
[19:02:56] <cmjohnson1>	 yes
[19:03:03] <gwicke>	 or maybe there is actually https there, wait..
[19:03:08] <gwicke>	 yup, works
[19:03:09] <matanya>	 ottomata: need help with users on stat1?
[19:03:17] <gwicke>	 andrewbogott, so +1 for nuking
[19:03:24] <ottomata>	 yes!
[19:03:28] <ottomata>	 matanya:  that would be amazing!
[19:03:52] <ottomata>	 matanya: https://rt.wikimedia.org/Ticket/Display.html?id=6789
[19:04:21] <icinga-wm>	 PROBLEM - Host analytics1019 is DOWN: PING CRITICAL - Packet loss = 100%  
[19:05:12] <ottomata>	 oh, i guess apergos has done work on this too
[19:05:21] <ottomata>	 matanya: , will you coordinate with apergos?
[19:06:53] <apergos>	 I think he knows bout that ticket and added a bunch of gerrit changesets actually
[19:09:17] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: add postgresql-client [operations/puppet] - 10https://gerrit.wikimedia.org/r/120574 
[19:09:32] <grrrit-wm>	 (03PS3) 10Dzahn: decom: remove searchidx2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120145 
[19:11:56] <yurik>	 paravoid, do you have any objects to my new calculations at https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Reducing_image_quality_for_mobile
[19:13:28] <grrrit-wm>	 (03PS1) 10Ottomata: Fixing Stefan's public key [operations/puppet] - 10https://gerrit.wikimedia.org/r/120575 
[19:13:50] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Fixing Stefan's public key [operations/puppet] - 10https://gerrit.wikimedia.org/r/120575 (owner: 10Ottomata)
[19:14:05] <grrrit-wm>	 (03PS1) 10Cmjohnson: updating dns entry for analytics1019 [operations/dns] - 10https://gerrit.wikimedia.org/r/120576 
[19:14:50] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] updating dns entry for analytics1019 [operations/dns] - 10https://gerrit.wikimedia.org/r/120576 (owner: 10Cmjohnson)
[19:15:10] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] decom: remove searchidx2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120145 (owner: 10Dzahn)
[19:16:04] <cmjohnson1>	 ottomata: yours to make if cfg changes
[19:16:36] <matanya>	 ottomata: i'll try to catch him tomorrow 
[19:19:52] <mutante>	 !log searchidx2 - revoked puppet cert,remove from puppet,icinga,salt...
[19:19:58] <morebots>	 Logged the message, Master
[19:22:17] <ottomata>	 ok thanks cmjohnson1, works great
[19:22:18] <ottomata>	 on to an20?
[19:22:22] <ottomata>	 ready?
[19:22:26] <cmjohnson1>	 sure
[19:23:04] <cmjohnson1>	 ye[
[19:23:06] <cmjohnson1>	 yes
[19:23:07] <ottomata>	 you can go ahead and change dns
[19:23:30] <ottomata>	 !log stopping hadoop service on analytics1020 and shutting down for move to Row D
[19:23:35] <morebots>	 Logged the message, Master
[19:24:29] <grrrit-wm>	 (03PS1) 10Cmjohnson: updating dns for analytics1020 [operations/dns] - 10https://gerrit.wikimedia.org/r/120579 
[19:25:01] <icinga-wm>	 PROBLEM - Host analytics1020 is DOWN: PING CRITICAL - Packet loss = 100%  
[19:25:12] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] updating dns for analytics1020 [operations/dns] - 10https://gerrit.wikimedia.org/r/120579 (owner: 10Cmjohnson)
[19:30:50] <cmjohnson1>	 ottomata: booting now
[19:36:21] <mutante>	 !log shutdown searchidx2
[19:36:27] <morebots>	 Logged the message, Master
[19:37:25] <grrrit-wm>	 (03PS1) 10DamianZaremba: Moving labs icinga from pmtpa to eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/120581 
[19:40:26] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] Moving labs icinga from pmtpa to eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/120581 (owner: 10DamianZaremba)
[19:41:10] <grrrit-wm>	 (03CR) 10Dzahn: [V: 032] Moving labs icinga from pmtpa to eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/120581 (owner: 10DamianZaremba)
[19:41:32] <grrrit-wm>	 (03PS2) 10Ori.livneh: Geo-cookie: enable on one production text Varnish (cp1066) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119098 
[19:41:45] <ori>	 bblack: shall we start with that ^ ? i can deploy if you like
[19:42:59] <grrrit-wm>	 (03PS1) 10RobH: removing stat1002 access for amir & santhosh [operations/puppet] - 10https://gerrit.wikimedia.org/r/120583 
[19:43:46] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] removing stat1002 access for amir & santhosh [operations/puppet] - 10https://gerrit.wikimedia.org/r/120583 (owner: 10RobH)
[19:43:58] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[19:44:55] <grrrit-wm>	 (03PS2) 10RobH: removing stat1002 access for amir & santhosh [operations/puppet] - 10https://gerrit.wikimedia.org/r/120583 
[19:45:14] <grrrit-wm>	 (03PS1) 10DamianZaremba: Updating NRPE allowed hosts to new icinga instance (pmtpa to eqiad migration) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120584 
[19:47:08] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Straightforward enough." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120584 (owner: 10DamianZaremba)
[19:48:55] <grrrit-wm>	 (03CR) 10RobH: [C: 032] removing stat1002 access for amir & santhosh [operations/puppet] - 10https://gerrit.wikimedia.org/r/120583 (owner: 10RobH)
[19:49:03] <grrrit-wm>	 (03CR) 10coren: [V: 032] "... why did jenkins-bot only +1?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120584 (owner: 10DamianZaremba)
[19:49:41] <robh>	 Damianz: im merging your change live cuz its mixed with mine on palladium  Coren ^
[19:49:52] <Coren>	 robh: I was about to tell you the same.
[19:49:53] <mutante>	 Coren: because Damianz is the author and not in trusted user regex
[19:49:56] <robh>	 hehe
[19:50:08] <Damianz>	 I love that feature of zuul
[19:51:08] <Coren>	 Ah.  Interesting.  That seems a little pointles since there is still need for a +2 review.
[19:51:09] <mutante>	 Damianz: $somebody should just add you:)
[19:51:32] <matanya>	 he is away
[19:51:58] <Damianz>	 I really don't commit to puppet enough tbh
[19:51:58] <matanya>	 robh: when you are done with that, mind looking into https://rt.wikimedia.org/Ticket/Display.html?id=6133 and answer my question?
[19:52:52] <robh>	 its mid migration last i checked, i'll find out in a bit
[19:53:09] <robh>	 working access requests at the moment.
[19:55:16] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "also not in Icinga anymore" [operations/dns] - 10https://gerrit.wikimedia.org/r/120062 (owner: 10Dzahn)
[19:55:43] <mutante>	 !log DNS update - removing tampa search pools and searchidx2
[19:55:50] <morebots>	 Logged the message, Master
[20:00:08] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[20:03:05] <icinga-wm>	 RECOVERY - Host analytics1019 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms  
[20:03:15] <icinga-wm>	 RECOVERY - Host analytics1020 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms  
[20:05:26] <grrrit-wm>	 (03PS3) 10Ottomata: RT 7090 - adding analytics contact group to researchdb icinga alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 
[20:05:34] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] RT 7090 - adding analytics contact group to researchdb icinga alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 (owner: 10Ottomata)
[20:06:35] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Simple package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120574 (owner: 10coren)
[20:07:00] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: split HBA config from execnode [operations/puppet] - 10https://gerrit.wikimedia.org/r/120589 
[20:07:54] <subbu>	 !log deployed Parsoid fa03dd20 with deploy repo sha e4d28e7e
[20:08:00] <morebots>	 Logged the message, Master
[20:08:03] <grrrit-wm>	 (03PS2) 10coren: Tool Labs: split HBA config from execnode [operations/puppet] - 10https://gerrit.wikimedia.org/r/120589 
[20:09:35] <gwicke>	 yay for a working sudo setup for parsoid deploys!
[20:09:42] <andrewbogott>	 gwicke: I've finished moving the instances… http://parsoid.wmflabs.org/ doesn't look so good but I haven't investigated yet.
[20:10:33] <gwicke>	 andrewbogott, the code is on /data/project
[20:10:37] <andrewbogott>	 gwicke: well, actually, it looks like port 80 was never open for that instance, so… I don't know what that was for :)
[20:10:39] <gwicke>	 in case that makes a difference
[20:10:43] <grrrit-wm>	 (03PS3) 10Dzahn: let yuvipanda upload mobile tarballs [operations/puppet] - 10https://gerrit.wikimedia.org/r/119336 
[20:10:54] <gwicke>	 andrewbogott, we opened it in the firewall
[20:11:12] <andrewbogott>	 gwicke: I can do that now, but it was definitely not open before.  *shrug*
[20:11:35] <gwicke>	 all ports were open before
[20:11:55] <icinga-wm>	 PROBLEM - Packetloss_Average on oxygen is CRITICAL: packet_loss_average CRITICAL: 9.65246402062  
[20:12:04] <gwicke>	 we are using a few other ports as well
[20:12:15] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "updated per ticket comments, should have been Yuvi, and not mix 2 people in one change" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119336 (owner: 10Dzahn)
[20:12:15] <gwicke>	 ssh, another http port for the deb repo etc
[20:12:28] <gwicke>	 but I can fix that too
[20:13:33] <andrewbogott>	 omg, you're right, ports 1-65000 were open.
[20:13:37] <andrewbogott>	 I would… not encourage that!
[20:15:49] <icinga-wm>	 RECOVERY - Packetloss_Average on oxygen is OK: packet_loss_average OKAY: 1.44533255102  
[20:15:57] <andrewbogott>	 gwicke: it's not obvious to me how this web server was set up… I encourage you to ssh in and see what you can see.
[20:19:58] <grrrit-wm>	 (03CR) 10coren: [C: 032] "What's the worse that could happen?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120589 (owner: 10coren)
[20:23:26] <grrrit-wm>	 (03CR) 10RobH: [C: 04-1] "The linked RT ticket in question has raised questions to the need for this level of sudo access. My vote is indicated as a blocker until " [operations/puppet] - 10https://gerrit.wikimedia.org/r/119790 (owner: 10Ori.livneh)
[20:27:41] <andrewbogott>	 gwicke: ?
[20:28:10] <gwicke>	 andrewbogott, looking into it
[20:28:14] <andrewbogott>	 ok, thanks
[20:29:11] <gwicke>	 http://208.80.155.167:8000/ is working
[20:30:29] <andrewbogott>	 gwicke: are other things not working?  Or is that what it's supposed to do?
[20:31:09] <andrewbogott>	 http://parsoid.wmflabs.org:8000/ does the same
[20:31:48] <gwicke>	 andrewbogott, is DNS already updated?
[20:31:53] <andrewbogott>	 it is for me
[20:32:25] <gwicke>	 on my host it still point to the old location
[20:32:30] <gwicke>	 *points
[20:33:05] <andrewbogott>	 ok -- it'll catch up.
[20:33:16] <andrewbogott>	 Can I mark this as 'finished' or are there other issues?
[20:33:34] <gwicke>	 port 80 is working now too
[20:33:53] <gwicke>	 andrewbogott, as far as I can tell things are fine
[20:33:59] <gwicke>	 so thanks!
[20:34:09] <andrewbogott>	 cool.  Let me know if you run into any trouble.
[20:37:25] <Jamesofur>	 hmm, so my wiki appears to have gone missing
[20:37:53] <Jamesofur>	 (legalteamwiki) redirecting to wikimediafoundation.org ... it was working for the past couple weeks ... not seeing any apache changes that would do it..
[20:38:31] <ottomata>	 mark, do you have a minute to revew the archiva module?  i really just want someone to tell me if the rsync server i'm setting up is ok
[20:38:35] <grrrit-wm>	 (03CR) 10BryanDavis: "I can't see the RT ticket, but my $0.02 is that sudo is needed on the boxes to restart the logstash server when it forgets what it's suppo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119790 (owner: 10Ori.livneh)
[20:38:44] <ottomata>	 i can point you to what i'd like an opinion on
[20:39:05] <hashar>	 oh why does git-deploy never complete the fetches :(
[20:39:18] <gwicke>	 andrewbogott_afk, I'll remove the tampa IP now; labs-ns0.wikimedia.org still returns both IPs which results in round-robin
[20:53:09] <grrrit-wm>	 (03PS1) 10Tpt: Activate the "other projects" sidebar managed by Wikibase in frwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 
[20:55:26] <grrrit-wm>	 (03CR) 10Lydia Pintscher: [C: 031] "Good from PM side :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt)
[20:56:35] <paravoid>	 Reedy: around?
[20:56:36] <matanya>	 robh: what is the fate of ES in tampa? es4,7,8 
[20:56:50] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] "Every configuration change should have a bug: Please open and and link it in the commit summary. Don't forget to note that you have commun" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt)
[20:57:21] <matanya>	 bd808: you might know as well
[20:57:43] <bd808>	 matanya: What's up?
[20:58:02] <matanya>	 hi there, all good, hope you too :)
[20:58:16] <matanya>	 i'm wondering about tampa es servers
[20:58:25] <matanya>	 es4,7,8
[20:58:49] <cmjohnson1>	 what do you mean by fate?
[20:58:59] <cmjohnson1>	 matanya ^
[20:59:08] <matanya>	 moved to eqiad, decommed, recycle etc
[20:59:16] <mutante>	 searchidx2, just shut it down
[20:59:24] <bd808>	 matanya: No idea.
[20:59:30] <mutante>	 support contract litereally ends in 2 days
[20:59:33] <mutante>	 :)
[20:59:54] <matanya>	 searchidx2 is es servers ?
[21:00:00] <mutante>	 no
[21:00:01] <cmjohnson1>	 the es servers are relocating
[21:00:35] <matanya>	 any reason to keep them up now ?
[21:01:09] <grrrit-wm>	 (03PS1) 10RobH: Add csteipp & aaron to admins::jenkins group [operations/puppet] - 10https://gerrit.wikimedia.org/r/120596 
[21:01:13] <matanya>	 mutante: any ticket on those?
[21:01:15] <cmjohnson1>	 I think we have a decom ticket for es1-4 now...not sure about the other 2
[21:02:06] <matanya>	 if there is, i don't see it linked to 6099
[21:02:07] <mutante>	 matanya: 6266 
[21:02:16] <matanya>	 oh
[21:02:19] <mutante>	 es = external storage
[21:02:25] <matanya>	 external storage, not elastic search
[21:02:29] <spagewmf>	 andrewbogott_afk: I ran `sudo puppetd -tv` on ee-flow-extra (labs-vagrant role) has been running for an hour
[21:02:54] <mutante>	 matanya: yea, common confusion, we even had a bot trigger to explain because of that :p
[21:02:57] <mutante>	 !es
[21:03:08] <matanya>	 that is dead :)
[21:03:14] <mutante>	 yes :(
[21:03:22] <mutante>	 it knew all the server prefixes
[21:03:29] <matanya>	 so, not going yet
[21:03:36] <mutante>	 nope
[21:03:53] <matanya>	 thanks for clarifying this 
[21:04:05] <mutante>	 6266 getting an update would still be great!
[21:04:15] <mutante>	 i don't know much more than that
[21:04:46] <matanya>	 who is in charge of that? aaron ?
[21:05:07] <Reedy>	 paravoid: Aye
[21:05:09] <grrrit-wm>	 (03PS2) 10John F. Lewis: Activate the "other projects" sidebar managed by Wikibase in frwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt)
[21:05:36] <grrrit-wm>	 (03CR) 10John F. Lewis: [C: 031] "Bug now attached, looks good." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt)
[21:05:41] <paravoid>	 Reedy: hey, we got an RT by James_F, legalteam wiki redirects to foundationwiki for some reason
[21:06:02] <grrrit-wm>	 (03CR) 10RobH: [C: 032] Add csteipp & aaron to admins::jenkins group [operations/puppet] - 10https://gerrit.wikimedia.org/r/120596 (owner: 10RobH)
[21:06:05] <paravoid>	 I can't reproduce it on the appservers, any ideas on what would generate this?
[21:06:16] <paravoid>	 I'm suspecting a stale appserver...
[21:06:18] <JohnLewis>	 paravoid: *jamesofur :p
[21:06:22] <mutante>	 matanya: i think springle.. but ..
[21:06:35] <paravoid>	 oh, sorry
[21:06:49] <mutante>	 paravoid: 10
[21:06:55] <paravoid>	 10?
[21:06:56] <mutante>	 oops, ignore that
[21:06:58] <matanya>	 late night for him, will try to ask him tomorrow
[21:07:11] <JohnLewis>	 But yeah - It keeps redirecting to foundationwiki. DNS, Apache and all that looks good but hey - I'm not a professional ops guy :p
[21:07:26] <matanya>	 mutante: if you want to get rid of hume: https://gerrit.wikimedia.org/r/#/c/74591/ is the last blocker
[21:07:36] <Reedy>	 paravoid: Have we got an example URL?
[21:07:38] <mutante>	 paravoid: nothing, wrong usage of irc client :p
[21:08:17] <matanya>	 actully, i should point Reedy to that blocker, not you mutante 
[21:08:34] <mutante>	 but yea, who moves the es boxes
[21:08:47] <matanya>	 Reedy: mind if i finish that one ? 
[21:09:08] <qchris>	 ottomata: Yes, everything in labs that I care about got migrated.
[21:09:08] <Reedy>	 sure
[21:09:18] <ottomata>	 great
[21:09:27] <matanya>	 thanks Reedy 
[21:09:40] <mutante>	 matanya: well, i got you some cleanup on maintenance.pp, rebase it
[21:09:49] <matanya>	 i saw 
[21:10:01] <matanya>	 i rebase more than any other git command
[21:12:12] <paravoid>	 found it
[21:12:13] <paravoid>	 mw1163
[21:13:39] <paravoid>	 Reedy: I'm resyncing apache-config to mw1163; can you double-check that mediawiki is deployed as it should
[21:13:43] <paravoid>	 also, this used to work...
[21:13:53] <paravoid>	 boxes coming back after downtime and getting the new config
[21:15:04] <grrrit-wm>	 (03PS2) 10BBlack: GeoIP cookie: expand deployment from Labs to production [operations/puppet] - 10https://gerrit.wikimedia.org/r/119014 (owner: 10Ori.livneh)
[21:16:57] <mutante>	 paravoid: Reedy, used apache-fast-test with pybal option?
[21:17:14] <mutante>	 it actually checks all pooled servers and if they are all the same it summarizes
[21:17:23] <paravoid>	 I did not, I used salt
[21:17:25] <paravoid>	 but good to know, thanks
[21:17:26] <mutante>	 it's quickest to detect rogue apache
[21:17:32] <Reedy>	 mutante: That's why I asked for an example url ;)
[21:18:01] <mutante>	 if you get only line as a result it means they were all identical
[21:18:07] <paravoid>	 (I fixed it)
[21:18:30] <grrrit-wm>	 (03PS8) 10Matanya: Properly puppeti[sz]e purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[21:18:37] <matanya>	 Reedy: mutante ^ 
[21:19:26] <matanya>	 arrg white space
[21:20:05] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] GeoIP cookie: expand deployment from Labs to production [operations/puppet] - 10https://gerrit.wikimedia.org/r/119014 (owner: 10Ori.livneh)
[21:20:13] <grrrit-wm>	 (03PS1) 10RobH: adding dumps.wikimedia.org certificate [operations/puppet] - 10https://gerrit.wikimedia.org/r/120598 
[21:21:16] <Reedy>	 dumps is getting a cert?
[21:21:36] <grrrit-wm>	 (03PS9) 10Matanya: Properly puppeti[sz]e purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[21:22:25] <robh>	 Reedy: apergos wants to support https so yep
[21:22:45] <grrrit-wm>	 (03CR) 10RobH: [C: 032] adding dumps.wikimedia.org certificate [operations/puppet] - 10https://gerrit.wikimedia.org/r/120598 (owner: 10RobH)
[21:22:58] <matanya>	 seems ready to me
[21:23:33] <robh>	 did someone merge my change on palladium?
[21:23:55] <Reedy>	 Sweet
[21:24:00] <MaxSem>	 considering lack of auth and pubicity of data published, the only thing someone can sniff is what dump someone is downloading. but that's also sniffable from the size of transfer:)
[21:24:05] <matanya>	 if someone will review this, i can go forward and decom hume. ops, please ? :)
[21:24:15] <Reedy>	 apergos: dataset2 doesn't seem to redirect to dumps :(
[21:24:15] <Reedy>	 http://dataset2.wikimedia.org/
[21:24:20] <apergos>	 no
[21:24:23] <apergos>	 it should not
[21:24:34] <apergos>	 people shouldn't really use those hostnames for web access
[21:24:38] <Reedy>	 how come?
[21:24:39] <Reedy>	 lol
[21:24:55] <apergos>	 because we might be using one server or another
[21:25:10] <matanya>	 apergos: I you want help with 6789 i'll be glad to help
[21:25:11] <apergos>	 like, is it dataset2? dataset1001? the server at $newdc when it happens?
[21:25:13] <Reedy>	 sorry
[21:25:20] <Reedy>	 I was meaning..
[21:25:24] <matanya>	 *if
[21:25:37] <Reedy>	 why doesn't dataset2 -> dumps. not why shouldn't people be using dataset2.wm.o
[21:25:42] <Reedy>	 Currently cleaning up
[21:25:42] <Reedy>	 <exclusion pattern="^http://(?:apt|brewster|bug-attachment|cs|cz|dataset2|download|ekrem|emery|search|sitemap|status|torrus|ubuntu|wiki-mail)\.wikimedia\.org" />
[21:25:52] <apergos>	 well it should not redirect there
[21:26:18] <apergos>	 http://dataset2.wikimedia.org/backup-index.html
[21:26:34] <apergos>	 if someone really wants the files and they use that hostname, there they are
[21:26:57] <apergos>	 but in fact those files are 2 hours out of date since they are rsynced from the primary (now in eqiad)
[21:27:12] <Reedy>	 right
[21:27:14] <apergos>	 what is this exclusion pattern?
[21:27:17] <Reedy>	 so it's gotta stay for now :(
[21:27:20] <apergos>	 maybe that's what I'm missing
[21:27:20] <Reedy>	 https everywhere
[21:27:22] <apergos>	 ah
[21:27:27] <apergos>	 ok so that's aother story
[21:27:32] <mutante>	 :)
[21:27:40] <mutante>	 likes anything that makes that regex shorter
[21:27:41] <apergos>	 dataset2, if someone ever uses it, yeah it's going to have a bad cert (one day)
[21:27:46] <Reedy>	 it had a whole host (hah!) of NXDOMAIN
[21:27:48] <apergos>	 and dataset1001 the same
[21:27:51] <mutante>	 and anything that updates https://wikitech.wikimedia.org/wiki/Httpsless_domains
[21:28:08] <apergos>	 but dumps and download will sooon be https, what file is that in? I need to make a note to update that
[21:28:19] <mutante>	 upstream ?
[21:28:21] <apergos>	 ah
[21:28:22] <mutante>	 EFF
[21:28:24] <Reedy>	 I'm just making an update to it now
[21:28:24] <apergos>	 yeah still
[21:28:42] <Reedy>	 guess I should leave download|dumps|sitemap in for now...
[21:28:43] <apergos>	 if you have a link I'll update it when download becomes httpsable
[21:29:16] <apergos>	 and I'm too tired to even groan at the host pun so nyah
[21:29:41] <Reedy>	 I make a commit in my fork on github and poke them on OFTC in #https-everywhere
[21:29:44] <Reedy>	 <exclusion pattern="^http://(?:apt|brewster|bug-attachment|cs|cz|dataset2|downloads|dumps|ekrem|emery|search|status|sitemap|torrus|ubuntu|wiki-mail)\.wikimedia\.org" />
[21:30:24] <apergos>	 MaxSem: it's not that ooooh someone could sniff, it's sthat people want to turn on httpseverywhere and I've gotten complaints (there's ben an open bug for awhile)
[21:30:59] <apergos>	 someday there will be no http.  and maybe that will make the internet a tiny bit safer
[21:31:04] <mutante>	 what's with "cs" and "cz"
[21:31:13] <apergos>	 or make the nsa spooks lose their hair a little sooner...
[21:32:17] <Reedy>	 https://github.com/reedy/https-everywhere/tree/Wikimedia24032014
[21:32:21] <apergos>	 I dunno, those are weird aren't they
[21:32:45] <apergos>	 ok reedy thanks for that
[21:33:17] <Reedy>	 Feel free to ping/poke me to make another commit when it's done and I'll get it upstreamed
[21:33:22] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: tweaks to xcrontab [operations/puppet] - 10https://gerrit.wikimedia.org/r/120600 
[21:34:11] <apergos>	 yay
[21:34:41] <matanya>	 Coren: icehouse next?
[21:44:55] <grrrit-wm>	 (03Abandoned) 10Dr0ptp4kt: WIP: DO NOT MERGE YET. Split up tagging for baselining period. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119781 (owner: 10Dr0ptp4kt)
[21:52:01] <matanya>	 mutante: you can update  yongle.wikimedia.org 
[21:52:09] <matanya>	 it doesn't exist
[21:53:50] <mutante>	 update where? you meant the regex?
[21:53:53] <mutante>	 then it's Reedy
[21:54:03] <Reedy>	 matanya: https://wikitech.wikimedia.org/w/index.php?title=Httpsless_domains&action=edit
[21:54:05] <matanya>	  https://wikitech.wikimedia.org/wiki/Httpsless_domains
[21:54:31] <matanya>	 Reedy: what is this??? a wiki? how do you edit it?
[21:54:42] <matanya>	 :P
[21:54:47] <Reedy>	 Ctrl + A
[21:54:48] <Reedy>	 Delete
[21:54:50] <Reedy>	 Save
[21:55:19] <matanya>	 the wiki way of rm -rf / :)
[21:55:40] <matanya>	 done
[21:55:41] <mutante>	 help, no visual editor?:)
[21:56:28] <matanya>	 kinda, apart from the fact i used VE about 20 times total
[21:57:47] <matanya>	 Reedy: most of those redirects should go away
[21:57:57] <matanya>	 esp those pointing to tampa
[21:58:59] <mutante>	 matanya: the RT links in that wiki page, they all used to work
[21:59:08] <mutante>	 somebody deleted/broke template or something
[21:59:44] <matanya>	 used to
[21:59:52] <matanya>	 well i'll check those too
[22:00:03] <matanya>	 my pile is growing lately
[22:01:10] <mutante>	 or, just replace with BZ links
[22:01:26] <matanya>	 i'll review it in saner hours
[22:01:35] <grrrit-wm>	 (03CR) 10coren: [C: 032] Tool Labs: tweaks to xcrontab [operations/puppet] - 10https://gerrit.wikimedia.org/r/120600 (owner: 10coren)
[22:02:54] <mutante>	 matanya: move it to the very bottom of the list:)
[22:03:51] <mutante>	 how's stats.wm doing 
[22:03:59] <mutante>	 i think i fixed that
[22:04:06] <mutante>	 some time ago
[22:04:28] <mutante>	 yep https://bugzilla.wikimedia.org/show_bug.cgi?id=32143
[22:07:44] <matanya>	 cool
[22:08:26] <matanya>	 ok, mutante in your own time please https://gerrit.wikimedia.org/r/#/c/74591/ then i'll push hume decom patch 
[22:09:48] <grrrit-wm>	 (03PS1) 10Dzahn: decom snapshot1-4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120615 
[22:10:55] <matanya>	 no RT linked in this ^ 7097
[22:11:42] <grrrit-wm>	 (03PS2) 10Matanya: decom snapshot1-4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120615 (owner: 10Dzahn)
[22:13:50] <mutante>	 Reedy: purge_checkuser  .so that should already exist, just wasnt puppetize. does that sound right to you
[22:14:00] <mutante>	 looks for it on terbium
[22:14:27] <mutante>	 i just see other "purge"things
[22:14:47] <mutante>	 oh, hume of course, i guess
[22:14:53] <mutante>	 that was the blocker mentioned earlier:)
[22:15:59] <grrrit-wm>	 (03PS1) 10Gilles: Add ?download parameter to images [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 
[22:17:22] <grrrit-wm>	 (03CR) 10Gilles: "This is untested because I don't know how to get varnish working on vagrant. If there's an easy way to set that up, please let me know." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[22:22:17] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "+1 for puppet code, but where is this actually running now and does it work?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[22:23:51] <grrrit-wm>	 (03CR) 10Reedy: "I think it's a manual cronjob entry somewhere on hume..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[22:24:44] <mutante>	 Reedy: can't find :p
[22:24:47] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: Make xcrontab support @syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/120682 
[22:28:47] <grrrit-wm>	 (03CR) 10Dzahn: "i can't seem to find it on hume, which user? tried apache,root,mwdeploy , /etc/cron.d/ ..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[22:29:44] <Jamesofur>	 robh: thanks for doing some of those mailing list bugs
[22:29:48] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] Add ?download parameter to images (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[22:30:04] <robh>	 Jamesofur: im looping though them all, quite welcome
[22:30:13] <JohnLewis>	 robh: If I knew it was you - I would have thanked you on IRC :p
[22:32:48] <grrrit-wm>	 (03CR) 10coren: [C: 032] Tool Labs: Make xcrontab support @syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/120682 (owner: 10coren)
[22:34:51] <robh>	 heh
[22:40:59] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Move the labs ganglia aggregator to eqiad. [operations/puppet] - 10https://gerrit.wikimedia.org/r/120687 
[22:41:01] <robh>	 when someone asks for a list, and we ask followup on what to call it
[22:41:07] <robh>	 and it sits since last july
[22:41:11] <robh>	 im just rejecting it...
[22:43:19] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Move the labs ganglia aggregator to eqiad. [operations/puppet] - 10https://gerrit.wikimedia.org/r/120687 (owner: 10Andrew Bogott)
[22:44:16] <icinga-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC  
[22:47:31] <mutante>	  89     fixed-address statistics1.wikimedia.org;
[22:47:33] <mutante>	 heh
[22:49:42] <grrrit-wm>	 (03PS1) 10Dzahn: decom ms5 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120689 
[22:53:15] <grrrit-wm>	 (03PS2) 10Gilles: Add ?download parameter to images [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 
[22:53:42] <grrrit-wm>	 (03CR) 10Gilles: Add ?download parameter to images (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[22:54:31] <grrrit-wm>	 (03PS1) 10Dzahn: remove "statistics1" from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/120694 
[22:57:50] <bd808>	 !log Forced /srv/scap to update to c771a46 across the cluster
[22:57:56] <morebots>	 Logged the message, Master
[22:58:37] <RoanKattouw>	 bd808: Are you doing the SWAT today?
[22:59:28] <aude>	 we have things to deploy
[22:59:36] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] "Now you set headers on the request object, not the response (that's not what you want). Also you will probably produce invalid request url" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[22:59:43] <bd808>	 RoanKattouw: No. Just pushing a fix for a bug Antoine found in the scap code itself
[23:00:00] <aude>	 making patch to update wmf19 submodule
[23:00:12] <RoanKattouw>	 bd808: OK. You know that maaaybe you shouldn't do that during a scheduled deployment window?
[23:00:13] <hashar>	 RoanKattouw: I commited a trivial PHP fatal error.
[23:00:18] <RoanKattouw>	 hah whoops
[23:00:21] <hashar>	 RoanKattouw: should have left bd808 to +2 the change though
[23:00:41] <hashar>	 RoanKattouw: double whops is that Jenkins only runs php -l on files ending with .php
[23:00:55] <bd808>	 RoanKattouw: You're right; I appologize
[23:00:57] <mutante>	 !quip
[23:01:13] <RoanKattouw>	 bd808: It's OK, it's just Yet Another Reason why we need a locking system
[23:01:26] * bd808  nods

[23:01:59] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "whatever this was, it's not stat1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120694 (owner: 10Dzahn)
[23:02:37] <bd808>	 Short of blocking ssh to tin it's going to be pretty hard to accomplish
[23:02:51] <hoo>	 wrong approach :P
[23:03:09] <bd808>	 Even with trebuchet something like what I just did would work fine
[23:03:35] <hoo>	 You could hack that using git hook (although these atm don't even seem to work properly)
[23:03:42] <hoo>	 * a git hook
[23:03:57] <greg-g>	 vim
[23:04:15] <bd808>	 A git hook won't stop modifications to the working copy which is what scap operates on
[23:04:30] <greg-g>	 that, in more words, than my simple: "vim"
[23:04:52] <hoo>	 bd808: Right, but no one is supposed to do that, right
[23:05:22] <bd808>	 trebuchet only stops `git deploy start` with it's lock
[23:05:39] <RoanKattouw>	 OK so who's doing the SWAT?
[23:05:56] <aude>	 https://gerrit.wikimedia.org/r/#/c/120697/
[23:06:07] <aude>	 i don't know how this works
[23:06:10] <hoo>	 I don't crea... I can do it
[23:06:27] <aude>	 do we deploy or does a swat deployer do it?
[23:06:44] <greg-g>	 a swatter
[23:06:57] <greg-g>	 ori: ebernhardson ^ ? 
[23:06:58] <bd808>	 hoo: The answer to your question is … complicated. Train deploys currently do make "live" changes on tin before adding them to gerrit and reverting the local changes to tin.
[23:07:06] <mutante>	 gerrit doesnt merge things for me.. did i miss out?
[23:07:26] <aude>	 greg-g: ok
[23:07:31] <hoo>	 bd808: Yeah... there are edge cases... also recreating the iw-map and stuff
[23:07:49] <grrrit-wm>	 (03CR) 10Dzahn: [V: 032] remove "statistics1" from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/120694 (owner: 10Dzahn)
[23:07:50] <greg-g>	 ori: ebernhardson now with content in my ping: time for a SWAT?
[23:09:17] <mutante>	 Submitted, Merge Pending
[23:09:35] <ori>	 am I needed to deploy something?
[23:09:45] <hoo>	 ori: yup, please do us
[23:09:50] <hoo>	 https://gerrit.wikimedia.org/r/#/c/120697/
[23:09:51] <mutante>	 let's see if you can merge
[23:09:53] <hoo>	 * our stuff
[23:09:55] <mutante>	 i cant
[23:09:58] <mutante>	 but other repo
[23:10:11] <aude>	 it's for test.wikidata only
[23:10:22] <ori>	 greg-g: ok with you?
[23:10:51] <greg-g>	 yeah
[23:11:32] <ori>	 cool. +2'd, gerrit running gate-and-submit.
[23:11:40] <aude>	 :)
[23:12:58] <grrrit-wm>	 (03CR) 10Dzahn: "still "Submitted, Merge Pending".." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120694 (owner: 10Dzahn)
[23:13:27] <ori>	 bblack: whoa, i completely missed the deployment
[23:13:28] <ori>	 thanks!
[23:13:29] <greg-g>	 TimStarling: you haven't filled up the disk on gallium or anything weird like that have you?
[23:13:35] <greg-g>	 :)
[23:14:12] <TimStarling>	 I haven't filled up the disk, hopefully I haven't done anything weird
[23:14:22] <TimStarling>	 I just ran some tests
[23:14:49] <greg-g>	 k, we'll give it a minute, just mutante has been having issues with jenkins in the last 15ish minutes
[23:15:11] <greg-g>	 there it goes
[23:15:24] <grrrit-wm>	 (03CR) 10Gilles: "The requests will be coming from Media Viewer, where it will be just "?download" and no other parameter. "=1" won't happen, it's not neces" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[23:16:01] <mutante>	 greg-g: TimStarling "16:15 < hashar> mutante: that is Gerrit JGit backend having trouble/delay  merging a commit apparently
[23:16:52] * greg-g  shrugs

[23:16:59] <mutante>	 greg-g: ignore me, my bad
[23:17:09] <mutante>	 that one had a dependency it should not have had
[23:17:12] <TimStarling>	 well, I may possibly be doing bad things to it
[23:17:22] <logmsgbot>	 !log ori synchronized php-1.23wmf19/extensions/Wikidata  'I6ffb304d3: Update Wikidata'
[23:17:26] <greg-g>	 TimStarling: yeah, where my mind went...
[23:17:28] <morebots>	 Logged the message, Master
[23:17:30] <TimStarling>	 like writing to a shared sqlite database when I shouldn't be
[23:17:34] <ori>	 ^ aude
[23:17:37] <aude>	 yay!
[23:17:47] * aude  checks test wikidata

[23:17:48] <hoo>	 ok
[23:17:52] <ori>	 hoo, aude: can you...yeah
[23:17:54] <ori>	 thanks :)
[23:18:44] <aude>	 gah, issue in PageTranslationHooks
[23:18:53] <aude>	 i'll poke around more
[23:18:57] <hoo>	 what?
[23:18:58] * greg-g  heads out

[23:19:10] <aude>	 give me a minute
[23:19:16] <greg-g>	 oh
[23:19:23] <ori>	 greg-g: should I sync the fix-up?
[23:19:25] <ori>	 I don't mind
[23:19:49] <greg-g>	 ori: yes, please
[23:19:54] <greg-g>	 thanks
[23:20:04] <ori>	 OK. make your train! :P
[23:20:10] <greg-g>	 :)
[23:20:12] <ori>	 the train-train I mean, not the dpeloyment train!
[23:20:15] <greg-g>	 (bus)
[23:20:22] <ori>	 deployment bus?
[23:20:28] <greg-g>	 not the same ring to it
[23:20:37] * greg-g  gos

[23:20:39] <greg-g>	 +e
[23:22:02] <aude>	 might make another submodule update
[23:22:30] <RoanKattouw>	 ori: So are you today's SWATter?
[23:23:06] <ori>	 i guess so, wouldn't mind splitting the job tho ;)
[23:23:16] <RoanKattouw>	 OK
[23:23:25] <RoanKattouw>	 Did you do all the Wikibase stuff already?
[23:23:28] <RoanKattouw>	 Then I can just do the VE stuff
[23:24:09] <grrrit-wm>	 (03PS1) 10Ori.livneh: CentralNotice: Re-set $wgCentralGeoScriptURL to false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120699 
[23:24:13] <hoo>	 aude: Do you have the ext. locally?
[23:24:27] <ori>	 RoanKattouw: wikidata needs to follow-up with a fix-up
[23:24:29] <RoanKattouw>	 OK
[23:24:31] <ori>	 still waiting for it
[23:24:44] <RoanKattouw>	 ori: Let me know when is a good time to start doing the VE stuff
[23:24:50] <aude>	 hoo: i don't but think i see the issue
[23:25:30] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] CentralNotice: Re-set $wgCentralGeoScriptURL to false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120699 (owner: 10Ori.livneh)
[23:25:40] <grrrit-wm>	 (03Merged) 10jenkins-bot: CentralNotice: Re-set $wgCentralGeoScriptURL to false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120699 (owner: 10Ori.livneh)
[23:25:50] <logmsgbot>	 !log ori updated /a/common to {{Gerrit|I42e9c8c97}}: CentralNotice: Re-set $wgCentralGeoScriptURL to false
[23:25:55] <hoo>	 aude: Has that been there before?
[23:25:56] <morebots>	 Logged the message, Master
[23:26:18] <aude>	 hoo: ?
[23:26:25] <logmsgbot>	 !log ori synchronized wmf-config/CommonSettings.php  'I42e9c8c97: CentralNotice: Re-set $wgCentralGeoScriptURL to false'
[23:26:31] <morebots>	 Logged the message, Master
[23:26:50] <hoo>	 aude: I mean the fatal, have you checked test before
[23:27:06] <hoo>	 I wasn't on it today
[23:27:10] <aude>	 it's new and have a paptch for it
[23:27:16] <aude>	 patch*
[23:27:34] <aude>	 no idea how i can't reproduce these issues on my test wiki, though don't have translate
[23:28:08] <aude>	 even on beta wikidata
[23:28:16] <ori>	 aude: would it make sense to have Roan proceed with the VE swat, and we'll ask to interject with your fix-up if/when it's ready?
[23:28:18] <aude>	 no issue
[23:28:26] <aude>	 ori: go ahead
[23:28:29] <ori>	 RoanKattouw: ^
[23:28:34] <aude>	 would take us time to make a new build
[23:28:37] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "that mount was already ensured absent and did not exist on hume, which is the only node using nfs::upload" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120689 (owner: 10Dzahn)
[23:33:29] <grrrit-wm>	 (03CR) 10Hoo man: "> The requests will be coming from Media Viewer, where it will be just "?download" and no other parameter. "=1" won't happen, it's not nec" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[23:34:33] <ori>	 RoanKattouw: you can go ahead, in case you missed my earlier message
[23:35:07] <aude>	 as always, takes time to wait for jenkins etc. so go ahead
[23:39:35] <grrrit-wm>	 (03PS1) 10Dzahn: decom - remove ms5 [operations/dns] - 10https://gerrit.wikimedia.org/r/120705 
[23:40:25] <grrrit-wm>	 (03CR) 10Gilles: "Yes, people will find out about those URLs. That's why I'm being cautious, for example not settings the response header based on the custo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[23:40:27] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "removed from puppet in I8530218e320364849d1ae1ccfd1cea7a18489e90" [operations/dns] - 10https://gerrit.wikimedia.org/r/120705 (owner: 10Dzahn)
[23:42:18] <grrrit-wm>	 (03CR) 10Gilles: "Re-reading your remarks, you might have missed the fact that I don't clean "=bla" because it doesn't get through, the only recognized URL " [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[23:49:33] <grrrit-wm>	 (03CR) 10Hoo man: "Yikes... that's what I get for only skimming over this change, sorry for the oversights." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles)
[23:49:40] <ori>	 aude: I have a hard stop in ten minutes
[23:49:46] <aude>	 ok
[23:49:53] <hoo>	 we will be there by then
[23:49:54] <aude>	 updating submodule now
[23:50:16] <RoanKattouw>	 ori: Crap, sorry
[23:50:36] <RoanKattouw>	 I went upstairs to fix a travel problem and forgot all about the SWAT thing
[23:50:38] <RoanKattouw>	 I'll do it now
[23:50:41] <ori>	 RoanKattouw: no worries at all
[23:51:38] <ori>	 !log Switched CentralNotice to GeoIP cookie rather than bits.wm.o/geoiplookup script tags
[23:51:43] <morebots>	 Logged the message, Master
[23:52:37] <aude>	 ori: https://gerrit.wikimedia.org/r/#/c/120710/
[23:52:39] <hoo>	 ori, RoanKattouw: https://gerrit.wikimedia.org/r/120710
[23:52:43] <aude>	 :)
[23:53:16] <James_F>	 Coren: Is the very slow speed of Beta Labs a more general issue, and does it have an end in sight? Was about to launch a round of user testing…
[23:53:25] <RoanKattouw>	 hoo: Ok I'll include that one
[23:53:57] <Coren>	 James_F: As far as I know, everything in labs is working fine.  This is the first I hear of it.
[23:54:07] <James_F>	 Coren: Ah. Oh dear. :-(
[23:54:25] <James_F>	 Coren: Yeah, it's been near-catastrophic for the past few hours.
[23:54:53] <James_F>	 Coren: E.g. beta-bits ERR_CONNECTION_REFUSED, exceedingly slow loads, and all that fun.
[23:55:01] <James_F>	 Coren: Sorry, I assumed it was known. :-(
[23:56:02] <ori>	 RoanKattouw: OK, seems like you're taking over that one (which is great for me), so I'm signing off
[23:56:09] <ori>	 unless you'd like me to stick around
[23:56:13] <aude>	 thanks ori 
[23:56:24] <RoanKattouw>	 ori: Yup I got it
[23:56:30] <Coren>	 A quick look at other labs projects doesn't reveal any performance issue.  It's something on beta afaict.  Have you talked to Hashar?  I know he's busy doing migration, there might be something he's doing that's consuming resources?
[23:56:30] <ori>	 thanks, *wave*
[23:56:34] <RoanKattouw>	 Sorry again for the unexplained absence
[23:57:14] <mutante>	 !log ms5 - removed from icinga,puppet,storedconfigs,salt...
[23:57:20] <morebots>	 Logged the message, Master
[23:57:51] <James_F>	 Coren: I didn't ask him before he disconnected for the night, no.
[23:58:19] <Coren>	 I seem to remember hearing him talk about a database dump; perhaps that's what is going on?
[23:58:35] <James_F>	 Ah.
[23:58:41] <MaxSem>	 Coren, staging.wmflabs.org is also slow as hell