[00:00:55] TimStarling: so, any perf concerns from moving HTTP_X_Forwarded_Proto earlier in all.conf? [00:01:17] TimStarling: so that it can be used in redirect targets for other conf files [00:01:20] is that a gerrit change? [00:01:24] not yet [00:02:00] i made some other changes to the redirects system. escaping and a new line at EOF [00:02:08] (those are already merged) [00:02:20] (03PS1) 10Lcarr: Adding in semicolon for proper ferm syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/107516 [00:02:46] you are referring to: [00:02:46] RewriteRule . - [E=HTTP_X_Forwarded_Proto:%{HTTP:X-Forwarded-Proto}] [00:02:46] RewriteCond %{ENV:HTTP_X_Forwarded_Proto} !=https [00:02:47] RewriteRule . - [E=HTTP_X_Forwarded_Proto:http] [00:03:02] ? [00:03:13] yes [00:03:41] PROBLEM - LibreNMS HTTPS on netmon1001 is CRITICAL: Connection timed out [00:03:46] jenkins you are so slow today..... a watched jenkins never code reviews [00:03:47] well, this cannot be in the server context [00:04:35] should i just copy it to other files? [00:04:38] ok, well the docs say it can be in the server context, but how does that make sense? [00:04:53] how not? [00:05:01] (03CR) 10Lcarr: [C: 032] Adding in semicolon for proper ferm syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/107516 (owner: 10Lcarr) [00:05:17] btw, a use case: https://wikipedia.org/foo [00:05:33] ah, here we go: "Note that rewrite configurations are not inherited by virtual hosts. This means that you need to have a RewriteEngine on directive for each virtual host in which you wish to use rewrite rules." [00:05:43] hah [00:05:43] makes more sense now, yes? [00:06:15] so a RewriteRule in the server context won't be executed if the request matches a [00:06:31] TimStarling: so what say you? copy it? [00:06:37] yes, copy it [00:08:49] hah, how old is this? wwwportals.conf:31: RewriteRule ^/(mailman|pipermail)(.*)$ http://mail.wikipedia.org/$1$2 [R=301,L] [00:15:11] PROBLEM - NTP on netmon1001 is CRITICAL: NTP CRITICAL: No response from NTP server [00:35:51] cajoel and i are looking at ntp on netmon1001 [00:39:06] !log on ssl1, testing CPU hotplug feature for possible power impact [00:39:11] Logged the message, Master [00:44:13] made no difference, for the record [00:45:13] it's the thought, etc. [00:50:11] (03PS1) 10Aaron Schulz: Try increasing scap fanout to 40 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107522 [00:57:11] TimStarling: Did you turn of NSA mode while you were there? [00:57:17] *off [00:58:52] don't be silly [00:59:09] you know there's no way to turn off NSA mode without physically reflashing the RAC firmware [01:00:05] (03PS2) 10Ori.livneh: Try increasing scap fanout to 40 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107522 (owner: 10Aaron Schulz) [01:00:14] (03CR) 10Ori.livneh: [C: 032 V: 032] Try increasing scap fanout to 40 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107522 (owner: 10Aaron Schulz) [01:01:30] at LCA there was a keynote by Matthew Garrett, he talked about boot security and how there's no way to tell whether or not you've been hacked by the NSA [01:02:12] he said that if you want to work with something that you can trust completely, you should work with sheep or cattle, not computers [01:07:38] Pfft [01:07:45] I'm not sure you can trust cattle [01:07:51] That can maul and trample you pretty easily [01:08:38] maul? [01:08:53] Not the right word.. [01:09:01] not the right cattle [01:10:05] AaronSchulz: ran puppet on tin, if you want to test [01:10:39] ok [01:10:56] There's a word for it... [01:11:01] Headbutted I guess explains it [01:11:35] or gored [01:11:50] What's the proper way to make tests for stuff that requires MediaWiki to be run by a web server? [01:12:03] uh, wrong channel. [01:12:21] I wasn't really meaning with horns [01:13:11] RECOVERY - NTP on netmon1001 is OK: NTP OK: Offset -0.001911044121 secs [01:21:12] springle: seem to be a lot of slow Block::newLoad queries on enwiki? Where some cold slaves brought in or something? [01:21:23] I don't think that's a new query [01:25:15] ok so lesliecarr and cajoel were working on ferm on netmon1001 to support pmacct, and the firewall changes blocked NTP for a while. for the moment i've manually added a rule to allow connections to udp dpt:123, looking at how do this the Ferm Way.. [01:25:39] I would remove ferm [01:25:42] it wasn't there before [01:26:09] we added it as an attempt to manage the nat redirects [01:26:21] (signing off for real now) [01:26:51] AaronSchulz: yep, db1049 got hammered by those yesterday on warm up. it was missing an index. but in the process of fixing I also found bug 60035 helping to make it all slow [01:27:19] jgage: hrm, it started blocking ? [01:27:46] the default ruleset it loaded did not have a hole for udp:123 [01:27:53] so, yes [01:28:21] :( [01:29:19] springle: I can look at the query generation [01:29:36] great! [01:29:43] i didn't get a chance [01:32:25] jgage: what do you think about adding udp123 as a default accept for ferm ? [01:33:38] seems reasonable [01:34:48] interesting that the input chain is default-deny, then allows all tcp, but udp only to 500. [01:36:27] which is weird.... [01:38:52] (03PS1) 10Lcarr: allow udp by default with ferm [operations/puppet] - 10https://gerrit.wikimedia.org/r/107532 [01:39:14] jgage: https://gerrit.wikimedia.org/r/#/c/107532/ ? [01:41:12] (03CR) 10Gage: [C: 032] allow udp by default with ferm [operations/puppet] - 10https://gerrit.wikimedia.org/r/107532 (owner: 10Lcarr) [01:41:38] :) [01:42:44] woot i puppet merged it [01:43:16] cool [01:43:50] IMHO that should have belonged in base::firewall. [01:45:18] (03PS1) 10Lcarr: dport is important [operations/puppet] - 10https://gerrit.wikimedia.org/r/107534 [01:45:50] ah [01:46:04] we should have these things in one location .... [01:46:09] that's it, i quit! [01:46:18] heh [01:46:24] (i am too easily amused because that's still not yet old) [01:46:29] (03CR) 10Lcarr: [C: 032 V: 032] dport is important [operations/puppet] - 10https://gerrit.wikimedia.org/r/107534 (owner: 10Lcarr) [01:48:05] ok woo [01:48:18] that is making puppet happy-ish [01:48:23] still wants ip6table love [01:53:38] looks like you just need to prepend domain (ip ip6) [01:54:34] well... that sounds like what i need to do tomorrow ;) [01:54:38] au revoir! [01:57:19] grrrit-wm: OK for me to grab an LD for Roan for tomorrow? Single VE change, backport. [01:57:21] Err. [01:57:29] greg-g: OK for me to grab an LD for Roan for tomorrow? Single VE change, backport. [01:57:35] * James_F sighs. [02:08:46] James_F: A) what kind of change? :) link? [02:08:48] email is good [02:08:57] post 5pm ;) [02:14:05] greg-g: It's a fix for a fatal JS error on save that occurs any time you edited a template [02:14:18] I've been a bit BOLD and already put it on the [[Deployments]] page [02:21:14] !log LocalisationUpdate completed (1.23wmf10) at Wed Jan 15 02:21:14 UTC 2014 [02:21:20] Logged the message, Master [02:28:10] springle: any clue how to repro bug 60035? [02:30:09] (03PS2) 10Mattflaschen: Enable GuidedTour on translated languages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105997 (owner: 10Reza) [02:30:49] (03CR) 10Mattflaschen: "I changed the URL in the commit message, since this is unrelated to pt-br." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105997 (owner: 10Reza) [02:40:10] !log LocalisationUpdate completed (1.23wmf9) at Wed Jan 15 02:40:09 UTC 2014 [02:40:16] Logged the message, Master [03:13:57] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 15 03:13:57 UTC 2014 [03:14:04] Logged the message, Master [03:17:42] AaronSchulz: none. although your question on the IPs was interesting. commented on bug 60035 [03:43:35] (03PS1) 10Springle: warm up db1040 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107542 [03:44:06] (03CR) 10Springle: [C: 032] warm up db1040 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107542 (owner: 10Springle) [03:44:15] (03Merged) 10jenkins-bot: warm up db1040 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107542 (owner: 10Springle) [03:45:09] !log springle synchronized wmf-config/db-eqiad.php 'warm up db1040' [03:45:16] Logged the message, Master [03:52:04] (03PS7) 10Ori.livneh: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/106169 (owner: 10BryanDavis) [03:52:11] (03CR) 10Ori.livneh: [C: 032 V: 032] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/106169 (owner: 10BryanDavis) [04:02:05] (03PS1) 10Ori.livneh: Fix Kibana init.pp file location [operations/puppet] - 10https://gerrit.wikimedia.org/r/107544 [04:02:38] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix Kibana init.pp file location [operations/puppet] - 10https://gerrit.wikimedia.org/r/107544 (owner: 10Ori.livneh) [04:06:54] springle: so will these 'contributions' servers have a very low 'load' value? [04:07:43] that way they wouldn't be picked by wfGetDB() normally (often) unless there was a the right $group...the low weights could still be adjusted relative to each other [04:07:52] correct [04:09:39] also, if it is sharded by user what about newbie contribs? [04:10:39] i don't follow. you mean new user ids? [04:10:42] meh, that had to filesort all results anyway [04:10:52] so I guess that doesn't make much difference probably [04:11:34] it's the "newest 1% of users" query (rev_user >= X ORDER BY rev_timestamp DESC) [04:11:57] ah gotcha [04:12:06] well that should be improved too [04:12:30] most likely is that partitioning by range(rev_user) is best for biggest wikis [04:12:52] so that would hit the last partition [04:15:09] I assumed it was module...is the range partitioning well balanced? [04:15:17] *modulo [04:15:34] though range would be good for that specific query...not as much for the other ones by user [04:17:37] i was testing hash & moduolo earlier [04:18:18] hash is nice and simple allowing good balancing for equality [04:18:53] range is potentially more flexible but needs custom ranges for each wiki depending on the distribution of user_id activity [04:19:59] although i find that mostly it's user ids < 100000 causing slow queries, maybe because of age or becuase it incudes most bots. =0 is also high [04:20:43] same trend for all the large wikis [04:21:05] is hash just the modulo of the hash (for non-integers)? [04:21:12] yes [04:21:17] right [04:21:30] RewriteRule ^/([a-z]{2}|meta)/(.*)$ %{ENV:HTTP_X_Forwarded_Proto}://$1.wikipedia.org/wiki/$2 [R=301,L,NE] [04:21:42] TimStarling: no way to do that with the new redirects scheme, right? [04:24:34] you're referring to a rewrite rule which I moved out of redirects.conf specifically because there was no way to do it in the new redirects scheme [04:27:12] good answer, danke :) [04:27:40] (03PS1) 10Springle: db1040 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107546 [04:29:11] (03CR) 10Springle: [C: 032] db1040 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107546 (owner: 10Springle) [04:29:19] (03Merged) 10jenkins-bot: db1040 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107546 (owner: 10Springle) [04:30:16] !log springle synchronized wmf-config/db-eqiad.php 'db1040 to full steam' [04:30:23] Logged the message, Master [04:38:37] TimStarling: maybe getReaderIndex shouldn't set mReadIndex if $group was set? [04:39:24] that sounds broken [04:40:34] paravoid: /srv/deployment/librenms is owned by you, should be 'sartoris' [04:40:41] aka git-deploy aka trebuchet aka trebuchet-deploy [04:40:54] it should only be set if both $group and $wiki were false [04:41:29] $this->mLoads[$i] > 0 [04:41:44] that condition is probably the reason it doesn't break everything completely [04:44:09] Is git-deploy interactive? [04:44:23] no [04:44:51] We have an interactive deploy script at work. It's pretty nice. [04:47:46] Gloria: interactive in which way? [04:48:29] Ryan_Lane: it muders people that break the deployment [04:48:34] *murders [04:48:47] the CI system should do that, not the deployment system ;) [04:48:54] Ryan_Lane: It gives a prompt asking "where do you want to deploy to?" [04:49:02] And you select S for staging, L for live, Q for quit. [04:49:10] Then you it gives a list of repos and says "which repo do you want to push?" [04:49:18] And you type in the repo name (with tab-completion). [04:49:39] are you likely to be eaten by a grue? [04:49:49] Then it clones the repo, does a git log diff, says "these are the users to be contacted if something goes wrong". [04:49:51] git deploy knows what you're trying to deploy, since it runs from within a repo [04:50:22] ori: Not usually. [04:50:23] I'd like to add stage selection (git deploy canary) [04:50:43] Well, everything gets pushed to staging first. [04:50:45] Which is nice. [04:50:53] Wikimedia would probably benefit from that. [04:50:55] Gloria: you should have it say "You are unlikely to be eaten by a grue", then [04:51:04] Gloria: it actually wouldn't much [04:51:12] That's basically what Beta Labs is. [04:51:18] So I imagine it would. ;-) [04:51:27] that's already automated [04:51:35] but it runs from master [04:51:39] Right. [04:51:59] test.wikipedia.org would be a canary [04:52:12] rather than beta [04:52:15] ori: I got a log added so that I could tail it on IRC. [04:52:37] Ryan_Lane: Yes, I suppose you're right. [04:52:54] I don't deploy, so it makes little difference to me. The process never sounded very enjoyable, though. [04:52:57] Ryan_Lane: https://dpaste.de/zHAw/raw everything ran successfully....nothing got deployed? [04:53:22] 2 minions pending (2 reporting) [04:53:24] pending [04:53:41] the reporting could be clearer [04:53:56] well [04:53:57] # NOTE : Sync ran ok! Everything looks good! Automatically finishing [04:54:00] ah, I should add a troubleshooting section for checking if minions are pinging [04:54:03] # YAY : 'sync' for 'kibana/kibana' completed successfully (now at kibana/kibana-20140115-044735) [04:54:11] ori: that's due to the perl git-deploy's stupidity [04:54:14] i usually take 'YAY' in all caps to be an indication of success [04:54:34] What about 'NOTE' and 'INFO' and 'USER'? [04:54:51] all of that output is crap [04:55:03] NOTE: all of that output is crap * [04:55:14] Or maybe that'd be INFO:. [04:55:26] we need to upgrade salt [04:55:48] there's some bug with grains and subscription [04:55:57] salt -G 'deployment_target:kibana' test.ping <-- not pinging [04:58:02] ori: ok, I restarted salt on the minions, it'll deploy now [04:58:09] adding to the troubleshooting section [04:58:30] YAY [04:58:37] oh, it was already there: https://wikitech.wikimedia.org/wiki/Trebuchet#Initial_fetches_are_failing_.28minions_forever_pending.29 [04:58:49] Wait, trebuchet wasn't a joke? [04:59:03] trebuchet is the name of the system [04:59:24] I thought that was sartoris. :-/ [04:59:28] git deploy is an interface to it. it'll soon be replaced by trebuchet trigger: https://github.com/trebuchet-deploy/trigger/ [04:59:36] sartoris never existed on wikimedia servers [04:59:48] We shall never speak of it again. [05:00:08] it was a rewrite of the perl git deploy in python [05:00:45] it was gpl licensed, but trebuchet is apache2 licensed, so I wrote trigger from scratch [05:03:01] wait, trebuchet trigger? [05:03:10] ryan! [05:03:16] you spent your name quota! [05:03:32] trigger is a part of trebuchet ;) [05:03:37] it's the frontend [05:03:45] trebuchet turtle [05:03:49] It's the recursive potion. [05:03:53] the backend is trebuchet [05:03:56] portion * [05:04:03] Typos, please stop ruining my jokes. [05:04:10] Gloria: :) [05:05:09] ori: heh. I send you a link to that ages ago, I'm surprised you didn't call me out on that then :) [05:05:13] *sent [05:07:11] puppet runs trebuchet trigger as sartoris to push to salt minions [05:07:17] any questions? [05:07:20] :) [05:07:26] I'll rename the user to trebuchet ;) [05:07:42] and also, the name of trigger doesn't matter. users will just know "git deploy " [05:07:47] i love giving you a hard time about this, i don't know why [05:08:00] it's not my fault it's been renamed so many times :( [05:08:14] we should make it a tradition [05:08:16] !blame [05:08:19] rename it every month [05:08:23] hahaha. no thanks. I'm the upstream now [05:08:50] do you know how much of a pain in the ass it is to set everything up? github, pypi, launchpad, etc, etc [05:10:52] is brewster no longer the apt host? [05:11:32] it isn't? [05:11:36] paravoid: who in the world blocked access to brewster from iron? [05:11:42] it's *killing* me [05:12:45] !log adding git-python to the apt repo. it comes from the saltstack ppa. [05:12:52] Logged the message, Master [05:13:00] !log make that python-git [05:13:06] Logged the message, Master [05:18:28] (03PS2) 10Jeremyb: clean up chapters redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106107 [05:18:30] (03PS2) 10Jeremyb: rm trailing slash from destinations where unneeded [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106110 [05:18:32] (03PS2) 10Jeremyb: final (I hope!) fix for protorel redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106109 [05:18:40] (03PS2) 10Jeremyb: move {jobs,careers}.wikimedia.org to redirects.dat [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106108 [05:20:06] (03PS6) 10Ori.livneh: Proxy logstash.wikimedia.org via misc varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [05:21:41] (03CR) 10Ori.livneh: "Faidon, is this OK for now?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [05:24:57] could use some QA on those changes, particularly https://gerrit.wikimedia.org/r/106109 [05:26:01] * jeremyb will poke Jeff tomorrow but maybe i can also get some extra sanity checking [05:26:27] mutante: ^ [05:26:39] and of course TimStarling :-) [05:26:43] * jeremyb sleeps [05:27:34] (/me already nominated Gloria for review :P) [05:30:16] (03PS7) 10BryanDavis: Proxy logstash.wikimedia.org via misc varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 [05:30:38] (03CR) 10Jeremyb: "hello world?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 (owner: 10Jeremyb) [05:43:34] (03CR) 10BryanDavis: Proxy logstash.wikimedia.org via misc varnish cluster (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [07:00:18] gerrit is... very... slow , both git review and the web interface [07:00:52] It's rebelling for having to work after 5pm [07:01:11] ori restarted gerrit a few days ago for it being too damn slow [07:01:13] I think [07:09:56] it's awful, isn't it? [07:10:21] gerrit? [07:10:22] yeah [07:10:27] who would've thunk [07:10:34] the performance regression, i mean [07:10:38] it's semi-unusable [07:10:45] it's as slow as ever for me [07:10:58] I thought it was for everyone until I tried to use it from SF, and boy did that make a difference [07:57:49] morning [08:10:07] morning paravoid [08:14:20] (03PS2) 10Faidon Liambotis: RT 3645 - give iodine a predictable IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 (owner: 10Jeremyb) [08:14:26] (03PS3) 10Faidon Liambotis: Give iodine a predictable IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 (owner: 10Jeremyb) [08:14:35] (03CR) 10Faidon Liambotis: [C: 032] Give iodine a predictable IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 (owner: 10Jeremyb) [08:15:10] looks like beta labs is down [08:16:07] (03PS8) 10Faidon Liambotis: Proxy logstash.wikimedia.org via misc varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [08:16:14] (03CR) 10Faidon Liambotis: [C: 032] Proxy logstash.wikimedia.org via misc varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [08:17:52] (03PS4) 10Faidon Liambotis: Add logstash.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 (owner: 10BryanDavis) [08:18:02] (03CR) 10Faidon Liambotis: [C: 032] Add logstash.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 (owner: 10BryanDavis) [08:25:28] (03CR) 10Physikerwelt: added basic hbase support (033 comments) [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [08:28:44] (03PS1) 10Faidon Liambotis: Revert "Initial commit of pmacct module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107550 [08:32:30] (03PS2) 10Faidon Liambotis: Revert "Initial commit of pmacct module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107550 [08:32:48] (03CR) 10Faidon Liambotis: [C: 032] Revert "Initial commit of pmacct module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107550 (owner: 10Faidon Liambotis) [08:38:40] (03PS5) 10Physikerwelt: added basic hbase support [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 [08:40:39] (03CR) 10Physikerwelt: [C: 04-1] "I did not test the recent changes... I'll come back to this later (i.e. my environment is a little bit different since I'm using vagrant)" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [08:42:08] (03CR) 10Faidon Liambotis: [C: 04-1] "One real easy to fix comment (breakage), plus two very OCD comments." (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [08:45:23] (03CR) 10Physikerwelt: "I'm not sure about the default values." (039 comments) [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [08:46:29] (03PS1) 10Faidon Liambotis: Give Nik & Chad root access to lucene search boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/107551 [08:47:35] (03CR) 10Faidon Liambotis: [C: 032] Give Nik & Chad root access to lucene search boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/107551 (owner: 10Faidon Liambotis) [08:53:50] (03CR) 10Faidon Liambotis: [C: 04-1] "I'm afraid we can do none of that at all. We don't do git clones on production boxes, especially from untrusted sources (GitHub) & we don'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 (owner: 10Addshore) [09:01:54] paravoid: regarding your comments on my patch; the path /srv/ocg/tmp wont have parents; but can I assume (because of trebuchet) that /tmp/deployment/ocg will be there for subfolders (and that iwll be safe?) [09:02:48] !log jenkins: python-git package receiving an "update" 0.3.2~RC1-1~precise1 -> 0.3.2.RC1-1 (same version assuming we repackaged it somehow) [09:02:54] Logged the message, Master [09:05:26] (03CR) 10KartikMistry: "Nitpick in changelog." (031 comment) [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [09:08:05] !log jenkins: restarting Zuul to make sure it uses the new python-git [09:08:12] Logged the message, Master [09:24:12] (03CR) 10Hashar: [C: 04-1] "What Faidon said, plus some other nitpicks :-]" (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 (owner: 10Addshore) [09:24:26] paravoid: and it is not like I told Wikidata team to NOT use github :-D [09:25:25] the good thing is that addshore is friendly and willing to comply [09:26:17] (03PS1) 10Spage: Enable Flow on meta and enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107553 [09:30:47] (03CR) 10Spage: "I think this could go out any time, but we definitely want it for the Thursday update of phase2 wikis to 1.23wmf10." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107553 (owner: 10Spage) [09:42:31] (03PS1) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [09:43:15] (03CR) 10jenkins-bot: [V: 04-1] gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [09:49:02] mwalker: I'm talking about /srv/ocg itself [09:49:56] (03PS2) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [09:57:11] jeremyb: i see requestors there [10:02:24] (03PS1) 10Ori.livneh: Kibana: use a plain block for /status [operations/puppet] - 10https://gerrit.wikimedia.org/r/107556 [10:02:52] (03CR) 10Ori.livneh: [C: 032 V: 032] Kibana: use a plain block for /status [operations/puppet] - 10https://gerrit.wikimedia.org/r/107556 (owner: 10Ori.livneh) [10:03:34] paravoid: i merged your sudo for chad / nik change [10:03:41] oh did I forget to do that [10:03:44] thanks [10:04:01] root@tungsten:~# ls -lah /var/lib/graphite/search_index [10:04:01] -rw------- 1 _graphite _graphite 2.5K Dec 12 20:53 /var/lib/graphite/search_index [10:04:07] tungsten's broken in this sense [10:04:20] I wonder how it works right now [10:04:22] that's not the real location, is it? [10:04:35] i think i have it in /var/lib/carbon [10:04:43] i was just nervous about deleting that one [10:04:50] oh, right [10:05:03] but it still only works by dint of an explicit chmod [10:05:15] so you're right that that needs a fix, i haven't forgotten [10:05:30] yeah, /usr/bin/graphite-build-search-index is locally modified [10:05:31] (03CR) 10Dzahn: download: lint clean (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107341 (owner: 10Matanya) [10:06:57] it was elves [10:07:12] pretty sure it was elves [10:07:24] your logstash change doesn't make any sense [10:07:26] i did revert the other one [10:07:33] (or i'm missing something) [10:08:21] well, tried logstash.wikimedia.org, varnish 503 [10:08:31] well, it's a right change, it just won't fix the effect you're seeing [10:08:44] varnishlog on cp1043 says backend is sick [10:08:51] it 404s [10:08:52] curl /status, get nada [10:09:09] GET /status HTTP/1.1 [10:09:09] Host: logstash.wikimedia.org [10:09:09] HTTP/1.1 404 Not Found [10:09:37] try again [10:09:47] well i haven't done logstash1003 yet [10:09:49] but the other two work [10:09:57] (03PS2) 10Matanya: download: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107341 [10:10:29] (03CR) 10Dzahn: [C: 031] cpufrequtils: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107346 (owner: 10Matanya) [10:12:05] you only tell me when i'm wrong, paravoid :P [10:12:14] hm? [10:12:21] it works now [10:12:32] by dint of my change which doesn't make any sense and won't fix the effect [10:12:39] yes, it works, you're right [10:13:08] i'm not sure why, though [10:13:20] missing RewriteEngine on? [10:13:26] does LocationMatch require it? [10:13:34] no, it shouldn't [10:13:41] no [10:14:37] anyways, i'm sufficiently embarrassed to go clean up the search index file permissions thing for good, bbiaf [10:14:55] ori: you should go to bed [10:15:05] it's harder to type in bed [10:15:46] (03CR) 10Dzahn: "inline comment and it should have a role class where you set "git.wikimedia.org" and then pass it as a variable to be used in init.pp and " (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:16:05] I suspect the regexp is already bound [10:16:08] and ^ $ don't work [10:16:20] not that this makes much sense [10:16:48] mutante: there is a role already [10:19:39] (03CR) 10Matanya: "the role class is in manifests/role/gitblit.pp and has the system role there. should i move to init.pp?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:21:32] (03CR) 10Dzahn: download: lint clean (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107341 (owner: 10Matanya) [10:22:46] paravoid: graphite-build-search-index is pretty annoying [10:23:06] it uses mktemp and then mvs it into place so it's not world-readable [10:23:26] we didn't have a reason to NOT have NRPE on virt0, did we? [10:23:33] and it chowns it to _graphite:_graphite but it makes sense to run carbon as _graphite and graphite-web as www-dev [10:24:31] you don't want graphite-web to be tampering with whisper files ever, so it makes sense to run it as a different user than carbon [10:24:41] (03CR) 10Dzahn: "ah, ok. no the system role should stay in role class, but the apache site and cert name should be set there instead of inside the module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 (owner: 10Matanya) [10:29:19] (03PS1) 10Faidon Liambotis: kibana: redirect to HTTPS [operations/puppet] - 10https://gerrit.wikimedia.org/r/107557 [10:29:29] (03PS3) 10Matanya: download: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107341 [10:29:43] (03CR) 10Dzahn: [C: 031] nrpe: enable on virt0 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107424 (owner: 10Gage) [10:30:50] (03CR) 10Dzahn: [C: 032] "thx for solving the path conflict, and fixes the feed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107376 (owner: 10Nemo bis) [10:31:22] (03CR) 10Dzahn: "what, "needs verified"? it already is..sigh" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107376 (owner: 10Nemo bis) [10:32:55] (03PS3) 10Matanya: gitblit: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107555 [10:35:44] (03CR) 10Dzahn: "< mutante> did we have any reason to NOT have NRPE on virt0? ..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/107424 (owner: 10Gage) [10:36:52] (03CR) 10Dzahn: "need to check firewalling before this (ferm) and/or maybe putting that on the replacement host right away" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107424 (owner: 10Gage) [10:37:28] (03CR) 10Dzahn: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107376 (owner: 10Nemo bis) [10:38:58] hashar: 107376 says it's Verified by jenkins but also "needs verified", what am i missing [10:39:34] touches an .erb [10:40:43] (03CR) 10Faidon Liambotis: [C: 032] kibana: redirect to HTTPS [operations/puppet] - 10https://gerrit.wikimedia.org/r/107557 (owner: 10Faidon Liambotis) [10:44:48] (03PS1) 10Faidon Liambotis: kibana: enable Apache2 rewrite module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107558 [10:45:03] (03CR) 10Faidon Liambotis: [C: 032] kibana: enable Apache2 rewrite module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107558 (owner: 10Faidon Liambotis) [10:45:17] (03CR) 10Faidon Liambotis: [V: 032] kibana: enable Apache2 rewrite module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107558 (owner: 10Faidon Liambotis) [10:45:31] (03CR) 10Dzahn: [C: 031] "heh, a 2-spaces -> 4-spaces lint, +hashar" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107347 (owner: 10Matanya) [10:49:55] (03CR) 10Dzahn: [C: 031] dynamicproxy: lint clean (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107339 (owner: 10Matanya) [10:50:11]