[00:01:36] hrmm, all my tests from various spots hit cp1044 and no loop [00:01:42] lemme see if 1043 is also workin [00:02:19] aude: Do you have full access to the rt ticket or only got the mail either? [00:02:34] I guess I'm going to reply to it to just have the stuff in the ticket itsekf [00:04:04] hoo: reply to the mail [00:19:25] For me "curl 'https://doc.wikimedia.org/VisualEditor/'" works = showing an Apache directory index. [00:23:09] RobH: :P [00:25:49] greg-g: ^_^ [00:25:59] be proud that you make newbs quake in fear! [00:26:48] hah, it's just the beard [00:45:49] (03PS3) 10Dzahn: decom professor, add decommissioning.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/109884 [00:47:36] (03CR) 10Cmjohnson: [C: 032] decom professor, add decommissioning.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/109884 (owner: 10Dzahn) [00:48:14] (03PS2) 10Dzahn: remove professor from DNS (decom) [operations/dns] - 10https://gerrit.wikimedia.org/r/109286 [00:49:23] (03CR) 10Cmjohnson: [C: 032] remove professor from DNS (decom) [operations/dns] - 10https://gerrit.wikimedia.org/r/109286 (owner: 10Dzahn) [01:00:32] (03PS1) 10Cmjohnson: removing pappas from dsh/dhcpd and site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/110281 [01:00:38] (03Abandoned) 10Cmjohnson: decom "pappas" (formerly fr bastion) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110158 (owner: 10Dzahn) [01:01:48] (03CR) 10Cmjohnson: [C: 032] removing pappas from dsh/dhcpd and site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/110281 (owner: 10Cmjohnson) [01:12:49] (03PS1) 10Cmjohnson: Removing dns entries for pappas [operations/dns] - 10https://gerrit.wikimedia.org/r/110286 [01:14:59] (03CR) 10Cmjohnson: [C: 032] Removing dns entries for pappas [operations/dns] - 10https://gerrit.wikimedia.org/r/110286 (owner: 10Cmjohnson) [01:18:21] beta cluster is dead [01:18:22] Exception from line 84 of /data/project/apache/common-local/php-master/extensions/Wikidata/extensions/Wikibase/repo/Wikibase.hooks.php: Wikibase: Incomplete configuration: $wgWBRepoSettings["entityNamespaces"] has to be set to an array mapping content model IDs to namespace IDs. See ExampleSettings.php for details and examples. [01:19:25] aude: ^ [01:19:27] damn [01:19:49] not again [01:25:51] hoo: aude: filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=60606 already [01:26:10] got it [01:26:13] MatmaRex: aude: hoo: https://bugzilla.wikimedia.org/60606 [01:26:17] making a fix [01:26:59] Already have it fixed [01:29:11] (03PS1) 10Reedy: Disable and remove ContactPageFundraiser [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110292 [01:33:25] beta works again [01:33:32] \o/ [02:02:05] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-30 02:02:05+00:00 [02:02:14] Logged the message, Master [02:02:54] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-30 02:02:54+00:00 [02:03:03] Logged the message, Master [02:05:14] MariaDB 5.5.35 [02:05:15] https://blog.mariadb.org/mariadb-5-5-35-now-available/ [02:09:15] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-30 02:09:15+00:00 [02:09:24] Logged the message, Master [03:19:18] (03PS1) 10Springle: repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110307 [03:20:21] (03CR) 10Springle: [C: 032] repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110307 (owner: 10Springle) [03:20:27] (03Merged) 10jenkins-bot: repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110307 (owner: 10Springle) [03:21:17] !log springle synchronized wmf-config/db-eqiad.php 'repool db1042' [03:21:26] Logged the message, Master [03:38:10] (03CR) 10Reedy: [C: 032] Stop making AdminSettings symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110259 (owner: 10Chad) [03:38:19] (03Merged) 10jenkins-bot: Stop making AdminSettings symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110259 (owner: 10Chad) [04:05:21] (03PS1) 10Springle: depool db1020 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110312 [04:08:55] (03CR) 10Springle: [C: 032] depool db1020 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110312 (owner: 10Springle) [04:09:02] (03Merged) 10jenkins-bot: depool db1020 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110312 (owner: 10Springle) [04:09:47] !log springle synchronized wmf-config/db-eqiad.php 'depool db1020 for schema changes' [04:09:54] Logged the message, Master [05:44:58] Well, this is a very basic problem… Ryan_Lane, on virt1000 "keystone endpoint-list" gets me a strange internal python keyerror [05:45:07] as though I need to upgrade some other component or db. But everything looks up to date... [06:49:17] (03PS1) 10Andrew Bogott: Inform keystone about our ldap schema for roles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110318 [06:49:19] (03PS1) 10Andrew Bogott: Include python-novaclient for nova/havana. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110319 [06:51:51] (03CR) 10Andrew Bogott: [C: 032] Inform keystone about our ldap schema for roles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110318 (owner: 10Andrew Bogott) [06:52:04] (03CR) 10Andrew Bogott: [C: 032] Include python-novaclient for nova/havana. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110319 (owner: 10Andrew Bogott) [07:07:32] (03PS1) 10Andrew Bogott: Use 'keystoneclient' instead of 'keystone'. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110320 [07:09:31] mornin [07:10:04] (03CR) 10Andrew Bogott: [C: 032] Use 'keystoneclient' instead of 'keystone'. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110320 (owner: 10Andrew Bogott) [08:07:55] (03PS1) 10Matanya: emery: move rsync api job to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110325 [08:18:29] (03PS1) 10Matanya: emery: remove api logs from emery [operations/puppet] - 10https://gerrit.wikimedia.org/r/110327 [08:20:29] (03PS1) 10Matanya: emery: move glam-nara logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110328 [08:27:22] PROBLEM - Check status of defined EventLogging jobs on vanadium is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/vanadium [08:30:19] that's e [08:31:42] I wish I knew how to fix that :-D [08:32:18] that's "e"? [08:32:30] *me [08:36:22] RECOVERY - Check status of defined EventLogging jobs on vanadium is OK: OK: All defined EventLogging jobs are runnning. [08:49:11] paravoid: mind looking at https://gerrit.wikimedia.org/r/#/c/109869/ please? [08:58:15] paravoid: akosiaris: do we have any easy way to convert ruby gems to debian packages? :D [08:58:22] gem2deb [08:58:56] I am not yet sure which path we will follow: either a git repo containing all our gems dependencies, or packaging them [09:01:00] for? [09:01:21] context is for browsertests [09:01:29] they are based on cucumber / ruby webdriver [09:01:34] and have a bunch of dependencies [09:02:12] we run then on a labs instance and run a command that install the dependencies from rubygems [09:02:26] hashar: packages, way better [09:02:29] which add a 2-3 minutes overhead to the job (have to download from ruby gems repo then install them) [09:02:50] if you wish, can package them for you [09:03:30] matanya: thanks for your offering :] [09:03:34] I find packaging to be painfull [09:03:49] will check with zeljkof and find out which gems we should package [09:06:53] oh hashar would love if you can review https://gerrit.wikimedia.org/r/#/c/108289/ [09:07:00] !gitweb mediawiki/selenium [09:07:00] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/selenium.git [09:07:06] * matanya lacks reviews [09:07:35] matanya: an example of gem dependencies: https://git.wikimedia.org/blob/mediawiki%2Fselenium.git/master/mediawiki-selenium.gemspec :D [09:07:44] mediawiki/selenium being a gem itself [09:08:00] ohhhh [09:08:09] thanks for the beta modularization [09:08:41] hashar: not an issue, i can package from the one with no deps, to the last one [09:08:46] matanya: you probably want to rebase your change already. I added a fatal monitoring script last week [09:08:57] i rebase daily :/ [09:11:15] :( [09:12:38] (03CR) 10Hashar: "sounds fine. You want to rebase this change again since I have added a fatal monitoring script last week :(" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [09:17:59] (03PS4) 10Matanya: beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 [09:20:27] ok, hashar whenever you are ready, you can review the rebase, and give me a list of gems [09:50:44] is cluster tier a specific variable for varnish, or it can be used elsewhere? [09:51:46] * matanya wonders if ori would know [10:31:11] (03PS1) 10Matanya: facilities: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/110339 [11:03:04] akosiaris: can poolcounter be modulrized? or any other plan for it? [11:03:53] a) yes b) no, c) go ahead :-) [11:04:06] * matanya is doing [11:06:14] akosiaris: can I include a cron directly in site.pp? [11:06:39] Nemo_bis: best not to do that. Unless it really really makes sense [11:06:41] surely adding exim::roled to mchenry is excessive https://bugzilla.wikimedia.org/show_bug.cgi?id=57890#c5 [11:07:25] and privateexim::aliases::private is ... private I guess, not in puppet so I don't think the cron can/should be added there [11:07:30] I don't know off hand. I 'd have to have a look [11:09:12] (03PS1) 10Matanya: poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 [11:09:39] Nemo_bis: the answer is no, don't do it [11:10:15] read in a nice manner please [11:10:17] then a new mail.pp class just for that? [11:10:31] came out too harsh, sorry [11:11:12] no Nemo_bis this : https://gerrit.wikimedia.org/r/#/c/68584/ will solve it once it is done [11:11:53] sounds like it will take ages [11:12:07] we need stats nowish [11:12:14] poke andrewbogott [11:12:46] he might know. i might take it over, if he can't but it looks nice, and i hope he can finish it [11:16:38] sigh, now I must go [11:16:44] too bad, I hoped to submit a patch [11:16:45] matanya: regarding poolcounter? [11:17:05] no, andrewbogott regarding exim module [11:17:18] poolcounter is already pushed [11:17:27] oh -- Yeah, I wrote a refactor once but I think mark didn't like it so… might be best to collaborate with him [11:17:33] not that he has a spare minute this week :( [11:17:48] ken related? [11:18:21] well [11:18:24] Ah, just, he's doing many things at once. [11:18:25] Faidon has agreed to take on the mail migration [11:18:32] so he'll probably be the point person for this new exim module [11:18:38] correct [11:18:49] Ah, ok then! [11:18:55] will we have DKIM everywhere? :) [11:19:05] yes [11:19:16] matanya: I don't know if there's much to salvage from my patch, probably it should be abandonded. [11:19:45] if you do that, i'll start a new one. too bad for my huge rebase on it :/ [11:20:17] and another question i had : is cluster tier a specific variable for varnish, or it can be used elsewhere? [11:20:29] i think paravoid or mark will know to answer this [11:20:48] grep the source :) [11:20:51] i did [11:20:57] where do you see it specified? [11:21:05] cluster_tier is role/cache.pp specific indeed [11:21:05] it is called cache.pp [11:21:22] if it's really useful elsewhere we could think about making it a global [11:21:53] it's hard to imagine "tiers" for other services I think [11:22:35] all around modules/protoproxy/templates/proxy.erb [11:22:52] 12 times. [11:23:25] tier wouldn't help you much there, since you need to reference the backend IPs anyway [11:23:26] so either i can replace it with a variable from the class it is called, or with some higher scope var [11:23:37] but in any case, the protoproxy stuff is eventually going away [11:23:51] this part anyway [11:23:53] egde case anyway. would be replaced by what? [11:23:58] with the localsslw ork [11:24:14] local ssl? where is that? [11:24:18] same module [11:24:22] but different template [11:24:25] oh [11:24:31] (and manifest) [11:24:43] so i'll ignore this puppet3 issue [11:24:56] no, let's not please tie these two migrations together [11:25:10] the syntax fixes are easy enough to do them now [11:25:34] hmm, it is, but it involves design issues [11:25:57] i.e. what we spoke some days ago [11:26:04] remind me, I forgot [11:26:25] openstack version iirc [11:27:25] andrewbogott: solved it at the end, but it wasn't clear at first [11:30:17] out for 15 min [11:37:51] (03CR) 10Alexandros Kosiaris: [C: 04-1] poolcounter: convert into a module (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [12:07:05] (03PS2) 10Matanya: poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 [12:07:42] (03CR) 10jenkins-bot: [V: 04-1] poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [12:08:44] (03PS3) 10Matanya: poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 [12:16:10] hashar: i would love to understand why manifests/zuul.pp gules modules/zuul and manifests/roles/zuul.pp [12:20:54] matanya: ahhh [12:21:05] matanya: so Zuul module is supposed to be independent / wikimedia agnostic [12:21:18] matanya: Zuul role just set system_role and basic stuff [12:21:31] matanya: manifests/zuul.pp carry wikimedia specific configuration [12:21:45] so why not put that into the role? [12:23:07] matanya: done that on my birthday 2 years [12:23:08] ago moved files definitions out of role::zuul to a new zuulwikimedia manifests in manifests/zuul.pp [12:23:09] zuulwikimedia calls the module class and then deploy Wikimedia specific configuration. [12:23:12] on https://gerrit.wikimedia.org/r/#/c/27611/ [12:24:48] and where is it now? [12:25:00] still the same layout [12:25:18] feel free to drop zuulwikimedia and put its content in the role class [12:25:24] might want to check with faidon first though [12:25:29] planning to change it yourself? [12:25:36] nop [12:25:41] if it works, don't break it! :-] [12:25:50] i'd love some input from paravoid [12:26:03] puppet-glue isn't very friendly :) [12:28:52] lunch needed [13:03:14] (03CR) 10Alexandros Kosiaris: [C: 04-1] poolcounter: convert into a module (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [13:07:25] (03PS4) 10Matanya: poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 [13:31:35] (03PS1) 10Andrew Bogott: Specify compute_driver as that seems required in havana. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110365 [13:34:35] (03CR) 10Andrew Bogott: [C: 032] Specify compute_driver as that seems required in havana. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110365 (owner: 10Andrew Bogott) [13:38:02] PROBLEM - DPKG on virt1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:39:02] RECOVERY - DPKG on virt1001 is OK: All packages OK [13:48:32] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:22] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 178565 bytes in 6.376 second response time [13:59:21] (03PS1) 10Matanya: certs: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/110366 [14:11:26] (03PS5) 10Alexandros Kosiaris: poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [14:13:15] matanya: ^^ that is what i meant [14:13:34] yeah, i see. thanks. sorry for extra work [14:15:36] i'll remove those too next time :) [14:18:35] (03CR) 10Alexandros Kosiaris: [C: 032] poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [14:18:54] (03CR) 10Alexandros Kosiaris: [C: 032] puppet-merge: Fix bug introduced in 4a79aa1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/110203 (owner: 10Alexandros Kosiaris) [14:19:25] (03CR) 10Alexandros Kosiaris: [V: 032] poolcounter: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/110340 (owner: 10Matanya) [14:22:15] (03PS12) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [14:22:25] (03CR) 10jenkins-bot: [V: 04-1] site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 (owner: 10Matanya) [14:26:19] (03CR) 10Alexandros Kosiaris: "Both approaches failed to take merges into account. In that case another commiter, namely gerrit, is also present in the git log output. W" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110203 (owner: 10Alexandros Kosiaris) [14:27:50] (03PS13) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [14:29:47] who will be at fosdem this weekend? [14:30:52] lots of people [14:30:58] (03CR) 10Guido.iaquinti: [C: 031] add FIXMEs for erzurumi references [operations/puppet] - 10https://gerrit.wikimedia.org/r/109655 (owner: 10Dzahn) [14:31:07] what is it that you're interested in specifically? [14:31:48] * AaronSchulz likes how dark it is outside [14:31:57] if someone of you will be at fosdem, just to meet and have a beer :) [14:32:10] i already know that a lots of people will be there :D [14:32:34] we'll have a stand this year [14:32:47] plus Erik is giving a talk about the "Wikipedia stack" [14:33:32] paravoid: i know, but who [14:33:38] I think the best bet would be to hang out at our stand, I'm sure lots of us will be going back and forth there :) [14:33:51] ok [14:34:01] i'm sure paravoid doesn't have a name list [14:34:05] I actually do [14:34:06] https://www.mediawiki.org/wiki/Events/FOSDEM/2014 [14:34:07] :) [14:34:17] always suprising paravoid :) [14:37:12] PROBLEM - Host virt1001 is DOWN: PING CRITICAL - Packet loss = 100% [14:37:34] oh? [14:38:10] volunteer to kick the machine? [14:39:19] that's me [14:39:40] andrewbogott: around? want to draft an email about openstack support and need to know what exactly we are looking for [14:41:00] drdee: I'll be awake for a bit. It's hard to know exactly what we'll need, other than 'troubleshooting help' [14:41:16] But I can describe the setup [14:42:22] RECOVERY - Host virt1001 is UP: PING OK - Packet loss = 0%, RTA = 0.65 ms [14:44:22] PROBLEM - RAID on virt1001 is CRITICAL: Connection refused by host [14:44:30] that's still me! [14:44:42] PROBLEM - Disk space on virt1001 is CRITICAL: Connection refused by host [14:45:02] PROBLEM - SSH on virt1001 is CRITICAL: Connection refused [14:45:02] PROBLEM - puppet disabled on virt1001 is CRITICAL: Connection refused by host [14:45:02] PROBLEM - DPKG on virt1001 is CRITICAL: Connection refused by host [14:52:15] mutante: mind if I add your instructions to create jgonera to the wikitech RT page? [14:53:40] (03PS1) 10Aude: add data license settings for wikibase [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110368 [14:53:42] (03PS1) 10Aude: enable wikidata build for test.wikidata and test2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110369 [14:54:13] (03CR) 10Ottomata: "Change the rsync job to point at erbium now too, ja?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110327 (owner: 10Matanya) [14:54:49] (03PS2) 10Matanya: emery: move glam-nara logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110328 [14:54:56] (03CR) 10Ottomata: [C: 032 V: 032] emery: move glam-nara logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110328 (owner: 10Matanya) [14:56:12] (03CR) 10Matanya: "done in https://gerrit.wikimedia.org/r/#/c/110325/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110327 (owner: 10Matanya) [14:56:38] oh matanya, didn't see that [14:56:39] cool [14:56:42] PROBLEM - NTP on virt1001 is CRITICAL: NTP CRITICAL: No response from NTP server [14:56:58] (03PS2) 10Matanya: emery: move rsync api job to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110325 [14:57:00] ottomata: told you i'm spiltting it up for easier review :) [14:57:03] (03CR) 10Ottomata: [C: 032 V: 032] emery: move rsync api job to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110325 (owner: 10Matanya) [14:59:26] Heja ops. Who would I ping for mailing list issues? See https://bugzilla.wikimedia.org/show_bug.cgi?id=60215 [15:00:11] * matanya suggestes rt ticket [15:00:35] (03PS2) 10Aude: enable wikidata build for test.wikidata and test2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110369 [15:03:21] (03PS2) 10Ottomata: emery: remove api logs from emery [operations/puppet] - 10https://gerrit.wikimedia.org/r/110327 (owner: 10Matanya) [15:03:36] (03CR) 10Ottomata: [C: 032 V: 032] emery: remove api logs from emery [operations/puppet] - 10https://gerrit.wikimedia.org/r/110327 (owner: 10Matanya) [15:04:42] ottomata: the left services on emery should also move to erbium, correct? [15:09:17] (03PS1) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:10:09] (03PS2) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:10:52] if i want to setup new cron jobs (but not enable yet), can i list them in puppet as "absent" [15:11:06] then change to "enabled" later [15:11:14] does that work? [15:11:14] no aude that won't do that [15:11:18] aww [15:11:22] then how? [15:11:24] absent removes them [15:11:28] yes [15:11:40] i could comment them out [15:11:46] you should use comment out [15:11:49] ok [15:12:28] then once need to use, remove the remark, and puppet will install them [15:13:27] matanya, yeah, to erbium should be good [15:13:33] i was just checking out load and stuff, i think we can handle it [15:13:41] one patch, or a few? [15:13:59] one is fine for the rest, but, for now, just leave the sampled-1000 filter on emery and add it to erbium [15:14:12] we'll just turn off the sampled-1000 filter when we decom emery [15:14:17] its kinda nice to have it as a little backup [15:14:25] (03PS3) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:14:44] it will remain on gadolinium only ottomata ? [15:14:44] like that ^ ? [15:14:55] PROBLEM - Host virt1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:03] ? [15:15:05] we want to stop the old jobs when we switch wikidata from wmf11 to wmf12 [15:15:13] go ahead and add it to erbium config now [15:15:14] then after switched, enable new cron jobs [15:15:22] but don't remove it from emery config yet [15:15:36] i'm going to go in and be really careful about those logs when the rsync job changes [15:15:45] to either make sure there is no overlap in the archived logs [15:15:52] or at least note that there is [15:15:59] ok [15:16:22] aude: yes, but it hurts my eyes (where is lint?) [15:16:32] what's wrong? [15:16:49] i see a tab [15:16:50] you want a proper review? [15:16:56] yes [15:17:01] ok [15:18:05] RECOVERY - SSH on virt1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [15:18:15] RECOVERY - Host virt1001 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [15:20:43] (03CR) 10Matanya: update wikibase cronjobs to use wikidata build [DNM yet] (0310 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 (owner: 10Aude) [15:21:15] thanks :) [15:21:18] aude: are you aware of the style guide? [15:21:20] (03CR) 10Jgreen: "All these references to erzurumi are long-deprecated--no need to replace them. The box has been simply a standby server for the past year." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109655 (owner: 10Dzahn) [15:21:26] no [15:21:36] * aude just copying code [15:21:47] sure it's poorly styled though [15:21:47] aude: docs.puppetlabs.com/guides/style_guide.html [15:21:53] thanks [15:21:58] please follow :) [15:22:11] i feel like the lint-nazi :P [15:23:10] aude: two execptions from the guide: [15:23:32] 1) we use 4 spaces instead of 2 in this guide [15:23:53] 2) ignore Inheritance at all cost :) [15:24:42] aude: and to nugde a bit more, are you aware of the puppet conventions on wikitech? [15:27:27] not sure i understand "$enabled ?{ " [15:27:52] can i just put 'present' or 'absent' ? [15:28:21] but we have "class misc::maintenance::wikidata( $enabled = false ) { " [15:28:23] http://puppet-lint.com/checks/selector_inside_resource/ [15:28:29] ok [15:29:07] aude: cron ensure => can get present or absent only [15:30:11] http://docs.puppetlabs.com/references/latest/type.html#cron [15:30:29] (03CR) 10Ori.livneh: [C: 04-1] "What's up with the removal of monitor_ganglia? It's not mentioned in the commit message, and it's used in several modules." [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 (owner: 10Matanya) [15:30:57] (03PS4) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:31:41] what do you mean by "to be on the safe side remark those lines too."? [15:31:44] add more comments? [15:32:55] PROBLEM - Varnish HTCP daemon on cp1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:32:55] PROBLEM - Varnish HTTP text-backend on cp1053 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:15] PROBLEM - Varnish traffic logger on cp1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:33:44] yes aude [15:34:00] (03PS3) 10Matanya: nagios: puppet 3 compatibility fix: fully qualify variables [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 [15:34:06] (03PS5) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:34:40] ok [15:35:03] (03CR) 10Matanya: "I some how did something wrong here. Not sure what. Thanks for pointing it out ori." [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 (owner: 10Matanya) [15:35:12] (03CR) 10Ori.livneh: [C: 04-1] "Rather than provision each file in /var/lib/gerrit2 as a separate resource, could you simply reproduce the hierarchy in a modules/gerrit/f" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109088 (owner: 10Matanya) [15:35:28] cmjohnson1: If you have a DC visit planned today, go ahead and drop the 10G nic into labnet1001. I'll be sleeping during most of the DC day anyway so won't notice the downtime :) [15:36:23] matanya: you added trailing whitespace on line 366 [15:36:33] of nagios.pp i mean [15:36:35] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:08] (03PS6) 10Aude: update wikibase cronjobs to use wikidata build [DNM yet] [operations/puppet] - 10https://gerrit.wikimedia.org/r/110371 [15:37:45] RECOVERY - Disk space on virt1001 is OK: DISK OK [15:37:55] RECOVERY - puppet disabled on virt1001 is OK: OK [15:38:05] RECOVERY - DPKG on virt1001 is OK: All packages OK [15:38:25] RECOVERY - RAID on virt1001 is OK: OK: Active: 16, Working: 16, Failed: 0, Spare: 0 [15:38:35] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 180599 bytes in 9.620 second response time [15:39:33] (03PS4) 10Matanya: nagios: puppet 3 compatibility fix: fully qualify variables [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 [15:39:35] RECOVERY - NTP on virt1001 is OK: NTP OK: Offset -0.01575934887 secs [15:42:32] paravoid, while Snaps is here, got a sec to opine about yajl1 vs yajl2 packages for kafkatee? [15:43:19] (03CR) 10Matanya: "There is no module yet. i fail to understand what you are referring to." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109088 (owner: 10Matanya) [15:45:49] (03CR) 10Ori.livneh: [C: 032] nagios: puppet 3 compatibility fix: fully qualify variables [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 (owner: 10Matanya) [15:48:35] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:25] (03PS1) 10Matanya: emery: RT #6143 move two logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110382 [15:51:05] RECOVERY - Varnish traffic logger on cp1053 is OK: PROCS OK: 2 processes with command name varnishncsa [15:51:05] PROBLEM - DPKG on virt1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [15:51:35] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 180469 bytes in 9.643 second response time [15:51:45] RECOVERY - Varnish HTCP daemon on cp1053 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [15:51:45] RECOVERY - Varnish HTTP text-backend on cp1053 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.001 second response time [15:52:05] RECOVERY - DPKG on virt1001 is OK: All packages OK [15:53:37] (03PS1) 10Aude: bump $wgCacheEpoch for wikidatawiki and test wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110384 [15:53:44] (03PS2) 10Matanya: gerrit: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109088 [15:56:29] (03CR) 10Mark Bergsma: [C: 031] Allow PUT method to hosts behind the misc Varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109330 (owner: 10BryanDavis) [15:56:38] ori: go ahead [15:56:41] needs a rebase though, looks like [15:57:18] mark: awesome, thanks. i'll flag it if the rebase is in any way problematic. [15:57:39] (03PS4) 10BryanDavis: Allow PUT method to hosts behind the misc Varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109330 [16:00:03] (03CR) 10Ori.livneh: [C: 032] "per mark's ok" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109330 (owner: 10BryanDavis) [16:16:49] ori, mark: Thanks! One step closer to logstash world domination. [16:17:09] bd808: [16:17:15] so verified works :) [16:17:56] ori: is logstash planned to replace udp2log or to be on top of it? [16:18:06] andrewbogott: sounds good I will take care of it..thx [16:18:23] cmjohnson1: cool, thanks! [16:19:08] matanya: Good question. For the foreseeable I think it's an addition rather than a replacement [16:19:12] matanya: replace. though the intent is to preserve file-based logging as well for grepping [16:20:00] * bd808 sees that ori has more complete plans :) [16:20:05] thank you both [16:20:21] <^d> bd808: This plan is incomplete. No plan is complete without an evil-plans.txt [16:20:36] <^d> :) [16:20:50] bd808: i got the redirect loop again btw a couple of minutes ago [16:20:59] git add evil-plans.txt [16:21:10] matanya: logstash is kinda different than udp2log [16:21:15] (03PS1) 10Manybubbles: Update Elasticsearch monitoring for 0.90.10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/110389 [16:21:20] yeah, i know ottomata [16:21:24] logstash index logs for fancy viewing and stats [16:21:30] udp2log is a transport mechanism [16:21:34] kafka is more the replacement for udp2log :p [16:21:46] have log stash at my $day_job [16:21:55] aye cool [16:22:09] ori: Yeah. It's a varnish issue caused by the way that we send the http -> https redirect. Not sure the best way to fix it yet. [16:22:18] build it from scratch (quite proud of it in fact) [16:22:26] :) [16:22:36] bd808: what's the issue, and why doesn't it affect other SSL services in misc-eqiad? [16:23:01] ori: https://bugzilla.wikimedia.org/show_bug.cgi?id=60488 [16:23:26] Varnish is caching the 301 redirect without respecting X-Forwarded-Proto [16:25:01] So when / is requested via http and the cache is empty the 301 to https is cached and served to both http and http requests until an Auth header is added that breaks the cache [16:26:23] I'm not sure that a similar problem wouldn't effect other vhosts behind the misc cluster if they had the same Apache behavior of redirecting to https as canonical [16:28:01] Adding X-Forwarded-Proto to the vcl_hash should fix it. I'm not sure if it could also be fixed by adding X-Forwarded-Proto to a Vary header. My Varnish fu is mostly search/grep based at this point. [16:28:55] Maybe mark and/or paravoid could read https://bugzilla.wikimedia.org/show_bug.cgi?id=60488 and tell us the right way to fix this cache issue [16:29:28] Vary: X-Forwarded-Proto [16:29:38] when emitting the 301 [16:29:57] hm, but the 301 is being generated from apache, right? [16:30:02] bd808: https://gerrit.wikimedia.org/r/#/c/23521/ [16:30:08] so apache can't do that [16:30:25]