[00:11:14] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:11] jdlrobson: Max mentioned that you had reservations about implementing .nomobile in CSS [00:19:31] can you elaborate on that (if true) [00:19:52] kaldari: not really [00:20:06] cool [00:32:35] !log mwalker synchronized php-1.22wmf19/extensions/CentralNotice/ 're pushing centralnotice to master on the CORRECT branch... grumble' [00:32:50] Logged the message, Master [00:34:04] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 00:33:54 UTC 2013 [00:34:14] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:56] (03PS8) 10Physikerwelt: Intial version of puppet script for LaTeXML [operations/puppet] - 10https://gerrit.wikimedia.org/r/61767 [01:13:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:35:12] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 01:35:09 UTC 2013 [01:35:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:11] * AaronSchulz gets ceph working locally again :) [01:42:48] * AaronSchulz uses xfs this time [02:04:42] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 02:04:32 UTC 2013 [02:04:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:45] !log LocalisationUpdate completed (1.22wmf19) at Tue Oct 1 02:13:45 UTC 2013 [02:13:57] Logged the message, Master [02:25:59] !log LocalisationUpdate completed (1.22wmf18) at Tue Oct 1 02:25:59 UTC 2013 [02:26:11] Logged the message, Master [02:33:52] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 02:33:48 UTC 2013 [02:34:52] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:50:55] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 1 02:50:54 UTC 2013 [02:51:10] Logged the message, Master [03:06:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [03:34:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 03:34:48 UTC 2013 [03:35:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [03:41:05] PROBLEM - Puppet freshness on vanadium is CRITICAL: No successful Puppet run in the last 10 hours [03:52:15] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Tue Oct 1 03:52:11 UTC 2013 [04:04:05] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 04:04:00 UTC 2013 [04:04:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:35:03] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 04:34:59 UTC 2013 [04:35:23] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:44:55] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [04:44:55] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:55] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:55] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:50] Elsie: can we change name? *Bot will be surely blocked somewhere [04:56:20] It requires a configuration change. [04:56:29] Yes, so it's easy. :) [04:56:29] I think legoktm wanted to use EdwardsBot eventually. [04:56:36] Where is *Bot blocked? [04:56:41] Surely a stupid wiki we can ignore. [04:56:52] On any stupid bureacratic wiki [04:56:55] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:55] PROBLEM - Puppet freshness on ms-be1004 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:14] I did [04:57:17] Fuck 'em. :-) [04:57:23] Yes, but the sysops' stupidity is generally not the message recipents' fault. [04:57:41] I'm not sure we need to accommodate stupidity. [04:57:48] Couldn't they equally just block the new name? [04:57:55] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: No successful Puppet run in the last 10 hours [04:58:18] It's going to be pretty difficult to come up with a name that makes sense in 280+ languages, and follows every single username policy [04:58:21] It's harder to turn on the block mental automatism [04:58:36] EdwardsBot is probably the best since they're already used to it [04:58:46] Don't have to respect username policy [04:58:55] I won't help paint this bikeshed. :-) [04:59:09] Well, that would be a legitimate reason to use a Bot username [04:59:27] It's not a bikeshed, it's survival semplification trick [04:59:57] is there a good reason to not just keep using EdwardsBot? [05:00:21] It pings me. [05:00:22] https://meta.wikimedia.org/wiki/Special:CentralAuth/EdwardsBot <-- only one block [05:00:24] :D [05:00:55] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:55] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:24] Nemo_bis: Any edits to https://meta.wikimedia.org/wiki/Global_message_delivery/Spam ? [05:03:55] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:25] Elsie: "much better replacement" sounds redundant [05:05:40] also, all this conversation should be in -tech [05:05:55] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:39] ah, I see the extension will also fulfil Billinghurts' request [05:10:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:34:05] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 05:33:56 UTC 2013 [05:34:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:39:14] (03PS1) 10Matanya: Removed pstack package since bug 48025 is resolved. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86812 [06:14:22] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:59] apergos: what ubuntu version in on fenari? [06:27:27] *is [06:28:18] might be precise [06:28:28] can you please check? [06:28:44] precise [06:28:51] what's up? [06:28:55] thanks! [06:29:11] i'm playing with some pupet, needed to know the version :) [06:34:42] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 06:34:32 UTC 2013 [06:35:22] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:58] ok, well fenari is on its way out sometime [06:36:07] so I wouldn't rely on that for whatever needs [06:40:16] apergos: what is the deploy host? [06:40:24] tin [07:08:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:19:06] (03PS1) 10Matanya: Remove lucid umask setting [operations/puppet] - 10https://gerrit.wikimedia.org/r/86816 [07:33:52] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 07:33:42 UTC 2013 [07:34:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:40:33] PROBLEM - search indices - check lucene status page on search1022 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 56441 bytes in 0.018 second response time [07:54:11] (03PS1) 10Matanya: Fundrising: removed non-WMF jenkins. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86818 [08:01:25] hashar: are all jenkins gerrit jobs jobs in /srv/ssd/gerrit on jenkins-gallium ? [08:06:28] matanya: that the git repositories [08:06:43] matanya: Gerrit send all repositories modifications to the Jenkins hosts under /srv/ssd/gerrit [08:06:46] as bare repositories [08:07:17] the Jenkins workspaces are in /srv/ssd/jenkins-slave/workspace [08:07:25] thanks, I looked in manifests/role/gerrit.pp [08:07:56] hashar: it seems to have a FIXME that can be applied, wanted to verify i get it correct. Do I ? [08:07:58] (03CR) 10Dzahn: [C: 031] "works for me on analytics machines:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/80055 (owner: 10ArielGlenn) [08:08:43] and BTW hashar I added you as a reviewer on other patch, please review it when you have time [08:10:04] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:28] matanya: let me look at gerrit.pp [08:11:56] matanya: yeah I still have some jobs depending on repositories being in /var/lib/git/ [08:13:49] ok, thanks hashar I won't commit this yet [08:13:52] *push [08:14:17] (03CR) 10Hashar: [C: 031] "Note that this does not actually remove the package. It merely move it out of puppet scope ;-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86812 (owner: 10Matanya) [08:15:04] hashar: i may be change to absent if you prefer [08:15:11] na it is ok [08:15:15] I can remove it manually [08:15:39] mutante: mind merging a cleanup change please? https://gerrit.wikimedia.org/r/#/c/86812/ rm a package I have added temporarily on gallium. Will clean up manually. [08:15:43] ok, cool. thanks [08:15:53] mutante: also Guten Tag :-] [08:18:10] (03CR) 10Dzahn: [C: 032] "sure, per comment it was just added for debugging" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86812 (owner: 10Matanya) [08:18:14] \O/ [08:18:46] mutante: danke! [08:18:52] hashar: c'est fait [08:25:24] RECOVERY - Disk space on ms1004 is OK: DISK OK [08:25:37] :) thx apergos [08:26:17] thanks hashar and mutante [08:27:25] mutante: do we have any hardy machines around? [08:27:27] purged on gallium [08:28:25] !log jenkins updating pyflakes jobs to run 'pyflakes .' instead of simply 'pyflakes' {{gerrit|86727}} [08:28:40] matanya: i think just a single one [08:28:41] now [08:28:44] Logged the message, Master [08:29:00] which one mutante ? [08:30:55] matanya: mchenry [08:31:21] the mail relay mutante ? [08:31:28] !log Jenkins: so pyflakes are now actually doing something useful instead of testing nothing. That causes a bunch of jobs to fail [08:31:36] matanya: yes [08:31:41] Logged the message, Master [08:33:44] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 08:33:42 UTC 2013 [08:34:04] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:38:31] (03CR) 10Hashar: "I have manually purged `pstack` on gallium." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86812 (owner: 10Matanya) [08:42:57] (03CR) 10Hashar: "Katie, Adam, Matthew : I suddenly discover you have a Jenkins installation :-) The installation I take care of (for Continuous Integra" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86818 (owner: 10Matanya) [08:45:20] hashar: ++ for not having a separate jenkins install from repo [08:45:32] mutante: yeah that part needs to die hard [08:45:49] mutante: I have added FR folks to the change, hopefully they will notice the Gerrit spam mail :-] [08:46:11] hashar: do you want to hire me? :P [08:46:13] (03CR) 10Dzahn: [C: 031] "++ for not having a separate install from repo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86818 (owner: 10Matanya) [08:49:54] so [08:50:01] lets get some work done instead of reviewing [08:56:59] (03PS1) 10Faidon Liambotis: bits: remove /w/ symlink, add a /w/404.php symlink [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 [08:57:08] (03CR) 10jenkins-bot: [V: 04-1] bits: remove /w/ symlink, add a /w/404.php symlink [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 (owner: 10Faidon Liambotis) [08:57:32] huh [08:57:36] absolute symlinks [08:57:37] fun [08:58:09] Could not open input file: docroot/bits/w/404.php [08:58:18] heh, yea, i was about to say, it's a thing when adding new symlinks [08:58:40] (03CR) 10Faidon Liambotis: [V: 031] "Ignore the Jenkins error, it fails because of the absolute symlink to /apache." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 (owner: 10Faidon Liambotis) [08:58:55] hashar: [09:07:12] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:08] !log killing nrpe on cp1045 and restarting, was running as icinga instead of nagios, fixes "NRPE: Unable to read output" Icinga Unknown's [09:11:22] Logged the message, Master [09:11:51] huh, weird [09:12:17] the gooood old bug hehe [09:15:42] ..but why the same on mw1173 but restarting doesnt fix it [09:18:08] PIDDIR=/var/run/icinga [09:18:30] and /var/run/nagios on others [09:23:20] (03PS1) 10Adamw: minor config file sync [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/86822 [09:23:27] !log mw1173 had wrong nrpe init script, synced the one from mw1172 and restarted (as nagios), fixes dpkg check [09:23:30] wth [09:23:38] Logged the message, Master [09:25:24] (03PS1) 10Adamw: gitignore things [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/86823 [09:25:52] mark or paravoid, Dan just contacted me because he found some urgent bug in zero, would it be ok if i push https://gerrit.wikimedia.org/r/#/c/86821/ to prod now? [09:26:07] (03PS2) 10Adamw: gitignore things [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/86823 [09:26:10] or if greg-g is around...? :) [09:26:23] what kind of bug? [09:27:01] when users navigate to zero.wikipedia.org without having X-CS set, they should NOT be redirected to www.wikipedia.org, but should be shown red banner instead [09:28:51] paravoid ^ [09:29:37] hi yurik [09:29:43] hey mark [09:29:45] what's your plan on properly solving that DfltLanguage issue? [09:30:04] re: the bug, looks very MW-related so I don't personally have an opinion [09:30:11] but I'm sure greg-g would like to see a BZ # [09:30:49] paravoid, i got a hangout msg from dan. Should i tell him you said he should file a bug first? :) [09:30:54] (03PS1) 10Adamw: cheap hack to qualify table with prefix [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/86824 [09:31:02] yes you should [09:31:18] this isn't (just?) us you need to coordinate with [09:31:33] it is [09:31:47] pretty straightforward really - just a minor logic revert [09:32:21] hm, dan is no longer online, i will put a bug in [09:32:49] I think greg-g would like to track emergency deploys, but you should talk with him and I shouldn't make guesses [09:33:09] as far as the change goes, I don't care about the php logic there, do as you will [09:33:56] i hear you, will follow proper procedure and ask dan to do it. For this case - I will create bz bug, link to it, and deploy [09:34:02] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 09:33:59 UTC 2013 [09:34:06] thanks [09:34:09] so, on the dfltlang stuff [09:34:12] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:15] what's your plan with that? [09:34:20] sec, let me push it out, will spell it out in a sec [09:42:46] mark, before i push it out - do you know that there are ~600 fatal errors in fatalmonitor? [09:43:04] i don't [09:43:10] 552 PHP Fatal error: Invalid host name (docroot=/usr/local/apache/common/docroot/bits/), can't determine language.#012 in /usr/local/apache/common-local/multiversion/MWMultiVersion.php on line 351 [09:43:32] don't know how critical this is [09:43:35] just letting you know [09:43:40] it's not critical to me :) [09:43:42] yurik: https://gerrit.wikimedia.org/r/#/c/86819/1 [09:44:05] mutante, thx [09:44:06] it's a known issue from yesterday [09:44:16] does anyone sleep around here? [09:44:18] basically, something is hitting MW on bits [09:44:19] just curious [09:44:42] bits.wikimedia.org [09:44:46] grrr [09:44:55] it's BZ #54805, I pushed a fix to gerrit [09:45:22] http://bits.wikimedia.org/w/index.php [09:45:33] * MaxSem bites Reedy [09:46:25] ok, syncing. [09:46:30] * YuviPanda|away has started using PhpStorm, with Vim keybindings [09:46:42] YuviPanda|away, enjoy proper debugging :) [09:46:42] looks like a nice medium, except for the entire 'too slow to run on my old Air' problem [09:46:54] yurik: AutoComplete is nice, yes. [09:47:06] yurik: I'm on the trial version tho [09:47:10] YuviPanda, omg macs are imperfect? liar! [09:47:16] MaxSem: :P [09:47:27] YuviPanda, soon you will start fixing yellow warnings (they are kinda important, but ppl are lazy) [09:47:28] MaxSem: still trying to find what non-apple laptop to buy. no dice yet :( [09:47:31] also, marktraceur will kill you for betrayal of vim [09:47:45] MaxSem: I switched to Emacs like, a month ago [09:47:49] MaxSem: Emacs with Vim keybindings :P [09:48:03] !log yurik synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/ [09:48:15] Logged the message, Master [09:49:13] (03CR) 10MaxSem: [C: 032] "Let's see if it will break all the consecutive tests:)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 (owner: 10Faidon Liambotis) [09:49:19] (03CR) 10jenkins-bot: [V: 04-1] bits: remove /w/ symlink, add a /w/404.php symlink [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 (owner: 10Faidon Liambotis) [09:49:52] (03CR) 10MaxSem: [V: 032] bits: remove /w/ symlink, add a /w/404.php symlink [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86819 (owner: 10Faidon Liambotis) [09:50:05] heh [09:50:08] max and his big hammer [09:50:16] !log yurik synchronized php-1.22wmf19/extensions/ZeroRatedMobileAccess/ [09:50:31] Logged the message, Master [09:50:32] * MaxSem hammers paravoid [09:51:31] mark, overall idea for dflt lang is to have some (preferably index.php) entry point that will redirect to the proper language wiki based on whatever carrier has defined as default, OR show red banner if its ZERO and no carrier, OR redirect to www.wikipedia.org if its m.wikipedia.org and no carrier [09:52:48] (03PS1) 10MaxSem: Quick test to see if tests still work [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86827 [09:52:58] now since i don't exactly know how this is done ( MaxSem showed me some magical script that uses meta page as raw HTML for the main page, but i will have to dig through it), I don't know how soon it will be implemented [09:53:02] paravoid, ^ [09:53:39] take that script apart and grab the interesting pieces? [09:54:10] (03Abandoned) 10MaxSem: Quick test to see if tests still work [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86827 (owner: 10MaxSem) [09:54:58] MaxSem, in order for me to do all that magic I will need to at least load zero config - which might or might not work without the entire framework [09:55:17] probably will but will need some testing [09:55:28] yurik: just make mediawiki handle m.wikipedia.org and zero. ? [09:55:31] and that's the real problem - no "full blown test env" :( [09:55:40] extract2.php has no problems with loading MW [09:56:17] MaxSem, cool [09:56:24] good to know [09:56:47] will try to implement it soon [09:58:18] only slightly unrelated, but where do you put MW code that's completely shared across all languages? [09:58:21] m.wikipedia.org [09:58:24] for example? [09:58:32] (or the proposed global shorturl service) [09:58:38] m.wikipedia.org currently gets redirected to wikipedia.org [09:58:48] which is really ugly and we should fix [09:58:48] !log maxsem synchronized docroot and w [09:58:59] Logged the message, Master [09:59:29] s/redirected/rewritten/ actually [09:59:34] hmm, so does http://bits.wikimedia.org/w/404.php mean that it works or not? :P [09:59:56] yurik: so will you just handle those URLs directly, without a varnish redirect? [10:00:14] yes, that's the general idea :)( [10:00:32] just need to figure out how www works and do the same here [10:00:34] so we can abandon that current patch [10:00:47] which seems like an ugly workaround [10:00:56] i'm talking about https://gerrit.wikimedia.org/r/#/c/86721/1 [10:01:14] mark, i think we can keep it and fix the final redirect target when needed. Dflts vars have to be removed regardless [10:01:24] i mean - don't merge it [10:01:41] i will fix update that patch later once i figure out how to do the whole thing in MW [10:01:45] I think we should fix it properly instead of changing it but not quite fixing it [10:01:56] ok, I'll -2 it now [10:02:01] MaxSem: it works [10:02:03] exactly [10:02:15] (03CR) 10Mark Bergsma: [C: 04-2] "This is just an ugly workaround instead of a real fix. Let's fix it properly." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86721 (owner: 10Yurik) [10:05:11] alright [10:05:13] have a nice flight [10:09:30] thx mark - its in 8 hrs.... need to pack i guess... will sleep on the plane [10:11:35] MaxSem, re weird meta request that timesout for all traffic -- http://meta.m.wikimedia.org/wiki/Special:RecordImpression?result=hide&reason=empty&country=US&uselang=en&project=wikipedia&db=enwiki&bucket=0&anonymous=true&device=unknown [10:11:42] http://meta.m.wikimedia.org/wiki/Special:BannerRandom?uselang=en&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=0&country=US&device=unknown&slot=15 [10:11:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [10:12:40] the first link gets accessed twice - once for meta.m.wikimedia.org and once meta.wikimedia.org; m - gets 200, meta gets 302\ [10:13:18] MaxSem, do you see that in your browser when you simulate moblie device UA? [10:13:28] Gotta love bugs in software, eh? [10:16:50] mhm, don't see it in exception.log - appears to be handled [10:23:28] MaxSem, i'm just looking at the networking tab of the FF's firebug [10:23:54] it makes the site spend extra 10 seconds loading it seems [10:25:09] ask mwalker|away [10:26:25] heh, was just a heads up. I'm more interested to know why zero doesn't show banner for the new carrier i just created :( [10:30:11] (03PS3) 10Hashar: contint::localvhost easily setup an apache listener [operations/puppet] - 10https://gerrit.wikimedia.org/r/86665 [10:31:32] mark: when you can, I have made Apache 'Listen …' statements to be written in a different conf file https://gerrit.wikimedia.org/r/#/c/86665/ That is really ugly though :( [10:34:25] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 10:34:23 UTC 2013 [10:34:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [10:48:03] hashar: is it considered ugly to depend on a different module when writting a module? [10:48:15] not necessarily [10:48:55] hashar: i'm working to split the packages in : modules/contint/manifests/packages.pp [10:49:38] but we have the java module, which takes care of most of the java related packages, my question is can I call it? or the packages should be independent? [10:49:48] (03CR) 10Mark Bergsma: "Meh, until we have a decent Apache module it's all ugly." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86665 (owner: 10Hashar) [10:53:51] (03CR) 10Mark Bergsma: [C: 032] "Meh, until we have a decent Apache module it's all ugly." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86665 (owner: 10Hashar) [10:58:47] matanya: yeah you can call that [11:01:52] it needs a rewrite, but it's fine for using it now [11:02:10] ottomata: btw, how's openjdk? :-) [11:03:04] (03CR) 10Mark Bergsma: [C: 032] Normalize and Vary on the forceHTTPS cookie [operations/puppet] - 10https://gerrit.wikimedia.org/r/86268 (owner: 10Mark Bergsma) [11:04:25] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 11:04:20 UTC 2013 [11:04:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:22] (03PS6) 10Hashar: contint: puppet class to setup browsertests slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/85264 [11:11:48] hashar: so using "imort java" and then calling java { 'java-6-openjdk': distribution => 'openjdk' version => 6, alternative => false } java { 'java-7-openjdk': distribution => 'openjdk' version => 7, alternative => false } should work, correct? [11:12:35] * matanya is driving hashar crazy today :| [11:16:16] (03PS1) 10Hashar: contint::localvhost is really a define [operations/puppet] - 10https://gerrit.wikimedia.org/r/86831 [11:16:34] (03CR) 10Hashar: [C: 031] contint::localvhost is really a define [operations/puppet] - 10https://gerrit.wikimedia.org/r/86831 (owner: 10Hashar) [11:16:40] .. [11:17:02] mark: I have created a Class when I wanted a define :( https://gerrit.wikimedia.org/r/86831 [11:17:06] Class -> define [11:17:20] had Invalid resource type contint::localvhost at modules/contint/manifests/website.pp:105 [11:20:05] but two weeks haven't passed yet since you submitted this patch, eh [11:20:10] so I can't possibly merge that yet [11:21:03] (03CR) 10Mark Bergsma: [C: 032 V: 032] contint::localvhost is really a define [operations/puppet] - 10https://gerrit.wikimedia.org/r/86831 (owner: 10Hashar) [11:21:10] :D [11:24:38] I should have done that patch differntly [11:25:47] (03PS1) 10Dzahn: add nrpe to node zirconium so we can monitor processes like etherpad [operations/puppet] - 10https://gerrit.wikimedia.org/r/86833 [11:25:48] (03PS1) 10Dzahn: use clientaddrUsesPort and a high port to send out outgoing SNMP requests per man snmp.conf so that non-root users can bind to it and we avoid getting SNMP permission denied errors when icinga runs these and we send the client IP since I9c7b1c11aeef2ff984 [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 [11:28:09] ArgumentError: comparison of Hash with Hash failed [11:28:10] niceeee [11:28:20] what? [11:28:21] awwww, Java :P [11:28:24] why didn't you test this in labs? [11:28:41] irb(main):003:0> [ { 'f' => 1 }, { 'b' => 3 } ].sort [11:28:42] ArgumentError: comparison of Hash with Hash failed [11:28:54] awww, Ruby! :P [11:28:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:29:06] mark: Iwas too lazy to setup fake SSL certificates on a labs instance to get apache to start [11:29:22] mark: and randomly flipping a coin because of GlusterFS is not that fun anymore :-] [11:30:34] I should rewrite that Listen madness anyway and use a basic template such as: [11:30:35] Listen 127.0.0.1:<%= port %-> http [11:30:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 10.262 second response time [11:30:35] Listen [::1]:<%= port %-> http [11:32:07] (03PS2) 10Dzahn: add :0 as port when sending outgoing SNMP requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 [11:33:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:36:35] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 11:36:28 UTC 2013 [11:36:40] (03PS3) 10Dzahn: add :0 as port when sending outgoing SNMP requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 [11:36:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:46] (03PS1) 10Hashar: contint: simplify apache listen template [operations/puppet] - 10https://gerrit.wikimedia.org/r/86836 [11:38:00] mark: over simplified my soup with https://gerrit.wikimedia.org/r/86836 :D [11:38:45] (03CR) 10Mark Bergsma: [C: 032] contint: simplify apache listen template [operations/puppet] - 10https://gerrit.wikimedia.org/r/86836 (owner: 10Hashar) [11:38:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 25.592 second response time [11:38:56] sorry :( [11:39:01] was too tired yesterday I guess [11:40:42] (03CR) 10ArielGlenn: [C: 031] "this is due to a bug in the client code (which shuld not try to bind to port 161), e.g. https://groups.google.com/forum/#!topic/mailing.un" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 (owner: 10Dzahn) [11:50:38] (03CR) 10Dzahn: [C: 032] add :0 as port when sending outgoing SNMP requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 (owner: 10Dzahn) [11:54:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:55:19] (03CR) 10Dzahn: "fixed PDU checks . https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?hostgroup=pdus&style=detail&nostatusheader" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86834 (owner: 10Dzahn) [11:56:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 27.050 second response time [11:59:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:00:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.404 second response time [12:06:45] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 12:06:36 UTC 2013 [12:07:30] (03Abandoned) 10Dzahn: add nrpe to node zirconium so we can monitor processes like etherpad [operations/puppet] - 10https://gerrit.wikimedia.org/r/86833 (owner: 10Dzahn) [12:07:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:07:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:08:02] (03CR) 10Hashar: [C: 04-1] "(11 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/61767 (owner: 10Physikerwelt) [12:08:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 18.746 second response time [12:14:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:15:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 17.612 second response time [12:33:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:38:05] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 12:37:59 UTC 2013 [12:38:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [12:41:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.620 second response time [12:45:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:46:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.083 second response time [12:48:06] Are dzahn, lcarr or rhalsell around? [12:49:26] the first one would mutante [12:49:45] LeslieCarr and RobH respectively, although they're SF-based [12:51:32] got him in pm [13:05:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 13:05:52 UTC 2013 [13:06:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:10:28] (03PS1) 10Faidon Liambotis: varnish mobile: don't override MW's X-Analytics [operations/puppet] - 10https://gerrit.wikimedia.org/r/86846 [13:10:39] mark: ^ [13:11:30] (03PS1) 10Matanya: Nrpe: /usr/lib/nagios/plugins/check_dpkg should be absent everywhere. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86847 [13:12:12] !log start xtrabackup s3 db1035 to db1038 [13:12:23] Logged the message, Master [13:14:51] (03CR) 10Mark Bergsma: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86846 (owner: 10Faidon Liambotis) [13:15:35] is the TTL 30 days? [13:15:55] I think so [13:17:11] hiyalls [13:17:14] https://gerrit.wikimedia.org/r/#/c/86746/ [13:17:20] mmmm need breakfast [13:18:39] (03PS2) 10Faidon Liambotis: varnish mobile: don't override MW's X-Analytics [operations/puppet] - 10https://gerrit.wikimedia.org/r/86846 [13:20:14] (03CR) 10Faidon Liambotis: [C: 032] varnish mobile: don't override MW's X-Analytics [operations/puppet] - 10https://gerrit.wikimedia.org/r/86846 (owner: 10Faidon Liambotis) [13:21:25] MaxSem: Again, I made a patch to only enable PHP for /w/404.php, not for the whole w dir :p [13:27:20] (03CR) 10Hashar: [C: 031] Shell access for Bryan Davis. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86764 (owner: 10BryanDavis) [13:29:51] (03CR) 10Dzahn: "wait, the check command is "check_dpkg" via NRPE , nrpe_check_dpkg and that is /usr/local/lib/nagios/plugins/check_dpkg. having it in /us" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86847 (owner: 10Matanya) [13:31:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:32:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 11.020 second response time [13:33:42] (03CR) 10Matanya: "Seems so. what I understood from it is was in an incorrect path. which should have been absent. looks like it is gone anyway." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86847 (owner: 10Matanya) [13:35:18] (03CR) 10Akosiaris: [C: 032] "Yes that was the original intent. nrpe/nagios plugins shipped by us should not be in /usr/etcetcetc but in /usr/local/etcetcetc. I meant t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86847 (owner: 10Matanya) [13:36:12] (03CR) 10Mark Bergsma: "text-varnish is text, served by Varnish, as opposed to Squid. We're currently migrating them one project at a time." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86746 (owner: 10Ottomata) [13:36:15] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Tue Oct 1 13:36:07 UTC 2013 [13:36:45] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [13:36:56] great, thanks mark, paravoid and I were talking about that yesterday and weren't sure [13:37:03] I am sure of what it is [13:37:14] I just said it doesn't matter where you'll put them now [13:37:22] (03CR) 10Dzahn: [C: 031] "key like on office page, added by User:BDavis (WMF). good point about the next UID" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86764 (owner: 10BryanDavis) [13:38:00] (03PS1) 10Akosiaris: Adding bacula module rspec tests [operations/puppet] - 10https://gerrit.wikimedia.org/r/86850 [13:40:20] (03CR) 10Akosiaris: [C: 032] Adding bacula module rspec tests [operations/puppet] - 10https://gerrit.wikimedia.org/r/86850 (owner: 10Akosiaris) [13:42:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:44:59] (03PS1) 10Akosiaris: Use generic Rakefile in nrpe module [operations/puppet] - 10https://gerrit.wikimedia.org/r/86852 [13:50:26] paravoid, ja i know [13:50:33]