[01:57:33] (03CR) 1020after4: "So we probably need to package things up instead of relying on git to deploy the code because there are 3 repositories that need to be kep" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/132505 (owner: 10Dzahn) [02:15:52] !log LocalisationUpdate completed (1.24wmf4) at 2014-05-22 02:14:49+00:00 [02:15:59] Logged the message, Master [02:25:16] (03CR) 10CSteipp: Improve nginx TLS/SSL settings. (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [02:30:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 11:30:29 PM UTC [02:39:21] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-22 02:38:18+00:00 [02:39:25] Logged the message, Master [03:00:38] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu May 22 03:00:28 UTC 2014 [03:38:37] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu May 22 03:37:31 UTC 2014 (duration 37m 30s) [03:38:41] Logged the message, Master [04:55:23] (03CR) 10Chad: [C: 031] "lgtm, merge when we're comfortable." [operations/software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/133771 (owner: 10Manybubbles) [04:56:30] (03CR) 10Chad: [C: 031] "lgtm, merge whenever." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134585 (owner: 10Manybubbles) [05:00:36] (03CR) 10Chad: "I think we should use trebuchet to deploy the repo(s) rather than relying on puppet to clone them." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132505 (owner: 10Dzahn) [05:11:26] (03PS2) 10MZMcBride: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [06:04:41] (03CR) 10Nemo bis: "By the way, you can compare with http://translatewiki.net/ which uses https://git.wikimedia.org/blob/translatewiki.git/HEAD/puppet%2Fmodul" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [06:18:39] do we have a preferred service supervisor / init system? upstart? systemd? plain old init.d scripts? [06:23:58] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:48] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.006 second response time [06:28:05] twentyafterfour: upstart [06:29:16] cool. So when puppetizing something I should include an upstart conf for the service and any related daemons in the puppet module? [06:30:37] yes [06:36:52] <_joe_> good morning [06:37:48] <_joe_> twentyafterfour: it strongly depends on what you are puppetizing [06:38:27] <_joe_> most external softwares should be packaged as debs and have the upstart job in the package [06:54:21] morning [06:56:52] <_joe_> good morning matanya [07:03:53] _joe_: ok that makes sense [07:04:31] <_joe_> twentyafterfour: the answer is always 'it depends on the use case :)' [07:05:08] do we have documentation on recommended/required packaging procedures? [07:09:03] <_joe_> twentyafterfour: can I ask what software/service you are talking about? [07:09:10] * _joe_ is now curious [07:09:40] _joe_: phabricator [07:10:21] thinking about the best way to package and deploy it. Phabricator doesn't follow many standard os conventions and instead reinvents a few wheels [07:10:46] <_joe_> twentyafterfour: let me check it [07:11:50] <_joe_> twentyafterfour: it's basically a LAMP software right? [07:12:17] yeah with a few daemons and dependencies [07:12:47] <_joe_> so for the lamp stack use the usual puppet setup [07:12:52] a few people have already been working on it, I'm just working on making it production ready [07:13:20] https://gerrit.wikimedia.org/r/#/c/132505/5 [07:13:22] <_joe_> I guess you ask chasemp for advice on that [07:13:32] <_joe_> when he's around [07:14:17] yeah I'm trying to finish some of it so he doesn't have to [07:14:35] <_joe_> well, the 'daemons' you referred to [07:14:46] <_joe_> what are they, separate projects? [07:14:57] <_joe_> written in what language? [07:15:02] they are php scripts that run as separate processes [07:15:05] background jobs [07:15:21] <_joe_> ok so install everything without packages [07:15:27] <_joe_> never package php, IMO [07:15:34] really? [07:15:44] <_joe_> php apps, sorry [07:15:57] <_joe_> non php the language or the execution environment [07:16:36] why not package them? deploying from git isn't ideal in this case but packaging is definitely a bit of a pain [07:16:51] <_joe_> I was looking at your comment in the patch [07:17:20] <_joe_> I guess libphutil, arcanist and phabricator do have a coordinated release schedule [07:17:48] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data exceeded the critical threshold [500.0] [07:18:06] yeah it's ad-hoc but they need to be in sync when one updates to a new revision it often requires the latest revision of another... [07:18:06] <_joe_> uhm gotta look into that sorry [07:18:57] <_joe_> ok we had a BIG spike around 7:15 UTC [07:19:02] <_joe_> checking that... [07:23:54] (03CR) 1020after4: "trebuchet looks like an ok way to go, as long as we can specify dependencies among the packages or just put libphutil and phabricator into" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132505 (owner: 10Dzahn) [07:24:51] <_joe_> twentyafterfour: sorry, looking into an issue [07:25:05] _joe_: no problem [07:25:14] <_joe_> It's already over, but if I don't try to understand it now I'll forget to do it later [07:25:31] indeed, take your time :) [07:30:48] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [07:33:19] <_joe_> ok I'm back [07:33:38] <_joe_> I know not enough about trebuchet to know if dependencies are supported [07:33:56] <_joe_> you could just release the three as a bundle via trebuchet I guess [07:57:53] good morning [08:00:01] <_joe_> ciao hashar [08:00:52] (03CR) 10Alexandros Kosiaris: [C: 032] "@dzahn, niah, matanya is correct. We don't even run the puppet daemon anymore (and for a long time now) and TBH I see no reason to run the" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131333 (owner: 10Matanya) [08:04:44] _joe_: 503's : [08:04:51] If you report this error to the Wikimedia System Administrators, please include the details below. [08:04:51] Request: GET http://he.wikipedia.org/wiki/%D7%A2%D7%9E%D7%95%D7%A1_6, from 10.128.0.108 via cp4016 cp4016 ([10.128.0.116]:3128), Varnish XID 1201529834 [08:04:52] Forwarded for: 62.0.53.15, 10.128.0.108, 10.128.0.108 [08:04:52] Error: 503, Service Unavailable at Thu, 22 May 2014 07:15:24 GMT [08:05:18] and : [08:05:21] If you report this error to the Wikimedia System Administrators, please include the details below. [08:05:22] Request: GET http://he.wikipedia.org/wiki/%D7%92%D7%90%D7%99%D7%94_(%D7%9C%D7%95%D7%95%D7%99%D7%99%D7%9F), from 10.128.0.108 via cp4017 cp4017 ([10.128.0.117]:3128), Varnish XID 16023081 [08:05:22] Forwarded for: 62.0.53.15, 10.128.0.108, 10.128.0.108 [08:05:23] Error: 503, Service Unavailable at Thu, 22 May 2014 07:15:24 GMT [08:06:25] akosiaris you can look as well :) [08:06:57] and more: Request: GET http://he.wikipedia.org/wiki/%D7%A6%27%D7%90%D7%A0%D7%92-%D7%90%D7%94_3, from 10.128.0.109 via cp4009 cp4009 ([10.128.0.109]:3128), Varnish XID 337384636 [08:07:14] <_joe_> matanya: pastebin!!! [08:07:24] oh, right, sorry :/ [08:07:30] <_joe_> matanya: now? [08:07:40] yes. see times [08:08:04] actually for last hour, on and off [08:09:22] <_joe_> matanya: I suspect it's some connectivity issue , lemme take a further look though [08:09:58] _joe_: saw your mail, i'm wonder why i go through ulsfo [08:10:07] it makes no sense to me [08:10:15] <_joe_> matanya: what is your ip? in query [08:13:02] matanya: I must say I have to congratulate you for being able to switch from left to right to right to left writing [08:13:29] I am looking at the pages you pasted and man, my brain is having a hard time [08:13:43] poor you :) i'm so use to it [08:14:36] <_joe_> you all switch alphabets as well [08:14:45] <_joe_> it's pretty impressive for me [08:14:46] anyway, more info on this so we can debug it ? [08:14:57] and gladly, gnome support separate key map for every window, so chat is en only [08:15:01] _joe_: hey you are right! I 'd never thought of that [08:15:39] akosiaris: why i'm i routed to ulsfo ? that is my first question [08:15:55] indeed [08:15:55] no caching in esams ? [08:16:09] nothing to do with that (hopefully) [08:16:16] it should be geodns [08:17:05] right, and in dns israel goes to esams (iirc) [08:17:35] so what does host he.wikipedia.org say for you ? [08:18:10] I think you should be seeing wikipedia-lb.wikimedia.org is an alias for text-lb.esams.wikimedia.org. [08:18:13] do you ? [08:18:20] yes, if do [08:18:24] *i do [08:18:42] but when refreshing a page and looking in firebug: cp1068 miss (0), cp4010 miss (0), cp4010 frontend miss (0) [08:19:35] oh, now it changed again [08:19:42] and you have not messed up with any firefox settings, any weird extensions setting Host: headers and things like that right ? [08:19:45] text-lb.ulsfo.wikimedia.org [08:19:56] i haven't [08:19:56] ah there we are [08:19:58] weird [08:20:25] connected to 198.35.26.96 [08:20:35] which is indeed ulsfo [08:21:06] <_joe_> matanya: what is your dns? [08:21:24] <_joe_> maybe it does something funny [08:21:36] outside of our lan it is google [08:21:44] 8.8.8.8 ? [08:21:54] <_joe_> ok [08:21:58] <_joe_> so that's the reason [08:22:01] <_joe_> lemme check [08:22:06] so it does not honour EDNS client subnet then ? [08:22:17] They had fixed that [08:22:32] <_joe_> akosiaris: it does [08:22:34] yes, 8.8.8.8 [08:22:36] <_joe_> just checked [08:22:46] <_joe_> matanya: dig he.wikipedia.org @8.8.8.8 please [08:22:49] http://dpaste.com/3677CW3/ [08:23:14] vs: http://dpaste.com/049TAF6/ [08:23:27] indeed google's fault [08:23:32] <_joe_> matanya: not so sure [08:23:54] <_joe_> matanya: google points me to esams [08:24:01] see the second paste [08:24:21] <_joe_> yes, I think this is something wrong in geoip data we get [08:24:42] <_joe_> because IL is supposed to be directed to esams [08:25:43] that is relatively easy to verify [08:26:06] matanya: your IP please ? [08:26:12] or at least your subnet [08:26:15] <_joe_> akosiaris: just checked [08:54:53] akosiaris: rights question, when one has sudo rights on a machine it mean he can run stuff like otto@box: sudo command if he has root on the same machine this mean he has root@box, not the right to do any command as sudo. is this right? this is the setup ? [08:56:08] matanya: well if has root he implicitly has the right to use sudo (not that he needs to) but yes you are correct [08:58:02] <_joe_> matanya: dig +trace he.wikipedia.org [08:58:29] <_joe_> do not paste the result here pls! [08:59:36] <_joe_> akosiaris: which california ip got esams as an answer? [09:00:18] the lb answer [09:00:28] 198.35.26.0 [09:00:43] but ns0.wikimedia.org answers text-lb.eqiad for this one [09:01:07] _joe_: https://dpaste.de/JzAr and https://dpaste.de/0v2Q [09:01:15] argh this keeps on changing [09:01:18] this is erratic... [09:11:26] <_joe_> matanya: no I asked you to do [09:11:36] <_joe_> "dig he.wikipedia.org +trace" [09:12:11] https://dpaste.de/Y2w9 [09:15:21] (03PS1) 10Matanya: admins: add otto as analytics root [operations/puppet] - 10https://gerrit.wikimedia.org/r/134793 [09:19:57] (03CR) 10Filippo Giunchedi: [C: 031] remove 'maerlant' from site.pp/dsh/dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134745 (owner: 10Dzahn) [09:20:37] (03PS1) 10Matanya: admins: move otto to class admin rather than sudo_user [operations/puppet] - 10https://gerrit.wikimedia.org/r/134794 [09:23:26] (03CR) 10Matanya: [C: 031] remove 'maerlant' from site.pp/dsh/dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134745 (owner: 10Dzahn) [09:23:50] (03PS2) 10Filippo Giunchedi: update legal terms for dumps.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/134121 [09:24:04] (03CR) 10Filippo Giunchedi: [C: 032] update legal terms for dumps.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/134121 (owner: 10Filippo Giunchedi) [09:24:14] (03CR) 10Filippo Giunchedi: [V: 032] update legal terms for dumps.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/134121 (owner: 10Filippo Giunchedi) [09:25:03] (03PS4) 10Giuseppe Lavagetto: puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 [09:36:43] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 (owner: 10Giuseppe Lavagetto) [09:40:19] (03PS1) 10Matanya: admins: add manybubbels and elasticsearch group [operations/puppet] - 10https://gerrit.wikimedia.org/r/134796 [09:44:41] (03CR) 10Alexandros Kosiaris: [C: 032] Fix parsoidCacheUri config in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134643 (owner: 10Catrope) [09:45:02] (03CR) 10Alexandros Kosiaris: [C: 032] Add hewiki to Parsoid config in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134649 (https://bugzilla.wikimedia.org/65490) (owner: 10Catrope) [09:45:04] (03PS1) 10Matanya: admins: add elasticsearch-roots to elasticsearch nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/134797 [09:46:24] (03PS2) 10Matanya: admins: add manybubbels and elasticsearch group [operations/puppet] - 10https://gerrit.wikimedia.org/r/134796 [09:46:51] (03PS3) 10Alexandros Kosiaris: let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [09:46:59] (03CR) 10Alexandros Kosiaris: [C: 032] let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [09:47:37] (03PS3) 10Alexandros Kosiaris: fix "read_only" setting in rsyncd setups [operations/puppet] - 10https://gerrit.wikimedia.org/r/133647 (owner: 10Dzahn) [09:49:03] phuedx: you are welcome :-) [09:49:11] :) [09:49:39] i was looking in to making the zuul-config change myself [09:49:47] (03CR) 10Alexandros Kosiaris: [C: 032] fix "read_only" setting in rsyncd setups [operations/puppet] - 10https://gerrit.wikimedia.org/r/133647 (owner: 10Dzahn) [09:49:53] phuedx: and welcome around :D I haven't spot you are working for Growth [09:50:02] ah sorry [09:50:03] and was trying to figure out if adding the test and lint sections would work [09:50:07] no no! [09:50:09] (03PS4) 10Alexandros Kosiaris: let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [09:50:10] thanks for doing it! [09:50:15] :D [09:50:18] I though you were US based so did it myself :-/ We can pair together for the next change if you want [09:50:21] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [09:50:33] that'd be great! thanks! [09:50:42] i'm london based [09:50:48] I wasn't sure whether I could add the test/gate-and-submit pipeline after the template. Seems Zuul supports it quite well nowadays. [09:50:51] \O/ [09:50:57] one more european dev yeahhh [09:51:09] I am in France myself, and attempt to maintain the CI infrastructure [09:51:21] so I am barely involved in mediawiki dev anymore [09:51:54] directly [09:51:56] but you facilitate it daily! [09:52:09] akosiaris: would you please have some bandwidth to +2 two changes for contint in labs please? I need a few iptables rules there to reach NATed instances. [09:52:28] sure [09:52:30] akosiaris: the changes are already deployed on the local puppet master : https://gerrit.wikimedia.org/r/127213 https://gerrit.wikimedia.org/r/129687 [09:52:42] andrew +2ed them yesterday but they needed a rebase hehe [09:52:56] <_joe_> hashar: I can help with that [09:53:06] race with alexandros so! [09:53:54] I was mumbling yesterday how I got too many patch opened. Found out I haven't asked reviews for them and whenever I asked review all of them got merged under a couple hours \O/ [09:54:41] \o/ [09:54:45] <_joe_> hashar: I'm building debian packages, which is just taking time between builds [09:54:51] ahh [09:55:07] I even got a bunch of Jenkins jobs to build debian packages on patch submission [09:55:09] hashar: so including base::firewall means you will got ferm with policy DROP there [09:55:13] is that ok with you ? [09:55:20] akosiaris: let me triple check [09:55:41] Quite probably you have already done it in another class just making sure [09:55:47] akosiaris: yeah that is fine. Ferm is already applied on the targeted hosts [09:55:52] cool [09:55:54] merging then [09:55:54] Chain INPUT (policy DROP) [09:55:59] thanks for the warning [09:56:08] (03CR) 10Alexandros Kosiaris: [C: 032] contint: apply beta natfix on Jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/127213 (owner: 10Hashar) [09:56:46] so both merged, anything else ? [09:57:08] the rests of the changes are experimental [09:57:11] before I have to decide whether I want to put my head into ganglia and/or diamond or OpenLDAP [09:57:32] thanks akosiaris ! happy ganglia/diamond digging hehe [09:57:49] thanks for giving me the answer to my dilema :-) [09:58:08] hmm. now i hit esams, did you change anything _joe_ ? [09:58:18] matanya: no we did not [09:58:24] <_joe_> matanya: your dns cache expired [09:58:25] <_joe_> :) [09:58:41] it seems it was some erratic behaviour on google's part [09:58:46] or maybe not [09:58:56] the lesson is we need some better DNS monitoring [09:59:35] (03PS8) 10Hashar: contint: split Zuul server and merger (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 [09:59:45] matanya: please ping us if you suddenly switch to another DC again [09:59:52] (03CR) 10Hashar: [C: 04-1] "rebased. Still a work in progress." [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 (owner: 10Hashar) [10:00:02] sure akosiaris [10:01:56] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[puppet-lint] is already defined in file /etc/puppet/manifests/role/labs.pp at line 33; cannot redefine at /etc/puppet/modules/contint/manifests/packages/labs.pp:66 on node i-000001bd.eqiad.wmflabs [10:01:58] pfff [10:02:02] i hate puppet [10:02:25] don't, at least you have a clear error message :) [10:02:58] <_joe_> hashar: it's a pity my compiler does not work on labs [10:03:00] he [10:03:11] _joe_: is london configured to hit eqiad or esams ? [10:03:25] so wouldn't you love it if you could do include 'packagename' ? [10:03:36] and it would work the same way as classes ? [10:03:45] I know I would be thrilled... [10:03:47] sounds like a dream to me [10:04:40] (03PS1) 10Hashar: contint: puppet-lint is now available in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 [10:05:04] _joe_: if you can https://gerrit.wikimedia.org/r/134799 in :-] That drop my dupe puppet-lint package which is now installed on all labs instances [10:05:25] maybe I should add a require Package['puppet-lint'] or something [10:05:40] <_joe_> akosiaris: we can try to do something like a define that does not declare the package if it's already declared [10:05:49] <_joe_> it will slow puppet down a lot, though [10:06:02] there is such a define in puppet stdlib [10:06:35] and how will you name the define ? [10:07:01] let's say it is define lala::package () { blah blah} [10:07:02] akosiaris _joe_ now london is hitting eqiad instead of esams [10:07:31] <_joe_> matanya: how did you determine that? [10:07:46] from another box i have [10:08:09] _joe_ : ensure_resource( 'package', 'puppet-lint', {'ensure' => 'present' }) :-D [10:08:10] <_joe_> matanya: ip please [10:08:11] https://forge.puppetlabs.com/puppetlabs/stdlib [10:08:23] <_joe_> hashar: ugly [10:08:26] yeah [10:08:43] I am not sure how to declare the dependency though [10:09:11] (03CR) 10Hashar: [C: 04-1] "Ideally I would want to declare the dependency on puppet-lint to make sure it is around." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 (owner: 10Hashar) [10:10:36] <_joe_> hashar: a Require [10:10:50] any clue what would be the exact syntax? [10:10:56] <_joe_> hashar: require => Package['puppet-lint'] [10:11:41] _joe_: ah but I would need to have it inside the class definition ( contint::packages::labs ) [10:11:50] _joe_: aka have contint::packages::labs to require the package [10:12:24] maybe i can just: require Package['puppet-lint'] [10:12:45] <_joe_> hashar: what needs this package? [10:12:56] modules/contint/manifests/packages/labs.pp [10:12:59] for contint purposes [10:13:10] <_joe_> no I mean [10:13:10] that class is only installed on labs and the package is nowadays provided on all instances [10:13:15] <_joe_> where do you use it? [10:13:23] <_joe_> in the contint module [10:13:52] it is included in role classes [10:14:16] (03CR) 10Nikerabbit: standardize a few things in admins.pp for conversion (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134394 (owner: 10Rush) [10:14:35] <_joe_> no, I mean, is there some part of the contint module that uses puppet-lint? [10:14:51] <_joe_> or some script that will not run without it? [10:15:14] yeah Jenkins will have some jobs failling [10:15:34] the contint::packages::labs is to be installed on Jenkins nodes in labs and ensure we have everything required by the Jenkins jobs [10:15:51] maybe I can Class['contint::packages::labs'] => Package['puppet-lint'] [10:18:27] <_joe_> hashar: let me take a look at the contint class [10:19:28] maybe https://dpaste.de/0Akt [10:19:37] <_joe_> hashar: you just include contint::packages::labs or a parent class? [10:20:04] I just include contint::packages::labs [10:20:09] <_joe_> oh ok [10:20:25] that is done in manifests/role/ci.pp role::ci::slave::labs [10:20:48] <_joe_> you know, I'm not sure what you wrote will work. It should, not sure it will [10:21:18] <_joe_> I have some kind of memory of these things failing with a Class declaration [10:21:26] <_joe_> let me build a test case though [10:22:15] I have a puppet master, trying it :) [10:22:43] <_joe_> hashar: you will not see if that does not do what advertised, though :) [10:24:11] notice: /Stage[main]/Role::Labs::Instance/Package[puppet-lint]/ensure: ensure changed 'purged' to 'present' [10:24:11] \O/ [10:24:17] (03PS4) 10Matanya: dns recurses: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/133515 [10:24:27] (03PS2) 10Hashar: contint: puppet-lint is now available in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 [10:24:32] <_joe_> hashar: and that is the labs class doing that [10:24:51] (03CR) 10Hashar: "Apparently declaring the dependency in the class works fine:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 (owner: 10Hashar) [10:25:05] _joe_: yup wich is fine [10:26:10] if I remove the package definition from the labs class. I get a dependency error which is nice [10:26:17] SERVER: Could not find resource 'Package[puppet-lint]' for relationship on 'Class[Contint::Packages::Labs]' [10:26:46] (03CR) 10Hashar: [V: 032] "If the puppet-lint package is not defined somewhere else, puppet complains with:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 (owner: 10Hashar) [10:27:37] sorry I am stealing a bunch of your brain cycles :( [10:28:25] <_joe_> hashar: yes, the only thing that is tricky is, that dependency will NOT ensure package puppet-lint is installed before class Contint::Packages::Labs is applied [10:28:40] <_joe_> which is definitely NOT what you would expect [10:29:14] <_joe_> with classes included with 'include', the -> works differently that it would do with defines [10:29:24] <_joe_> or with explicitly declared classes [10:29:26] <_joe_> FUN! [10:30:54] ohh [10:31:27] <_joe_> https://dpaste.de/mPE9 for reference [10:33:47] <_joe_> from the puppet manual: '-> (ordering arrow) Causes the resource on the left to be applied before the resource on the right' [10:34:19] so that will ensure puppet-lint is realized first isn't it ? [10:34:38] the ordering is not much of an issue anyway. I just need to make sure puppet-lint is present [10:34:41] <_joe_> it should. It will not! [10:36:41] <_joe_> it will do what you need though :) [10:37:32] \O/ [10:37:40] so I guess you can land https://gerrit.wikimedia.org/r/#/c/134799/ in :) [10:37:45] <_joe_> btw, when you start writing python in the middle of a puppet manifest, your brain is really trying to tell you something [10:38:03] ahah [10:39:40] <_joe_> oh no, I will have to give your change -1 I guess [10:39:54] <_joe_> -1 "puppet is dumb" [10:40:24] <_joe_> the dependency gets declared only when the class and/or define is applied, it seems [10:41:07] <_joe_> well, since your needs are met, who cares :) [10:41:29] yeah I guess that is good enough [10:41:47] it is not like that part of the code is processing millions of dollars :] [10:43:11] (03CR) 10Giuseppe Lavagetto: [C: 032] contint: puppet-lint is now available in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134799 (owner: 10Hashar) [10:43:49] <_joe_> hashar: done [10:44:42] I gotta learn italian now [10:44:42] <_joe_> (btw, I hate having my RL name printed in bold here everytime I do something on gerrit, it kinda feels weird to see my RL name in IRC) [10:44:47] to figure out a way to say thanks [10:45:05] <_joe_> hashar: grazie == mercy [10:45:06] that is probably your labs account name [10:45:08] <_joe_> ehm [10:45:12] <_joe_> s/y/i/ [10:45:19] <_joe_> hashar: it is [10:45:19] mine is Hashar for whatever reason [10:45:23] instead of my real name :-( [10:45:38] <_joe_> hashar: not a real issue, it's just weird :P [10:46:04] <_joe_> well, it feels weird. [10:47:31] we [10:47:41] we can most probably have Gerrit to use the real name field in LDAP for display purposes [10:59:15] (03CR) 10Nemo bis: Gather all soft-disabled uploads wikis in one config item (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134400 (owner: 10Nemo bis) [10:59:37] seems like paravoid is signing a lot of pgp keys. He is near the center of the Greek web of trust http://www.void.gr/kargig/blog/wp-content/GreekWoT_2012_02_network_v2.png D [10:59:51] (purple disc in the middle, slightly on the left of a huge red disk) [10:59:55] disc [10:59:56] ahahaha [11:00:17] (03PS5) 10Nemo bis: Gather all soft-disabled uploads wikis in one config item [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134400 [11:04:12] the whole blog for our Greek ops http://www.void.gr/kargig/blog/2013/07/08/greek-pgp-web-of-trust-2012-edition/ :) [11:06:52] Anyone around who can look into email logs for password reset email status? [11:07:17] There's yet another user quite annoyed that they are locked out of their account [11:07:30] But supposedly-sent password reset emails do not arrive in their inbox [11:09:39] Krenair: I can try. pm me with details [11:09:55] email is probably wrong / not expected :-( [11:10:06] user.user_email is a good source :) [11:10:19] ? [11:10:24] hashar, actually I copied their address they contacted us from into the password reset form myself [11:10:35] But they reported no email arriving [11:13:15] (03CR) 10Gilles: Introduce finer-grained Media Viewer EventLogging sampling [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [11:46:30] The password reset form often lies, to avoid leaks [11:49:20] Are these cases documented? [11:57:56] I don't think so, it just says that an email was sent unconditionally, even when no email was sent at all or the email didn't exist etc. IIRC. [12:01:06] Easily tested [12:17:58] PROBLEM - mysqld processes on db1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [12:18:23] Reedy, perhaps you can check to see whether a given email address is confirmed against an account? [12:19:25] <_joe_> Nemo_bis: there is no easy way to monitor the delivery of an email message from an application perspective. So even if the local MTA accepted the message, you have no guarantee of the correctness of the email address and/or of delivery [12:40:22] (03PS1) 10Gilles: Launch Media Viewer for all users on Italian wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134808 [12:40:24] (03PS1) 10Gilles: Launch Media Viewer for all users on Russian wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134809 [12:40:26] (03PS1) 10Gilles: Launch Media Viewer for all users on all Wikisources [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134810 [12:40:28] (03PS1) 10Gilles: Launch Media Viewer for all users on German wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134811 [12:40:30] (03PS1) 10Gilles: Launch Media Viewer for all users on English wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134812 [12:41:18] Those are literally atomic commits :D [12:42:45] (03CR) 10Gilles: [C: 04-2] "Scheduled May 29th" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134810 (owner: 10Gilles) [12:43:02] (03CR) 10Gilles: [C: 04-2] "Scheduled June 3rd" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134811 (owner: 10Gilles) [12:43:09] (03CR) 10Gilles: [C: 04-2] "Scheduled June 3rd" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134812 (owner: 10Gilles) [12:45:23] (03Abandoned) 10Gilles: FUTURE: Eighth batch of pilot sites for Media Viewer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/129828 (owner: 10MarkTraceur) [13:11:10] (03CR) 10Ottomata: [C: 032] "Shall I merge?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [13:12:57] ottomata: I was wondering whether that is the good way to use rsync :) [13:13:17] (03CR) 10Hashar: "Merely wondering whether that is how to setup a rsync server :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [13:13:28] I think that's fine, I use it like that for a few things [13:21:28] ottomata: I still have to confirm a few things though. Such as whether connections between labs projects are firewalls [13:21:30] firewalled [13:24:35] aye, you might have to set up a public proxy port to your hosts and use the public hostname [13:25:51] (03PS1) 10Alexandros Kosiaris: WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 [13:27:29] (03CR) 10jenkins-bot: [V: 04-1] WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 (owner: 10Alexandros Kosiaris) [13:28:01] ottomata: ho it is all for internal use :) [13:28:19] oh, just in labs? [13:29:02] hashar: May 6 [13:29:02] 15:43 hashar: rebooting aggregator, it is stalled somehow: There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused [13:29:05] ottomata: I would like to remove the hardcoded hosts_allow line [13:29:10] had that fixed it ? [13:29:14] akosiaris: yup [13:29:19] cool [13:29:24] akosiaris: gotta rebuild that to a stronger instance though [13:29:25] I am doing it again then :-) [13:29:38] the gmond collectors are eating all the cpu iirc [13:30:11] aaaa nice... [13:30:22] hashar, if you remove the hosts_allow line, then anyone who can access the port will be able to sync files to that rsync module [13:30:38] well it should be puppetized so it is just a question of rsync the rrds [13:30:49] ottomata: yeah and I am not sure what is able to access it [13:30:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Thu 22 May 2014 10:30:34 AM UTC [13:31:16] akosiaris: and have to update the collector IP address. I would also like to split the web frontend and aggregator to different instances. But not sure how to achieve that. [13:31:47] (03PS6) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [13:31:55] aren't they already split up ? I see ganglia-web and aggregator [13:32:12] and aggregator1 and aggregator02 but that is a different question [13:32:36] (03CR) 10Hashar: "rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [13:32:57] akosiaris: there is only one instance really in use i think [13:33:05] sigh [13:33:47] if only we could resize a nova instance to give it more CPU ... but that doesn't work (andrew tried) [13:34:42] ganeti ftw :P [13:35:04] (03PS2) 10Alexandros Kosiaris: WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 [13:35:26] no clue what ganeti is :) [13:36:31] (03CR) 10jenkins-bot: [V: 04-1] WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 (owner: 10Alexandros Kosiaris) [13:36:42] Oh I am gonna kill jenkins .... [13:37:31] it is complaining about a file I did not even touch! [13:37:37] fun [13:37:52] https://integration.wikimedia.org/ci/job/operations-puppet-pep8/4116/violations/file/modules/deployment/files/modules/deploy.py/ [13:37:55] seems modules/postgresql/files/ganglia/postgresql.py does not pass pep8 though [13:38:01] honestly... I did not even touch that file [13:38:12] yes it does not but I excluded it for now [13:38:30] plus it is upstream's plugin, not mine [13:39:10] anyway I will test it in labs anyway... [13:39:42] will have a look at it [13:39:56] * hashar blames folks for putting code in puppet [13:40:08] ahahah [13:40:28] be thankful we don't deploy code with puppet [13:40:32] cause I have tried that [13:40:38] and it fails so so so so miserably [13:40:54] which is of course to be expected. But you know. hammer and nails [13:42:15] (03PS3) 10Alexandros Kosiaris: WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 [13:43:41] (03CR) 10jenkins-bot: [V: 04-1] WIP: postgresql ganglia plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/134817 (owner: 10Alexandros Kosiaris) [13:48:38] (03PS7) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [13:49:15] (03CR) 10Hashar: [V: 032] "I removed the hardcoded hosts_allow list entirely. Only labs instance part of the integration project will be able to rsync." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [13:49:36] ottomata: https://gerrit.wikimedia.org/r/#/c/134608/ (rsync for contint) can land in :-) Others labs project can't talk with it [13:49:51] no the pep8 issue [13:50:15] mmmk [13:50:41] (03CR) 10Ottomata: [C: 032 V: 032] contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [13:50:59] done. [13:52:21] akosiaris: so https://gerrit.wikimedia.org/r/#/c/126846 did fail pep8 but got force merged [13:53:12] shame on ottomata :P [13:53:13] ah pfff [13:53:34] not that I wouldn't do the same [13:53:43] that E128 pep8 issue is very very useless [13:54:19] way too strict about over/under indentation [13:54:51] we can ignore it somehow [13:54:53] (03PS1) 10Ottomata: Fixing lines in deploy.py to make pep8 happy [operations/puppet] - 10https://gerrit.wikimedia.org/r/134821 [13:57:13] (03PS1) 10Hashar: pep8: ignore E128 error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134822 [13:58:11] bah that makes things even worth [13:58:42] (03CR) 10jenkins-bot: [V: 04-1] pep8: ignore E128 error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134822 (owner: 10Hashar) [13:59:00] (03Abandoned) 10Hashar: pep8: ignore E128 error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134822 (owner: 10Hashar) [14:00:12] (03CR) 10Hashar: [C: 031] Fixing lines in deploy.py to make pep8 happy [operations/puppet] - 10https://gerrit.wikimedia.org/r/134821 (owner: 10Ottomata) [14:02:00] (03CR) 10Ottomata: [C: 032 V: 032] Fixing lines in deploy.py to make pep8 happy [operations/puppet] - 10https://gerrit.wikimedia.org/r/134821 (owner: 10Ottomata) [14:08:26] (03CR) 10Rush: "Ok so, for now, I don't want to use trebuchet and based on the conversations I have had with other operations folks that is a shared opini" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132505 (owner: 10Dzahn) [14:08:52] (03PS1) 10Giuseppe Lavagetto: puppet: pin packages to the correct version [operations/puppet] - 10https://gerrit.wikimedia.org/r/134823 [14:19:44] (03CR) 10Ryan Lane: [C: 04-1] Improve nginx TLS/SSL settings. (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [14:26:53] ottomata: you set up the client/server aspect of role::puppet::self, right? Can you help me debug a setup? [14:27:11] sure! [14:27:34] So… in theory I should just set role::puppet::self on the server and role::puppet::self + the name of the server labs instance on the client, yes? [14:28:23] I'm going to add you to this project so you can log in… what's your wikitech username? [14:28:32] Reedy: bug 64885 -- is that something we can run before the (re)merge? [14:29:24] yes, in theory [14:29:32] if you want to do a self hosted master with clients [14:29:36] then you configure them all the same [14:29:39] with the puppetmaster's hostname [14:30:03] the master will notice that the configured puppetmaster hostname matches $::hostname (or $::fqdn, actually), and configure itself according [14:30:05] accordingly [14:30:32] oh my wikitech username [14:30:32] is ottomata [14:30:39] wait, do you mean I configure the master with its own name? [14:31:10] !log pushing new swift rings [14:31:15] Logged the message, Master [14:31:15] (03CR) 10Odder: [C: 04-1] "See comment." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134400 (owner: 10Nemo bis) [14:31:23] yes [14:31:32] ottomata: master is 'puppet-localmaster' and client is 'trusty-puppet-client' [14:31:37] andrewbogott: all nodes within a puppet uhhh 'cluster' [14:31:42] shoudl have puppetmaster set to the same fqdn [14:32:13] ottomata: but it looks to me like the role class decides if I am a master or a client based on the presence or absence of the 'puppetmaster' setting... [14:32:23] andrewbogott: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_a_multi-instance_self_hosted_puppetmaster [14:32:29] lemme check [14:33:02] andrewbogott: https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/puppet.pp#L56 [14:33:31] # If localhost or if $server matches this node's [14:33:31] # $fqdn, then this is a puppetmaster. [14:33:31] if ($server == 'localhost' or $server == $::fqdn) { [14:33:31] class { 'puppet::self::master': [14:33:34] oh, you're right. [14:33:39] OK, I'll try this again :) [14:33:41] k [14:35:53] (03CR) 10Anomie: "I think we'll have to do this in two stages: stage 1 would add the new variables but not remove the definition of wgMediaViewerSamplingFac" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [14:36:29] Reedy: Would you mind double-checking my comment on https://gerrit.wikimedia.org/r/#/c/134343/ ? [14:50:54] gi11es: SWAT in 10 minutes. I'm not sure about 134343 as-is, or about 134804 without (the first stage of) 134343. [14:52:05] anomie: it's fine if we don't log events between the two deploy windows [14:52:17] when the config variables are absent, logging is just turned off [14:52:29] that's why I didn't make a backport for 1.24wmf4 [14:52:38] gi11es: Well, if you are fine with losing logging events then we can go for it. [14:53:10] I am, our graphs have been b0rked since last weekend: http://multimedia-metrics.wmflabs.org/dashboards/mmv [14:53:10] gi11es: (please reply to 134343 with that, just for the record) [14:53:13] will do [14:53:30] * anomie likes permanent record of stuff like this [14:54:52] (03CR) 10Gilles: "It's fine if we lose EventLogging tracking for a couple of hours between the deploy windows. That's why I didn't write a backport specific" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [14:55:09] Hi tgr. gi11es is the point person, but SWAT of some patches you were involved with is starting in about 5 minutes in case you wanted to follow along. [14:56:05] which ones? you mean the fix of the one tgr deployed yesterday? [14:57:25] ottomata: looks good now… apparently I just needed to rtfm [14:57:28]