[00:01:24] also makes sense... we would have probably at least gotten a tcp connection, even if the bot dislikes us :P [00:01:58] (03CR) 10Ori.livneh: [C: 04-1] "Change the value in manifests/role/tcpircbot.pp, not the module" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134279 (owner: 10Dzahn) [00:02:03] ok, so did anyone say it should actually work from terbium? [00:02:24] mutante: mh, not sure... the script is where, so I would at least expect it [00:02:28] * there [00:03:02] I was just idling on terbium at that time, that's why I tested there [00:03:11] It think it's only meaningful from tin (eg the deploy server) [00:05:05] include misc::deployment::scap_scripts [00:05:10] why are those on terbium even? [00:06:31] because scap works by sshing into each target nodes and running a script to pull code changes [00:07:08] so there isn't a clear-cut separation of code meant for the deployment server and code meant for the deployment targets [00:07:29] ori: The application servers don't seem to have this class [00:07:34] only tin and terbium have it [00:07:36] AFAIS [00:08:28] The mediawiki::sync class has the files that need to be on the scap target hosts. [00:09:09] yep, but the scap scripts don't need to be on terbium [00:09:29] Not for any reason that I'm aware of, no. [00:10:47] git-blame the line in site.pp [00:11:49] dangerous territory [00:12:54] dates back from fb609118 [00:13:09] doesn't give a hint [00:13:25] sure it does [00:13:41] the ability to set explicitly will also be helpful if the maint [00:13:42] host in the primary site dies. [00:14:25] that refers to the the misc:maint classes [00:14:53] I guess he just copied that over from hume back then [00:16:32] (03PS1) 10Hoo man: Remove misc::deployment::scap_scripts from terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/134282 [00:16:40] yeah, that sounds right [00:19:48] https://www.mail-archive.com/mediawiki-commits@lists.wikimedia.org/msg67233.html [00:20:05] the IPv4-mapped IPv6 doesn't map on tin i suppose [00:20:20] it currently has the ::ffff compat format again [00:21:40] but tin doesn't have a v6 interface added via interface::add_ip6_mapped [00:21:58] so either first add the actual interface to make it mapped [00:22:05] or back to your change making it use v4 [00:22:42] '::ffff:10.64.0.196/128', # tin this isn't it [00:26:05] (03PS1) 10Dzahn: let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 [00:27:35] (03CR) 10Dzahn: "totally true, should have touched the role class only, but looking at that i think the fix is rather Change-Id: Iadd9b978" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134279 (owner: 10Dzahn) [00:27:47] (03Abandoned) 10Dzahn: allow tin and terbium Ipv6 IP to talk to tcpircbot [operations/puppet] - 10https://gerrit.wikimedia.org/r/134279 (owner: 10Dzahn) [00:28:39] (03CR) 10Dzahn: "also see: Change-Id: Iadd9b978" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [00:29:19] (03CR) 10Dzahn: "also see: Id688a26dbf5c71" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [00:29:53] ori: gah, i forgot 100% of that [00:30:07] we even reverted that with "Connection from ('::ffff:10.64.0.196' matches tin and is what is trying to connect in /var/log/upstart/tcpircbot-logmsgbot.log" [00:30:26] i'd say the root cause is that the v6 address is not mapped [00:30:33] but gotta run for now [00:33:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [00:43:27] (03CR) 10Hoo man: "bump" [operations/puppet] - 10https://gerrit.wikimedia.org/r/126027 (owner: 10Hoo man) [00:49:14] (03PS3) 10Withoutaname: Reduce string URLs to defined constant [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131914 (https://bugzilla.wikimedia.org/48618) [01:00:35] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon May 19 22:00:21 2014 [01:25:05] PROBLEM - MySQL Processlist on db1064 is CRITICAL: CRIT 86 unauthenticated, 0 locked, 0 copy to table, 0 statistics [01:26:05] RECOVERY - MySQL Processlist on db1064 is OK: OK 27 unauthenticated, 0 locked, 0 copy to table, 0 statistics [01:51:07] anyone happen to know why a patch would not have made it to the deploy schedule? expected patch https://gerrit.wikimedia.org/r/#/c/127839/ to be part of https://www.mediawiki.org/wiki/MediaWiki_1.24/wmf5#GWToolset ... [02:05:28] dan-nl: it should be, gerrit thinks it is [02:06:15] hmm, if something is on testwiki today but not enwiki, it should be on enwiki on thursday, right? [02:06:40] dan-nl: so does gitblit: http://git.wikimedia.org/log/mediawiki%2Fextensions%2FGWToolset.git/refs%2Fheads%2Fwmf%2F1.24wmf5 [02:06:48] YuviPanda: right [02:06:59] cool, ty grrrit-wm [02:07:02] gah [02:07:03] greg-g: [02:07:05] :) [02:07:19] dan-nl: apparently the script that generates that wikipage is finicky [02:07:22] * YuviPanda remembers to never write a bot with name yuppybot [02:07:37] thanks for checking, just didn't see it on https://www.mediawiki.org/wiki/MediaWiki_1.24/wmf5#GWToolset. where would i see it in gerrit? [02:07:54] dan-nl: click on the "Included In" arrow on that gerrit change [02:08:05] it'll take a second, but it'll show all branches that it thinks the change is in [02:08:22] ah, cool, thanks greg-g [02:09:46] np :) [02:13:25] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3789 MB (3% inode=99%): [02:20:36] !log LocalisationUpdate completed (1.24wmf4) at 2014-05-20 02:19:33+00:00 [02:20:43] Logged the message, Master [02:21:25] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3433 MB (3% inode=99%): [02:30:15] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Tue May 20 02:30:07 UTC 2014 [02:34:34] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-20 02:33:31+00:00 [02:34:39] Logged the message, Master [03:00:25] RECOVERY - Disk space on virt0 is OK: DISK OK [03:31:17] Coren: I'm considering writing a tool to make handling Oversight requests that sucks less than OTRS. Think more AFTish, rather than EmailUser. [03:31:30] Coren: is this something to host on wmflabs, or... [03:32:16] lfaraone: It's a reasonable place to host it, but you'll have to be careful about logging and permissions. [03:32:36] lfaraone: We'd sure as hell rather you host it here than on some random third party hosting. :-) [03:33:38] lfaraone: Also there are significant privacy issues, so you'll need to (a) make sure only oversighters have access and (b) authenticate properly. [03:34:06] speaking of permissions… is there a bugzilla bug for "please enable 2FA on enwp and friends and make it mandatory for functionaries"? :P [03:34:41] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [03:38:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue May 20 03:37:32 UTC 2014 (duration 37m 31s) [03:38:42] Logged the message, Master [03:54:52] lfaraone: What's wrong with OTRS? [03:56:12] Gloria: its mostly a workflow problem for us; we often miss requests, users are surprisingly bad at identifying the actual items to be Oversighted, its hard to escalate to "discuss with the Oversight team" when an individual OS looks at a ticket and goes "uhm, not sure" [04:08:27] lfaraone: Perhaps the Oversight extension needs love? [04:08:45] It could probably better integrate with Flow and Echo. [04:11:41] Gloria: sure. the problem is I really don't want to learn how to PHP :) [04:12:24] Well, Oversight already has a supported and approved authentication scheme. [04:12:31] And it's already live in production and has been for some time. [04:12:43] So it's likely a much better starting place than a brand new tool. [04:22:03] you can do exciting things with OAuth these days [04:22:25] plus, isn't suppression MW core these days? Not sure a request queue is really in scope for that. [04:26:48] (03CR) 10Ori.livneh: [C: 031] Remove misc::deployment::scap_scripts from terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/134282 (owner: 10Hoo man) [04:27:09] (03CR) 10Ori.livneh: "should be accompanied by the removal of the scripts. this won't actually purge them." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134282 (owner: 10Hoo man) [06:00:37] (03PS1) 10Giuseppe Lavagetto: icinga: stop pages from 5xx alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134301 [06:02:36] (03CR) 10Giuseppe Lavagetto: [C: 032] icinga: stop pages from 5xx alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134301 (owner: 10Giuseppe Lavagetto) [06:07:14] morning [06:07:32] * springle waves [06:07:37] _joe_: isn't it like 8am over there ? [06:07:45] springle: I won't even ask ... [06:07:47] <_joe_> akosiaris: it is [06:07:58] <_joe_> akosiaris: yesterday I started at around 7 am [06:08:24] <_joe_> I am moving and I try to begin early to be free in the afternoon [06:08:55] ok that makes sense. As long as it does not become a habbit :P [06:09:02] <_joe_> also, I'm at my new flat waiting for Telecom Italia to come and connect my DSL, as they 'forgot' [06:09:22] pfff DSL... who needs DSL ? you got fibers over there :P [06:09:29] you didn't bribe the right people? [06:09:36] <_joe_> akosiaris: it will be upgraded [06:09:47] springle: ahahah :-) [06:09:48] <_joe_> springle: no, it's not like that. this is Telecom [06:10:07] <_joe_> no way to make them work correctly, no matter how much you bribe someone [06:10:13] you always need to bribe [06:10:37] IIRC Italy had a company that had FTTH (but the rest of the network was...) [06:11:01] <_joe_> yesterday the technician called me 5 (f i v e) times and every call started with "so... can you explain me what your problem is?" [06:11:15] I even remember having a public IP at my friend's home and the next hop being a 10.x.x.x [06:11:30] or maybe the next-next hop [06:11:35] <_joe_> akosiaris: my DSL will be upgraded to fiber [06:11:49] i tried to ignore it the first time you said it [06:11:55] THAT IS UNFAIR!!! [06:11:56] <_joe_> akosiaris: did I tell you there is *no* ipv6 offering for businesses here? [06:12:03] <_joe_> no serious one at least [06:12:25] I think we got that now [06:12:49] some ISPs do provide IPv6 [06:12:55] <_joe_> akosiaris: it will be a shitty fiber with the Iran-style internet filters and the ransom-based peering italian ISPs are famous for :) [06:13:00] some in pilot, some not in pilot [06:13:05] <_joe_> so no reason to hate me :) [06:13:48] heh. Still it gives you hope [06:13:52] <_joe_> oh yes telecom has an "ipv6 pilot", a friend who worked as a consultant setting it up told me it's one router in a basement :P [06:14:09] internet filters are only one court ruling away from being called illegal [06:14:20] <_joe_> in greece? [06:14:23] well a "final" court ruling [06:14:33] <_joe_> in Italy they're all loved by almost everyone [06:14:38] ah, in greece that translates to around 12-15 years [06:14:45] <_joe_> ahhh ok [06:14:56] <_joe_> akosiaris: they should unify greece and italy, it seems [06:14:58] <_joe_> :P [06:15:02] ahaha [06:15:17] <_joe_> btw, I was amazed by how clean Athens is compared to Rome [06:15:41] is it ? [06:15:43] <_joe_> and no, this is not a compliment for Athens :) [06:15:51] ah, ok [06:15:54] <_joe_> akosiaris: it is, Rome has become filthy [06:16:09] <_joe_> well it's not-so-dirty anyway [06:16:40] <_joe_> it seems you don't consider the public streets as your private litter/ashtray as much as we do [06:17:02] (03CR) 10Alexandros Kosiaris: [C: 032] let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 (owner: 10Dzahn) [06:18:05] somehow I think you got a skewed version of reality while you were here. Good, we managed to hide the truth well! [06:19:21] <_joe_> eheh [06:19:52] (03CR) 10Alexandros Kosiaris: [C: 04-2] "Let's fix whatever problem IPv6 has instead of working around it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134277 (owner: 10Hoo man) [06:22:09] akosiaris: i saw some reality last time. this trip was much better :) you local guides did well [06:22:42] :-) :-) :-) [06:25:40] <_joe_> akosiaris: athens looks a lot like big sicilian cities, btw [06:25:54] <_joe_> only, slightly less chaotic [06:26:14] <_joe_> (yes, they *are* that chaotic) [06:29:55] hmmm did we upgrade RT or something ? [06:30:08] I was logged out and when I logged in again [06:30:19] it started displaying everything in greek [06:30:33] which is nice and all... but still ... [06:31:00] oh my god I know what it is ... my 1 year cookie just expired... [06:31:44] i like the new greek enabled RT better :P [06:35:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [06:40:02] <_joe_> lol [06:40:24] <_joe_> akosiaris: so it's 1 year today? [06:40:32] <_joe_> linkedin told me, in fact [06:41:14] yup. one year and 1 week to be exact [06:55:26] (03CR) 10Alexandros Kosiaris: [C: 032] puppet:self::master use newer package names [operations/puppet] - 10https://gerrit.wikimedia.org/r/134143 (owner: 10Alexandros Kosiaris) [07:04:18] akosiaris: firewall question please ? [07:04:54] matanya: sure [07:05:16] (03CR) 10Alexandros Kosiaris: [C: 032] Avoid connection tracking for DNS recursors [operations/puppet] - 10https://gerrit.wikimedia.org/r/134071 (owner: 10Alexandros Kosiaris) [07:05:56] looking at hafnium.wikimedia.org - it has role::eventlogging::graphite which has input and output, does those ports need to be open too ? [07:10:42] matanya: not sure where the too goes but in this specific case, neither [07:11:14] the software works by connecting to input, consuming and then connecting to output and writing them [07:11:42] not the best description but the point I want to make is that both connections are outgoing and not incoming [07:12:12] so no listening ports on hafnium due to that role [07:12:26] akosiaris: a suggestion to make it simpler: can get a list of all iptables current rules for the host listed in the etherpad ? i'm just guessing in the fog now [07:14:21] hafnium ? It has no rules right now if this is what you are asking [07:15:36] i was asking about all hosts. [07:15:57] <_joe_> matanya: man iptables [07:16:03] <_joe_> that usually tend to work [07:16:05] <_joe_> :) [07:16:06] ? [07:16:44] <_joe_> "can get a list of all iptables current rules" -> as in, you need someone to run iptables in prod for you? [07:16:53] yes [07:17:18] <_joe_> ok did not get that, it seemed you were asking how to get the rules :) [07:17:31] <_joe_> matanya: I have absolutely no time for that, sorry [07:17:40] I am not sure I understand what that would accomplish [07:18:09] i can look at what has allow in the rules and transform that into ferm rules [07:18:22] <_joe_> akosiaris: having a list of rules already in place, wich given we're moving from default-> all open to default -> all dropped does not serve its purpose [07:18:58] _joe_: my point exactly [07:19:19] matanya: so the answer is easy. Do not assume any rules exist for any machine unless puppet says so [07:19:33] good answer [07:20:21] so my next question would be role::webperf - is that incoming or out going ? [07:22:01] outgoing [07:22:35] so just apply ferm on the host, no rules needed :) [07:22:51] for example if you follow the path from role::webperf you will probably end up to ve.py for example [07:22:58] modules/webperf/files/ve.py [07:23:15] which clearly has sock.sendto calls inside [07:23:56] so it is sending udp packets [07:24:04] and not receiving [07:24:18] hence outgoing [07:25:10] thank you, too tired to think about stuff more complicated than "include something" :P [07:25:20] so to verify your sentence, yes no rules needed, just apply ferm [07:26:44] (03PS1) 10Matanya: hafnium: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/134304 [07:39:54] (03PS1) 10Giuseppe Lavagetto: compare-puppet-catalogs: bugfixes [operations/software] - 10https://gerrit.wikimedia.org/r/134305 [07:39:57] (03CR) 10jenkins-bot: [V: 04-1] compare-puppet-catalogs: bugfixes [operations/software] - 10https://gerrit.wikimedia.org/r/134305 (owner: 10Giuseppe Lavagetto) [07:51:30] (03PS2) 10Giuseppe Lavagetto: compare-puppet-catalogs: bugfixes [operations/software] - 10https://gerrit.wikimedia.org/r/134305 [08:00:57] (03PS2) 10Giuseppe Lavagetto: erbium: fix template variable scoping [operations/puppet] - 10https://gerrit.wikimedia.org/r/134060 [08:12:27] <_joe_> Oh snap, the old catalog differ did not take into account missing resources [08:12:43] <_joe_> the new one does otoh [08:12:55] <_joe_> so, finding new issues [08:12:57] <_joe_> argh [09:02:16] (03CR) 10Filippo Giunchedi: [C: 031] "I'll merge this on Wed if no objections arise" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134258 (owner: 10Dzahn) [09:02:51] (03PS1) 10Ottomata: Allowing root and hdfs as well as hue to submit oozie -doas commands as any user [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/134315 [09:03:26] (03CR) 10Ottomata: [C: 032 V: 032] Allowing root and hdfs as well as hue to submit oozie -doas commands as any user [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/134315 (owner: 10Ottomata) [09:04:09] (03PS1) 10Ottomata: Updating cdh4 module with oozie -doas change [operations/puppet] - 10https://gerrit.wikimedia.org/r/134317 [09:04:18] (03PS2) 10Ottomata: Updating cdh4 module with oozie -doas change [operations/puppet] - 10https://gerrit.wikimedia.org/r/134317 [09:04:26] (03CR) 10Ottomata: [C: 032 V: 032] Updating cdh4 module with oozie -doas change [operations/puppet] - 10https://gerrit.wikimedia.org/r/134317 (owner: 10Ottomata) [09:04:53] (03CR) 10Filippo Giunchedi: [C: 031] "to be merged on Wed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134251 (owner: 10Dzahn) [09:08:56] <_joe_> ottomata: you may want to take a look at http://puppet-compiler.wmflabs.org/html/analytics1027.eqiad.wmnet.html [09:09:15] <_joe_> it's not clear to me if those number changes around is a problem [09:11:41] um, first of all [09:11:43] THAT IS AWESOME [09:11:45] 2nd of all [09:12:15] random nums diff [09:12:18] oo, brb... [09:13:44] <_joe_> ok np :) [09:18:33] hm yea so hm [09:19:06] ah, hm, _joe_, its ok [09:19:28] that will be different every time, but will only run if something is missing in hdfs [09:19:40] <_joe_> ottomata: ok that's fair [09:19:48] actually, i should change that to use tempfile [09:19:57] <_joe_> I figured that, but I wanted a confirmation [09:20:00] $oozie_sharelib_tmpdir = inline_template('/tmp/oozie_sharelib_install.<%= rand() %>') [09:20:07] pretty silly now that I'm looking at it [09:20:52] _joe_, so, i tried to use puppet comparator for this [09:20:52] https://gerrit.wikimedia.org/r/#/c/133695/ [09:21:00] but it was kinda hard, because it was a submodule change [09:21:09] <_joe_> oh, god [09:21:11] <_joe_> :) [09:21:14] hehe [09:21:16] i copied down a catalog from palladium [09:21:25] then manually updated the submodule the vm's checkuot [09:21:27] <_joe_> ottomata: I think it should handle it [09:21:29] <_joe_> let me check [09:21:42] <_joe_> which nodes should this affect? [09:21:49] cp1052 is a good one [09:21:53] it shouldn't change anything [09:21:58] but that's the one i was testing with [09:22:47] <_joe_> ottomata: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/lastBuild/ [09:23:05] ? [09:23:15] <_joe_> ottomata: it's in jenkins now! [09:23:23] coool! [09:23:27] <_joe_> you can trigger builds of the compiler from jenkins [09:23:27] but wait, this change hasn't been merged... [09:23:29] ohhh [09:23:30] <_joe_> it's not finished [09:23:32] hm [09:23:43] <_joe_> yes, I do fetch the change from gerrit [09:23:57] hm, yeah, probably it would work when ops/puppet actually changes the sha that the submodule points to [09:24:00] but, this is just the submodule commit [09:24:04] so nothing has changed in ops/puppet yet [09:24:15] <_joe_> oh, ok [09:24:34] <_joe_> right. [09:24:44] <_joe_> yeah, no way to test submodules ATM [09:24:55] <_joe_> create me a ticket, I'll add that ability :) [09:25:16] <_joe_> it will require some time and is not high on my priority list now [09:25:32] <_joe_> for now, just create a dependent change on operations/puppet and test that [09:26:50] yeah, but i can't until the submodule change is merged! [09:27:14] <_joe_> fair enough. [09:27:24] <_joe_> ottomata: yeah I don't have an instant solution for that [09:28:04] <_joe_> we should add command-line switches for this, also we would need to code a little around [09:30:55] (03PS3) 10Giuseppe Lavagetto: erbium: fix template variable scoping [operations/puppet] - 10https://gerrit.wikimedia.org/r/134060 [09:31:09] (03CR) 10Giuseppe Lavagetto: [C: 032] erbium: fix template variable scoping [operations/puppet] - 10https://gerrit.wikimedia.org/r/134060 (owner: 10Giuseppe Lavagetto) [09:31:29] ok well, at least the submodule won't go live in prod until we make a specific merge for it [09:31:35] just gotta get a few folks to review this first [09:36:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [09:40:56] <_joe_> ottomata: I have a trick for you [09:41:51] (03CR) 10Hashar: [C: 04-1] "I have already packaged a statsd pure python module. It is available in apt.wikimedia.org and being used by Zuul." [operations/debs/python-statsd] - 10https://gerrit.wikimedia.org/r/131449 (owner: 10Gage) [09:42:44] <_joe_> ottomata: oh sorry, nevermind, it won't [09:43:02] ha [09:43:06] well, i mean, i think i got it to work locally [09:43:14] by manually checking out that patch inthe submodule [09:43:21] and then running comparator [09:43:28] although..i think i had to mess with the python cod emaybe [09:43:38] to get it to compare just 2.7 [09:44:12] hm, i ust did +PUPPET_VERSIONS = [('2.7', 'production')] [09:44:17] because I didn't want to compare with 3.0 now [09:45:43] <_joe_> ottomata: no that is wrong [09:46:03] oh? [09:46:13] (03CR) 10Hashar: "Forgot to add, adding this package with the name python-statsd will conflict with the version Zuul is depending upon and break CI :-(" [operations/debs/python-statsd] - 10https://gerrit.wikimedia.org/r/131449 (owner: 10Gage) [09:46:15] <_joe_> ottomata: do you have the latest version of the code? [09:46:23] hmm, as of late last week [09:46:27] haven't tried this wek [09:46:28] week [09:46:32] will pull now [09:46:55] oo many changes ) [09:46:56] :) [09:47:11] <_joe_> ottomata: yeah sorry, you will also need to reprovision vagrant [09:47:30] ah, s'ok np [09:47:32] <_joe_> as they (as in puppetlabs) changed the catalog differ in an incompatible way [09:47:46] <_joe_> gotta love them [09:52:21] <_joe_> ottomata: working on your needs now [09:52:32] haha :) [09:52:41] <_joe_> it's gonna be kind of a workaround for now [09:52:54] <_joe_> we'll add proper submodule support later. [10:15:10] (03PS3) 10Giuseppe Lavagetto: compare-puppet-catalogs: bugfixes, jenkins integration [operations/software] - 10https://gerrit.wikimedia.org/r/134305 [10:22:26] (03CR) 10Giuseppe Lavagetto: [C: 032] compare-puppet-catalogs: bugfixes, jenkins integration [operations/software] - 10https://gerrit.wikimedia.org/r/134305 (owner: 10Giuseppe Lavagetto) [10:45:43] (03CR) 10Mark Bergsma: "One minor comment. Feel free to merge after you've resolved that." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/132495 (owner: 10Rush) [10:48:06] (03PS1) 10Giuseppe Lavagetto: creation of role::puppet_compiler [operations/puppet] - 10https://gerrit.wikimedia.org/r/134327 [10:50:25] if only that meant we were going to natively compiled puppet code ;-) [10:50:52] <_joe_> mark: he. [10:51:21] <_joe_> mark: on a positive note, the new puppet compiler showed clearly some differences we were not shown before [10:51:33] mark: i saw you said something about firewall yesterday, are you not pleased with firewalling some of the hosts other than lvs ? [10:53:36] godog: https://rt.wikimedia.org/Ticket/Display.html?id=6845 does this mean i should mail each and every one of them ? [10:55:33] (03PS1) 10Yuvipanda: tools: Tune nginx to handle higher load [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 [10:55:39] hello! [10:55:43] anyone who can merge ^? [10:55:48] tools proxy is having perf issues [10:55:51] these should help [10:55:56] am going to test them live now [10:58:13] <_joe_> YuviPanda: reviewing [10:59:05] _joe_: actually no, test failed. fixing [10:59:26] <_joe_> YuviPanda: I said reviewing, not merging :) [10:59:31] ah :) [11:00:11] hmm, it's a bit weird. I did put them in the right context but nginx tells me they aren't [11:00:36] matanya: if that's an updated list of people (and they still require prod access) then yes they should be poked. I think an email will be fine [11:04:14] ah, hmm [11:19:16] (03PS2) 10Yuvipanda: tools: Tune nginx to handle higher load [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 [11:20:12] _joe_: better now ^ [11:20:13] I should eventually just use the nginx module in ops/puppet, but that's a fair chunk of work [11:21:54] (03CR) 10Giuseppe Lavagetto: [C: 04-1] tools: Tune nginx to handle higher load (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 (owner: 10Yuvipanda) [11:22:11] <_joe_> YuviPanda: see my comment, apart from that it seems fair [11:22:24] <_joe_> still need to review the syntax of the whole file [11:22:56] _joe_: I just picked a randomly high number. I know that this is causing issues because I keep seeing errors about not enough workers in the error logs. [11:22:59] _joe_: let me look into it [11:23:40] <_joe_> YuviPanda: ok that is a little high, but my main comment was about its effectiveness [11:23:54] _joe_: yeah, true. let me both reduce the number and look into the upstart file [11:24:28] _joe_: hmm, I don't think nginx comes with an upstart script. I just see a normal init script. [11:24:47] <_joe_> oh ok sorry I assumed it was ported to upstart [11:27:38] <_joe_> YuviPanda: then it's /etc/security/limits.conf [11:27:48] <_joe_> sorry gotta go, lunch! [11:28:25] _joe_: I'll poke around. Thanks for the review! [12:02:52] (03CR) 10TTO: "Although this is a true constant unlike $stdlogo and friends, I wonder if it would be neater to do this via CommonSettings.php line 210ff?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131914 (https://bugzilla.wikimedia.org/48618) (owner: 10Withoutaname) [12:02:55] (03PS1) 10Giuseppe Lavagetto: compare-puppet-catalogs: jenkins integration [operations/software] - 10https://gerrit.wikimedia.org/r/134335 [12:04:32] (03CR) 10Giuseppe Lavagetto: [C: 032] compare-puppet-catalogs: jenkins integration [operations/software] - 10https://gerrit.wikimedia.org/r/134335 (owner: 10Giuseppe Lavagetto) [12:04:36] (03CR) 10TTO: "This is surely not redundant. If you want to deny a permission, don't you use $wgRevokePermissions?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134155 (owner: 10Nemo bis) [12:05:14] (03CR) 10TTO: [C: 031] Disable query pages for closed wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) (owner: 10Withoutaname) [12:05:53] (03CR) 10TTO: [C: 04-1] Disable query pages for closed wikis (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) (owner: 10Withoutaname) [12:08:33] (03CR) 10Giuseppe Lavagetto: [C: 032] "This does not affect any other role." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134327 (owner: 10Giuseppe Lavagetto) [12:11:06] (03PS3) 10Yuvipanda: tools: Tune nginx to handle higher load [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 [12:11:08] _joe_: updated [12:11:39] _joe_: /etc/security/limits.conf doesn't actually have any content by default, so this should be safe. [12:11:45] and I didn't find a limits module in our puppet [12:22:00] (03Abandoned) 10Nemo bis: Remove plwiki permission for 'autoconfirmed' denied for 'user' [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134155 (owner: 10Nemo bis) [12:27:35] (03PS1) 10Gilles: Introduce finer-grained Media Viewer EventLogging sampling [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 [12:28:00] (03CR) 10Gilles: [C: 04-2] "Depends on https://gerrit.wikimedia.org/r/134064" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [12:37:35] PROBLEM - Puppet freshness on maerlant is CRITICAL: Last successful Puppet run was Fri May 16 06:03:33 2014 [12:41:36] (03CR) 10Andrew Bogott: tools: Tune nginx to handle higher load (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 (owner: 10Yuvipanda) [12:42:22] (03PS4) 10Yuvipanda: tools: Tune nginx to handle higher load [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 [12:42:27] andrewbogott: fixed [12:43:35] (03CR) 10Andrew Bogott: [C: 032] tools: Tune nginx to handle higher load [operations/puppet] - 10https://gerrit.wikimedia.org/r/134328 (owner: 10Yuvipanda) [12:46:36] (03CR) 10Hashar: [C: 031] adding .gitreview [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/95144 (owner: 10AzaToth) [12:51:10] (03PS1) 10Giuseppe Lavagetto: role::puppet_compiler - lint and fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/134349 [12:52:52] (03PS2) 10Ottomata: adding .gitreview [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/95144 (owner: 10AzaToth) [12:53:04] (03CR) 10Ottomata: [C: 032 V: 032] adding .gitreview [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/95144 (owner: 10AzaToth) [12:56:41] (03PS2) 10Giuseppe Lavagetto: role::puppet_compiler - lint and fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/134349 [12:56:52] (03CR) 10Giuseppe Lavagetto: [C: 032] role::puppet_compiler - lint and fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/134349 (owner: 10Giuseppe Lavagetto) [13:20:41] !log git-deploy: Deploying integration/slave-scripts I4a4e2a4c90fb6 [13:20:46] Logged the message, Master [13:47:02] (03PS1) 10Giuseppe Lavagetto: puppet_compiler: easier jenkins launch with parameters [operations/puppet] - 10https://gerrit.wikimedia.org/r/134354 [13:48:44] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet_compiler: easier jenkins launch with parameters [operations/puppet] - 10https://gerrit.wikimedia.org/r/134354 (owner: 10Giuseppe Lavagetto) [13:51:00] (03PS2) 10Andrew Bogott: Renumber many users to match labs: [operations/puppet] - 10https://gerrit.wikimedia.org/r/133968 [13:55:08] hellp anskar [13:55:11] *o [13:55:19] hello [13:55:39] ops, anskar is here to get help with an rc bot flooding [13:56:11] :) thanks matanya [13:56:54] May 17 15:44:54 [13:24:34] Greetings, can someone help me with a bot in #da.wikipedia ? - It keeps getting "Excess Flood", so that mean it keep joining and leaveing. :) [13:56:54] May 17 15:44:54 [13:26:38] There's nothing we can do about it [13:56:54] May 17 15:44:54 [13:26:55] Oka, have a nice day then. [13:57:26] Reedy, that wasn't the most helpful of answers... Why can't anyone fix it? This sounds like a normal IRC ops task. Who can oper on irc.wm.o? [13:57:46] the bot now is wikichanges-bf5a6c68-7bd8-4fcf-b331-2912f3fb013a (~nodebot@anonymous.user) [13:57:59] * matanya pokes _joe_ ^ [13:58:10] but change nick [13:58:31] it's possible block wikichanges-*-*-*-*-*!~nodebot@anonymous.user ? [14:05:58] (03CR) 10Andrew Bogott: [C: 032] Renumber many users to match labs: [operations/puppet] - 10https://gerrit.wikimedia.org/r/133968 (owner: 10Andrew Bogott) [14:07:34] (03PS4) 10Rush: ircd-ratbox and udpmxircecho puppetized [operations/puppet] - 10https://gerrit.wikimedia.org/r/132495 [14:08:24] (03PS5) 10Rush: ircd-ratbox and udpmxircecho puppetized [operations/puppet] - 10https://gerrit.wikimedia.org/r/132495 [14:09:17] (03CR) 10Rush: [C: 032 V: 032] "fixed the last outlier." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132495 (owner: 10Rush) [14:26:11] (03PS1) 10Rush: wm-rc-irc echo and not relay typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/134358 [14:27:10] (03CR) 10Rush: [C: 032 V: 032] "typo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134358 (owner: 10Rush) [14:27:35] (03PS6) 10Hashar: Describe Math related packages in a class [operations/puppet] - 10https://gerrit.wikimedia.org/r/115133 (https://bugzilla.wikimedia.org/61090) [14:33:22] (03PS1) 10Giuseppe Lavagetto: puppet_compiler: add naggen installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/134359 [14:34:36] (03CR) 10jenkins-bot: [V: 04-1] puppet_compiler: add naggen installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/134359 (owner: 10Giuseppe Lavagetto) [14:37:18] (03PS2) 10Giuseppe Lavagetto: puppet_compiler: add naggen installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/134359 [14:39:11] (03PS3) 10Giuseppe Lavagetto: puppet_compiler: add naggen installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/134359 [14:41:14] (03CR) 10Giuseppe Lavagetto: [C: 032] "Adding forgotten naggen installation." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134359 (owner: 10Giuseppe Lavagetto) [14:43:05] PROBLEM - RAID on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:15] PROBLEM - Disk space on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:15] PROBLEM - puppet disabled on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:25] PROBLEM - check if dhclient is running on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:35] PROBLEM - twemproxy port on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:35] PROBLEM - DPKG on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:44:25] PROBLEM - check configured eth on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:44:25] PROBLEM - twemproxy process on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:44:32] likely down for disk replacement [14:45:18] PROBLEM - RAID on ms-be3003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:18] cmjohnson1: ^ ? [14:45:39] godog: looking [14:45:56] cmjohnson1: oh ok, just curious if you were replacing mw1151's disk [14:47:35] !log running 'find' commands on many hosts to chown files for users with new UIDs. [14:47:40] Logged the message, Master [14:47:55] godog: no I am not. working on mw1163 for the millionth time though ...different rack [14:47:56] RECOVERY - RAID on ms-be3003 is OK: OK: optimal, 12 logical, 12 physical [14:50:20] cmjohnson1: kk, thanks [14:51:05] PROBLEM - SSH on mw1151 is CRITICAL: Server answer: [14:52:25] RECOVERY - twemproxy port on mw1151 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [14:52:35] RECOVERY - DPKG on mw1151 is OK: All packages OK [14:53:05] RECOVERY - SSH on mw1151 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [14:53:15] RECOVERY - Disk space on mw1151 is OK: DISK OK [14:53:15] RECOVERY - puppet disabled on mw1151 is OK: OK [14:53:32] ok so ms-be3003 hiccup might have been andrewbogott's find, I found sdk on it was complaining though [14:53:56] RECOVERY - RAID on mw1151 is OK: OK: no RAID installed [14:54:01] <_joe_> godog: if swat is running, it will kill mw1151 as it has a disk failure [14:54:09] <_joe_> so, expected [14:54:12] godog: I didn't run the find on the ms-be boxes [14:54:19] or… at least I tried not to [14:54:25] salt -E '^(.(?!ms-be|labstore|snapshot))*$' [14:54:51] mh there are several of these [14:54:52] root 9804 0.0 0.0 4404 612 ? S 14:41 0:00 \_ /bin/sh -c find / -user 605 -print0 | xargs -0 chown -h 2454 [14:55:11] <_joe_> we may decide to shut it down and put it in scheduled downtime [14:55:15] RECOVERY - check configured eth on mw1151 is OK: NRPE: Unable to read output [14:55:17] dammit! [14:55:25] I tested that regext to death [14:55:31] godog: feel free to kill that [14:55:35] PROBLEM - twemproxy port on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:55:35] PROBLEM - DPKG on mw1151 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:56:15] RECOVERY - twemproxy process on mw1151 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [14:56:15] RECOVERY - check if dhclient is running on mw1151 is OK: PROCS OK: 0 processes with command name dhclient [14:56:19] andrewbogott: can salt do that? likely it is running elsewhere too? [14:56:25] RECOVERY - twemproxy port on mw1151 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [14:56:25] RECOVERY - DPKG on mw1151 is OK: All packages OK [14:56:37] godog: I can 'killall find'... [14:56:44] <_joe_> godog: if it's running on mw1151 [14:56:48] <_joe_> andrewbogott: no! [14:57:05] <_joe_> your finds are not the only one running at a given moment [14:57:11] _joe_: yeah, that's what I figured :) [14:57:28] lemme see what I can do... [14:59:39] _joe_: can you suggest a way for me to selectively kill jobs like '/bin/sh -c find / -user 603 -print0 | xargs -0 chown -h 2400' [14:59:41] ? [14:59:56] (03PS1) 10Giuseppe Lavagetto: puppet_compiler: fix user in differ [operations/puppet] - 10https://gerrit.wikimedia.org/r/134367 [15:00:15] <_joe_> andrewbogott: not really... some ugly grep on ps -ef maybe [15:00:31] <_joe_> killall is unreliable IMO [15:00:56] andrewbogott: I suspect salt doesn't write anywhere its minion's pids? [15:01:22] godog: I don't know. I don't think so [15:02:03] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet_compiler: fix user in differ [operations/puppet] - 10https://gerrit.wikimedia.org/r/134367 (owner: 10Giuseppe Lavagetto) [15:02:06] (03CR) 10Physikerwelt: [C: 031] Describe Math related packages in a class [operations/puppet] - 10https://gerrit.wikimedia.org/r/115133 (https://bugzilla.wikimedia.org/61090) (owner: 10Hashar) [15:03:16] (03PS1) 10Cmjohnson: Adding dns entries for test HP servers ...called it hp1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/134369 [15:03:38] trivia: salt-minion won't create a new process group, so its process group is its father's, so killing all the related process groups will do the trick [15:05:02] godog: so you mean if I kill /usr/bin/salt-minion... [15:05:10] that seems about as bad as killing all 'find' jobs [15:05:59] andrewbogott: no, collecting all salt-minion processes that have spawned find and kill their process group (i.e. likely the pid-1) [15:06:20] godog: i assigned a bunch of RT to you, don't be shocked.. it doesn't mean you have to resolve all of them and they should be more like just following up with people if things can be closed or not yet [15:06:32] godog: and it also means "please ask me" about any of them [15:06:40] if any questions [15:06:47] mutante: yup, will do! [15:07:10] godog: ok… I still don't follow. How would I 'collect all salt-mion processes that have...' [15:10:31] andrewbogott: kill $(ps aux | grep find | awk '{print $2}') ? [15:10:37] andrewbogott: heh, good question [15:11:44] or something better than "grep find", but the print $2 gets you the PIDs [15:15:34] !log running salt "ms-be*" cmd.run "kill $(ps aux | grep 'find / -user' | awk '{print $2}')" to kill runaway 'finds' on swifts [15:15:36] (03CR) 10Cmjohnson: [C: 032] Adding dns entries for test HP servers ...called it hp1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/134369 (owner: 10Cmjohnson) [15:15:40] Logged the message, Master [15:17:25] well, that didn't work at all [15:19:39] andrewbogott: try to find "saltutil.killjob" [15:19:59] http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html [15:20:15] * andrewbogott is in quoting hell [15:20:20] ah [15:20:27] <_joe_> ok, done. puppet-compiler can be run via jenkins now [15:20:42] \o/ ! [15:20:45] !:)) [15:21:21] <_joe_> https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/13/console for a live stream of compilation of 'production' in puppet 2.7 vs 3 for a smart subset of all nodes [15:21:48] <_joe_> it runs one job at a time, so don't expect to be able to use it in the near future [15:21:52] <_joe_> :) [15:22:12] _joe_: does it needs rights to be run for changes ? [15:24:57] <_joe_> matanya: no [15:25:05] <_joe_> matanya: not that I'm aware of [15:25:21] <_joe_> but it's limited to 1 concurrent run [15:25:21] (03PS2) 10Dzahn: let tin have a proper IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/134284 [15:25:31] <_joe_> so for now any new job will be queued [15:25:36] yes, i don't have rights to run it [15:25:54] <_joe_> matanya: we should speak with hashar about this :) [15:26:06] i guess with greg-g [15:26:10] <_joe_> I am deep down the $mariadb dynamic lookup hole [15:26:14] based on membership in ops LDAP group? [15:26:18] nda issues and the like [15:26:30] <_joe_> matanya: in case just ask me :( [15:26:35] matanya: it comes up at the meetings though...:) [15:26:40] it did again yesterday [15:27:00] i'm blocked in so many ways :) [15:27:30] i'll push a patch to ldap: group : volunteer ou : wikimedia [15:27:45] matanya: Scott Lee .. do you know him? [15:28:03] he is trying to contribute [15:28:26] known by irc nick pancake so something similar [15:28:55] didn't see any conrib yet [15:29:15] don't talk to me about NDAs :P [15:29:27]