[00:00:32] <^d> Any roots around who mind doing a few minutes of spelunking for me? [00:00:53] <^d> (It's easy, I promise) [00:01:06] (03PS2) 10Ori.livneh: puppet-merge: warn if multiple committers [operations/puppet] - 10https://gerrit.wikimedia.org/r/110104 [00:01:18] ^d: possibly -- what do you need? [00:01:26] <^d> I'm trying to move the purge-checkuser cron from hume to terbium. I'm not sure how often and when it currently runs on hume. [00:02:28] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.586 second response time [00:02:55] hume:/etc/cron.d/mw-purge-checkuser doesn't exist [00:03:44] <^d> Grrr :\ [00:06:28] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 310587 bytes in 7.431 second response time [00:06:35] (03PS2) 10Danny B.: skwiki: Configure transwiki import sources. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109723 [00:06:40] (03CR) 10Reedy: [C: 032] skwiki: Configure transwiki import sources. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109723 (owner: 10Danny B.) [00:06:47] (03Merged) 10jenkins-bot: skwiki: Configure transwiki import sources. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109723 (owner: 10Danny B.) [00:09:55] (03PS5) 10Odder: Enable per-wiki addition to 'translationadmin' group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109689 [00:10:27] (03Abandoned) 10Chad: WIP: Move purge-checkuser script off hume and to terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/108165 (owner: 10Chad) [00:16:44] (03CR) 10Reedy: [C: 032] Enable per-wiki addition to 'translationadmin' group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109689 (owner: 10Odder) [00:16:51] (03Merged) 10jenkins-bot: Enable per-wiki addition to 'translationadmin' group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109689 (owner: 10Odder) [00:17:30] (03PS6) 10Chad: Properly puppeti[sz]e purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy) [00:18:05] (03CR) 10jenkins-bot: [V: 04-1] Properly puppeti[sz]e purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy) [00:18:38] (03PS11) 10Dereckson: Throttle now handles IP ranges. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 [00:18:43] (03CR) 10Reedy: [C: 032] Throttle now handles IP ranges. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 (owner: 10Dereckson) [00:19:24] (03Merged) 10jenkins-bot: Throttle now handles IP ranges. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 (owner: 10Dereckson) [00:19:47] (03PS7) 10Chad: Properly puppeti[sz]e purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy) [00:20:11] !log reedy synchronized wmf-config/ [00:20:19] Logged the message, Master [00:20:34] (03PS1) 10Ori.livneh: gdash: fix capitalization of dashboard name [operations/puppet] - 10https://gerrit.wikimedia.org/r/110109 [00:20:56] (03CR) 10Ori.livneh: [C: 032 V: 032] gdash: fix capitalization of dashboard name [operations/puppet] - 10https://gerrit.wikimedia.org/r/110109 (owner: 10Ori.livneh) [00:22:28] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:56] <^d> Well crap. [00:23:02] You stole my commit [00:23:03] ! [00:23:13] <^d> Reedy: Yes I did :) [00:24:23] :( [00:24:53] what's going on with gitblit? [00:25:20] ori: Hump Day Strike [00:26:28] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 321746 bytes in 7.279 second response time [00:28:54] !log gitblit on antimony crashed with org.eclipse.jetty.io.EofException. trace: . lots of java.lang.NullPointerException due to malformed URLs, but these appear to happen continuously. [00:29:02] Logged the message, Master [00:29:20] <^d> Yeah I saw that recently. [00:29:40] <^d> Something's not encoding the /s in repo names to %2Fs [00:32:02] Is somebody looking at "PHP Warning: dba_fetch() expects parameter 2 to be resource, boolean given in /usr/local/apache/common-local/wmf-config/missing.php on line 76" [00:32:18] lol [00:32:19] No [00:32:22] That'll have been me [00:32:26] "me" [00:33:28] damn it [00:34:15] !log reedy synchronized wmf-config/missing.php [00:34:22] Logged the message, Master [00:34:31] !log another recurrent error in antimony:/var/log/upstart/gitblit.log : "org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed." repeats for each repository. traces: [00:34:39] Logged the message, Master [00:34:39] !log reedy updated /a/common to {{Gerrit|I2294bac73}}: Throttle now handles IP ranges. [00:34:42] (03PS1) 10Reedy: Move function_exists( 'dba_open' ) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110111 [00:34:47] Logged the message, Master [00:35:05] (03CR) 10Reedy: [C: 032] Move function_exists( 'dba_open' ) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110111 (owner: 10Reedy) [00:35:11] (03Merged) 10jenkins-bot: Move function_exists( 'dba_open' ) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110111 (owner: 10Reedy) [00:40:58] ori: That org.eclipse.jetty.io.EofException is probably just log noise. It means that the user-agent closed the socket before the response was sent (i.e. the client timed out waiting for the resource) [00:43:05] <^d> ori: I have jgit gc turned on for the gitblit replicas. [00:43:13] <^d> I wonder if we should gc using normal c git instead. [00:43:18] <^d> Might be...less painful [00:48:50] bd808: *shrug* I'm just logging it so ^d / opsen are aware; I can't troubleshoot it further atm. [01:42:13] * Eloquence waves to chasemp2 [01:51:11] cmjohnson1: hi [01:51:19] cmjohnson1: are you on-site perhaps? [02:10:25] (03PS8) 10Yurik: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 [02:16:57] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-29 02:16:57+00:00 [02:17:07] Logged the message, Master [02:31:30] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-29 02:31:30+00:00 [02:31:37] Logged the message, Master [02:49:16] ottomata: I think it should work with yajl1 aswell. [02:58:21] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-29 02:58:21+00:00 [02:58:29] Logged the message, Master [03:04:45] (03PS1) 10Andrew Bogott: Allow caller to specify Pin-Priority in apt::repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/110124 [03:08:21] 04:56 < Betacommand> save the .log as a txt file [03:08:21] 04:57 < Betacommand> open AWB [03:08:24] er [03:08:31] middle-click, sorry [03:08:38] paravoid: what? [03:08:42] sorry, wrong paste [03:33:26] (03PS1) 10Andrew Bogott: Pin the ubuntu-cloud repo so that dependencies work [operations/puppet] - 10https://gerrit.wikimedia.org/r/110126 [03:33:49] Coren, puppet+havana is cheered up by ^ and ^^. Thanks for diagnosing. [03:58:07] Does anyone feel like helping me debug a strange puppet logic thing? [03:58:16] re: https://dpaste.de/POHr [04:15:59] andrewbogott: What's the strangeness? [04:16:13] scfc_de: The 'unless' clause doesn't work [04:16:19] It always does it no matter what [04:16:27] When I run that 'unless' line on the cmdline it works properly [04:16:51] Um, wait, I pasted the wrong thing. One moment... [04:17:01] Have you tried replacing it with /bin/true? [04:17:18] OK, this is the good bit [04:17:19] https://dpaste.de/p7Vf [04:17:38] I will try [04:18:25] So… 'unless true' means that it should not run [04:18:37] In my understanding, yes. [04:18:40] Somehow the meaning of 'unless' keeps flipping in my head. been looking at this too long :) [04:19:30] I'm not sure if you could use "onlyif", or if there are additional differences to "!unless" :-). [04:20:03] I tried using onlyif, the behavior then was that the exec never ran. [04:20:15] :-) [04:20:22] So I think puppet understands what unless and onlyif means, but the actual shell command is not evaluating as I'd expect. [04:20:30] I can give you a login if you're interested in tinkering with the cmdline [04:20:54] Possibly ${glance_db_name} is empty in the 'unless' clause? That would explain the behavior I see [04:21:08] But, I can see that it is set properly in the exec command that immediately follows [04:21:21] andrewbogott: if you're just looking to avoid this without tinkering with unless, note there is CREATE DATABASE IF NOT EXISTS [04:21:59] The advantage of "unless" is that otherwise Puppet will log the command on every run. [04:22:22] fair enough [04:22:25] Also I would like to understand... [04:22:46] Although, the thirst for wisdom often leads to damnation [04:23:12] And if you replace the unless command with "echo ${glance_db_name} > /var/tmp/log.txt"? [04:23:27] There [04:23:47] scfc_de: That's a good one! Will try next... [04:23:53] is also an option to show what commands Puppet is actually executing, but I forgot it. [04:24:06] (It takes five mins or so to run each test) [04:24:50] scfc_de: more than just -v [04:24:51] ? [04:25:15] Don't remember, sorry. [04:27:31] ok, unless => '/bin/true' does not create the database. So that is encouraging [04:35:49] andrewbogott: what would the $HOME and $PATH be for the command when run by puppet? [04:36:04] springle: I'm not sure -- do they matter? [04:36:12] Shouldn't everything that matters be in my.cnf? [04:36:22] -uroot would only detect the /root/.my.cnf if $HOME was set properly [04:37:20] Ah, which contains the password… hm [04:40:03] although if unless => /bin/true caused the command to run, which also uses -uroot, probably not the problem :) nm me [04:43:00] why does /bin/true use -uroot? [04:43:32] And the command is failing, could be for lack of password rather than (as I assumed) due to the presence of the db [04:43:40] although then how would the db exist in the first place? Hm... [04:44:42] springle, https://dpaste.de/vRHg [04:44:45] Here is what I think: [04:45:05] - when puppet runs due to the periodic puppet cron, $home is correct, so db is created properly. [04:45:28] - when I run puppet via puppetd -tv, $home is wrong (it's mine) so the 'unless' clause fails [04:45:41] So… this is bad behavior is only visible when I look for it. [04:45:44] Sound credible? [04:46:07] sounds possible [04:46:23] i like puppet puzzles. is this one complicated to explain, or is there a code snippet to look at it? [04:46:26] *to look at [04:46:37] * ori is just joining in, my kibbitzer sense tingled. [04:46:50] ori, the original snip is https://dpaste.de/p7Vf [04:47:07] if you use -uroot as 'andrew', you would need to have /home/andrew/.my.cnf with root credentials [04:47:14] And the problem is -- when I do puppetd -tv, puppet tries to create the db every single time, whether or not it really exists. [04:48:10] so, let's see… is there sudo syntax that says 'use root env instead of mine'? [04:48:24] other than sudo su - etc. [04:50:37] you can set the HOME environment variable for that exec [04:50:53] yeah, but... [04:51:21] now I'm thinking that my standard debug process ($ sudo puppetd -tv) is inherently flawed. If it differs in behavior from the cron puppet run [04:51:26] then I must reform my process! [04:51:32] * andrewbogott is testing to make sure this is the case [04:51:48] just use an explicit --defaults-file=/root/.my.cnf each time? assuming it's readable by both the final puppet run and your test env... [04:52:14] sorry, who would I be passing --defaults-file to? mysql? [04:52:39] yes [04:52:48] that's better than messing with shell environment [04:52:50] /usr/bin/mysql -uroot --defaults-file=/root/.my.cnf [04:53:10] it's not --defaults-extra-file ? [04:53:15] hmm [04:53:22] oh no, it's --defaults-file [04:53:30] Ok, I've verified that if I sudo su - first then all is well. [04:53:38] no, don't [04:53:43] springle's solution is way less hacky [04:54:13] But… now I'm thinking this isn't actually a problem at all. [04:54:18] Since 'real' puppet runs as root anyway. [04:54:24] It's only my test process that scrambles the env [04:54:30] it's still ugly. [04:54:48] you were confused by it, right? which means that its behavior is hard to reason about, right? so make it explicit. [04:54:56] Patching the environment for every single exec in our codebase… also ugly? [04:55:12] sudo relies on the shell environment being just so; --defaults-extra-file reads the global configuration file first, which is another environment factor. --defaults-file means the exact behavior is fully specified in the command line; there are no hidden forces. [04:55:40] I guess I can fix the bits I'm looking at without having to modify the whole codebase. Halfmeasure better than no measure at all [04:57:21] full measure better than half measure :P [04:58:36] no more half measures, ori [04:58:52] go big or go home! [04:59:05] * ori readies the nukes. [05:03:48] hm, any chance --defaults-file isn't supported by my version of mysql? [05:05:37] andrewbogott: You're right. [05:05:40] ah, nm, it just doesn't like it after -uroot [05:06:09] You have to use "--defaults-file=/root/something" -- the "=" is important. [05:06:40] MySQL has some very strange manners. [05:09:02] Hrm. [05:09:09] Bye, Ken. [05:15:19] (03PS1) 10Andrew Bogott: Pass in explicit --defaults-file=/root/.my.cnf to db creation calls. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110128 [05:16:30] ori, ^ makes my puppet runs all quiet and happy [05:17:55] (03CR) 10Ori.livneh: [C: 032] "I'll let you merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/110128 (owner: 10Andrew Bogott) [05:18:27] to avoid this sort of repetition i create a 'sql' resource for the mysql module in mediawiki-vagrant [05:18:53] so you can write, e.g.: mysql::sql { 'add user': sql => "create user 'monty'@'localhost'", unless => "select 1 from mysql.user where user = 'monty'", } [05:18:59] https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/mysql/manifests/sql.pp [05:21:12] yeah, that's much easier to read [05:24:21] ori (and departed scfc_de and springle-afk) thanks for help w/sorting that [05:39:19] (03PS2) 10Andrew Bogott: Allow caller to specify Pin-Priority in apt::repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/110124 [05:39:21] (03PS2) 10Andrew Bogott: Pin the ubuntu-cloud repo so that dependencies work [operations/puppet] - 10https://gerrit.wikimedia.org/r/110126 [05:39:23] (03PS1) 10Andrew Bogott: Openstack Havana in eqiad, baby step: [operations/puppet] - 10https://gerrit.wikimedia.org/r/110130 [05:43:03] (03CR) 10Andrew Bogott: [C: 032] Allow caller to specify Pin-Priority in apt::repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/110124 (owner: 10Andrew Bogott) [05:43:11] why do you need to pin? [05:43:36] because of https://gerrit.wikimedia.org/r/#/c/110126/ [05:43:49] but why pinning them? [05:43:51] Um… which, otherwise apt refuses to install the needed dependencies from the ubuntu cloud archive [05:44:00] why? [05:44:09] the regular apt repo should be in the same priority as the ubuntu-cloud one [05:44:11] Because the standard brewster repo is already pinned [05:44:39] right, so, it's ours (apt.wikimedia.org) > ubuntu precise, and ubuntu precise == ubuntu-cloud [05:44:52] yes... [05:45:01] which packages do we have in our repo that conflict? [05:45:43] there were several I believe… if I give me a few minutes I can reproduce the problem, maybe there's a better solution. [05:45:52] um… if you give me :) [05:46:18] sure [05:46:20] I wonder why [05:46:36] maybe they were imported for the existing openstack cluster instead of using ubuntu-cloud? [05:46:56] pinning is ok too, but it might bite you in the future [05:47:09] if you need to override e.g. a single package from ubuntu-cloud [05:47:17] Yeah, might be cruft from before the ubuntu cloud existed [05:54:06] (03CR) 10Andrew Bogott: [C: 04-1] Openstack Havana in eqiad, baby step: (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110130 (owner: 10Andrew Bogott) [06:28:24] (03PS1) 10Ori.livneh: graphite: fix storage aggregation patterns [operations/puppet] - 10https://gerrit.wikimedia.org/r/110133 [06:28:51] (03CR) 10Ori.livneh: [C: 032 V: 032] graphite: fix storage aggregation patterns [operations/puppet] - 10https://gerrit.wikimedia.org/r/110133 (owner: 10Ori.livneh) [06:29:36] andrewbogott: should i merge your change? [06:30:01] did I forget to run puppet-merge? If so then go ahead and merge... [06:30:09] * ori does [06:30:09] If you're talking about the patch in gerrit then, not yet please [06:30:18] no, the former [06:32:33] thx [06:37:35] ok… paravoid, do you want to log in and tinker with apt yourself? on labs host puppet-testing-6 you can see the pinning problem by doing apt-get install nova-api [06:37:43] 'nova-api : Depends: nova-common (= 1:2013.2-0ubuntu1~cloud0) but it is not going to be installed' [06:39:27] python-nova : Depends: python-jsonschema (>= 1.3.0) but 1.1.0-1~precise1 is to be installed [06:40:01] if you force that, it works [06:40:09] now let's find out why/where do we use this [06:40:58] python-jsonschema is hosted on brewster [06:41:08] So that would do it. Hard to know /why/ it's on brewster though... [06:41:19] whoa, python-nova is using python-jsonschema? [06:41:24] https://rt.wikimedia.org/Ticket/Display.html?id=4474 [06:41:27] ori requested it [06:41:42] yeah, it's used by eventlogging [06:42:23] right [06:42:31] so, I see the following solutions, andrewbogott [06:42:46] upgrade the package on brewster? [06:43:00] a) do the pinning thing you did, with the drawback that you have no way (other than pinning a specific package even higher) to override ubuntu-cloud [06:43:52] b) upgrade python-jsonschema in our repo to 1.3.0 (or ubuntu-cloud's version as-is), with the drawback that they may get out of sync again and break python-nova (unlikely, imho) [06:44:17] Yeah, not really worried about it breaking nova stuff… ori, will it mess with you if I upgrade that package? [06:44:23] c) downpin ptyhon-jsonschema to 500 specifically in the nova manifests (which is kind of ugly, but will work) [06:44:29] (03Abandoned) 10Matanya: emery: move left emery udp2log logs and sync jobs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/109957 (owner: 10Matanya) [06:44:41] possibly. i need to walk the dog, back in 5. [06:44:49] whoah, you have a dog too? [06:44:53] paravoid, c) involves downpinning particular package for particular repo? [06:44:54] man, where do you find the time [06:45:06] andrewbogott: yeah [06:45:43] paravoid: seems like b) is the preferred option if we have a good understanding of everyone else who is currently using that package. [06:45:53] I think so too [06:45:59] all of them suck, (b) sucks less :) [06:46:15] yeah b would be the way to go imho [06:47:08] paravoid: have sec for pm? [06:47:15] The question with b) is if we manually upgrade everything that currently uses that package so that version is consistent with future installs... [06:47:39] trusty has python-jsonschema 2.3.0 fwiw [06:47:51] you can test test that in labs andrewbogott [06:47:52] and it's 2-3 months away [06:49:02] hm, well… that might argue for a or c, and then just undoing later. Dunno, will wait for ori to comment. [06:50:00] (03PS1) 10Springle: depool db1042 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110136 [06:50:26] (03CR) 10Springle: [C: 032] depool db1042 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110136 (owner: 10Springle) [06:50:32] (03Merged) 10jenkins-bot: depool db1042 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110136 (owner: 10Springle) [06:51:10] (03PS1) 10Matanya: emery: remove one udp2log logger. [operations/puppet] - 10https://gerrit.wikimedia.org/r/110137 [06:51:19] !log springle synchronized wmf-config/db-eqiad.php 'depool db1042 for schema changes' [06:51:28] Logged the message, Master [06:56:06] (03PS1) 10Matanya: emery: remove rsync arab banner job [operations/puppet] - 10https://gerrit.wikimedia.org/r/110138 [06:57:34] ori, suddenly a work crew is setting up ladders in my room and I'm also a few hours late for lunch, so will have to catch your answer on the backscroll. Presuming that the upgrade won't break things for you, I will make an RT bug to track the by-hand upgrades. [06:58:24] andrewbogott_afk: i have no idea if it will or not [06:58:30] i need to read the changelog and test it [06:59:43] andrewbogott_afk: it looks fine, based on . i'd still prefer to be around when you upgrade. [07:01:34] (03PS1) 10Matanya: emery: move api logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110139 [07:02:47] (03PS1) 10Ori.livneh: graphite: re-enable logging of VE performance counters [operations/puppet] - 10https://gerrit.wikimedia.org/r/110140 [07:03:26] (03CR) 10Ori.livneh: [C: 032 V: 032] graphite: re-enable logging of VE performance counters [operations/puppet] - 10https://gerrit.wikimedia.org/r/110140 (owner: 10Ori.livneh) [07:04:05] i regret the assault of tiny commits this past week [07:04:33] there was a long tail of graphite / statsd / metric module niggles to discover and fix [07:16:00] hi mutante [07:17:19] when you are around, i'd like your help in decoming erzurumi (idle) [07:17:19] loudon (active secondary central logger) [07:17:19] payments1 (active paymentsdb master) [07:17:19] payments2 (idle) [07:17:19] payments3 (idle) [07:17:20] payments4 (idle) [07:17:22] db78 (active db+archive [07:17:24] pappas (active bastion) [07:18:04] jeff green approved in 6635 [07:22:48] PROBLEM - MySQL InnoDB on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:48] PROBLEM - MySQL disk space on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:48] PROBLEM - Full LVS Snapshot on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:48] PROBLEM - MySQL Recent Restart on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:58] PROBLEM - mysqld processes on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:08] PROBLEM - puppet disabled on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:08] PROBLEM - Disk space on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:28] PROBLEM - MySQL Idle Transactions on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:28] PROBLEM - MySQL Processlist on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:28] PROBLEM - RAID on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:31] how interesting [07:23:32] springle: i assume you're aware [07:23:39] :) [07:25:56] paravoid: any point in fixing Dynamic lookup of $cluster at /etc/puppet/manifests/ganglia.pp:168 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes. [07:26:04] since we new have a module? [07:26:07] *now [07:27:21] yes, it's small and trivial to verify [07:28:02] (03CR) 10Dzahn: [C: 032] "removes Arabic Wikipedia Banner Pages" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110137 (owner: 10Matanya) [07:29:09] (03CR) 10Dzahn: [C: 032] "removes the cron for Arabic banners" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110138 (owner: 10Matanya) [07:29:11] ori: where does cluster come from? [07:29:34] no, mutante you shloudn't have merged that yet :/ [07:29:51] git grep 'cluster =' [07:30:09] matanya: siggggh?! [07:30:11] RT #6143 says: [07:30:11] Those should be fine to delete. Thanks again for checking! [07:30:12] - Jonathan [07:31:27] yes, but otto wanted to let the cron to run one more day, in order to remove all left logs [07:31:39] oh well, not critical [07:31:55] well, it didn't say so on the change :/ [07:31:59] revert or not [07:32:08] no [07:32:11] ok [07:32:17] see his comments on https://gerrit.wikimedia.org/r/109957 [07:33:13] i see, well i didnt notice because that was abanonded [07:33:19] at least he says "I'd rather move these filters one at a time" [07:33:22] and that's what we did [07:33:25] right [07:45:56] !log powercycled unresponsive db1042, /a tank data mount failed on boot, vgchange -a y + mount + xfs_check. still investigating [07:46:03] Logged the message, Master [08:12:23] (03CR) 10Lydia Pintscher: "@Peachey88: No. Wikidata relies on ULS more than any other of our projects. It has language built into its core like no other. We are prob" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 (owner: 10Dereckson) [08:14:00] (03PS1) 10Matanya: deployment: puppet 3 compatibility fix: full path to puppet file server [operations/puppet] - 10https://gerrit.wikimedia.org/r/110145 [08:23:01] ori: can you please merge https://gerrit.wikimedia.org/r/#/c/100760/ ? [08:24:38] andrewbogott: did you have a chance glancing at my etherpad module? [08:25:44] (03PS13) 10Matanya: svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 [08:25:46] matanya: Not yet, sorry, focused on labs stuff [08:26:23] OK, get your chips [08:26:27] how many fix-up commits? [08:26:30] for SVN [08:26:43] i hope not [08:26:54] it happens to everyone [08:26:59] you can bet 0 if you like [08:27:10] not liely thoght [08:27:13] i say 3 [08:27:14] i'm going with 2 [08:27:19] *likely [08:27:23] look at that, you're more pessimistic than i am! [08:27:37] since i did the change :) [08:27:56] (03CR) 10Ori.livneh: [C: 032] svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [08:28:09] ori, if you're pulling a late night then we can try a python-jsonschema update now... [08:28:19] Or if you can suggest a good test to run on labs I'm happy to set that up [08:28:36] i have to shepherd matanya's svn patch now, let's see how well that goes first [08:28:45] what's your pick, btw, andrewbogott? [08:29:12] can we hedge 1 and 2? [08:29:13] For svn fixups? I'll go with 1, I like long odds. [08:29:15] unfair, you can split the fixes or squash them to adjust the winning number [08:29:22] hehe :P [08:29:31] mutante: ori always wins :P [08:29:46] we'll see, maybe not [08:29:57] mutante, that's risky -- even if you squash everything you might wind up with another one on the end. [08:29:57] that's what the people who always win always say as well :P [08:30:24] yuvipanda: i don't commit the fixups, matanya does [08:30:28] i report puppet failures [08:30:49] hmm, seems fair enough actually [08:31:06] no fucking way [08:31:27] matanya: http://p.defau.lt/?mXU0M674u0Z_AJY86Xx5_A [08:31:49] nice, good job matanya :) [08:31:54] sweet, i wanted it to be "subversion":) [08:31:57] wow! [08:32:02] matanya: thanks!! [08:32:07] wow! [08:32:09] nobody bet 0 [08:32:20] settle down, i have to do the other svn hosts too [08:32:52] one was just installing client [08:32:56] the other server [08:33:10] formey and antimony are both server [08:33:32] ok, then there was one more where it just uses the client role, right [08:34:42] antimony was a no-op run too aside from the motd, so that's 2/2 so far [08:34:59] ori: i meant subversion::client [08:35:02] is on bast1001 [08:35:07] yeah, puppet already running [08:35:11] cool! [08:37:21] no-op on bast1001 too [08:37:43] :) [08:37:56] fenari's last [08:39:38] should we still have svn clients on bastions? [08:39:57] i have no idea what they're doing there in the first place [08:40:16] Reedy probably knows [08:40:22] agree [08:41:32] fenari too [08:41:52] nice job, matanya! [08:42:11] & thanks for the patch [08:42:25] thank you :) some credit to mutante too [08:45:55] andrewbogott: there's a labs instance, yeah [08:46:08] Host deployment-eventlogging.labs [08:46:08] Hostname = I-00000733 [08:46:26] If I upgrade the package there will you be able to tell w/not it broke something? [08:46:39] we may need a ticket for "twemproxy on fenari" [08:46:39] And, are you pretty confident that eventlogging is the only thing that's using the package? [08:46:39] upgrade the package and run "eventloggingctl restart" [08:46:42] i'll be able to tell [08:46:46] linked to "out of Tampa" [08:47:12] i think so, yeah [08:47:26] it's pretty obscure [08:48:23] matanya: check doc.wm.org now for auto-generated docs it takes from module structure:) [08:48:33] Hm… using an 'asia' mirror when downloading to a tampa host? Not really the model of efficiency [08:48:51] https://doc.wikimedia.org/puppet/classes/subversion.html [08:51:15] (03PS2) 10Matanya: deployment: puppet 3 compatibility fix: full path to puppet file server [operations/puppet] - 10https://gerrit.wikimedia.org/r/110145 [08:51:24] ori: OK, upgraded and restarted. [08:51:40] * ori looks at the logs to be safe [08:51:47] (03CR) 10Ori.livneh: [C: 032] deployment: puppet 3 compatibility fix: full path to puppet file server [operations/puppet] - 10https://gerrit.wikimedia.org/r/110145 (owner: 10Matanya) [08:52:43] nice mutante, thaks :) [08:53:00] (03CR) 10ArielGlenn: [C: 032] snapshots: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109653 (owner: 10Matanya) [08:53:11] so many pings [08:53:26] matanya: if a .pp file has comments on the line right before a class{} or define{} it finds them, but only if there is no newline [08:53:58] should add some docs to some modules [08:54:03] and there are cases where just due to the newline it doesnt show up there [08:54:06] ori, going to merge your salt master changes [08:54:07] !log applying NTP access lists on cr{1,2}-{esams,knams,eqiad,pmtpa,sdtpa,ulsfo}, csw2-esams, pfw1-eqiad [08:54:11] er puppet-merge them [08:54:15] Logged the message, Master [08:54:24] apergos: i already did [08:54:49] we must have hit it the same time, palladium asked me to do them [08:55:18] kk, thanks either way [08:55:53] ori, so, that package is ensure=>present, so when I update brewster nothing will happen… for now. [08:56:19] until when? [08:56:26] this sounds like a good candidate for RT 135 [08:56:29] oh, until it gets reprovisioned [08:56:32] Until a new server uses that module in which case it will have a different version [08:56:35] or newly provisioned after a hw failure [08:56:36] right [08:56:41] Yeah, so might be best to force an upgrade just so we aren't surprised later. [08:56:49]