[00:00:03] I have a performing client now, much better to use multipart/form-data [00:01:30] grr, the dumps I downloaded expand to > 412G [00:01:59] (03CR) 10Faidon Liambotis: [C: 032] cassandra: add sysctl.d tunable [operations/puppet] - 10https://gerrit.wikimedia.org/r/91326 (owner: 10Faidon Liambotis) [00:02:35] paravoid: i still think managing sysctl.d recursively is right; it's better for this to be in puppet [00:02:54] I don't disagree [00:03:19] it's a package bug + a sysctl::conffile limitation that it always prepends priority [00:03:25] paravoid: how are thumbnails coming along? [00:03:43] paravoid: hm -- what's the prepending priority issue? [00:03:43] AaronSchulz: copying at 10-15MB/s [00:04:04] ori-l: the package's postinst does [00:04:04] if ! sysctl -p /etc/sysctl.d/cassandra.conf; then [00:04:07] (warnings) [00:04:17] rm -v /etc/sysctl.d/cassandra.conf [00:04:22] fi [00:04:38] if that file doesn't exist, configure (i.e. install, upgrade etc.) fails [00:04:39] ugh [00:04:51] that's a bug in itself, I'm going to file it and suggest them to do rm -vf [00:05:06] but sysctl::conffile can't be tweaked to install at that location [00:05:23] externally I mean, we could always fix it :) [00:05:28] i can fix that [00:05:29] but it wasn't an unreasonable decision [00:06:59] RECOVERY - Disk space on xenon is OK: DISK OK [00:08:01] god, I hate jira [00:11:40] ori-l: this seems like something random you would know off the top of your head -- the pybal data files -- do you know where the canonical location for them is? [00:11:56] it doesn't seem to be in the public puppet stuff [00:12:04] no idea at all, i'm sure paravoid knows [00:12:25] oh, um, actually [00:12:32] i think they're on fenari somewhere, no? [00:12:34] yes [00:12:43] fenari:/h/w/conf/pybal [00:12:48] ah; awesome! [00:12:49] what he said [00:12:51] thanks :) [00:12:56] * mwalker writes that down [00:13:20] they're being served as http://noc.wikimedia.org/pybal/ [00:13:37] same thing really [00:13:50] only roots can write to them, though. [00:13:55] anything in particular I can help with? [00:14:12] I just need an updated copy -- I have a script that queries all the text caches for content [00:14:26] I'll looking for stray URLs in CentralNotice that keep getting redirected to foundation wiki [00:14:43] which group are you using? [00:14:49] we're migrating to varnish for text [00:14:55] there's a separate pybal group for this [00:15:04] ("text-varnish" vs. "text") [00:15:08] currently 'text' [00:15:10] and ah [00:15:14] so I'll need to load both [00:15:33] nod [00:15:39] expect them to have overlap though [00:15:57] we have squids in text-varnish as well, with enabled = False [00:16:16] it's easier to just vi text-varnish and convert a few Trues to False and vice-versa than switching groups [00:16:45] hehe -- the magic of python means I just add the two lists together [00:17:18] though; actually; that probably does a true addition... /me looks further [00:17:24] not sure what your script does, but careful :) [00:17:34] *nods* [00:17:59] text would have cp1001 as enabled=True, text-varnish would have cp1001 as enabled=False & cp1052 as enabled=True [00:18:46] it takes a prototype CentralNotice URL; and queries every cache server for it's current content under that prototype -- it's not exactly rate limited -- but we're only talking 30 queries per server -- and it's not threaded [00:19:32] (03PS1) 10Ryan Lane: Namespace mediawiki repos for deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/91327 [00:27:08] (03PS1) 10Springle: warmup db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91332 [00:29:25] (03CR) 10Springle: [C: 032] warmup db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91332 (owner: 10Springle) [00:30:37] !log springle synchronized wmf-config/db-eqiad.php 'warmup db1034' [00:30:49] Logged the message, Master [00:34:53] paravoid: as a first test, I'll import three history dumps in parallel from the three machines [00:35:14] each only hitting localhost [00:35:16] cool [00:36:50] that's around 900G of input text [00:39:46] ori-l: I highly suggest http://chimera.labs.oreilly.com/books/1230000000545/ [00:40:10] it was out of stock in amazon UK but it appeared in stock for 3 whole days so I managed to order it [00:40:16] but I've started reading it out of the html [00:40:34] nothing new/shocking so far, but it's a nice overview of everything [00:40:48] I haven't read it all [00:41:19] it's a bit strange that it talks about the different variants of mobile networks in a book called "high performance *browser* networking" [00:41:35] so it does get a bit too unnecessarily deep into some subjects imho [00:42:12] (the author is quite well known too) [00:42:45] Loads of copies in stock ;) [00:43:06] "Only 11 left in stock (more on the way)." is what amazon.co.uk says now [00:43:16] it was out of stock last week [00:43:38] released a month ago [00:43:59] Ahh [00:44:58] I pinged ori-l specifically because I thought he might care the most -- but it's definitely something that could be of broader interest [00:53:41] paravoid: yeah, ilya gregorik is awesome [00:54:16] paravoid: the import is running, but the client is still the bottleneck [00:54:25] since you're already reading it, if you come away from reading it with any ideas, let me know? [00:54:45] sure. nothing groundbreaking so far [00:54:46] I'm heading out, will work on speeding up the client tomorrow [01:51:00] (03PS1) 10Reedy: Move "RewriteEngine On" earlier in www.wikimedia.org vhost [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91339 [01:52:53] (03PS2) 10Reedy: Move "RewriteEngine On" earlier in www.wikimedia.org vhost [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91339 [02:03:23] Reedy: you still around? [02:03:28] I'm guessing probably not [02:04:16] Bugzilla seems wonky. [02:04:20] Reedy is always around. [02:04:38] I did just try to leave... [02:04:49] gj [02:04:59] same here, but wfm [02:05:39] im not able to look at scary Apache vhosts now [02:05:46] so some improvement [02:06:02] mwalker: wassup? [02:06:12] Elsie: wonky? looks ok to me [02:06:36] I'm seeing about 1/4 of requests to Special:BannerRandom still being redirected to wmfwiki [02:06:41] It gave me an Apache error a few minute ago and it hung a bit. [02:06:55] seems to be all hosts with the name sq[0-9]{2}.wikimedia.org that are doing it [02:06:59] Could just be me, though. [02:07:47] I can provide some example URLs if needed -- but what were you doing before that was clearing them? [02:08:28] Elsie: seems to be working fine for me [02:08:51] I've been able to use it, it just seems a bit wonky. :-) [02:11:16] *nods* not that I'll be able to fix it -- but what's wonky about it? queries being slow; pages rendering odd? [02:12:08] It gave me an Apache error a few minutes ago and it hung a bit. [02:12:14] Could just be me, though. [02:12:38] mwalker: Daniel had been running a script to clear the Squid caches. [02:12:41] purgeList.php [02:12:44] It's on the relevant bug. [02:12:58] He did the small *.wikimedia.org wikis. [02:13:00] but not so useful for wildcards etc [02:13:06] But probably didn't do sq0? [02:13:09] What is that? [02:13:16] squid [02:13:33] squid has an exposed public hostname? [02:13:38] https://bugzilla.wikimedia.org/show_bug.cgi?id=56006#c12 is the bug, mwalker. [02:13:45] it'll only purge a list of URLs [02:13:49] well; I have a live stream of shit that's getting 302'd -- for centralnotice [02:13:56] varnish does it a lot better [02:14:02] so... I can work with the restriction of a purge list [02:14:21] File a bug. :-) [02:14:23] relatively easily fixed then [02:14:35] you can run it yourself even! [02:14:40] I marked 56006 as fixed and recommended others file a bug. [02:14:49] For new issues such as this. [02:14:55] Reedy: where do we run this type of script from? [02:14:58] tin [02:15:05] or terbium [02:15:17] doesn't matter a great deal..quick to run [02:15:23] kk [02:15:32] quick to run over 28k urls? :p [02:15:49] ..and then my phone shut down and i lost my mobile internetz [02:16:07] !log LocalisationUpdate completed (1.22wmf22) at Wed Oct 23 02:16:06 UTC 2013 [02:16:31] Logged the message, Master [02:17:10] it is [02:17:40] without writing to console, no sleep 12k URLs was very quick [02:17:45] ok; preparing the list for blasting [02:17:54] including database queries too ;) [02:18:35] omg dos!!!!!! [02:18:44] That's the response I usually get. [02:19:03] purging all of outreachwiki took seconds :D [02:19:23] Elsie: glancing at bugzilla server i dont see anything obvious.. [02:19:23] how is it doing that? [02:19:23] Should've null edited and you would've purged Squid and refreshed the *links. ;-) [02:19:37] mutante: Okay, no worries. Nobody else is complaining. [02:19:42] well; how does it know about URL parameters? [02:19:49] yes, --all worked ok on small wikis [02:19:50] mwalker: purgeList.php has a --wiki parameter. [02:19:53] quality.wm too [02:19:57] Elsie: ok [02:20:08] put the full URL [02:20:18] http://en.wikipedia.org [02:20:23] arffghj [02:20:37] /WIKI/FOOBAR [02:20:47] stupid phone [02:21:07] yep; they all look something like this: http://meta.wikimedia.org/wiki/Special:BannerRandom?uselang=es&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=1&country=CO&device=desktop&slot=4 [02:21:11] I'm impressed you were able to start a line with a slash from your phone whilst raging. [02:21:26] press / twice [02:21:28] does it care about SSL varients? [02:21:48] meta isn't forced SSL, so use just http [02:21:53] gotcha [02:22:11] My IRC client went a bit goofy and joined every channel twice. [02:22:25] at worst, do it once, run it, http -> https and run it again ;) [02:24:39] !log LocalisationUpdate completed (1.22wmf21) at Wed Oct 23 02:24:39 UTC 2013 [02:24:57] Logged the message, Master [02:26:55] Reedy: Were there any reports of Meta-Wiki's API returning a 404? [02:27:08] I seem to have a lot of e-mails from cron. [02:29:04] 404s? no... [02:30:07] Reedy: damn; that was fast [02:30:16] It'll be trying to hit /w/api.php... [02:30:23] * Elsie shrugs. [02:30:29] also; /me grumbles about scp having stupid numeric options that I can never remember [02:32:31] add --verbose and it will be a lot slower! [02:33:59] heh [02:34:00] ya [02:36:40] Reedy: presumably the timeout for these 301s is the standard 30 days? [02:36:56] probably [02:37:25] *nods* I'll have to script this then; slowly whittle my horrific cache explosion of doom away [02:37:32] thanks for your help though :) [02:40:20] I think 301s are cached client-side for much longer than 30 days. [02:47:16] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 23 02:47:16 UTC 2013 [02:47:28] Logged the message, Master [03:09:11] (03PS1) 10Kaldari: Changing default wmgMinimumVideoPlayerSize from 200 to 800. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91342 [03:48:52] (03PS1) 10Legoktm: Enable MassMessage on all wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91344 [05:00:31] Unless anyone objects, in about half an hour I'm going to sync a small config change (https://gerrit.wikimedia.org/r/#/c/90265/) that adds purge/thumbnail rate limits. It's not urgent but I said I'd do it today and I'd prefer to stand by that commitment. [05:08:30] ori-l: do you know if that limit will be proactively revisted? or not? [05:09:42] ori-l: but, I don't object, just curious if a follow up bug/something needs to be filed. [05:25:43] RECOVERY - Host srv291 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [05:27:53] PROBLEM - Apache HTTP on srv291 is CRITICAL: Connection refused [05:28:53] RECOVERY - Apache HTTP on srv291 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.400 second response time [05:30:23] !log powercycled srv291, unresponsive to ping, no login at mgmt console [05:30:39] Logged the message, Master [05:38:33] PROBLEM - Puppet freshness on mw125 is CRITICAL: No successful Puppet run in the last 10 hours [05:40:07] (03CR) 10Ori.livneh: [C: 032] Added some purge/thumbnail rate limits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90265 (owner: 10Aaron Schulz) [05:40:17] (03Merged) 10jenkins-bot: Added some purge/thumbnail rate limits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90265 (owner: 10Aaron Schulz) [05:53:49] !log ori synchronized wmf-config/InitialiseSettings.php 'I11c90ed5a: Added some purge/thumbnail rate limits' [05:54:04] Logged the message, Master [05:54:25] !log sync-file: 'Could not resolve hostname mw125: Name or service not known' [05:54:39] Logged the message, Master [06:07:15] (03CR) 10Ori.livneh: [C: 032] Timestamp log messages [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/71315 (owner: 10Ori.livneh) [06:22:03] PROBLEM - Disk space on cp1050 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 12345 MB (3% inode=99%): /srv/sdb3 13582 MB (4% inode=99%): [06:27:15] (03PS1) 10ArielGlenn: put mw125 back in (range off by one error) [operations/dns] - 10https://gerrit.wikimedia.org/r/91347 [06:28:12] (03CR) 10ArielGlenn: [C: 032] put mw125 back in (range off by one error) [operations/dns] - 10https://gerrit.wikimedia.org/r/91347 (owner: 10ArielGlenn) [06:29:34] ori-l: wanna resync that file now? [06:29:38] ^^ [06:30:03] apergos: sure [06:30:09] thanks [06:30:28] well I saw it ding my daily checks, but happened also to notice your comment in the logs [06:31:18] wonder what else has been synced in that time, I'd better check [06:31:39] right, might have to scap [06:31:50] in which case it might be better to keep it out of circulation until morning PDT [06:33:05] I can do the sync on that host only [06:34:59] * apergos tries something clever over there [06:35:05] yeah, that's right [06:35:14] sure, go for it [06:35:47] hm puppet not exactly running, that's a bit of a problem. well apache is off for the moment, and I'll poke atit [06:37:02] PROBLEM - Apache HTTP on mw125 is CRITICAL: Connection refused [06:37:43] yeah yeah we know, and it will stay that way too for a bit [06:43:10] ugh, I see... gotta wait a while [06:45:55] have to wait for it to fall out of the tampa recursors [06:46:52] this is a great time to go to the bank, brb [06:57:58] back [07:37:44] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Wed Oct 23 07:37:39 UTC 2013 [07:39:04] RECOVERY - Apache HTTP on mw125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.274 second response time [07:39:32] good morning [07:39:52] rats, I just started another puppet run without seeing in here that it already went [07:40:05] well I'll check the log as soon as this one completes [07:40:51] mw-sync went, rsync went [07:41:22] so that's sync-common [07:41:31] that sohuld get everything under the sun, problem solved [07:43:27] !log mw125: stopped apache shortly after adding back to dns, (needed to wait for an hour for the update to reach the pmtpa recursors so puppet could run), at first successful puppet run mw-sync completed so this host should be good to go now [07:43:41] Logged the message, Master [07:50:01] apergos: I think you can tail the puppet log in /var/log/puppet.log or something [07:50:16] apergos: I do that whenever the cron tasks kicks off [07:50:25] might even have colors \O/ [07:50:31] well I was not in screen, I was already running puppet [07:50:44] and too lazy to log in via anotehr terminal, that's why I wasn't tailing the log [07:51:14] I had been watching the clock for the dns ttl to expire, see [08:26:39] (03PS4) 10Hashar: ganglia wrapper for py plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 [08:26:40] (03PS1) 10Hashar: ganglia: diskstat.py plugin [operations/puppet] - 10https://gerrit.wikimedia.org/r/91351 [08:26:44] (03PS1) 10Hashar: contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 [08:28:02] (03CR) 10Hashar: "I have renamed the define to ganglia::python::plugin per Andrew." [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 (owner: 10Hashar) [08:33:57] (03PS5) 10Ori.livneh: ganglia wrapper for py plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 (owner: 10Hashar) [08:34:25] (03PS6) 10Ori.livneh: ganglia wrapper for py plugins [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 (owner: 10Hashar) [08:35:28] (03CR) 10Ori.livneh: [C: 032] "PS5/6: tiny whitespace / spelling changes; remove reference to bug 36994 from commit message. (Not strictly related any longer.)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 (owner: 10Hashar) [08:36:22] (03PS2) 10Ori.livneh: ganglia: diskstat.py plugin [operations/puppet] - 10https://gerrit.wikimedia.org/r/91351 (owner: 10Hashar) [08:36:44] (03CR) 10Ori.livneh: [C: 032] "Gmetric module code LGTM." [operations/puppet] - 10https://gerrit.wikimedia.org/r/91351 (owner: 10Hashar) [08:38:20] I am wondering how many weeks it will takes for me to exhaust Ori :-] [08:40:11] a few, at least :) is there a role where you could put the diskstat resource declaration? [08:40:13] as opposed to site.pp [08:41:07] ori-l: someone commented on my first patch how it was adding unnecessary glue [08:41:08] https://gerrit.wikimedia.org/r/#/c/85669/1/manifests/ganglia.pp,unified [08:41:23] I had a ganglia::plugin::diskstat wrapper [08:41:25] comment is: [08:41:31] This is already nice enough: [08:41:32] ganglia::pyplugin { 'diskstat': [08:41:33] opts => { [08:41:34] devices => [ 'sda', 'sdb' ], [08:41:35] }, [08:41:36] } [08:41:56] yes, it's a question of whether it goes in site.pp or the contint role i guess [08:42:04] it should go in 'standard' eventually, but it's good to make sure it works well first on a select group of machines, so i see the logic of applying it to gallium and lanthanum first, but having it in site.pp seems [08:42:14] yeah that was my idea [08:42:16] *gross [08:42:17] try it out for contint [08:42:25] also the role class might not end up always be applied [08:42:35] i.e. role::contint::master is only on gallium [08:42:41] role::contint::slave is on both [08:43:02] but anyway, both classes rely on a SSD drive being mounted to /srv/ssd which is done at the node level [08:43:08] so I though the monitoring was making more sense there [08:43:27] could it go in ::slave, with a comment saying it is being evaluated for the standard role? [08:44:42] amending :-] [08:45:04] oh, i see what you're saying [08:45:08] (03PS2) 10Hashar: contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 [08:45:20] IMO the mounts should move to contint as well, but that's for you to decide :P [08:45:48] yeah I tried to mount it under contint [08:45:56] does not play well cause role::ci::master require the mount [08:46:04] and role::ci::slave requires it as well [08:46:13] but gallium include both classes so you end up with a duplicate definition [08:46:15] and [08:46:42] it makes more sense to me to handle the mount stuff at the node level, makes it more obvious what the hardware conf is on then ode [08:46:43] node [08:46:59] want me to move the ganglia::plugin::python{'diskstat': } to role::ci::slave ? [08:48:33] hmm. well, i see the logic of having the mounts at the node level [08:48:50] at any rate it looks like faidon did it for swift and i'm sure he knows what he's doing [08:49:05] hopefully [08:49:13] maybe ask him? [08:49:15] or we will end up loosing TB of commons pictures :] [08:49:32] apergos made sure it's all on archive.org anyway :P [08:49:48] no, that was Nemo_bis [08:50:04] oh [08:50:21] I would just go for node [08:50:30] and later on add to standard if that work for them [08:50:31] or [08:50:41] I can move the calls to contint::slave if you prefer, patch is ready to be send :-] [08:50:57] I don't really care one way or another [08:52:02] well, I don't want you to amend the patch just to satisfy my preferences, and I'm suppose to be taking it nice & slow with the Puppet stuff anyway, so maybe we should wait to ask someone else? (apergos, do you have an opinion, if you've been following along?) [08:52:10] *supposed [08:52:16] if we want to add diskstat for mysql server, we will most probably add it to the role db class as well :-] [08:52:21] I haven't, sorry, working on something else right now [08:52:28] I am only looking if somone pings me [08:52:39] got it, sorry [08:53:15] hashar: maybe wait to ask p-void? that way we both learn something [08:53:37] (03PS3) 10Hashar: contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 [08:53:38] na na [08:53:40] role is fine :-] [08:53:50] that is only one line added now [08:54:02] and will make sure all jenkins CI slaves have the diskstat :] [08:54:08] https://gerrit.wikimedia.org/r/#/c/91352/3/manifests/role/ci.pp,unified [08:54:16] grr missing a space [08:55:59] (03PS4) 10Hashar: contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 [08:56:50] also between python + {diskstat [08:58:12] i'll amend it [08:59:55] (03PS5) 10Ori.livneh: contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 (owner: 10Hashar) [09:00:53] (03CR) 10Ori.livneh: [C: 032] contint: monitor CI server diskstats in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/91352 (owner: 10Hashar) [09:02:03] hashar: merged [09:02:19] \O/ [09:02:40] might have a patch for you too in a bit :P [09:04:44] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type ganglia::plugin::python at /etc/puppet/manifests/role/ci.pp:138 on node gallium.wikimedia.org [09:04:45] :( [09:04:46] seriously [09:05:19] puppet [09:05:20] please [09:05:21] :-D [09:05:52] do you have another class named ganglia in that scope? [09:06:11] plugin::python versuse python::plugin [09:06:21] or not [09:07:28] should be plugin::python [09:08:06] (03PS1) 10Hashar: ganglia::python::plugin --> ganglia::plugin::python [operations/puppet] - 10https://gerrit.wikimedia.org/r/91358 [09:08:09] ^^^^ [09:08:12] sorry :( [09:08:42] (03PS2) 10Ori.livneh: ganglia::python::plugin --> ganglia::plugin::python [operations/puppet] - 10https://gerrit.wikimedia.org/r/91358 (owner: 10Hashar) [09:09:16] (03CR) 10Ori.livneh: [C: 032] ganglia::python::plugin --> ganglia::plugin::python [operations/puppet] - 10https://gerrit.wikimedia.org/r/91358 (owner: 10Hashar) [09:10:02] ok,merged [09:15:27] puppet runnig [09:15:41] info: /Stage[main]/Role::Ci::Slave/Ganglia::Plugin::Python[diskstat]/File[/etc/ganglia/conf.d/diskstat.pyconf]: Scheduling refresh of Service[gmond] [09:15:44] \O/ [09:19:56] /usr/sbin/gmond[27120]: [PYTHON] Can't call the metric_init function in the python module [diskstat].#012 [09:19:57] /usr/sbin/gmond[27120]: Unable to find any metric information for 'diskstat_(.+)'. Possible that a module has not been loaded.#012 [09:20:01] you should get to bed ori :-] [09:28:21] hashar: you should set a 'devices' param [09:28:41] http://paste.debian.net/60759/ [09:28:53] from my manual testing earlier it was not needed [09:28:54] :( [09:33:56] you need either 'devices' or 'device_mapper' in params [09:33:59] the sequence is: [09:34:33] line 383-384: else: DEVICES = params.get('devices') [09:34:50] >>> assert {}.get('nonexistent') is None [09:34:51] ori-l: SyntaxError: Unexpected token { [09:35:07] yeah and I am wondering why gmond pass device to it [09:35:10] maybe it s a default [09:36:18] ori-l: so that sounds like a bug in there :D [09:36:38] the get should use '' as a default I guess [09:37:13] wouldn't fix it, too late at that point [09:37:44] the 'fix' would be to change 377 to: [09:37:52] (currently: if params.get('device-mapper') == 'true': ) [09:38:39] to: if params.get('device-mapper') == 'true' or 'device' not in params: [09:39:02] even that is a bit stupid [09:39:04] it should just be [09:39:05] that would make the plugin use device-mapper per default so [09:39:07] if 'device' not in params: [09:39:37] well, if the intention was to make it require an explicit param one way or the other, it should throw an intelligible error in metric_init [09:40:06] if not 'device' in params and not 'device-mapper' in params: log.exception('one of "device" or "device-mapper" must be set!') [09:40:35] i suggest we "fix" this in puppet [09:41:07] well there a bunch of DEVICES != '' [09:41:33] it's not too late to use tim's module, you know :P [09:41:40] falling back to empty string when fetching devices did the job: DEVICES = params.get('devices', '') [09:41:44] hehe [09:43:42] meh, i don't love the tortuous argument-handling and the gratuitous use of globals [09:43:45] but up to you [09:44:16] I could make DEVICE to default to None hehe [09:44:23] will report upstream anyway [09:44:33] (03PS1) 10Hashar: ganglia: diskstats plugin using wrong default [operations/puppet] - 10https://gerrit.wikimedia.org/r/91360 [09:44:57] i still think you should fix it in puppet for now rather than fork [09:45:08] by setting device-mapper in the pyconf file [09:45:14] the puppet change is https://gerrit.wikimedia.org/r/91360 [09:45:20] will submit a pull request upstream [09:45:28] making DEVICES to be null and updating all the logic [09:45:33] no no, i meant in the pyconf erb template [09:45:37] until upstream merges [09:45:55] add param device-mapper { value = 'true ' } [09:45:58] like https://github.com/ganglia/gmond_python_modules/blob/master/diskstat/conf.d/diskstat.pyconf [09:46:05] except not commented out of course [09:46:18] well device-mapper look for block devices under /dev/mapper [09:46:24] and that does not exist on the CI servers :/ [09:48:41] ugh, yeah. in that case, '' is right [09:48:51] (03PS2) 10Ori.livneh: ganglia: diskstats plugin using wrong default [operations/puppet] - 10https://gerrit.wikimedia.org/r/91360 (owner: 10Hashar) [09:49:26] (03PS1) 10ArielGlenn: remove asher, py from icinga access; use wikitech names for authz [operations/puppet] - 10https://gerrit.wikimedia.org/r/91361 [09:49:54] (03CR) 10Ori.livneh: [C: 032] ganglia: diskstats plugin using wrong default [operations/puppet] - 10https://gerrit.wikimedia.org/r/91360 (owner: 10Hashar) [09:51:27] (03CR) 10ArielGlenn: [C: 032] remove asher, py from icinga access; use wikitech names for authz [operations/puppet] - 10https://gerrit.wikimedia.org/r/91361 (owner: 10ArielGlenn) [09:58:57] (03CR) 10Hashar: "Proper fix sent upstream: https://github.com/ganglia/gmond_python_modules/pull/120" [operations/puppet] - 10https://gerrit.wikimedia.org/r/91360 (owner: 10Hashar) [09:59:05] ori-l: patch proposed upstream https://github.com/ganglia/gmond_python_modules/pull/120 [09:59:12] might take a few months for them to process it :-] [10:01:28] ori-l: you are such a hacker [10:15:12] thank you very much [10:45:10] (03PS1) 10ArielGlenn: remove htpasswd from icinga config, not used (see r51315) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91367 [10:46:39] (03CR) 10ArielGlenn: [C: 032] remove htpasswd from icinga config, not used (see r51315) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91367 (owner: 10ArielGlenn) [11:37:48] hey apergos [11:37:51] saw the patch? [12:00:31] (03CR) 10Akosiaris: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/91043 (owner: 10Dzahn) [12:01:57] YuviPanda: no, not yet [12:02:04]