[00:13:10] 3Wikimedia Labs / 3deployment-prep (beta): Issues connecting to https://meta.wikimedia.beta.wmflabs.org - 10https://bugzilla.wikimedia.org/73680 (10Kunal Mehta (Legoktm)) 3NEW p:3Unprio s:3normal a:3None $ curl "https://meta.wikimedia.beta.wmflabs.org" curl: (7) Failed connect to meta.wikimedia.beta.... [00:16:08] 3Wikimedia Labs / 3deployment-prep (beta): beta labs no longer listens for HTTPS - 10https://bugzilla.wikimedia.org/68387#c9 (10Bryan Davis) *** Bug 73680 has been marked as a duplicate of this bug. *** [00:16:10] 3Wikimedia Labs / 3deployment-prep (beta): Issues connecting to https://meta.wikimedia.beta.wmflabs.org - 10https://bugzilla.wikimedia.org/73680#c1 (10Bryan Davis) 5NEW>3RESO/DUP https doesn't work in beta. *** This bug has been marked as a duplicate of bug 68387 *** [01:06:04] Coren: is there an ETA for the complete move to trusty? [01:08:07] Betacommand: Strictly speaking, not for a very long time - I'll keep precise around until nobody uses it - but I'll probably switch teh default to trusty within a few months. [01:42:41] (03PS1) 10Krinkle: Enforce coding style [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174894 [01:56:55] [13nagf] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/wikimedia/nagf/commit/f23a507528575700877e0afdbff53aef99f54e8c [01:56:55] 13nagf/06master 14f23a507 15Timo Tijhof: build: Move test script from .travis.yml to composer.json and package.json [02:57:00] (03PS1) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:08:23] (03PS2) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:08:26] (03PS1) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:11:19] (03PS2) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:11:21] (03PS3) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:12:45] (03PS3) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:12:47] (03PS4) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:12:49] (03PS1) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:13:56] (03PS2) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:13:58] (03PS4) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:14:00] (03PS5) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:17:06] (03PS3) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:17:08] (03PS5) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:17:10] (03PS6) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:19:23] (03PS4) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:19:25] (03PS6) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:19:27] (03PS7) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:22:38] (03PS5) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:22:40] (03PS7) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:22:42] (03PS8) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:26:25] (03PS6) 10Krinkle: gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 [03:26:27] (03PS8) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:26:29] (03PS9) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:27:15] (03PS9) 10Krinkle: settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 [03:27:17] (03PS10) 10Krinkle: Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 [03:29:18] (03CR) 10Krinkle: [C: 032 V: 032] Enforce coding style [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174894 (owner: 10Krinkle) [03:29:33] (03CR) 10Krinkle: [C: 032 V: 032] gitignore: Ignore cache/*.json [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174904 (owner: 10Krinkle) [03:29:44] (03CR) 10Krinkle: [C: 032 V: 032] settings: Read config from replica.my.cnf [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174903 (owner: 10Krinkle) [03:29:56] (03CR) 10Krinkle: [C: 032 V: 032] Use spaces instead of underscores in display of page title [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/174902 (owner: 10Krinkle) [06:29:19] Is there anyway I can change my login shell to my own compiled gnu screen on each instance? [13:55:11] andrewbogott_afk: are we using ldap for ssh keys in trusty already? [13:55:18] does this mean an NFS outage will not affect them? [14:16:11] YuviPanda|zzzz: That'll never actually happen as homes are on NFS anyways. [14:16:45] ah, heh, true [14:17:24] clearly, we should switch back to glusterfs [14:23:56] Weren't you advocating for CIFS earlier? :-) [14:26:35] Coren: I was. CIFS over Thunderbolt devices. clearly that shall be fastest [14:26:41] shinken-wm: well, hello there! [14:26:49] shinken-wm: now to wait and see if you work this time :) [14:27:00] * YuviPanda got side-tracked by what ended up being a red herring for a couple of days [14:42:28] I think IRCEcho needs to die. [15:06:56] * YuviPanda continues throwing things at ircecho [15:24:19] have labs changed something with the puppet certificate names ? [15:24:29] on a puppet client I have: Server hostname 'deployment-salt.eqiad.wmflabs' did not match server certificate; expected i-0000015c.eqiad.wmflabs [15:24:54] it has: puppet.agent = server = deployment-salt.eqiad.wmflabs [15:25:01] I guess the master has the wrong certificate name now :d [15:44:46] hashar: deployment-salt is self-hosted isn't it? So it's not talking to any master other than itself? [15:58:30] andrewbogott: I fixed it :] [15:58:42] so yeah deployment-salt is the puppet master [15:58:52] I somehow managed to screw up the master key [15:59:05] I regenerated it and all agents are happy now :] [16:05:29] great! [16:29:11] andrewbogott: shinken caught this as well pretty early. I'm close to considering initial deployment stable once I get ircecho working [16:29:30] Will get spam here also [16:29:33] :) [16:29:47] 'this' meaning hashar's puppet failure? [16:36:55] YuviPanda: do you have a few minutes to help me sort out a hiera/labs question? [16:37:57] andrewbogott: yeah hashar's change [16:38:07] andrewbogott: in a restaurant eating food. Will be back in 1h [16:38:10] ok! [17:52:11] it seems tools-login is stuffed with non-interactive tasks that all in all consume much memory [17:52:30] at least i think that's why memory usage is what it is [17:52:35] and i think it is too high [17:55:26] YuviPanda: back? I think my first question (possibly only question) is if you'd expect the wikitech heira pages to work with self-hosted puppet? [17:55:38] annika_: Mem: 4049952k total, 2903464k used, 1146488k free [17:56:02] and the most memory consuming process is someone's ls [17:56:34] hm, i just run into no available memory once in a while, but that could be my own fault [17:57:16] annika_: err, on tools-login or when you submit a task to the grid? [17:58:21] tools-login [17:59:22] annika_: that would suggest you're using 1GB of memory yourself :-p [17:59:25] andrewbogott: it does work [17:59:38] Deployment prep and integration projects use that [18:00:08] YuviPanda: in that case… can you help me sort out what's happening in the 'testlabs' project, instance testlabs-hieratests? [18:00:53] Sure. In 10mins. Walking home [18:01:34] thanks [18:15:45] andrewbogott: back [18:15:47] andrewbogott: looking now [18:15:59] thanks [18:21:30] andrewbogott: interesting. is the project name testlabs? or is it test-labs or something like that? [18:21:33] * YuviPanda checks [18:21:37] testlabs, I think [18:21:40] could be some case mismatch [18:22:03] nah [18:22:12] no case mismacth [18:27:18] YuviPanda: so, just to confirm… it's not working for you either, right? No obvious user error? [18:27:23] yeah [18:27:26] no obvious user error [18:27:27] I'm trying a similar thing in the 'openstack' project, also to no avail. [18:27:38] Dunno if it's useful for you to have a second example. [18:27:44] which host? [18:27:57] wikitech-test-network and wikitech-test-frontend [18:50:30] YuviPanda: I'm going to eat some lunch but won't stray too far from my keyboard [18:51:03] andrewbogott: ok. I'm not really sure what's happening on the wikitech instances, no local commits, not sure what to look out for. [18:51:10] (and got distracted by gwicke) [18:51:12] :) [18:51:43] YuviPanda: use of openstack::version in heira is active in production [18:51:47] So, no local commits is… that. [18:51:55] aaaah [18:51:59] is this project OpenStack? [18:52:10] * YuviPanda checks [18:52:18] right [18:52:18] it is [18:53:02] testing by changing to icehouse now [18:53:36] Well, I want it to be havana, but it is defaulting to icehouse due to the setting in prod hiera. [18:53:37] I think. [18:53:47] oh, hmm [18:53:52] well, wait [18:54:01] no, probably not due to production heira, probably just due to defaults in puppet [18:54:15] andrewbogott: this might be a bug in our hiera code. [18:58:07] andrewbogott: file a phab task, maybe? :) and then we can poke _joe_ with it [18:58:47] YuviPanda: With topic 'hiera no work'? [18:59:06] But, ok :) [18:59:10] andrewbogott: yeah :) [18:59:15] I'm still betting on some sort of case-sensitivity. [18:59:34] andrewbogott: it's weird, becuase it does work in other places. [18:59:50] andrewbogott: hmm, openstack is Hiera:Openstack, and deployment-prep is Hiera:Deployment-prep [19:00:02] because mediawiki forces all titles to have an uppercase character for first character [19:00:09] but also gives you the right answer if you send the wrong case [19:00:13] (Yay mediawiki) [19:01:18] andrewbogott: might also end up being non-obvious user error, but I can't see any user errors there [19:23:27] Coren: the "no webservice" error page for the tool giftbot says Tools.giftbot is a maintainer of the tool, others don't have it [19:24:13] annika_: That's because, for some reason, they put the tool as one of its own maintainers. It's not harmful, but it's also not very useful either. [19:25:04] euh [19:25:52] self-maintaining tools! [19:26:09] skynet, etc [19:28:01] annika_: you can add service groups as maintainers, too. This can be useful for something like subprojects, where one group of people maintains multiple tools [19:28:36] ya, now i see it [19:38:31] andrewbogott: yay, shinken-wm is now talking properly :) [19:39:24] great! [19:39:38] andrewbogott: have no way to 'acknowledge' alerts tho [19:40:09] hm, that may get annoying :) [19:40:51] andrewbogott: yup [19:40:58] andrewbogott: it's set to re-alert only once in a day tho [19:41:08] Ah, that's maybe fine then [19:41:28] YuviPanda: do you know where the code is that slurps up heira settings from wikitech? Is it in puppet itself? [19:41:37] andrewbogott: yeah, it is. [19:43:10] andrewbogott: 57b9f91795ba714b90023d78dd31d814c7973e3e [19:43:21] Thank you! [19:43:22] andrewbogott: in the puppetmaster module [19:44:00] yay, ruby [19:44:35] Do you know how I can turn on Hiera.debug or where it logs to? [19:45:03] ah [19:45:04] hmm [19:45:05] no idea :| [19:45:58] andrewbogott: puppetmaster log? [19:45:59] maybe? [19:46:19] andrewbogott: http://puppetlabs.com/blog/debugging-hiera maybe? [19:46:33] yeah, reading that now [20:07:04] YuviPanda: on deployment-salt, I'm trying to demonstrate the hiera is working: [20:07:06] puppet apply -e '$somevar = hiera(ocg::temp_dir) notify { $somevar: }' [20:07:07] But I get nothing [20:07:14] Any idea what mistake I'm making? [20:07:54] andrewbogott: try hiera('has_ganglia')? [20:07:59] also fails [20:08:22] hmm, tha'ts weird. [20:08:33] it's possible puppet apply doesn't do the things that agent -tv does? [20:08:41] try putting that in site.pp default? [20:09:34] andrewbogott: I think it's still working, since without it there'd be more puppet errors for the cache machines and stuff that will notify [20:11:37] I don't think I should mess with beta's site.pp :) [20:17:12] hehe [20:20:43] YuviPanda: what's the other project where you think it works? [20:21:19] Integration [20:21:24] And soon shinken [20:21:34] Tools too [20:21:40] Hm, guess I don't want to mess with integration's site.pp either [20:21:44] You are optimistic! [20:22:39] If it didn't we would get tools dying every day from corr dumps [20:54:11] valhallasw`cloud: for gerrit-patch-uploader, did you create a ssh key so it could push to gerrit? [20:57:51] legoktm: yup, suchabot also has ssh key [20:57:57] okay [20:58:00] with no passphrase? [20:58:07] or just in ssh-agent? [20:59:28] YuviPanda: ^ ? [20:59:34] no passphrase [21:16:55] legoktm: yes [21:17:04] but I could have done https, too [21:17:27] but indeed key without passphrase [21:17:35] ok [21:17:46] are you planning to impersonate it? :-p [21:20:22] nope, got something eviler planned [21:20:42] oh dear [21:21:06] legoktm: but, as mentioned, you can also just use the https password [21:21:39] well then you have to use one of git's credential thingies [21:21:40] https://stackoverflow.com/questions/5343068/is-there-a-way-to-skip-password-typing-when-using-https-github [21:21:42] ssh is much easier [21:22:16] legoktm: nah, just https://username:password@etc [21:22:33] then you have to make sure your .git dir is not readable [21:22:46] same with your private key :-p [21:23:02] yes but there's only one private key, and a bajillion git repos [21:23:07] fair point [21:23:18] gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -fPIC -I/usr/include/python3.2mu -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.2/lib/sqlalchemy/cextension/processors.o [21:23:18] lib/sqlalchemy/cextension/processors.c:10:20: fatal error: Python.h: No such file or directory [21:23:19] :| [21:23:22] no python3-dev module? [21:23:33] where are you compiling this? [21:23:36] I think we had it [21:23:44] tools-login :P [21:23:54] woah [21:23:56] apparently we don't [21:23:59] let me fix that [21:24:10] ty :D [21:25:54] legoktm: waitig for jenkins. [21:26:09] legoktm: but I also just hand-installed python3-dev on -login [21:27:33] ugh [21:27:36] legoktm: you should be good to go now [21:27:41] okay :D [21:27:42] ty! [21:27:46] ugh was for something else :P [21:27:49] heh [21:28:34] awwww [21:28:37] this is python3.2 [21:29:03] legoktm: there's a trusty instance :) [21:29:04] legoktm: use that [21:29:16] legoktm: and -l release=trusty on your jsub to run on trusty :) [21:29:59] legoktm: I'm going to mess up extdist1 for a moment [21:30:06] okay :/ [21:30:10] and tools-login-trusty, or something like that, as login/dev node [21:30:42] trusty.tools.wmflabs.org [21:34:31] legoktm: tell me if you get a extdist1 service email [21:35:39] valhallasw`cloud: https://gerrit.wikimedia.org/r/175113 [21:38:19] is that what I think it is? [21:38:52] probably :P [21:38:54] https://github.com/legoktm/gerrit-gh [21:38:57] legoktm: verry nice [21:39:03] but how will it work with phab? :-p [21:39:17] (then again, that didn't stop me from building pywikibugs) [21:39:44] I have a github issues->phab thing in progress too :P [21:40:16] YuviPanda: got the emails [21:40:22] ** PROBLEM alert - extdist1/Free space - all mounts is CRITICAL ** [21:40:26] CRITICAL: extdist.extdist1.diskspace._var.byte_percentfree.value (<22.22%) [21:40:35] has hearts ^.^ [21:40:47] legoktm: did you break mediawiki again with your json? :-p [21:40:53] lolol [21:41:10] for comments, I want to do what facebook does for hhvm stuff [21:41:37] "All comments and reviews will happen in Phabricator" and they leave a comment with a link to the review on the github PR [21:42:22] ah ok [21:43:56] https://github.com/wikimedia/mediawiki-extensions-examples/pull/2 <-- yay for github auto-closing :D [21:44:24] legoktm: :) [21:45:52] legoktm: cool, should recover in a few mins and send a recovery email. [21:45:55] legoktm: and also notify this channel [21:46:35] if that works, then \o/ [21:46:37] legoktm: you don't need to explicitly close it, right? [21:46:47] legoktm: a commit with Closes: XX in it will close the pull request [21:46:52] yup [21:46:56] that's what it did :) [21:46:57] oh wait [21:47:04] wmfgerrit is what's called github on gerrit [21:47:13] oh no, it's gerritadmin [21:47:30] wmfgerrit is the account that replicates gerrit to github [21:47:47] ohhh, I see [21:47:51] not anything I set up [21:47:59] but this bot will need a github account to comment on things [21:48:12] *nod* [21:48:31] https://github.com/wmfgerrit?tab=activity i see :P [21:51:18] * legoktm goes to find food [21:57:18] YuviPanda: got the fixed email [21:57:25] damn [21:57:31] shinken-wm: y u keep quiet :( [22:00:29] YuviPanda: also I received 2 emails for critical, and 2 for recovery. [22:00:43] legoktm: yeah, one from shinken test server a nd one from real server [22:00:47] ah [22:00:48] need to find way to kill all test server email