[00:19:33] <grrrit-wm>	 (03PS2) 10Ori.livneh: Add abacist module & role; provision on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/181110 
[00:21:48] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Add abacist module & role; provision on stat1001 [puppet] - 10https://gerrit.wikimedia.org/r/181110 (owner: 10Ori.livneh)
[00:26:56] <icinga-wm>	 RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57863 bytes in 0.084 second response time  
[00:34:36] <hoo>	 ori: here?
[00:34:52] <hoo>	 Can I run sync-common on osmium without having to worry about the world ending?
[00:35:13] <hoo>	 I need a hhvm production host to test a two lines of code
[00:36:25] <ori>	 hoo: sure, go for it
[00:52:32] <icinga-wm>	 PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 2 failures  
[00:52:43] <icinga-wm>	 PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 1 failures  
[00:55:54] <icinga-wm>	 PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures  
[00:59:56] <icinga-wm>	 PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 3 failures  
[01:00:24] <icinga-wm>	 PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:01:03] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [500.0]  
[01:01:24] <icinga-wm>	 PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:02:03] <icinga-wm>	 PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 4 failures  
[01:03:23] <icinga-wm>	 RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures  
[01:06:13] <icinga-wm>	 RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures  
[01:06:45] <icinga-wm>	 RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[01:11:12] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[01:13:34] <icinga-wm>	 RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[01:13:43] <icinga-wm>	 RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[01:14:44] <icinga-wm>	 RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[01:15:32] <icinga-wm>	 RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[01:17:37] <hoo>	 !log Ran mysql:wikiadmin@db1033 [centralauth]> DELETE FROM bug_54847_password_resets WHERE r_username = 'Stilfehler';
[01:17:47] <morebots>	 Logged the message, Master
[01:38:19] <grrrit-wm>	 (03PS1) 10Hoo man: Fix Bug54847.php for broken hashes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181710 
[04:54:14] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Fix Bug54847.php for broken hashes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181710 (owner: 10Hoo man)
[05:21:17] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] beta: honor log sampling for logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181349 (owner: 10BryanDavis)
[06:33:46] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:33:57] <icinga-wm>	 PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:35:30] <icinga-wm>	 PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:35:33] <icinga-wm>	 PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 3 failures  
[06:35:34] <icinga-wm>	 PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:36:06] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:36:17] <icinga-wm>	 PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:36:40] <icinga-wm>	 PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:45:40] <icinga-wm>	 RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures  
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures  
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures  
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[06:47:49] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[06:47:55] <icinga-wm>	 RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[06:48:07] <icinga-wm>	 RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[06:49:16] <icinga-wm>	 RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures  
[08:15:17] <godog>	 matanya: did you investigate how to upload larger files to mw?
[08:15:37] <matanya>	 godog: nit really, i just split them
[08:15:40] <matanya>	 *not
[08:15:52] <matanya>	 the upper limit is 5GB
[08:15:57] <matanya>	 and done server side
[08:16:10] <matanya>	 client side is limited to 1GB
[08:18:12] <godog>	 matanya: ack, and the videos reenconded to smaller size too?
[08:18:26] <matanya>	 no
[08:18:42] <matanya>	 but i can do that as well, though it is a pain
[08:20:59] <andrewbogott>	 matanya: if you don't mind being a lab rat, I can create a big image for your video project on one of the new servers.
[08:21:01] <godog>	 yeah I was wondering if it makes sense to have videos raw on commons as opposed to just reencoded and raw on e.g. archive
[08:21:02] <andrewbogott>	 What specs do you want?
[08:21:48] <matanya>	 andrewbogott: feel free to experiment on me 16 or more CPU, 16GB of ram
[08:22:09] <matanya>	 godog: i can just copy from youtube then, and no need for reecoding
[08:23:21] <godog>	 matanya: that'll do too yeah
[08:23:39] <matanya>	 so that is the new approche ?
[08:27:58] <andrewbogott>	 matanya: disk space?
[08:28:07] <godog>	 matanya: merely a suggestion, perhaps somebody from multimedia team?
[08:28:11] <matanya>	 andrewbogott: 80 GB ?
[08:28:54] <matanya>	 godog: the raw isn't higher quality from a 720p file
[08:29:36] <matanya>	 https://commons.wikimedia.org/wiki/File:Evaluation_I_Metrics_p1-001.webm
[08:30:05] <icinga-wm>	 PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures  
[08:30:21] <matanya>	 vs https://www.youtube.com/watch?v=0Z6Hv91pIEs godog 
[08:30:30] <andrewbogott>	 matanya: want to create an instance of type 'm1.gigantic' and make sure it gets scheduled on virt1010 or virt1011?
[08:30:41] <matanya>	 yes
[08:31:38] <godog>	 matanya: nice, looks good
[08:32:17] <matanya>	 godog: so the one on commos in 10mbps and the one on youtube is approx 2mbps
[08:32:22] <matanya>	 and no real diff
[08:33:18] <matanya>	 andrewbogott: Failed to create instance. 
[08:37:51] <andrewbogott>	 matanya: hm, I will look
[08:39:06] <andrewbogott>	 oh, quota
[08:40:07] <andrewbogott>	 matanya: try again?
[08:40:41] <matanya>	 yes, now it is being created
[08:44:03] <icinga-wm>	 RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures  
[08:44:35] <_joe_>	 good morning
[08:44:53] <_joe_>	 I'm on/off IRC this morning (on a train) but I'm working
[08:46:08] <andrewbogott>	 matanya: if that new instance holds up for a few days then please delete the old smaller one if you can spare it.
[08:46:14] <_joe_>	 so if you need me, drop me an email
[08:46:39] <matanya>	 ok, andrewbogott i'll need to install some tools
[08:46:42] <andrewbogott>	 matanya: the only bad behavior we've seen on virt1011 so far is that an instance turned itself off at one point.  No idea if that's a hosting problem or was just bad behavior within the instance.
[08:46:57] <matanya>	 i can live with that
[08:47:46] <matanya>	 btw, andrewbogott can you automaticlly install all packages on one instance on another one ?
[08:48:17] <andrewbogott>	 there's certainly no automated way to do it, although dpkg --list and a bit of sed work might get you a full list to jam into apt-get install
[08:48:48] <matanya>	 i used dpkg --get-selection
[08:49:01] <matanya>	 but that was 100% by hand
[08:49:38] <_joe_>	 andrewbogott: will you find time to fix the openstack templates anytime soon?
[08:49:54] <_joe_>	 they're the last big offender when it comes to variable access
[08:50:16] <_joe_>	 (the puppet templates I mean)
[08:50:28] <andrewbogott>	 _joe_: sure, I just now put it on my list-of-things-to-do-soon
[08:50:42] <andrewbogott>	 but I'm out for the next couple of weeks, so it won't be before mid-January.
[08:52:12] <_joe_>	 eheh me too
[08:52:21] <_joe_>	 whoever does that first I'd say
[08:52:40] <_joe_>	 this is utterly-low-priority of course, it's just my OCD 
[08:59:12] <matanya>	 andrewbogott: just FYI, puppet is failing on this host
[08:59:33] <andrewbogott>	 matanya: I'll look
[08:59:41] <matanya>	 nfs issues
[08:59:49] <matanya>	 running it again to verify
[09:00:00] <andrewbogott>	 oh, in that case it probably just needs a reboot.
[09:00:11] <andrewbogott>	 I thought that race was fixed, but… I saw the same problem earlier today
[09:04:17] <matanya>	 rebooting
[09:14:44] <matanya>	 thanks andrewbogott fixed
[09:14:52] <andrewbogott>	 cool.  Hope it holds up
[09:18:15] <grrrit-wm>	 (03PS2) 10KartikMistry: WIP: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 
[09:37:16] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "Minor inline comments, rest LGTM" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/181080 (owner: 10Filippo Giunchedi)
[09:53:20] <_joe_>	 can someone name me a good reason why we don't include admin in standard?
[09:53:40] <_joe_>	 apart from the fact we didn't have hiera at the time
[09:55:52] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Fix image_commands in the jessie manifest. [puppet] - 10https://gerrit.wikimedia.org/r/181722 
[09:57:13] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Fix image_commands in the jessie manifest. [puppet] - 10https://gerrit.wikimedia.org/r/181722 (owner: 10Andrew Bogott)
[09:57:47] <akosiaris>	 _joe_: we could not override per host the parameters, which pretty much boils down to we did not have hiera
[09:57:57] <akosiaris>	 which means we now can :-)
[09:58:10] <akosiaris>	 there is my useless answer for the year ;-)
[09:58:54] <_joe_>	 not useless
[09:59:01] <_joe_>	 you just confirmed I was right
[09:59:09] <_joe_>	 :)
[10:06:06] <_joe_>	 but well, I won't work on that today anyways
[10:06:45] <_joe_>	 gosh I almost have 2 weeks without working, I'm not used to this
[10:07:02] <YuviPanda>	 _joe_: whenever I’ve had that I’ve never managed to not work
[10:07:20] * YuviPanda wishes _joe_ good luck
[10:07:24] <matanya>	 _joe_: you can volunteer :)
[10:07:42] <_joe_>	 matanya: I already do, I'm paid for a 40 hour week
[10:08:07] <YuviPanda>	 hehe
[10:08:16] * matanya grants _joe_ the honor of all wikimedians
[10:08:42] <YuviPanda>	 the line is somewhat blurry. All my ‘side projects’ have somehow been very related to my ‘get paid for this’ stuff
[10:09:00] <_joe_>	 also, I have other projects I'm a bit negleticting lately I should really give some love to
[10:09:59] <_joe_>	 but well, I'll mostly eat and meet some old friends for a few days
[10:10:25] <_joe_>	 so I'll probably manage not to work. 
[10:11:33] <YuviPanda>	 nice
[10:22:30] <matanya>	 akosiaris: for the HP devices, you also redirect console output ?
[10:22:49] <akosiaris>	 yeah
[10:23:34] <matanya>	 thanks
[11:51:27] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: backups: Create an offsite pool [puppet] - 10https://gerrit.wikimedia.org/r/181725 
[11:51:29] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: backups: Support next pool directive [puppet] - 10https://gerrit.wikimedia.org/r/181726 
[11:51:31] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: backups: define next_pool for the production pool [puppet] - 10https://gerrit.wikimedia.org/r/181727 
[11:51:33] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: backups: add offsite job template [puppet] - 10https://gerrit.wikimedia.org/r/181728 
[12:31:37] <grrrit-wm>	 (03PS1) 10Yuvipanda: shinken: Provide ldap information for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/181732 
[13:16:02] <icinga-wm>	 PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail  
[13:29:42] <icinga-wm>	 RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:57:16] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] backups: add offsite job template [puppet] - 10https://gerrit.wikimedia.org/r/181728 (owner: 10Alexandros Kosiaris)
[13:58:05] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] backups: define next_pool for the production pool [puppet] - 10https://gerrit.wikimedia.org/r/181727 (owner: 10Alexandros Kosiaris)
[13:59:38] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] backups: Support next pool directive [puppet] - 10https://gerrit.wikimedia.org/r/181726 (owner: 10Alexandros Kosiaris)
[13:59:51] <Jeff_Green>	 !log package updates and reboots for several fundraising servers...
[13:59:57] <morebots>	 Logged the message, Master
[13:59:57] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] backups: Create an offsite pool [puppet] - 10https://gerrit.wikimedia.org/r/181725 (owner: 10Alexandros Kosiaris)
[14:17:44] <grrrit-wm>	 (03PS1) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:18:46] <grrrit-wm>	 (03PS2) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:27:09] <grrrit-wm>	 (03PS3) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:27:24] <grrrit-wm>	 (03PS4) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:29:26] <paravoid>	 YuviPanda: isn't there any openstack API you could query instead?
[14:30:58] <YuviPanda>	 paravoid: not for puppet classes, no.
[14:31:06] <YuviPanda>	 err
[14:31:12] <YuviPanda>	 puppet roles paplied to each instance via wikitech
[14:31:13] <YuviPanda>	 that’s in LDAP
[14:34:16] <_joe_>	 we need puppetdb!
[14:36:36] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "I was about to comment something about perhaps using the metadata service to encode the tenant/project ID instead of the separate LDAP que" [puppet] - 10https://gerrit.wikimedia.org/r/181535 (owner: 10Andrew Bogott)
[14:38:21] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "I've killed a bunch of provider => upstart all over the tree as well. Older versions of puppet were buggy and did not do the right thing w" [puppet] - 10https://gerrit.wikimedia.org/r/181540 (owner: 10Andrew Bogott)
[14:40:22] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 031] "The only thing that base::instance-upstarts does is to set up a getty under ttyS0 (= serial console). This isn't needed with systemd as it" [puppet] - 10https://gerrit.wikimedia.org/r/181541 (owner: 10Andrew Bogott)
[14:42:01] <grrrit-wm>	 (03PS1) 10RobH: setting mgmt info for stat2001 [dns] - 10https://gerrit.wikimedia.org/r/181741 
[14:42:36] <grrrit-wm>	 (03PS2) 10RobH: setting mgmt info for stat2001 [dns] - 10https://gerrit.wikimedia.org/r/181741 
[14:44:57] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "Clearly the case. I'm wondering what we're using it for, though. I don't think it's needed for unattended upgrades. If it is, how do they " [puppet] - 10https://gerrit.wikimedia.org/r/181539 (owner: 10Andrew Bogott)
[14:45:43] <grrrit-wm>	 (03CR) 10RobH: [C: 032] setting mgmt info for stat2001 [dns] - 10https://gerrit.wikimedia.org/r/181741 (owner: 10RobH)
[14:50:16] <grrrit-wm>	 (03PS5) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:51:48] <paravoid>	 _joe_: puppetdb wouldn't help much here
[14:52:06] <paravoid>	 YuviPanda: it's kind of a pity that we call out to LDAP as often we are :/
[14:52:29] <paravoid>	 considering we have this infrastructure that is supposed to be built on well-defined restful APIs :)
[14:52:33] <_joe_>	 paravoid: I will use the <sarcasm> tag next time :)
[14:52:52] <paravoid>	 heh
[14:52:58] <akosiaris>	 we need moar clojure!!!
[14:53:04] <valhallasw`cloud>	 paravoid: add an LDAP api to wikitech! :-p
[14:53:05] <YuviPanda>	 we have infrastructure that’s built on well defined restful APIs?!
[14:53:06] <YuviPanda>	 where?
[14:53:21] <valhallasw`cloud>	 also I don't think the mediawiki API classified as RESTful :-p
[14:53:31] <YuviPanda>	 valhallasw`cloud: he was referring to OpenStack, I think
[14:53:34] <paravoid>	 I was refering to openstack
[14:53:41] <valhallasw`cloud>	 ah, fair enough
[14:54:03] <YuviPanda>	 I think we can get puppet roles into OS’s metadata service somehow, but that somehow feels like a much bigger project than hitting LDAP :D
[14:54:05] <_joe_>	 valhallasw`cloud: the mediawiki api is everything but restful
[14:54:06] <YuviPanda>	 maybe along with Horizon.
[14:54:23] <YuviPanda>	 _joe_: spiteful. that’s what it is.
[14:54:26] <paravoid>	 I was researching this just now for a different reason
[14:54:31] <paravoid>	 (https://gerrit.wikimedia.org/r/181535)
[14:54:44] <akosiaris>	 so Horizon is that new shiny beatiful thing that will solve all of our problems and bring world peace along as well ?
[14:54:50] <paravoid>	 but the metadata service seems to only be about an instance querying its metadata
[14:55:00] <akosiaris>	 why I am so ready to be disappointed ? :-/
[14:55:01] <paravoid>	 not a host querying all instances
[14:55:01] <grrrit-wm>	 (03PS6) 10Yuvipanda: Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 
[14:55:32] <YuviPanda>	 akosiaris: mostly switching to horizon will require so much work we hope there is enough time and energy to do that properly instead of like the last time around
[14:55:44] <valhallasw`cloud>	 akosiaris: !xkcd standards
[14:55:52] <valhallasw`cloud>	 https://xkcd.com/927/
[14:55:57] <akosiaris>	 hehe
[14:56:15] <YuviPanda>	 paravoid: hmm, puppet failure status is tracked through the metadata service (and is completely useless also), so I’m guessing we can do something like that..
[14:56:22] <YuviPanda>	 put a JSON blob in there or something
[14:57:28] <akosiaris>	 !log disabled puppet on helium while testing copy jobs
[14:57:32] <morebots>	 Logged the message, Master
[14:58:22] <YuviPanda>	 also, err
[14:58:29] <YuviPanda>	 home NFS on labs is 94% full?
[14:58:41] <YuviPanda>	 CorenAFK: ^
[14:58:57] <YuviPanda>	 at least something’s fuller than /var
[15:00:21] <paravoid>	 YuviPanda: puppet master's api is a restful API
[15:00:40] <paravoid>	 I'm not sure if you can query for included classes
[15:00:48] <paravoid>	 and how long would that take (probably a full catalog generation)
[15:01:02] <YuviPanda>	 paravoid: that, plus self hosted puppetmasters
[15:01:12] <paravoid>	 grumble
[15:01:15] <YuviPanda>	 I also don’t know if it can query for the classes that are explicitly included
[15:01:30] <paravoid>	 right
[15:02:03] <YuviPanda>	 in some ways this might be somewhat like role() that _joe_ has for prod :D 
[15:02:07] <paravoid>	 hmm
[15:02:13] <YuviPanda>	 (or not, I have only vague ides what role() does)
[15:02:47] <grrrit-wm>	 (03PS2) 10Yuvipanda: shinken: Provide ldap information for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/181732 
[15:02:49] <YuviPanda>	 hmm, except now shinken throws up the equivalent of a NullPointerException
[15:02:52] * YuviPanda shakes fist at shinken
[15:02:57] <paravoid>	 if it can query the classes that are explicitly included (i.e. query the ldap terminus), then it wouldn't need a full catalog compilation and it would work for selfs
[15:04:39] <YuviPanda>	 ‘it’?
[15:04:56] <YuviPanda>	 and yeah, just hitting ldap works for selfs and also doesn’t involve any compilation
[15:05:58] <YuviPanda>	 what the fuck, shinken
[15:06:04] <YuviPanda>	                     if hg is None:
[15:06:07] <YuviPanda>	                         hg.configuration_errors.append(err)
[15:06:09] <YuviPanda>	 ...
[15:06:47] <_joe_>	 lol
[15:09:28] <_joe_>	 oh man, I'm trying to apply role() in site.pp, and I keep finding edge cases
[15:09:37] <_joe_>	 that don't really need to be there btw
[15:10:40] <CorenAFK>	 Hm.  I think I need to look at storage and see where it's going away.
[15:11:06] <Coren>	 It's probably one or two outliers consuming most of this.
[15:11:20] <paravoid>	 YuviPanda: root@virt1000:~# curl -k -H "Accept: pson" https://virt1000.wikimedia.org:8140/production/node/i-00000615.eqiad.wmflabs | jq '.data.classes'
[15:11:52] <paravoid>	 it needed a auth.conf stanza to allow virt1000 to query for this
[15:12:09] <paravoid>	 (plus an apache2 allow from, to allow virt1000 to query itself)
[15:12:26] <paravoid>	 but you can easily make this better
[15:13:32] <_joe_>	 paravoid: it think that implies a full catalog compilation
[15:13:40] <paravoid>	 no it doesn't
[15:13:41] <_joe_>	 *I think it
[15:13:54] <_joe_>	 paravoid: it uses cached catalogs?
[15:14:07] <paravoid>	 no
[15:14:18] <paravoid>	 this is just the node information
[15:14:35] <paravoid>	 most of it is facts
[15:14:37] <paravoid>	 from the database
[15:14:42] <_joe_>	 oh and .data.classes includes information from ldap?
[15:14:42] <paravoid>	 but I think classes is from the ldap terminus
[15:14:54] <_joe_>	 nice, didn't know that
[15:14:56] <paravoid>	 we can clear the database for an instance and check, though
[15:15:20] <_joe_>	 even if you get those info from the db, it's still good enough
[15:17:34] <YuviPanda>	 hmm, this plus hitting nova / wikitech would work as well...
[15:17:57] <YuviPanda>	 I can also make the wikitech API hit the puppetmaster, but that sounds horrible + there’s no way to test any changes to wikitech atm...
[15:18:56] <paravoid>	 wikitech wants to also write and this isn't suitable for this
[15:19:39] <paravoid>	 you could write a new API for getting/setting classes, though, and use an ENC instead of the LDAP terminus
[15:19:44] <paravoid>	 and abstract LDAP from puppet altogether
[15:22:10] <Coren>	 That may be counterproductive since the intent is to get rid of wikitech.
[15:22:40] <paravoid>	 you need something like this anyway
[15:22:50] <YuviPanda>	 yeah, if we didn’t want to get rid of wikitech, setting up a YAML backed page with a UI for both puppetclasses + puppetvars would work with an ENC
[15:22:54] <paravoid>	 since you need a UI that allows people to pick puppet classes they want applied to hosts
[15:23:10] <paravoid>	 whether that's wikitech, a horizon module or a third-party app
[15:23:14] <YuviPanda>	 hmm
[15:23:32] <icinga-wm>	 PROBLEM - Host thulium is DOWN: PING CRITICAL - Packet loss = 100%  
[15:24:22] <YuviPanda>	 paravoid: that actually sounds like a nice solution, and will also give us ways to define roles / puppetvars for project-wide application
[15:24:30] <_joe_>	 why do I get paged for machines I don't have access to?
[15:24:33] <YuviPanda>	 hmm, and we can probably get rid of puppetvars, and just put those in hiera instead.
[15:24:48] <_joe_>	 YuviPanda: we should, in fact
[15:24:52] <paravoid>	 Jeff_Green: ping
[15:24:53] <YuviPanda>	 paravoid: the only problem with all this is that we don’t have a way to test any changes to wikitech atm.
[15:25:35] <mark>	 thulium is frack right?
[15:25:39] <paravoid>	 yes
[15:25:43] <_joe_>	 right
[15:26:09] <Coren>	 YuviPanda: Andrew's test wikitech instance is broken?
[15:26:18] <YuviPanda>	 Coren: has been for a while.
[15:26:24] <Jeff_Green>	 paravoid: hi
[15:26:37] <paravoid>	 Jeff_Green: we are getting paged about thulium being down
[15:27:01] <Jeff_Green>	 yeah, i updated the kernel and rebooted, and the root partition is corrupt
[15:27:23] <mark>	 did you set downtime?
[15:27:26] <Jeff_Green>	 i already marked it down for maintenance in icinga, not sure how to stop it paging
[15:27:33] <mark>	 hm
[15:27:54] <akosiaris>	 it should not page anymore then
[15:28:07] <Jeff_Green>	 I've been wondering why the maintenance mode doesn't work in icinga, could it be because it's all passive checks?
[15:28:27] <_joe_>	 it works, I have used it extensively
[15:29:07] <Jeff_Green>	 did I do the wrong host or something?
[15:30:10] <Jeff_Green>	 nope, thulium
[15:30:24] <Jeff_Green>	 i selected all services and scheduled downtime for all of them
[15:31:05] <icinga-wm>	 RECOVERY - Host thulium is UP: PING OK - Packet loss = 0%, RTA = 2.76 ms  
[15:31:07] <Jeff_Green>	 should be good for another 45 min, yet it's still paging
[15:32:00] <akosiaris>	 but not the server, right ? 
[15:32:19] <robh>	 yea thats gotten me in past
[15:32:23] <robh>	 gotta do both services and server =P
[15:32:24] <Jeff_Green>	 akosiaris: where do you do the server?
[15:32:37] <robh>	 host summary view iirc
[15:33:07] <akosiaris>	 if you click on the server name it has Host Commands on the right and a "Schedule downtime for this host and all services"
[15:33:19] <YuviPanda>	 paravoid: Coren filed https://phabricator.wikimedia.org/T85279?workflow=create
[15:33:46] <Jeff_Green>	 I'm probably a buffoon, but I don't see anything to that effect
[15:33:57] <akosiaris>	 I did it
[15:34:11] <akosiaris>	 it now added the ballon and the grey/more grey thingy
[15:34:18] <akosiaris>	 next to the server name
[15:34:25] <Jeff_Green>	 ok but I'd like to know for future reference
[15:34:44] <Jeff_Green>	 i find the server summary by search, then click the server name
[15:35:03] <akosiaris>	 yes, not the services, the service name explicitly
[15:35:06] <Jeff_Green>	 from there I see a lot of "View..." but nothing about Schedule
[15:35:22] <Jeff_Green>	 oh there I see it
[15:35:27] <akosiaris>	 ok
[15:35:34] <Jeff_Green>	 ok great, I've been doing it wrong. this explains a lot :-)
[15:35:37] <Jeff_Green>	 thx
[15:35:46] <akosiaris>	 don't mention it
[15:36:10] <Jeff_Green>	 now back to figuring out how to repair the root partition :-)
[15:39:21] <grrrit-wm>	 (03PS3) 10Yuvipanda: shinken: Provide ldap information for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/181732 
[15:44:06] <grrrit-wm>	 (03PS1) 10Gage: strongswan: puppet module [puppet] - 10https://gerrit.wikimedia.org/r/181742 
[15:45:46] <YuviPanda>	 paravoid: I think I’m going to just hit LDAP for now. shouldn’t cause *that* much additional load, I think. Will move it over to the ENC one when we do move to using the ENC.
[15:46:00] <paravoid>	 it's not about of load at all
[15:46:06] <paravoid>	 s/of//
[15:46:25] <YuviPanda>	 true, but I don’t want to switch to one thing now (nova + puppetmaster) and then switch again.
[15:46:39] <YuviPanda>	 hmm
[15:46:43] <YuviPanda>	 it won’t be *that* much of a switch
[15:47:00] <YuviPanda>	 the *real* reason, of course, is I’m lazy, and wanted this so I can start writing checks against roles :)
[15:47:14] <paravoid>	 :-)
[15:47:14] <YuviPanda>	 and don’t want to start yakshaving now.
[15:47:24] <paravoid>	 no opinion
[15:47:27] <paravoid>	 I don't care all that much :P
[15:47:50] <YuviPanda>	 ah, heh :)
[15:48:30] <Coren>	 "Yak shaving"?
[15:52:18] <_joe_>	 YuviPanda: yak shaving meaning?
[15:52:49] <YuviPanda>	 https://en.wiktionary.org/wiki/yak_shaving
[15:53:02] <YuviPanda>	 uh
[15:53:04] <YuviPanda>	 that’s a terrible page
[15:53:22] <YuviPanda>	 http://www.urbandictionary.com/define.php?term=yak%20shaving
[15:53:25] <YuviPanda>	 >  Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on. 
[15:53:42] <YuviPanda>	 original story I remember hearing involves someone deciding to knit a sweater
[15:53:55] <YuviPanda>	 and then 4 days later finds themselves in the freezing himalayas trying to shave a yak
[15:54:27] <YuviPanda>	 Coren: _joe_ ^
[15:54:46] <Coren>	 Ah.
[15:55:57] <YuviPanda>	 Coren: want to +1? https://gerrit.wikimedia.org/r/#/c/181732/
[15:56:36] <YuviPanda>	 this creates hostgroups with all the puppet roles ever applied directly to a node, making it easy to target nodes with particular set of roles for monitoring
[15:56:45] <YuviPanda>	 all tools proxies should have a check_http, for example
[15:57:38] <YuviPanda>	 hmm, my power is going to die. brb.
[15:57:57] <grrrit-wm>	 (03CR) 10coren: [C: 031] "That looks (a) sane, and (b) like a good idea" [puppet] - 10https://gerrit.wikimedia.org/r/181732 (owner: 10Yuvipanda)
[15:58:32] <_joe_>	 lol
[16:08:18] <Coren>	 Gah!  Someone changed something on the puppetmaster config that breaks everything.
[16:08:36] * Coren hunts for the changeset.
[16:33:15] <Coren>	 _joe_: Puppet fails in labs because " Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL" but the only reference to an ACL of that name I can find in puppet is a change you made some time ago to /modules/puppet/templates/auth-self.conf.erb which shouldn't matter to virt1000.  Any insight?
[16:45:39] <grrrit-wm>	 (03PS1) 10BBlack: Install mcelog and intel-microcode everywhere [puppet] - 10https://gerrit.wikimedia.org/r/181743 
[16:47:08] <bblack>	 sometimes yak shaving has negative connotations, too
[16:47:50] <bblack>	 like we when you start looking for a real bug, but end up correcting trivial nits all over the codebase as you go, generating lots of pointless commit traffic that does nothing to solve the real problem :)
[16:56:33] <YuviPanda|brb>	 bblack: that sounds more like yak nailpainting :)
[16:56:55] <YuviPanda|brb>	 Also I'm locked out of the house. 
[16:57:05] <Reedy>	 gj
[16:57:11] <Coren>	 YuviPanda|brb: That seems like a bad idea.  Why did you do that?
[16:58:04] <YuviPanda|brb>	 Coren: looks like everyone went to a Christmas party without telling me and took the keys 
[16:58:26] <YuviPanda|brb>	 So much fun 
[16:58:38] <bblack>	 http://www.capricorn.org/~akira/home/lockpick/mitlg-a4.pdf
[16:58:51] <Coren>	 Oh, if they're at a party I'm sure they /are/ having so much fun.  :-)
[16:59:23] <jgage>	 bblack i got in trouble in college for slipping printed copies of that under the door of everyone in my dorm ;)
[16:59:52] <bblack>	 I built my tools by grinding down tiny allen keys and such when I read that doc in highschool :)
[17:00:06] <jgage>	 street sweeper bristles!
[17:00:17] <YuviPanda|brb>	 Heh
[17:00:54] <YuviPanda|brb>	 So I get to sit outside and IRC from my.phone until that runs out
[17:09:21] <grrrit-wm>	 (03PS1) 10Sn1per: Add version number 3.0 to CC license Phab footer [puppet] - 10https://gerrit.wikimedia.org/r/181744 
[17:12:23] <Coren>	 Yuvi|LockedOut: At least, you are someplace where exposure is not an issue.
[17:25:58] <Yuvi|LockedOut>	 Coren: true except for the mosquitoes. Lots. And lots of mosquitoes
[17:29:03] <chasemp>	 I hear if you hop on one foot mosquitoes won't bite you
[17:29:09] <chasemp>	 just a thought in case you wanted to try it :)
[17:29:44] <bd808>	 I have this toy on my desk to remind me about yak shaving -- www.amazon.com/Palisades-Ren-Stimpy-Shaven-Action/dp/B000302CQM/ -- I tend to use the term in the negative connotation. Why shave a yak when you can just buy yarn?
[17:41:27] <grrrit-wm>	 (03PS1) 10RobH: setting mac info for codfw mw servers [puppet] - 10https://gerrit.wikimedia.org/r/181746 
[17:42:22] <grrrit-wm>	 (03CR) 10RobH: [C: 032] setting mac info for codfw mw servers [puppet] - 10https://gerrit.wikimedia.org/r/181746 (owner: 10RobH)
[18:13:08] <Yuvi|LockedOut>	 chasemp: heh. I usually just let them bite. Being zen about it is probably the only thing that's going to work
[18:13:22] <Yuvi|LockedOut>	 Also when these guys come back I'm going to murder them one by one slowly 
[18:14:43] <grrrit-wm>	 (03PS2) 10Rush: Add version number 3.0 to CC license Phab footer [puppet] - 10https://gerrit.wikimedia.org/r/181744 (owner: 10Sn1per)
[18:15:04] <grrrit-wm>	 (03CR) 10Rush: [C: 032] "per https://phabricator.wikimedia.org/T644#13818 this should be gtg" [puppet] - 10https://gerrit.wikimedia.org/r/181744 (owner: 10Sn1per)
[18:15:28] <grrrit-wm>	 (03CR) 10Rush: [V: 032] "per https://phabricator.wikimedia.org/T644#13818 this should be gtg" [puppet] - 10https://gerrit.wikimedia.org/r/181744 (owner: 10Sn1per)
[18:42:11] <paravoid>	 !log manually running debmirror on carbon to sync over the holidays; "pkill -f debmirror" should suffice if there is a problem
[18:42:15] <morebots>	 Logged the message, Master
[18:50:25] <Coren>	 paravoid: Syncing Jessie?
[19:10:20] <spagewmf>	  /last greg-g 
[19:36:04] <spagewmf>	 any deploy gods around? PageTriage is still pretty badly broken on enwiki. It's an easy 1-line fix https://gerrit.wikimedia.org/r/#/c/181753 , but no greg-g around.
[19:45:58] <spagewmf>	 YuviPanda|futex my browsers get locked out all the time
[19:46:11] <YuviPanda>	 spagewmf: heh :D
[20:04:10] <ori>	 spagewmf: just do it
[20:05:28] <spagewmf>	 ori: OK
[23:15:12] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 28847 seconds ago, expected  28800  
[23:20:16] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 29148 seconds ago, expected  28800  
[23:25:05] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 29448 seconds ago, expected  28800  
[23:30:06] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 29748 seconds ago, expected  28800  
[23:35:10] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 30047 seconds ago, expected  28800  
[23:40:09] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 30348 seconds ago, expected  28800  
[23:43:10] <icinga-wm>	 PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 1 failures  
[23:43:10] <icinga-wm>	 PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 2 failures  
[23:44:40] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[23:44:40] <icinga-wm>	 PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 2 failures  
[23:45:10] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 30648 seconds ago, expected  28800  
[23:48:22] <icinga-wm>	 PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 1 failures  
[23:48:44] <icinga-wm>	 PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures  
[23:49:28] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] Use LDAP as source for instance and project info [software/shinkengen] - 10https://gerrit.wikimedia.org/r/181737 (owner: 10Yuvipanda)
[23:49:42] <grrrit-wm>	 (03PS4) 10Yuvipanda: shinken: Provide ldap information for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/181732 
[23:50:11] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 30948 seconds ago, expected  28800  
[23:50:47] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] shinken: Provide ldap information for shinkengen [puppet] - 10https://gerrit.wikimedia.org/r/181732 (owner: 10Yuvipanda)
[23:54:37] <spagewmf>	 greg-g: you're not around, but Collaboration team is going to deploy a 2-line JS fix to PageTriage. I'll send e-mail
[23:55:17] <icinga-wm>	 PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet last ran 31247 seconds ago, expected  28800  
[23:55:20] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[23:57:08] <icinga-wm>	 RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[23:57:08] <icinga-wm>	 RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[23:58:48] <icinga-wm>	 RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[23:59:18] <icinga-wm>	 RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures