[00:24:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 33.33% of data above the critical threshold [0.0]
[01:14:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0]
[02:05:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 55.56% of data above the critical threshold [0.0]
[02:35:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0]
[05:26:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 44.44% of data above the critical threshold [0.0]
[05:51:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0]
[06:02:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 22.22% of data above the critical threshold [0.0]
[06:32:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0]
[06:53:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-trusty is CRITICAL 33.33% of data above the critical threshold [0.0]
[07:23:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-trusty is OK Less than 1.00% above the threshold [0.0]
[08:54:27] <shinken-wm>	 PROBLEM - Puppet staleness on tools-mailrelay-01 is CRITICAL 100.00% of data above the critical threshold [43200.0]
[09:28:33] <yuvipanda>	 !log tools cleared and depooled tools-exec-02 and -13. only job running was deadlocked for a long, long time (week)
[09:28:39] <labs-morebots>	 Logged the message, Master
[09:30:58] <shinken-wm>	 PROBLEM - Host tools-exec-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.31)
[09:32:57] <shinken-wm>	 PROBLEM - Host tools-exec-13 is DOWN: CRITICAL - Host Unreachable (10.68.17.184)
[12:10:53] <valhallasw`nuage>	 yuvipanda: it took me a while, but it's working now
[12:12:55] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Add rdbtools build script [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 
[12:23:40] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: Add rdbtools build script [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) 
[13:05:17] <valhallasw`nuage>	 Coren: errr, how do I build stuff in labs/toollabs?
[13:06:02] <valhallasw>	 I'm trying aclocal, autoconf, autoheader (error: AC_CONFIG_HEADERS not found in configure.ac), automake (error: required directory ./build-aux does not exist)
[13:15:03] <ashley>	 Coren: any idea what's going on in here -- my instance (http://social-tools.wmflabs.org/) is down, can't ssh to it or anything; instance status on Special:NovaInstance is "SHUTOFF" but clicking on the reboot link there fails with the error message "Failed to reboot instance social-tools1."
[13:16:34] * valhallasw just realized he had to actually read the error messages
[13:23:31] <valhallasw>	 not that that helped, apparently I needed the magic 'dpkg-buildpackage -B' instead
[13:34:58] <grrrit-wm>	 (03PS3) 10Merlijn van Deen: Add rdbtools build script [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) 
[14:13:11] <grrrit-wm>	 (03PS4) 10Merlijn van Deen: Add rdbtools build script [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) 
[14:13:13] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Add redistools package [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209975 (https://phabricator.wikimedia.org/T98641) 
[14:13:45] <valhallasw>	 well, that only took me multiple hours to get to work
[14:13:53] <valhallasw>	 hurray for debian package madness
[14:19:40] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 04-1] "sudo trick doesn't work, fixing..." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209975 (https://phabricator.wikimedia.org/T98641) (owner: 10Merlijn van Deen)
[15:16:42] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: Add redistools package [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209975 (https://phabricator.wikimedia.org/T98641) 
[15:16:44] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Add python-ipaddress package [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209978 (https://phabricator.wikimedia.org/T86015) 
[19:16:24] <yuvipanda>	 valhallasw: that script only makes sense on the redis hosts no? You could have / should have just put it in ops puppet and provisioned it to there...
[19:16:32] <valhallasw>	 yuvipanda: ??
[19:16:45] <valhallasw>	 yuvipanda: that's why it's a seperate package, smartass :P
[19:17:27] <yuvipanda>	 valhallasw: why a separate package? It is just one file...
[19:17:36] <valhallasw>	 because packaging
[19:17:42] <yuvipanda>	 But why 
[19:18:01] <yuvipanda>	 So every time you wanna change that you will have to fiddle with debian packaging 
[19:18:07] <valhallasw>	 because that seemed like the sensible thing to do
[19:18:17] <yuvipanda>	 Nope. Not for 1 file.
[19:18:21] <valhallasw>	 yes, and otherwise I have to fiddle with a puppet repository where it takes weeks to get a +2
[19:18:33] <valhallasw>	 so that's pretty much equal
[19:18:41] <valhallasw>	 two files
[19:18:43] <yuvipanda>	 That security patch is an anomaly I promise 
[19:18:44] <valhallasw>	 and dependencies
[19:19:08] <Krenair>	 can confirm, review on puppet for non-ops is ridiculous
[19:19:23] <yuvipanda>	 I do it whenever poked for tools stuff 
[19:19:46] <valhallasw>	 yuvipanda: you mean that sudo patch that actually hasn't been merged yet? :P
[19:20:09] <yuvipanda>	 valhallasw: yes because I last saw that on a weekend...
[19:20:17] <yuvipanda>	 And it needed rebasing.
[19:20:27] <valhallasw>	 which you were going to do :p
[19:20:32] <yuvipanda>	 On Monday 
[19:20:38] <valhallasw>	 anyway.
[19:20:58] <valhallasw>	 the same arguments can be made for most scripts in that deb package
[19:21:07] <valhallasw>	 in the end, it doesn't really matter
[19:21:15] <valhallasw>	 if you want it in ops/puppet, please go ahead and move it
[19:21:26] <yuvipanda>	 Yes everything that's not a file that needs compiling there needs to move I think 
[19:21:38] <yuvipanda>	 Unless it has advantages of being in a python package 
[19:21:56] <valhallasw>	 they're all not packages, but scripts
[19:21:59] <Krenair>	 It took me adding someone who had just quickly uploaded merged their own dotfiles changes as a reviewer to get my own .gitconfig patch through -.-
[19:22:03] <yuvipanda>	 I moved all the webservice related scripts out of it when I touched them 
[19:22:17] <valhallasw>	 I'd rather move everything to the .deb than to ops/puppet tbh
[19:22:27] <valhallasw>	 but also there I can't really be bothered
[19:22:44] <valhallasw>	 I'd even rather have it as a python package, then .deb-ifying that, I think
[19:22:49] <yuvipanda>	 I'll move it around and merge on Monday 
[19:22:59] <valhallasw>	 because then we still have sane dependency management /and/ we get an easy local development env
[19:23:00] <Krenair>	 and I still have two open puppet patches for other trivial things
[19:23:21] <valhallasw>	 You should just self-merge within five minutes! [/oh wait]
[19:23:27] <yuvipanda>	 Meh
[19:23:33] <yuvipanda>	 I'm going back to bed
[19:24:06] <valhallasw>	 yuvipanda: I might come over annoyed, because I am somewhat annoyed after spending hours to get python-to-.deb working sensibly
[19:25:01] <valhallasw>	 which doesn't relate to you, obviously, just to stdeb
[19:49:55] <wikibugs>	 10Tool-Labs: make https://dumps.wikimedia.org/other/wikidata/ available on tool labs - https://phabricator.wikimedia.org/T98655#1275135 (10valhallasw) p:5Triage>3Normal
[19:50:15] <wikibugs>	 10Tool-Labs: Clean up huge logs on toollabs - https://phabricator.wikimedia.org/T98652#1275137 (10valhallasw) p:5Triage>3High
[19:50:25] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Puppetize redis usage reporting tools - https://phabricator.wikimedia.org/T98641#1275138 (10valhallasw) p:5Triage>3Normal
[19:50:46] <wikibugs>	 10Tool-Labs: Convert tomcat-starter to python - https://phabricator.wikimedia.org/T98442#1275139 (10valhallasw) p:5Triage>3Normal
[19:50:52] <wikibugs>	 10Tool-Labs: Unify / simplify webservice code - https://phabricator.wikimedia.org/T98440#1275141 (10valhallasw) p:5Triage>3Normal
[19:52:04] <wikibugs>	 10Tool-Labs: xtools-ec has multiple webservices running - https://phabricator.wikimedia.org/T98432#1275144 (10valhallasw) p:5Triage>3High Currently 17.
[19:52:08] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1275146 (10valhallasw) p:5Triage>3Low
[19:52:37] <wikibugs>	 10Tool-Labs: Add shinken admin accounts for tools ops - https://phabricator.wikimedia.org/T97862#1275150 (10valhallasw) p:5Triage>3Normal a:3yuvipanda
[20:00:23] <wikibugs>	 10Tool-Labs: Convert lighttpd-starter from bash to python - https://phabricator.wikimedia.org/T98441#1275160 (10valhallasw) p:5Triage>3Normal
[20:03:17] <wikibugs>	 10Tool-Labs: Tool Labs: Install php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1275165 (10valhallasw)
[20:03:18] <wikibugs>	 10Tool-Labs, 7Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#1275164 (10valhallasw)
[20:03:54] <wikibugs>	 10Tool-Labs: Tool Labs: Install php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1275166 (10valhallasw) p:5Triage>3High
[20:04:04] <wikibugs>	 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Setup a tools checker service that can check all internal services for availability - https://phabricator.wikimedia.org/T97748#1275168 (10valhallasw) p:5Triage>3Normal
[20:04:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 33.33% of data above the critical threshold [0.0]
[20:04:57] <wikibugs>	 10Tool-Labs: New tool creation broken - https://phabricator.wikimedia.org/T97740#1275172 (10valhallasw) p:5Triage>3Unbreak! Is this fixed now?
[20:07:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:07:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 55.56% of data above the critical threshold [0.0]
[20:07:24] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:07:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:07:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:07:51] <yuvipanda>	 meh
[20:07:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:08:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:08:37] <wikibugs>	 10Tool-Labs: Fix oscillation between 'purged' and 'latest' for several packages on toollabs - https://phabricator.wikimedia.org/T97628#1275177 (10valhallasw) p:5Triage>3High ``` valhallasw@tools-bastion-01:/var/log$ aptitude show libtime-local-perl tclthread libthreads-shared-perl libthreads-perl No current...
[20:08:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:08:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:09:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:09:14] <valhallasw>	 ...
[20:09:20] <valhallasw>	 nfs/ldap broken again? :P
[20:09:23] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 33.33% of data above the critical threshold [0.0]
[20:09:24] <yuvipanda>	 valhallasw: no
[20:09:26] <valhallasw>	 or my broken patch...
[20:09:28] <yuvipanda>	 valhallasw: python3-flake8
[20:09:29] <valhallasw>	 ...bah
[20:09:31] <valhallasw>	 oh
[20:09:35] <yuvipanda>	 am committing a followup
[20:09:35] <valhallasw>	 that only works for 14xx
[20:09:37] <valhallasw>	 doh.
[20:09:37] <yuvipanda>	 precise
[20:09:38] <yuvipanda>	 yeah
[20:09:39] <valhallasw>	 sorry :(
[20:09:48] <valhallasw>	 I think https://phabricator.wikimedia.org/T97628 is also precise v trusty
[20:10:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL 62.50% of data above the critical threshold [0.0]
[20:11:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:11:27] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:12:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:12:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:14:27] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:14:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL 75.00% of data above the critical threshold [0.0]
[20:16:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:17:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 44.44% of data above the critical threshold [0.0]
[20:17:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:17:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:18:13] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL 22.22% of data above the critical threshold [0.0]
[20:19:53] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:21:01] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Fix oscillation between 'purged' and 'latest' for several packages on toollabs - https://phabricator.wikimedia.org/T97628#1275198 (10valhallasw) a:3valhallasw
[20:21:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:23:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL 55.56% of data above the critical threshold [0.0]
[20:24:22] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 66.67% of data above the critical threshold [0.0]
[20:27:40] <wikibugs>	 10Tool-Labs: Migrate individual tools to trusty to relieve pressure on older precise nodes - https://phabricator.wikimedia.org/T88228#1275203 (10valhallasw) p:5Triage>3Normal
[20:28:20] <andrewbogott>	 dammit, all those alerts and as soon as I look everything is fine
[20:29:19] <wikibugs>	 10Tool-Labs: Package mwlib on tool labs - https://phabricator.wikimedia.org/T98712#1275205 (10valhallasw) 3NEW
[20:29:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0]
[20:30:49] <yuvipanda>	 andrewbogott_afk: sorry, that was just us
[20:32:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1214 is OK Less than 1.00% above the threshold [0.0]
[20:32:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0]
[20:32:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1201 is OK Less than 1.00% above the threshold [0.0]
[20:32:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1219 is OK Less than 1.00% above the threshold [0.0]
[20:33:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0]
[20:33:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1213 is OK Less than 1.00% above the threshold [0.0]
[20:33:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1203 is OK Less than 1.00% above the threshold [0.0]
[20:35:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1218 is OK Less than 1.00% above the threshold [0.0]
[20:36:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0]
[20:36:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1206 is OK Less than 1.00% above the threshold [0.0]
[20:37:08] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1210 is OK Less than 1.00% above the threshold [0.0]
[20:37:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK Less than 1.00% above the threshold [0.0]
[20:37:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK Less than 1.00% above the threshold [0.0]
[20:38:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1205 is OK Less than 1.00% above the threshold [0.0]
[20:39:00] <wikibugs>	 10Tool-Labs: Package mwlib on tool labs - https://phabricator.wikimedia.org/T98712#1275220 (10valhallasw) p:5Triage>3Low I'm not sure if we can do this, because of the following dependency:   * pyparsing>=1.4.11,<1.6  while trusty bundles   * python-pyparsing 2.0.1+dfsg1-1build1  We also need to build:  * ti...
[20:39:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK Less than 1.00% above the threshold [0.0]
[20:39:18] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK Less than 1.00% above the threshold [0.0]
[20:39:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0]
[20:39:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK Less than 1.00% above the threshold [0.0]
[20:40:03] <wikibugs>	 10Tool-Labs: Useful graphite metrics to be tracked for Tool labs (tracking) - https://phabricator.wikimedia.org/T69879#1275222 (10valhallasw) p:5Triage>3Normal
[20:41:33] <wikibugs>	 10Tool-Labs: Package mwlib on tool labs - https://phabricator.wikimedia.org/T98712#1275231 (10yuvipanda) I'm inclined to close this as declined. Use virtualenv.
[20:41:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1202 is OK Less than 1.00% above the threshold [0.0]
[20:42:31] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1215 is OK Less than 1.00% above the threshold [0.0]
[20:42:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK Less than 1.00% above the threshold [0.0]
[20:42:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1208 is OK Less than 1.00% above the threshold [0.0]
[20:43:52] <wikibugs>	 10Tool-Labs: zoomviewer seems to be down - https://phabricator.wikimedia.org/T97790#1275244 (10valhallasw) 5Open>3Resolved a:3valhallasw
[20:44:37] <wikibugs>	 6Labs, 10Tool-Labs: Unable to "Create New Tool" from tools.wmflabs.org webpage - https://phabricator.wikimedia.org/T91246#1275255 (10valhallasw) 5Open>3Resolved a:3valhallasw
[20:44:46] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Improve & force sudo lecture - https://phabricator.wikimedia.org/T95882#1275257 (10valhallasw) 5Open>3Resolved
[20:44:51] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK Less than 1.00% above the threshold [0.0]
[20:46:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0]
[20:47:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0]
[20:48:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1204 is OK Less than 1.00% above the threshold [0.0]
[20:49:25] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK Less than 1.00% above the threshold [0.0]
[21:07:27] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Fix oscillation between 'purged' and 'latest' for several packages on toollabs - https://phabricator.wikimedia.org/T97628#1275269 (10valhallasw) 5Open>3Resolved
[21:09:19] <wikibugs>	 10Tool-Labs: Package mwlib on tool labs - https://phabricator.wikimedia.org/T98712#1275271 (10Ricordisamoa) 5Open>3declined a:3Ricordisamoa Use virtualenv, at least before T95142 is implemented.
[21:09:46] <wikibugs>	 10Tool-Labs: Package mwlib on tool labs - https://phabricator.wikimedia.org/T98712#1275277 (10Ricordisamoa) a:5Ricordisamoa>3None
[21:12:44] <valhallasw>	 yuvipanda: I sinned today. I found one existing platform too complicated, and one not good enough, so I wrote something inbetween
[21:12:53] <valhallasw>	 :D
[21:13:03] <valhallasw>	 (the scripts that build packages in toollabs/packages)
[21:13:29] <yuvipanda>	 link?
[21:14:00] <valhallasw>	 https://gerrit.wikimedia.org/r/#/c/209968/4/packages/rdbtools.sh
[21:14:51] <valhallasw>	 if you're wondering 'why bash', the answer is 'lsb-release', if you're wondering 'why write stdeb.cfg', the answer is 'because stdeb doesn't actually do command-line arguments very well'
[21:15:16] <grrrit-wm>	 (03CR) 10Yuvipanda: "I'll just wait on other people to comment :)" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) (owner: 10Merlijn van Deen)
[21:15:34] <yuvipanda>	 valhallasw: I added a bunch of people
[21:15:58] <valhallasw>	 (there's probably a way to read exports in python somehow, but I couldn't really be bothered to find out how)
[21:16:04] <grrrit-wm>	 (03CR) 10Yuvipanda: "I think the current traditional way (for things in prod) is to import it into gerrit and put things there." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) (owner: 10Merlijn van Deen)
[21:16:24] <valhallasw>	 ...that's what I tried first, remember :P
[21:17:24] <valhallasw>	 but that's really labor-intensive to get set up
[21:17:55] <valhallasw>	 and forces you to do things the traditional debian way (update changelog, etc)
[21:18:44] <yuvipanda>	 valhallasw: it does, it does. People consider that the right thing
[21:19:10] <yuvipanda>	 I have +0 opinions on all htese
[21:19:15] <yuvipanda>	 added people who do have opinions
[21:19:34] <valhallasw>	 what I wrote corresponds to what we currently do
[21:19:45] <valhallasw>	 except for the part where I wrote a script in source control instead of dumping a .deb :P
[21:20:14] <yuvipanda>	 probably
[21:20:45] <yuvipanda>	 I explicitly don’t have an opinion so I don’t have to think / argue about this. 
[21:20:53] <valhallasw>	 :D
[21:20:57] <yuvipanda>	 I will say that labs/toollabs is already containing too many unrelated things.
[21:21:04] <yuvipanda>	 and then I’ll go take a shower and hunt for food
[21:21:09] <yuvipanda>	 or continue watching Justice League...
[21:21:10] <yuvipanda>	 unsure
[21:21:24] <yuvipanda>	 but one thing I’ll not do is offer opinions on ways to build debian packages :P
[21:21:30] <eranroz>	 how to kill bigbrother/ stop it from flodding on fail to start a job I already removed from .bigbrotherrc?
[21:21:33] <yuvipanda>	 been there, done that, had my fights
[21:21:35] <valhallasw>	 it contains stuff that builds .debs, plus the website, but you also wanted to debianize that I think
[21:22:03] <yuvipanda>	 eranroz: sadly you can’t. it needs a bigbrother restart, which I just did
[21:22:06] <yuvipanda>	 so it should be gone now
[21:22:23] <eranroz>	 thanks :)
[21:22:24] <yuvipanda>	 I was hoping to finish up all of service manifests by now but lots more firefighting
[21:22:25] <yuvipanda>	 oh well
[21:26:56] <yuvipanda>	 valhallasw: started a page to document service manifests: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Service_manifests
[21:26:58] <yuvipanda>	 needs more work
[21:27:04] <valhallasw>	 yuvipanda: \o/
[21:27:22] <valhallasw>	 yuvipanda: I'm still thinking about switching to something like sphinx
[21:27:56] <yuvipanda>	 docs? mmm
[21:28:06] <yuvipanda>	 spagewmf is working on something that doesn’t suck, I’m told
[21:28:08] <yuvipanda>	 not sure what that is
[21:28:50] <valhallasw>	 yuvipanda: main advantage is that we can move it into the same repo as the puppet classes, so doc changes and infra changes are together
[21:29:11] <yuvipanda>	 valhallasw: I think there might already be a CI job that does that, but is also… eugh
[21:29:19] <yuvipanda>	 no idea where it is :)
[21:29:32] <valhallasw>	 although part of that is in that /toollabs rep, so that doesn't work completely
[21:29:36] <valhallasw>	 ugh.
[21:29:46] <yuvipanda>	 valhallasw: no, service manifests are not in toollabs repo. they’re in tools-services.
[21:30:00] <valhallasw>	 'part of that' = part of our infra/scripts
[21:30:03] <yuvipanda>	 right
[21:30:40] <valhallasw>	 you mean services.pp, I assume?
[21:31:18] <valhallasw>	 it's in a tools-manifest .deb, apparently? *confused*
[21:31:37] <yuvipanda>	 https://github.com/wikimedia/operations-software-tools-manifest
[21:31:53] <valhallasw>	 I... see.
[21:32:37] <yuvipanda>	 anyway, these are all just python packages + init scripts
[21:33:03] <yuvipanda>	 I think ideal is: if it is 1 script + 3rd party package, put it in ops/puppet, else make its own repo and build deb
[21:33:38] <valhallasw>	 as in: put that build tool in ops/puppet? or?
[21:33:51] <valhallasw>	 because maintaining our own repo for something we don't really maintain is a pita
[21:33:51] <yuvipanda>	 what ‘build tool’?
[21:34:11] <valhallasw>	 https://gerrit.wikimedia.org/r/#/c/209968/4/packages/rdbtools.sh
[21:34:29] <yuvipanda>	 I dunno, you just pull from upstream and push if you want to.
[21:34:37] <yuvipanda>	 it’s also a requirement in prod because you can’t hit the wider internet
[21:34:39] <yuvipanda>	 to clone things
[21:34:49] <yuvipanda>	 so you don’t really have a choice there :)
[21:35:05] <valhallasw>	 yeah, but prod is irrelevant, because the packages can't be pushed to prod anyway
[21:35:16] <yuvipanda>	 we don’t have policies set in labs / tools, so we’ve been flaking and just doing it manually and putting it on the repo
[21:35:28] <valhallasw>	 I'm ok with using prod policies, though, I guess.
[21:35:41] <yuvipanda>	 I think we should, as close to possible
[21:35:51] <yuvipanda>	 but we already have a bunch of debs that I have no idea how they were built...
[21:35:57] <valhallasw>	 *nod*
[21:36:18] <valhallasw>	 however, git-buildpackage is super complicated, so I'm not really happy with that
[21:36:24] <yuvipanda>	 you don’t need git-buildpackage
[21:36:30] <yuvipanda>	 you can just do what you did on your git repo
[21:36:33] <yuvipanda>	 have a debian/ folder
[21:36:37] <yuvipanda>	 make it a ‘native’ package
[21:36:40] <yuvipanda>	 that’s what I do too
[21:36:46] <yuvipanda>	 and just put it on gerrit
[21:36:46] <yuvipanda>	 instead of github
[21:36:57] <yuvipanda>	 so you can just pull and rebase whenever you want to update / rebuild
[21:37:13] <yuvipanda>	 this works as long as you don’t have to maintain local patches, which we shouldn’t anyway.
[21:37:43] <yuvipanda>	 the worst part is that you can’t create gerrit repos yourself, but I can so I can do that if you want to
[21:40:03] <yuvipanda>	 valhallasw: that would be a valid strategy in prod too
[21:41:21] <valhallasw>	 *nod* ok
[21:41:35] <valhallasw>	 just copy the /debian dir from another repo, hack and slash, commit
[21:41:37] <valhallasw>	 that's doable
[21:41:48] <yuvipanda>	 or just use stdeb to generate it, hack slash, commit :)
[21:41:51] <yuvipanda>	 whatever you prefer
[21:41:54] <yuvipanda>	 debian dir is debian dir
[21:42:00] <valhallasw>	 yuvipanda: except the stdeb version is apparently crappy
[21:42:11] <yuvipanda>	 good starting point to hack out :)
[21:42:12] <valhallasw>	 also, we should get build infra
[21:42:32] <valhallasw>	 not necessarily bettre than 'other package that already had shit hacked out' :P
[21:42:36] <yuvipanda>	 tools-dev should suffice? or are you thinking of something else?
[21:42:44] <valhallasw>	 we need trusty and precise
[21:42:48] <yuvipanda>	 oh yes
[21:42:56] <yuvipanda>	 in fact, we need a tools-precise host
[21:42:59] <yuvipanda>	 no bastions with precise atm
[21:43:15] <valhallasw>	 and I need to figure out a way to ssh within labs because non-mosh-ssh is undoable from here
[21:43:25] <valhallasw>	 and mosh doesn't do agent forwarding
[21:43:30] <valhallasw>	 :{
[21:43:43] <yuvipanda>	 it is worse from europe than from India? :(
[21:44:06] <wikibugs>	 10Tool-Labs: New tool creation broken - https://phabricator.wikimedia.org/T97740#1275293 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Oops, yes, fixed.
[21:44:40] <valhallasw>	 yuvipanda: dunno, but it was horribly slow. Often multi-second lags etc.
[21:44:43] <wikibugs>	 10Tool-Labs: Setup tools-precise as precise bastion - https://phabricator.wikimedia.org/T98713#1275296 (10yuvipanda) 3NEW
[21:44:50] <yuvipanda>	 valhallasw: hmm, definitely sounds worse than from here...
[21:44:50] <yuvipanda>	 err
[21:44:51] <yuvipanda>	 from India
[21:44:53] <valhallasw>	 with mosh, I only have to wait for tabs
[21:45:12] <yuvipanda>	 you should ask Tim L what he does
[21:45:26] <yuvipanda>	 valhallasw: hmm, HBA? so then you can mosh to tools-login and then ssh to all other hosts from there
[21:45:31] <yuvipanda>	 if HBA is setup properly
[21:45:51] <wikibugs>	 10Tool-Labs: Setup tools-precise as precise bastion - https://phabricator.wikimedia.org/T98713#1275303 (10yuvipanda) Or we don't as incentive to move people to trusty but that seems a bit backhanded.
[21:46:06] <valhallasw>	 that would be good, yes
[21:47:15] <yuvipanda>	 so that should solve problems at least inside of toollabs
[21:47:40] <yuvipanda>	 another thing would be to allow / disallow HBA from bastion to all of labs - this we can also do beause IIRC we lock down root on bastion project as well
[21:48:00] <valhallasw>	 uhuh
[21:49:49] <wikibugs>	 10Tool-Labs: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275308 (10valhallasw) 3NEW
[21:50:11] <wikibugs>	 10Tool-Labs: Setup tools-precise as precise bastion - https://phabricator.wikimedia.org/T98713#1275316 (10valhallasw) p:5Triage>3High
[21:51:07] <valhallasw>	 yuvipanda: protip: create tasks in the backlog column by using the create task butting here: https://phabricator.wikimedia.org/project/view/539/
[21:51:15] <valhallasw>	 also set priority ;-)
[21:51:38] <valhallasw>	 (I'm ok with triaging every now and then, but it helps to keep the triage column a bit cleaner)
[21:51:41] <yuvipanda>	 I dunno, maybe we *should* not have a precise bastion
[21:51:45] <yuvipanda>	 err
[21:51:47] <yuvipanda>	 *not*
[21:51:55] <yuvipanda>	 hmm
[21:52:03] <yuvipanda>	 ok that’s backhanded.
[21:52:16] <valhallasw>	 I'll rename it to a -dev host :P
[21:52:20] <yuvipanda>	 valhallasw: will do! I rarely use that view, so I guess I don’t bothe ras much
[21:52:33] <yuvipanda>	 valhallasw: hmm, tools-precise-dev? and not give it a public IP, maybe?
[21:53:05] <valhallasw>	 *nod* v
[21:53:07] <yuvipanda>	 valhallasw: https://phabricator.wikimedia.org/tag/toollabs-goals-q4/ is what I’m keeping clean
[21:53:08] <wikibugs>	 10Tool-Labs: Setup tools-dev-precise as precise dev host - https://phabricator.wikimedia.org/T98713#1275321 (10valhallasw)
[21:53:12] <yuvipanda>	 valhallasw: wanna do it? :)
[21:53:27] <valhallasw>	 I'm going to bed, but I can push some buttons
[21:53:49] <yuvipanda>	 valhallasw: nah, go to bed :) I’ll get it started
[21:53:52] <valhallasw>	 ok
[21:53:57] <yuvipanda>	 valhallasw: thanks! <3 have a good night
[21:54:01] <valhallasw>	 thx
[22:06:41] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "In general, I'm a huge fan of reproducibility, so any script is better than none :-). But relying on external sources (that may go away) " [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) (owner: 10Merlijn van Deen)
[22:07:59] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "(labs/toollabs/deb/* would be fine as well, just the repository => package mechanism should be common.)" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209968 (https://phabricator.wikimedia.org/T98641) (owner: 10Merlijn van Deen)
[22:08:45] <yuvipanda>	 !log tools created tools-precise-dev instance
[22:08:49] <labs-morebots>	 Logged the message, Master
[22:19:48] <Earwig>	 Coren: are you around?
[22:21:24] <wikibugs>	 10Tool-Labs: xtools-ec has multiple webservices running - https://phabricator.wikimedia.org/T98432#1275344 (10scfc) 5Open>3Resolved a:3scfc There was a `cron` job (the infamous `webwatcher.sh`).  I disabled it with the note:  ``` # Disabled per https://phabricator.wikimedia.org/T98432.  Your # webservice i...
[22:22:26] <yuvipanda>	 Earwig: hey! I’m around if you need anything
[22:23:46] <Earwig>	 howdy, but I'm not even sure if this is a labs issue at all... and I'm not sure who else than Coren would know what's up
[22:24:01] <Earwig>	 it's regarding the Yahoo BOSS thing
[22:24:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 20.00% of data above the critical threshold [0.0]
[22:25:39] <yuvipanda>	 ah
[22:25:50] <yuvipanda>	 then I’ll just let you wait for him :)
[22:25:54] <Earwig>	 alright
[22:27:11] <Earwig>	 I think I'll send him an email
[22:29:36] <shinken-wm>	 RECOVERY - Puppet failure on tools-precise-dev is OK Less than 1.00% above the threshold [0.0]
[22:33:12] <wikibugs>	 10Tool-Labs: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275350 (10scfc) That's a bit misleading: It should already be enabled on all exec nodes & Co., but that has to be done manually, and where it hasn't been done, that was probably an oversight.  So: "Make sure that HBA i...
[22:37:02] <wikibugs>	 10Tool-Labs: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275352 (10yuvipanda) I think we should just find a way to have it set automatically for all of the exec hosts.
[22:40:44] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275359 (10yuvipanda) Untested, not sure how it'll interact with denying access to some hosts.
[22:45:28] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275363 (10valhallasw) This is not just for exec nodes; I'd also like to be able to ssh into e.g. -mail and -redis.
[22:46:09] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275364 (10yuvipanda) Yes, so ideally we'll enable hba for all hosts and just disable it for people not in tools.admin for non-exec nodes.
[22:46:31] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275365 (10yuvipanda) Come to think of it, why do we allow HBA to exec nodes for users?
[23:18:05] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275386 (10scfc) I don't know if I ever asked, but I assumed it was so that debugging grid jobs/web services didn't require you to set up proper `ProxyCommand`s, but you could just `ssh` to `tools-l...
[23:20:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 22.22% of data above the critical threshold [0.0]
[23:27:33] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: setup host-based auth for tools hosts - https://phabricator.wikimedia.org/T98714#1275387 (10yuvipanda) Hmm, right - so I guess the user can attach a debugger or somesuch to it. However, I'm not sure if that's what people do - vs just running it on tools-login / dev and just atta...