[02:04:53] <mutante>	 !log phabricator - add member Chad, make Chad project admin
[06:52:47] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%)  
[07:00:08] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[07:37:09] <YuviPanda>	 !ping
[07:37:09] <wm-bot>	 !pong
[08:40:07] <Ynhockey_>	 hi, is anyone here involved in the CatScan rewrite?
[08:41:09] <Ynhockey_>	 (or know how to contact Magnus Manske)
[09:21:05] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[09:25:53] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275#c1 (10Antoine "hashar" Musso (WMF)) Well operations/mediawiki-config has:    $ mwscript eval.php --wiki=enwiki   > print $wmfUdp2logDe...
[10:17:59] <Ynhockey>	 anyone either involved in CatScan rewrite or know how to contact Magnus Manske? (just asked same question but not sure if it registered in the channel)
[13:52:28] <^d>	 hashar: Thx for doing the puppetmaster/salt stuff for the 2 new elastic nodes.
[13:52:36] <^d>	 Was going to finish that this morning :)
[13:52:50] <hashar>	 ^d: you are welcome!
[13:53:01] <hashar>	 ^d: they probably aren't pooled properly unless puppet handle it
[13:53:13] <hashar>	 the whole ElasticSearch setup is a magical mystery to me
[13:53:24] <^d>	 Well it's not very magical, puppet does all the work :)
[13:53:36] <^d>	 I was stuck on ganglia which Filippo merged for us overnight.
[13:53:44] <^d>	 (Otherwise you wouldn't have gotten to the cert errors :))
[13:53:49] <hashar>	 I am sure :D
[13:54:09] <hashar>	 don't you like waking up with stuff fixed up overnight?  :-D
[13:54:17] <hashar>	 moree time to enjoy coffee and breakfast this way!
[13:54:26] <^d>	 Indeed :)
[13:55:09] <^d>	 Now I can decom elastic01-03 today since 05-07 are running happy :)
[13:56:49] <hashar>	 \O/
[15:06:38] <wikibugs>	 3Tool Labs tools / 3[other]: merl tools (tracking) - 10https://bugzilla.wikimedia.org/67556 (10merl)
[15:44:20] <andrewbogott>	 Coren: fyi, https://gerrit.wikimedia.org/r/#/c/167822/ <- this one seems to actually work
[16:00:48] <Coren>	 That seems...  not optimistic.  :-)
[16:02:12] <Coren>	 Thankfully, I don't overprovision ram on normal exec nodes, so my bunch of instances isn't making matters worse.
[16:02:41] <Coren>	 Then again, buffering.
[16:13:50] <Coren>	 Oh noes.  Sometimes I *HATE* git.
[16:22:31] <darkblue_b>	 hehe
[16:30:56] <valhallasw`cloud>	 Coren: reset --hard all the things? :-p
[16:32:20] <^d>	 !log deploment-prep decom'd and deleted elastic01-03. 05-07 are their replacements.
[17:14:19] <godog>	 btw diamond on testlabs-schedule-test4 is spamming
[17:14:23] <godog>	 testlabs-schedule-test4 : Oct 21 17:12:34 : diamond : unable to resolve host testlabs-schedule-test4
[17:18:02] <andrewbogott>	 Hm, I'll look.
[17:19:32] <andrewbogott>	 godog and/or YuviPanda where does diamond find its list of hosts?  Because that host was deleted a while ago
[17:21:30] <godog>	 andrewbogott: mh the message is originating from the host itself testlabs-schedule-test4 / 10.68.17.93 afaict
[17:21:45] <godog>	 there's no diamond list of hosts, it gets installed in every host and that's it
[17:21:47] <andrewbogott>	 ok….
[17:25:04] <andrewbogott>	 godog:  nova denies that any such instance exists :(  Is it still complaining?
[17:26:59] <godog>	 andrewbogott: looks like it :( Date: Tue, 21 Oct 2014 17:26:36 +0000
[17:27:07] <andrewbogott>	 what the heck
[17:27:35] <Tpt>	 andrewbogott: Hi! If you have some time, could you take a look to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/ProgVal , please?
[17:29:53] <andrewbogott>	 Tpt: added to tools project.  That's all you need, right?
[17:30:49] <Tpt>	 andrewbogott: yes, Thanks a lot!
[17:43:22] <wikibugs>	 3Wikimedia Labs: Discrepancy between enwiki_p.pagelinks on labs and production - 10https://bugzilla.wikimedia.org/71176 (10Russell Blau) s:5normal>3major
[17:43:54] <wikibugs>	 3Wikimedia Labs: Discrepancy between enwiki_p.pagelinks on labs and production - 10https://bugzilla.wikimedia.org/71176 (10Russell Blau)
[17:43:54] <wikibugs>	 3Wikimedia Labs / 3tools: Missing page revisions on enwiki - 10https://bugzilla.wikimedia.org/72226 (10Russell Blau)
[18:11:12] <mutante>	 Coren: i'd merge your change to firstboot.sh
[18:11:15] <mutante>	 want me to?
[18:11:38] <Coren>	 mutante: Which one?
[18:11:45] <mutante>	 https://gerrit.wikimedia.org/r/#/c/166221/1
[18:12:01] <mutante>	 more verbose output from parted
[18:12:41] <Coren>	 Ah, yes, useful for much debug.  That'll require rebuilding images though to be useful (and I want to sneak in a ssh key patch with Andrew before we do that).  But no harm in merging it now.
[18:12:52] <mutante>	 ok :) doing so
[18:16:29] <mutante>	 Coren: hah, another one just FYI, ever saw this? https://gerrit.wikimedia.org/r/#/c/15561/
[18:16:36] <mutante>	 and see the dates :p
[18:16:51] <mutante>	 it's all related to the SSL cert install
[18:16:57] <mutante>	 but wayy old
[18:17:14] <mutante>	 i think rebasing attempt will be hopeless for Roan :p
[18:17:31] <Coren>	 I'm pretty sure that changeset is doomed.
[18:32:38] <wikibugs>	 3Wikimedia Labs / 3tools: Missing page revisions on enwiki - 10https://bugzilla.wikimedia.org/72226 (10Russell Blau)
[18:32:39] <wikibugs>	 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Russell Blau)
[19:45:47] <mutante>	 Coren: https://gerrit.wikimedia.org/r/#/c/167889/1/manifests/role/labstools.pp
[19:47:11] <cscott>	 !log deployment-prep updated OCG to version 523c8123cd826c75240837c42aff6301032d8ff1
[20:07:54] <Coren>	 mutante: Might as well add codfw indeed.
[20:08:33] <Coren>	 But what I did instead is to point to ${::instanceproject}-master.${::site}}.wmflabs in my patch
[20:08:48] <Coren>	 mutante: https://gerrit.wikimedia.org/r/#/c/167852/
[20:14:29] <mutante>	 Coren: ah! i see, so what do you say, should i just abandon mine because it conflicts, or merge it and your other change needs rebase
[20:28:36] <Coren>	 I'd abandon it; it seems a little silly to merge in a change we know is going to be explicitly overwritten by the next one.  :-)
[20:29:36] <Coren>	 But also moar eyeballs on my patch would be good.
[20:42:08] <cscott>	 !log deployment-prep turned off puppet on deployment-pdf01, manually fixed broken /etc/ocg/mw-ocg-service.js
[20:42:30] <cscott>	 !log deployment-prep _joe_ promises to fix this properly tomorrow am
[20:47:53] <bd808>	 cscott: When you disabled puppet, did you happen to give a message? It's really handy to do `puppet agent --disable "describe the reason puppet is disabled"` so that attempts to run puppet echo the reason back.
[20:48:17] <cscott>	 bd808: now i know!
[20:48:28] <bd808>	 It's kind of a hidden secret
[20:48:29] <cscott>	 i actually just typed in the command which _joe_ told me to type
[20:48:35] <cscott>	 which didn't have a reason string
[20:48:38] * bd808 nods
[20:49:02] * bd808 will retrain the world one user at a time
[20:49:08] <cscott>	 i can probably enable and redisable puppet on the box if you think it would be helpful
[20:49:18] <cscott>	 i assume 'puppet agent --enable' works
[20:49:38] <cscott>	 bd808: do you know why the !log bot doesn't seem to be running in here?
[20:49:52] <bd808>	 it died and nobody fixed it :(
[20:50:08] <cscott>	 can I !log deployment-prep over in #ops?  will that work?
[20:50:12] <bd808>	 andrewbogott: can you fix the !log bot?
[20:50:26] <bd808>	 cscott: Nope, but you can just !log in #-qa
[20:50:32] <andrewbogott>	 um… yes, one second...
[20:50:44] <andrewbogott>	 morebots, you there?
[20:51:39] <andrewbogott>	 labs-morebots, yt?
[20:51:39] <labs-morebots>	 I am a logbot running on tools-exec-01.
[20:51:39] <labs-morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[20:51:39] <labs-morebots>	 To log a message, type !log <msg>.
[20:52:14] <bd808>	 !log logstash testing morebots
[20:52:20] <labs-morebots>	 Logged the message, Master
[20:53:06] <bd808>	 thanks andrewbogott :)
[21:13:02] <hashar>	 andrewbogott: that is a net split causing the issue again ?
[21:13:21] <andrewbogott>	 hashar: I presume so.  It always looks like it's running fine when I check the bot
[21:13:28] <andrewbogott>	 So I presume it's fine, but elsewhere
[21:17:12] <hashar>	 I guess
[21:27:29] <wikibugs>	 3Wikimedia Labs: Discrepancy between enwiki_p.pagelinks on labs and production - 10https://bugzilla.wikimedia.org/71176 (10Kunal Mehta (Legoktm))
[21:27:29] <wikibugs>	 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Kunal Mehta (Legoktm))
[21:27:30] <wikibugs>	 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Kunal Mehta (Legoktm))
[21:42:54] <matanya>	 are xtools down?
[21:44:14] <matanya>	 andrewbogott: ^ ?
[21:44:49] <andrewbogott>	 matanya: not that I know of.  What are you seeing?
[21:45:02] <matanya>	 blank page
[21:45:31] <andrewbogott>	 um...
[21:45:40] <andrewbogott>	 is xtools a particular tool, or a particular /kind/ or tool, or…?
[21:45:50] <matanya>	 https://tools.wmflabs.org/xtools/echo/
[21:45:52] <matanya>	 e.g
[21:46:53] <andrewbogott>	 Looks like the maintainers are Tparis, cyperpower678, hedonil
[21:47:51] <matanya>	 yes
[21:50:08] <andrewbogott>	 That's all I know :(
[22:16:36] <grrrit-wm1>	 (03PS1) 10Dzahn: add (fake) tor control password [labs/private] - 10https://gerrit.wikimedia.org/r/167965 
[22:17:52] <grrrit-wm1>	 (03CR) 10Dzahn: [C: 032] add (fake) tor control password [labs/private] - 10https://gerrit.wikimedia.org/r/167965 (owner: 10Dzahn)
[22:18:10] <grrrit-wm1>	 (03CR) 10Dzahn: [V: 032] add (fake) tor control password [labs/private] - 10https://gerrit.wikimedia.org/r/167965 (owner: 10Dzahn)