[00:01:51] <icinga-wm>	 RECOVERY - Disk space on ms-be7 is OK: DISK OK  
[00:02:54] <logmsgbot>	 !log maxsem synchronized php-1.23wmf9/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/105108'
[00:03:14] <morebots>	 Logged the message, Master
[00:03:58] <MaxSem>	 I'm done
[00:04:27] <greg-g>	 woohoo
[00:21:23] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 032] Set $wgULSFontRepositoryBasePath to protocol-relative URL [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105115 (owner: 10Ori.livneh)
[00:21:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set $wgULSFontRepositoryBasePath to protocol-relative URL [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105115 (owner: 10Ori.livneh)
[00:23:23] <logmsgbot>	 !log tstarling synchronized wmf-config/CommonSettings.php
[00:23:53] <morebots>	 Logged the message, Master
[00:25:05] <ori>	 TimStarling: thanks
[00:28:34] <TimStarling>	 ok, well it still aborts
[00:31:04] <TimStarling>	 it aborts the connection and then starts a new request for the same URL after 100ms
[00:36:12] <TimStarling>	 do you think we could check connection_aborted() during MW request shutdown, and log to a special channel if it is?
[00:45:08] <ori>	 TimStarling: maybe, do you have any reason to suspect the problem is prevalent?
[00:45:38] <ori>	 I haven't been able to reproduce it, so it could still be a browser bug
[00:51:14] <paravoid>	 ori: what's up?
[00:51:57] <ori>	 paravoid: can you be around if I flip the graphite CNAME to tungsten?
[00:51:59] <ori>	 also, hi
[00:52:13] <paravoid>	 happy new year
[00:52:21] <ori>	 same to you :)
[00:52:35] <paravoid>	 yes, I can be around
[00:52:39] <paravoid>	 is everything ready?
[00:53:03] <ori>	 no, not yet. how much longer do you think you'll be around?
[00:53:54] <paravoid>	 half an hour maybe?
[00:54:05] <paravoid>	 but I can stay for more
[00:54:12] <paravoid>	 how can I help?
[00:55:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 7.271 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[00:56:51] <icinga-wm>	 RECOVERY - MySQL Slave Running on db68 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[00:58:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[01:02:34] <ori>	 paravoid: argh, I need to edit all the graph definition files in files/graphite/gdash/dashboards to change all FooMetric -> MediaWiki.FooMetric
[01:02:44] <ori>	 since they're namespaced under 'MediaWiki' in the new instance
[01:03:04] <ebernhardson>	 sed -i ?
[01:03:06] <ebernhardson>	 :)
[01:03:16] <ebernhardson>	 (i know, error prone :P)
[01:03:17] <ori>	 FooMetric can be anything, sadly
[01:03:34] <ori>	 e.g.: :data  => 'cactiStyle(substr(EditPage.*.tp90,1,2))'
[01:03:39] <ori>	 ('EditPage' in this case)
[01:04:11] <ori>	 paravoid: sorry, I forgot about that. Probably best that I just ping you tomorrow.
[01:14:08] <grrrit-wm>	 (03PS4) 10Aaron Schulz: Added a separate scap-rebuild-cdbs phase to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/105110 
[01:18:01] <ori>	 AaronSchulz: since you're setting it to 1 you might as well explicitly check for that value
[01:18:55] <AaronSchulz>	 I did that before and then didn't, meh
[01:19:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 8.612 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[01:21:04] <grrrit-wm>	 (03PS5) 10Aaron Schulz: Added a separate scap-rebuild-cdbs phase to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/105110 
[01:21:05] <AaronSchulz>	 meh
[01:22:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[01:27:06] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Added a separate scap-rebuild-cdbs phase to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/105110 (owner: 10Aaron Schulz)
[01:28:03] <Reedy>	 ori: https://gerrit.wikimedia.org/r/#/c/105021/
[01:28:06] <Reedy>	 eeaseh
[01:30:15] <logmsgbot>	 !log reedy started scap: Fix Disambiguator hewiki magicwords
[01:30:19] <ori>	 Reedy: why two separate messages?
[01:30:47] <ori>	 also, why remove the code to print it to stdout?
[01:31:02] <morebots>	 Logged the message, Master
[01:31:42] <Reedy>	 does it not print and assign?
[01:32:30] <Reedy>	 "The -v option causes the output to be assigned to the variable var rather than being printed to the standard output."
[01:32:31] <Reedy>	 srsly
[01:33:10] <ori>	 it's like printf / sprintf
[01:33:32] <ori>	 you could just change one line, the current dologmsg
[01:34:39] <ori>	 well, no. print it to a var, echo the var, and interpolate it into the log msg format
[01:39:12] <grrrit-wm>	 (03PS2) 10Reedy: Make logmsgbot report scap length to irc channel, but not log it [operations/puppet] - 10https://gerrit.wikimedia.org/r/105021 
[01:42:41] <Reedy>	 mw103: mwdeploy is not in the sudoers file.  This incident will be reported.
[01:42:41] <Reedy>	 mw103: mwdeploy is not in the sudoers file.  This incident will be reported.
[01:42:41] <Reedy>	 mw103: Done
[01:42:48] <Reedy>	 There's quite a few of these for different servers...
[01:42:58] <Reedy>	 I'm really on santas naughty list now
[01:52:18] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Do not try to run as mwdeploy in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105132 
[01:52:36] <ori>	 yep.
[01:52:53] <Reedy>	 Didn't even last a fortnight :(
[01:53:24] <AaronSchulz>	 ori: ^ stupid c/p error
[01:54:21] <Reedy>	 Is my "localisation update" not going to have worked then?
[01:55:19] <AaronSchulz>	 it would have aborted out
[01:55:21] <grrrit-wm>	 (03PS3) 10Ori.livneh: Make logmsgbot report scap length to irc channel, but not log it [operations/puppet] - 10https://gerrit.wikimedia.org/r/105021 (owner: 10Reedy)
[01:55:29] <AaronSchulz>	 anything after the failed sudo doesn't happen
[01:55:48] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Do not try to run as mwdeploy in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105132 (owner: 10Aaron Schulz)
[01:55:58] <Reedy>	 Right
[01:56:00] <Reedy>	 Sigh
[01:57:07] <grrrit-wm>	 (03PS4) 10Ori.livneh: Make logmsgbot report scap length to irc channel, but not log it [operations/puppet] - 10https://gerrit.wikimedia.org/r/105021 (owner: 10Reedy)
[01:57:12] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Make logmsgbot report scap length to irc channel, but not log it [operations/puppet] - 10https://gerrit.wikimedia.org/r/105021 (owner: 10Reedy)
[01:57:24] <logmsgbot>	 !log reedy finished scap: Fix Disambiguator hewiki magicwords
[01:57:39] <Reedy>	 scap completed in 31m 28s.
[01:58:02] <morebots>	 Logged the message, Master
[02:01:50] <Reedy>	 Hmmm
[02:02:03] <Reedy>	 Localisation update is due to start nowish..
[02:03:09] <ori>	 i ran puppet on tin
[02:03:13] <ori>	 anything else i can do to help?
[02:03:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 7.726 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[02:03:52] <Reedy>	 I sorta need to run scap again... But it's going to be running at a very similar time to localisation update
[02:04:59] <AaronSchulz>	 so the /upstream dir must have been synced, so I can just rebuild the cdbs
[02:05:18] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Our nameservers do not provide a recursion service anyway, so yes, these are all noops and this discussion is a orthogonal, indeed." [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn)
[02:05:57] <Ryan_Lane>	 paravoid: you aren't awake are you?
[02:06:02] <paravoid>	 I am
[02:06:11] <Ryan_Lane>	 it's absurdly late there ;)
[02:06:28] <Ryan_Lane>	 I'm thinking of ways to move away from dns for ldap/puppet
[02:06:34] <paravoid>	 were you afraid of someone hacking my gerrit account and posting code reviews? :)
[02:06:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[02:06:43] <Reedy>	 AaronSchulz: WFM
[02:07:08] <paravoid>	 I'm not sure I understand
[02:07:12] <AaronSchulz>	 I think it's done already, just hanging on the last few boxes
[02:07:14] <Ryan_Lane>	 one sec
[02:07:21] <Ryan_Lane>	 need to describe it :)
[02:07:27] <paravoid>	 okay :)
[02:07:38] <Ryan_Lane>	 I was thinking of storing puppet data in DNS, and writing an ENC that reads it from there
[02:07:40] <AaronSchulz>	 searchidx1001...always the last to the party
[02:07:54] <Ryan_Lane>	 then writing everything into DNS using designate
[02:08:51] <AaronSchulz>	 Reedy: I didn't do wmf9 though, does that matter?
[02:09:01] <paravoid>	 why?
[02:09:20] <Ryan_Lane>	 because I'd like the web interface to stop writing puppet and dns info into ldap
[02:09:41] <paravoid>	 ah
[02:09:58] <Ryan_Lane>	 this would make it easier to open api access
[02:10:21] <paravoid>	 no generic key/value openstack service yet I'm assuming :)
[02:10:44] <Ryan_Lane>	 well, there is, kind of
[02:10:51] <Ryan_Lane>	 it wouldn't really be much of a help there, though
[02:11:02] <Reedy>	 AaronSchulz: Shouldn't really. localisation update will fix that one long before it's used on hewiki
[02:11:15] <Ryan_Lane>	 since it really just provides the ability to create them in tenants
[02:11:37] <Ryan_Lane>	 this needs to be globally accessible and tenant writable
[02:11:59] <Ryan_Lane>	 I could write a nova plugin for puppet to write it into LDAP, too
[02:12:12] <Ryan_Lane>	 it doesn't necessarily need to be designate
[02:13:26] <Ryan_Lane>	 DNS can be extended to be a key/value for things that don't write a lot and don't need to be queried easily.
[02:13:36] <Ryan_Lane>	 not sure if it's an insane idea or not, though :D
[02:13:38] <paravoid>	 switching to designate for DNS sounds kind of obvious
[02:13:50] <Ryan_Lane>	 yeah, we're doing that either way
[02:13:56] <paravoid>	 using DNS for puppet classes... dunno, I'm not thrilled by the idea :)
[02:14:02] * Ryan_Lane  nods

[02:14:17] <paravoid>	 how would you do variables?
[02:14:24] <paravoid>	 arbitrary TXT records or something?
[02:14:28] <Ryan_Lane>	 yep
[02:15:56] <Ryan_Lane>	 extending nova client and server api is likely easy enough too, though
[02:16:14] <paravoid>	 and then what? nova server api writing to ldap?
[02:16:17] <Ryan_Lane>	 yep
[02:16:44] <Ryan_Lane>	 I want to move away from LDAP for DNS because pdns's implementation sucks
[02:17:12] <Ryan_Lane>	 and it's unmaintained
[02:17:20] <paravoid>	 and there's an openstack service for it now
[02:17:25] <Ryan_Lane>	 yep
[02:17:32] <Ryan_Lane>	 which makes things way easier
[02:17:32] <paravoid>	 that alone sounds a good enough reason to me
[02:17:38] <paravoid>	 sounds like*
[02:17:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 7.843 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[02:17:44] <Ryan_Lane>	 mhoover has a working prototylpe
[02:17:55] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf8) at Fri Jan  3 02:17:55 UTC 2014
[02:17:56] <paravoid>	 do you know what is wrong with ^^^ btw? :)
[02:18:05] <paravoid>	 speaking of labs DNS
[02:18:06] <Ryan_Lane>	 has that been flapping?
[02:18:09] <paravoid>	 yes
[02:18:10] <paravoid>	 all day
[02:18:14] <paravoid>	 I haven't investigated
[02:18:18] <morebots>	 Logged the message, Master
[02:18:27] <Ryan_Lane>	 opendj has an issue of some variety
[02:18:38] <Ryan_Lane>	 I'm betting something absurd is querying it poorly
[02:18:39] <paravoid>	 and even the recovery is 7.8s, doesn't sound very recovered to me either
[02:19:01] <Ryan_Lane>	 indeed
[02:19:31] <Ryan_Lane>	 checking opendj on virt1000
[02:19:42] <Ryan_Lane>	 seems fine there
[02:20:23] <Ryan_Lane>	 hm. still a problem on virt0
[02:21:23] <ori>	 Reedy / AaronSchulz: I'm about to head out. Is there anything else that I should hang around for?
[02:21:23] <grrrit-wm>	 (03PS1) 10Springle: sanitarium scripts built during Sep13 data leak event, based on .sql files in Asher's old $HOME. [operations/software/redactatron] - 10https://gerrit.wikimedia.org/r/105135 
[02:21:29] <Ryan_Lane>	 hm. someone's doing a pretty expensive query really often
[02:21:37] <AaronSchulz>	 not for me, I'm heading out too
[02:22:21] <Reedy>	 I'm hoping to head to bed soon..
[02:22:21] <grrrit-wm>	 (03CR) 10Springle: [C: 032] sanitarium scripts built during Sep13 data leak event, based on .sql files in Asher's old $HOME. [operations/software/redactatron] - 10https://gerrit.wikimedia.org/r/105135 (owner: 10Springle)
[02:22:41] <ori>	 Reedy: I can stick around if there's something I could do to assist
[02:23:04] <Ryan_Lane>	 we should really have a dedicated ldap server for wikitech/gerrit/other web interfaces
[02:23:05] <Reedy>	 ori: Nothing to do, just been confirmed as fixed :)
[02:23:12] <ori>	 wee.
[02:23:19] <Ryan_Lane>	 so that labs instances can't affect it
[02:23:23] <Reedy>	 bar some purging it seems
[02:23:27] <paravoid>	 yeah definitely
[02:23:39] <paravoid>	 but the discussion above sounds very appropriate too :)
[02:23:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[02:24:47] <paravoid>	 between the two, I think I prefer nova writing to LDAP using the puppet/ldap schema, rather than trying to use domain records to encode puppetClass/puppetVar
[02:24:58] <Ryan_Lane>	 agreed
[02:25:09] <Ryan_Lane>	 either way I need to write a plugin for some service
[02:33:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 9.447 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[02:36:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[02:47:35] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf9) at Fri Jan  3 02:47:34 UTC 2014
[02:47:53] <morebots>	 Logged the message, Master
[02:52:31] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 0.117 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[02:53:37] <Ryan_Lane>	 paravoid: ^^ well seems clearing that cron may have fixed the situation
[03:03:17] <grrrit-wm>	 (03PS1) 10Ryan Lane: Specify the command to remove for manage-exports [operations/puppet] - 10https://gerrit.wikimedia.org/r/105136 
[03:08:16] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Specify the command to remove for manage-exports [operations/puppet] - 10https://gerrit.wikimedia.org/r/105136 (owner: 10Ryan Lane)
[03:20:58] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan  3 03:20:58 UTC 2014
[03:21:20] <morebots>	 Logged the message, Master
[03:23:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[03:24:07] <springle>	 !log start schema changes for bug 59236, indexing only, ipblocks ipb_parent_block_id
[03:24:54] <morebots>	 Logged the message, Master
[03:33:40] <grrrit-wm>	 (03PS1) 10Ryan Lane: Fix misplaced closing brace for manage-exports [operations/puppet] - 10https://gerrit.wikimedia.org/r/105137 
[03:33:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 6.466 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[03:35:28] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Fix misplaced closing brace for manage-exports [operations/puppet] - 10https://gerrit.wikimedia.org/r/105137 (owner: 10Ryan Lane)
[03:37:41] <icinga-wm>	 PROBLEM - Auth DNS on labs-ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call  
[03:39:41] <icinga-wm>	 RECOVERY - Auth DNS on labs-ns1.wikimedia.org is OK: DNS OK: 5.797 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219  
[04:24:14] <Aaron|home>	 TimStarling: is there any use keeping the postConnectionBackoff stuff around?
[04:27:34] <Ryan_Lane>	 !log added python-keystone-redis to apt repo
[04:27:35] <Aaron|home>	 actually if getLagTimes is the only thing used, then that could just go in LB and LM can be deleted
[04:27:50] <morebots>	 Logged the message, Master
[04:28:02] <TimStarling>	 I guess postConnectionBackoff can go
[04:28:32] <Aaron|home>	 it's a whole separate hierarchy that really only supports mysql
[04:28:34] <TimStarling>	 I figured it was better to reduce the max_threads on the server, and let the clients get a connection error and die
[04:28:47] <TimStarling>	 well, the idea was that more subclasses would be added to support other DBMSes
[04:29:07] <TimStarling>	 and, ideally, some other information system outside of MySQL
[04:29:22] <TimStarling>	 that doesn't require you to actually connect to the server to check whether it is overloaded
[04:30:14] <Aaron|home>	 can't other replication systems have a meaningful getLag() implementation? I'd hope that any replica DB has some way to get the lag somehow.
[04:30:39] <Aaron|home>	 so the only real use would be something that grabs async stats without a direct DB query
[04:31:30] <Aaron|home>	 we kind of hack around that with $wgMemc, the add() locks, and using stale data while locked
[04:31:40] <TimStarling>	 yes, we have memcached already
[04:31:58] <TimStarling>	 ideally you don't want to have to connect to a server in order to decide whether or not to connect to it
[04:32:04] <grrrit-wm>	 (03PS1) 10Ryan Lane: Add redis support to keystone [operations/puppet] - 10https://gerrit.wikimedia.org/r/105139 
[04:32:05] <TimStarling>	 it is kind of inefficient
[04:32:38] <TimStarling>	 I imagined LoadMonitor being a client for some system which held information on all mysql servers
[04:32:50] <TimStarling>	 updated by regular polling
[04:36:22] <Aaron|home>	 it would be nice to have a non-mysql specific one...actually the mysql one might be as long as getLag is implemented
[04:37:28] <TimStarling>	 yeah, once postConnectionBackoff() is removed, it is not really MySQL-specific
[04:38:55] <Aaron|home>	 of course, the DatabasePostgres class for example, would need config flags for the type of replication (http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling#Introduction)
[04:39:10] <Aaron|home>	 with mysql, there aren't so many choices
[04:39:33] <Aaron|home>	 TimStarling: so when was that backoff disabled?
[04:39:41] <Aaron|home>	 I see it commented out with --TS
[04:43:59] <TimStarling>	 after 2009 and before 2013
[04:46:31] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1026 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: Error Deadlock found when trying to get lock: try restarting transac  
[04:46:39] <TimStarling>	 pity the history of this repo wasn't imported into git from subversion
[04:47:31] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1026 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[04:48:39] <TimStarling>	 looks like it was around 2011-06-23, r1905 in the old repo
[04:50:29] <TimStarling>	 it was during incident response on that day
[04:50:49] <TimStarling>	 https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_18#June_23
[04:51:01] <TimStarling>	  16:32 RoanKattouw: Site came back up instantly after Tim disabled max_threads
[04:51:01] <TimStarling>	 16:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'disabled max threads' 
[05:00:58] <springle>	 haproxy nodes in front of slaves, perhaps. it can monitor both rep lag and outages
[05:08:11] <Aaron|home>	 aaron@fluorine:~/mw-log$ grep -P -o 'Error connecting to .+ ' dberror.log | grep -P -o '\d+\.\d+\.\d+\.\d+' | sort | uniq -c
[05:08:13] <Aaron|home>	    4312 10.64.0.6
[05:08:14] <Aaron|home>	    8743 10.64.16.23
[05:08:16] <Aaron|home>	     718 10.64.16.29
[05:08:48] <Aaron|home>	 springle: I always wonder if there is some way to avoid that error spam (though it only matters to the user if all the slaves give it)
[05:09:43] <Aaron|home>	 either the weights are off, the max connections on the server are too low, or there are too many CDN misses
[05:10:59] <springle>	 db1002, db1034, and db1040
[05:14:44] <springle>	 max_connections is already high. imo it should be lower
[05:15:24] <Aaron|home>	 well that or more slaves are needed
[05:15:35] <Aaron|home>	 so 4 possibilities ;)
[05:15:37] <springle>	 db1002 and 34 are both s2 and have same hardware/config. probably a spike
[05:15:49] <springle>	 yes definitely. have ordered some
[05:15:59] <Aaron|home>	 these flood of errors happen every day
[05:16:23] <Aaron|home>	 springle: move some from tampa too? >:D
[05:16:43] <springle>	 yep :) i have 9 coming from tampa ;)
[05:16:44] <Aaron|home>	 hmm, that emoticon looks wrong in BZ
[05:17:12] <Aaron|home>	 dastardly smiley face is just regular angry face
[05:17:21] <Aaron|home>	 *CZ
[05:37:40] <Aaron|home>	 query: SELECT /* EditPage::getLastDelete EvilFreD */ 
[05:37:59] <Aaron|home>	 silly that such a simple query falls over...must be those missing log indexes
[06:57:42] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[06:58:52] <icinga-wm>	 PROBLEM - LVS HTTP IPv6 on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:59:21] <icinga-wm>	 PROBLEM - LVS HTTPS IPv6 on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:59:31] <icinga-wm>	 PROBLEM - Host amssq62 is DOWN: PING CRITICAL - Packet loss = 61%, RTA = 2217.21 ms  
[06:59:51] <icinga-wm>	 RECOVERY - LVS HTTP IPv6 on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64254 bytes in 5.294 second response time  
[06:59:53] <icinga-wm>	 RECOVERY - Host amssq62 is UP: PING OK - Packet loss = 16%, RTA = 118.64 ms  
[07:00:21] <icinga-wm>	 RECOVERY - LVS HTTPS IPv6 on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64390 bytes in 7.692 second response time  
[07:00:51] <icinga-wm>	 PROBLEM - LVS HTTPS IPv4 on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:01:51] <icinga-wm>	 RECOVERY - LVS HTTPS IPv4 on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64390 bytes in 5.604 second response time  
[07:04:21] <icinga-wm>	 PROBLEM - LVS HTTPS IPv6 on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:05:03] <icinga-wm>	 PROBLEM - Host amssq62 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2161.02 ms  
[07:05:03] <icinga-wm>	 PROBLEM - Host amssq56 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2214.52 ms  
[07:05:11] <icinga-wm>	 PROBLEM - Host amssq48 is DOWN: PING CRITICAL - Packet loss = 73%, RTA = 2058.84 ms  
[07:05:32] <icinga-wm>	 RECOVERY - Host amssq48 is UP: PING WARNING - Packet loss = 37%, RTA = 111.89 ms  
[07:05:32] <icinga-wm>	 RECOVERY - Host amssq56 is UP: PING WARNING - Packet loss = 37%, RTA = 146.32 ms  
[07:05:41] <icinga-wm>	 RECOVERY - Host amssq62 is UP: PING WARNING - Packet loss = 37%, RTA = 118.10 ms  
[07:06:12] <icinga-wm>	 RECOVERY - LVS HTTPS IPv6 on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64390 bytes in 0.655 second response time  
[07:06:22] <mark>	 !log Disabled OSPF3 on csw2-knams:xe-1/1/0.0
[07:06:31] <icinga-wm>	 PROBLEM - Packetloss_Average on erbium is CRITICAL: CRITICAL: packet_loss_average is 14.6136152381 (gt 8.0)  
[07:06:40] <morebots>	 Logged the message, Master
[07:11:01] <icinga-wm>	 PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 8.32331555556 (gt 8.0)  
[07:16:31] <icinga-wm>	 RECOVERY - Packetloss_Average on erbium is OK: OK: packet_loss_average is 0.48323969697  
[07:17:01] <icinga-wm>	 PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 8.43920567568 (gt 8.0)  
[07:21:01] <icinga-wm>	 RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 0.629932833333  
[07:21:11] <icinga-wm>	 PROBLEM - Host streber is DOWN: PING CRITICAL - Packet loss = 100%  
[07:27:01] <icinga-wm>	 RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is -0.644574864865  
[07:37:03] <apergos>	 !log streber from mgmt console reports eth0 link down (hence it appears down to icinga and ganglia)
[07:37:20] <morebots>	 Logged the message, Master
[07:37:52] <apergos>	 no carrier... 
[07:38:07] * apergos  looks around for a mark

[07:39:26] <apergos>	 mark, around?  any ideas about streber no carrier?
[07:45:07] <apergos>	 nm I see your email nw
[08:09:41] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[09:07:07] <hashar>	 goooood morning
[09:07:24] <ori>	 heyo
[09:10:26] <grrrit-wm>	 (03CR) 10Hashar: [WIP] Kibana puppet class (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis)
[09:16:32] <icinga-wm>	 RECOVERY - Host streber is UP: PING OK - Packet loss = 0%, RTA = 26.88 ms  
[09:18:31] <icinga-wm>	 PROBLEM - Host streber is DOWN: PING CRITICAL - Packet loss = 100%  
[09:22:31] <icinga-wm>	 RECOVERY - Host streber is UP: PING OK - Packet loss = 0%, RTA = 26.90 ms  
[09:25:11] <icinga-wm>	 PROBLEM - Host streber is DOWN: PING CRITICAL - Packet loss = 100%  
[09:27:31] <icinga-wm>	 RECOVERY - Host streber is UP: PING OK - Packet loss = 0%, RTA = 26.91 ms  
[09:28:01] <icinga-wm>	 PROBLEM - Disk space on wtp1023 is CRITICAL: DISK CRITICAL - free space: / 264 MB (2% inode=72%):  
[09:31:01] <icinga-wm>	 RECOVERY - Disk space on wtp1023 is OK: DISK OK  
[09:31:51] <icinga-wm>	 PROBLEM - Parsoid on wtp1023 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:31:51] <icinga-wm>	 PROBLEM - Host streber is DOWN: PING CRITICAL - Packet loss = 100%  
[09:33:31] <icinga-wm>	 RECOVERY - Host streber is UP: PING OK - Packet loss = 0%, RTA = 26.89 ms  
[09:33:41] <icinga-wm>	 PROBLEM - Disk space on wtp1010 is CRITICAL: DISK CRITICAL - free space: / 199 MB (2% inode=72%):  
[09:36:11] <icinga-wm>	 PROBLEM - Host streber is DOWN: PING CRITICAL - Packet loss = 100%  
[09:38:41] <icinga-wm>	 PROBLEM - Parsoid on wtp1010 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:47:41] <icinga-wm>	 RECOVERY - Parsoid on wtp1023 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.002 second response time  
[09:49:31] <icinga-wm>	 RECOVERY - Parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.006 second response time  
[09:49:41] <icinga-wm>	 RECOVERY - Disk space on wtp1010 is OK: DISK OK  
[09:50:51] <apergos>	 !log restarted parsoid on wtp1010 and 1023, several gigs of logs full of "Error: Can't set headers after they are sent." from ServerResponse.OutgoingMessage.setHeader 
[09:51:09] <morebots>	 Logged the message, Master
[09:58:23] <grrrit-wm>	 (03CR) 10saper: "Oh, very imporant reasons:" [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn)
[11:12:13] <grrrit-wm>	 (03PS1) 10Ori.livneh: gdash: Prefix all metric names with 'MediaWiki.' [operations/puppet] - 10https://gerrit.wikimedia.org/r/105163 
[11:16:10] <hashar>	 ori: friendly reminder that it is 3am and you should sleep :-D
[11:16:52] <ori>	 allllllllmost
[11:17:14] <hashar>	 ori: and I resist the envy of bike shedding about prefixing metrics with MediaWiki :D
[11:17:45] <grrrit-wm>	 (03PS1) 10Ori.livneh: point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 
[11:17:55] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 (owner: 10Ori.livneh)
[11:18:36] <hashar>	 bah
[11:19:00] <grrrit-wm>	 (03PS2) 10Ori.livneh: point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 
[11:49:05] <grrrit-wm>	 (03CR) 10Aklapper: [C: 031] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103525 (owner: 10Dzahn)
[12:14:30] <Krinkle>	 !log Jenkins is showing failures for tests executed on integration-slave01 (remote file system failing)
[12:14:46] <morebots>	 Logged the message, Master
[12:14:56] <Nemo_bis>	 uhuh, morebots is here :)
[12:43:48] <Krinkle>	 !log Jenkins still unable to use integration-slave01 (restarted the node in labs, and disconnected/re-launched slave agent afterwards, too; no effect)
[12:44:02] <morebots>	 Logged the message, Master
[13:30:21] <icinga-wm>	 PROBLEM - MySQL Processlist on db1009 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 92 copy to table, 36 statistics  
[13:34:21] <icinga-wm>	 RECOVERY - MySQL Processlist on db1009 is OK: OK 1 unauthenticated, 0 locked, 6 copy to table, 4 statistics  
[13:35:46] <springle>	 !log killed msnbot spike on s2
[13:36:04] <morebots>	 Logged the message, Master
[13:44:07] <paravoid>	 again?
[13:45:49] <Steinsplitter>	 msn o_O
[13:56:11] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:57:01] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical  
[14:08:31] <icinga-wm>	 PROBLEM - HTTP on aluminium is CRITICAL: Connection refused  
[14:11:31] <icinga-wm>	 RECOVERY - HTTP on aluminium is OK: HTTP OK: HTTP/1.1 302 Found - 557 bytes in 0.001 second response time  
[14:14:51] <icinga-wm>	 PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100%  
[14:15:51] <icinga-wm>	 RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.38 ms  
[14:18:11] <icinga-wm>	 PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused  
[14:19:12] <icinga-wm>	 RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.446 second response time  
[14:23:14] <hashar>	 !log restarting Jenkins , some git plugin are misbehaving
[14:23:30] <morebots>	 Logged the message, Master
[14:30:13] <brainwane>	 hi akosiaris - per https://wikitech.wikimedia.org/wiki/Server_access_responsibilities#SSH_key_access I need my key revoked. Do you want the fingerprint, the public key, or something else?
[14:30:39] <brainwane>	 (the laptop is with WMF office IT for repair)
[14:31:02] <brainwane>	 (I assume I should contact akosiaris because that's the nick I see for "on RT duty" in the /topic)
[14:31:53] <akosiaris>	 I need username and public key (whether the fingerprint or the entire key... does not matter for revocation)
[14:32:24] <brainwane>	 akosiaris -- username: sumanah     
[14:32:43] <akosiaris>	 after you issue a new one, that one you need to upload somewhere trusted and I will fetch it from there (office wiki for example)
[14:32:44] <brainwane>	 akosiaris: fingerprint: 54:66:e6:49:fd:47:1e:16:19:d8:85:94:cd:61:d3:1c
[14:32:47] <brainwane>	 akosiaris: got it
[14:33:05] <akosiaris>	 oh.. and welcome back!!
[14:33:08] <brainwane>	 thank you akosiaris!
[14:33:42] <brainwane>	 I'm sorry for not revoking the key IMMEDIATELY; the laptop was being carried via a trusted person to WMF OIT and so I thought the risk was ok, but then I read the wikitech wiki page and saw that it was not.
[14:35:38] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 
[14:36:09] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Revoke sumanah keys after her request. [operations/puppet] - 10https://gerrit.wikimedia.org/r/105186 
[14:37:41] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 (owner: 10Mark Bergsma)
[14:39:23] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Revoke sumanah keys after her request. [operations/puppet] - 10https://gerrit.wikimedia.org/r/105186 (owner: 10Alexandros Kosiaris)
[14:39:34] <grrrit-wm>	 (03PS2) 10Mark Bergsma: Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 
[14:39:40] <akosiaris>	 LOST ?
[14:39:55] <brainwane>	 akosiaris: huh?
[14:40:08] <akosiaris>	 jenkins is voting LOST ...
[14:40:13] <brainwane>	 oh, jenkins. Weird!
[14:40:16] <akosiaris>	 on all jobs... weird...
[14:40:22] <akosiaris>	 will force the commit
[14:40:24] <Krinkle>	 <hashar>: !log restarting Jenkins 
[14:40:34] <Krinkle>	 Just wait a few minutes, CR+2 it now and it'll be handled when it is ready.
[14:40:42] <hashar>	 i should stop Zuul probably :D
[14:40:51] <akosiaris>	 17 minutes ago... does it take that long ?
[14:40:54] <Krinkle>	 (unless it needs to be deployed right now)
[14:41:00] <Krinkle>	 Jenkins takes a long time to reboot.
[14:41:07] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Revoke sumanah keys after her request. [operations/puppet] - 10https://gerrit.wikimedia.org/r/105186 (owner: 10Alexandros Kosiaris)
[14:41:14] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 (owner: 10Mark Bergsma)
[14:42:46] <akosiaris>	 brainwane: I just merged the chage, key will be revoked from anywhere in the cluster in 30 mins top. Thanks for letting us know.
[14:43:53] <akosiaris>	 !log zapped /vol/root export from nas-1001-a
[14:44:10] <morebots>	 Logged the message, Master
[14:44:16] <brainwane>	 thanks akosiaris - that seems sufficient to me
[14:44:16] <mark>	 entirely?
[14:45:19] <akosiaris>	 mark: we can change it to another host if someone uses it. 
[14:45:32] <mark>	 occasionally to do netapp config changes
[14:45:38] <mark>	 VERY annoying that you almost have to do that over nfs
[14:45:46] <mark>	 but yeah, not on streber anymore ;p
[14:45:50] <akosiaris>	 I always use the CLI
[14:46:00] <mark>	 does it have an editor?
[14:46:20] <mark>	 or do you use that cat equivalent ;p
[14:46:22] <akosiaris>	 I think it does in 8.something something
[14:46:30] <akosiaris>	 but I don't use it
[14:46:32] <mark>	 why can't they just ship vi or smt
[14:48:16] <akosiaris>	 anyway we can always set the export to another host when we need it
[14:48:24] <mark>	 absolutely
[14:53:25] <hashar>	 !log jenkins restarted
[14:53:43] <morebots>	 Logged the message, Master
[14:57:15] <grrrit-wm>	 (03PS3) 10Hashar: Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 (owner: 10Mark Bergsma)
[15:00:48] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Fix indentation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105185 (owner: 10Mark Bergsma)
[15:00:51] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1017 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[15:01:41] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1017 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[15:20:15] <grrrit-wm>	 (03PS8) 10Physikerwelt: Add Mathoid module (TeX -> MathML / SVG conversion web service) [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 
[15:23:20] <grrrit-wm>	 (03PS1) 10Hashar: webproxies service entries in wmnet. [operations/dns] - 10https://gerrit.wikimedia.org/r/105189 
[15:23:38] <grrrit-wm>	 (03PS2) 10Hashar: webproxies service entries in wmnet. [operations/dns] - 10https://gerrit.wikimedia.org/r/105189 
[15:24:51] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] webproxies service entries in wmnet. [operations/dns] - 10https://gerrit.wikimedia.org/r/105189 (owner: 10Hashar)
[15:37:51] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Setup RANCID on netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105191 
[15:39:03] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Setup RANCID on netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105191 (owner: 10Mark Bergsma)
[15:39:49] <grrrit-wm>	 (03PS1) 10Tim Landscheidt: Tools: Install requested package python-pyexiv2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105192 
[15:47:18] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add systemuser creation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105193 
[15:48:23] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add systemuser creation [operations/puppet] - 10https://gerrit.wikimedia.org/r/105193 (owner: 10Mark Bergsma)
[16:04:47] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add /etc/rancid/rancid.conf to config management [operations/puppet] - 10https://gerrit.wikimedia.org/r/105199 
[16:14:15] <logmsgbot>	 !log reedy synchronized php-1.23wmf9/extensions/
[16:14:32] <morebots>	 Logged the message, Master
[16:15:22] <logmsgbot>	 !log reedy synchronized php-1.23wmf8/extensions/
[16:15:38] <morebots>	 Logged the message, Master
[16:22:23] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add RANCID cron job [operations/puppet] - 10https://gerrit.wikimedia.org/r/105203 
[16:23:09] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add /etc/rancid/rancid.conf to config management [operations/puppet] - 10https://gerrit.wikimedia.org/r/105199 (owner: 10Mark Bergsma)
[16:23:27] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add RANCID cron job [operations/puppet] - 10https://gerrit.wikimedia.org/r/105203 (owner: 10Mark Bergsma)
[16:24:16] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "scap shuffles the list to avoid overloading rsync servers, as far as I can see, so this won't produce the effect you want it to produce. I" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105006 (owner: 10Reedy)
[16:27:12] <grrrit-wm>	 (03CR) 10Reedy: "Outage last night; waiting for redeployment of updated l10n cache files via sync-dir to fix the problem. EQIAD was last, so had to wait fo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105006 (owner: 10Reedy)
[16:28:14] <paravoid>	 Reedy: outage report? :)
[16:28:33] <Reedy>	 commons and wikivoyage were broken ;)
[16:28:40] <twkozlowski>	 and Wikidata
[16:28:46] <Reedy>	 because of wikidata
[16:29:11] <Reedy>	 Due to the somewhat stupid way we handle missing magic words
[16:29:37] <Reedy>	 or, as in this case; we don't
[16:29:52] <paravoid>	 can you send a detailed report via email or even document it on wikitech under the incident section?
[16:30:16] <Reedy>	 Yeah
[16:30:25] <paravoid>	 not too detailed, just what happened/what was the bug, when did it happen, why did it happen, what can we do to fix it :)
[16:30:32] <Reedy>	 I'd had more than enough after the long deploy and unrelated database outage we had afterwards
[16:30:52] <greg-g>	 :)
[16:31:05] <paravoid>	 I understand, it's not urgent, but I was lacking context for your change
[16:31:06] <greg-g>	 it was a fun day
[16:31:21] <paravoid>	 oh the fun continued today, don't you worry
[16:31:38] <Reedy>	 paravoid: it was literally a wtf, why do we do tampa first for anything like that
[16:32:05] <Reedy>	 In theory, reversing the list would've reduced the time to deploy the fix by anything upto 50%
[16:32:08] <paravoid>	 so sync-common-files takes the list as-is
[16:32:18] <Reedy>	 yup
[16:32:18] <paravoid>	 sync-common-file
[16:32:21] <Reedy>	 line by line via dsh
[16:32:25] <paravoid>	 although it is dsh -c
[16:32:26] <Reedy>	 X at a time etc
[16:32:29] <paravoid>	 so in theory, concurrent
[16:32:43] <paravoid>	 scap shuffles, though
[16:33:08] <Reedy>	 yeah because of tims proxy fan out niceness
[16:35:35] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Are we going to do both kibana.wm.org & logstash.wm.org?" [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 (owner: 10BryanDavis)
[16:37:05] <bd808>	 paravoid: Would you rather have logstash as the name for the search GUI? I don't care
[16:37:24] <paravoid>	 hi
[16:37:27] <paravoid>	 are we going to do both?
[16:37:45] <paravoid>	 logstash has a web intf too, does it not?
[16:37:49] <bd808>	 Hi and happy new year :)
[16:37:54] <greg-g>	 I don't think so paravoid 
[16:37:56] <paravoid>	 oh yeah :)
[16:38:07] <bd808>	 Ah. It used to but has pretty much been deprecated
[16:38:09] <paravoid>	 it used to at least
[16:39:03] <bd808>	 It had a really crappy UI and then the ruby rewrite of kibana (v2). Now it seems to be dead tech
[16:39:54] <bd808>	 So I was just planning on the on the kibana3 UI with logstash being backend middlewear for parsing and shipping logs
[16:40:15] <paravoid>	 shipping?
[16:40:21] <paravoid>	 shipping how?
[16:40:26] <Reedy>	 ocean going liners
[16:40:52] <bd808>	 Into elasticsearch and any other backends we end up wanting
[16:41:48] <bd808>	 log source -> logstash -> elasticsearch
[16:42:06] <paravoid>	 log source being?
[16:42:18] <bd808>	 php app or syslog or ?
[16:42:47] <paravoid>	 okay, "log shipping" confused me :)
[16:42:58] <paravoid>	 lumberjack etc.
[16:43:17] <bd808>	 Ah. Yes. That is a possible but not probably use case for us
[16:43:22] <paravoid>	 http://cookbook.logstash.net/recipes/log-shippers/
[16:43:51] <bd808>	 It think logstash is little heavy for running on most log generating nodes
[16:44:12] <paravoid>	 yup
[16:48:53] <^d>	 bd808: Heh, http://www.elasticsearch.org/blog/logging-elasticsearch-events-with-logstash-and-elasticsearch/
[16:50:48] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Fix rancid file modes [operations/puppet] - 10https://gerrit.wikimedia.org/r/105209 
[16:50:49] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add RANCID .cloginrc file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105210 
[16:51:04] <bd808>	 ^d: Feeding back in to the same cluster :)
[16:51:39] <bd808>	 I've done that before in testing but never considered it in production
[16:51:45] <^d>	 It sounds insane!
[16:51:46] <^d>	 :)
[16:52:08] <greg-g>	 paravoid: just read _security.....
[16:52:15] <paravoid>	 I know
[16:52:16] <^d>	 bd808: Logging elastic (for cirrus) in logstash would be a good idea though, and easy.
[16:52:21] <gi11es>	 just before I left dA logstash was rolled out on all the web servers
[16:52:37] <gi11es>	 with ES and kibana as the frontend (not sure what version)
[16:52:59] <gi11es>	 didn't seem to be an issue to run in production, then again I don't know the performance details
[16:53:06] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Fix rancid file modes [operations/puppet] - 10https://gerrit.wikimedia.org/r/105209 (owner: 10Mark Bergsma)
[16:53:27] <^d>	 gi11es: We're explicitly having different clusters for logstash's elasticsearch and production search's elasticsearch.
[16:53:36] <^d>	 :)
[16:53:40] <gi11es>	 sounds sane
[16:54:07] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add RANCID .cloginrc file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105210 (owner: 10Mark Bergsma)
[16:55:21] <^d>	 bd808: Do we have a preferred host to log to in the logstash setup?
[16:55:53] <^d>	 (or, maybe we could put it behind lvs like we did with elastic10[nn] so clients don't care)
[16:56:25] <bd808>	 ^d: We haven't got that bit nailed down yet
[16:57:02] <bd808>	 There is going to be a redis input path that the php code will probably use
[16:57:02] <^d>	 I think we should do lvs like we did on the other one. It makes things wayyyyy easier for the clients (and is pretty trivial to setup)
[16:57:18] <^d>	 Then it won't matter what boxen is moved around or renamed.
[16:57:52] <bd808>	 That would be good. I'm always a fan of service names vs host names
[16:58:04] <paravoid>	 lvs for?
[16:58:08] <^d>	 logstash
[16:58:11] <paravoid>	 for logging you mean?
[16:58:17] <^d>	 ya
[16:58:59] <mark>	 logging behind lvs? is that really necessary? :)
[16:59:05] <gi11es>	 are you going to allow logstash usage from php-land? or will it be for purely traditional syslog entries?
[16:59:21] <paravoid>	 I was hoping all clients will support writing to multiple servers
[16:59:34] <^d>	 paravoid, mark: Maybe not necessary, I just threw the idea out there.
[16:59:39] <^d>	 :)
[16:59:39] <bd808>	 gi11es: We will be logging from php for sure
[16:59:45] <paravoid>	 gi11es: the first deployment is going to be mediawiki actually, syslog will come later :)
[17:00:02] <paravoid>	 and I'm not even sure how are we going to do that yet tbh
[17:00:07] <gi11es>	 awesome. is there a legacy equivalent? or is it the first time you do something like this?
[17:00:08] <mark>	 logging is rather basic infrastructure, I'd rather not have that depend on something like lvs if we can avoid it
[17:00:20] <paravoid>	 as logstash doesn't really have proper access controls
[17:00:40] <^d>	 mark: Fair enough :)
[17:00:41] <paravoid>	 and some syslog data are sensitive enough
[17:01:05] <^d>	 Anyway, anything that's java-base we can easily get in elasticsearch.
[17:01:16] <^d>	 log4j should *just work* with socket appender stuff.
[17:01:18] <paravoid>	 gi11es: we have a plain old syslog writing to files now, plus at least one other specialized thing for syslog data
[17:02:16] <bd808>	 gi11es: We are basically trying to move beyond "ssh to fluorine and use grep"
[17:02:24] <gi11es>	 and syslog was being written to by php?
[17:03:13] <bd808>	 There is a somewhat elaborate php -> udp -> files path for production application logs
[17:05:07] <gi11es>	 stop me if I ask too many questions :) and are these oldschool application logs searchable/graphed?
[17:05:43] <paravoid>	 currently, no, unless you count grep as "searchable"
[17:05:49] <bd808>	 :)
[17:06:16] <bd808>	 There are some "rate of message generation" graphs too but nothing very fancy
[17:07:06] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Fix RANCID directory modes [operations/puppet] - 10https://gerrit.wikimedia.org/r/105212 
[17:07:17] <bd808>	 gi11es: You can play with the proof of concept system in labs: https://logstash.wmflabs.org/
[17:07:39] <gi11es>	 wow, yeah logstash + kibana is going to be like going from horse carriages to spaceships
[17:08:20] <paravoid>	 that's the hope
[17:09:03] <ottomata>	 whoaa bd808 kibana is like graphs for logstash?
[17:09:15] <paravoid>	 it's a better web interface
[17:09:24] <paravoid>	 initially in php, then ruby, now in-browser javascript 
[17:09:29] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Fix RANCID directory modes [operations/puppet] - 10https://gerrit.wikimedia.org/r/105212 (owner: 10Mark Bergsma)
[17:09:33] <paravoid>	 developed by someone else than the logstash author
[17:09:39] <paravoid>	 but then both authors were hired by elasticsearch
[17:09:46] <paravoid>	 full circle
[17:09:58] <gi11es>	 something we did that could be useful here is we moved chat logging to logstash
[17:10:18] <ottomata>	 awesoome
[17:10:28] <paravoid>	 there's a bunch of things we can do, but one at a time :)
[17:10:35] <ottomata>	 uhhhh, how much data can we push to this thang?
[17:10:43] <ottomata>	 (webrequest, ahem ahem?)
[17:10:44] <ottomata>	 heheh :p
[17:10:59] <paravoid>	 the possibilities are endless, but let's pace ourselves :)
[17:11:01] <ottomata>	 heheh
[17:11:29] <paravoid>	 we've been talking about this for over a year now
[17:12:05] <bd808>	 And hopefully we are within a few days of seeing something that works :)
[17:28:23] <MaxSem>	 hey opsen, can I turn your attention to https://rt.wikimedia.org/Ticket/Display.html?id=4028 ? :)
[17:36:26] <ottomata>	 heyaa manybubbles, you've got elastic search icinga alerts up, right?
[17:36:48] <ottomata>	 do you have any that come directly from JMX?  or are they all just http curls ?
[17:48:15] <grrrit-wm>	 (03PS1) 10Edenhill: json: output NaN as null for '\!num' modifiers [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105216 
[17:48:42] <grrrit-wm>	 (03PS2) 10Edenhill: json: output NaN as null for '\!num' modifiers [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105216 
[17:52:04] <hashar>	 !log Jenkins downgrading git plugin from 2.0 to 1.5  , we might be hit by https://issues.jenkins-ci.org/browse/JENKINS-21057
[17:52:20] <morebots>	 Logged the message, Master
[17:54:57] <grrrit-wm>	 (03CR) 10BryanDavis: "I was planning on only using the kibana interface served via apache. Logstash has the ability to serve a web interface but it just serves " [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 (owner: 10BryanDavis)
[18:04:09] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1050 is CRITICAL: CRIT replication delay 308 seconds  
[18:04:09] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1037 is CRITICAL: CRIT replication delay 308 seconds  
[18:04:18] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1051 is CRITICAL: CRIT replication delay 312 seconds  
[18:04:28] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1052 is CRITICAL: CRIT replication delay 330 seconds  
[18:04:28] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 330 seconds  
[18:04:38] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1051 is CRITICAL: CRIT replication delay 337 seconds  
[18:04:38] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1049 is CRITICAL: CRIT replication delay 337 seconds  
[18:04:39] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1052 is CRITICAL: CRIT replication delay 336 seconds  
[18:04:39] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 336 seconds  
[18:04:39] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 336 seconds  
[18:04:39] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1037 is CRITICAL: CRIT replication delay 336 seconds  
[18:04:39] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db63 is CRITICAL: CRIT replication delay 338 seconds  
[18:04:40] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db63 is CRITICAL: CRIT replication delay 339 seconds  
[18:04:48] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1049 is CRITICAL: CRIT replication delay 342 seconds  
[18:04:48] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db67 is CRITICAL: CRIT replication delay 343 seconds  
[18:04:58] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1050 is CRITICAL: CRIT replication delay 352 seconds  
[18:04:58] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 358 seconds  
[18:05:28] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1052 is OK: OK replication delay 140 seconds  
[18:05:38] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1051 is OK: OK replication delay 147 seconds  
[18:05:38] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1052 is OK: OK replication delay -0 seconds  
[18:05:38] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db63 is OK: OK replication delay 148 seconds  
[18:05:38] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db63 is OK: OK replication delay 149 seconds  
[18:05:45] <Nemo_bis>	 s1 quite unhappy
[18:05:55] <greg-g>	 blugh
[18:06:04] <Reedy>	 I can guess why.. But it shouldn't be
[18:06:08] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1050 is OK: OK replication delay 72 seconds  
[18:06:15] <Reedy>	 Unless someone broke wfWaitForSlaves()...
[18:06:18] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1051 is OK: OK replication delay -0 seconds  
[18:06:23] <Reedy>	 -0 seconds! result
[18:06:33] <greg-g>	 that's fast
[18:06:38] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1049 is OK: OK replication delay 0 seconds  
[18:06:38] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay -1 seconds  
[18:06:48] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1049 is OK: OK replication delay -0 seconds  
[18:06:58] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1050 is OK: OK replication delay -0 seconds  
[18:07:04] <greg-g>	 getting reports in -tech
[18:07:14] <greg-g>	 of lag
[18:07:25] <grrrit-wm>	 (03PS1) 10Odder: Add more settings related to page imports on hewikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105221 
[18:07:28] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1043 is OK: OK replication delay 0 seconds  
[18:07:30] <Reedy>	 yeah, it'll take a little while for it to catch up
[18:07:50] <greg-g>	 s/reports/drive-by report/
[18:07:56] <MaxSem>	 wtfWaitForSlaves()...
[18:07:58] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1033 is OK: OK replication delay 123 seconds  
[18:08:38] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay -1 seconds  
[18:08:38] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1037 is OK: OK replication delay -1 seconds  
[18:09:08] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1037 is OK: OK replication delay 0 seconds  
[18:10:24] <Reedy>	 [17:17:56] <anomie> Hmm. It looks like it'll break DatabaseMysqlBase::masterPosWait() for non-fakeSlaveLag too. Yep, errors when I try to run update.php for wikis in the labs oauth project.
[18:11:28] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db67 is CRITICAL: CRIT replication delay 360 seconds  
[18:11:32] <Reedy>	 But that's master..
[18:11:47] <anomie>	 Reedy: The problematic patch I was referring to didn't make it into wmf9 though, I don't think.
[18:11:50] <hashar>	 !log Jenkins downgrading git plugin client to 1.4.6 and restarting jenkins
[18:11:51] <hashar>	 sorry
[18:11:58] <Reedy>	 anomie: Right
[18:12:02] <Reedy>	 And enwiki is on wmf8 anyway
[18:12:06] <morebots>	 Logged the message, Master
[18:12:13] <Reedy>	 Mostly checking as it was similar area
[18:13:28] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db67 is OK: OK replication delay 0 seconds  
[18:13:48] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db67 is OK: OK replication delay -1 seconds  
[18:16:28] <icinga-wm>	 PROBLEM - jenkins_service_running on gallium is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war  
[18:20:28] <icinga-wm>	 RECOVERY - jenkins_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war  
[18:21:24] <ori>	 paravoid: graphite dns is https://gerrit.wikimedia.org/r/#/c/105166/
[18:22:30] <paravoid>	 hmm
[18:22:36] <paravoid>	 on a friday...
[18:22:43] <paravoid>	 I wonder if I should press this button
[18:23:40] <Reedy>	 friday
[18:23:41] <Reedy>	 friday
[18:24:17] <paravoid>	 ori: what about the gdash dashboards that you mentioned?
[18:24:38] <ori>	 https://gerrit.wikimedia.org/r/#/c/105163/
[18:25:50] <paravoid>	 ouch
[18:25:56] <paravoid>	 this must haven't been fun to make
[18:26:24] <ori>	 i was going to say... that was more manual than i care to admit
[18:26:33] <paravoid>	 so
[18:26:37] <paravoid>	 we didn't swap TTLs to 60s
[18:26:52] <paravoid>	 which means that now we'll be in the awkward situation that traffic will gradually switch over the course of 1 hour
[18:27:09] <ori>	 right, i figured just !log the fact that gdash will be wacky for a bit.
[18:27:22] <Reedy>	 It won't affect that many people
[18:27:25] <paravoid>	 so, let's merge it, but puppetd --disable on professor so that gdash dashboards don't get picked up
[18:27:38] <ori>	 gdash.wikimedia.org is already tungsten
[18:27:56] <paravoid>	 oh right
[18:28:52] <ori>	 i wonder if i could just move everything in professor under MediaWiki/ too
[18:29:03] <paravoid>	 what do we need gdash.pmtpa.wmnet for?
[18:29:27] <ori>	 i don't know, probably nothing
[18:29:33] <ori>	 i just updated it since it was there
[18:29:45] <paravoid>	 how about graphite.pmtpa.wmnet? log target?
[18:31:05] <ori>	 it is not used in that way currently (thought it'd make sense to). i figured it was just a handy alias if graphite is failing and the on-duty person doesn't happen to remember which host it is on
[18:31:14] <paravoid>	 nah
[18:31:16] <paravoid>	 let's drop both
[18:31:42] <paravoid>	 and let's make the graphite CNAME TTL to 600 for now
[18:31:51] <paravoid>	 and I'll merge now
[18:31:55] <paravoid>	 want to do it or should I?
[18:32:39] <ori>	 i can update the patch, sure
[18:33:01] <ori>	 is there more to it than just removing the lines?
[18:33:08] <paravoid>	 nope
[18:35:56] <grrrit-wm>	 (03PS3) 10Ori.livneh: Point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 
[18:36:20] <logmsgbot>	 !log aaron started scap: active Testing timing
[18:36:37] <morebots>	 Logged the message, Master
[18:36:39] <paravoid>	 templates/apache/sites/graphite.wikimedia.org.erb:              ProxyPass http://graphite.pmtpa.wmnet:81/
[18:37:09] <ori>	 yeah, that's all going away
[18:37:16] <paravoid>	 but not for another hour
[18:37:19] <paravoid>	 do you care?
[18:37:56] <ori>	 well, i guess it doesn't hurt to change it
[18:38:11] * ori  updates that, too

[18:38:12] <paravoid>	 do we write to both carbons atm?
[18:38:18] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 (owner: 10Ori.livneh)
[18:38:37] <Reedy>	 jenkins is telling lies
[18:41:04] <MaxSem>	 ^d & manybubbles, will Cirrus fix http://thedailywtf.com/Articles/Lightspeed-is-Too-Slow-for-MY-Luggage.aspx ? :P
[18:41:18] <paravoid>	 it obviously doesn't matter much now and I won't hold this back, but what we generally do is set TTL to some low-value first (e.g. 1m), wait until the old TTL expires, swap while keeping the old TTL, wait to see if everything works, bump TTL back to 1h
[18:41:42] <Nemo_bis>	 MaxSem: see #wikimedia-dev
[18:42:17] <^d>	 Search is hard.
[18:42:33] <logmsgbot>	 !log aaron started scap
[18:42:35] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] Point graphite CNAME at tungsten in eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/105166 (owner: 10Ori.livneh)
[18:42:35] <ori>	 paravoid: no, let's do this right
[18:42:41] <ori>	 heh
[18:42:43] <MaxSem>	 people inventing a dozen of similar looking quotes are harder
[18:42:50] <morebots>	 Logged the message, Master
[18:42:54] <paravoid>	 and I worked around the graphite.pmtpa.wmnet issue, don't bothe
[18:42:55] <paravoid>	 r
[18:43:22] <ori>	 ok, thank you
[18:43:26] <grrrit-wm>	 (03PS1) 10Manybubbles: Cirrus config updates [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105228 
[18:44:24] <paravoid>	 ok
[18:44:26] <paravoid>	 gerrit is broken
[18:44:29] <paravoid>	 yay
[18:44:51] <paravoid>	 oh wait
[18:44:59] <paravoid>	 scratch that, it was pebcak
[18:45:11] <AaronSchulz>	 ori: yeah, MW_SCAP_BETA makes do difference, it still does the same stuff
[18:45:19] <AaronSchulz>	 *no
[18:45:38] <ori>	 it is probably not being correctly exported to the subshell in which one of the dependent scripts is executing
[18:46:11] <AaronSchulz>	 even for the local sync-common
[18:46:31] <ori>	 maybe it's just old and set in its ways
[18:46:53] <AaronSchulz>	 probably why the --versions thing seemed to not have a effect sometimes
[18:47:02] * AaronSchulz  gets the python urge again

[18:47:12] <paravoid>	 ori: seems to work here
[18:47:17] <twkozlowski>	 https://gerrit.wikimedia.org/r/#/c/103107/ has effectively been abandoned by the GCI students, any admins willing to press the button?
[18:47:40] * ori  hugs paravoid

[18:47:54] <ori>	 wee, thanks. i thought professor was going to bury me
[18:47:58] <paravoid>	 oh wait
[18:48:11] <paravoid>	 https://graphite.wikimedia.org/render/?title=HTTP%205xx%20Responses%20-8hours&from=-8hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=2&lineMode=connected&target=color%28cactiStyle%28alias%28reqstats.500,%22500%20resp/min%22%29%29,%22red%22%29&target=color%28cactiStyle%28alias%28reqstats.5xx,%225xx%20resp/min%22%29%29,%22blue%22%29
[18:48:16] <paravoid>	 "no data"
[18:48:49] <ori>	 hmm.
[18:48:56] * ori  investigates where reqstats.* come from

[18:48:59] <paravoid>	 no MediaWiki. there
[18:49:12] <ori>	 well, I haven't merged the gdash change yet
[18:49:21] <paravoid>	 no, but I mean, this isn't mediawiki
[18:49:41] <paravoid>	 it's the squid logs -> carbon
[18:49:59] <paravoid>	 my $carbon_server = "10.0.6.30";
[18:49:59] <paravoid>	 my $carbon_port = 2003;
[18:50:00] <paravoid>	 yay
[18:50:55] <ori>	 oh, christ. let's not wait a week for that.
[18:51:37] <ori>	 i'll update it so that new stats go to tungsten
[18:51:42] <paravoid>	 already on it
[18:51:44] <twkozlowski>	 ori: https://gerrit.wikimedia.org/r/#/c/103107/ abandon please? :-)
[18:51:54] <paravoid>	 can you see if you can copy the data from professor?
[18:52:34] <ori>	 yep
[18:53:01] <twkozlowski>	 also https://gerrit.wikimedia.org/r/#/c/103355/ was dropped by the student :-(
[18:53:45] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Move sqstat (udp2log to carbon) to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/105230 
[18:54:18] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: gdash: Prefix all metric names with 'MediaWiki.' [operations/puppet] - 10https://gerrit.wikimedia.org/r/105163 (owner: 10Ori.livneh)
[18:54:25] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] gdash: Prefix all metric names with 'MediaWiki.' [operations/puppet] - 10https://gerrit.wikimedia.org/r/105163 (owner: 10Ori.livneh)
[18:54:33] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: Move sqstat (udp2log to carbon) to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/105230 
[18:54:39] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] Move sqstat (udp2log to carbon) to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/105230 (owner: 10Faidon Liambotis)
[18:54:57] <paravoid>	 oh wait, this is wrong
[18:54:58] <paravoid>	 damn
[18:55:22] <AaronSchulz>	 ori: doing sudo -u mwdeploy makes them fall off
[18:55:35] <ori>	 paravoid: i know, i'm fixing
[18:55:54] <ori>	 i didn't know about reqstats
[18:57:04] <paravoid>	 MediaWiki.reqstatsedits.en_wikipedia_org.tp99,
[18:57:05] <paravoid>	 also typo
[18:59:34] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Fix gdash reqstats dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/105232 
[18:59:37] <paravoid>	 ori: ^^
[18:59:48] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[19:00:40] <paravoid>	 ori: although ideally we'd put it under cdn.* or something
[19:01:03] <hashar>	 !log apparently jenkins is back up and happy. Had to revert the git plugin to a previous version ...
[19:01:18] <morebots>	 Logged the message, Master
[19:01:35] <ori>	 oh, you're quick
[19:02:17] <paravoid>	 should had caught it in review, sorry about that
[19:02:19] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Fix gdash reqstats dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/105232 (owner: 10Faidon Liambotis)
[19:03:38] <paravoid>	 ori: did you find a way to copy the data?
[19:04:54] <ori>	 sorry, my isp had a hickup
[19:07:57] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: gdash: fix owner/group/mode for templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/105237 
[19:08:49] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] gdash: fix owner/group/mode for templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/105237 (owner: 10Faidon Liambotis)
[19:09:03] <paravoid>	 ...and now graphite times out
[19:09:04] <ori>	 thanks
[19:09:24] <paravoid>	 https://graphite.wikimedia.org/render/?title=10%20Most%20Deviant%20API%20Methods%20by%20Call%20Rate%20log(2)%20-8hours&from=-8hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&logBase=2&lineWidth=1&lineMode=connected&target=cactiStyle(substr(mostDeviant(10,maximumAbove(MediaWiki.API.*.count,1)),0,2))
[19:09:29] <paravoid>	 for example
[19:09:35] <ori>	 yeah, i got a 500 just now
[19:09:45] <ori>	 don't fix everything, let me figure it out
[19:09:56] <paravoid>	 lol, ok
[19:11:28] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Fixed scap variable exporting [operations/puppet] - 10https://gerrit.wikimedia.org/r/105238 
[19:11:36] <grrrit-wm>	 (03PS1) 10Jgreen: add root@backup4001's ssh key to fundraising backupmover auth keys [operations/puppet] - 10https://gerrit.wikimedia.org/r/105239 
[19:11:56] <AaronSchulz>	 ori: ^
[19:12:08] <paravoid>	 we're a little busy with breaking graphite :P
[19:13:58] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[19:16:33] <paravoid>	 ori: hey
[19:16:39] <ori>	 yep?
[19:16:45] <grrrit-wm>	 (03CR) 10Jgreen: [C: 032 V: 031] add root@backup4001's ssh key to fundraising backupmover auth keys [operations/puppet] - 10https://gerrit.wikimedia.org/r/105239 (owner: 10Jgreen)
[19:16:54] <paravoid>	 are you debugging?
[19:17:04] <ori>	 yeah, a bit stumped, but give me a moment
[19:17:14] <paravoid>	 it's uwsgi, did you find that?
[19:18:34] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] json: output NaN as null for '!num' modifiers [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105216 (owner: 10Edenhill)
[19:18:38] <ori>	 the uwsgi log files report successful reqs
[19:18:39] <ottomata>	 Snaps: thanks
[19:18:45] <ottomata>	 I'll rebuild that on Monday and deploy it then
[19:18:53] <ori>	 oh, no it doesn't
[19:18:58] <paravoid>	 nope
[19:19:04] <paravoid>	 and I also got the error myself
[19:19:09] <paravoid>	 <h1>uWSGI Error</h1>Python application not foundConnection closed by foreign host.
[19:19:31] <paravoid>	 (it's a 500, varnish thinks it's a transient error and it retries indefinitely, hence the timeout)
[19:21:57] <paravoid>	 it's the graphite search index I think
[19:22:08] <paravoid>	 it's _graphite:_graphite
[19:22:16] <paravoid>	 and the app runs as www-data, so it can't write to it
[19:22:35] <paravoid>	 ori: ^
[19:22:53] <paravoid>	 you just chowned it
[19:23:00] <paravoid>	 I'll take that as a "yes" :)
[19:23:18] <ori>	 hey
[19:23:21] <ori>	 yeah, i just figured that out
[19:23:49] <ori>	 that needs to be fixed more permanently, i'll take care of that
[19:23:50] <paravoid>	 did you see the comment in the /usr/share/graphite-web/graphite.wsgi source?
[19:24:32] <ori>	 sigh, no. that is a little obscure.
[19:26:39] <paravoid>	 professor has 16 x 70G disks
[19:26:42] <paravoid>	 lol
[19:27:15] <paravoid>	 73G 15k RPM SAS
[19:27:18] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[19:28:11] <ori>	 i'm confused. is it just choking on the requests at the moment?
[19:28:22] <ori>	 because memcached is empty or something?
[19:28:59] <paravoid>	 Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
[19:29:02] <paravoid>	 sda               0.00     2.80    1.20 3550.00     7.20 14211.20     8.01   143.65   40.39   58.67   40.39   0.28 100.00
[19:29:05] <paravoid>	 100% i/o
[19:29:09] <paravoid>	 while rendering
[19:29:46] <paravoid>	 and, presumably, flushing data to disk
[19:29:56] <paravoid>	 I've seen professor go 100% i/o too at times
[19:30:53] <paravoid>	 ganglia says tungsten is better iowait-wise
[19:30:55] <ori>	 but why does reloading a graph after it has been generated make it hang? it should be caching, no?
[19:32:02] <paravoid>	 I don't know how graphite works tbh
[19:32:04] <ori>	 'tcpdump tcp port 11211' doesn't show any traffic, so i must have misconfigured it somewhere
[19:32:43] <paravoid>	 (on an unrelated note, pypy with the version in precise doesn't seem like a great idea, it's really old and Debian explicitly didn't want to release with that version)
[19:33:01] <ori>	 it won't be needed after today
[19:33:13] <ori>	 !log Some migration pains while moving graphite from professor to tungsten; expect graphite & gdash flakiness
[19:33:28] <morebots>	 Logged the message, Master
[19:34:12] <domas>	 another system with lots of files!
[19:38:00] <paravoid>	 are you sure graphite uses memcache for graphs?
[19:38:42] <paravoid>	 If set, enables the caching of calculated targets (including applied functions) and rendered images. 
[19:38:42] <ori>	 MEMCACHE_HOSTS: "If set, enables the caching of calculated targets (including applied functions) and rendered images. "
[19:38:45] <ori>	 heh
[19:38:46] <paravoid>	 yeah :)
[19:39:45] <paravoid>	 found it
[19:39:57] <paravoid>	 if MEMCACHE_HOSTS: CACHE_BACKEND = 'memcached://' + ';'.join(MEMCACHE_HOSTS) + ('/?timeout=%d' % DEFAULT_CACHE_DURATION)
[19:40:01] <paravoid>	 that's django 1.2
[19:40:03] <paravoid>	 we run django 1.3
[19:40:10] <paravoid>	 which has moved to a CACHES dictionary
[19:40:32] <paravoid>	 hm, django might have a backwards compatibility stanza though
[19:41:17] <paravoid>	 if not settings.CACHES: legacy_backend = getattr(settings, 'CACHE_BACKEND', None) if legacy_backend:
[19:41:32] <domas>	 how does graphite have 500G of data
[19:41:34] <domas>	 heh
[19:42:09] <ori>	 paravoid: tail -f /var/log/graphite-web/cache.log
[19:43:17] <paravoid>	 domas: logging time series data forever for lots of things
[19:43:30] <paravoid>	 with no rrd-like aggregation
[19:43:54] <AaronSchulz>	 heh
[19:45:58] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[19:46:14] <paravoid>	 ori: found it
[19:46:18] <paravoid>	 oh wait no
[19:46:21] <paravoid>	 dammit :)
[19:46:32] <paravoid>	 W11: Warning: File "render/datalib.py" has changed since editing started
[19:46:52] <ori>	 i made the log output the port it's trying to connect on
[19:47:57] <ori>	 for later: 'service uwsgi stop' leaves uwsgi-core instances running
[19:49:19] <ori>	 is it supposed to be trying to hit the pickle_receiver_port?
[19:49:55] <paravoid>	 netstat -nap |grep 2204 shows multiple TIME_WAIT, plus one established
[19:51:52] <paravoid>	 wait
[19:51:56] <paravoid>	 so it pops from connectionPool
[19:52:45] <paravoid>	 if it errors out, it nevers puts the connection back again, which makes sense
[19:55:34] <paravoid>	   File "/usr/lib/python2.7/dist-packages/graphite/render/datalib.py", line 191, in recv_exactly
[19:55:37] <paravoid>	     data = conn.recv( num_bytes - len(buf) )
[19:55:40] <paravoid>	 timeout: timed out
[19:55:42] <paravoid>	 there you go
[19:55:55] <paravoid>	 that's why connections keep getting added to the pool
[19:56:34] <paravoid>	 Fri Jan 03 19:47:06 2014 :: Retrieval of cactiStyle(substr(mostDeviant(10,maximumAbove(MediaWiki.API.*.count,1)),0,2)) took 219.277028
[19:56:37] <paravoid>	 Fri Jan 03 19:47:07 2014 :: Rendered PNG in 0.453549 seconds
[19:56:40] <paravoid>	 Fri Jan 03 19:47:07 2014 :: Total rendering time 219.770963 seconds
[19:56:43] <paravoid>	 220 seconds
[20:02:29] <ori>	 paravoid: templates/varnish/graphite.inc.vcl.erb sets explicit TTLs
[20:02:57] <paravoid>	 ok?
[20:03:03] <ori>	 paravoid: that's not the config that is active on tungsten; tungsten's is in templates/varnish/misc.inc.vcl.erb
[20:03:09] <paravoid>	 I know
[20:03:31] <paravoid>	 220s to render a graph doesn't sound great in any case
[20:03:35] <ori>	 do you suppose that would help? i guess not, yeah.
[20:03:52] <paravoid>	 and there's no traffic on the memcached port on localhost
[20:03:59] <paravoid>	 hm, now there is
[20:04:05] <paravoid>	 but very minimal, still
[20:06:49] <paravoid>	 the graphs are empty now, are you aware of this?
[20:06:59] <paravoid>	 ah that's the reqstats ones
[20:10:06] <paravoid>	 let's increase CARBONLINK_TIMEOUT for starters
[20:10:18] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[20:10:20] <paravoid>	 now it's 1s
[20:10:33] <paravoid>	 so that times out, the cache lookup times out, and it's rendered every time
[20:12:40] <paravoid>	 it times out even with 20
[20:12:56] <paravoid>	 so the cache isn't very much of a cache probably
[20:13:18] <ori>	 something is badly misconfigured somewhere
[20:13:22] <ori>	 performance should not be this bad
[20:13:47] <ori>	 i want to investigate it more before modifying the configuration for longer timeouts
[20:14:06] <paravoid>	 uhm
[20:14:14] <paravoid>	 I think I found it...
[20:14:28] <ori>	 i have both face and palm ready
[20:14:38] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[20:15:47] <paravoid>	 fixed
[20:16:34] <ori>	 ......
[20:17:49] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: graphite: use the correct port for carbonlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/105250 
[20:18:08] <paravoid>	 hm, maybe that's not right
[20:18:24] <paravoid>	 well, the port is the right thing to do, but I'm looking if that hash is used elsewhere
[20:19:13] <paravoid>	 yes it is, in carbon.conf
[20:20:47] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "This is wrong as it changes carbon's [relay] as well." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105250 (owner: 10Faidon Liambotis)
[20:22:50] <paravoid>	 ori: 1) the above, correct port for carbonlink, 2) copying reqstats data from professor, 3) using apache's mod_expires to set some more sensible TTLs (the 120/600s seemed ok to me)
[20:23:16] <ori>	 no, wait
[20:23:23] <ori>	 paravoid: 
[20:23:35] <ori>	 i moved reqstats/ backed to archived/, just move it one dir up
[20:23:38] <ori>	 it is freshly rsync'ed
[20:23:55] <ori>	 and let's try it out with the correct config before changing apache
[20:23:56] <paravoid>	 oh and 4) http://graphite.wikimedia.org/ fails right away
[20:24:05] <paravoid>	 why archived?
[20:24:11] <paravoid>	 the hierarchy is the same, we didn't change it
[20:24:33] <ori>	 i just moved it out of the way at one point because i suspected there was something about it that was causing this
[20:24:42] <ori>	 the mismatch between the declared aggregation configs and the files themselves
[20:24:44] <paravoid>	 and I did change CARBONLINK_HOSTS by hand and it just works now
[20:24:54] <paravoid>	 (puppet is disabled for now)
[20:26:25] <ori>	 do you grok the 'tip' box in https://graphite.readthedocs.org/en/latest/config-carbon.html ?
[20:26:28] <ori>	 i'm very confused by it
[20:28:46] <paravoid>	 I think it means that if you had a cache listening on 2003 & 2004
[20:29:05] <paravoid>	 you can move 2003 & 2004 to the relay, create some new port numbers for your cache and then set them to the relay's destination
[20:29:24] <paravoid>	 it's poorly phrased in any case
[20:29:34] <ori>	 which is how this is configured
[20:29:37] <paravoid>	 correct
[20:29:44] <paravoid>	 graphite-web was misconfigured
[20:29:55] <paravoid>	 to talk with the pickle port, instead of the cache query port
[20:30:13] <paravoid>	 carbon is just fine I think
[20:30:37] <paravoid>	 (my puppet commit above is wrong, as it changes both, hence my -1)
[20:30:54] <grrrit-wm>	 (03PS2) 10Manybubbles: Cirrus config updates [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105228 
[20:32:41] <grrrit-wm>	 (03PS3) 10Manybubbles: Cirrus config updates [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105228 
[20:34:34] <ori>	 paravoid: yes, you are right; http://rcrowley.org/articles/federated-graphite.html confirms too
[20:34:45] <ori>	 i'll change local_settings.py.erb to grab the correct port number
[20:37:39] <paravoid>	 it's called "cache query" for a reason I thought
[20:42:21] <ori>	 don't be mean; i was mislead by advice that the two settings should exactly match
[20:42:36] <paravoid>	 oh sorry, I didn't intend to be mean
[20:42:44] <paravoid>	 I'm explaining my thought process
[20:42:58] <paravoid>	 that it was more intuition because of that, rather than something else
[20:43:41] <ori>	 well, also 11:49 <ori> is it supposed to be trying to hit the pickle_receiver_port?
[20:43:46] <ori>	 but i didn't follow it
[20:43:48] <paravoid>	 hehe
[20:43:52] <paravoid>	 happens!
[20:46:46] <ori>	 i need a couple more minutes for the erb, thanks for the patience
[20:48:45] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: graphite: use both virt0/virt1000 for AuthLDAPURL [operations/puppet] - 10https://gerrit.wikimedia.org/r/105381 
[20:49:04] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] graphite: use both virt0/virt1000 for AuthLDAPURL [operations/puppet] - 10https://gerrit.wikimedia.org/r/105381 (owner: 10Faidon Liambotis)
[20:49:42] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [V: 032] graphite: use both virt0/virt1000 for AuthLDAPURL [operations/puppet] - 10https://gerrit.wikimedia.org/r/105381 (owner: 10Faidon Liambotis)
[20:51:47] <paravoid>	 ori: another very tiny optimization that you might consider making is switching to pylibmc
[20:52:14] <paravoid>	 i.e. apt-get install python-pylibmc + switch from MEMCACHE_HOSTS to a django 1.3's CACHES with the pylibmc backend
[20:53:02] <paravoid>	 probably isn't worth it considering the memcache traffic I'm seeing atm
[20:57:01] <grrrit-wm>	 (03PS2) 10Ori.livneh: graphite-web: use the correct port for carbonlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/105250 (owner: 10Faidon Liambotis)
[20:58:13] <ori>	 i don't have a VM provisioned for testing that at the moment
[20:58:59] <ori>	 but I think it's correct. the DESTINATIONS setting was right; it's only graphite-web's CARBONLINK_HOSTS that needs to be changed
[20:59:17] <paravoid>	 yes
[20:59:21] <paravoid>	 that's my understanding as well
[20:59:27] <ori>	 and it should match the relay's DESTINATIONS in terms of the order of carbon caches instances
[21:00:18] <ori>	 it's a bit unfortunate that we infer CARBONLINK_HOSTS from DESTINATIONS, but we don't infer DESTINATIONS from the set of [cache:*] declarations
[21:00:42] <ori>	 it might call for a graphite::carbon_cache puppet resource
[21:00:42] <paravoid>	 I... was about to say this..
[21:01:05] <ori>	 or it might call for going in the other direction and making things repetitive but non-magical
[21:01:19] <ori>	 i prefer the former, as you might have guessed :P
[21:01:52] <ori>	 because my taste for erb magic has never, ever bitten me
[21:02:33] <ori>	 anyways, I think that change is fine
[21:02:36] <ori>	 ok if I merge?
[21:03:04] <paravoid>	 yes
[21:03:20] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Add carbon-relay & statsd service aliases [operations/dns] - 10https://gerrit.wikimedia.org/r/105383 
[21:03:28] <paravoid>	 ^^^ what do you think of that?
[21:03:44] <paravoid>	 and before you ask me why it's carbon-relay and not carbon, we have a server named carbon...
[21:04:14] <paravoid>	 I supposed restoring "graphite" would be equally as good, although technically the protocol is carbon, not graphite
[21:04:20] <paravoid>	 but I won't bikeshed, whatever you want :)
[21:04:36] <paravoid>	 but I want to git grep tungsten in puppet & mediawiki-config and use service hostnames instead
[21:05:01] <ori>	 the problem with having a statsd service alias is that statsd instances have a one-size-fits-all configuration for how data should be aggregated, which summary metrics should be computed, and which backends they should be routed to
[21:05:29] <paravoid>	 what do you mean?
[21:05:29] <ori>	 i worked around it by exploiting the fact that the configuration file is loaded and evaluated to hack in a plugin-like system into the puppet module
[21:05:46] <paravoid>	 I want to use it in all the places we've hardcoded tungsten
[21:05:46] <ori>	 but it's still the case that we probably want multiple instances running
[21:06:06] <ori>	 i think we need more than one statsd instance on the cluster is what i'm saying
[21:06:08] <paravoid>	 ah, I see
[21:06:31] <ori>	 for example: https://github.com/etsy/statsd/blob/master/exampleConfig.js
[21:06:39] <ori>	 the 'histogram' config
[21:06:52] <paravoid>	 ugh
[21:07:34] <ori>	 the bins for memcached latency are not going to be the same for page load timing
[21:07:35] <ori>	 etc.
[21:09:20] <ori>	 i would like to make it easier to specify aggregation policies by metric regex, like graphite, and submit that upstream, but that'll be a while
[21:09:32] <ori>	 is jenkins dead? https://gerrit.wikimedia.org/r/#/c/105250/
[21:09:38] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] graphite-web: use the correct port for carbonlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/105250 (owner: 10Faidon Liambotis)
[21:11:21] <ori>	 anyways, let me publicly embarrass you for a moment and acknowledge that you've been awesome & extremely helpful with the graphite work in general and the past hour in particular
[21:12:09] <paravoid>	 shush
[21:12:31] <paravoid>	 your change is broken
[21:12:34] <paravoid>	 :P
[21:13:09] <paravoid>	 no :a :b :c :d
[21:13:10] <ori>	 really? it seems to have made the right change
[21:13:17] <ori>	 ugh
[21:13:24] <ori>	 so that's another reason it was broken before, too
[21:13:25] <ori>	 -CARBONLINK_HOSTS = ["127.0.0.1:7102:a", "127.0.0.1:7202:b", "127.0.0.1:7302:c", "127.0.0.1:7402:d"]
[21:13:26] <ori>	 +CARBONLINK_HOSTS = ["127.0.0.1:2104:a", "127.0.0.1:2204:b", "127.0.0.1:2304:c", "127.0.0.1:2404:d"]
[21:13:34] <paravoid>	 -CARBONLINK_HOSTS = ["127.0.0.1:2104:a", "127.0.0.1:2204:b", "127.0.0.1:2304:c", "127.0.0.1:2404:d"]
[21:13:37] <paravoid>	 +CARBONLINK_HOSTS = ["127.0.0.1:7102", "127.0.0.1:7202", "127.0.0.1:7302", "127.0.0.1:7402"]
[21:13:40] <paravoid>	 is what I got
[21:13:44] <paravoid>	 you probably run puppet before it was merged
[21:13:54] <paravoid>	 before you run puppet-merge I suppose, because I did that
[21:14:00] <ori>	 oh, right, ugh
[21:14:11] <ori>	 well, it's your change, so your mistake, as far as the historical record is concerned
[21:14:17] <paravoid>	 right!
[21:14:17] <ori>	 i'll now swoop in for the fix
[21:16:26] <paravoid>	 so, shall we put reqstats under CDN.* or something?
[21:16:55] <ori>	 why CDN?
[21:17:06] <paravoid>	 because gdash says "(cdn)" before the graphs
[21:17:07] <grrrit-wm>	 (03PS1) 10Ori.livneh: graphite-web: Specify instance names in CARBONLINK_HOSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/105385 
[21:17:19] <paravoid>	 and I didn't feel like inventing a name
[21:18:00] <paravoid>	 but I'm okay with anything else, I'm just saying that now that it's all neat with the mediawiki hierarchy, maybe it makes sense to not put reqstats in root
[21:18:05] <paravoid>	 under root I mean
[21:18:52] <paravoid>	 wouldn't that make it "cache:a" instead of "a"?
[21:19:01] <paravoid>	 oh wait, no
[21:19:22] <grrrit-wm>	 (03PS5) 10BryanDavis: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 
[21:19:50] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Can we name it "instance" instead of "name"? (and probably "server" instead of "host", too, to match what Graphite source's uses)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105385 (owner: 10Ori.livneh)
[21:19:59] <ori>	 I think it's odd to put reqstats in root, yeah. I don't like 'CDN', though I don't have a better idea. Can we defer that decision?
[21:20:15] <paravoid>	 sure, I don't mind
[21:20:21] <paravoid>	 as long as these graphs start working again, though :)
[21:20:30] <paravoid>	 so it's going to have to be root for now
[21:20:37] <paravoid>	 (they're very useful to me)
[21:21:29] <grrrit-wm>	 (03PS2) 10Ori.livneh: graphite-web: Specify instance names in CARBONLINK_HOSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/105385 
[21:22:00] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] graphite-web: Specify instance names in CARBONLINK_HOSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/105385 (owner: 10Ori.livneh)
[21:22:07] <ori>	 i'll run puppet
[21:22:13] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [V: 032] graphite-web: Specify instance names in CARBONLINK_HOSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/105385 (owner: 10Ori.livneh)
[21:22:24] <paravoid>	 ok
[21:27:27] <ori>	 well, things look a lot better
[21:27:35] <grrrit-wm>	 (03CR) 10BryanDavis: "Antoine created https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/kibana. Someone with Push rights in that project still" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis)
[21:28:05] <ori>	 reqstats will take a bit to look right
[21:28:16] <paravoid>	 ?
[21:28:34] <paravoid>	 I run puppet on emery but haven't restarted udp2log yet, so I don't think the change has taken effect
[21:28:43] <ori>	 ah, i see
[21:28:44] <paravoid>	 but I'd like to restore professor's data first
[21:28:52] <ori>	 i did
[21:28:54] <paravoid>	 oh
[21:29:04] <ori>	 oh, hm
[21:29:21] <paravoid>	 it still says nan?
[21:29:36] <ori>	 not for everything, the 1wk have data
[21:29:43] <ori>	 i must not have rsynced properly
[21:29:55] <ori>	 re-syncing
[21:31:07] <ori>	 ok, i have to run to the office. i'm still at home and i'm interviewing someone in half an hour.
[21:31:15] <paravoid>	 eek
[21:31:17] <ori>	 rsync is running. paravoid, ok if i take off from your perspective?
[21:31:25] <paravoid>	 ok
[21:31:57] <ori>	 thanks again (x100) for the help
[22:09:59] <paravoid>	 ori: I'm restarting rsync, it was wrong
[22:11:28] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[22:17:01] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: gdash dashboards: s/white/black/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/105392 
[22:17:16] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] gdash dashboards: s/white/black/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/105392 (owner: 10Faidon Liambotis)
[22:17:28] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [V: 032] gdash dashboards: s/white/black/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/105392 (owner: 10Faidon Liambotis)
[22:42:22] <Nemo_bis>	 ah, the colour wasn't intentional :)
[23:28:28] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx  
[23:30:30] <greg-g>	 I thought that was fixed? ^^