[00:00:17] * rdwrer  looks at greg-g

[00:01:42] <icinga-wm>	 PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 7.318 second response time  
[00:03:46] <rdwrer>	 THIS IS THOROUGHLY USELESS
[00:03:56] <rdwrer>	 greg-g's calendar is marked as "appt"
[00:04:25] <rdwrer>	 Should I just go with the LD, or...
[00:05:46] <bd808>	 rdwrer: No known cluster issues I think. If you have an LD and some opsen around to help if things go bad… Be Bold!
[00:06:06] <rdwrer>	 *shrug* 'kay whatever
[00:06:20] <rdwrer>	 There will be logs
[00:06:34] <grrrit-wm>	 (03CR) 10MarkTraceur: [C: 032] Start sampling detailed network performance for Multimedia Viewer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112452 (owner: 10Gilles)
[00:06:40] <grrrit-wm>	 (03Merged) 10jenkins-bot: Start sampling detailed network performance for Multimedia Viewer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112452 (owner: 10Gilles)
[00:07:38] <logmsgbot>	 !log mholmquist updated /a/common to {{Gerrit|I1319008cb}}: Start sampling detailed network performance for Multimedia Viewer
[00:07:45] <morebots>	 Logged the message, Master
[00:08:42] <icinga-wm>	 RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 297412 bytes in 7.561 second response time  
[00:09:00] <rdwrer>	 LIGHTENING DEPLOYYYYYYYYY
[00:09:04] <logmsgbot>	 !log mholmquist synchronized wmf-config/CommonSettings-labs.php  'Set MMV network performance sampling level to 100% on labs'
[00:09:11] <morebots>	 Logged the message, Master
[00:09:43] <rdwrer>	 AND AGAIIIIIIIN
[00:10:00] <logmsgbot>	 !log mholmquist synchronized wmf-config/CommonSettings.php  'Set MMV network performance sampling level to 0.1% on production wikis'
[00:10:08] <morebots>	 Logged the message, Master
[00:10:08] <rdwrer>	 gi11es: Safety is overrated, your config patch is out
[00:10:12] <rdwrer>	 Praise Thor
[00:11:03] <rdwrer>	 Everything working still; good stuff
[00:12:15] <rdwrer>	 https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=100039&oldid=100034
[00:12:18] <rdwrer>	 That is all
[00:13:33] <ori>	 gi11es, rdwrer: mmv.performance.js is awesome as hell.. if it works well we should adapt it for inclusion in core imo
[00:14:46] <rdwrer>	 No objection here
[00:14:56] <paravoid>	 hrmmmm
[00:15:00] <paravoid>	 this looks very interesting
[00:15:01] <paravoid>	 nice job
[00:15:26] <rdwrer>	 Anyone else itching to push out in the LD or should I stop paying attention
[00:15:28] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Wikiversities and itwikiquote are done building [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114169 (owner: 10Chad)
[00:15:36] <grrrit-wm>	 (03Merged) 10jenkins-bot: Wikiversities and itwikiquote are done building [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114169 (owner: 10Chad)
[00:16:09] <logmsgbot>	 !log demon synchronized wmf-config/InitialiseSettings.php  'Wikiversities and itwikiquote get Cirrus as beta'
[00:16:17] <morebots>	 Logged the message, Master
[00:16:23] <^d>	 rdwrer: Well I was itching, but I scratched it
[00:16:35] <rdwrer>	 ^d: I was about to say, don't announce yourself or anything :P
[00:17:51] <^d>	 I HAD A CHANGE TO DEPLOY BUT I'M DONE NOW KTHNX
[00:47:01] <bd808>	 !log Testing changes to mw-update-l10n on tin
[00:47:09] <morebots>	 Logged the message, Master
[00:47:36] <bd808>	 I'm going to test a script on tin; please don't scap while I'm in there
[00:51:19] <bd808>	 !log mw-update-l10n test on tin completed successfully
[00:51:26] <morebots>	 Logged the message, Master
[01:01:21] <bd808>	 One more mw-update-l10n test incoming; no scaps please
[01:01:30] <bd808>	 !log Testing changes to mw-update-l10n on tin
[01:01:38] <morebots>	 Logged the message, Master
[01:04:56] <bd808>	 !log mw-update-l10n test on tin completed
[01:05:05] <morebots>	 Logged the message, Master
[01:23:22] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:25:12] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.960 second response time  
[01:33:04] <grrrit-wm>	 (03PS3) 10Reedy: Remove UserThrottle extension and log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113897 
[01:33:08] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Remove UserThrottle extension and log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113897 (owner: 10Reedy)
[01:33:15] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove UserThrottle extension and log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113897 (owner: 10Reedy)
[01:42:03] <grrrit-wm>	 (03PS1) 10Chad: Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 
[01:51:22] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:53:11] <^d>	 I think wikitechwiki is dead ^
[01:53:15] <^d>	 People reporting can't login, etc.
[01:56:33] <icinga-wm>	 PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:58:23] <icinga-wm>	 RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.081 second response time  
[01:58:30] <andrewbogott>	 ^d:  better?
[01:58:44] <rdwrer>	 Doesn't look like it actually
[01:58:58] <^d>	 Still can't login here either.
[01:59:43] <rdwrer>	 OK so it's not virt0?
[02:00:13] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.172 second response time  
[02:00:26] <andrewbogott>	 ^d, what behavior do you see?
[02:01:14] <^d>	 Login page loads, clicking submit just hangs forever.
[02:01:20] <^d>	 "Waiting for wikitech.wikimedia.org..."
[02:03:22] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:03:25] <^d>	 Now the page won't even load. Times out.
[02:03:33] <icinga-wm>	 PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:04:23] <andrewbogott>	 ok, I see it too now.
[02:04:27] <andrewbogott>	 Can't tell what's happening...
[02:05:55] <grrrit-wm>	 (03PS2) 10Chad: Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 
[02:06:12] <andrewbogott>	 Coren, still around?
[02:09:29] <andrewbogott>	 66.249.66.163 - - [20/Feb/2014:02:00:56 +0000] "GET /index.php?title=Software_deployments&diff=50404&oldid=prev HTTP/1.1" 302 601 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
[02:15:51] <andrewbogott>	 ^d, if I want to forbit acess to all special pages in robots.txt...
[02:15:59] <andrewbogott>	 Disallow: /wiki/Special:*  ?
[02:16:04] <andrewbogott>	 I don't know if wildcards work in that context
[02:16:17] <^d>	 That'd work, yes.
[02:16:22] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.651 second response time  
[02:16:23] <icinga-wm>	 RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 1.305 second response time  
[02:16:25] <andrewbogott>	 ok.  Will see if that helps...
[02:16:26] <^d>	 But I thought we already fixed wikitech's robots.
[02:16:40] <andrewbogott>	 Yes, but I'm being more strident.  Right now special pages are still allowed...
[02:16:43] <^d>	 Oh, just some of them.
[02:16:44] <^d>	 Got it.
[02:16:54] <andrewbogott>	 I don't know why it's happening since that implies that googlebot has a login...?
[02:17:23] <andrewbogott>	 but ldap is very busy!
[02:18:51] <^d>	 There were requests from googlebot to special:userlogin?
[02:20:42] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Stop bots from crawling all special pages. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114391 
[02:21:36] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Stop bots from crawling all special pages. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114391 (owner: 10Andrew Bogott)
[02:22:04] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf14) at 2014-02-20 02:22:04+00:00
[02:22:12] <morebots>	 Logged the message, Master
[02:25:02] <^d>	 andrewbogott: Seems much more responsive now. Thanks!
[02:25:17] <andrewbogott>	 I hope it was my change and not just googlebot getting tired out
[02:26:27] <^d>	 googlebot has a bad habit of making non-cluster-wiki things fall over :\
[02:41:15] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf13) at 2014-02-20 02:41:15+00:00
[02:41:22] <morebots>	 Logged the message, Master
[03:02:30] <grrrit-wm>	 (03PS1) 10Chad: WIP: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[03:03:33] <grrrit-wm>	 (03PS2) 10Chad: WIP: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[03:17:52] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:18:42] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1067 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[03:18:42] <icinga-wm>	 PROBLEM - Varnish HTCP daemon on cp1067 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[03:26:31] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-20 03:26:31+00:00
[03:26:38] <morebots>	 Logged the message, Master
[03:33:33] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1067 is OK: PROCS OK: 2 processes with command name varnishncsa  
[03:33:33] <icinga-wm>	 RECOVERY - Varnish HTCP daemon on cp1067 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd  
[03:33:42] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1067 is OK: HTTP OK: HTTP/1.1 200 OK - 188 bytes in 0.290 second response time  
[03:46:20] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Attempt to get ssh keys working, pre-puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/114408 
[03:48:29] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Syslog is broken on first launch, maybe because of this? [operations/puppet] - 10https://gerrit.wikimedia.org/r/114410 
[03:51:04] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Syslog is broken on first launch, maybe because of this? [operations/puppet] - 10https://gerrit.wikimedia.org/r/114410 
[03:52:19] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Syslog is broken on first launch, maybe because of this? [operations/puppet] - 10https://gerrit.wikimedia.org/r/114410 (owner: 10Andrew Bogott)
[03:56:09] <grrrit-wm>	 (03PS1) 10coren: Labs: Kick sshd when /public/keys is mounted [operations/puppet] - 10https://gerrit.wikimedia.org/r/114413 
[03:57:43] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 031] Labs: Kick sshd when /public/keys is mounted [operations/puppet] - 10https://gerrit.wikimedia.org/r/114413 (owner: 10coren)
[03:59:41] <grrrit-wm>	 (03CR) 10coren: [C: 04-1] "I really very much dislike the idea of the first boot being dependent on the NFS server; even with the bg mount option, it could potential" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114408 (owner: 10Andrew Bogott)
[04:01:18] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Labs: Kick sshd when /public/keys is mounted [operations/puppet] - 10https://gerrit.wikimedia.org/r/114413 (owner: 10coren)
[04:35:26] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Further attempt to redirect syslog to console for labs instances. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114414 
[04:49:12] <grrrit-wm>	 (03PS1) 10coren: labs_vmbuilder: make sure console=ttyS0 is there [operations/puppet] - 10https://gerrit.wikimedia.org/r/114415 
[04:54:34] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] labs_vmbuilder: make sure console=ttyS0 is there [operations/puppet] - 10https://gerrit.wikimedia.org/r/114415 (owner: 10coren)
[04:57:16] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Revert "Syslog is broken on first launch, maybe because of this?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114416 
[04:58:24] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Revert "Syslog is broken on first launch, maybe because of this?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114416 (owner: 10Andrew Bogott)
[05:21:15] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Add redis config for keystone in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/104322 (owner: 10Ryan Lane)
[05:24:19] <grrrit-wm>	 (03PS2) 10Ryan Lane: Add redis support to keystone [operations/puppet] - 10https://gerrit.wikimedia.org/r/105139 
[05:24:42] <grrrit-wm>	 (03PS3) 10Ryan Lane: Add redis support to keystone in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/105139 
[05:27:53] <grrrit-wm>	 (03PS4) 10Ryan Lane: Add redis support to keystone in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/105139 
[05:28:05] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Add redis support to keystone in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/105139 (owner: 10Ryan Lane)
[06:00:46] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Turns out every two minutes is not often enough. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114422 
[06:17:14] <grrrit-wm>	 (03PS1) 10Ryan Lane: Enable keystone redis driver for eqiad. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114423 
[06:17:22] <Ryan_Lane>	 andrewbogott: ^^
[06:18:44] <andrewbogott>	 Ryan_Lane: if it was in common before...
[06:19:01] <Ryan_Lane>	 I wanted to limit it by region
[06:19:18] <andrewbogott>	 ah, ok.  So this patch is meant to subtract, not add.
[06:19:21] <andrewbogott>	 Looks good, in that case.
[06:19:31] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 031] Enable keystone redis driver for eqiad. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114423 (owner: 10Ryan Lane)
[06:19:33] <grrrit-wm>	 (03PS2) 10KartikMistry: Remove $wgULSNoWebfontsSelectors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114131 
[06:19:35] <Ryan_Lane>	 it's meant to move it from common to site specific config
[06:19:43] <Ryan_Lane>	 then I can test it in a single place
[06:19:58] <Ryan_Lane>	 we'll want to make sure it's working properly for eqiad
[06:22:31] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Enable keystone redis driver for eqiad. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114423 (owner: 10Ryan Lane)
[06:22:51] <inahandizha>	 http://VisitsToMoney.com/index.php?refId=386970
[06:23:39] <Ryan_Lane>	 bleh. I forgot something
[06:28:49] <grrrit-wm>	 (03PS1) 10Ryan Lane: Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 
[06:29:16] <Ryan_Lane>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/114424
[06:29:26] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 (owner: 10Ryan Lane)
[06:29:37] <Ryan_Lane>	 -_-
[06:32:34] <grrrit-wm>	 (03PS2) 10Ryan Lane: Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 
[06:33:11] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 (owner: 10Ryan Lane)
[06:34:18] <Ryan_Lane>	 wtf am I doing wrong?
[06:35:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:36:20] <Ryan_Lane>	 andrewbogott: do you know why jenkins is complaining about that line?
[06:37:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:38:02] <andrewbogott>	 not immediately...
[06:38:52] <grrrit-wm>	 (03PS3) 10Ryan Lane: Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 
[06:38:55] <Ryan_Lane>	 I hate puppet
[06:39:27] <andrewbogott>	 well… I don't understand what the logic is there
[06:39:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:39:56] <andrewbogott>	 It's syntax I haven't seen before
[06:40:02] <Ryan_Lane>	 basically it's just setting a different hash if it's in labs or production
[06:40:04] <Ryan_Lane>	 based on a case
[06:40:31] <ori>	 well, for one, it looks like the way the redis module uses $replication is different
[06:40:43] <ori>	 it expects either a 'site' or a 'hostname' key on the hash
[06:40:52] <ori>	 see redis.conf.erb
[06:41:09] <ori>	 you're mapping one hostname to another hostname so i'm not sure how that would work
[06:41:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:41:40] <Ryan_Lane>	 I'm following the example of the other uses of that
[06:41:50] <Ryan_Lane>	 unless it's been changed since the original changeset
[06:42:07] <Ryan_Lane>	 the error I was getting had nothing to do with that, though
[06:42:11] <Ryan_Lane>	 it had to do with my syntax
[06:42:23] <ori>	 http://docs.puppetlabs.com/puppet/latest/reference/lang_conditional.html#selectors
[06:42:31] <ori>	 Values may be any of the following:
[06:42:35] <ori>	 Any literal value, with the exception of hash literals 
[06:43:00] <Ryan_Lane>	 for instance:
[06:43:00] <Ryan_Lane>	     $redis_replication = {
[06:43:00] <Ryan_Lane>	         'rdb1002' => 'rdb1001',
[06:43:00] <Ryan_Lane>	         'rdb1004' => 'rdb1003',
[06:43:00] <Ryan_Lane>	     }
[06:43:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:43:49] <Ryan_Lane>	 bleh. you can do the equivalent of a case, but you can't use a hash as the value?
[06:43:58] <ori>	 but that's being passed to role::db::redis, not ::redis
[06:43:58] <Ryan_Lane>	 this is why it's a terrible idea to make your own shitty DSL
[06:45:12] <Ryan_Lane>	 in role::db::redis:
[06:45:12] <Ryan_Lane>	                 class { "::redis":
[06:45:12] <Ryan_Lane>	                         maxmemory => $maxmemory,
[06:45:12] <Ryan_Lane>	                         persist => "aof",
[06:45:12] <Ryan_Lane>	                         redis_replication => $redis_replication,
[06:45:12] <Ryan_Lane>	                         password => $passwords::redis::main_password,
[06:45:12] <Ryan_Lane>	                 }
[06:45:15] <ori>	 they fucked it up so badly though, i think it's letting it off the hook too lightly to suggest that it just boils down to grammars being hard
[06:45:22] <Ryan_Lane>	 you realize I've tested this in labs already, right?
[06:45:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:45:40] <ori>	 well, you haven't tested the hash literals thing, because it couldn't have worked
[06:45:54] <Ryan_Lane>	 replication is working there
[06:45:58] <Ryan_Lane>	 so it must be
[06:46:06] <Ryan_Lane>	 and I'm calling it exactly like in production
[06:46:18] <ori>	 	if $::realm == "labs" { 
[06:46:21] <ori>	 			redis_replication	  => undef, 
[06:46:25] <ori>	 in redisdb.pp
[06:46:37] <Ryan_Lane>	 I'm not using redisdb
[06:47:17] <Ryan_Lane>	 ori: https://gerrit.wikimedia.org/r/#/c/104322/3/manifests/role/keystone.pp,unified
[06:47:22] <Ryan_Lane>	 that's what I was using
[06:47:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:48:12] <Ryan_Lane>	 see node /^rdb100[1-4]\.eqiad\.wmnet/
[06:48:19] <Ryan_Lane>	 then trace through to redisdb
[06:48:55] <Ryan_Lane>	 and see that it's using $redis_replication directly. so, unless I'm missing something, I should be doing this correctly
[06:49:04] <Ryan_Lane>	 or production is not
[06:49:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:51:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:51:37] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Combine labs and production classes for keystone redis and include with keystone server [operations/puppet] - 10https://gerrit.wikimedia.org/r/114424 (owner: 10Ryan Lane)
[06:53:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:54:28] <grrrit-wm>	 (03PS1) 10Ryan Lane: Don't split keystone token password config by realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/114425 
[06:55:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:56:12] <grrrit-wm>	 (03PS2) 10Ryan Lane: Move the keystone token driver password back into common [operations/puppet] - 10https://gerrit.wikimedia.org/r/114425 
[06:57:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[06:57:36] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Move the keystone token driver password back into common [operations/puppet] - 10https://gerrit.wikimedia.org/r/114425 (owner: 10Ryan Lane)
[06:59:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 06:30:35 AM UTC  
[07:00:53] <icinga-wm>	 RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Thu Feb 20 07:00:52 UTC 2014  
[07:02:33] <icinga-wm>	 PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Thu 20 Feb 2014 07:00:52 AM UTC  
[07:05:18] <Ryan_Lane>	 bleh. I need to use hostname and not fqdn
[07:08:08] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Specific metadata_host, in case that helps. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114428 
[07:08:13] <grrrit-wm>	 (03PS1) 10Ryan Lane: Change redis config hash key to hostname rather than fqdn [operations/puppet] - 10https://gerrit.wikimedia.org/r/114429 
[07:09:47] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Change redis config hash key to hostname rather than fqdn [operations/puppet] - 10https://gerrit.wikimedia.org/r/114429 (owner: 10Ryan Lane)
[07:19:55] <grrrit-wm>	 (03PS1) 10Ryan Lane: Add base::firewall to openstack's firewall class [operations/puppet] - 10https://gerrit.wikimedia.org/r/114430 
[07:20:55] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032 V: 032] Add base::firewall to openstack's firewall class [operations/puppet] - 10https://gerrit.wikimedia.org/r/114430 (owner: 10Ryan Lane)
[07:26:46] <grrrit-wm>	 (03PS1) 10Ryan Lane: Add http/https to ferm rules on wikitech [operations/puppet] - 10https://gerrit.wikimedia.org/r/114431 
[07:27:26] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 031] "yes please!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114431 (owner: 10Ryan Lane)
[07:27:31] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Add http/https to ferm rules on wikitech [operations/puppet] - 10https://gerrit.wikimedia.org/r/114431 (owner: 10Ryan Lane)
[07:27:37] <grrrit-wm>	 (03CR) 10Ryan Lane: [V: 032] Add http/https to ferm rules on wikitech [operations/puppet] - 10https://gerrit.wikimedia.org/r/114431 (owner: 10Ryan Lane)
[07:30:55] <icinga-wm>	 RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Thu Feb 20 07:30:47 UTC 2014  
[07:39:03] <grrrit-wm>	 (03PS1) 10Ryan Lane: Fix labs_nodes subnet in firewall config [operations/puppet] - 10https://gerrit.wikimedia.org/r/114434 
[07:40:20] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032 V: 032] Fix labs_nodes subnet in firewall config [operations/puppet] - 10https://gerrit.wikimedia.org/r/114434 (owner: 10Ryan Lane)
[07:45:57] <grrrit-wm>	 (03PS1) 10Ryan Lane: Add redis config to havana's keystone config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/114435 
[07:48:53] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Add redis config to havana's keystone config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/114435 (owner: 10Ryan Lane)
[08:20:13] <grrrit-wm>	 (03PS1) 10Ryan Lane: Add DNS ferm rules for labs DNS [operations/puppet] - 10https://gerrit.wikimedia.org/r/114436 
[08:20:22] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032 V: 032] Add DNS ferm rules for labs DNS [operations/puppet] - 10https://gerrit.wikimedia.org/r/114436 (owner: 10Ryan Lane)
[08:26:11] <grrrit-wm>	 (03PS1) 10Ryan Lane: Switch back to sql driver for keystone in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114437 
[08:31:40] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Switch back to sql driver for keystone in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114437 (owner: 10Ryan Lane)
[08:55:23] <grrrit-wm>	 (03PS1) 10Ori.livneh: Remove now-duplicate scap scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114441 
[09:01:40] <grrrit-wm>	 (03PS2) 10Ori.livneh: Remove now-duplicate scap scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114441 
[09:03:13] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Remove now-duplicate scap scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114441 (owner: 10Ori.livneh)
[09:50:55] <icinga-wm>	 PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset -299.9772283 secs  
[10:00:35] <icinga-wm>	 PROBLEM - Disk space on virt10 is CRITICAL: DISK CRITICAL - free space: /var/lib/nova/instances 44119 MB (3% inode=99%):  
[10:19:55] <icinga-wm>	 RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.0003002882004 secs  
[10:33:29] <grrrit-wm>	 (03CR) 10Nikerabbit: [C: 031] Remove $wgULSNoWebfontsSelectors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114131 (owner: 10KartikMistry)
[10:38:55] <grrrit-wm>	 (03PS1) 10Ori.livneh: bits: gzip image/svg+xml & image/x-icon [operations/puppet] - 10https://gerrit.wikimedia.org/r/114448 
[11:06:59] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-2] "Duplicate of" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114448 (owner: 10Ori.livneh)
[11:07:07] <paravoid>	 :D
[11:07:38] <ori>	 ah, that's great actually
[11:07:48] <ori>	 brion: thanks!
[11:09:08] <ori>	 paravoid: if you were excited to review the patch and were disappointed to realize it was a dupe, there are always the geoip patches :P
[11:09:58] <grrrit-wm>	 (03CR) 10Nemo bis: Work in progress: set .svg and .ico files to be compressed on bits.wikimedia.org (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113687 (owner: 10Brion VIBBER)
[11:15:02] <grrrit-wm>	 (03CR) 10Nemo bis: "So here all requests to the sites send GeoIP information, even if not used?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113935 (owner: 10Ori.livneh)
[11:16:15] <ori>	 Nemo_bis: all page requests include a <script src="//bits.wikimedia.org/geoiplookup">, which pulls GeoIP information via a separate request
[11:16:48] <ori>	 encoding it as a session cookie means you can eliminate that request in the vast majority of cases
[11:17:15] <ori>	 because you just let the info piggy-back on the primary page request
[11:17:59] <ori>	 it's a blocking request in the critical path (before <body>) so eliminating it would be great
[11:25:33] <scfc_de>	 What is the GeoIP information used for at the moment?
[11:29:00] <ori>	 fundraiser is foremost, but there is a long tail of random gadgets and bits of site JS
[11:30:27] <ori>	 i think ULS uses it for localization.. that's probably the most valid use-case tho
[11:32:30] <scfc_de>	 k
[11:54:59] <qchris>	 When looking at EventLogging data, I see a >100ms drop in connection setup time between 2014-02-13 and 2014-02-14 for SE Asia.
[11:55:04] <qchris>	 Were there some changes around that time that could explain that?
[11:55:45] <paravoid>	 are you sure it's between the 13th and 14th?
[11:55:56] <paravoid>	 on the 12th I switched Hong Kong to ulsfo
[11:55:57] <qchris>	 EventLogging tells me so.
[11:56:22] <qchris>	 Ok I'll double check.
[11:56:25] <qchris>	 Thanks.
[11:56:54] <paravoid>	 CommitDate: Wed Feb 12 17:04:16 2014 +0200
[11:56:56] <paravoid>	 specifically
[11:57:27] <paravoid>	 and on the 14th we had a ulsfo outage which had us reverting to eqiad for approx. 3 hours
[11:57:59] <matanya>	 thu united layer thing?
[11:58:05] <matanya>	 *the
[11:58:25] <paravoid>	 indeed
[11:59:04] <ori>	 on the 14th I changed NT to capture mw load time (load time via js timers) from all browsers, not just the ones that don't support navigation timing
[11:59:12] <matanya>	 btw paravoid started packaging the deps of jsduck for the debian archive
[11:59:26] <ori>	 but if it's b/w 13 & 14, it sounds attributable to faidon's change
[11:59:50] <matanya>	 so i hope at the end jsduck will be in debian archive (will take some days, i guess)
[12:07:16] <qchris>	 Ok. Thanks a lot!
[12:09:58] <ori>	 np, it's cool to have more people scrutinize this data
[12:10:07] * ori  sleeps

[12:10:40] <qchris>	 Scrutinize is a too big word. "Just taking a random peek" would more closely reflect reality :-)
[13:43:44] <grrrit-wm>	 (03PS1) 10coren: Tweaks to instance build postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114458 
[14:10:44] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] "I can't think of any reason we wouldn't want this in the long run." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114458 (owner: 10coren)
[14:23:39] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Turns out every two minutes is not often enough. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114422 (owner: 10Andrew Bogott)
[14:24:01] <grrrit-wm>	 (03Abandoned) 10Andrew Bogott: Specific metadata_host, in case that helps. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114428 (owner: 10Andrew Bogott)
[14:39:31] <Snaps>	 hey ottomata: did you reach any conclusion from the tcptracing yesterday?
[14:44:25] <ottomata>	 Snaps: no
[14:44:33] <drdee>	 Snaps: we did notice continuous shrinking/expanding of the ISR on both brokers because the batch size was larger than the max lag messages and that was fixed but we won't know for a while if that solves it
[14:44:37] <ottomata>	 drdee and I did find that with our batch size and with teh replica.max.lag setting
[14:44:39] <ottomata>	 ah yes
[14:44:39] <ottomata>	 :)
[14:44:54] <ottomata>	 so yeah, but i doubt that it is the problem
[14:44:58] <ottomata>	 drdee thinks it was
[14:44:59] <drdee>	 Snaps: did you see tha graphs i generated?
[14:45:00] <ottomata>	 so now we have a bet
[14:45:40] <Snaps>	 oh, interesting. so for each full batch the replicas would fall so far behind that they would not be considered in-sync?
[14:45:41] <drdee>	 well it's not really a bet if the outcome is known ahead of time
[14:46:16] <drdee>	 we ran into this issue: https://issues.apache.org/jira/browse/KAFKA-766
[14:46:18] <Snaps>	 drdee: I saw colored lines, nice colored lines, but Im not sure I understood them
[14:46:31] <drdee>	 it's art
[14:46:37] <drdee>	 you superimpose meaning on top of it
[14:48:31] <ottomata>	 haha
[14:48:34] <ottomata>	 Snaps, ja so
[14:48:41] <ottomata>	 batch size is 6000 msgs currently
[14:48:54] <ottomata>	 replica.max.lag is 4000 by default
[14:49:07] <Snaps>	 thats funny
[14:49:09] <ottomata>	 so occaisionally and very briefly
[14:49:30] <drdee>	 Snaps: but more seriously, i think the cause is probably not in the network but with the broker itself
[14:49:30] <ottomata>	 a replica would look to be out of sync, and be removed from the ISR
[14:49:38] <drdee>	 that's why we started digging in the log files
[14:49:40] <ottomata>	 only to be re-added a few milliseconds to a couple of seconds later
[14:50:08] <ottomata>	 yeah, i tend to agree there, the fact that this only happens on an22 is either a fluke or means there is something weird with an22
[14:50:17] <ottomata>	 it could still be networking to an22, but we haven't seen anything wrong there
[14:51:14] <Snaps>	 it seems likely that ISR changes might cause internal broker ops that delay request handling, yeah?
[14:51:55] <ottomata>	 its possible, yeah, but, i think that is run by the controller, which i think is a separate thread, no?
[14:52:04] <ottomata>	 also, only the followers were removed from the ISR
[14:52:05] <ottomata>	 no leaders
[14:52:12] <ottomata>	 so there shouldn't have been any production problems
[14:52:20] <Snaps>	 and reques.required.acks is 1, yeah?
[14:52:55] <ottomata>	 yeah
[14:53:19] <Snaps>	 ok
[14:53:53] <ottomata>	 also, just fyi, i reduced kafka.queue.buffering.max.messages to 500000 from the 2M we had it at
[14:53:58] <ottomata>	 i don't think it was helping us at all
[14:54:06] <ottomata>	 and it only caused vk to use more memory and to take a long time to dequeue
[14:55:24] <Snaps>	 it means that you are more likely to drop messages in case of broker unavailability though, but I guess thats okay
[14:56:29] <grrrit-wm>	 (03PS1) 10coren: More fixes to postinst.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114470 
[14:59:53] <Snaps>	 so, what do you guys think? problem solved?
[15:00:01] <drdee>	 that's the bet
[15:00:03] <drdee>	 i think it is
[15:00:09] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] More fixes to postinst.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114470 (owner: 10coren)
[15:00:10] <drdee>	 ottomata disagrees
[15:01:11] <ottomata>	 naw i doubt it
[15:01:15] <drdee>	 i am sure ottomata will be right after all -- but the prospect of him buying  me a couple of beers was good enough to make a fool of myself
[15:01:30] <ottomata>	 Snaps, at 6000 msgs a second,  500000 messages is 83 seconds of buffer
[15:01:31] <ottomata>	 which is plenty
[15:01:38] <ottomata>	 if the broker is out for more than that, then that is a bigger problem
[15:01:45] <drdee>	 what i would like is to speed up the debug cycle :)
[15:01:46] <ottomata>	 we woudl rather drop than fill up memory on varnishes
[15:02:13] <Snaps>	 fair enough
[15:02:28] <Snaps>	 but you dont think that the ISR flapping is the root of the issue here?
[15:02:56] <Snaps>	 and seeing how only a few of the varnishen have seen this problem (cp3019 was it?) I tend to agree
[15:16:28] <drdee>	 is it always cp3019 and that's the only varnish which had this problem?
[15:19:11] <drdee>	 ottomata: would turn on verbose logging on an22 for a couple of hours be useful?
[15:19:43] <ottomata>	 it happens to any of the bits varnishes
[15:19:46] <ottomata>	 sometimes more than one at a time
[15:19:47] <ottomata>	 but not always
[15:19:56] <ottomata>	 bits esams varnishes onlyu
[15:20:12] <ottomata>	 and it happens only for messages bound to analytics1022
[15:21:45] <drdee>	 so cp3019 <-> an21 is fine?
[15:25:58] <ottomata>	 everything <-> an21 is fine
[15:26:17] <ottomata>	 cp3019-cp3022 <-> occasionally (usually during high load) have this problem
[15:26:21] <ottomata>	 <-> an22
[15:26:22] <ottomata>	 *
[15:31:35] <Snaps>	 drdee: one of your charts from yesterday showed a very jagged tcp session, while the other one was more smooth
[15:31:40] <Snaps>	 did you make anything of that?
[15:32:17] <drdee>	 i showed them to mark as well so the conclusion was "yes we have bursty traffic, but we don't know why"
[15:39:18] <Snaps>	 so comparing outstanding data for cp3021 and ..19. whats the Y unit and which color does it correlate to?
[15:39:27] <Snaps>	 (your outstanding_data.png)
[15:40:22] <ottomata>	 so, bursty there makes sense
[15:40:27] <ottomata>	 i think irght?
[15:40:29] <ottomata>	 because at the time
[15:40:41] <ottomata>	 cp3021 had a full buffer
[15:40:47] <ottomata>	 so, it dropped any new messages
[15:40:54] <ottomata>	 and cp3021's buffer was full of traffic bound to an22
[15:41:17] <ottomata>	 so, vk periodicalyy (every retry interval?) try to resend messages that had failed in the buffer
[15:41:26] <ottomata>	 does that sound right snaps?
[15:42:15] <drdee>	 snaps: see http://www.tcptrace.org/manual/node12_tf.html for details on the outstanding data graph
[15:45:16] <drdee>	 Snaps, ottomata : i can also share the xplot data files with you, then you can zoom-in -- it's much more useful than those screenshots
[15:45:39] <Snaps>	 yes please
[15:46:10] <drdee>	 do you have dropbox?
[15:46:19] <drdee>	 the files are a couple of mb's
[15:48:35] <Snaps>	 ottomata: is txretries (in rdk stats) higher for those hosts?
[15:50:36] <Snaps>	 drdee: nope, lets hold with that for a bit
[15:50:45] <ottomata>	 Snaps: 
[15:50:45] <ottomata>	 http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=&vl=&x=&n=&hreg%5B%5D=cp30(19%7C2%5B0-2%5D)&mreg%5B%5D=kafka.rdkafka.brokers.*.txretries&gtype=line&glegend=show&aggregate=1
[15:51:05] <Snaps>	 is this rdk 0.8.3 or 0.8.1?
[15:51:24] <ottomata>	 0.8.1
[15:54:03] <Snaps>	 okay, so message retries didnt really work in 0.8.1, it was fixed in 0.8.3
[15:55:00] <ottomata>	 hm, ok, but according to ganglia there haven't been any retries?
[15:55:02] <ottomata>	 is that wrong?
[15:55:10] <Snaps>	 exactly :)
[15:55:21] <ottomata>	 ah, ha, ok
[15:55:39] <ottomata>	 really?  huh, so what does that mean for messages in the buffer that fail or timeout?
[15:55:42] <ottomata>	 in 0.8.1?
[15:55:56] <Snaps>	 and I would guess that varnishkafka.log says: Kafka message delivery failed: Local: Message timed out ?
[15:58:58] <Snaps>	 a message is subject to multiple timeouts: the message.timeout.ms (how long rdk will hold on to the message in its queues), request.timeout.ms (the broker's request handling timeout, not enforced in rdk),  socket.timeout.ms (how long we wait for an ack)
[15:59:26] <Snaps>	 the socket.timeout.ms is set to 60s default, and I dont think youve changed that?
[15:59:57] <Snaps>	 Which means that if the broker does not respond with an ack in 60s the produce request will be considered failed and the messages will be subject to transmission retry.
[16:00:25] <Snaps>	 however, in <0.8.3 this retry never took place so the messages were timed out instead, disregarding a higher message.timeout.ms setting.
[16:00:47] <Snaps>	 (message.timeout.ms defaults to 300s)
[16:01:53] <ottomata>	 Yes, Snaps, lots of Local: message timed out
[16:01:58] <ottomata>	 when this happens
[16:02:19] <ottomata>	 you think I should just upgrade?
[16:03:27] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Include puppet, nfs-client, sudo-ldap packages in base image. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114485 
[16:03:52] <andrewbogott>	 Coren, any reason to think ^ won't work?
[16:04:07] * Coren  reads

[16:04:38] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Include puppet, nfs-client, sudo-ldap packages in base image. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114485 
[16:04:41] <Snaps>	 ottomata: this is an error, I dont know if its the error.  
[16:05:09] <ottomata>	 aye
[16:05:13] <andrewbogott>	 Coren, do you think installing sudo-ldap there will get us around the no-root-password thing?
[16:05:33] <andrewbogott>	 Seems like it would, but I don't know quite how those bits work
[16:05:37] <Snaps>	 ottomata: but it would be interesting to upgrade one or two of the troublesome varnishes
[16:06:17] <Coren>	 I'm not sure.  It might break the build, I think the protection against removing sudo when there is no root password cannot be circumvented.  But it might well allow it since it's in build and not a live system.
[16:06:28] <grrrit-wm>	 (03CR) 10coren: [C: 032] "And Andrew saw everything that he had made, and behold, it was very good." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114485 (owner: 10Andrew Bogott)
[16:06:36] <ottomata>	 yeah, hm
[16:06:42] <andrewbogott>	 And I wonder if the addpkg list pulls in deps as well?
[16:06:47] <andrewbogott>	 If not then just 'puppet' is insufficient
[16:06:48] <ottomata>	 it would be Snaps, but that would still be hard to isolate
[16:07:04] <ottomata>	 i haven't noticed any pattern as to which of the esams bits do this
[16:07:06] <Coren>	 I'm pretty sure addpkg uses apt-get
[16:07:09] <ottomata>	 it is usually just one of them
[16:07:10] <ottomata>	 at a time
[16:07:12] <andrewbogott>	 ok
[16:07:14] <ottomata>	 but it could be any one of them
[16:07:19] <andrewbogott>	 I'll merge and start a new build.  Ready for a break anyway
[16:07:40] <ottomata>	 Snaps: how about I wait until Tuesday next week and then upgrade all of them
[16:07:57] <ottomata>	 that way we can at least make a guess as to if the change that drdee and I made fixed anything
[16:08:08] <ottomata>	 (and that way I can win the bet if it didn't :p)
[16:08:23] <ottomata>	 Actually, I have the better end of this bet, hmm, don't I?
[16:08:30] <ottomata>	 if it breaks again, then win
[16:08:32] <ottomata>	 if it doesn't
[16:08:35] <ottomata>	 we'll never know :p
[16:08:39] <ottomata>	 bwaahhha
[16:08:40] <drdee>	 then i win
[16:09:26] <Snaps>	 :)
[16:11:27] <drdee>	 Snaps: would you know a good solution for https://issues.apache.org/jira/browse/KAFKA-766 ?
[16:11:58] <ottomata>	 uh, isn't the solution to keep the isr check larger than batch size?
[16:12:46] <drdee>	 yeah but isn't that still fragile?
[16:13:30] <Snaps>	 maybe the isr check should be time based rather than message based.  i.e., are the replicas no longer than 30s behind, then insync, else not.
[16:14:47] <ottomata>	 it is time and message
[16:14:49] <ottomata>	 it has both
[16:14:51] <ottomata>	 whichever comes first
[16:15:04] <Snaps>	 then message should be infinite
[16:16:13] <Snaps>	 there is really no reliable way of adjusting the isr check msg cnt based on max incoming batch sizes, since there might be a multitude of batches rolling in.
[16:17:21] <grrrit-wm>	 (03PS1) 10Ottomata: Adding ISRShrinks and ISRExpands to Ganglia kafka view [operations/puppet] - 10https://gerrit.wikimedia.org/r/114487 
[16:17:39] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding ISRShrinks and ISRExpands to Ganglia kafka view [operations/puppet] - 10https://gerrit.wikimedia.org/r/114487 (owner: 10Ottomata)
[16:18:04] <^d>	 ottomata: I tweaked the elasticsearch LVS check last night.
[16:18:17] <^d>	 Still a WIP, but I think it's a step in the right direction instead of just pinging on port 9200
[16:27:12] <grrrit-wm>	 (03PS3) 10Chad: WIP: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[16:28:41] <grrrit-wm>	 (03CR) 10Chad: "Ok, with PS3 we check the response for the hostname. If you query the _nodes API with something it doesn't recognize you'll get an empty-i" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 (owner: 10Chad)
[16:30:01] <Bsadowski1>	 PlayStation 3?
[16:30:11] <grrrit-wm>	 (03PS4) 10Chad: WIP: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[16:30:18] <hoo>	 Bsadowski1: Patch set 3
[16:30:22] <Bsadowski1>	 Ah :p
[16:30:27] <hoo>	 ;)
[16:30:34] <grrrit-wm>	 (03PS1) 10Ottomata: Fixing metric for ISR expands and shrinks [operations/puppet] - 10https://gerrit.wikimedia.org/r/114489 
[16:30:43] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Fixing metric for ISR expands and shrinks [operations/puppet] - 10https://gerrit.wikimedia.org/r/114489 (owner: 10Ottomata)
[16:31:04] <^d>	 Bsadowski1: I've had every Playstation ;-)
[16:31:39] <Bsadowski1>	 Even the new one?
[16:31:51] <Snaps>	 ottomata,drdee: Im soon off for today, anything else?
[16:31:59] <ottomata>	 not at the moment, Snaps, thanks!
[16:32:03] <ottomata>	 right now I think we are just waiting
[16:32:04] <^d>	 Bsadowski1: Yeah. I took a vacation day on the day it arrived so I could play :p
[16:32:12] <Snaps>	 ottomata: okay, drop me a mail if theres anything. cheers!
[16:32:17] <ottomata>	 ho
[16:32:18] <ottomata>	 Snaps: 
[16:32:25] <Snaps>	 ?
[16:32:31] <ottomata>	 if I fix your comments for my kt_log change in kafkatee
[16:32:34] <ottomata>	 can I merge?
[16:32:48] <ottomata>	 want to rebuild that and work on it today
[16:32:56] * ^d  spies an ottomata

[16:32:59] <Snaps>	 removing the fprintfs? yeah!
[16:33:24] <grrrit-wm>	 (03PS2) 10Ottomata: Fixing metric for ISR expands and shrinks [operations/puppet] - 10https://gerrit.wikimedia.org/r/114489 
[16:33:25] <ottomata>	 cool danke
[16:33:29] <ottomata>	 heheheh, yup hi ^d
[16:33:31] <ottomata>	 with you in a sec!
[16:33:37] <^d>	 okie dokie
[16:33:45] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Fixing metric for ISR expands and shrinks [operations/puppet] - 10https://gerrit.wikimedia.org/r/114489 (owner: 10Ottomata)
[16:41:52] <grrrit-wm>	 (03CR) 10Ottomata: "Other than the one comment LGTM" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 (owner: 10Chad)
[16:43:51] <ottomata>	 drdee, still happening: http://ganglia.wikimedia.org/latest/graph_all_periods.php?hreg[]=analytics102%5B12%5D.%2A&mreg[]=kafka.server.ReplicaManager.ISRShrinks.FiveMinuteRate&z=large&gtype=line&title=kafka.server.ReplicaManager.ISRShrinks.FiveMinuteRate&aggregate=1&r=hour
[16:43:52] <ottomata>	 :/
[16:43:56] <ottomata>	 on an22 it looks like
[16:45:27] <drdee>	 weird
[16:45:48] <drdee>	 so that suggests that they do get out of sync for real
[16:49:39] <ottomata>	 i dunno, they are only out of sync for such a sort period of time though!
[16:49:47] <ottomata>	 maybe magnus is rigth
[16:50:03] <ottomata>	 we shoudl increase replica.lag.max.messages to a huge number, and just have it be a timeout
[16:50:12] <ottomata>	 5 or 10 seconds should be good
[16:50:40] <ottomata>	 but also, this is still strange that is is only happening on an22
[16:51:56] <greg-g>	 uhhh, is the login prompt for graphite wrong? It says "Wikimedia LDAP" which isn't clear (thought that was my google/OIT LDAP) but it's really my wikitech user/pass
[16:52:21] <Ryan_Lane>	 greg-g: nothing uses your google/oit ldap password
[16:52:38] <greg-g>	 "Wikimedia LDAP" is confusing then
[16:52:56] <greg-g>	 it used to say "you wikitech user/pass" or something of the sort
[16:53:03] * Ryan_Lane  nods

[16:53:41] <greg-g>	 either way, I'm in, minor nit, just noting somewhere informal :)
[16:54:26] <bd808>	 greg-g: Patches welcome. The label would be in the graphite role in operations/puppet somewhere
[16:54:46] <drdee>	 ottomata: yeah maybe go with Snaps' advice
[16:55:29] <greg-g>	 bd808: ty, seriously
[16:56:15] <grrrit-wm>	 (03CR) 10Chad: WIP: Improve LVS check for Elasticsearch (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 (owner: 10Chad)
[16:56:25] <grrrit-wm>	 (03PS5) 10Chad: WIP: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[16:56:29] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|I19a8c3298}}: Remove UserThrottle extension and log group
[16:56:38] <bd808>	 greg-g: templates/graphite/graphite.apache.erb: AuthName 'Wikimedia LDAP'
[16:56:38] <morebots>	 Logged the message, Master
[16:56:41] <greg-g>	 yep
[16:56:42] <^d>	 bd808: If you wouldn't mind chiming in on 114399 too ^
[16:56:42] <greg-g>	 found it
[16:56:53] <^d>	 It'd be useful to make sure I'm not insane on the _nodes api usage.
[16:57:02] * greg-g  is s/Wikimedia/Wikitech/ :)

[16:57:10] <grrrit-wm>	 (03PS6) 10Chad: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[16:57:16] <grrrit-wm>	 (03PS1) 10Reedy: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114493 
[16:57:18] <grrrit-wm>	 (03PS1) 10Reedy: Wikipedias to 1.23wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114494 
[16:57:20] <grrrit-wm>	 (03PS1) 10Reedy: group0 wikis to 1.23wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114495 
[16:57:33] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114493 (owner: 10Reedy)
[16:57:47] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114493 (owner: 10Reedy)
[16:57:57] * greg-g  goes to meeting and doesn't upload patch

[16:58:52] <logmsgbot>	 !log reedy Started scap: testwiki to 1.23wmf15 and build l10n cache
[16:58:58] <morebots>	 Logged the message, Master
[17:02:23] <^d>	 I'm getting invalid certificate errors from etherpad.wm.o
[17:03:09] <grrrit-wm>	 (03CR) 10Ottomata: "Chad, looks like the | -> ! change got reverted from patchset 5 to 6." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 (owner: 10Chad)
[17:03:16] <^d>	 Blahhhh
[17:03:19] <^d>	 I'll fix.
[17:04:07] <grrrit-wm>	 (03PS7) 10Chad: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[17:04:16] <^d>	 Ok, correct check, correct commit message.
[17:05:04] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Remove sudo-ldap and puppet, we need to do this a different way. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114496 
[17:06:58] <grrrit-wm>	 (03PS1) 10Odder: Change Luis Villa's blog URL on the English Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/114498 
[17:07:01] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Remove sudo-ldap and puppet, we need to do this a different way. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114496 
[17:07:34] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Remove sudo-ldap and puppet, we need to do this a different way. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114496 (owner: 10Andrew Bogott)
[17:14:58] <ottomata>	 ^d lgtm, shall I merge?
[17:15:29] <^d>	 Let's try it :)
[17:16:09] <grrrit-wm>	 (03PS1) 10coren: Add comment to clarify part of the postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 
[17:16:36] <grrrit-wm>	 (03PS2) 10coren: Add comment to clarify part of the postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 
[17:22:17] <grrrit-wm>	 (03PS8) 10Chad: Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 
[17:22:22] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Improve LVS check for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/114399 (owner: 10Chad)
[17:22:26] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 04-1] Add comment to clarify part of the postint.sh (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 (owner: 10coren)
[17:22:42] <ottomata>	 ^d, merged, where should I run puppet to check?
[17:23:15] <^d>	 I guess on the lvs boxes?
[17:23:32] <^d>	 I really don't know though. I'm guessing.
[17:42:41] <bd808>	 !log mw1151 & mw1152 throwing PHP Fatal error that looks to be APC or other cache corruption
[17:42:50] <morebots>	 Logged the message, Master
[17:47:32] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Clobber resolv.conf after we're done with our apt magic. [operations/puppet] - 10https://gerrit.wikimedia.org/r/114501 
[17:50:24] <grrrit-wm>	 (03PS1) 10Greg Grossmeier: Modify login credential hint [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 
[17:51:52] <greg-g>	 (after a wikidata meeting and a call from my dad telling me they're putting an offer in on a retirement house... finally git review'd)
[17:55:44] <^d>	 ottomata: I rethought it just now over a late breakfast. This isn't really what I wanted.
[17:55:47] <^d>	 Can we revert?
[17:55:51] <ottomata>	 haha
[17:55:54] <ottomata>	 whaaaa
[17:56:01] <ottomata>	 sure, submit a  patch :p
[17:57:29] <grrrit-wm>	 (03PS1) 10Chad: Revert "Improve LVS check for Elasticsearch" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114504 
[17:57:51] <^d>	 I think I explained myself
[17:58:27] <awjr>	 i am getting a fatal on mw.o: [129daf16] 2014-02-20 17:57:50: Fatal exception of type MWException
[17:58:29] <awjr>	 has that been reported yet?
[17:58:49] <^d>	 in -tech
[17:58:51] <awjr>	 Reedy: 	^
[17:58:52] <awjr>	 k
[17:59:00] <Reedy>	 Yeah
[17:59:23] <grrrit-wm>	 (03CR) 10BryanDavis: "Maybe we should settle on a standard label for this credential pool. It seems that various vhosts use different wording: "WMF Labs (use wi" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[17:59:24] <Reedy>	 Considering the cdb files shouldn't have been touched
[18:01:08] <^d>	 ottomata: https://gerrit.wikimedia.org/r/#/c/114504/ is the straight revert.
[18:02:22] <ottomata>	 ^d, are you just trying to get alerts of individual node status?
[18:02:58] <^d>	 I'm trying to improve what LVS uses to pool.
[18:03:05] <^d>	 But I realize that's in a totally different place.
[18:03:42] <^d>	 This check is checking the status of the front of lvs from icinga--could be a smarter test like I said but definitely shouldn't care about node-level.
[18:03:47] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Revert "Improve LVS check for Elasticsearch" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114504 (owner: 10Chad)
[18:04:37] <bd808>	 Reedy: My patch for the branch initialization bug "bootstrapped" the en l10n cache. It may not have picked up any extension messages.
[18:04:44] <bd808>	 That would be a problem
[18:05:08] <hexmode>	 Why https://bugzilla.wikimedia.org/61689 ???
[18:05:19] <hexmode>	 This seems new today.
[18:05:34] <hexmode>	 And I keep getting them
[18:06:05] <grrrit-wm>	 (03PS1) 10Ryan Lane: Fix virt1000 hostname in redis config [operations/puppet] - 10https://gerrit.wikimedia.org/r/114505 
[18:06:20] <bd808>	 Reedy: That's almost definately the problem. l10n_cache-en.cdb.json is way too small
[18:06:29] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Fix virt1000 hostname in redis config [operations/puppet] - 10https://gerrit.wikimedia.org/r/114505 (owner: 10Ryan Lane)
[18:06:33] <Reedy>	 Gah
[18:08:03] <bd808>	 So deleting /a/common/php-1.23wmf15/cache/l10n/l10n_cache-en.cdb and re-running mw-update-l10n should recreate.
[18:08:19] <bd808>	 Then you could push just the l10n_cache-en.cdb out
[18:08:38] <greg-g>	 hexmode: I can confirm, but mw.org hasn't updated yet... so this isn't new code :/
[18:09:27] <grrrit-wm>	 (03PS3) 10coren: Fix inconsistencies in postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 
[18:09:40] <bd808>	 greg-g: For some reason it seems that mediawikiwiki is looking at the php-1.23wmf15 l10n cache
[18:09:40] <grrrit-wm>	 (03PS1) 10Ryan Lane: Reenable redis for keystone in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114506 
[18:09:41] <Reedy>	 bd808: Noting scap hasn't finished yet though ;)
[18:09:52] <greg-g>	 bd808: whh......
[18:09:55] <Reedy>	 mediawikiwki is still on 1.23wmf15
[18:10:02] <hexmode>	 greg-g: ty for confirmation
[18:10:05] <Reedy>	 Or is that the new one?
[18:10:08] <Reedy>	 I'M CONFUSED
[18:10:13] <bd808>	 That's the new one
[18:10:32] <greg-g>	 15 is new, mw should still be pointing to 14
[18:10:42] <Reedy>	 reedy@tin:/a/common$ grep mediawiki wikiversions.dat
[18:10:42] <Reedy>	 mediawikiwiki php-1.23wmf14
[18:10:48] <greg-g>	 since you just did testwiki, afaict
[18:10:48] <Lcawte>	 I'm assuming the WMF icinga config is in a repo somewhere?
[18:10:49] <Reedy>	 It's having an identity crisis
[18:10:58] <greg-g>	 aren't we all
[18:10:58] <Reedy>	 Lcawte: Puppet repo
[18:11:08] <Lcawte>	 Ty :)
[18:14:03] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 031] Fix inconsistencies in postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 (owner: 10coren)
[18:14:14] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 032] Reenable redis for keystone in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/114506 (owner: 10Ryan Lane)
[18:15:22] <grrrit-wm>	 (03PS4) 10coren: Fix inconsistencies in postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 
[18:16:21] <^d>	 Weird. SSL error on etherpad seems transient. Gone now.
[18:17:55] <bd808>	 Reedy: Comparing again it seems that the php-1.23wmf1[345] l10n_cache-es.cdb.json sizes are consistent
[18:18:07] <bd808>	 Where is scap at in it's process?
[18:18:23] <bd808>	 Has it generated the cdb files from the json yet?
[18:18:27] <Reedy>	 Nope
[18:18:30] <Reedy>	 still pushing to both datacentres seemingly
[18:18:37] <bd808>	 So that may be the problem.
[18:18:37] <grrrit-wm>	 (03PS2) 10Ottomata: Kafkatee puppet module [operations/puppet/kafkatee] - 10https://gerrit.wikimedia.org/r/110650 
[18:18:42] <Reedy>	 mw67: 18:12:42 INFO - Finished rsync common (duration: 15m 38s)
[18:18:42] <Reedy>	 mw1147: 18:12:43 DEBUG - Copying to mw1147.eqiad.wmnet from tin.eqiad.wmnet
[18:19:07] <Reedy>	 Oh
[18:19:10] <Reedy>	 That was 7 minutes ago
[18:19:12] <bd808>	 Code is wanting new messages but the cdb is out of data
[18:19:15] <Reedy>	 No output since
[18:19:16] <kaldari>	 This graph doesn't look good: https://gdash.wikimedia.org/dashboards/reqerror/
[18:19:29] <Reedy>	 kaldari: only mediawikiwiki
[18:19:41] <kaldari>	 ah good :)
[18:19:58] <grrrit-wm>	 (03PS1) 10Brion VIBBER: Enable lower-res .ogv transcodes for fallback player on slow machines [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114507 
[18:21:13] <grrrit-wm>	 (03PS2) 10Brion VIBBER: Enable lower-res .ogv transcodes for fallback player on slow machines [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114507 
[18:21:19] <bd808>	 Reedy: if there has been no other output lets hope that mw1147 is the last scap-1 fetch
[18:21:42] <Reedy>	 there's 43 rsyncs still running it seems
[18:21:46] <greg-g>	 bah, gdash's 'show code deploys' tick isn't working
[18:21:47] <bd808>	 blerg
[18:22:06] <bd808>	 greg-g: It doesn't tick until the scap is over
[18:22:18] <bd808>	 which is a bug IMHO
[18:22:31] <bd808>	 It should mark start+end
[18:22:39] <greg-g>	 bd808: yeah, 1) do both started and finished, but in general, I see no code deploy lines for the past month
[18:26:00] <Reedy>	 mw1212: 18:12:45 DEBUG - Copying to mw1212.eqiad.wmnet from tin.eqiad.wmnet
[18:26:00] <Reedy>	 mw1212: 18:12:45 DEBUG - Started rsync common
[18:26:00] <Reedy>	 mw1099: 18:25:01 INFO - Finished rsync common (duration: 15m 31s)
[18:26:00] <Reedy>	 mw106: 18:25:02 DEBUG - Copying to mw106.pmtpa.wmnet from tin.eqiad.wmnet
[18:26:43] <Reedy>	 I wonder if it just timed that most of them finished, meaning the next ones started would take a further 15 minutes or so
[18:27:03] <Reedy>	 snapshot3 isn't happy
[18:27:06] <Reedy>	 not that we care about it
[18:27:29] <bd808>	 Reedy: That's the time on each individual host.
[18:27:40] <Reedy>	 Right
[18:27:44] <Reedy>	 But that's what I mean
[18:27:49] <Reedy>	 All of the running ones finished
[18:28:07] <Reedy>	 More were started, and because there was no staggered start of the last batch,t here was nothing for numerous minutes
[18:28:27] <Reedy>	 but it's still syncing to EQIAD hosts
[18:28:34] <Reedy>	 (as well as tampa)
[18:28:54] <bd808>	 "Copying to mw1212.eqiad.wmnet from tin.eqiad.wmnet"
[18:29:00] <bd808>	 Are they all pulling from tin?
[18:29:14] <bd808>	 That would be shitty.
[18:29:22] <Reedy>	 Haha
[18:29:23] <Reedy>	 Yup
[18:29:48] <bd808>	 Fffffuuuuuu
[18:29:57] <Reedy>	 https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Miscellaneous+eqiad&h=tin.eqiad.wmnet&host_regex=&max_graphs=0&tab=m&vn=&hide-hf=false&sh=1&z=small&hc=4
[18:29:58] <Reedy>	 Poor tin
[18:31:01] <grrrit-wm>	 (03CR) 10Greg Grossmeier: "Agreed. So we appear to have:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[18:33:02] <grrrit-wm>	 (03CR) 10Ryan Lane: [C: 031] Fix inconsistencies in postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 (owner: 10coren)
[18:33:40] <bd808>	 Everything pulling from tin is Ori and my fault. I'll submit a patch to back out our replacement for use-cluster_run
[18:34:18] <Reedy>	 Think you may have set a record for the worlds slowest scap :D
[18:36:17] <Reedy>	 What order does it process the dsh list?
[18:36:31] <yuvipanda>	 bd808: Reedy I suppose you are aware of ' No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php.'
[18:36:43] <brion>	 testwiki seems dead: "[70918cab] /wiki/Main_Page Exception from line 468 of /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php."
[18:36:48] <Reedy>	 For 40 minutes or so
[18:36:49] <bd808>	 Reedy: randomized
[18:36:52] <brion>	 fun times
[18:36:53] <Reedy>	 ...
[18:36:54] <Reedy>	 WHY!?
[18:37:12] <James_F>	 brion: Same on MediaWiki.org
[18:37:15] <bd808>	 Because legacy behavior
[18:37:32] <Reedy>	 I thought the dsh list was done in machine order?
[18:38:26] <Reedy>	 And for that reason I made a change to do all the EQIAD hosts first
[18:39:17] <bd808>	 Reedy: From scap-old `shuf < /etc/dsh/group/mediawiki-installation > "$NODEFILE"`
[18:39:23] <Reedy>	 :/
[18:39:26] <Reedy>	 :(
[18:39:35] <Reedy>	 Which also begs the question, again, can we stop pushing code to PMTPA?
[18:39:35] <bd808>	 "# Do the main code update in random order to avoid overloading any given rsync server"
[18:40:00] <Reedy>	 Progress: rand() %
[18:40:11] <bd808>	 Which I obviously did badly
[18:40:16] <greg-g>	 :/
[18:40:22] <ori>	 :(
[18:40:34] <ori>	 can we stop having a sad and falling on swords?
[18:40:44] <bd808>	 ori: Review https://gerrit.wikimedia.org/r/#/c/114511 plz
[18:40:47] * ori  does

[18:41:19] <greg-g>	 ori: but we have all these swords, what do YOU want to do with them?
[18:41:34] <greg-g>	 "Kill all the scap demons" is the right answer
[18:41:35] <bd808>	 Sheath them in the blood of our enemies?
[18:41:54] <greg-g>	 bd808's is close enough/the meaning is the same.
[18:42:10] <Reedy>	 cmjohnson1: Can you just get all of tampa turned off? Kthx
[18:42:30] <cmjohnson1>	 consider it done
[18:42:32] <ori>	 bd808: is sync_from set?
[18:42:33] <greg-g>	 I hear the DC has a feature for that, called "Cut ur cablez". It's a free service, too.
[18:42:38] <cmjohnson1>	 !log flipping the switch to off in tampa
[18:42:47] <morebots>	 Logged the message, Master
[18:43:04] <greg-g>	 that's going to scare tomasz
[18:43:13] <cmjohnson1>	 greg-g that sounds like a great service! 
[18:43:30] <ori>	 bd808: no, it's not
[18:43:45] <ori>	 oh hm
[18:44:06] <ori>	 yeah, it's not
[18:44:06] <ori>	 sys.exit(scap.sync_common())
[18:44:13] <ori>	 so sync_from is None
[18:44:17] <ori>	 so the function never gets called
[18:44:43] <bd808>	 ori: That's main.sync_common
[18:44:51] <dr0ptp4kt>	 ori, you around and able to spare 8 minutes?
[18:44:57] <bd808>	 Which becomes asks.sync_common(args.cfg, args.servers)
[18:45:08] <bd808>	 *tasks
[18:45:14] <ori>	 there are two sync_commons?
[18:45:24] * bd808  hangs head in shame

[18:45:44] <ori>	 dr0ptp4kt: free between 3pm and 3:30pm
[18:45:53] <dr0ptp4kt>	 ori, k, will put it on calendar
[18:46:34] <Reedy>	 mw1010: 18:44:17 INFO - Finished rsync common (duration: 00m 25s)
[18:46:38] <Reedy>	 2nd sync-common?
[18:47:02] <bd808>	 Now it's going to tell them all to rebuild cdb
[18:47:30] <Reedy>	 It's not finished doing the first one yet
[18:48:34] <bd808>	 mw1010 is in both the proxy group and the installation group
[18:48:35] <ori>	 bd808: sync_common(dict(args.cfg.items() + env.items()))
[18:48:44] <bd808>	 That's the local sync
[18:48:46] <ori>	 def sync_common(cfg, sync_from=None):
[18:49:10] <ori>	 where is sync_from parameter getting passed?
[18:49:25] <bd808>	 Nowhere in that call. That's tin to tin
[18:49:28] <bd808>	 utils.dsh(
[18:49:28] <bd808>	                         '/usr/local/bin/scap-1 "%s"' % ' '.join(scap_proxies),
[18:49:28] <bd808>	                         tmp.name, env)
[18:49:56] <bd808>	 Inside the tempfile with is the apache sync
[18:50:04] <grrrit-wm>	 (03CR) 10Reedy: "This apparently was essentially a no-op." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105006 (owner: 10Reedy)
[18:50:24] <ori>	 mhmm
[18:50:39] <ori>	 and how is that making it to the sync_from argument?
[18:51:07] <bd808>	 scap.main.sync_common: asks.sync_common(args.cfg, args.servers)
[18:51:12] <bd808>	 *tasks
[18:51:22] <bd808>	 cunt-n-paste is hard
[18:51:27] <bd808>	 sorry
[18:51:39] <ori>	 oh, damn it
[18:52:27] <bd808>	 It's the list change
[18:52:47] <bd808>	 the task expects a list now and it's getting all hosts in one arg
[18:53:32] <bd808>	 We added the quotes for the old behavior and then I didn't remove them in the updated patch
[18:54:14] <ori>	 yep
[18:58:03] <huh>	 hi
[18:58:06] <huh>	 progress?
[18:58:20] <greg-g>	 huh: in-progress
[18:58:30] <huh>	 k
[18:58:34] <greg-g>	 I assume you're referring to the exceptions on mw.org?
[18:58:39] <greg-g>	 (and test.wikipedia)
[18:59:03] <huh>	 Yes.
[18:59:12] <greg-g>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=61689
[18:59:24] <huh>	 Yeah, saw the bug
[18:59:35] <huh>	 even commented but I don't know what I'm talking about
[18:59:59] <greg-g>	 oh, you're PiR2, gotcha
[19:00:03] <ori>	 bd808: are you updating?
[19:00:06] <bd808>	 ori: So now what? Kill the scap and start over?
[19:00:48] <ori>	 yeah
[19:00:55] <ori>	 but what happened?
[19:01:11] <bd808>	 ori: error: insufficient permission for adding an object to repository database .git/objects
[19:01:42] <grrrit-wm>	 (03PS1) 10Hashar: contint: prevent user-agent ZumBot on Jenkins [operations/puppet] - 10https://gerrit.wikimedia.org/r/114516 
[19:02:16] <bd808>	 ori: All hosts fell back to rsync from tin. So slooooow
[19:02:46] <ori>	 bd808: how do we go from there to fatals on mediawiki.org?
[19:02:55] <bd808>	 So there are l10n updates that are crawling over as json and haven't been turhned into cdb yet
[19:03:04] <Reedy>	 mediawikiwiki is having an identity crisis
[19:03:33] <huh>	 Has anyone ever considered using more specific error messages? ;)
[19:03:49] <greg-g>	 huh: not always safe
[19:04:14] <Reedy>	 If people copy the whole error message, it's as easily looked up
[19:04:18] <huh>	 A full traceback would not be necessary, and could have private info
[19:04:26] <huh>	 but what's wrong with something more specific than "Fatal exception of type MWException" ?
[19:04:27] <bd808>	 Did I somehow mess up the scap-2 rsync too? Meaning did it grab the multiversion control file early?
[19:04:45] <Reedy>	 No
[19:05:08] <Reedy>	 I only changed testwiki over in the /a/common w/c
[19:05:27] <bd808>	 ori: I can't `git update` in tin:/srv/scap
[19:05:40] <Reedy>	 I've no idea where that's come from
[19:05:43] <bd808>	 "error: insufficient permission for adding an object to repository database .git/objects"
[19:05:52] <ori>	 bd808: i updated everywhere
[19:06:00] <ori>	 bd808: i'll look at the permission in a moment
[19:06:15] <bd808>	 That change only really matters on tin
[19:06:19] <bd808>	 Thanks
[19:07:56] <ori>	 i'll bbiab
[19:08:07] <ori>	 something is wrong if not getting fatals is tied to the speed with which files are synchronized
[19:08:23] <Reedy>	 !log For those still following along, scap is still running
[19:08:24] <ori>	 they should be synchronized and the cdbs built before wikiversions is synchronized
[19:08:31] <morebots>	 Logged the message, Master
[19:08:40] <Reedy>	 But I still didn't change mediawikiwiki over
[19:09:04] <ori>	 so why are there fatals?
[19:09:17] <Reedy>	 I've nfi
[19:09:32] <greg-g>	 so, what are the fatals?
[19:09:40] <greg-g>	 Reedy: can you copy one to the bug, plz?
[19:09:42] <Reedy>	 No en localisation cache
[19:09:46] <greg-g>	 k
[19:09:47] <bd808>	 So mw1067 is one of the node throwing the error. grep php-1.23wmf15 wikiversions.dat there returns `testwiki php-1.23wmf15`
[19:10:02] <Reedy>	 lol
[19:10:05] <Reedy>	 awesome
[19:10:12] <Reedy>	 Is scap fixed now?
[19:10:16] <Reedy>	 Can I kill and run it?
[19:10:21] <ori>	 yes
[19:10:26] <bd808>	 yes
[19:10:26] <Reedy>	 Then we can get out of this state we're in
[19:10:28] <logmsgbot>	 !log reedy scap aborted: testwiki to 1.23wmf15 and build l10n cache (duration: 131m 36s)
[19:10:34] <greg-g>	 heh
[19:10:36] <morebots>	 Logged the message, Master
[19:10:40] <logmsgbot>	 !log reedy Started scap: testwiki to 1.23wmf15 and build l10n cache (take 2)
[19:10:48] <morebots>	 Logged the message, Master
[19:11:03] <ori>	 you let this go on for 131 mins?
[19:11:15] <ori>	 and 36 seconds
[19:11:36] <Reedy>	 There wasn't a solution for some time of it
[19:11:57] <ori>	 "flip mw.org and testwiki back to 1.23wmf14"?
[19:12:13] <greg-g>	 mw.org *was* on 14
[19:12:18] <Reedy>	 mw.o still wasn't changed in wikiversions
[19:12:26] <Reedy>	 And testwiki only went when it finally got round to mw1017
[19:12:42] <logmsgbot>	 !log reedy scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '/usr/bin/rsync', '-a', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/usr/local/apache/common-local']' returned non-zero exit status 23 (duration: 02m 01s)
[19:12:50] <morebots>	 Logged the message, Master
[19:12:57] <Reedy>	 !bug
[19:13:43] <Reedy>	 http://p.defau.lt/?OJ3BM45H0jZQqCKgn4OHnA
[19:14:27] <bd808>	 wtf is that
[19:14:30] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Stuff back to their deployed versions
[19:14:39] <morebots>	 Logged the message, Master
[19:14:48] <Reedy>	 That fixes those 2 for nwo
[19:14:51] <ori>	 thanks, that takes care of the immediate problem
[19:16:27] <greg-g>	 Reedy: mind copying one of the backtraces to that bug, just for record keeping
[19:18:06] <greg-g>	 Reedy: ty
[19:18:47] <bd808>	 ori: Does that rsync error message make any sense to you?
[19:19:38] <bd808>	 Spot checking files it names shows them on disk an seeming to have sane permissions
[19:19:58] <Reedy>	 The froms don't seem to exist
[19:20:20] <bd808>	 Sure. They would have been cleaned up when rsync exited
[19:20:47] <Reedy>	 So why is it trying to copy them away?
[19:21:13] <logmsgbot>	 !log reedy Started scap: testwiki to 1.23wmf15 and build l10n cache (take 3)
[19:21:21] <morebots>	 Logged the message, Master
[19:21:23] <bd808>	 The --delay-updates on rsync does that
[19:21:44] <bd808>	 Copies into a tmp dir then renames to place the files
[19:21:55] <Reedy>	 Seems to be working this time
[19:22:36] <Reedy>	 Updating LocalisationCache for 1.23wmf13...
[19:22:45] <Reedy>	 We should just pipe it all to IRC
[19:22:46] <Reedy>	 much easier
[19:23:09] <greg-g>	 )
[19:23:09] <greg-g>	 :)
[19:23:14] <Reedy>	 Though
[19:23:20] <Reedy>	 That's not going to have the desired result
[19:23:30] <Reedy>	 I didn't swap testwiki back again
[19:23:58] <ori>	 no, do it after the scap
[19:24:25] <Reedy>	 But the it won't build the wmf15 l10n cahce?
[19:24:34] <Reedy>	 Updating LocalisationCache for 1.23wmf13... Updated 366 JSON file(s) in '/a/common/php-1.23wmf13/cache/l10n'.
[19:24:35] <Reedy>	 done
[19:24:35] <Reedy>	 19:24:22 INFO - Finished mw-update-l10n (duration: 02m 37s)
[19:24:35] <Reedy>	 19:24:22 DEBUG - Started scap-1 to proxies
[19:24:40] <Reedy>	 Which defeats the point of running scap
[19:24:42] <logmsgbot>	 !log reedy scap aborted: testwiki to 1.23wmf15 and build l10n cache (take 3) (duration: 03m 29s)
[19:24:50] <morebots>	 Logged the message, Master
[19:25:02] <Reedy>	 And also means the summary is wrong
[19:25:12] <logmsgbot>	 !log reedy Started scap: testwiki to 1.23wmf15 and build l10n cache (take 4)
[19:25:20] <morebots>	 Logged the message, Master
[19:27:34] <Reedy>	 stupid apc
[19:29:32] * Reedy  finds things to cross

[19:31:37] <Reedy>	 ori: bd808: Everything is still copying from tin
[19:31:41] <logmsgbot>	 !log reedy scap aborted: testwiki to 1.23wmf15 and build l10n cache (take 4) (duration: 06m 29s)
[19:31:48] <morebots>	 Logged the message, Master
[19:32:00] <bd808>	 Had it gotten to the apache pulls?
[19:32:19] <Reedy>	 Yup
[19:32:20] <Reedy>	 19:30:31 DEBUG - Started update apaches
[19:32:20] <Reedy>	 mw1056: 19:30:33 DEBUG - Copying to mw1056.eqiad.wmnet from tin.eqiad.wmnet
[19:32:20] <Reedy>	 mw1056: 19:30:33 DEBUG - Started rsync common
[19:32:21] <Reedy>	 mw1032: 19:30:33 DEBUG - Copying to mw1032.eqiad.wmnet from tin.eqiad.wmnet
[19:32:23] <Reedy>	 mw1032: 19:30:33 DEBUG - Started rsync common
[19:32:25] <Reedy>	 mw1179: 19:30:33 DEBUG - Copying to mw1179.eqiad.wmnet from tin.eqiad.wmnet
[19:32:27] <Reedy>	 mw1179: 19:30:33 DEBUG - Started rsync common
[19:32:30] <Reedy>	 mw1130: 19:30:33 DEBUG - Copying to mw1130.eqiad.wmnet from tin.eqiad.wmnet
[19:32:31] <Reedy>	 mw1130: 19:30:33 DEBUG - Started rsync common
[19:32:34] <Reedy>	 etc
[19:32:35] <bd808>	 ok. blerg
[19:32:49] <grrrit-wm>	 (03CR) 10Jforrester: [C: 031] "Community aware." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 (owner: 10Jforrester)
[19:32:59] <Reedy>	 And things like
[19:32:59] <Reedy>	 mw109: 19:30:34 DEBUG - Copying to mw109.pmtpa.wmnet from tin.eqiad.wmnet
[19:33:06] <Reedy>	 Don't seem so likely
[19:33:08] <grrrit-wm>	 (03PS2) 10Jforrester: Enable VisualEditor by default on iewiki as well as iawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 
[19:34:12] <sumanah>	 (thx for the fix, all)
[19:35:55] <grrrit-wm>	 (03CR) 10Ottomata: Adding git-fat support for trebuchet deployment (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/112944 (owner: 10Ottomata)
[19:36:39] <Reedy>	 !log Running sync-common on mw1017 for testwiki in the meantime
[19:36:46] <morebots>	 Logged the message, Master
[19:38:19] <grrrit-wm>	 (03PS2) 10Ottomata: Adding git-fat support for trebuchet deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/112944 
[19:41:30] <bd808>	 The host selector works for me locally. Will try debugging on a cluster node
[19:41:35] <greg-g>	 bd808: Reedy ori so, I have to step out for an interview in 20, I assume bd808 and/or ori are currently diagnosing. If you're comfortable, try again, but revert the deploy if it isn't working (as Reedy just did).
[19:41:47] <greg-g>	 aha, assumption 1 is confirmed :)
[19:41:57] <greg-g>	 oh, forgot to add #obviousthings to that message
[19:42:30] <Reedy>	 file has vanished: "/php-git/modules/extensions/wikihiero/refs/remotes/origin/HEAD" (in common)
[19:42:35] <Reedy>	 Isn't this a barrel of laughs
[19:44:33] <huh>	 Reedy: I7aa5f2d454e72733391c590bba198323b71953ac when do you add to rebuild/dump interwikis, when to [[m:IWM]] ?
[19:44:55] <Reedy>	 Randomly mostly
[19:44:59] <Reedy>	 Usually when someone asks
[19:44:59] <huh>	 I submitted https://gerrit.wikimedia.org/r/#/c/112920/ (DNM), but someone called Technical 13 says it is wrong
[19:45:03] <huh>	 Does it matter?
[19:45:14] <Reedy>	 Depends if they're right
[19:45:43] <huh>	 It is exactly equivalent to the one you merged
[19:46:28] <huh>	 or not
[19:46:38] <Reedy>	 Now, where's the wikiversions cdb build script
[19:47:17] <Reedy>	 multiversion/refreshWikiversionsCDB
[19:49:18] <Reedy>	 !log I'm making a note here: HUGE SUCCESS. testwiki is on 1.23wmf15
[19:49:26] <morebots>	 Logged the message, Master
[19:49:45] <Reedy>	 Why do we backup wikiversions?
[19:49:52] <Reedy>	 It's in version control
[19:52:04] <grrrit-wm>	 (03PS1) 10Reedy: Don't backup wikiversions.dat [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114521 
[19:53:06] <yuvipanda>	 Reedy: was expecting 'GREAT SUCCESS', was dissapointed
[19:54:46] <Reedy>	 Why GREAT? That's WRONG
[19:55:05] <bd808>	 Reedy: What's the "test" apache? I'd like to run scap-1 manually from there to see the output
[19:55:10] <Reedy>	 mw1017
[19:55:15] <bd808>	 thanls
[19:55:21] <bd808>	 thanks even
[19:56:05] <ori>	 thanl10s
[19:58:22] <bd808>	 So that worked. It picked the right host (not tin) and finished in 1:16
[19:58:40] <bd808>	 `/usr/local/bin/scap-1 mw1010.eqiad.wmnet mw1070.eqiad.wmnet mw40.pmtpa.wmnet srv270.pmtpa.wmnet mw10.pmtpa.wmnet`
[20:00:04] * greg-g  goes to the interview

[20:04:59] <Reedy>	 Shall we try again?
[20:05:08] <ori>	 yes
[20:05:12] <ori>	 well, wait for bd808 to reply
[20:05:28] <bd808>	 I don't have any objection, but I haven't changed anything
[20:05:36] <paravoid>	 ottomata: have you seen the little "x" next to a reviewer's name?
[20:05:45] <bd808>	 So unless we are voting for sunspots as the cause...
[20:05:48] <paravoid>	 or is it some special access privilege that I have maybe?
[20:05:57] <ottomata>	 yesh i see it
[20:06:02] <ottomata>	 if i remove you I can merge maybe?
[20:06:03] <ottomata>	 i will try
[20:06:04] <ottomata>	 :p
[20:06:05] <paravoid>	 I think so
[20:06:16] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Initial deb packaging [operations/debs/check_ganglia] (debian) - 10https://gerrit.wikimedia.org/r/107723 (owner: 10Ottomata)
[20:06:17] <ottomata>	 haha
[20:06:18] <ottomata>	 that worked
[20:06:21] <paravoid>	 :)
[20:06:22] <ottomata>	 haha
[20:06:29] <ottomata>	 hahaha
[20:06:34] <ottomata>	 feels vindictive
[20:06:43] <ottomata>	 "-2 eh?!  TAKE THAT"
[20:06:46] <ottomata>	 Xed.
[20:06:50] <paravoid>	 :)
[20:07:10] <bd808>	 Reedy: If we ran it now would it be a no-op scap or is there config change that would be pushed?
[20:07:33] <Reedy>	 bd808: Not everything is synced yet
[20:07:38] <Reedy>	 ie json l10n files
[20:08:04] <bd808>	 I'd like to drive on the next attempt to see what happens up close and personal
[20:08:42] <bd808>	 Reedy: Ok with you if I try?
[20:08:49] <grrrit-wm>	 (03CR) 10Ottomata: "Soooo, +1? +2? NE1? :)" [operations/debs/git-fat] (debian) - 10https://gerrit.wikimedia.org/r/113018 (owner: 10Ottomata)
[20:08:53] <Reedy>	 Yeah
[20:08:53] <Reedy>	 scap testwiki to 1.23wmf15 and build l10n cache \(take 5\) | tee /tmp/wmf15-5.log
[20:09:09] <bd808>	 Add a 2>&1 in there somewhere
[20:09:14] <bd808>	 before the tee
[20:09:21] <Reedy>	 You're the one running it this time :p
[20:09:28] <bd808>	 ok :)
[20:09:35] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "Yeah +1 as far as I am concerned :-] You would need a Debian guru to proof check the package though!" [operations/debs/git-fat] (debian) - 10https://gerrit.wikimedia.org/r/113018 (owner: 10Ottomata)
[20:10:13] <logmsgbot>	 !log bd808 Started scap: testwiki to 1.23wmf15 and build l10n cache (take 5)
[20:10:21] <morebots>	 Logged the message, Master
[20:10:35] <bd808>	 Follow along at /tmp/wmf15-5.log if you'd like
[20:10:55] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|I5cb2f67ea}}: Add symlinks
[20:10:59] <grrrit-wm>	 (03PS1) 10Reedy: Fixup populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114528 
[20:11:04] <morebots>	 Logged the message, Master
[20:11:10] <grrrit-wm>	 (03CR) 10Ottomata: "Matanya, just curious, why?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114136 (owner: 10Matanya)
[20:12:58] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] testing for Jenkins / debian-glue [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/113114 (owner: 10Hashar)
[20:13:02] <grrrit-wm>	 (03Abandoned) 10Ori.livneh: bits: gzip image/svg+xml & image/x-icon [operations/puppet] - 10https://gerrit.wikimedia.org/r/114448 (owner: 10Ori.livneh)
[20:13:10] <bd808>	 Copying to srv261.pmtpa.wmnet from mw10.pmtpa.wmne
[20:13:14] <bd808>	 It's working
[20:13:52] <Reedy>	 fsck computers
[20:13:59] <hashar>	 ottomata: the change I made for operations/debs/vips need to be squashed I think :)
[20:14:46] <grrrit-wm>	 (03CR) 10Ottomata: "Should we get another +1 on this? This change should be a no-op." [operations/puppet] - 10https://gerrit.wikimedia.org/r/110454 (owner: 10Matanya)
[20:14:59] <ottomata>	 oh hashar, ok
[20:15:00] <ottomata>	 i mean
[20:15:05] <ottomata>	 there is a change you made before it that isn't merged
[20:15:07] <hashar>	 will do right now
[20:15:12] <ottomata>	 we could just merge them both
[20:15:37] <grrrit-wm>	 (03PS5) 10Hashar: git buildpackage basic configuration [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/113098 
[20:15:52] <grrrit-wm>	 (03Abandoned) 10Hashar: testing for Jenkins / debian-glue [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/113114 (owner: 10Hashar)
[20:15:59] <bd808>	 Each rsync is taking about 2m which isn't horrible. It's not great but it's not horrible
[20:16:23] <hashar>	 ottomata: https://gerrit.wikimedia.org/r/#/c/113098/ might work.  Gotta wait for Jenkins to complete the debian-glue job
[20:16:44] <hashar>	 it is building athttps://integration.wikimedia.org/ci/job/operations-debs-vips-debian-glue/5/console :-D
[20:26:19] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "There is still a bunch of lintian failures: https://integration.wikimedia.org/ci/job/operations-debs-vips-debian-glue/5/testReport/" [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/113098 (owner: 10Hashar)
[20:28:29] <Reedy>	 bd808: has it finished all the rsyncing yet?
[20:28:50] <bd808>	 Nope. Still chugging along
[20:29:12] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Fixup populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114528 (owner: 10Reedy)
[20:29:18] <grrrit-wm>	 (03Merged) 10jenkins-bot: Fixup populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114528 (owner: 10Reedy)
[20:30:26] <bd808>	 40 concurrent, 2-3 minutes per rsync, 400 total… takes a while
[20:32:49] <grrrit-wm>	 (03PS2) 10Reedy: WIP: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[20:32:52] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] WIP: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 (owner: 10Reedy)
[20:33:17] <Reedy>	 STFU jenkins
[20:41:07] <bd808>	 Reedy: 20:40:53 INFO - Finished scap-1 to apaches (duration: 27m 55s)
[20:41:20] <bd808>	 next up is l10n rebuild
[20:41:58] <Reedy>	 yay
[20:43:18] <bd808>	 Programming sucks. Two characters I left in a file wasted hours and hours of time.
[20:44:00] <bd808>	 The next time I put in two characters that save hours and hours of time it's unlikely to be noticed
[20:49:39] <hoo>	 bd808: 99 little bugs in the code
[20:49:40] <hoo>	 99 little bugs in the code
[20:49:40] <hoo>	 Take one down, patch it around
[20:49:40] <hoo>	 117 little bugs in the code ;)
[20:49:56] <bd808>	 hoo: :)
[20:50:07] <bd808>	 That's been my career at WMF so far
[20:52:08] <grrrit-wm>	 (03PS5) 10coren: Fix inconsistencies in postint.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 
[20:53:02] <grrrit-wm>	 (03CR) 10coren: [C: 032] "A +1 from Ryan is like unto a high-five from God." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114499 (owner: 10coren)
[20:56:56] <grrrit-wm>	 (03PS3) 10Reedy: WIP: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:03:14] <greg-g>	 [5~
[21:03:54] <greg-g>	 (oops)
[21:05:39] <greg-g>	 gah!
[21:05:55] <greg-g>	 netsplit
[21:06:11] <Eloquence>	 o rly
[21:06:19] <grrrit-wm>	 (03PS1) 10Hashar: contint: webproxy for maven on CI production slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/114597 
[21:07:13] <greg-g>	 I have no bd808 :/
[21:08:16] <greg-g>	 bd808: welcome back
[21:08:31] <bd808>	 I was here the whole time. You got lost :)
[21:08:33] <greg-g>	 though we've lost Reedy
[21:08:39] <logmsgbot>	 !log bd808 Finished scap: testwiki to 1.23wmf15 and build l10n cache (take 5) (duration: 58m 26s)
[21:08:46] <bd808>	 \o/
[21:08:47] <greg-g>	 oo, look at that, perfect timing
[21:08:47] <morebots>	 Logged the message, Master
[21:09:06] <bd808>	 Blerg. "Finished scap-rebuild-cdbs (duration: 27m 31s)"
[21:09:32] <bd808>	 Half of the time of the scap was l10n.json to l10n.cdb
[21:09:35] <RoanKattouw>	 Holy crap, a 58-minute scap?
[21:10:17] <bd808>	 RoanKattouw: 3 version l10n update
[21:10:30] <RoanKattouw>	 Why 3 versions?
[21:10:38] <Nemo_bis>	 and how much of that time was downtime
[21:10:48] <bd808>	 Only test has moved up to the new branch
[21:10:51] <Reedy>	 netsplit
[21:11:09] <greg-g>	 RoanKattouw: test on 15, group2 on 13, group1 on 14
[21:11:10] <bd808>	 That 58 minutes was all "good" work
[21:11:11] <RoanKattouw>	 Right, so you guys do the test++ change before the Wikipedias++ change?
[21:11:27] <bd808>	 RoanKattouw: Yes
[21:11:29] <greg-g>	 RoanKattouw: yeah, and test.wiki is before test2/mw/etc
[21:11:33] <RoanKattouw>	 Right
[21:12:06] <bd808>	 If we re-run right now with no l10n update I'd put my money on a 5m runtime
[21:12:22] <RoanKattouw>	 Nemo_bis: People who actually designed this system can answer better, but I would assume that there was at the very most 58/3 minutes of downtime per individual wiki
[21:12:32] <RoanKattouw>	 Which is still 19 minutes, which would be bad
[21:12:40] <greg-g>	 bd808: well, we will in a second, right, to get test2/mw.org/testwikidata?
[21:13:03] <RoanKattouw>	 That doesn't require rerunning scap, does it?
[21:13:05] <bd808>	 I'll let Reedy drive for that one
[21:13:24] <Nemo_bis>	 RoanKattouw: I surely was unable to use mediawiki.org for longer than that
[21:13:30] <bd808>	 RoanKattouw: I think it's just the multiversion sync
[21:13:43] <RoanKattouw>	 Yeah exactly
[21:13:53] <Reedy>	 Where are we upto?
[21:13:57] <Reedy>	 I netsplitted
[21:14:08] <bd808>	 Reedy: scap done
[21:14:16] <Reedy>	 yay
[21:14:21] <Reedy>	 should be pretty quick now
[21:14:22] <bd808>	 <logmsgbot>	 !log bd808 Finished scap: testwiki to 1.23wmf15 and build l10n cache (take 5) (duration: 58m 26s)
[21:14:34] <bd808>	 It finished just as the netsplit ended
[21:14:35] <greg-g>	 right, no scap needed, but we could do one :)
[21:14:37] <grrrit-wm>	 (03PS4) 10Reedy: WIP: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:14:44] <Reedy>	 ori: ^ Nearly done!
[21:15:31] <grrrit-wm>	 (03PS2) 10Reedy: Wikipedias to 1.23wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114494 
[21:15:34] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Wikipedias to 1.23wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114494 (owner: 10Reedy)
[21:15:43] <grrrit-wm>	 (03Merged) 10jenkins-bot: Wikipedias to 1.23wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114494 (owner: 10Reedy)
[21:16:13] <grrrit-wm>	 (03PS1) 10Hashar: contint: slaves now have openjdk-{6,7}-jdk [operations/puppet] - 10https://gerrit.wikimedia.org/r/114619 
[21:16:40] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf14
[21:16:48] <morebots>	 Logged the message, Master
[21:18:08] <Reedy>	 All is quiet
[21:18:12] <grrrit-wm>	 (03PS2) 10Reedy: group0 wikis to 1.23wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114495 
[21:18:15] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] group0 wikis to 1.23wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114495 (owner: 10Reedy)
[21:18:23] <grrrit-wm>	 (03Merged) 10jenkins-bot: group0 wikis to 1.23wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114495 (owner: 10Reedy)
[21:19:49] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "I would like to avoid using terms like "WMF Labs LDAP (use wiki login name not shell)" because the "access denied" page ("This server coul" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[21:19:59] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf15
[21:20:07] <morebots>	 Logged the message, Master
[21:20:48] <Reedy>	 apc spam as usual
[21:21:01] <grrrit-wm>	 (03PS2) 10Reedy: Don't backup wikiversions.dat [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114521 
[21:21:06] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Don't backup wikiversions.dat [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114521 (owner: 10Reedy)
[21:21:12] <grrrit-wm>	 (03Merged) 10jenkins-bot: Don't backup wikiversions.dat [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114521 (owner: 10Reedy)
[21:23:33] <greg-g>	 Alchimista: I think your client is having issues, mind killing the autojoin and fixing your connection?
[21:28:32] <grrrit-wm>	 (03PS5) 10Reedy: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:28:33] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 (owner: 10Reedy)
[21:32:00] <Reedy>	 I think that's it! ;)
[21:32:47] <grrrit-wm>	 (03PS6) 10Reedy: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:32:47] <grrrit-wm>	 (03PS1) 10Reedy: Delete wikiversions dat files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114621 
[21:37:35] <grrrit-wm>	 (03PS1) 10Reedy: Reuse readDbListFile in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114625 
[21:39:52] <grrrit-wm>	 (03PS2) 10Reedy: Reuse readDbListFile in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114625 
[21:40:33] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Reuse readDbListFile in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114625 (owner: 10Reedy)
[21:40:40] <grrrit-wm>	 (03Merged) 10jenkins-bot: Reuse readDbListFile in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114625 (owner: 10Reedy)
[21:48:50] <grrrit-wm>	 (03PS7) 10Reedy: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:50:46] <grrrit-wm>	 (03PS1) 10Reedy: Fixup versions in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114630 
[21:55:03] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Fixup versions in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114630 (owner: 10Reedy)
[21:55:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Fixup versions in populateWikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114630 (owner: 10Reedy)
[21:55:27] <grrrit-wm>	 (03PS8) 10Reedy: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:57:37] <grrrit-wm>	 (03PS9) 10Reedy: Change wikiversions to use json [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 
[21:57:45] <Reedy>	 That's the loose ends tied up
[22:00:03] <bd808__>	 Reedy: did I miss anything fun on the other side of the netsplit?
[22:00:28] <Reedy>	 Nope
[22:00:45] <Reedy>	 I finished the wikiversions dat -> json stuff though
[22:01:04] <bd808__>	 Cool. I'll peek at it.
[22:04:17] <grrrit-wm>	 (03CR) 10Odder: "Where was this discussed?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114088 (owner: 10EBernhardson)
[22:04:27] <odder>	 Nemo_bis: ^^
[22:15:13] <grrrit-wm>	 (03CR) 10Nemo bis: "-me" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114088 (owner: 10EBernhardson)
[22:15:54] * Nemo_bis  hammers odder 

[22:16:00] <Nemo_bis>	 wwhere is my X button
[22:16:15] <odder>	 Wut
[22:17:11] <Nemo_bis>	 oic, disappears on merged changes
[22:20:11] <Reedy>	 !log Created EducationProgram tables on nlwiki
[22:20:17] <grrrit-wm>	 (03PS3) 10Siebrand: Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 
[22:20:20] <morebots>	 Logged the message, Master
[22:20:25] <grrrit-wm>	 (03PS4) 10Reedy: Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 (owner: 10Siebrand)
[22:20:30] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 (owner: 10Siebrand)
[22:20:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 (owner: 10Siebrand)
[22:20:39] <Reedy>	 Rebase and commit summary update in one change would be nice
[22:21:15] <grrrit-wm>	 (03PS5) 10Whym: Enable autopatrol and patrolling of RecentChanges on Japanese Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113338 
[22:21:20] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable autopatrol and patrolling of RecentChanges on Japanese Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113338 (owner: 10Whym)
[22:21:38] <Nemo_bis>	 I still don't get why it doesn't rebase on its own when +2, if it already knows it merges
[22:21:51] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable autopatrol and patrolling of RecentChanges on Japanese Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113338 (owner: 10Whym)
[22:22:05] <Reedy>	 There might be a setting about rebasing
[22:22:15] <Reedy>	 I wish it would though. Save some of the merge commitss
[22:22:37] <grrrit-wm>	 (03PS2) 10Jean-Frédéric: Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113500 
[22:22:42] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113500 (owner: 10Jean-Frédéric)
[22:22:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113500 (owner: 10Jean-Frédéric)
[22:23:11] <grrrit-wm>	 (03PS2) 10Odder: Create an ArbCom group on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113542 
[22:23:15] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Create an ArbCom group on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113542 (owner: 10Odder)
[22:23:22] <odder>	 oh, that would make the French happy
[22:23:25] <grrrit-wm>	 (03Merged) 10jenkins-bot: Create an ArbCom group on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113542 (owner: 10Odder)
[22:24:05] <Reedy>	 odder: https://gerrit.wikimedia.org/r/#/c/113334/
[22:24:47] <grrrit-wm>	 (03PS3) 10Brion VIBBER: Enable lower-res .ogv transcodes for fallback player on slow machines [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114507 
[22:24:55] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable lower-res .ogv transcodes for fallback player on slow machines [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114507 (owner: 10Brion VIBBER)
[22:25:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable lower-res .ogv transcodes for fallback player on slow machines [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114507 (owner: 10Brion VIBBER)
[22:25:17] <grrrit-wm>	 (03PS3) 10Jforrester: Enable VisualEditor by default on iewiki as well as iawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 
[22:25:21] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable VisualEditor by default on iewiki as well as iawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 (owner: 10Jforrester)
[22:25:29] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable VisualEditor by default on iewiki as well as iawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 (owner: 10Jforrester)
[22:25:48] <grrrit-wm>	 (03CR) 10Reedy: [C: 04-1] Remove C: namespace alias (for categories) from hiwiki config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113656 (owner: 10Gerrit Patch Uploader)
[22:26:30] <grrrit-wm>	 (03CR) 10Reedy: "https://wikitech.wikimedia.org/wiki/Close_a_wiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113878 (owner: 10Jeremyb)
[22:26:50] <grrrit-wm>	 (03PS3) 10KartikMistry: Remove $wgULSNoWebfontsSelectors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114131 
[22:26:57] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Remove $wgULSNoWebfontsSelectors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114131 (owner: 10KartikMistry)
[22:27:06] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove $wgULSNoWebfontsSelectors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114131 (owner: 10KartikMistry)
[22:27:43] <odder>	 Reedy: I know; I don't care
[22:28:01] <grrrit-wm>	 (03PS5) 10Jforrester: Factor out Parsoid config from VisualEditor config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114100 
[22:28:12] <odder>	 they managed to piss me off by not anwering my questions for weeks, whatever they do with the favicon is whatever
[22:28:16] <grrrit-wm>	 (03PS3) 10Tim Landscheidt: Revert "Add Apple Touch icon for Labs" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113334 
[22:28:21] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Revert "Add Apple Touch icon for Labs" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113334 (owner: 10Tim Landscheidt)
[22:28:25] <Reedy>	 :(
[22:28:28] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Add Apple Touch icon for Labs" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113334 (owner: 10Tim Landscheidt)
[22:28:52] <greg-g>	 Reedy: so, after a couple hours to think: are we comfortable with non-Platform people running scap now?
[22:30:03] <logmsgbot>	 !log reedy synchronized database lists files:
[22:30:10] <morebots>	 Logged the message, Master
[22:30:37] <logmsgbot>	 !log reedy synchronized wmf-config/
[22:30:45] <morebots>	 Logged the message, Master
[22:33:20] <grrrit-wm>	 (03CR) 10BryanDavis: "A couple of questions inline." (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 (owner: 10Reedy)
[22:34:44] <grrrit-wm>	 (03PS1) 10MaxSem: Don't collapse sections on Wiktionaries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114634 
[22:35:00] <grrrit-wm>	 (03CR) 10GWicke: [C: 031] Factor out Parsoid config from VisualEditor config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114100 (owner: 10Jforrester)
[22:39:43] <grrrit-wm>	 (03CR) 10Reedy: Change wikiversions to use json (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114108 (owner: 10Reedy)
[22:40:43] <logmsgbot>	 !log csteipp synchronized php-1.23wmf15/includes  'bugs 60771, 61346, 61362'
[22:40:50] <morebots>	 Logged the message, Master
[22:46:00] <logmsgbot>	 !log csteipp synchronized php-1.23wmf14/includes  'bugs 60771, 61346, 61362'
[22:46:17] <morebots>	 Logged the message, Master
[22:49:46] <odder>	 greg-g: I'd like to link to anything that announces the re-enabling of ULS for Tech News
[22:49:59] <odder>	 is there any patch or mail about it anywhere?
[22:50:18] <odder>	 I'm looking at https://wikitech.wikimedia.org/wiki/Deployments#Week_of_February_17th
[22:50:49] <Reedy>	 greg-g: In theory, no one should be doing as "complex" as my thursday deploys... If the fanout issue is fixed, I suspect all should be ok. I suspect for most of them, someone from core will likely be around
[22:51:17] <greg-g>	 Reedy: right, but Monday?
[22:51:27] <greg-g>	 odder: one sec
[22:51:28] <Reedy>	 What about monday?
[22:51:30] <ori>	 it should be OK, imo
[22:52:36] <greg-g>	 thanks ori
[22:52:56] <greg-g>	 Reedy: just if we think we're all alright with not platform people running scap as is
[22:53:38] <bd808>	 greg-g: Are there still issues?
[22:53:57] <bd808>	 The fanout is fixed. I have logs to prove it
[22:54:01] <logmsgbot>	 !log ori synchronized php-1.23wmf15/extensions/Echo  'Update Schema:Echo revision to r7572295 (bug 61698)'
[22:54:09] <morebots>	 Logged the message, Master
[22:54:13] <odder>	 greg-g: I think I might have missed a Labs outage that happened on... Sunday, was it?
[22:54:30] * odder  can't keep up with the outages these days

[22:55:14] <logmsgbot>	 !log ori synchronized php-1.23wmf14/extensions/Echo  'Update Schema:Echo revision to r7572295 (bug 61698)'
[22:55:22] <morebots>	 Logged the message, Master
[22:58:17] <grrrit-wm>	 (03CR) 10Ryan Lane: "Nothing should ever say OIT LDAP. We don't trust the office for anything in production." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[23:00:40] <greg-g>	 odder: is the notice at the top of https://www.mediawiki.org/wiki/UniversalLanguageSelector useful?
[23:00:47] <greg-g>	 (the pages linked from, specifically)
[23:01:47] <odder>	 it appears to be incorrect
[23:02:06] <Reedy>	 See also: Bug 1
[23:02:08] <grrrit-wm>	 (03CR) 10Greg Grossmeier: "Ryan: touche. Then the problem is smaller than I think (good)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[23:02:27] <odder>	 I should be seeing ULS on Commons if it's supposed to go out with wmf14
[23:02:47] <Reedy>	 WFM
[23:03:01] <AaronSchulz>	 paravoid: I wonder why there are a good chunk of container HEAD 503s each day...that info is heavily cached so queries should not be frequent at all
[23:03:27] <Reedy>	 Though, on the CyanogenMod wiki its pretty screwed up
[23:03:29] <Reedy>	 http://wiki.cyanogenmod.org/w/Main_Page
[23:03:36] <Reedy>	 chinese and czech
[23:03:37] <AaronSchulz>	 I wonder if it's a sqlite3 concurrency limitation...which can also cause 401 on PUT
[23:03:52] <legoktm>	 ori: did you also update the mediawiki-config repo for echo?
[23:03:54] <AaronSchulz>	 (object PUT)
[23:04:17] <legoktm>	 ori: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php#L2308
[23:04:31] <odder>	 I guess it helps not to hide #pt-uls in one's CSS
[23:04:38] * odder  bangs his head on the wall

[23:12:06] <grrrit-wm>	 (03PS3) 10BryanDavis: Send Vary header on http to http redirect [operations/puppet] - 10https://gerrit.wikimedia.org/r/111917 
[23:12:30] <grrrit-wm>	 (03PS1) 10Legoktm: Update Schema:Echo to revision 7572295 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114650 
[23:12:43] <legoktm>	 ori: ^
[23:15:42] <greg-g>	 Ryan_Lane2: is there one Labs LDAP group that is a surrogate for access to eg graphite?
[23:18:46] <bd808>	 greg-g: cn=wmf,ou=groups,dc=wikimedia,dc=org
[23:19:38] <greg-g>	 huh, so we are limited to only wmf'ers?
[23:20:06] <greg-g>	 yak, I don't want to shave you
[23:20:09] <bd808>	 yeah
[23:20:21] <bd808>	 We talked about it in the Core quarterly review
[23:20:36] * greg-g  kinda remembers that

[23:20:42] <greg-g>	 was it vis a vis aude ?
[23:20:53] <greg-g>	 audo/wikidata folks
[23:20:58] <bd808>	 Sumana asked aobut wmf-de access, Ken said it was in legal's hands
[23:20:59] <greg-g>	 audo? heh
[23:21:03] <greg-g>	 yeah
[23:21:15] * greg-g  waves to aude, nothing to worry about in here

[23:21:27] <bd808>	 It may be laying on the floor now :(
[23:21:31] <greg-g>	 yep :(
[23:21:36] * greg-g  peers into RT

[23:22:07] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Update Schema:Echo to revision 7572295 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114650 (owner: 10Legoktm)
[23:22:22] <logmsgbot>	 !log ori updated /a/common to {{Gerrit|I62011ada8}}: Update Schema:Echo to revision 7572295
[23:22:30] <morebots>	 Logged the message, Master
[23:22:59] <logmsgbot>	 !log ori synchronized wmf-config/CommonSettings.php  'I62011ada8: Update Schema:Echo to revision 7572295'
[23:23:07] <morebots>	 Logged the message, Master
[23:24:58] <greg-g>	 bd808: https://rt.wikimedia.org/Ticket/Display.html?id=6293
[23:25:03] <greg-g>	 not sureif you can see it, but yeah
[23:25:15] <legoktm>	 ori: thanks!
[23:25:52] <AaronSchulz>	 gi11es: << grep -r 'transcode' runJobs.log | grep 'error=' >> shows a fair amout of "Source not found" failures for TMH...not sure who should look at that
[23:26:21] <logmsgbot>	 !log reedy Started scap: no op scap is no op?
[23:26:29] <morebots>	 Logged the message, Master
[23:28:21] <greg-g>	 bd808: 'wmde' group created via https://rt.wikimedia.org/Ticket/Display.html?id=5023, requst for that group to have graphite/etc access in https://rt.wikimedia.org/Ticket/Display.html?id=6293, which was attempted by then "wait, we'll have a better solution soon" in https://gerrit.wikimedia.org/r/#/c/95333/
[23:28:27] <greg-g>	 so, effing yak
[23:28:42] <MaxSem>	 !log No it's not, you're adding to global warming
[23:28:50] <morebots>	 Logged the message, Master
[23:29:44] <bd808>	 greg-g: Yeah. Looks like this cause (getting anyone under NDA access to logs/stats) needs a new internal champion
[23:29:49] * bd808  not it

[23:29:55] <grrrit-wm>	 (03PS4) 10Hashar: Send Vary header on http to http redirect [operations/puppet] - 10https://gerrit.wikimedia.org/r/111917 (owner: 10BryanDavis)
[23:30:40] <grrrit-wm>	 (03CR) 10Hashar: "Followed up on my comment from patchset1:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/111917 (owner: 10BryanDavis)
[23:31:29] <grrrit-wm>	 (03CR) 10Greg Grossmeier: "Tim: Not all WMF Employees have access though, but I understand your frustration/point with this. It's WMF employees who have Wikitech acc" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114503 (owner: 10Greg Grossmeier)
[23:31:44] <bd808>	 hashar: \o/ Now you just have to find someone willing to merge and babysit
[23:32:15] <hashar>	 bd808: yeah and need to get that merged when I am not half asleep :-]
[23:32:39] <hashar>	 bd808: will do with greek ops during monday europe morning :-D
[23:32:40] <bd808>	 hashar: RobH was interested at some point but then was out of touch for a while
[23:32:52] <hashar>	 yeah and we worked on that during the mw summit
[23:33:02] <hashar>	 so I was not carefully reviewing / verifying :(
[23:33:22] <hashar>	 then Krinkle complained (for good reasons) :-D
[23:33:24] <RobH>	 ?
[23:33:40] <hashar>	 RobH: some weird issue with http/https on integration.wikimedia.org and doc.wikimedia.org
[23:33:49] <hashar>	 something we deployed during mw summit a month or so ago
[23:33:50] <RobH>	 looking at ps now
[23:33:51] <bd808>	 RobH: https://gerrit.wikimedia.org/r/111917
[23:34:00] <hashar>	 oh men
[23:34:07] <hashar>	 I sense I am not going to go to sleep anytime soon :-]
[23:34:21] <RobH>	 i rather merge at start of the day rather than end of the day
[23:34:27] <RobH>	 mind if it sits and merges tomorrow my AM?
[23:34:40] <hashar>	 will do on monday
[23:34:43] <hashar>	 I am not there tomorrow :-(
[23:34:49] <RobH>	 sounds good to me
[23:34:49] <hashar>	 but yeah better be careful
[23:34:51] <grrrit-wm>	 (03CR) 10Krinkle: [C: 031] "I approve this message." [operations/puppet] - 10https://gerrit.wikimedia.org/r/111917 (owner: 10BryanDavis)
[23:34:56] <RobH>	 lets plan to push on monday then
[23:35:02] <RobH>	 when you and i are both abouts
[23:35:19] <bd808>	 \o/
[23:35:26] <grrrit-wm>	 (03PS1) 10Hoo man: Simplify the AbuseFilter configuration a little [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114656 
[23:36:01] * Reedy  eyes hoo

[23:36:15] <hashar>	 RobH: will do with Greek ops during european morning :-]
[23:36:26] <hashar>	 RobH: Iam busy all the afternoon
[23:36:40] <RobH>	 good enough, then when they go to bed im aware of it if it gets wonky
[23:37:01] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "It is the end of the day in SF and I am not there this friday to babysit this change. Friday deploy are evils, and the sites are not THA" [operations/puppet] - 10https://gerrit.wikimedia.org/r/111917 (owner: 10BryanDavis)
[23:37:12] <hashar>	 RobH: should be fine :]
[23:37:22] <hoo>	 :P
[23:37:39] <hashar>	 I should write an Icinga script to monitor the http / https redirections for integration.wikimedia.org
[23:42:02] <Krinkle>	 !log Reloading zuul to deploy I15c43ec458b053a9
[23:42:10] <morebots>	 Logged the message, Master
[23:43:32] <grrrit-wm>	 (03CR) 10Nemo bis: "I think there's no harm in leaving the "overrides" which equal the default, when a bug report is mentioned. Makes it easier to change defa" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114656 (owner: 10Hoo man)
[23:44:21] <grrrit-wm>	 (03Abandoned) 10Hoo man: Simplify the AbuseFilter configuration a little [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114656 (owner: 10Hoo man)
[23:44:51] <Nemo_bis>	 aww
[23:47:02] <hoo>	 so far for the plans to actually remove that file... I think there's even a bug about that :P