[00:09:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[00:09:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[00:09:57] <greg-g>	 whew, all better ;)
[00:12:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103317)  
[00:12:31] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103314)  
[00:12:35] <grrrit-wm>	 (03PS1) 10Ryan Lane: Make appserver common a mediawiki deploy target [operations/puppet] - 10https://gerrit.wikimedia.org/r/94832 
[00:28:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[00:28:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[00:32:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103838)  
[00:32:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103831)  
[00:44:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[00:44:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[01:05:50] <icinga-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours  
[01:10:49] <grrrit-wm>	 (03PS4) 10Ryan Lane: Add recursive submodule support to trebuchet [operations/puppet] - 10https://gerrit.wikimedia.org/r/94688 
[01:12:22] <grrrit-wm>	 (03PS4) 10Ryan Lane: Add mediawiki module for fetch and checkout hooks [operations/puppet] - 10https://gerrit.wikimedia.org/r/94682 
[01:12:41] <grrrit-wm>	 (03PS5) 10Ryan Lane: Add recursive submodule support to trebuchet [operations/puppet] - 10https://gerrit.wikimedia.org/r/94688 
[01:27:29] <springle>	 !log restart db1050 mariadb after outage, let repl catch up. new lvm snaps mount ok. leave out of pool for now
[01:27:49] <morebots>	 Logged the message, Master
[01:34:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103110)  
[01:34:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (103081)  
[01:37:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[01:37:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[01:40:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (100854)  
[01:40:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (100846)  
[01:42:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[01:42:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[01:46:50] <icinga-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:49:36] <grrrit-wm>	 (03PS1) 10Springle: track client/user/table/index stats for audit. disable excess warnings for mariadb until we switch to RBR. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94841 
[01:50:49] <grrrit-wm>	 (03CR) 10Springle: [C: 032] track client/user/table/index stats for audit. disable excess warnings for mariadb until we switch to RBR. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94841 (owner: 10Springle)
[01:58:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (109860)  
[01:58:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (110003)  
[02:05:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[02:05:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[02:15:04] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf3) at Tue Nov 12 02:15:03 UTC 2013
[02:15:22] <morebots>	 Logged the message, Master
[02:16:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104802)  
[02:16:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104789)  
[02:20:54] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf2) at Tue Nov 12 02:20:54 UTC 2013
[02:21:10] <morebots>	 Logged the message, Master
[02:25:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[02:25:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[02:33:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (100432)  
[02:33:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (100426)  
[02:34:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[02:34:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[02:39:30] <icinga-wm>	 PROBLEM - MySQL Processlist on db1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[02:41:20] <icinga-wm>	 RECOVERY - MySQL Processlist on db1002 is OK: OK 1 unauthenticated, 0 locked, 4 copy to table, 9 statistics  
[02:42:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104180)  
[02:42:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104103)  
[02:49:20] <icinga-wm>	 PROBLEM - MySQL Processlist on db1002 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 7 copy to table, 222 statistics  
[02:50:20] <icinga-wm>	 RECOVERY - MySQL Processlist on db1002 is OK: OK 1 unauthenticated, 0 locked, 4 copy to table, 7 statistics  
[02:52:10] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[02:55:00] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1002 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[03:00:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:00:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[03:05:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (108151)  
[03:06:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104566)  
[03:09:30] <icinga-wm>	 PROBLEM - MySQL Processlist on db1002 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 5 copy to table, 312 statistics  
[03:11:30] <icinga-wm>	 RECOVERY - MySQL Processlist on db1002 is OK: OK 0 unauthenticated, 0 locked, 5 copy to table, 1 statistics  
[03:11:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:11:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[03:12:05] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 12 03:12:05 UTC 2013
[03:12:23] <morebots>	 Logged the message, Master
[03:14:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (102136)  
[03:14:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (102073)  
[03:21:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:21:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[03:24:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (101860)  
[03:24:40] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:24:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (101814)  
[03:26:30] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.001 second response time  
[03:29:30] <icinga-wm>	 PROBLEM - MySQL Processlist on db1002 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 6 copy to table, 152 statistics  
[03:32:30] <icinga-wm>	 RECOVERY - MySQL Processlist on db1002 is OK: OK 0 unauthenticated, 0 locked, 4 copy to table, 2 statistics  
[03:32:36] <grrrit-wm>	 (03PS1) 10Springle: aim for at least 3 equivalent slaves on shards not using groupLoadsByDB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94843 
[03:33:11] <grrrit-wm>	 (03CR) 10Springle: [C: 032] aim for at least 3 equivalent slaves on shards not using groupLoadsByDB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94843 (owner: 10Springle)
[03:34:27] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'slave balancing'
[03:34:44] <morebots>	 Logged the message, Master
[03:37:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:37:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[03:38:40] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:39:30] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.939 second response time  
[03:49:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104886)  
[03:49:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (104649)  
[03:52:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:52:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[03:55:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (102874)  
[03:55:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (102817)  
[03:57:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[03:57:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[04:05:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (106488)  
[04:05:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (106405)  
[04:14:30] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[04:14:40] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[04:17:30] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (101970)  
[04:17:40] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (102936)  
[05:04:10] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[05:04:10] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[05:06:10] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1002 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[05:06:10] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1002 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[06:09:43] <grrrit-wm>	 (03PS1) 10Tim Starling: Disable client idle disconnection [operations/puppet] - 10https://gerrit.wikimedia.org/r/94848 
[06:29:20] <icinga-wm>	 RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 100,000  
[06:32:20] <icinga-wm>	 PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[06:58:08] <grrrit-wm>	 (03PS1) 10Springle: move recache jobs to snapshot host in future [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94851 
[06:59:45] <grrrit-wm>	 (03CR) 10Springle: [C: 032] move recache jobs to snapshot host in future [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94851 (owner: 10Springle)
[07:02:01] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'recache jobs on S2 to db1018'
[07:02:21] <morebots>	 Logged the message, Master
[07:23:11] <grrrit-wm>	 (03CR) 10Ori.livneh: "Useful thread from the redis mailing list:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94848 (owner: 10Tim Starling)
[07:51:58] <grrrit-wm>	 (03PS1) 10Raimond Spekking: Temporary lift of IP cap for WikiCon 2013 in de/en.WP, Commons, de/en.wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 
[07:59:50] <icinga-wm>	 PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:38:56] <grrrit-wm>	 (03CR) 10Tim Starling: "On rdb1003, netstat shows only 312 connections at present, 82 in TIME_WAIT, and tcpdump shows about 125 connections per second, so we are " [operations/puppet] - 10https://gerrit.wikimedia.org/r/94848 (owner: 10Tim Starling)
[08:48:20] <icinga-wm>	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 100,000  
[08:51:21] <icinga-wm>	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[08:55:39] <grrrit-wm>	 (03PS1) 10ArielGlenn: monitoring for text-varnish wikipedia in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/94868 
[08:57:18] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] monitoring for text-varnish wikipedia in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/94868 (owner: 10ArielGlenn)
[09:14:40] <icinga-wm>	 RECOVERY - Puppet freshness on neon is OK: puppet ran at Tue Nov 12 09:14:34 UTC 2013  
[09:22:37] <icinga-wm>	 PROBLEM - Host sq48 is DOWN: PING CRITICAL - Packet loss = 100%  
[09:23:09] <paravoid>	 morning
[09:23:18] <paravoid>	 springle: still here?
[09:24:25] <springle>	 paravoid: yep
[09:24:32] <paravoid>	 hey
[09:25:04] <paravoid>	 so, on sunday, even after your pt-kill jobs, there was still some load spikes on databases
[09:25:23] <paravoid>	 90% CPU and such, they're apparent in ganglia
[09:25:31] <paravoid>	 I only saw them after the fact
[09:25:47] <paravoid>	 there were tons of "too many connection" errors in dberror, look at around 14:00 UTC iirc
[09:26:04] <springle>	 yes the 30s kill limit was too long
[09:26:18] <paravoid>	 my very cursory ishmael digging only showed the logpager query as an outlier, with 1.1% of queries / 72% of time (wtf...)
[09:28:23] <springle>	 actually, to be accurate, 30s was too long and the pt-kill interval at 5s was too long to catch the surge of SpecialAllpages::showToplevel
[09:28:43] <springle>	 twofold issue. they're now shorter
[09:29:43] <paravoid>	 ok
[09:29:49] <paravoid>	 as long as you're aware of it :)
[09:30:18] <springle>	 paravoid: did you see this http://aerosuidae.net/paste/22/52807966 (was in an email in Problem SQl thread)
[09:31:55] <paravoid>	 I didn't
[09:32:22] <paravoid>	 is that ishmael?
[09:32:30] <paravoid>	 hmmm, that looks like a much nicer way than the web intf
[09:32:35] <mark>	 hmmz
[09:32:39] <mark>	 now amssq58 is in trouble
[09:32:46] <mark>	 [497924.126440] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[09:32:49] <mark>	 again
[09:34:06] <paravoid>	 hm
[09:34:17] <paravoid>	 ms-be1003 had very weird symptoms last night
[09:34:29] <paravoid>	 kernel locks spinning forever
[09:34:34] <paravoid>	 50% cpu 50% iowait
[09:34:44] <paravoid>	 load 190
[09:34:53] <springle>	 paravoid: yes, ishmael plus a cross-slave view
[09:34:56] <paravoid>	 maybe related?
[09:35:39] <paravoid>	 springle: so, what am I looking at?
[09:36:22] <springle>	 the last seven days aggregating query review history from one non-snapshot slave per cluster, looking for queries based on total time and rows examined
[09:36:48] <paravoid>	 showToplevel isn't even close to the top?
[09:37:06] <springle>	 no, this doesn't account for spikes
[09:37:13] <paravoid>	 right
[09:37:14] <springle>	 only volume over 7 days
[09:39:55] <springle>	 of these, the problem ones are some forms of LogPager (paging all user history forever is madness), Wikibase\TermSqlIndex::getMatchingIDs (fixed with reindexing mid last week), and SpecialAllpages::showToplevel 
[09:40:07] <springle>	 problems == cause spikes
[09:40:21] <paravoid>	 nod
[09:40:41] <paravoid>	 logpager and showtoplevel are not new, though, right?
[09:41:18] <paravoid>	 maybe someone's crawling as theorized on list, who knows...
[09:41:42] <springle>	 have been fighting with logpager for months. showtoplevel is new in that i've not had it show up on this sort of list before
[09:41:50] <springle>	 but it's an old query
[09:45:43] <mark>	 !log rebooting amssq58 with sysrq-trigger
[09:46:02] <morebots>	 Logged the message, Master
[09:46:21] <springle>	 paravoid: a parallel issue that's affected slave cpu is updatespecialpages jobs on terbim. working on getting those onto the snapshot slaves along with dumps
[09:46:58] <paravoid>	 right, the one I inadvertently fixed the other day
[09:47:54] <paravoid>	 mark: "cdn" 5xx are elevated since yesterday
[09:48:12] <paravoid>	 http://gdash.wikimedia.org/dashboards/reqerror/
[09:48:27] <icinga-wm>	 RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Tue Nov 12 09:48:22 UTC 2013  
[09:50:00] <mark>	 i don't... see that?
[09:50:53] <paravoid>	 it's not very apparent, I only noticed because I looked at those graphs yesterday too
[09:51:02] <paravoid>	 so if you look at the -1 day one
[09:51:36] <paravoid>	 the blue line is usually close to zero, there's  a small bump around 13:00 UTC which is the esams congestion issue
[09:51:51] <mark>	 i'll believe it, I think especially those gzip errors are suspicious
[09:51:54] <paravoid>	 then it's fixed shortly after that (by your change) and goes back to zero for a while
[09:52:02] <paravoid>	 then it starts going up around 16:00 again
[09:52:42] <paravoid>	 little before
[09:52:52] <paravoid>	 you switched traffic to varnish at 15:30
[10:16:52] <mark>	 so I think these amssq* boxes are dying with kmem_alloc errors because they're under memory pressure due to too many dirty pages
[10:16:58] <mark>	 I guess we should change the thresholds a little
[10:28:20] <grrrit-wm>	 (03PS3) 10Faidon Liambotis: Remove references to 'olivneh' account from node defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/92267 (owner: 10Ori.livneh)
[10:28:32] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Remove references to 'olivneh' account from node defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/92267 (owner: 10Ori.livneh)
[10:30:38] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: Slight restructure for java module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/74380 (owner: 10Ottomata)
[10:32:40] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "The text in parentheses is actually quite useful. We have a very confusing (to some :) rule that we've even debated in the past that we us" [operations/puppet] - 10https://gerrit.wikimedia.org/r/81630 (owner: 10Ori.livneh)
[10:35:13] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "Why do we neeed (tool)labsbeta.pp? Just kill that and rename labs.pp to toollabs.pp." [operations/puppet] - 10https://gerrit.wikimedia.org/r/84926 (owner: 10Yuvipanda)
[10:35:32] <grrrit-wm>	 (03PS1) 10ArielGlenn: remove rose (long gone); fix range comments [operations/dns] - 10https://gerrit.wikimedia.org/r/94876 
[10:36:42] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Is this a -1 (improve by moving to dev_environ) or a -2 (do not submit)? If it's the latter, then abandon the change since it's over a mon" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon)
[10:37:06] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] remove rose (long gone); fix range comments [operations/dns] - 10https://gerrit.wikimedia.org/r/94876 (owner: 10ArielGlenn)
[10:38:02] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: Hopefully fix the Parsoid Varnishes not showing up as such in Ganglia [operations/puppet] - 10https://gerrit.wikimedia.org/r/69443 (owner: 10Catrope)
[10:39:40] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-1] "-1 because of what Reedy said. Max, you recently said you've been using this successfully, so let's fix it up and merge it?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/38252 (owner: 10MaxSem)
[10:40:05] <MaxSem>	 ahhh, remembering fail:P
[10:41:16] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "Any progress, Andrew?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata)
[10:43:37] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: Adding IPv4/6 networks in ferms defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/89791 (owner: 10Akosiaris)
[10:47:05] <paravoid>	 apergos: there's quite a lot of beta patchsets submitted by hashar, are you handling these?
[10:47:09] <paravoid>	 you're the one doing beta now, aren't you? :)
[10:47:19] <apergos>	 uhh
[10:47:31] <paravoid>	 (and ci)
[10:47:40] <apergos>	 what is ci?
[10:47:47] <paravoid>	 contint
[10:48:10] <apergos>	 I can do thebeta ones, he usually adds me as a reviewer if he wants me to look 
[10:48:11] <paravoid>	 ci is the more industry-known abbreviation
[10:48:24] <apergos>	 but I can rmeind him of that
[10:48:28] <apergos>	 *remind
[10:48:36] <paravoid>	 he's here :)
[10:48:39] <apergos>	 hashar: 
[10:48:42] <paravoid>	 hashar: heeelllo :)
[10:48:56] <hashar>	 hello
[10:49:02] <apergos>	 if you want reviews from me on puppet changesets having to do with beta, add me as a reviewer
[10:49:09] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 031] "I like this and despite plans to deprecate decom.pp, I don't see this happening very soon. Rebase & merge?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/93930 (owner: 10ArielGlenn)
[10:49:18] <hashar>	 I would apergos  :-]
[10:49:19] <apergos>	 nagging me doesn't hurt but I do look at the dash once every few days
[10:49:32] <hashar>	 been a bit too busy with CI for the last 5-6 weeks or so though
[10:49:47] <paravoid>	 who's reviewing those, hashar?
[10:50:14] <paravoid>	 https://gerrit.wikimedia.org/r/#/q/project:operations/puppet+owner:%22Hashar+%253Chashar%2540free.fr%253E%22+status:open,n,z
[10:50:37] <hashar>	 ops ? :-D
[10:50:43] <hashar>	 I haven't nagged anyone for them 
[10:51:00] <hashar>	 3 of them are related to an upgrade of Zuul, I have wrote them lsat week
[10:51:09] <hashar>	 haven't written down the migration plan yet
[10:51:21] <paravoid>	 as MaxSem and others know well, we generally suck at processing open patchsets with no reviewers set :)
[10:51:29] <grrrit-wm>	 (03PS3) 10Faidon Liambotis: rake validate now let puppet output colors [operations/puppet] - 10https://gerrit.wikimedia.org/r/77381 (owner: 10Hashar)
[10:51:43] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] rake validate now let puppet output colors [operations/puppet] - 10https://gerrit.wikimedia.org/r/77381 (owner: 10Hashar)
[10:51:48] <hashar>	 one is about tweaking jobrunner / videoscaler roles  which Ariel reviewed last week, still have to follow up though
[10:52:11] <paravoid>	 I just went through the list and you're on top of it ;)
[10:52:17] <hashar>	 well
[10:52:21] <hashar>	 because I send a ton of patches
[10:52:48] <hashar>	 nowadays, most of my puppet changes are reviewed /merged quite fast
[10:53:22] <aude>	 hi apergos 
[10:53:32] <hashar>	 there is  https://gerrit.wikimedia.org/r/65254  which is all about setting a symlink on beta :]
[10:54:42] <apergos>	 hello aude
[10:54:53] <aude>	 apergos: i want to ask about json dumps for wikidata
[10:54:54] <aude>	 per https://bugzilla.wikimedia.org/show_bug.cgi?id=54369
[10:55:12] <aude>	 what can we do to move it forward :)
[10:55:14] <aude>	 ?
[10:55:23] <aude>	 would an RT ticket be helpful?
[10:56:10] <apergos>	 no, its not something I can just fold into the regular dumps, so it's another maintenance script that needs to be put somewhere, tested, then we argue about the frequency, where the output goes, etc
[10:56:22] <aude>	 where is that somewhere?
[10:56:39] <aude>	 terbium (?) or arsenic(?)
[10:56:46] <apergos>	 maintenance scripts run on terbium I think (don't they?) 
[10:56:49] <aude>	 ok
[10:57:08] <apergos>	 I'm not sure what arsenic is being used for exactly, I know some cirrussearch stuff was happening there
[10:57:09] <aude>	 and then where the output goes?
[10:57:12] <aude>	 cirrus
[10:57:17] <aude>	 it's a new box for cirrus
[10:57:32] <aude>	 but might be used for cron jobs and scripts
[10:57:33] <apergos>	 ok well this script doesn't belong there then
[10:57:38] <apergos>	 oh
[10:57:40] <apergos>	 :-D
[10:57:43] <aude>	 i'd have to ask chad
[10:57:47] <apergos>	 right
[10:58:00] <aude>	 we can try terbium and if it's too much load, then find somewhere else
[10:58:27] <aude>	 we can test the setup with test.wikidata 
[10:58:32] <grrrit-wm>	 (03Restored) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 (owner: 10Hashar)
[10:58:32] <aude>	 although test.wikidata is very small
[10:58:36] <grrrit-wm>	 (03PS4) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 
[10:58:48] <apergos>	 if you run it on that at least you flush out silly errors
[10:58:56] <aude>	 yes
[10:59:05] <apergos>	 do you have terbium access?
[10:59:08] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 (owner: 10Hashar)
[10:59:15] <aude>	 i don't but can use puppet
[11:00:03] <apergos>	 well first steps would be to run by hand over there once in a screen session and see how it behaves, for test.wikidata that is
[11:00:09] <aude>	 ok
[11:00:11] <apergos>	 and then the same for wikidatawiki 
[11:00:17] <aude>	 i could ask about shell access :)
[11:00:25] <aude>	 don't know if it's possible but think it would help
[11:00:28] <apergos>	 how large do we expect these files to be? any notion?
[11:00:45] <aude>	 i don't know
[11:00:47] <aude>	 well.....
[11:00:54] <aude>	 similar to the current pages xml
[11:01:06] <aude>	 probably, though it's not quite as verbose
[11:01:20] <apergos>	 let's see how big those are
[11:01:20] <aude>	 wikidatawiki-20131006-pages-meta-current.xml.bz2 1.8 GB
[11:01:27] <apergos>	 ok that's not too bad
[11:01:29] <aude>	 and just items / properties
[11:01:47] <apergos>	 then there's the matter of where it should get put when done
[11:01:56] <apergos>	 and how often it should get produced
[11:01:59] <aude>	 then we'll need to make something like http://dumps.wikimedia.org/wikidatawiki/
[11:02:04] <aude>	 yep
[11:02:17] <aude>	 once a week is probably fine
[11:02:28] <apergos>	 it's going to land in other/something
[11:02:36] <apergos>	 we try not to clutter up the toplevel
[11:02:40] <aude>	 ok
[11:02:42] <apergos>	 note the word "try"
[11:02:55] <aude>	 looks like just directory listing
[11:02:58] <apergos>	 anyways, can we discuss this on the bug report? maybe summarize what's been said here so far
[11:03:02] <aude>	 ok
[11:03:09] <apergos>	 and next step
[11:04:28] <aude>	 added to the bug report
[11:04:49] <aude>	 to try in terbium, it's either i try to get shell access and do it or need help
[11:05:20] <aude>	 thanks apergos :)
[11:05:58] <apergos>	 to try in terbium, someone gives good instructions (see my comment just now on the report) 
[11:06:00] <apergos>	 yw
[11:06:08] <aude>	 ok
[11:10:28] <apergos>	 hashar: 
[11:10:56] <apergos>	 is /data/project/apache or /data/project/apache/common-local set up anywhere? 
[11:11:05] <apergos>	 in the puppet manifests that is
[11:16:06] <aude>	 apergos: put the command in
[11:16:19] <grrrit-wm>	 (03PS1) 10Hashar: rake validate was failling on non tty [operations/puppet] - 10https://gerrit.wikimedia.org/r/94880 
[11:17:04] <hashar>	 apergos: not y
[11:17:08] <hashar>	 grr
[11:17:22] <apergos>	 aude, thanks
[11:18:19] <hashar>	 apergos: /data/project… are not used directly
[11:18:20] <hashar>	 we have symlinks all over the place
[11:18:20] <apergos>	 not going to shard for this round. let's see what the script does normally.
[11:19:16] <hashar>	 bah puppet linting is broken :D  the rake file used to validate the lints is wrong. https://gerrit.wikimedia.org/r/#/c/94880/ should fix it
[11:19:37] <apergos>	 I don't care that it's not used drectly, that's not an issue
[11:19:59] <aude>	 sharding not needed for test wikidata
[11:20:06] <apergos>	 but puppet should manage those two directories (in case you ever set up a new box right?)
[11:20:11] <aude>	 and can be experimented with for wikidata
[11:20:30] <apergos>	 hmm just a second
[11:20:39] <apergos>	 we ant to pipe to bzip2 
[11:20:49] <aude>	 apergos: yes
[11:21:08] <apergos>	 --output  won't let medo that
[11:21:18] <aude>	 you can omit it probably
[11:21:22] <aude>	 and then just piple
[11:21:23] <aude>	 pipe
[11:21:40] <apergos>	 where do progress messages go in that case?
[11:21:51] <aude>	 hmmmm
[11:21:53] <aude>	 ok :)
[11:22:04] <aude>	 looking
[11:22:09] <apergos>	 thanks
[11:24:20] <grrrit-wm>	 (03PS2) 10Dereckson: Temporary lift of IP cap for WikiCon 2013 in de/en.WP, Commons, de/en.wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 (owner: 10Raimond Spekking)
[11:24:40] <grrrit-wm>	 (03CR) 10Dereckson: [C: 031] Throttle rule for WikiCon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 (owner: 10Raimond Spekking)
[11:27:03] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 100,000  
[11:28:47] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Throttle rule for WikiCon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 (owner: 10Raimond Spekking)
[11:28:59] <grrrit-wm>	 (03Merged) 10jenkins-bot: Throttle rule for WikiCon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 (owner: 10Raimond Spekking)
[11:29:57] <logmsgbot>	 !log hashar synchronized wmf-config/throttle.php  'thottle rule for WikiCon {{gerrit|94860}}'
[11:30:03] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (110729)  
[11:30:15] <morebots>	 Logged the message, Master
[11:30:18] <logmsgbot>	 !log hashar synchronized wmf-config/InitialiseSettings.php  'thottle rule for WikiCon {{gerrit|94860}}'
[11:30:19] <grrrit-wm>	 (03CR) 10Hashar: "deployed in production" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94860 (owner: 10Raimond Spekking)
[11:30:34] <morebots>	 Logged the message, Master
[11:33:49] <mark>	 !log Manually set dirty_background_ratio to 5 (from 10) on amssq58
[11:34:06] <morebots>	 Logged the message, Master
[11:36:15] <hashar>	 I am off for lunch / nap
[11:36:23] <hashar>	 ops/puppet validation is broken right now,  https://gerrit.wikimedia.org/r/#/c/94880/  should fix it
[11:43:43] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "This happens because redis first attempts to set the limit to 10032 (default maxclients = 10000 + 32 fds reserved for internal usage), fai" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94848 (owner: 10Tim Starling)
[11:44:17] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] rake validate was failling on non tty [operations/puppet] - 10https://gerrit.wikimedia.org/r/94880 (owner: 10Hashar)
[11:49:57] <grrrit-wm>	 (03CR) 10ArielGlenn: "I would like to see declarations for the directories /data/project/apache and /data/project/apache/common-local, just as you have done for" [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 (owner: 10Hashar)
[12:28:30] <grrrit-wm>	 (03PS3) 10Mark Bergsma: Allow caching of login.wikimedia.org requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94765 
[12:28:31] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Filter out some noise requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94886 
[12:30:04] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032 V: 032] Filter out some noise requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94886 (owner: 10Mark Bergsma)
[12:34:01] <grrrit-wm>	 (03PS4) 10Mark Bergsma: Allow caching of login.wikimedia.org requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94765 
[12:34:02] <grrrit-wm>	 (03PS1) 10Mark Bergsma: req.request instead of req.method [operations/puppet] - 10https://gerrit.wikimedia.org/r/94887 
[12:34:26] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032 V: 032] req.request instead of req.method [operations/puppet] - 10https://gerrit.wikimedia.org/r/94887 (owner: 10Mark Bergsma)
[13:12:15] <icinga-wm>	 PROBLEM - MySQL Processlist on db1002 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 178 statistics  
[13:13:15] <icinga-wm>	 RECOVERY - MySQL Processlist on db1002 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 3 statistics  
[13:25:43] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Allow caching of login.wikimedia.org requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94765 (owner: 10Mark Bergsma)
[13:39:40] <grrrit-wm>	 (03PS7) 10Hashar: beta: symlink /a/common [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 
[13:40:00] <grrrit-wm>	 (03CR) 10Hashar: "defines /data/project/apache and /data/project/apache/common-local as belonging to mwdeploy:mwdeploy." [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 (owner: 10Hashar)
[13:40:05] <grrrit-wm>	 (03PS8) 10Hashar: beta: symlink /a/common [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 
[13:47:53] <paravoid>	 eww
[13:49:21] <grrrit-wm>	 (03PS1) 10Akosiaris: Moving esams to new puppet infrastructure [operations/dns] - 10https://gerrit.wikimedia.org/r/94898 
[13:50:11] <paravoid>	 heh interesting
[13:50:15] <paravoid>	 a wikimedia.org CNAMEd to wmnet
[13:50:23] <paravoid>	 not sure how I feel about that
[13:50:32] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032] Moving esams to new puppet infrastructure [operations/dns] - 10https://gerrit.wikimedia.org/r/94898 (owner: 10Akosiaris)
[13:50:52] <akosiaris>	 lol
[13:51:07] <akosiaris>	 well... it is weird to say the least....
[13:51:32] <akosiaris>	 how would you feel about a wikimedia.org A record pointing to an 10.x address ?
[13:51:46] <akosiaris>	 not much different...
[13:53:24] <paravoid>	 I guess
[13:53:33] <paravoid>	 maybe we should just set server = explicitly? :)
[13:54:44] <akosiaris>	 wouldn't that make it more difficult to make such changes ?
[13:54:54] <akosiaris>	 hmmm well not really now that i think about it 
[13:55:04] <paravoid>	 anyway, let's think about it when you're done
[13:55:09] <paravoid>	 not the right time now I guess :)
[13:55:16] <akosiaris>	 we would have to maintain a hash in puppet for at least the DCs
[13:55:25] <akosiaris>	 yeah ok
[13:55:53] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "Fine to me, thank you :-] Feel free to merge at anytime." [operations/puppet] - 10https://gerrit.wikimedia.org/r/94257 (owner: 10Andrew Bogott)
[13:56:13] <grrrit-wm>	 (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 (owner: 10Hashar)
[14:03:04] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1021 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 19428654 for key PRIMARY on query. Defaul  
[14:03:04] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1026 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 19428654 for key PRIMARY on query. Defaul  
[14:03:14] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1045 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 19428654 for key PRIMARY on query. Defaul  
[14:03:15] <icinga-wm>	 PROBLEM - MySQL Slave Running on db73 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 19428654 for key PRIMARY on query. Defaul  
[14:03:56] <springle>	 wft
[14:03:57] <paravoid>	 that doesn't sound very good
[14:04:07] <paravoid>	 oh, heh, hey sean
[14:04:08] <springle>	 OSC gone wrong
[14:04:40] <paravoid>	 OSC?
[14:04:51] <paravoid>	 ah, schema change?
[14:05:11] <paravoid>	 anyway, I'll shut up, let me know if you need anything
[14:06:16] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1005 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 19428655 for key PRIMARY on query. Defaul  
[14:07:05] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 321 seconds  
[14:07:15] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1021 is CRITICAL: CRIT replication delay 329 seconds  
[14:07:16] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1045 is CRITICAL: CRIT replication delay 331 seconds  
[14:07:55] <icinga-wm>	 RECOVERY - check_job_queue on arsenic is OK: JOBQUEUE OK - all job queues below 100,000  
[14:08:15] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1045 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[14:08:15] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1045 is OK: OK replication delay -0 seconds  
[14:08:36] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032] More fixes for file permissions/ownerships [operations/puppet] - 10https://gerrit.wikimedia.org/r/94777 (owner: 10Akosiaris)
[14:09:05] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1026 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[14:09:05] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1021 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[14:09:15] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1005 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[14:09:15] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay -0 seconds  
[14:10:56] <icinga-wm>	 PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 99,999 jobs: , Total (108724)  
[14:11:05] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db73 is OK: OK replication delay -0 seconds  
[14:11:15] <icinga-wm>	 RECOVERY - MySQL Slave Running on db73 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[14:17:10] <springle>	 !log paused externallinks OSC jobs after replication glitch on dewiki. original table and data remain untouched
[14:17:28] <morebots>	 Logged the message, Master
[14:17:32] <springle>	 *sigh*
[14:18:09] <grrrit-wm>	 (03PS2) 10Faidon Liambotis: Remove misc::maintenance::foundationwiki cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/91676 (owner: 10Reedy)
[14:19:15] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Remove misc::maintenance::foundationwiki cronjobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/91676 (owner: 10Reedy)
[14:19:16] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Ignore Range requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94902 
[14:20:34] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Ignore Range requests [operations/puppet] - 10https://gerrit.wikimedia.org/r/94902 (owner: 10Mark Bergsma)
[14:20:41] <paravoid>	 I have them open
[14:20:50] <paravoid>	 merged
[14:21:00] <paravoid>	 (them = sockpuppet + palladium)
[14:21:36] <mark>	 we need to merge in both places now?
[14:21:43] <paravoid>	 temporarily
[14:21:44] <akosiaris>	 for a couple of days
[14:21:46] <paravoid>	 alex mailed about that :)
[14:21:51] <akosiaris>	 sorry :-(
[14:22:14] <akosiaris>	 I could switch everything at once... but I get a feeling it wouldn't be nice
[14:22:37] <mark>	 not a problem
[14:35:48] <springle>	 paravoid: the client IPs on these recent spikes are msnbot :)
[14:36:19] <paravoid>	 22:07 < paravoid> second time I'm seeing this msnbot IP
[14:36:25] <springle>	 it just hit wikidata with a load of LogPager queries with absurd LIMIT offsets
[14:36:40] <paravoid>	 22:28 < paravoid> 536184595       wikiuser        10.64.32.55:52792       dewiki  Query   208     Sending data    SELECT /* IndexPager::buildQueryInfo (LogPager) 157.55.32.209 */  log_id,log_type,log_action,log_timestamp,log_user,log_user_text,lo
[14:36:44] <paravoid>	 g_namespace,log_title,log_comment,log_params,log_deleted,user_id,user_name,user_editcount,ts_tags  FROM `logging` LEFT JOIN `user` ON ((log_user=user_id)) LEFT JOIN `tag_summary` ON ((ts_log_id=log_id))  WHERE (log_type NOT IN ('suppress','spambl
[14:36:48] <paravoid>	 acklist')) AND log_user = '676408' AND ((log_deleted & 4) = 0) AND (log_type != 'review')  ORDER BY log_timestamp DESC LIMI
[14:36:51] <paravoid>	 T 51       0.000
[14:36:53] <paravoid>	 this is from the day of the outage
[14:36:53] <aude>	 ban it!
[14:36:55] <paravoid>	 22:28 < paravoid> is one of the ones I captured before
[14:36:58] <paravoid>	 22:29 < paravoid> the IP is msnbot's, don't worry about leaking it
[14:37:00] <paravoid>	 :)
[14:37:02] <springle>	 paravoid: ah :)
[14:38:25] <springle>	 i wish i was around at the time. so far behind
[14:38:33] <paravoid>	 heh, sorry
[14:38:43] <paravoid>	 that was from ishmael, btw
[14:38:58] <paravoid>	 the outage lasted for about 10', I didn't get enough time to go digging in databases in realtime
[14:39:08] <paravoid>	 time to notice, time to pinpoint it to a db issue etc.
[14:39:45] <paravoid>	 so I did see your email about running 'show full processlist' etc. and I usually do that (although usually not show engine innodb), but it was too late at the time anyway
[14:40:30] <paravoid>	 hm, or not, the above was from a show full processlist
[14:40:38] <paravoid>	 but it was after the site was back up
[14:43:41] <springle>	 i checked a couple from ishmael but got no reverse dns at the time. seeing these now in the pt-kill stderr
[14:51:50] <grrrit-wm>	 (03CR) 10Ottomata: [C: 031] Remove Kraken-specific varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/91492 (owner: 10Ori.livneh)
[15:01:31] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032] Remove references to /etc/puppet/software [operations/puppet] - 10https://gerrit.wikimedia.org/r/94779 (owner: 10Akosiaris)
[15:07:33] <grrrit-wm>	 (03CR) 10Hashar: "tested in labs :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/93457 (owner: 10Hashar)
[15:21:37] <manybubbles>	 does anyone have a chance to look at Bug 56936?  en.wp search failures that I can't reproduce or find logs for.
[15:25:21] <ottomata>	 akosiaris:  hiii
[15:25:27] <ottomata>	 got some time to figure out the varnishkafka.log thing?
[15:25:50] <akosiaris>	 ottomata: yes
[15:26:28] <akosiaris>	 gimme a sec to read so I remember what we are talking about again
[15:30:07] <ottomata>	 k
[15:30:19] <ottomata>	 akosiaris: https://gerrit.wikimedia.org/r/#/c/78782/16/debian/rules
[15:30:40] <paravoid>	 ottomata: are you going to deploy https://gerrit.wikimedia.org/r/#/c/91492/ ?
[15:30:50] <ottomata>	 paravoid: sure shall I?
[15:30:59] <paravoid>	 why not? :)
[15:31:02] <ottomata>	 k
[15:31:04] <ottomata>	 on it
[15:31:14] <paravoid>	 mark only gave it a -1 because of making sure it gets removed
[15:31:19] <ottomata>	 paravoid, the other thing I want to work on today is IPv6
[15:31:21] <paravoid>	 as long as you do that, it's fine, imho
[15:31:21] <ottomata>	 k yeah, i'll do that
[15:31:34] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/93983/
[15:31:41] <ottomata>	 i got your email about the AAAA taking priority 
[15:31:42] <ottomata>	 so ok
[15:31:49] <ottomata>	 i don't have a full grasp of everything that needs to happen here though
[15:32:41] <paravoid>	 what are you missing?
[15:32:43] <ottomata>	 like, i'm not sure how the dhcp bit is supposed to work i think
[15:32:46] <paravoid>	 dhcp?
[15:32:48] <paravoid>	 there's no dhcp
[15:32:49] <ottomata>	 ummm
[15:32:51] <ottomata>	 ok
[15:32:55] <ottomata>	 how are the IPs actaully assigned?
[15:33:09] <ottomata>	 these are just dns records, right?
[15:33:10] <paravoid>	 the boxes have IP(v6)s already but that's due to autoconfiguration
[15:33:21] <paravoid>	 that is, they have the network + the mac address suffix
[15:33:23] <ottomata>	 are they inferred from the dns lookup?
[15:33:26] <paravoid>	 no
[15:33:33] <paravoid>	 you need to use interface::add_ip6_mapped
[15:33:40] <paravoid>	 that infers the IPv6 from the IPv4
[15:33:43] <ottomata>	 ah!
[15:33:54] <ottomata>	 ahhhh
[15:33:56] <ottomata>	 that is what I'm missing
[15:33:56] <ottomata>	 ok
[15:33:58] <ottomata>	 great
[15:34:02] <ottomata>	 so that will get IPs and names all set up
[15:34:08] <paravoid>	 not names
[15:34:10] <ottomata>	 well
[15:34:11] <ottomata>	 my change
[15:34:12] <paravoid>	 names is your commit
[15:34:13] <ottomata>	 is names
[15:34:13] <paravoid>	 yes
[15:34:16] <paravoid>	 exactly
[15:34:22] <ottomata>	 so those two together
[15:34:22] <paravoid>	         $ipv6_address = inline_template("<%= require 'ipaddr'; (IPAddr.new(scope.lookupvar(\"::ipaddress6_${intf}\")).mask(64) | IPAddr.new(ip4_address.gsub('.', ':'))).to_s() %>")
[15:34:23] <ottomata>	 ok
[15:34:28] <ottomata>	 great
[15:34:35] <ottomata>	 ok, and then does anything need to happen for public routing to work?
[15:34:53] <paravoid>	 so the boxes will have ipv6 connectivity (they already do)
[15:35:00] <paravoid>	 but we have ACLs in place, since these are private zones
[15:35:18] <paravoid>	 so you need to make a hole in the ACL, Leslie can probably help you with that
[15:35:32] <paravoid>	 I can too, but I had my fair share of net ops for a while :P
[15:35:52] <ottomata>	 haha
[15:35:53] <ottomata>	 ok
[15:36:07] <ottomata>	 ok great
[15:36:42] <akosiaris>	 ottomata: varnishkafka.log is what ?
[15:36:56] <ottomata>	 just the process logs
[15:37:01] <ottomata>	 error messages, startmessages, stats
[15:37:01] <ottomata>	 etc.
[15:37:02] <akosiaris>	 so errors etc
[15:37:03] <akosiaris>	 ok
[15:37:04] <ottomata>	 ja
[15:37:34] <ottomata>	 also, paravoid: https://gerrit.wikimedia.org/r/#/c/94148/
[15:37:38] <paravoid>	 ottomata: btw, as we discussed on that other ticket
[15:37:42] <akosiaris>	 /var/log/varnishkafka/varnishkafka.log for starters and your 1st problem is solved (with rsyslog not being able to write to /var/log/varnish
[15:37:43] <paravoid>	 we need to move one of the brokers to a different row
[15:37:47] <ottomata>	 right yeah
[15:37:49] <paravoid>	 this means different IP
[15:37:50] <paravoid>	 and IPv6
[15:37:51] <ottomata>	 oof
[15:37:54] <ottomata>	 ok
[15:37:54] <akosiaris>	 and postinst indeed for creating files etc
[15:37:59] <paravoid>	 (each row is a different network)
[15:38:03] <paravoid>	 so you might want to do that first
[15:38:06] <ottomata>	 oof
[15:38:10] <akosiaris>	 posting that to the change as well
[15:38:10] <ottomata>	 yeah, rats
[15:38:10] <ottomata>	 ok
[15:38:15] <ottomata>	 that is going to take a while I betcha though
[15:38:16] <ottomata>	 sigh
[15:38:39] <paravoid>	 it's just chris unracking and racking again
[15:38:50] <paravoid>	 it's work, but nothing insurmountable
[15:38:58] <ottomata>	 is there enough room?  do we have 3 rows?  does he have to move around other boxes too?
[15:38:59] <paravoid>	 but I'll leave scheduling to you ;)
[15:39:13] <paravoid>	 we have 3 rows and we're in the process of setting up a fourth
[15:39:16] <paravoid>	 (row D)
[15:39:27] <paravoid>	 no idea about room, have a look at racktables
[15:39:33] <ottomata>	 k
[15:39:44] <paravoid>	 it's a little more complicated than that, some racks have only 10gbe ports
[15:40:42] <paravoid>	 you can see the chassis topology for each row from the asw, and from the model you can see if it's a 10gbe switch or not
[15:41:08] <ottomata>	 akosiaris: how does /var/log/varnishkafka help me?
[15:41:40] <paravoid>	 akosiaris: rules is the wrong place for this, but creating a logfile in postinst is also wrong
[15:42:00] <akosiaris>	 you don't have to give rsyslog special access to /var/log/varnish ?
[15:42:23] <akosiaris>	 paravoid: why if I may ask ?
[15:42:53] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "Yay :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94148 (owner: 10Ottomata)
[15:43:00] <ottomata>	 akosiaris: not to /var/log/varnish, all I did was create a first varnishkafka.log file that was syslog:adm writeable
[15:43:01] <akosiaris>	 altough in this case having rsyslog create the file might be better
[15:43:15] <ottomata>	 if /var/log/varnish{kafka} was syslog writeable, it'd be fine
[15:43:18] <ottomata>	 wouldn't have to create file
[15:43:38] <paravoid>	 akosiaris: it's very counterintuitive to rm varnishkafka.log and have it never being able to be created again by itself, for starters
[15:44:10] <akosiaris>	 ottomata: /var/log/varnish/varnishkafka.log i don't like anyway... it is counter-intuitive
[15:44:25] <paravoid>	 I don't remember the details much, but I think the problem was reusing /var/log/varnish?
[15:44:26] <ottomata>	 i'm fine with /var/log/varnishkafka
[15:44:28] <ottomata>	 just not sure how it would help
[15:44:51] <ottomata>	 paravoid, I don't think so, unless we were to make /var/log/varnishkafka somehow writeable by rsyslog
[15:45:08] <ottomata>	 the problem is that rsyslog doesn't run as root so it can't just write files wherever
[15:45:18] <paravoid>	 right
[15:45:34] <ottomata>	 akosiaris:  i think we ahve varnishkafka running as the varnishlog user right now
[15:45:38] <ottomata>	 should we create a new user too?
[15:45:39] <ottomata>	 and if so
[15:45:48] <ottomata>	 we could perhaps add syslog user to the varnishkafka group
[15:45:48] <paravoid>	 if so, please decide on the name first :)
[15:45:52] <akosiaris>	 lol
[15:45:54] <ottomata>	 and make the dir group writeable
[15:45:54] <ottomata>	 oh
[15:45:56] <ottomata>	 ha
[15:46:06] <ottomata>	 paravoid: due to upcoming changes in varnishncsa
[15:46:12] <ottomata>	 and possible deprecation of varnishkafka one day
[15:46:15] <ottomata>	 we've decided not to change the name
[15:46:31] <Snaps>	 depreWHAT?!
[15:46:33] <paravoid>	 how is that related?
[15:46:39] <ottomata>	 hahha, Snaps, right?
[15:46:48] <ottomata>	 wait I'm not making this up I thought we talked about this?
[15:46:56] <Snaps>	 yeah, otto is right
[15:47:57] <ottomata>	 I don't care either way actually
[15:48:02] <ottomata>	 should we rename it to varnishmill after all?
[15:49:34] <paravoid>	 I don't really mind what name you decide
[15:49:43] <paravoid>	 I do care about not renaming while in production though
[15:50:33] <ottomata>	 indeed
[15:50:42] <ottomata>	 we had decided not to change it, Snaps, do we stick to that decision?
[15:51:01] <Snaps>	 ottomata: nah, lets stick with varnishkafka. If it catches on and becomes the de facto varnish generalizator of world domination then we'll rename it.
[15:51:10] <paravoid>	 okay
[15:51:52] <ottomata>	 ok phew
[15:52:01] <akosiaris>	 :-)
[15:52:13] <akosiaris>	 and not that we are at it
[15:52:16] <ottomata>	 ok so akosiaris, what about making varnishkafka user & group and then addign syslog to the varnishkafka group and making /var/log/varnishkafka group writeable?
[15:52:19] <akosiaris>	 now*
[15:52:32] <akosiaris>	 let the file be created by rsyslog
[15:52:44] <akosiaris>	 rsyslog can create files in /var/log/ 
[15:52:47] <ottomata>	 that would do it, but the dir need sto be writable by syslgo then
[15:52:49] <ottomata>	 hmmmMMMmmmm
[15:53:01] <ottomata>	  /var/log/varnishkafkfa.log then?
[15:53:05] <akosiaris>	 yes
[15:53:08] <ottomata>	 done.
[15:53:08] <ottomata>	 easy
[15:53:11] <akosiaris>	 and no groups/users nothing :-)
[15:53:27] <paravoid>	 akosiaris: nope
[15:53:28] <akosiaris>	 just a /etc/rsyslog.d/varnishkafka file ? 
[15:53:39] <paravoid>	 rsyslog cannot write to /var/log
[15:53:45] <paravoid>	 ...Ubuntu.
[15:53:48] <ottomata>	 yeah taht's this
[15:53:48] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/78782/16/debian/70-varnishkafka.conf
[15:53:49] <akosiaris>	 yes it can
[15:53:53] <akosiaris>	 i just tested it
[15:54:11] <akosiaris>	 # cat lala.conf 
[15:54:11] <akosiaris>	 *.*	/var/log/all.log
[15:54:16] <akosiaris>	 and the file just got created
[15:54:44] <paravoid>	 I can't see how
[15:54:45] <ottomata>	 # sudo -u syslog touch /var/log/test.log
[15:54:45] <ottomata>	 touch: cannot touch `/var/log/test.log': Permission denied
[15:54:56] <paravoid>	 did you test it in a Debian box perhaps? :)
[15:55:02] <paravoid>	 Debian isn't as silly
[15:55:14] <akosiaris>	 Distributor ID:	Ubuntu
[15:55:14] <akosiaris>	 Description:	Ubuntu 13.04
[15:55:15] <akosiaris>	 Release:	13.04
[15:55:15] <akosiaris>	 Codename:	raring
[15:55:16] <ottomata>	 drwxr-xr-x 17 root root 4096 Nov 12 06:25 /var/log/
[15:55:27] <akosiaris>	 it probably has privileges on restart
[15:55:30] <ottomata>	 Ubuntu 12.04.2 LTS \n \l
[15:55:33] <akosiaris>	 that it drops afterwards
[15:55:36] <ottomata>	 hm
[15:55:50] <paravoid>	 we care about logrotate/sighup too, though
[15:56:00] <paravoid>	 (even if that's the case)
[15:56:40] <akosiaris>	 logrotate's job then 
[15:56:50] <akosiaris>	 tell rsyslog to restart... 
[15:56:55] <akosiaris>	 won't be the first 
[15:57:00] <paravoid>	 eww :)
[15:57:23] <akosiaris>	 /etc/logrotate.d/rsyslog
[15:57:31] <akosiaris>	 you are already doing it here anyway
[15:57:36] <akosiaris>	 	postrotate
[15:57:36] <akosiaris>	 		reload rsyslog >/dev/null 2>&1 || true
[15:57:40] <paravoid>	 reload, not restart
[15:57:43] <paravoid>	 sighup
[15:58:12] <paravoid>	 anyway
[15:58:24] <paravoid>	 too many people on a single problem
[15:58:34] <paravoid>	 I'll stay out of your away :)
[15:59:10] <akosiaris>	 :-)
[15:59:30] <akosiaris>	 lemme check what happens on rsyslog reload... damned upstart jobs
[15:59:50] <manybubbles>	 resending spam:
[15:59:52] <manybubbles>	 does anyone have a chance to look at Bug 56936?  en.wp search failures that I can't reproduce or find logs for.
[15:59:53] <ottomata>	 that's what we are doing now anyway?
[15:59:54] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/78782/16/debian/varnishkafka.logrotate
[16:00:25] <akosiaris>	 ottomata: ew.... pgrep ?
[16:00:26] <paravoid>	 manybubbles: I don't think anyone has looked at it, no
[16:00:36] <akosiaris>	 reload ? restart ?
[16:01:17] <manybubbles>	 paravoid: yeah, I've looked at it but no one else has had a chance far as I can tell.
[16:01:37] <paravoid>	 manybubbles: ops typically don't look at bugzilla much, even less so if there's no "ops" keyword....
[16:01:50] <ottomata>	 akosiaris:  i think there wa sa reason for that…not remembering though
[16:02:07] <paravoid>	 manybubbles: I can tell you, though, that ulsfo (or esams) has nothing to do with all that
[16:02:29] <paravoid>	 as far as I know, mediawiki appservers interface with the search cluster and we have appservers only in one place (at at time)
[16:02:32] <paravoid>	 at a*
[16:02:41] <paravoid>	 the other DCs are just frontend HTTP caching
[16:02:51] <ottomata>	 akosiaris: rsyslogd does a full daemon restart when it is HUPed.
[16:02:55] <ottomata>	 that's from here
[16:02:55] <ottomata>	 http://blog.gerhards.net/2008/10/new-rsyslog-hup-processing.html
[16:02:59] <ottomata>	 but maybe that is old, dunno
[16:02:59] <paravoid>	 so your best luck would be to investigate on the mediawiki and or lsearchd layer
[16:03:37] <manybubbles>	 paravoid: that is what I thought.  I did investigate the mediawiki/lsearchd layer and I can't reproduce or find logs of it.  I'll go whine to -dev:)
[16:03:43] <akosiaris>	 ottomata: sure ... but if you wanted to restart, why not call... restart rsyslog ?
[16:03:59] <akosiaris>	 the init/upstart scripts are going to do it better anyway
[16:03:59] <ottomata>	 i can
[16:04:07] <ottomata>	 yeah i don't know why we did this right now….
[16:04:08] <ottomata>	 fine with that
[16:04:11] <paravoid>	 manybubbles: you could add wfDebug calls wherever "An error has occurred while searching: The search backend returned an error: " is printed out, perhaps?
[16:04:12] <ottomata>	 cept paravoid says 'eww'
[16:04:13] <ottomata>	 :p
[16:04:26] <ottomata>	 and i agree, but not sure what else to do
[16:04:32] <paravoid>	 "reload syslog", not "restart syslog" please
[16:04:33] <ottomata>	 seems silly to restart rsyslog on all rotates
[16:04:35] <paravoid>	 (you summoned me!)
[16:04:46] <ottomata>	 sure
[16:04:46] <ottomata>	 but
[16:04:49] <ottomata>	 just fyi: rsyslogd does a full daemon restart when it is HUPed.
[16:05:09] <ottomata>	 (unless that is not true anymore, that quote is from 2008)
[16:05:23] <akosiaris>	 i expect it to be true
[16:05:23] <ottomata>	 googling more
[16:06:27] <akosiaris>	 /lib/systemd/system/rsyslog.service
[16:06:31] <akosiaris>	 heh... I wanna cry
[16:06:44] <paravoid>	 hahaha
[16:06:45] <akosiaris>	 thankfully not used ...
[16:07:04] <ottomata>	 oo
[16:07:04] <ottomata>	 In v4, we provide some support for the old-style semantics. We introduced a setting $HUPisRestart which may be set to "on" (tradional, heavy operation) or "off" (new, lightweight "file close only" operation). The initial versions had the default set to traditional behavior, but starting with 4.5.1 we are now using the new behavior as the default.
[16:07:46] <akosiaris>	 huh
[16:07:49] <akosiaris>	 so file close only ?
[16:07:51] <akosiaris>	 perfect
[16:08:02] <akosiaris>	 it will open the new file then
[16:08:03] <ottomata>	 ok phew
[16:08:03] <ottomata>	 Please note that restart-type HUP is depricated and will go away in rsyslog v5.
[16:08:12] <ottomata>	 we are Version: 5.8.6-1ubuntu8
[16:08:24] <ottomata>	 yeah ok 
[16:08:29] <ottomata>	 so will use reload
[16:09:02] <akosiaris>	 ok. however the first time it will not work
[16:09:26] <akosiaris>	 after that and with logrotate creating the new files that rsyslog will open after reload
[16:09:38] <akosiaris>	 it will be fine
[16:09:48] <akosiaris>	 so rsyslog will need a restart the very first time
[16:10:00] <akosiaris>	 that might be a good case for postinst of varnishkafka
[16:10:09] <ottomata>	 welllllllllllhm.
[16:10:18] <ottomata>	 will rsyslog create the file even if there is no output yet?
[16:10:23] <ottomata>	 varnishkafka isn't going to start on install
[16:10:27] <akosiaris>	 yes
[16:10:28] <ottomata>	 ok
[16:10:51] <akosiaris>	 so a single restart rsyslog call in configure state of postinst is ok
[16:12:04] <ottomata>	 ok
[16:12:22] <ottomata>	 akosiaris: 
[16:12:27] <ottomata>	 is the pgrep for rsyslog bad?
[16:12:28] <ottomata>	  /usr/bin/pgrep -P 1 rsyslogd >/dev/null
[16:12:29] <ottomata>	 ?
[16:12:36] <akosiaris>	 yap
[16:12:44] <akosiaris>	 go for a single reload rsyslog there
[16:12:46] <ottomata>	 how to test?  status doesn't seem to change error goes
[16:12:48] <ottomata>	 codes*
[16:12:50] <ottomata>	 exit val
[16:12:53] <ottomata>	 always 0 
[16:12:54] <akosiaris>	 it will do the right thing anyways
[16:13:01] <ottomata>	 but what if rsyslogd isn't running
[16:13:02] <icinga-wm>	 PROBLEM - Host analytics1013 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:13:02] <ottomata>	 that's why the check
[16:13:09] <ottomata>	 like if someone shut it down on purpose
[16:13:27] <akosiaris>	 then logrotate should not restart it
[16:13:35] <akosiaris>	 it would be wrong
[16:13:50] <ottomata>	 riiiight, so it needs to check if rsyslog is running before calling reload
[16:13:52] <ottomata>	 right?
[16:14:07] <ottomata>	 oh reload won't start if stopped
[16:14:14] <ottomata>	 will it?
[16:14:15] <ottomata>	  hm checking
[16:14:35] <ottomata>	 looks like it won't ok
[16:14:35] <akosiaris>	 it won't
[16:14:49] <akosiaris>	 that is why i said use restart, reload commands
[16:14:54] <akosiaris>	 those problems are already solved
[16:14:58] <ottomata>	 ok
[16:16:02] <icinga-wm>	 RECOVERY - Host analytics1013 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms  
[16:16:09] <apergos>	 is anyone looking at 1013 or should I... huh.  that's interesting
[16:16:36] <akosiaris>	 telepathy ?
[16:16:51] <apergos>	 it knew I was going to give it a talking to
[16:17:23] <apergos>	 rebooted
[16:17:27] <ottomata>	 hah
[16:17:34] <ottomata>	 uh, 1013 wassup, still power problem?
[16:17:43] <apergos>	 if that's not any of you then *cough* power issues*cough*?
[16:17:51] <ottomata>	 yeah not me
[16:17:55] <apergos>	 yeah that's my first thought but let's look at things like ganglia to make sure right?
[16:19:06] <apergos>	 not swap, lots of disk busy at the last (atop)
[16:19:52] <icinga-wm>	 PROBLEM - Host analytics1011 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:19:53] <apergos>	 also analytics1011? 
[16:19:57] <akosiaris>	 ok ...
[16:19:59] <apergos>	 so.
[16:20:01] <akosiaris>	 now this is problem
[16:20:13] <akosiaris>	 i suppose same rack ?
[16:20:19] <apergos>	 same 4 as last time
[16:20:21] <ottomata>	 same rack yeah, these are the same machines 
[16:20:24] <akosiaris>	 same machines as 4 days ago
[16:20:34] <akosiaris>	 nice...
[16:20:53] <akosiaris>	 PDU problems again...
[16:21:01] <akosiaris>	 how did we make sure last time ?
[16:21:06] <apergos>	 and we just lost 1013 again
[16:21:09] <akosiaris>	 yes
[16:21:10] <apergos>	 chris was able to tell last time
[16:21:18] <apergos>	 physical presence I guess
[16:21:23] <akosiaris>	 heh
[16:21:53] <apergos>	 occam says it's the same thing as last time, that's for sure
[16:22:06] <apergos>	 lemme find that ticket
[16:22:42] <icinga-wm>	 PROBLEM - Host analytics1013 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:23:12] <icinga-wm>	 RECOVERY - Host analytics1011 is UP: PING OK - Packet loss = 0%, RTA = 2.87 ms  
[16:23:38] <apergos>	 cmjohnson1: around?  we have it seems more of the pdu issue causing reboots
[16:24:16] <apergos>	 (I have commented on the ticket, 6238, just for the record)
[16:25:54] <cmjohnson1>	 apergos: yes..i am swapping the pdu with one of the new ones destined for row d
[16:26:02] <icinga-wm>	 RECOVERY - Host analytics1013 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms  
[16:26:14] <apergos>	 ok, crossing fingers
[16:26:42] <apergos>	 what's your eta for that?
[16:26:46] <cmjohnson1>	 it will be easier for me to t-shoot it or get it replaced...cuz it's going to be a vendor thing and we really can't afford for that
[16:26:54] <cmjohnson1>	 my eta is about an hour or so
[16:27:08] <ottomata>	 awesome
[16:27:16] <ottomata>	 cmjohnson1: lemme know when you want to talk about row reshuffling
[16:27:22] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on amssq60 is CRITICAL: HTTP CRITICAL - No data received from host  
[16:28:15] <apergos>	 ok, thanks a lot
[16:28:28] <cmjohnson1>	 ottomata: not going to row reshuffle...going to try and move w/out powering down....moving all side B to new pdu and then will move side A
[16:29:02] <cmjohnson1>	 unless something is backwards which i suspect an1013 is ...all should move without interruption 
[16:29:07] <apergos>	 the reshuffle is for later; there's a ticket about splitting the hosts up so we have some redundancy, later
[16:30:41] <ottomata>	 yeah cmjohnson1, that is to avoid potential dataloss problems if we get a row failure that affects all nodes in it
[16:30:47] <ottomata>	 basically all analytics production nodes are in the same row right now
[16:30:51] <ottomata>	 quote 'production'
[16:30:51] <cmjohnson1>	 !log swapping ps1-c7-eqiad...one side at a time...notifications pending 
[16:31:05] <akosiaris>	 yippi
[16:31:08] <morebots>	 Logged the message, Master
[16:31:32] <icinga-wm>	 PROBLEM - Host analytics1014 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:31:33] <akosiaris>	 heh... the ganglia_new module is so new that it uses tabs 
[16:31:42] <icinga-wm>	 PROBLEM - Host analytics1011 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:32:22] <icinga-wm>	 RECOVERY - Host analytics1011 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms  
[16:32:37] <grrrit-wm>	 (03PS17) 10Ottomata: (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[16:33:22] <icinga-wm>	 RECOVERY - Host analytics1014 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms  
[16:34:02] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[16:34:52] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.003 second response time  
[16:40:07] <grrrit-wm>	 (03CR) 10Edenhill: "(1 comment)" [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[16:44:30] <ottomata>	 Snaps:  i just set it to the librdkafka defaults 
[16:45:30] <Snaps>	 ottomata: ah, okay. I think its better to comment them out then so that librdkafka defaults will propogate properply. But leave them commented out to indicate what might be worth fiddling with. ?
[16:46:19] <ottomata>	 yeah that is good
[16:47:11] <ottomata>	 will do that for the other vals too
[16:49:23] <grrrit-wm>	 (03PS18) 10Ottomata: (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[16:52:48] <grrrit-wm>	 (03PS1) 10Manybubbles: CirrusSearch as secondary for nlwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94925 
[16:54:49] <grrrit-wm>	 (03PS1) 10Ottomata: Updating varnishkafka.conf with a few changes and using some actual librdkakfa defaults. [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/94927 
[16:55:11] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Updating varnishkafka.conf with a few changes and using some actual librdkakfa defaults. [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/94927 (owner: 10Ottomata)
[16:58:07] <grrrit-wm>	 (03PS4) 10Ottomata: Setting up varnishkafka on mobile varnish caches. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 
[16:58:39] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Setting up varnishkafka on mobile varnish caches. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 (owner: 10Ottomata)
[16:58:45] <grrrit-wm>	 (03CR) 10Edenhill: [C: 031] (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[17:01:22] <grrrit-wm>	 (03CR) 10Edenhill: [C: 031] Setting up varnishkafka on mobile varnish caches. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 (owner: 10Ottomata)
[17:01:52] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Use descriptive heredoc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94303 (owner: 10Chad)
[17:02:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use descriptive heredoc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94303 (owner: 10Chad)
[17:02:53] <^d>	 manybubbles: You sync'ing and running the scripts or am I? :)
[17:03:10] <manybubbles>	 ^d:  I'll sync and run if you'll +2
[17:03:17] <grrrit-wm>	 (03CR) 10Chad: [C: 032] CirrusSearch as secondary for nlwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94925 (owner: 10Manybubbles)
[17:03:26] <greg-g>	 uhhhh
[17:03:26] <manybubbles>	 ^d: starting
[17:03:27] <grrrit-wm>	 (03Merged) 10jenkins-bot: CirrusSearch as secondary for nlwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94925 (owner: 10Manybubbles)
[17:03:29] <greg-g>	 :)
[17:03:37] <greg-g>	 hi there :)
[17:03:40] <^d>	 howdy
[17:03:42] <manybubbles>	 greg-g: ?
[17:04:05] <greg-g>	 "sync and run" where I thought "run" meant "run away from the computer" ;)
[17:04:23] <^d>	 Deploy then flee!
[17:04:24] * apergos  snickers

[17:06:44] <grrrit-wm>	 (03PS19) 10Ottomata: (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[17:08:47] <grrrit-wm>	 (03PS5) 10Ottomata: Setting up varnishkafka on mobile varnish caches. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 
[17:09:35] <logmsgbot>	 !log manybubbles synchronized wmf-config/InitialiseSettings.php  'Enable Cirrus as secondary on nlwiki'
[17:09:49] <morebots>	 Logged the message, Master
[17:09:59] <grrrit-wm>	 (03CR) 10Edenhill: [C: 031] (WIP) Initial Debian version [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis)
[17:11:30] <grrrit-wm>	 (03CR) 10Edenhill: [C: 031] Setting up varnishkafka on mobile varnish caches. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 (owner: 10Ottomata)
[17:13:29] <grrrit-wm>	 (03PS1) 10Ottomata: Using interface::add_ip6_mapped on analytics Kafka brokers. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94933 
[17:14:13] <ottomata>	 paravoid: ^
[17:15:04] <icinga-wm>	 PROBLEM - Host ps1-c7-eqiad is DOWN: PING CRITICAL - Packet loss = 100%  
[17:18:30] <manybubbles>	 ^d: search index is under constrution
[17:18:36] <manybubbles>	 *construction
[17:18:59] <^d>	 mmk
[17:19:00] <greg-g>	 [[File:Construction_man_digging.gif]]
[17:19:45] <manybubbles>	 greg-g: my favorite.  also, I'm done syncing files.  just needed the one.
[17:20:21] <greg-g>	 manybubbles: thanks for the heads up
[17:25:54] <icinga-wm>	 PROBLEM - Host wtp1004 is DOWN: PING CRITICAL - Packet loss = 100%  
[17:30:34] <icinga-wm>	 RECOVERY - Host wtp1004 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms  
[17:31:21] <^d>	 greg-g: I think you wanted https://commons.wikimedia.org/wiki/File:Under_Construction.jpeg :)
[17:32:05] <^d>	 https://commons.wikimedia.org/wiki/File:Lavori.gif is pretty fun
[17:33:04] <icinga-wm>	 PROBLEM - Host wtp1016 is DOWN: PING CRITICAL - Packet loss = 100%  
[17:33:08] <greg-g>	 ^d: actually, no, that's not animated :)
[17:33:18] <^d>	 Lavori.gif is :)
[17:33:24] <greg-g>	 ooo, that is good!
[17:33:35] <^d>	 Hah
[17:33:36] <^d>	 https://commons.wikimedia.org/wiki/File:Enobras.gif
[17:38:37] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Enable mwsearch logs'
[17:38:43] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|I4baa719b9}}: CirrusSearch as secondary for nlwiki
[17:38:51] <morebots>	 Logged the message, Master
[17:38:55] <Reedy>	 Lies logmsgbot
[17:38:56] <Reedy>	 All lies
[17:39:04] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[17:39:04] <grrrit-wm>	 (03PS1) 10Reedy: Enable mwsearch logs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94934 
[17:39:05] <morebots>	 Logged the message, Master
[17:39:13] <^d>	 Livehacks are bad yo :p
[17:39:14] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable mwsearch logs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94934 (owner: 10Reedy)
[17:39:24] <^d>	 lobmsgbot just exposes livehackers :p
[17:39:26] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable mwsearch logs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94934 (owner: 10Reedy)
[17:40:04] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 5.036 second response time  
[17:42:19] <ottomata>	 cmjohnson1: how's the pdu? :)
[17:43:34] <grrrit-wm>	 (03PS2) 10Yurik: Revert "Revoke Yuri's shell access" and change pub key [operations/puppet] - 10https://gerrit.wikimedia.org/r/94780 
[17:44:42] <yurik>	 anyone around to +2 pls ^
[17:44:52] <yurik>	 i have no shell access in the mean time
[17:45:02] <yurik>	 can validate that its me on hangout
[17:46:40] * greg-g  can't

[17:47:21] <yurik>	 greg-g, ?
[17:48:28] <yurik>	 mark disabled it a week ago due to some possible security issues, but i'm not sure if he's around now to +2 - isn't he on vacation?
[17:48:44] <icinga-wm>	 PROBLEM - Host terbium is DOWN: PING CRITICAL - Packet loss = 100%  
[17:49:05] <yurik>	 paravoid ^
[17:49:19] <greg-g>	 I can't +2 in operations/
[17:49:50] <yurik>	 oh :)
[17:50:14] <icinga-wm>	 RECOVERY - Host terbium is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms  
[17:53:12] <cmjohnson1>	 ottomata: the new one is in...trying to get all the power correct. probably about 30 mins more to get power right...then some configuration is needed 
[17:54:05] <paravoid>	 cmjohnson1: I'm not sure if you saw/had time to reply me on list, how come this comes with downtime? don't those machines have redundant PSUs/supplies?
[17:54:16] <paravoid>	 and cables coming from different PDUs?
[17:54:30] <ksnider>	 I was wondering that too
[17:54:32] <logmsgbot>	 !log reedy synchronized php-1.23wmf2/extensions/MWSearch/
[17:54:45] <morebots>	 Logged the message, Master
[17:55:30] <logmsgbot>	 !log reedy synchronized php-1.23wmf3/extensions/MWSearch/
[17:55:40] <grrrit-wm>	 (03PS3) 10Ottomata: Revert "Revoke Yuri's shell access" and change pub key [operations/puppet] - 10https://gerrit.wikimedia.org/r/94780 (owner: 10Yurik)
[17:55:47] <morebots>	 Logged the message, Master
[17:55:50] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Revert "Revoke Yuri's shell access" and change pub key [operations/puppet] - 10https://gerrit.wikimedia.org/r/94780 (owner: 10Yurik)
[17:56:18] <yurik>	 ottomata, thx
[17:56:22] <ottomata>	 yurik^
[17:56:22] <ottomata>	 yup!
[17:56:39] <Aaron|home>	 did terbium restart?
[17:57:13] <Aaron|home>	 paravoid: ah, well I can manage :)
[17:57:19] <paravoid>	 ?
[17:57:31] <paravoid>	 what?
[17:57:39] <Aaron|home>	 17:57:23 up 7 min,  2 users,  load average: 0.65, 0.86, 0.52
[17:57:48] <paravoid>	 fallout from the PDU replacement?
[17:57:49] <paravoid>	 cmjohnson1: ?
[17:57:58] <paravoid>	 terbium is on C7, yes
[17:58:17] <paravoid>	 but this was unexpected
[17:58:30] <Aaron|home>	 anyway, that stash cleaning script finished a run
[17:58:32] <paravoid>	 we also have a couple of more important systems on that rack, e.g ytterbium, rdb1002
[17:58:46] <Aaron|home>	 deleted like 1.4 million objects
[17:58:48] <paravoid>	 Aaron|home: yup, I noticed, we're fully synced
[17:58:56] <paravoid>	 Aaron|home: I ran another thumb sync
[17:59:06] <Aaron|home>	 what about timeline-render and score-render?
[17:59:10] <paravoid>	 the diff now is 20G, which I think may just be "lost" files
[17:59:12] <paravoid>	 yeah, those too
[17:59:18] * Aaron|home  was going to make sure those were done today...ah good

[17:59:18] <paravoid>	 and transcoded
[17:59:25] <paravoid>	 feel free to double check :)
[18:00:42] <paravoid>	 ^wikipedia-commons-local-thumb.[0-9a-f]{2}$
[18:00:42] <paravoid>	 ^wikipedia-..-local-thumb.[0-9a-f]{2}$
[18:00:42] <paravoid>	 ^wik[a-z]+-.*-local-thumb$
[18:00:42] <paravoid>	 ^global-.*$
[18:00:42] <paravoid>	 ^.*-timeline-render$
[18:00:45] <paravoid>	 ^.*-transcoded(.[0-9a-f]{2})?$
[18:00:47] <paravoid>	 is what I have
[18:00:59] * Aaron|home  already did captcha

[18:01:12] <Aaron|home>	 well that's global-*
[18:01:40] <mark>	 !log Rebooting amssq60, stuck in xfs kmem_alloc deadlock
[18:01:48] <Aaron|home>	 should be fine then
[18:01:58] <morebots>	 Logged the message, Master
[18:02:32] <Aaron|home>	 paravoid: wait why is it 20G?
[18:02:49] <paravoid>	 my guess?
[18:03:10] <paravoid>	 files that are in the listings (and added to the filesize counts) but don't actually exist to be copied
[18:03:23] <paravoid>	 as sad as that sounds
[18:05:54] <Aaron|home>	 Skipped non-stash 8/81/11mpb7ab2ktg.x71daf.2463293..16
[18:06:06] <Aaron|home>	 still some files not deleted due to the regex not catching stuff
[18:06:18] <icinga-wm>	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 100,000  
[18:06:27] <Aaron|home>	 why are there so many different formats??
[18:09:27] <icinga-wm>	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[18:12:57] <icinga-wm>	 PROBLEM - Host analytics1017 is DOWN: PING CRITICAL - Packet loss = 100%  
[18:18:15] <grrrit-wm>	 (03PS2) 10Ottomata: Remove Kraken-specific varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/91492 (owner: 10Ori.livneh)
[18:18:21] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Remove Kraken-specific varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/91492 (owner: 10Ori.livneh)
[18:20:32] <grrrit-wm>	 (03CR) 10Ottomata: "Ran this after merging:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/91492 (owner: 10Ori.livneh)
[18:21:08] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Reduce threshold for flushing of dirty pages [operations/puppet] - 10https://gerrit.wikimedia.org/r/94943 
[18:22:21] <paravoid>	 mark: interesting.
[18:22:51] <mark>	 dunno if it will help, but worth a shot
[18:23:03] <mark>	 after 3 boxes with identical symptoms, i'm pretty sure it will happen again ;-)
[18:23:05] <grrrit-wm>	 (03PS2) 10Ottomata: Cleaning up analytics role classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/94148 
[18:23:10] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Cleaning up analytics role classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/94148 (owner: 10Ottomata)
[18:24:19] <grrrit-wm>	 (03PS2) 10Mark Bergsma: Reduce threshold for flushing of dirty pages [operations/puppet] - 10https://gerrit.wikimedia.org/r/94943 
[18:24:42] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032 V: 032] Reduce threshold for flushing of dirty pages [operations/puppet] - 10https://gerrit.wikimedia.org/r/94943 (owner: 10Mark Bergsma)
[18:27:24] <paravoid>	 ottomata: I don't think that e.g. the puppetmaster work is relevant to the SoS
[18:27:38] <paravoid>	 (that's you, right?)
[18:27:44] <ottomata>	 yeah probably not
[18:27:53] <ottomata>	 hmm
[18:34:07] <icinga-wm>	 RECOVERY - Host analytics1017 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms  
[18:34:12] <andrewbogott>	 ottomata: is it too late for me to send you a SoS update?
[18:34:21] <ottomata>	 naw we're in it now
[18:34:22] <ottomata>	 send away!
[18:34:38] <andrewbogott>	 Not that I've done anything that should matter to anyone anyway… come to think of it.
[18:35:19] <andrewbogott>	 so, nevermind!
[18:35:26] <grrrit-wm>	 (03CR) 10GWicke: "Ping ;)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/93527 (owner: 10Lcarr)
[18:35:57] <icinga-wm>	 PROBLEM - Host analytics1011 is DOWN: PING CRITICAL - Packet loss = 100%  
[18:38:27] <icinga-wm>	 RECOVERY - Host analytics1011 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms  
[18:39:03] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Reexec procps on changes [operations/puppet] - 10https://gerrit.wikimedia.org/r/94949 
[18:41:16] <grrrit-wm>	 (03CR) 10Mark Bergsma: "Yep, I'll have a version of this merged before the end of the week. ;)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/93527 (owner: 10Lcarr)
[18:52:26] <cmjohnson1>	 ottomata: the power is stable in ps1-c7-eqiad now..sill have to cfg the pdu but we have redundancy again
[18:52:48] <ottomata>	 cooool
[18:52:49] <ottomata>	 danke
[18:53:27] <paravoid>	 cmjohnson1: hey
[18:53:32] <paravoid>	 cmjohnson1: did you see my question above?
[18:53:48] <paravoid>	 19:54 < paravoid> cmjohnson1: I'm not sure if you saw/had time to reply me on list, how come this comes with downtime? don't those machines have redundant PSUs/supplies?
[18:53:52] <paravoid>	 19:54 < paravoid> and cables coming from different PDUs?
[18:53:57] <paravoid>	 (also, we lost terbium in the process)
[18:54:04] <paravoid>	 (but it's back up now)
[18:54:31] <cmjohnson1>	 the pdu lost one side so we didn't have redundancy 
[18:54:51] <paravoid>	 don't we have *two* PDUs?
[18:55:19] <cmjohnson1>	 we have a single pdu with 2 sides that are on separate receptacles
[18:55:35] <paravoid>	 okay
[18:55:38] <paravoid>	 and we lost redundancy
[18:55:43] <cmjohnson1>	 yes
[18:55:49] <paravoid>	 why did we lose the analytics boxes, though?
[18:56:28] <cmjohnson1>	 i don't know why we lost analytics boxes...if power was lost in the last couple of hours it happened during the pdu swap
[18:56:40] <cmjohnson1>	 example terbium
[18:56:50] <paravoid>	 no, I mean, don't these boxes have two power supplies?
[18:57:04] <cmjohnson1>	 yes..they do have 2 power supplies
[18:57:29] <cmjohnson1>	 each power supply is connected to it's own power source ...side A or side B of the pdu
[18:57:32] <paravoid>	 right
[18:57:41] <cmjohnson1>	 that way if we lose 1 side the other remains 
[18:57:48] <paravoid>	 so why did they poweroff, both today and the other day when the problem originally presneted?
[18:57:52] <cmjohnson1>	 i did not see any of them off
[18:58:21] <cmjohnson1>	 I don't know why any of would of powered off
[18:58:43] <apergos>	 we had reboots as I noted on the ticket
[18:59:21] <apergos>	 both then (multiple times) and today (multiple times)
[18:59:21] <cmjohnson1>	 apergos: i rebooted it on Friday 1x
[19:00:34] <cmjohnson1>	 something is up with that pdu...not sure what...I swapped it with one destined for row D so we'll either need to order another set of the new ones if see if we can the broken one replaced. 
[19:01:03] <paravoid>	 cmjohnson1: there's a mail thread on the list about this btw :)
[19:01:26] <cmjohnson1>	 due to the fact we bought through dell and servertech couldn't find a match on the S/N swapping was the fastest and best thing to do
[19:01:48] <cmjohnson1>	 paravoid: i read a bit of this morning. once i finish I will go through and comment 
[19:02:11] <paravoid>	 k, thanks :)
[19:04:12] <ottomata>	 cmjohnson1: while you are here, this is kinda unrelated to the PDU problem, but I'd really like to do an analytics box reshuffle as soon as possible
[19:05:10] <ottomata>	 https://rt.wikimedia.org/Ticket/Display.html?id=6279
[19:05:44] <cmjohnson1>	 ottomata: ok...only issue is space at the moment.  Probably will need to wait for row D
[19:06:05] <cmjohnson1>	 in the meantime create a ticket and put it in queue
[19:08:18] <grrrit-wm>	 (03PS1) 10Dereckson: Undeploy SimpleAntiSpam extension. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94956 
[19:11:50] <icinga-wm>	 RECOVERY - Host wtp1016 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms  
[19:17:08] <ottomata>	 which queue?
[19:17:12] <ottomata>	 cmjohnson1: 
[19:17:16] <ottomata>	 this is the ticket
[19:17:17] <ottomata>	 https://rt.wikimedia.org/Ticket/Display.html?id=6279
[19:18:03] <Reedy>	 !log Created EducationProgram tables on elwiki bug 56771
[19:18:21] <morebots>	 Logged the message, Master
[19:26:22] <cmjohnson1>	 oh..okay...sorry ottomata been busy didn't look for it
[19:30:01] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Undeploy SimpleAntiSpam extension. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94956 (owner: 10Dereckson)
[19:37:34] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Move android::sdk and packages::ant18 into contint module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94257 
[19:40:04] <logmsgbot>	 !log reedy synchronized php-1.23wmf3  'Support CIDR ranges in $wgSquidServersNoPurge'
[19:40:22] <morebots>	 Logged the message, Master
[19:44:12] <logmsgbot>	 !log reedy synchronized php-1.23wmf2  'Support CIDR ranges in $wgSquidServersNoPurge'
[19:44:26] <morebots>	 Logged the message, Master
[19:48:11] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|I1bb030fce}}: Enable mwsearch logs
[19:48:18] <grrrit-wm>	 (03PS1) 10Reedy: Enable EducationProgram on elwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94964 
[19:48:26] <morebots>	 Logged the message, Master
[19:48:27] <Reedy>	 ori-l: That's starting to get slightly annoying as it's wrong...
[19:48:37] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Enable EducationProgram on elwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94964 (owner: 10Reedy)
[19:49:02] <ori-l>	 Reedy: it's wrong? did you file a bug?
[19:49:36] <Reedy>	 Bug for what? It being wrong?
[19:49:40] <icinga-wm>	 RECOVERY - Host ps1-c7-eqiad is UP: PING OK - Packet loss = 0%, RTA = 2.12 ms  
[19:51:49] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] fix pdf servers in dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/94307 (owner: 10Dzahn)
[19:53:40] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable EducationProgram on elwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94964 (owner: 10Reedy)
[19:53:47] <Reedy>	 Oh sod off
[19:54:16] <grrrit-wm>	 (03CR) 10Reedy: [V: 032] Enable EducationProgram on elwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94964 (owner: 10Reedy)
[19:55:30] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Iab47779a2c0f9fe239676d75a279336156353c4b'
[19:55:48] <morebots>	 Logged the message, Master
[19:56:17] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] Move android::sdk and packages::ant18 into contint module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94257 (owner: 10Andrew Bogott)
[19:57:56] <MatmaRex>	 lol Reedy.
[20:01:08] <mark>	 so, why is application server cpu up 30% since 19:50? :)
[20:01:51] <Reedy>	 Uhh
[20:02:17] <Reedy>	 EducationProgram isn't that bad is it?
[20:02:26] <greg-g>	 mark: that isn't due to your varnish stuff, right?
[20:02:45] <mark>	 shouldn't be, I didn't do anything ;)
[20:02:49] <Reedy>	 [all] [test2] [1.22wmf19] [1.22wmf18] [stats/all] [1.23wmf2] [1.23wmf3] [stats/1.22wmf13] [1.23wmf1] [stats/1.22wmf19] [stats/1.22wmf18] [thumb-1.23wmf1] [thumb-1.23wmf3] [thumb-1.23wmf2] [thumb-1.22wmf22] [thumb-1.22wmf21] [thumb-1.22wmf20] [1.22wmf22] [1.22wmf20] [1.22wmf21] [thumb-1.22wmf18] [thumb-1.22wmf19] [stats/1.22wmf20] [stats/1.22wmf21] [stats/1.22wmf22] [stats/1.23wmf1] [stats/1.23wmf2] [stats/1.23wmf3] [ showing 50 
[20:02:49] <Reedy>	 events, show more ]
[20:02:56] <Reedy>	 helpful profiling is helpful
[20:03:25] <greg-g>	 mark: oh, thought you said there'd be an increase in api server usage due to reduced cache hit for loginwiki
[20:03:26] <logmsgbot>	 !log reedy cleared profiling data
[20:03:41] <morebots>	 Logged the message, Master
[20:03:43] <mark>	 greg-g: that was yesterday, and as of this afternoon that does get cached
[20:03:48] <mark>	 didn't make much difference
[20:03:56] <greg-g>	 ah, cool, sorry to interrupt then :)
[20:04:19] <mark>	 it had actually been uncached by varnish for a long time, but squid did cache
[20:04:28] <mark>	 so should be the same on varnish now
[20:04:38] <mark>	 hopefully.
[20:04:39] <grrrit-wm>	 (03PS5) 10Dzahn: (bug 56412) Make all sidebar phrases on Planet translatable [operations/puppet] - 10https://gerrit.wikimedia.org/r/94433 (owner: 10Odder)
[20:04:48] <mark>	 varnish is a bit liberal with caching headers sometimes
[20:05:30] <RoanKattouw>	 When you say liberal, what does that mean exactly?
[20:06:14] <mark>	 that it ignores some RFC 2616 stuff because it considers itself not a http cache "in the traditional sense"
[20:06:23] <mark>	 and as such it used to ignore no-cache, private
[20:06:28] <mark>	 until we configured it to not ignore in vcl
[20:06:33] <mark>	 i think it's fine now
[20:07:45] <RoanKattouw>	 Oh, right, that
[20:07:54] <RoanKattouw>	 Cache-Control: no-cache in the *request* header, right?
[20:07:58] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] (bug 56412) Make all sidebar phrases on Planet translatable [operations/puppet] - 10https://gerrit.wikimedia.org/r/94433 (owner: 10Odder)
[20:08:17] <RoanKattouw>	 I think I once cited that as an example of an HTTP spec I anecdotally knew was being ignored by caching software for DoS protection
[20:08:22] <mark>	 RoanKattouw: nope, response
[20:08:29] <mark>	 also in the request yes
[20:08:29] <RoanKattouw>	 wtf
[20:08:41] <RoanKattouw>	 Well in the request it should ignore it
[20:08:48] <RoanKattouw>	 But in the response it absolutely should not
[20:09:01] <Reedy>	 clearing profiling data is taking an age
[20:09:38] <mutante>	 ori-l: do you know this one on puppet-merge? error: cannot open /var/lib/git/operations/puppet/.git/modules/modules/kafka/FETCH_HEAD: Permission denied
[20:09:40] <mark>	 RoanKattouw: yeah, so we're now taking care of that in the wikimedia.vcl
[20:09:51] <mutante>	 ori-l: it just happened once but next time it worked normal again
[20:10:52] <yurik>	 has puppet ran in the past few hours on the bast1001? I can't login with the new key
[20:13:18] <jeremyb>	 yurik: i thought only some fraction of puppet runs do keys?
[20:13:23] <RoanKattouw>	 Excellent
[20:14:11] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Move android::sdk and packages::ant18 into contint module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/94257 (owner: 10Andrew Bogott)
[20:14:43] <logmsgbot>	 !log reedy cleared profiling data
[20:14:45] <apergos>	 yes it has run recently
[20:14:48] <Reedy>	 That's just a lie
[20:15:00] <apergos>	 your key has been created over there
[20:15:15] <Reedy>	 ori-l: Done much with professor?
[20:15:26] <apergos>	 and only some fraction of runs do host keys
[20:15:51] <yurik>	 apergos, i am still getting pubkey error, and it merged 2+ hours ago
[20:16:18] <yurik>	 should i wait a bit longer?
[20:16:36] <mark>	 the answer to that is always: yes! :)
[20:16:53] <yurik>	 :)
[20:17:30] <apergos>	 well like I say it your key was added to authorized_keys over there
[20:17:58] <yurik>	 apergos, yes, but i still can't login :(
[20:18:27] <jeremyb>	 tail -f /var/log/auth.log :)
[20:18:28] <apergos>	 waiting isn't going to fix that
[20:18:45] <jeremyb>	 yurik: btw, your commit msg really sucks
[20:18:46] <yurik>	 yes, but if it will make mark happy...
[20:19:36] <yurik>	 jeremyb, :(
[20:20:04] <apergos>	 3FYSE2s=  is the last part of your correct public key right?
[20:20:07] <yurik>	 it was an automatic msg at first, and later i added that i regened the key
[20:20:52] <yurik>	 apergos, yes
[20:21:01] <apergos>	 double check your private key then
[20:21:20] <yurik>	 apergos, i am able to login into the bastion with the same key
[20:21:36] <apergos>	 into which bastion
[20:21:46] <yurik>	 labs
[20:21:46] <apergos>	 sorry but weren't we just talking about bast1001? 
[20:21:52] <yurik>	 correct
[20:21:53] <apergos>	 uh wait
[20:22:00] <apergos>	 you are using the same key in labs?
[20:22:15] <yurik>	 i added it to test, will remove once it works
[20:22:35] <yurik>	 i have two keys - one for labs, one for prod
[20:22:49] <yurik>	 labs is easier to manage - adding it via web interface
[20:24:33] <grrrit-wm>	 (03PS2) 10Reedy: (bug 56760) Update logo for Korean Wikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94413 (owner: 10Odder)
[20:24:40] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] (bug 56760) Update logo for Korean Wikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94413 (owner: 10Odder)
[20:26:13] <grrrit-wm>	 (03Merged) 10jenkins-bot: (bug 56760) Update logo for Korean Wikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94413 (owner: 10Odder)
[20:26:27] <apergos>	 well I have nothing more useful I can tell you from this end; the key is there, in the right file, with the right permissions, and the log tells me that you have a public key issue 
[20:28:23] <grrrit-wm>	 (03CR) 10Hashar: "Filled https://bugzilla.wikimedia.org/show_bug.cgi?id=56955 about it with a patch for wmerrors in https://gerrit.wikimedia.org/r/94978" [operations/puppet] - 10https://gerrit.wikimedia.org/r/93009 (owner: 10Tim Starling)
[20:28:31] <andrewbogott>	 !log deployed new AbortNewAccount hook on wikitech
[20:28:50] <morebots>	 Logged the message, Master
[20:29:26] <jeremyb>	 yurik: did you try with ssh -vvv -i ?
[20:30:07] <jeremyb>	 yurik: also, ssh-keygen -lf path/to/file/passed/to/-i
[20:30:22] <yurik>	 jeremyb, thx, checking
[20:31:13] <apergos>	 it might be worth doing that for your labs login too, to compare
[20:31:14] <grrrit-wm>	 (03PS2) 10Reedy: (bug 56761) Add shortcut for NS_PROJECT for kowiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94417 (owner: 10Odder)
[20:31:21] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] (bug 56761) Add shortcut for NS_PROJECT for kowiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94417 (owner: 10Odder)
[20:31:46] <yurik>	 apergos, jeremyb this is weird - i was able to login via SSH command, but not via putty, even though putty works to labs.
[20:32:21] <jeremyb>	 putty has a window where you can get more details
[20:32:21] <apergos>	 so first things first: note to yourself to test in future with ssh :-D
[20:32:44] <jeremyb>	 apergos: no, note to self to not use windows! :-D
[20:32:53] <matanya>	 yurik: in the putty windows check the setting under ssh section
[20:32:54] <yurik>	 heh, true - but on Windows one doesn't usually test that way as putty generally has a good level of trust
[20:33:09] <apergos>	 if I thought I could win that one, I'd give him the pen and paper for that note :-D
[20:33:20] <yurik>	 :)
[20:33:23] <apergos>	 putty has a key format issue or has had in the past
[20:33:43] <Reedy>	  Yeah
[20:33:50] <yurik>	 i would switch to linux in a heartbeat the moment it has 1) total commander (mc is nowhere near as powerful), and 2) winmerge
[20:34:05] <Reedy>	 It can do an openssh key or a ssh.com key
[20:34:11] <apergos>	 http://unix.stackexchange.com/questions/74545/what-difference-between-openssh-key-and-putty-key
[20:34:29] <jeremyb>	 Reedy: also it's own putty format?
[20:34:37] <matanya>	 yurik: to solve the key stuff mobaextrem might help you
[20:35:05] <matanya>	 * mobaxterm
[20:35:14] <Reedy>	 Yeah
[20:35:17] <Reedy>	 puttygen will fix it
[20:35:38] <matanya>	 as for the rest of the switching stuff, all i can say is blah
[20:36:17] <grrrit-wm>	 (03CR) 10Andrew Bogott: "If you have tested and validated this on labs, let me know and I will merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/90117 (owner: 10Nemo bis)
[20:36:28] <apergos>	 I'm gonna check out (basically I'm pretty checked out already, but now I'm officially non-productive) ... might peek in here and there but not be useful
[20:36:43] <apergos>	 you should be able to get it sorted now
[20:38:04] <grrrit-wm>	 (03CR) 10Nemo bis: "andrewbogott, how does one do that?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90117 (owner: 10Nemo bis)
[20:38:59] <andrewbogott>	 Nemo_bis, do you want me to talk you through it right now?
[20:39:20] * jeremyb  still has to point to putty's window with detailed logs. that should be able to tell you e.g. which key it's try to use.

[20:39:42] <matanya>	 andrewbogott: might i try and see if your training worked for me and i'll guide him? :)
[20:40:01] <andrewbogott>	 matanya: yeah, sounds good!  If he's around.
[20:40:12] * andrewbogott  adds Nemo_bis to puppet project

[20:40:47] <grrrit-wm>	 (03Merged) 10jenkins-bot: (bug 56761) Add shortcut for NS_PROJECT for kowiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94417 (owner: 10Odder)
[20:40:49] <andrewbogott>	 …maybe
[20:41:07] <grrrit-wm>	 (03CR) 10Matanya: "https://wikitech.wikimedia.org/wiki/Puppet_coding#labs_testing" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90117 (owner: 10Nemo bis)
[20:41:20] <matanya>	 at least for now.
[20:41:25] <yurik>	 ok, works now, i think its a weird bug in putty. jeremyb & apergos, & matanya, thank you for your help!!!
[20:42:03] <jeremyb>	 yurik: great.
[20:42:06] <matanya>	 yurik: now you can review my patch on the pywikibot :) 
[20:43:21] <grrrit-wm>	 (03CR) 10Nemo bis: "I'm not sure how that applies here: is there a cluster available for me to test on in labs, with a maintenance host for crontabs where to " [operations/puppet] - 10https://gerrit.wikimedia.org/r/90117 (owner: 10Nemo bis)
[20:43:31] <grrrit-wm>	 (03PS2) 10Reedy: (bug 56807) Localize logo for Welsh Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94438 (owner: 10Odder)
[20:43:37] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] (bug 56807) Localize logo for Welsh Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94438 (owner: 10Odder)
[20:43:38] <jeremyb>	 yurik: re msg: right. i'm thinking maybe you shouldn't use automatic msg if you're mixing in manual tweaks? or maybe the double negative confused me? ("revert revoke access" instead of just saying "restore access") also i think maybe i misread the first iteration of it yesterday
[20:45:12] <apergos>	 strictly speaking it's not a revert since it's a new key
[20:45:24] * apergos  whacks self for continuing to chime in in the work channel

[20:45:25] <apergos>	 gone!
[20:45:43] * yurik  shoots himself

[20:46:08] <jeremyb>	 yurik: please revert the shooting
[20:47:24] <yurik>	 jeremyb, only after https://gerrit.wikimedia.org/r/#/c/88261/ :)
[20:47:39] <yurik>	 otherwise dfoy & dr0ptp4kt won't let me stay alive
[20:53:03] <matanya>	 andrewbogott: regarding https://gerrit.wikimedia.org/r/#/c/90760/  I havn't found that .pep8 file, mind showing me where it might be?
[20:53:10] <grrrit-wm>	 (03CR) 10Reedy: [V: 032] (bug 56807) Localize logo for Welsh Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94438 (owner: 10Odder)
[20:53:55] <Nemo_bis>	 sigh https://gdash.wikimedia.org/dashboards/reqerror/
[20:54:01] <grrrit-wm>	 (03PS2) 10Reedy: (bug 56899) Extra NS for Collection on enwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94723 (owner: 10Odder)
[20:54:06] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] (bug 56899) Extra NS for Collection on enwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94723 (owner: 10Odder)
[20:56:41] <matanya>	 ksnider: hi, any news for me (rt access)? 
[20:57:29] <ksnider>	 matanya: Yes, actually - Legal literally got me the NDA 30 minutes ago. :)
[20:57:58] <matanya>	 ksnider: cool, thanks. great timing :)
[20:57:58] <andrewbogott>	 matanya, do you have the pep8 tool installed so you can tell what the failure is?
[20:58:14] <matanya>	 yes andrewbogott, longer than 80 car's
[20:58:26] <andrewbogott>	 Yeah, ok, probably fine to suppress that then.  Just a second...
[20:58:27] <grrrit-wm>	 (03CR) 10Reedy: [V: 032] (bug 56899) Extra NS for Collection on enwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94723 (owner: 10Odder)
[20:58:44] <Reedy>	 What's up with Jenkins tonight?
[20:58:53] <matanya>	 he is tired
[20:58:53] <jeremyb>	 andrewbogott: you can see the failure in the web UI
[20:58:55] <grrrit-wm>	 (03PS2) 10Reedy: Undeploy SimpleAntiSpam extension. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94956 (owner: 10Dereckson)
[20:59:07] <andrewbogott>	 matanya: cat puppet/files/mirror/.pep8  <- ?
[20:59:12] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Undeploy SimpleAntiSpam extension. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94956 (owner: 10Dereckson)
[20:59:55] <matanya>	 andrewbogott: damn, i'm stupid
[21:00:07] <andrewbogott>	 Well, it is hidden after all :)
[21:00:47] <grrrit-wm>	 (03CR) 10Reedy: [V: 032] Undeploy SimpleAntiSpam extension. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94956 (owner: 10Dereckson)
[21:02:03] <logmsgbot>	 !log reedy synchronized wmf-config/
[21:02:18] <morebots>	 Logged the message, Master
[21:04:05] <Reedy>	 !log Profiling data clearing seems to work, but echo "-truncate" | nc -q0 -u professor.pmtpa.wmnet 3811 doesn't return...
[21:04:22] <morebots>	 Logged the message, Master
[21:04:47] <jeremyb>	 Reedy: ori-l was doing /something/ to professor recently
[21:05:02] <Reedy>	 ori-l: How dare you.
[21:06:08] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1055 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:06:58] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend on cp1055 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 2.977 second response time  
[21:14:02] <ori-l>	 what's up with professor?
[21:14:07] <Aaron|home>	 Ryan_Lane: were is the code for the stuff you were working on?
[21:15:04] <ori-l>	 Aaron|home: he says it's checked in but that he can add you as a reviewer
[21:15:12] <ori-l>	 he says he also has a few things he needs to commit
[21:17:08] <icinga-wm>	 RECOVERY - Host analytics1012 is UP: PING OK - Packet loss = 0%, RTA = 1.34 ms  
[21:21:15] <logmsgbot>	 !log aaron synchronized php-1.23wmf3/maintenance/cleanupUploadStash.php  '6461a943ece41ac6914fc60ec16854f0591f0f2b'
[21:21:36] <morebots>	 Logged the message, Master
[21:30:57] <Reedy>	 ori-l: The clear-profile script seems to just hang. Stealing the netcat command and running it manually also hangs
[21:31:09] <Reedy>	 It used to only take a few seconds
[21:31:25] <Reedy>	 I thought originally it might've been because it had a lot of data (many mw versions etc)
[21:31:32] <Reedy>	 But running it again gives the same behaviour
[21:34:04] <logmsgbot>	 !log aaron cleared profiling data
[21:34:25] <morebots>	 Logged the message, Master
[21:40:54] <grrrit-wm>	 (03PS1) 10Matanya: Merge "Move android::sdk and packages::ant18 into contint module." into production [operations/puppet] - 10https://gerrit.wikimedia.org/r/95055 
[21:42:04] <grrrit-wm>	 (03Abandoned) 10Matanya: Merge "Move android::sdk and packages::ant18 into contint module." into production [operations/puppet] - 10https://gerrit.wikimedia.org/r/95055 (owner: 10Matanya)
[21:44:57] <hashar>	 andrewbogott: thank you for the ant18 modularization :-]
[21:45:51] <grrrit-wm>	 (03PS9) 10Matanya: download: convert into a module and clean up [operations/puppet] - 10https://gerrit.wikimedia.org/r/90760 
[21:48:19] <matanya>	 andrewbogott: ^
[21:48:48] <grrrit-wm>	 (03CR) 10Dzahn: "Odder, thanks for this. It works and is live now. -- Daniel" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94433 (owner: 10Odder)
[21:50:07] <awight>	 argh.  Need apache config handholding :( ... This directive is declared in one VirtualHost, but somehow affecting the other VirtualHost.  RewriteRule ^/$ /w/index.php
[21:50:58] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on amssq48 is CRITICAL: HTTP CRITICAL - No data received from host  
[21:51:08] <icinga-wm>	 PROBLEM - Varnish HTCP daemon on amssq48 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[21:51:28] <icinga-wm>	 PROBLEM - Varnish traffic logger on amssq48 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[21:52:04] <mutante>	 awight: where are you looking at that?
[21:52:31] <mutante>	 that's like the "Primary wiki redirector" in every wiki vhost
[21:53:09] <jeremyb>	 or what URL are you using as a test that's not behaving correctly?
[21:53:18] <awight>	 mutante: I'm hacking on mediawiki/vagrant, for the fundraising-tech team.  I've added a second server...
[21:53:48] <mutante>	 what are the ServerName's 
[21:53:55] <awight>	 fwiw, https://gerrit.wikimedia.org/r/#/c/94950/
[21:53:58] <mutante>	 since you say one affects the other
[21:54:01] <awight>	 the servernames are "devwiki" and "crm"
[21:54:15] <grrrit-wm>	 (03PS10) 10Matanya: download: convert into a module and clean up [operations/puppet] - 10https://gerrit.wikimedia.org/r/90760 
[21:54:28] <awight>	 apachectl -S gives me:
[21:54:29] <awight>	 wildcard NameVirtualHosts and _default_ servers:
[21:54:29] <awight>	 *:80                   is a NameVirtualHost default server mediawiki-vagrant.dev (/etc/apache2/sites-enabled/000-default:1) port 80 namevhost mediawiki-vagrant.dev (/etc/apache2/sites-enabled/000-default:1) port 80 namevhost crm (/etc/apache2/sites-enabled/crm:8) port 80 namevhost devwiki (/etc/apache2/sites-enabled/devwiki:8)
[21:55:37] <jeremyb>	 awight: errrrr, you could mention from the beginning that you're talking about vagrant :)
[21:55:43] <grrrit-wm>	 (03PS1) 10Cmjohnson: Removing cp1021-1036 mgmt ip [operations/dns] - 10https://gerrit.wikimedia.org/r/95059 
[21:55:52] <awight>	 hehe that would have closed the door on further help
[21:55:53] <mutante>	 eh, i don't really know about the vagrant part
[21:56:02] <mutante>	 but it sounds like you're talking to the default vhost
[21:56:03] <awight>	 well, these are just puppet rules
[21:56:06] <mutante>	 instead of the one you want
[21:56:16] <awight>	 nah, the default vhost is the standard apache uselessness
[21:56:32] <mutante>	 yea, we disable it in some puppet manifests
[21:56:32] <awight>	 I get that if the hostname is not in (crm,devwiki)
[21:56:48] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] Removing cp1021-1036 mgmt ip [operations/dns] - 10https://gerrit.wikimedia.org/r/95059 (owner: 10Cmjohnson)
[21:57:18] <mutante>	 but if you go to devwiki you get ?
[21:57:26] <cmjohnson1>	 !log dns update
[21:57:43] <morebots>	 Logged the message, Master
[21:57:49] <awight>	 mutante: http://devwiki/ is correct, but http://crm/ forwards to crm/w/index.php
[21:58:19] <mutante>	 awight: paste the whole sites-enabled/crm somewhere"
[22:00:53] <awight>	 mutante: http://paste2.org/aOJh51F0
[22:09:05] * awight  is looking for an apache flag to dump configuration lines in the order they are loaded

[22:12:01] <matanya>	 mutante: pm me please when you can
[22:14:26] <awight>	 self: mod_info reports that only devwiki has the rewrite rule.
[22:16:47] <mutante>	 awight: how about     curl -H 'Host: crm' localhost ?. also.. see any of these maybe in logs "   Could not reliably determine the server's fully qualified domain name" ?
[22:16:53] <mutante>	 matanya: ok, doi
[22:16:58] <mutante>	 ng so
[22:17:42] <awight>	 mutante: thanks, I think I'm narrowing it down to a Firefox problem !
[22:18:01] <awight>	 It doesn't like the hostname with a single component
[22:19:37] <awight>	 Yep.
[22:31:42] <grrrit-wm>	 (03PS1) 10Nemo bis: Use log scale for 5xx errors in "(cdn) HTTP Error Rate" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95064 
[22:36:28] <grrrit-wm>	 (03PS3) 10Ryan Lane: Add shadow_reference support to trebuchet [operations/puppet] - 10https://gerrit.wikimedia.org/r/94680 
[22:45:08] <grrrit-wm>	 (03PS1) 10Nemo bis: Also add 2 months and 1 year graphs in "(cdn) HTTP Error Rate" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95068 
[22:47:28] <logmsgbot>	 !log maxsem synchronized php-1.23wmf2/extensions/MobileFrontend/  'Fix fatal'
[22:47:43] <morebots>	 Logged the message, Master
[22:49:07] <grrrit-wm>	 (03PS5) 10Ryan Lane: Add mediawiki module for fetch and checkout hooks [operations/puppet] - 10https://gerrit.wikimedia.org/r/94682 
[22:50:20] <logmsgbot>	 !log maxsem synchronized php-1.23wmf3/extensions/MobileFrontend/  'Fix fatal'
[22:50:39] <morebots>	 Logged the message, Master
[22:51:17] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Task 1355: Enable the infobox experiment (story 1301) on enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93988 (owner: 10Jdlrobson)
[22:51:29] <grrrit-wm>	 (03Merged) 10jenkins-bot: Task 1355: Enable the infobox experiment (story 1301) on enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93988 (owner: 10Jdlrobson)
[22:53:54] <Eloquence>	 ksnider, LeslieCarr - FYI just added a notice to the RFP that the deadline has passed and we're now in the process of selecting a vendor. https://wikimediafoundation.org/w/index.php?title=RFP/2013_Datacenter&diff=94296&oldid=93993
[22:54:10] <ksnider>	 Eloquence: Thanks
[22:54:22] <grrrit-wm>	 (03CR) 10GWicke: [C: 031] Enable VisualEditor on board, collab, office and wikimaniateam wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93780 (owner: 10Jforrester)
[22:54:59] <grrrit-wm>	 (03CR) 10GWicke: "As discussed on IRC and logged on https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Parsoid was deployed last Thursday." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93780 (owner: 10Jforrester)
[22:55:00] <grrrit-wm>	 (03CR) 10Catrope: [C: 031] Create visualeditor-default.dblist to simplify config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94471 (owner: 10Jforrester)
[22:55:16] <logmsgbot>	 !log maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/93988/'
[22:55:27] <grrrit-wm>	 (03CR) 10Catrope: [C: 031] Enable VisualEditor on board, collab, office and wikimaniateam wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93780 (owner: 10Jforrester)
[22:55:29] <morebots>	 Logged the message, Master
[22:56:14] <grrrit-wm>	 (03CR) 10Catrope: [C: 04-2] "Do not deploy until December 2nd, when I'll have the time to actually monitor the fallout of this change (in theory there should be none, " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94472 (owner: 10Jforrester)
[22:56:21] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Disable slow UW queries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94404 (owner: 10MaxSem)
[22:56:30] <grrrit-wm>	 (03Merged) 10jenkins-bot: Disable slow UW queries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94404 (owner: 10MaxSem)
[22:58:48] <grrrit-wm>	 (03CR) 10Catrope: [C: 031] Make VisualEditor namespaces extend, not replace, default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94420 (owner: 10Jforrester)
[22:58:56] <grrrit-wm>	 (03PS6) 10Ryan Lane: Add recursive submodule support to trebuchet [operations/puppet] - 10https://gerrit.wikimedia.org/r/94688 
[22:59:01] <logmsgbot>	 !log maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/94404'
[22:59:17] <morebots>	 Logged the message, Master
[23:03:39] <grrrit-wm>	 (03PS2) 10Ryan Lane: Make appserver common a mediawiki deploy target [operations/puppet] - 10https://gerrit.wikimedia.org/r/94832 
[23:06:24] <aude>	 "Your cache administrator is nobody."
[23:06:34] <aude>	 https://git.wikimedia.org/ is not working :/
[23:06:38] <LeslieCarr>	 thanks Eloquence 
[23:08:20] <Reedy>	 ^d: ^^
[23:08:38] <Reedy>	 Error: 500, Internal Server Error at Tue, 12 Nov 2013 23:08:29 GMT
[23:08:55] <Reedy>	 aude: Looked at the link of nobody? :P
[23:09:57] <aude>	 heh
[23:10:07] <logmsgbot>	 !log maxsem Started syncing Wikimedia installation... : Demo deployment, no actual changes
[23:10:23] <morebots>	 Logged the message, Master
[23:10:42] <jeremyb>	 aude: root@wikimedia.dev.null
[23:10:48] <aude>	 :)
[23:17:40] <Ryan_Lane>	 !log upgrading wikitech to 1.23wmf3
[23:17:55] <morebots>	 Logged the message, Master
[23:18:00] <^d>	 Reedy: Restarting.
[23:18:54] <Ryan_Lane>	 RoanKattouw_away: done
[23:20:50] <^d>	 !log gitblit process had hung on antimony, restarted
[23:21:06] <morebots>	 Logged the message, Master
[23:21:23] <jeremyb>	 gitblit wfm
[23:23:26] <^d>	 jeremyb: Yeah cuz I restarted it :)
[23:23:45] <jeremyb>	 ^d: yeah, just confirming it's working :)
[23:24:17] <Reedy>	 aude: fixededed
[23:25:10] <grrrit-wm>	 (03PS7) 10Ryan Lane: Add recursive submodule support to trebuchet [operations/puppet] - 10https://gerrit.wikimedia.org/r/94688 
[23:25:47] <aude>	 Reedy: great
[23:25:52] <aude>	 thanks ^d
[23:25:54] <^d>	 yw
[23:26:10] <aude>	 errr, thanks nobody ;)
[23:28:51] <grrrit-wm>	 (03PS3) 10Ryan Lane: Make appserver common a mediawiki deploy target [operations/puppet] - 10https://gerrit.wikimedia.org/r/94832 
[23:31:37] <Ryan_Lane>	 !log upgrading wikitech-static to 1.23wmf3
[23:31:51] <morebots>	 Logged the message, Master
[23:33:06] <logmsgbot>	 !log maxsem Finished syncing Wikimedia installation... : Demo deployment, no actual changes
[23:33:25] <morebots>	 Logged the message, Master
[23:33:46] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Load ZeroRatedMobileAccess only where currently supported. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94250 (owner: 10Dr0ptp4kt)
[23:34:31] <grrrit-wm>	 (03Merged) 10jenkins-bot: Load ZeroRatedMobileAccess only where currently supported. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94250 (owner: 10Dr0ptp4kt)
[23:36:37] <logmsgbot>	 !log maxsem synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/94250/'
[23:36:51] <morebots>	 Logged the message, Master
[23:57:14] <Krinkle>	 This url is responding with a Wikimedia Tech error
[23:57:14] <Krinkle>	 https://git.wikimedia.org/blob/oojs%2Fcore.git
[23:57:21] <Krinkle>	 Forwarded for: 2001:980:a565:1:79ef:9a58:f00b:db94, 127.0.0.1
[23:57:21] <Krinkle>	 Error: 500, Internal Server Error at Tue, 12 Nov 2013 23:56:58 GMT
[23:57:36] <Krinkle>	 Request: GET http://git.wikimedia.org/blob/oojs%2Fcore.git, from 127.0.0.1 via cp1044 cp1044 ([127.0.0.1]:80), Varnish XID 1195859957 
[23:57:43] <RoanKattouw>	 That seems to be because gitblit doesn't know about that repo
[23:57:48] <RoanKattouw>	 It doesn't show up in the search for instance
[23:57:53] <Krinkle>	 https://git.wikimedia.org/summary/oojs%2Fcore.git
[23:57:57] <Krinkle>	 that url works fine
[23:58:03] <RoanKattouw>	 wtf
[23:58:14] <Krinkle>	 the url is bad, I provided blob without a file or branch
[23:58:22] <RoanKattouw>	 aha
[23:58:22] <Krinkle>	 btu it shouldn't fail like that 
[23:58:27] <RoanKattouw>	 That still shouldn't be a 500
[23:58:27] <Krinkle>	 gitblib has a 404 handler
[23:58:29] <RoanKattouw>	 Maybe a 400
[23:58:31] <Krinkle>	 indeed