[00:51:11] <icinga-wm>	 PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 308 seconds  
[00:51:26] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 323 seconds  
[00:53:26] <icinga-wm>	 RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds  
[00:53:37] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds  
[00:59:50] <icinga-wm>	 PROBLEM - puppet last run on cp1008 is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:17:54] <icinga-wm>	 RECOVERY - puppet last run on cp1008 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures  
[01:42:11] <grrrit-wm>	 (03CR) 10Springle: [C: 031] Add defines for working with mysql config files, and mysql client settings [puppet] - 10https://gerrit.wikimedia.org/r/169722 (owner: 10Ottomata)
[02:04:46] <icinga-wm>	 PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: puppet fail  
[02:24:57] <icinga-wm>	 RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures  
[04:12:06] <grrrit-wm>	 (03PS7) 10Springle: prepare dbproxy1002 for trials [puppet] - 10https://gerrit.wikimedia.org/r/164293 
[04:15:27] <grrrit-wm>	 (03PS8) 10Springle: prepare dbproxy1002 for trials [puppet] - 10https://gerrit.wikimedia.org/r/164293 
[04:16:53] <grrrit-wm>	 (03CR) 10Springle: [C: 032] prepare dbproxy1002 for trials [puppet] - 10https://gerrit.wikimedia.org/r/164293 (owner: 10Springle)
[04:44:27] <paravoid>	 springle: hey
[04:45:06] <paravoid>	 springle: is there any reason to keep "mariadb::monitor_replication" under role::mariadb::dbstore?
[04:46:18] <springle>	 paravoid: the alternative being?
[04:46:40] <springle>	 it's defined in the submodule
[04:46:59] <paravoid>	 isn't dbstore purposefully lagged?
[04:47:08] <springle>	 oh right
[04:47:12] <springle>	 one is, one isn't
[04:47:15] <paravoid>	 oh
[04:47:25] <springle>	 i have yet to split it up
[04:48:25] <paravoid>	 I see some support for increased thresholds
[04:48:39] <paravoid>	 but it probably doesn't work, icinga has been in a WARNING state for weeks now
[04:48:41] <springle>	 this is about the icinga warnings?
[04:49:10] <paravoid>	 yeah :)
[04:49:17] <paravoid>	 sorry, XY problem etc. :)
[04:49:18] <springle>	 the threshholds do work. the issue is that the current lag mechanism stops the slave, which makes seconds_behind_master null
[04:49:58] <springle>	 the check script then warns because sql thread is stopped and/or lag is null (ie, wrong, though it doesn't know why)
[04:50:23] <paravoid>	 I see a lot of that
[04:50:25] <paravoid>	 but I also see
[04:50:26] <paravoid>	 WARNING slave_sql_lag Seconds_Behind_Master: 95730 
[04:50:36] <paravoid>	 log_warn is 90000
[04:50:52] <paravoid>	 that's 2.5h of additional lag, right?
[04:51:00] <springle>	 yes
[04:51:21] <springle>	 yeah ok
[04:51:35] <springle>	 it shouldn't be doing that
[04:51:52] <paravoid>	 so is this a legitimate warning?
[04:52:34] <springle>	 with the thresholds we've set, yes. though i don't care unless it gets days behind :)
[04:52:38] <springle>	 i'll sort it out
[04:53:18] <paravoid>	 we should probably fix the checks :)
[04:53:27] <springle>	 i'd rather fix the lag
[04:53:33] <paravoid>	 heh
[05:11:04] * springle adds $happy_paravoid flag to replag check script :P
[05:11:08] <paravoid>	 :P
[05:23:06] <grrrit-wm>	 (03PS1) 10Springle: Option to silence some Icinga warnings for MariaDB replication. [puppet] - 10https://gerrit.wikimedia.org/r/170653 
[05:29:39] <grrrit-wm>	 (03PS1) 10Springle: Class parameter to pass --no-warn-stopped to check_mariadb.pl [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170654 
[05:32:03] <grrrit-wm>	 (03PS2) 10Springle: Class parameter to pass --no-warn-stopped to check_mariadb.pl [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170654 
[05:37:39] <grrrit-wm>	 (03PS1) 10Springle: Fix incorrect role parameter. [puppet] - 10https://gerrit.wikimedia.org/r/170656 
[05:38:38] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Fix incorrect role parameter. [puppet] - 10https://gerrit.wikimedia.org/r/170656 (owner: 10Springle)
[05:55:43] <paravoid>	 \o/ \o/
[05:55:44] <paravoid>	 :)
[06:29:52] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:59] <icinga-wm>	 PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:22] <icinga-wm>	 PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:37:35] <paravoid>	 _joe__: icinga is full of HHVM-related alerts too btw
[06:42:19] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 031] "WFM" [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170654 (owner: 10Springle)
[06:42:29] <icinga-wm>	 PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:46:34] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures  
[06:48:04] <icinga-wm>	 RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures  
[06:49:19] <icinga-wm>	 RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures  
[06:56:49] <_joe__>	 paravoid: again?
[06:56:56] <paravoid>	 yeah
[06:57:12] <_joe__>	 sorry just woke up
[06:57:44] <paravoid>	 68 Matching Service Entries Displayed
[06:57:46] <paravoid>	 *sigh*
[06:57:51] <grrrit-wm>	 (03PS2) 10Springle: Option to silence some Icinga warnings for MariaDB replication. [puppet] - 10https://gerrit.wikimedia.org/r/170653 
[06:57:53] <_joe__>	 uh?
[06:57:59] <paravoid>	 not just hhvm
[06:58:03] <paravoid>	 it's just a sad picture
[06:58:08] <paravoid>	 lots of alerts
[06:58:40] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Option to silence some Icinga warnings for MariaDB replication. [puppet] - 10https://gerrit.wikimedia.org/r/170653 (owner: 10Springle)
[06:59:00] <_joe__>	 paravoid: hhvm I just see the last three installed servers that have graphite checks failing
[06:59:27] <_joe__>	 but that's because some changes must have broken the puppettization, clearly
[07:00:33] <icinga-wm>	 RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[07:01:01] <paravoid>	 I see: mw1029 (check_procs), mw1030/mw1031/mw1032 (busy threads/queue size), mw1189 (same)
[07:01:27] <icinga-wm>	 RECOVERY - DPKG on mw1113 is OK: All packages OK  
[07:01:40] <paravoid>	 and mw1163 nutcrackefr
[07:01:54] <icinga-wm>	 RECOVERY - nutcracker port on mw1163 is OK: TCP OK - 0.000 second response time on port 11212  
[07:02:08] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Class parameter to pass --no-warn-stopped to check_mariadb.pl [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170654 (owner: 10Springle)
[07:02:54] <_joe__>	 paravoid: yes the queue size ones are to investigate, but it's a problem with the check]
[07:04:15] <_joe__>	 and nutcracker was on a depooled host, but I don't see other alarms there
[07:05:11] <_joe__>	 anyways, I'll take a shower and see why the latest appservers have failing icinga checks for busy threads and queue size.
[07:05:18] <paravoid>	 :)
[07:05:48] <icinga-wm>	 PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: puppet fail  
[07:08:44] <grrrit-wm>	 (03PS1) 10Springle: dbproxy: use full path for included /etc/haproxy/conf.d config files [puppet] - 10https://gerrit.wikimedia.org/r/170662 
[07:12:11] <grrrit-wm>	 (03PS1) 10Springle: WIP dbproxy monitoring [puppet] - 10https://gerrit.wikimedia.org/r/170663 
[07:19:32] <grrrit-wm>	 (03PS1) 10Springle: update mariadb submodule [puppet] - 10https://gerrit.wikimedia.org/r/170664 
[07:20:06] <grrrit-wm>	 (03CR) 10Springle: [C: 032] update mariadb submodule [puppet] - 10https://gerrit.wikimedia.org/r/170664 (owner: 10Springle)
[07:22:56] <springle>	 oh heh too tricky for own good
[07:23:29] <icinga-wm>	 PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: puppet fail  
[07:24:18] <icinga-wm>	 PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: puppet fail  
[07:24:30] <icinga-wm>	 PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: puppet fail  
[07:24:59] <icinga-wm>	 PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: puppet fail  
[07:25:28] <icinga-wm>	 RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures  
[07:25:48] <grrrit-wm>	 (03PS1) 10Springle: Don't get tricky with reusing variable names. [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170665 
[07:25:49] <icinga-wm>	 PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: puppet fail  
[07:25:59] <icinga-wm>	 PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: puppet fail  
[07:26:28] <icinga-wm>	 PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail  
[07:27:37] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Don't get tricky with reusing variable names. [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170665 (owner: 10Springle)
[07:27:48] <icinga-wm>	 PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail  
[07:27:52] <icinga-wm>	 PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail  
[07:27:52] <icinga-wm>	 PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail  
[07:28:38] <icinga-wm>	 PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail  
[07:28:59] <icinga-wm>	 PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: puppet fail  
[07:29:31] <_joe__>	 eheh
[07:29:58] <icinga-wm>	 PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail  
[07:29:59] <icinga-wm>	 PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: puppet fail  
[07:30:24] <grrrit-wm>	 (03PS1) 10Springle: Unbreak puppet on DBs [puppet] - 10https://gerrit.wikimedia.org/r/170666 
[07:30:38] <icinga-wm>	 PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: puppet fail  
[07:30:39] <icinga-wm>	 PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail  
[07:30:47] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Unbreak puppet on DBs [puppet] - 10https://gerrit.wikimedia.org/r/170666 (owner: 10Springle)
[07:31:20] <icinga-wm>	 PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail  
[07:32:42] <icinga-wm>	 PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail  
[07:33:08] <icinga-wm>	 RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures  
[07:33:39] <icinga-wm>	 RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures  
[07:33:48] <icinga-wm>	 PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: puppet fail  
[07:33:59] <icinga-wm>	 PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: puppet fail  
[07:34:21] <icinga-wm>	 PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: puppet fail  
[07:41:49] <icinga-wm>	 RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures  
[07:43:28] <icinga-wm>	 RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures  
[07:43:38] <icinga-wm>	 RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures  
[07:44:09] <icinga-wm>	 RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[07:44:18] <icinga-wm>	 RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures  
[07:45:12] <icinga-wm>	 RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[07:46:09] <icinga-wm>	 RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures  
[07:46:29] <icinga-wm>	 RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures  
[07:46:39] <icinga-wm>	 RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures  
[07:46:39] <icinga-wm>	 RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures  
[07:47:38] <icinga-wm>	 RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures  
[07:48:29] <icinga-wm>	 RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures  
[07:48:30] <icinga-wm>	 RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures  
[07:48:39] <icinga-wm>	 RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures  
[07:50:09] <icinga-wm>	 RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures  
[07:50:09] <icinga-wm>	 RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[07:52:01] <icinga-wm>	 RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[07:52:19] <icinga-wm>	 RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures  
[07:52:38] <icinga-wm>	 RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures  
[07:53:08] <grrrit-wm>	 (03PS1) 10Springle: Disable Icinga warnings from dbstore1001 delayed slave. [puppet] - 10https://gerrit.wikimedia.org/r/170667 
[07:56:50] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Disable Icinga warnings from dbstore1001 delayed slave. [puppet] - 10https://gerrit.wikimedia.org/r/170667 (owner: 10Springle)
[08:05:37] <grrrit-wm>	 (03PS2) 10Springle: dbproxy: use full path for included /etc/haproxy/conf.d config files [puppet] - 10https://gerrit.wikimedia.org/r/170662 
[08:05:44] <grrrit-wm>	 (03CR) 10Springle: [C: 032] dbproxy: use full path for included /etc/haproxy/conf.d config files [puppet] - 10https://gerrit.wikimedia.org/r/170662 (owner: 10Springle)
[08:11:00] <springle>	 enough breaking stuff for today
[08:11:34] <_joe__>	 eww I hate collected resources in nagios
[08:14:36] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: HHVM: fix monitoring. [puppet] - 10https://gerrit.wikimedia.org/r/170060 
[08:14:47] <paravoid>	 trailing dot!
[08:14:48] <_joe__>	 paravoid: ^^ I plainly forgot to merge this :/
[08:17:24] <grrrit-wm>	 (03PS1) 10QChris: Install maven on analytics clients [puppet] - 10https://gerrit.wikimedia.org/r/170668 
[08:18:20] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] HHVM: fix monitoring. [puppet] - 10https://gerrit.wikimedia.org/r/170060 (owner: 10Giuseppe Lavagetto)
[08:22:06] <grrrit-wm>	 (03PS2) 10QChris: Install maven on analytics clients [puppet] - 10https://gerrit.wikimedia.org/r/170668 
[08:25:38] <_joe__>	 springle: are we using haproxy in prod now for databases?
[08:33:38] <grrrit-wm>	 (03CR) 10QChris: Install maven on analytics clients (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170668 (owner: 10QChris)
[08:40:04] <paravoid>	 what's the status with CirrusSearch?
[08:47:18] <andre__>	 last that I know is that Nik disabled insource:// support as some regex searches could bring it down
[08:47:23] <andre__>	 paravoid, ^
[08:47:38] <paravoid>	 oh, thanks
[09:01:43] <_joe__>	 paravoid: yes, it seems like there is a memleak in lucene that was triggered by regex searches, nik wrote an email I guess
[09:01:55] <_joe__>	 but cirrus is up and running, and lemme check the heap size there
[09:04:03] <_joe__>	 http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Elasticsearch%20cluster%20eqiad&h=elastic1001.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1415005330&v=8266078072&m=es_heap_used&vl=bytes&ti=es_heap_used&z=large looks like a few major GCs went on without issues durning the weekend
[09:04:37] <paravoid>	 I'm wondering why we got old-search pages during the weekend
[09:05:37] <paravoid>	 root@search1019:/a/search/log# ls -lh
[09:05:37] <paravoid>	 total 47G
[09:05:45] <paravoid>	 -rw-r--r-- 1 lsearch search 23M Oct 30 23:59 log.2014-10-30
[09:05:45] <paravoid>	 -rw-r--r-- 1 lsearch search 22G Oct 31 23:59 log.2014-10-31
[09:05:45] <paravoid>	 -rw-r--r-- 1 lsearch search 26G Nov  1 23:59 log.2014-11-01
[09:05:45] <paravoid>	 -rw-r--r-- 1 lsearch search 20M Nov  2 23:59 log.2014-11-02
[09:06:02] <paravoid>	 (/a 100% full)
[09:27:50] <_joe__>	 smells like java stacktraces
[10:21:37] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: "looks like we're duplicating work with https://gerrit.wikimedia.org/r/#/c/170300/ ?" [puppet] - 10https://gerrit.wikimedia.org/r/147487 (owner: 10Reedy)
[10:43:14] <icinga-wm>	 RECOVERY - Disk space on search1019 is OK: DISK OK  
[10:56:04] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: apt: do not download Translations [puppet] - 10https://gerrit.wikimedia.org/r/170678 
[10:56:11] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: hiera: mediawiki-based backend for labs (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/168984 (owner: 10Giuseppe Lavagetto)
[11:02:19] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: apt: do not download Translations [puppet] - 10https://gerrit.wikimedia.org/r/170678 (owner: 10Faidon Liambotis)
[11:03:41] <hoo>	 paravoid: Then do acquire::languages en?
[11:04:08] <paravoid>	 that's the default
[11:04:14] <paravoid>	 that's what we fetch now
[11:04:31] <paravoid>	 (well, unless you have a different LC_MESSAGES, which we don't)
[11:04:56] <hoo>	 ah ok, then I didn't get the point
[11:05:01] <hoo>	 thought we were fetching more stuff
[11:07:36] <grrrit-wm>	 (03PS3) 10Giuseppe Lavagetto: hiera: mediawiki-based backend for labs [puppet] - 10https://gerrit.wikimedia.org/r/168984 
[11:08:22] <_joe_>	 godog: care to take another look? this should be ready to go live on labs
[11:09:29] <_joe_>	 YuviPanda: is there any documentation of the yaml files for the instance projects in wikitech? We should probably have a link in the project page under "configure"
[11:09:53] <_joe_>	 and encourage people to use it instead of declaring top-scope variables whenever possible.
[11:10:06] <_joe_>	 well, once this is merged
[11:10:16] <godog>	 _joe_: sure, I should be able to get to it before end of today
[11:10:18] * YuviPanda waves
[11:10:20] <YuviPanda>	 _joe_: nope, there's no docs...
[11:10:22] <YuviPanda>	 true.
[11:10:24] <YuviPanda>	 true
[11:10:26] <YuviPanda>	 also I wonder if I should get +2 today :)
[11:10:37] <_joe_>	 YuviPanda: eheh right!
[11:10:43] <_joe_>	 are you officially ops now?
[11:10:47] <YuviPanda>	 _joe_: I think so
[11:10:59] <YuviPanda>	 _joe_: mark should send an email to wmfall at some point, I think
[11:12:33] <_joe_>	 YuviPanda: anyways, I'm going to merge this now in beta, do a few tests, then merge it fully
[11:12:41] <YuviPanda>	 _joe_: cool! :)
[11:13:26] <godog>	 hey YuviPanda 
[11:13:31] <YuviPanda>	 hi godog!
[11:14:03] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-2] "I'm not sure if I understand this." [puppet] - 10https://gerrit.wikimedia.org/r/145997 (https://bugzilla.wikimedia.org/67957) (owner: 10Ori.livneh)
[11:26:13] <akosiaris>	 !log disable puppet on labsdb1004, labsdb1005 for postgresql reinitialization
[11:26:24] <morebots>	 Logged the message, Master
[11:28:39] <_joe_>	 and of course the ruby-httpclient version we have on precise has a different api that the one on trusty
[11:28:47] * _joe_ sings in joy
[11:28:53] <akosiaris>	 ruby ftw!
[11:29:41] <_joe_>	  Error 400 on SERVER: undefined method `ssl_version=' for #<HTTPClient::SSLConfig:>
[11:30:11] <_joe_>	 (yes, ruby-httpclient uses ssl3 by default, in 2014)
[11:30:43] <YuviPanda>	 well, ruby
[11:31:54] <paravoid>	 hey YuviPanda 
[11:31:56] <paravoid>	 I have a question
[11:32:12] <paravoid>	 what's with labmon1001 and all the labs checks alerting us in production icinga?
[11:32:21] <paravoid>	 is this temporary until you set up shinken in labs?
[11:32:35] <paravoid>	 is it something you're aware of, in general?
[11:46:13] <YuviPanda>	 paravoid: hey!
[11:46:15] <YuviPanda>	 paravoid: yup, until shinken. which is fairly close, today or tomorrow
[11:50:36] <icinga-wm>	 PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: puppet fail  
[11:50:48] <paravoid>	 ok, fair enough
[11:53:53] <grrrit-wm>	 (03PS4) 10Giuseppe Lavagetto: hiera: mediawiki-based backend for labs [puppet] - 10https://gerrit.wikimedia.org/r/168984 
[11:55:03] <_joe_>	 the simple fact that I have to inspect the httpclient library to know how to set the SSL option I want, because subsequent versions of the library are mutually incompatible, says a lot.
[11:55:19] <paravoid>	 a recruiter is pestering me for a ruby position
[11:55:32] <paravoid>	 I didn't even respond to the first mail and I got a followup
[11:55:41] <paravoid>	 i can pass your name on if you're interested :P
[11:55:46] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: osm: Fix rsync monitoring [puppet] - 10https://gerrit.wikimedia.org/r/170681 
[11:55:55] <paravoid>	 "Exciting Ruby opportunities"
[11:56:15] <akosiaris>	 i am excited :-)
[11:56:50] <_joe_>	 paravoid: ahahahah
[11:57:23] <YuviPanda>	 FTR I'll probably work somewhat close to EST hours
[11:57:24] <_joe_>	 paravoid: I have a couple of stalkers too, but not for ruby shops.
[11:57:51] <_joe_>	 paravoid: I can give you a PHP stalker in exchange for the ruby one
[12:05:20] <akosiaris>	 he... so I need a scala stalker to beat you both ...
[12:05:40] <akosiaris>	 well something jvm like... clojure would do I suppose
[12:09:25] <icinga-wm>	 RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures  
[12:12:35] <icinga-wm>	 PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:19:15] <icinga-wm>	 RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures  
[12:26:04] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: labs: specify a admin user for postgres [puppet] - 10https://gerrit.wikimedia.org/r/170684 
[12:44:24] <grrrit-wm>	 (03PS5) 10Giuseppe Lavagetto: hiera: mediawiki-based backend for labs [puppet] - 10https://gerrit.wikimedia.org/r/168984 
[12:58:10] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] "Works well in beta." [puppet] - 10https://gerrit.wikimedia.org/r/168984 (owner: 10Giuseppe Lavagetto)
[13:01:11] <_joe_>	 YuviPanda: https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep works well
[13:03:33] <_joe_>	 mmmh virt1000 is a labs puppetmaster, right? It doesn't seem to work as expected (it gets the prod hiera config)
[13:04:50] <_joe_>	 and... I think I was the one doing the dumb choice
[13:08:28] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: puppetmaster: use the correct hiera config [puppet] - 10https://gerrit.wikimedia.org/r/170689 
[13:18:08] <akosiaris>	 !log depool wtp1024.eqiad.wmnet in preparation for reimaging to trusty
[13:18:13] <akosiaris>	 and let's see how this goes ...
[13:18:15] <morebots>	 Logged the message, Master
[13:19:03] <paravoid>	 \o/
[13:37:00] <_joe_>	 parsoid on trusty means a not-too-old node version, right?
[13:37:16] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] puppetmaster: use the correct hiera config [puppet] - 10https://gerrit.wikimedia.org/r/170689 (owner: 10Giuseppe Lavagetto)
[13:38:16] * _joe_ off to lunch
[13:49:44] <YuviPanda>	 _joe_: \o/!
[13:50:15] <_joe_>	 YuviPanda: and it works, which wasn't so obvious :P
[13:50:22] <_joe_>	 YuviPanda: we need docs and UI
[13:51:52] <YuviPanda>	 _joe_: yeah. On the wiki side it is just 'edit Hiera:<projectname>'. Only projectadmins can do it
[13:52:12] <_joe_>	 yes I have seen the code
[13:52:22] <_joe_>	 but it should be doable from the configure project link
[13:52:34] <_joe_>	 and docs for using labs should be updated accordingly
[13:53:29] <_joe_>	 I evaluate the emotional cost of writing 1 line of ruby as 10 lines of docs. So you owe me several hundreds lines of docs :P
[13:53:59] * _joe_ shamelessly tries to strongarm Yuvi 
[13:55:26] <icinga-wm>	 PROBLEM - Host wtp1024 is DOWN: CRITICAL - Plugin timed out after 15 seconds  
[13:56:17] <YuviPanda>	 _joe_: heh
[13:57:09] <YuviPanda>	 _joe_: I'll add a patch for a link from 'manage projects'
[13:57:54] <_joe_>	 YuviPanda: quite simply, as we move to hiera more and more, it would be crucial to have this at our disposal
[14:00:47] <icinga-wm>	 RECOVERY - Host wtp1024 is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms  
[14:06:18] <grrrit-wm>	 (03PS5) 10Rush: Make the 'real name' user profile field optional in phabricator [puppet] - 10https://gerrit.wikimedia.org/r/170433 (owner: 1020after4)
[14:07:52] <grrrit-wm>	 (03CR) 10Rush: [C: 032 V: 032] Make the 'real name' user profile field optional in phabricator [puppet] - 10https://gerrit.wikimedia.org/r/170433 (owner: 1020after4)
[14:11:51] <grrrit-wm>	 (03PS5) 10Rush: Change phab_update_tag script to remove library lock file [puppet] - 10https://gerrit.wikimedia.org/r/166406 (owner: 10Christopher Johnson (WMDE))
[14:36:00] <grrrit-wm>	 (03PS1) 10ArielGlenn: force daily rotation of nginx logs [puppet] - 10https://gerrit.wikimedia.org/r/170700 
[14:38:07] <akosiaris>	 !log repool wtp1024 with a weight of 1 instead of 15 for now
[14:38:15] <morebots>	 Logged the message, Master
[14:38:18] <akosiaris>	 !log wtp1024 re-installed as trusty
[14:38:24] <morebots>	 Logged the message, Master
[14:40:16] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] force daily rotation of nginx logs [puppet] - 10https://gerrit.wikimedia.org/r/170700 (owner: 10ArielGlenn)
[15:00:04] <jouncebot>	 manybubbles, anomie, ^d, marktraceur: Respected human, time to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141103T1500). Please do the needful.
[15:00:15] <manybubbles>	 uh, bot?
[15:00:30] <manybubbles>	 your timezone seems off
[15:00:32] <anomie>	 I think the bot didn't handle DST ending correctly
[15:00:42] <YuviPanda>	 hehe
[15:01:02] <manybubbles>	 neither did the wiki:  https://wikitech.wikimedia.org/wiki/Deployments#Monday.2C.C2.A0November.C2.A003
[15:01:10] <Reedy>	 It is 1500
[15:01:11] <manybubbles>	 oh no!
[15:01:24] <manybubbles>	 it just thinks I'm in UTC-5 instead of america/new york
[15:01:27] <manybubbles>	 I can fix that one
[15:07:48] * anomie fixes Deployments to use PST, but sees the module's "us_in_dst" function is still b0rken
[15:11:29] <marktraceur>	 anomie: Screw that, UTC or nothing
[15:12:59] <godog>	 yes please!
[15:15:37] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: puppetmaster: install ruby-httpclient in labs [puppet] - 10https://gerrit.wikimedia.org/r/170707 
[15:15:56] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] puppetmaster: install ruby-httpclient in labs [puppet] - 10https://gerrit.wikimedia.org/r/170707 (owner: 10Giuseppe Lavagetto)
[15:16:23] <marktraceur>	 Hm, a MatmaRex patch. But no MatmaRex in-channel.
[15:16:23] <marktraceur>	 I guess he has like 50 minutes to get in
[15:17:12] <Reedy>	 marktraceur: Fancy doing another easy one?
[15:19:28] <marktraceur>	 Reedy: I didn't claim yet...but put 'er in
[15:19:40] <Reedy>	 Oh, is it not window time yet?
[15:19:45] <grrrit-wm>	 (03PS3) 10Reedy: Add export-0.10.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170351 (https://bugzilla.wikimedia.org/72417) 
[15:19:56] <marktraceur>	 No, jouncebot is Wrong
[15:20:14] <marktraceur>	 It's only 07:00 in SF
[15:21:33] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: hiera: fix logging, caching of non-existence in mwyaml [puppet] - 10https://gerrit.wikimedia.org/r/170708 
[15:22:49] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] hiera: fix logging, caching of non-existence in mwyaml [puppet] - 10https://gerrit.wikimedia.org/r/170708 (owner: 10Giuseppe Lavagetto)
[15:49:15] <anomie>	 marktraceur: I took Matma's patch off this morning's SWAT since the master-branch patch isn't merged yet.
[15:49:34] <marktraceur>	 'kay
[15:54:33] <icinga-wm>	 PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail  
[16:00:04] <bd808>	 anomie: csteipp emailed saying he'd review Matma's patch this morning. We'll get it out post-swat.
[16:00:05] <jouncebot>	 manybubbles, anomie, ^d, marktraceur: Respected human, time to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141103T1600). Please do the needful.
[16:00:39] <marktraceur>	 Better.
[16:00:49] <marktraceur>	 Who's doing it?
[16:00:58] <anomie>	 marktraceur: You? ;)
[16:01:06] <anomie>	 bd808: Or it could be done in the 16:00 PST slot
[16:01:37] <anomie>	 marktraceur: I can do it if you don't want to
[16:01:54] <marktraceur>	 anomie: I kinda don't
[16:02:15] * anomie starts SWAT
[16:02:22] <grrrit-wm>	 (03CR) 10Anomie: [C: 032] Add export-0.10.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170351 (https://bugzilla.wikimedia.org/72417) (owner: 10Reedy)
[16:02:34] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add export-0.10.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170351 (https://bugzilla.wikimedia.org/72417) (owner: 10Reedy)
[16:02:42] <Reedy>	 anomie: should be good with a sync-docroot :)
[16:03:01] <logmsgbot>	 !log anomie Synchronized docroot and w: (no message) (duration: 00m 10s)
[16:03:09] <morebots>	 Logged the message, Master
[16:03:10] <anomie>	 Reedy: I didn't know about sync-docroot, thanks. I'd have just done sync-dir docroot/mediawiki/xml
[16:03:18] <anomie>	 Reedy: ^^^ Test please
[16:03:29] <Reedy>	 https://www.mediawiki.org/xml/export-0.10.xsd
[16:03:30] <Reedy>	 LGTM
[16:03:37] * anomie is done with SWAT
[16:03:40] <Reedy>	 https://www.mediawiki.org/xml/
[16:03:41] <Reedy>	 Thanks
[16:06:37] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] osm: Fix rsync monitoring [puppet] - 10https://gerrit.wikimedia.org/r/170681 (owner: 10Alexandros Kosiaris)
[16:06:47] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] labs: specify a admin user for postgres [puppet] - 10https://gerrit.wikimedia.org/r/170684 (owner: 10Alexandros Kosiaris)
[16:07:11] <greg-g>	 oo, Reedy had something swat'd out?
[16:07:29] <Reedy>	 heh
[16:08:07] <greg-g>	 :)
[16:08:13] <icinga-wm>	 PROBLEM - Host ms-be2007 is DOWN: PING CRITICAL - Packet loss = 100%  
[16:11:05] <grrrit-wm>	 (03CR) 10ArielGlenn: "so besides using copytruncate for now, since the api cluster at least with hhvm is a bit touchy, we should also toss 'notifempty', to forc" [puppet] - 10https://gerrit.wikimedia.org/r/130296 (owner: 10ArielGlenn)
[16:13:03] <icinga-wm>	 RECOVERY - Host ms-be2007 is UP: PING OK - Packet loss = 0%, RTA = 43.61 ms  
[16:13:03] <icinga-wm>	 RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures  
[16:17:13] <icinga-wm>	 PROBLEM - swift-account-replicator on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-replicator  
[16:17:15] <icinga-wm>	 PROBLEM - swift-object-auditor on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor  
[16:17:21] <marktraceur>	 Oh, I was going to ask, why isn't MatmaRex just deploying his own crap
[16:17:22] <marktraceur>	 Surely now we're paying him (right?) we can trust him with the servers
[16:17:23] <icinga-wm>	 PROBLEM - swift-object-replicator on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-replicator  
[16:17:24] <icinga-wm>	 PROBLEM - swift-account-server on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-server  
[16:17:24] <icinga-wm>	 PROBLEM - swift-container-server on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server  
[16:17:37] <icinga-wm>	 PROBLEM - swift-object-server on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server  
[16:17:37] <icinga-wm>	 PROBLEM - swift-account-auditor on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-auditor  
[16:17:43] <icinga-wm>	 PROBLEM - swift-container-updater on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-updater  
[16:17:53] <icinga-wm>	 PROBLEM - swift-account-reaper on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-reaper  
[16:17:59] <icinga-wm>	 PROBLEM - swift-container-auditor on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor  
[16:17:59] <icinga-wm>	 PROBLEM - swift-object-updater on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-updater  
[16:17:59] <icinga-wm>	 PROBLEM - swift-container-replicator on ms-be2007 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-replicator  
[16:18:04] <godog>	 that's me
[16:18:47] <paravoid>	 yeah I figured :)
[16:21:38] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Added Yuvi to 'roots' [puppet] - 10https://gerrit.wikimedia.org/r/170717 
[16:21:50] <andrewbogott>	 mark:  +1?  ^
[16:22:41] <grrrit-wm>	 (03PS2) 10Krinkle: contint: Minor clean up [puppet] - 10https://gerrit.wikimedia.org/r/168629 
[16:22:48] <andrewbogott>	 !log added yuvi to 'Ops' ldap group
[16:22:55] <morebots>	 Logged the message, Master
[16:24:38] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Add Yuvi to 'roots' [puppet] - 10https://gerrit.wikimedia.org/r/170717 
[16:25:08] <grrrit-wm>	 (03PS3) 10Krinkle: contint: Clean up order of statements [puppet] - 10https://gerrit.wikimedia.org/r/168629 
[16:25:32] <grrrit-wm>	 (03CR) 10Krinkle: contint: Clean up order of statements (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/168629 (owner: 10Krinkle)
[16:25:41] <grrrit-wm>	 (03PS3) 10Krinkle: contint: Move /srv/localhost/qunit resource out of qunit_localhost class [puppet] - 10https://gerrit.wikimedia.org/r/168630 
[16:25:47] <grrrit-wm>	 (03PS3) 10Krinkle: [WIP] contint: Apply contint::qunit_localhost to labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/168631 
[16:25:48] <godog>	 !log reboot ms-be2007, disk replaced but no corresponding raid0 LD
[16:25:54] <morebots>	 Logged the message, Master
[16:27:18] <andrewbogott>	 springle: if you are working late tonight, can you have a look at https://bugzilla.wikimedia.org/show_bug.cgi?id=72908 ?
[16:28:41] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 1 [puppet] - 10https://gerrit.wikimedia.org/r/170718 
[16:28:43] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/170719 
[16:28:45] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 3 [puppet] - 10https://gerrit.wikimedia.org/r/170720 
[16:28:47] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/170721 
[16:28:54] <icinga-wm>	 RECOVERY - swift-container-updater on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-updater  
[16:28:54] <icinga-wm>	 RECOVERY - swift-account-auditor on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-auditor  
[16:28:54] <icinga-wm>	 RECOVERY - swift-object-server on ms-be2007 is OK: PROCS OK: 101 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server  
[16:29:03] <icinga-wm>	 RECOVERY - swift-account-reaper on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper  
[16:29:03] <icinga-wm>	 RECOVERY - swift-container-auditor on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor  
[16:29:03] <icinga-wm>	 RECOVERY - swift-object-updater on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-updater  
[16:29:04] <icinga-wm>	 RECOVERY - swift-container-replicator on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-replicator  
[16:29:10] <_joe_>	 don't worry, I left the interesting parts behind for now.
[16:29:33] <icinga-wm>	 RECOVERY - swift-account-replicator on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-replicator  
[16:29:45] <icinga-wm>	 RECOVERY - swift-object-auditor on ms-be2007 is OK: PROCS OK: 3 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor  
[16:29:46] <icinga-wm>	 RECOVERY - swift-object-replicator on ms-be2007 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator  
[16:29:46] <icinga-wm>	 RECOVERY - swift-account-server on ms-be2007 is OK: PROCS OK: 25 processes with regex args ^/usr/bin/python /usr/bin/swift-account-server  
[16:29:47] <icinga-wm>	 RECOVERY - swift-container-server on ms-be2007 is OK: PROCS OK: 25 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server  
[16:29:50] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 1 [puppet] - 10https://gerrit.wikimedia.org/r/170718 
[16:30:58] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] monitoring: move stuff out of nagios.pp, part 1 [puppet] - 10https://gerrit.wikimedia.org/r/170718 (owner: 10Giuseppe Lavagetto)
[16:31:34] <icinga-wm>	 RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures  
[16:34:03] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/170719 
[16:34:53] <_joe_>	 !log rolling-restarting hhvm appservers
[16:34:58] <morebots>	 Logged the message, Master
[16:38:19] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] monitoring: move stuff out of nagios.pp, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/170719 (owner: 10Giuseppe Lavagetto)
[16:39:33] <icinga-wm>	 PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [90.0]  
[16:40:04] <_joe_>	 it works, nice.
[16:41:06] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 3 [puppet] - 10https://gerrit.wikimedia.org/r/170720 
[16:41:15] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] monitoring: move stuff out of nagios.pp, part 3 [puppet] - 10https://gerrit.wikimedia.org/r/170720 (owner: 10Giuseppe Lavagetto)
[16:47:32] <grrrit-wm>	 (03CR) 10BBlack: [C: 031] remove en2.wikipedia.org [dns] - 10https://gerrit.wikimedia.org/r/170138 (owner: 10Dzahn)
[16:49:21] <grrrit-wm>	 (03CR) 10BBlack: [C: 031] Switch *.{wap,mobile}.wikipedia.org to wikipedia-lb [dns] - 10https://gerrit.wikimedia.org/r/98055 (owner: 10Faidon Liambotis)
[16:50:38] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: monitoring: move stuff out of nagios.pp, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/170721 
[16:53:45] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] monitoring: move stuff out of nagios.pp, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/170721 (owner: 10Giuseppe Lavagetto)
[16:54:52] <icinga-wm>	 RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [60.0]  
[16:57:25] <akosiaris>	 !log repool wtp1024 at regular weight
[16:57:32] <morebots>	 Logged the message, Master
[16:59:44] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Add Yuvi to 'roots' [puppet] - 10https://gerrit.wikimedia.org/r/170717 
[17:06:44] <grrrit-wm>	 (03PS5) 10ArielGlenn: script to monitor, clean up salt keys of deleted labs instances [puppet] - 10https://gerrit.wikimedia.org/r/168601 
[17:07:35] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] script to monitor, clean up salt keys of deleted labs instances [puppet] - 10https://gerrit.wikimedia.org/r/168601 (owner: 10ArielGlenn)
[17:10:40] <grrrit-wm>	 (03PS4) 10Krinkle: [WIP] contint: Apply contint::qunit_localhost to labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/168631 
[17:11:13] <Nemo_bis>	 hello bblack 
[17:11:16] <Nemo_bis>	 Did you see https://bugzilla.wikimedia.org/show_bug.cgi?id=72856
[17:17:19] * _joe_ off until ops meeting
[17:18:35] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: monitoring: convert monitor_group to monitoring::group [puppet] - 10https://gerrit.wikimedia.org/r/170727 
[17:21:09] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Add class and role for Openstack Horizon [puppet] - 10https://gerrit.wikimedia.org/r/170340 
[17:21:22] <YuviPanda>	 andrewbogott: ^ yay
[17:22:38] <andrewbogott>	 YuviPanda: I need to figure out if it will coexist with mediawiki… and also we should discuss 2fa during the Ops meeting today
[17:23:02] <YuviPanda>	 andrewbogott: yeah. I haven't touched it at all yet, though
[17:41:50] <grrrit-wm>	 (03PS2) 10Yuvipanda: shinken: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/168133 
[17:42:45] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] shinken: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/168133 (owner: 10Yuvipanda)
[17:42:51] <YuviPanda>	 ^ yay :)
[17:45:56] <grrrit-wm>	 (03PS1) 10RobH: setting mgmt ips for ms-be2013-2015 [dns] - 10https://gerrit.wikimedia.org/r/170736 
[17:47:01] <Reedy>	 YuviPanda: zomg, h4x
[17:52:34] <grrrit-wm>	 (03CR) 10RobH: [C: 032] setting mgmt ips for ms-be2013-2015 [dns] - 10https://gerrit.wikimedia.org/r/170736 (owner: 10RobH)
[17:56:19] <grrrit-wm>	 (03PS2) 10Dereckson: Improving comments for wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170526 
[17:58:14] <grrrit-wm>	 (03Draft2) 10Dereckson: Improving comments for wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170526 
[17:58:30] <grrrit-wm>	 (03CR) 10Dereckson: "PS2: ebased" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170526 (owner: 10Dereckson)
[17:59:11] <YuviPanda>	 Reedy: heh :)
[18:00:46] <grrrit-wm>	 (03PS1) 10Dereckson: Adding tools.wikimedia.pl to Commons wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170740 (https://bugzilla.wikimedia.org/72897) 
[18:08:12] <icinga-wm>	 PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail  
[18:22:31] <grrrit-wm>	 (03PS1) 10Gilles: Enable JPG thumbnail chaining on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170747 (https://bugzilla.wikimedia.org/67525) 
[18:24:59] <grrrit-wm>	 (03PS1) 10Dzahn: svn - move certificate installation into role [puppet] - 10https://gerrit.wikimedia.org/r/170748 
[18:26:31] <icinga-wm>	 RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[18:27:23] <grrrit-wm>	 (03PS1) 10Dzahn: gitblit - move certificate installation into role [puppet] - 10https://gerrit.wikimedia.org/r/170751 
[18:31:26] <grrrit-wm>	 (03PS1) 10Dzahn: svn - move from antimony to zirconium [puppet] - 10https://gerrit.wikimedia.org/r/170752 
[18:32:14] <MaxSem>	 mutante, why zirconium instead of /dev/null ? :P
[18:33:03] <mutante>	 because we want all the history
[18:37:18] <grrrit-wm>	 (03CR) 10Manybubbles: [C: 031] Adjust number of content shards for largest wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170173 (owner: 10Chad)
[18:37:27] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Include the vendor repo in mediawiki checkouts. [puppet] - 10https://gerrit.wikimedia.org/r/170755 
[18:38:32] <grrrit-wm>	 (03CR) 10Manybubbles: [C: 04-1] "iirc we made it so you had to specify each of the index types of you use the array for of replica count specification. Right?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170182 (owner: 10Chad)
[18:39:16] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Include the vendor repo in mediawiki checkouts. [puppet] - 10https://gerrit.wikimedia.org/r/170755 (owner: 10Andrew Bogott)
[18:45:22] <MaxSem>	 Reedy, does an extension needs to be enabled on testwiki for localisation to be built properly these days?
[18:45:40] <Reedy>	 Um
[18:45:44] <Reedy>	 Definitely needs to be in extension-list
[18:46:48] <MaxSem>	 I remember we had this bug before, but not sure what the status of it
[18:47:13] <grrrit-wm>	 (03CR) 10Chad: "I know, that's why the overrides are posted using the + syntax so the config will merge the arrays. Did that because I figured listing 'ge" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170182 (owner: 10Chad)
[18:57:21] <gwicke>	 akosiaris: memory looks good so far; lets continue to monitor it until later tonight & proceed with the roll-out tomorrow?
[18:57:31] <grrrit-wm>	 (03PS1) 10Dzahn: blog - remove puppet and apache config [puppet] - 10https://gerrit.wikimedia.org/r/170760 
[18:57:40] <akosiaris>	 gwicke: yup
[18:57:53] <gwicke>	 cool, thanks!
[19:00:04] <jouncebot>	 kaldari: Dear anthropoid, the time has come. Please deploy Mobile (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141103T1900).
[19:05:24] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Add WikiGrok [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170450 (owner: 10MaxSem)
[19:05:52] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add WikiGrok [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170450 (owner: 10MaxSem)
[19:08:31] <logmsgbot>	 !log maxsem Started scap: Build localization cache for WikiGrok
[19:08:37] <morebots>	 Logged the message, Master
[19:20:58] <grrrit-wm>	 (03CR) 10RobH: [C: 04-1] "I say no, as the server is still online. So we need to keep it in puppet." [puppet] - 10https://gerrit.wikimedia.org/r/170760 (owner: 10Dzahn)
[19:21:32] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "welcome YuviPanda" [puppet] - 10https://gerrit.wikimedia.org/r/170717 (owner: 10Andrew Bogott)
[19:22:00] <grrrit-wm>	 (03CR) 10Manybubbles: [C: 031] "Didn't realize that is what it did. Cool." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170182 (owner: 10Chad)
[19:39:01] <icinga-wm>	 PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [90.0]  
[19:43:25] <grrrit-wm>	 (03CR) 10RobH: "-1 revoked since it turns out the manifests didnt actually put in latest requirement for any package, so I suppose this really can sit in " [puppet] - 10https://gerrit.wikimedia.org/r/170760 (owner: 10Dzahn)
[19:43:38] <robh>	 mutante: comment revoked, kill with fire ^
[19:43:41] <logmsgbot>	 !log maxsem Finished scap: Build localization cache for WikiGrok (duration: 35m 09s)
[19:43:46] <morebots>	 Logged the message, Master
[19:44:10] <grrrit-wm>	 (03CR) 10RobH: [C: 031] "meant to replace my -1 with +1" [puppet] - 10https://gerrit.wikimedia.org/r/170760 (owner: 10Dzahn)
[19:45:16] <awight>	 Friends!  AndyRussG and I are talking about a minor skin change to support a new CentralNotice feature, and we're wondering if there is documentation somewhere about how long these changes take to propagate--also how expensive it is to purge the entire cache in case we need to do an emergency rollback.
[19:46:26] <MaxSem>	 awight, 1) 1 week since nearest Wednesday 2) prohibitively costly
[19:46:56] <Reedy>	 Which cache?
[19:47:12] <awight>	 Reedy: all the varnish...
[19:47:12] <MaxSem>	 http, judging from context
[19:47:16] <MaxSem>	 yup
[19:47:20] <Reedy>	 lol
[19:47:27] <awight>	 MaxSem: thanks!  is there a page about this...
[19:47:31] <Reedy>	 yeah, that's really not gonna happen
[19:47:47] <_joe_>	 clearing the whole cache is not advisable
[19:47:49] <awight>	 Reedy: which, the rollback?  yeah I assume that's a terrible situation
[19:48:05] <_joe_>	 that is, if we want wikipedias to be reachable
[19:48:12] <awight>	 but for a normally propagated change, Wednesday plus a week...
[19:48:25] <Reedy>	 yeah, it's now tuesday/wednesday deploys
[19:48:34] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Enable WikiGrok on test and test2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170451 (owner: 10MaxSem)
[19:48:43] <Reedy>	 so as long as you get it in before branching, it'll be on all wikipedias by the end of the next wednesday
[19:48:45] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable WikiGrok on test and test2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170451 (owner: 10MaxSem)
[19:48:59] <awight>	 Reedy: ok rad
[19:49:12] <awight>	 Sounds like breaking the site is easier than I thought :D
[19:49:55] <MaxSem>	 yup, just insert die( 'die motherfuckers' ) somewhere :P
[19:50:10] <awight>	 nooo not subtle enough
[19:50:36] <awight>	 how about <script lang=ecmascript src="slowly melt, world">
[19:51:09] <logmsgbot>	 !log maxsem Synchronized wmf-config/InitialiseSettings.php: Enable WikiGrok on test and test2 (duration: 00m 04s)
[19:51:14] <morebots>	 Logged the message, Master
[19:52:55] <grrrit-wm>	 (03CR) 10RobH: [C: 031] "This was just approved in our ops meeting." [puppet] - 10https://gerrit.wikimedia.org/r/170717 (owner: 10Andrew Bogott)
[19:52:59] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] "Mark says "Let's move forward on this"" [puppet] - 10https://gerrit.wikimedia.org/r/170717 (owner: 10Andrew Bogott)
[19:55:01] <icinga-wm>	 RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [60.0]  
[19:58:41] <icinga-wm>	 PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: puppet fail  
[20:01:55] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 031] Enable JPG thumbnail chaining on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170747 (https://bugzilla.wikimedia.org/67525) (owner: 10Gilles)
[20:09:09] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] blog - remove puppet and apache config [puppet] - 10https://gerrit.wikimedia.org/r/170760 (owner: 10Dzahn)
[20:10:59] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] bugzilla: linting fixes [puppet] - 10https://gerrit.wikimedia.org/r/170485 (owner: 10John F. Lewis)
[20:16:12] <grrrit-wm>	 (03CR) 10Dzahn: "hmm.. it kind of looks like it was aligned before but afterwards it's not" [puppet] - 10https://gerrit.wikimedia.org/r/170474 (owner: 10John F. Lewis)
[20:17:18] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] backup: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170474 (owner: 10John F. Lewis)
[20:17:40] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] archiva: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170472 (owner: 10John F. Lewis)
[20:17:53] <matanya>	 YuviPanda: you now have the powaa!!!
[20:18:52] <icinga-wm>	 RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[20:21:00] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] "failed attempt of the --fix option of puppet-lint ?" [puppet] - 10https://gerrit.wikimedia.org/r/170493 (owner: 10John F. Lewis)
[20:21:54] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Enable WikiGrok on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170452 (owner: 10MaxSem)
[20:22:06] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable WikiGrok on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170452 (owner: 10MaxSem)
[20:23:03] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] diamond: lint fixes (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/170494 (owner: 10John F. Lewis)
[20:23:38] <logmsgbot>	 !log maxsem Synchronized wmf-config/InitialiseSettings.php: Enable WikiGrok on enwiki (duration: 00m 04s)
[20:23:45] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] dynamicproxy: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170495 (owner: 10John F. Lewis)
[20:23:46] <morebots>	 Logged the message, Master
[20:25:07] <JohnFLewis>	 mutante: blame puppet-lint, it demands I do that :(
[20:25:10] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] elasticsearch: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170496 (owner: 10John F. Lewis)
[20:27:29] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] ferm: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170498 (owner: 10John F. Lewis)
[20:30:16] <robh>	 jgage: since you are on duty, i've updated the access requests docs
[20:30:32] <robh>	 outlining what has 3 day wait, when does said wait start (after all mgmt and project approvals are in), etc...
[20:30:49] <robh>	 and linked references between the clinic duty page and it
[20:32:03] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] dataset: lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170492 (owner: 10John F. Lewis)
[20:32:19] <grrrit-wm>	 (03CR) 10Ori.livneh: "paravoid, ping :)" [puppet] - 10https://gerrit.wikimedia.org/r/165779 (owner: 10Ori.livneh)
[20:32:50] <grrrit-wm>	 (03PS1) 10MaxSem: fatalmonitor: up the number of lines processed [puppet] - 10https://gerrit.wikimedia.org/r/170787 
[20:33:01] <MaxSem>	 jgage, ^^
[20:33:13] <grrrit-wm>	 (03CR) 10Reedy: [C: 031] fatalmonitor: up the number of lines processed [puppet] - 10https://gerrit.wikimedia.org/r/170787 (owner: 10MaxSem)
[20:37:42] <grrrit-wm>	 (03CR) 10ArielGlenn: "I pretty much detest the wrapping style as in nfs.pp, but if the linter really really wants the continuation to start at col 1 I'll grit m" [puppet] - 10https://gerrit.wikimedia.org/r/170492 (owner: 10John F. Lewis)
[20:41:55] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] apparmor: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170469 (owner: 10John F. Lewis)
[20:42:46] <grrrit-wm>	 (03PS5) 10Krinkle: [WIP] contint: Apply contint::qunit_localhost to labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/168631 
[20:42:48] <grrrit-wm>	 (03PS1) 10Krinkle: contint: Fix allow/deny rules for Apache 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/170792 
[20:44:42] <icinga-wm>	 PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch inactive shards 10 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 4, utimed_out: False, uactive_primary_shards: 46, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 82, uinitializing_shards: 6, unumber_of_data_nodes: 3}  
[20:45:12] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] admin: puppet lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170463 (owner: 10John F. Lewis)
[20:45:53] <bd808>	 manybubbles: Is there an easy way to tell elasticsearch not to rebalance the shards? The logstash nodes are juggling shards constantly today and killing search/indexing perfomance.
[20:46:22] <bd808>	 manybubbles: This seems to be a unintended side effect of dropping the replica count
[20:46:22] <icinga-wm>	 PROBLEM - ElasticSearch health check for shards on logstash1001 is CRITICAL: CRITICAL - elasticsearch inactive shards 10 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 4, utimed_out: False, uactive_primary_shards: 46, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 82, uinitializing_shards: 6, unumber_of_data_nodes: 3}  
[20:46:22] <icinga-wm>	 PROBLEM - ElasticSearch health check for shards on logstash1002 is CRITICAL: CRITICAL - elasticsearch inactive shards 10 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 4, utimed_out: False, uactive_primary_shards: 46, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 82, uinitializing_shards: 6, unumber_of_data_nodes: 3}  
[20:46:40] <manybubbles>	 bd808: it doesn't look like they are juggling the shards - it looks like they are unassigning them somehow
[20:46:48] <manybubbles>	 juggling doesn't make the cluster yellow
[20:46:55] <jgage>	 meh it went yellow again
[20:48:35] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Add author campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170453 (owner: 10MaxSem)
[20:48:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add author campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170453 (owner: 10MaxSem)
[20:49:02] <bd808>	 logstash1001 tried to send today's secondary shard to logstash1002. logstash1002 seems to have decided that to finish that copy it needs to drop a lot of other secondaries
[20:49:17] <jgage>	 bleh
[20:49:25] <jgage>	 how did you determine that, bd808?
[20:49:38] <logmsgbot>	 !log maxsem Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/170453/ (duration: 00m 03s)
[20:49:46] <morebots>	 Logged the message, Master
[20:50:05] <bd808>	 jgage: I've been watching /_plugin/whatson/ since MaxSem reported kibana being slow to me earlier
[20:50:37] <jgage>	 aha. so many magic urls in elasticsearch.
[20:52:33] <icinga-wm>	 RECOVERY - ElasticSearch health check for shards on logstash1002 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 3, timed_out: False, active_primary_shards: 46, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 83, initializing_shards: 6, number_of_data_nodes: 3  
[20:52:33] <icinga-wm>	 RECOVERY - ElasticSearch health check for shards on logstash1001 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 3, timed_out: False, active_primary_shards: 46, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 83, initializing_shards: 6, number_of_data_nodes: 3  
[20:53:03] <icinga-wm>	 RECOVERY - ElasticSearch health check for shards on logstash1003 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 3, timed_out: False, active_primary_shards: 46, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 83, initializing_shards: 6, number_of_data_nodes: 3  
[20:54:41] <grrrit-wm>	 (03PS2) 10Dzahn: contint: linting fixes and autoload module [puppet] - 10https://gerrit.wikimedia.org/r/170489 (owner: 10John F. Lewis)
[20:55:04] <bd808>	 jgage: Did you open an RT ticket to get more disk for these boxes by any chance?
[20:55:35] <jgage>	 we discussed it in the ops meeting but there wasn't resolution. i'll open a ticket today.
[20:55:43] <bd808>	 cool
[20:56:06] <jgage>	 i just noticed that fatalmonitor-unknowns dashboard is also sorting on normalized_message.raw, which manybubbles identified as problematic
[20:56:37] <jgage>	 i don't see a way to disable a dashboard component so i'm going to remove it for now
[20:56:44] <manybubbles>	 jgage: all those magic urls are documented :)
[20:56:54] <jgage>	 good :)
[20:56:55] <bd808>	 jgage: remove what?
[20:57:14] <jgage>	 the dashboard component which is sorting on normalized-message.raw
[20:57:40] <manybubbles>	 bd808: wtf - it really doesn't have the right to drop secondaries like that.....  I'd be pissed if the cirrus cluster did it.
[20:57:41] <jgage>	 s/component/panel/
[20:57:43] <bd808>	 The most frequent list?
[20:57:58] <jgage>	 yeah
[20:58:00] <manybubbles>	 bd808: yeah - maybe we should use docvalues on those to prevent the cache from getting super duper full
[20:58:46] <bd808>	 I'm sure it could all be made better with tuning, but it worked fine until somebody took treaffic from 4G/day to 25G/day without adding resources. :(
[20:58:52] <bd808>	 *traffic
[20:59:31] * bd808 pouts about his toy getting to popular for it's own good
[20:59:45] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "identical compilation - http://puppet-compiler.wmflabs.org/474/change/170489/html/" [puppet] - 10https://gerrit.wikimedia.org/r/170489 (owner: 10John F. Lewis)
[21:00:05] <jouncebot>	 gwicke, cscott, subbu: Dear anthropoid, the time has come. Please deploy Parsoid/OCG (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141103T2100).
[21:00:28] <jgage>	 it's parsoid, but that should be pretty clear from the default dashboard
[21:00:39] <jgage>	 we could throw away INFO level messages in teh interim, like i'm doing for hadoop
[21:03:00] <bd808>	 "started parsing" messages seem quite spammy.
[21:09:56] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] authdns: lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170473 (owner: 10John F. Lewis)
[21:10:53] <icinga-wm>	 PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail  
[21:13:37] <jgage>	 ok i've created a procurement ticket for logstash and cc'd the relevant parties (#8805)
[21:13:45] * jgage -> lunch
[21:15:03] <grrrit-wm>	 (03PS1) 10RobH: adding sni.wikimedia.org crt to repo [puppet] - 10https://gerrit.wikimedia.org/r/170805 
[21:16:15] <preilly>	 I’d like to get some #wikimedia-operations folks to come out to https://news.ycombinator.com/item?id=8553490
[21:17:18] <bd808>	 jgage: ty
[21:18:01] <grrrit-wm>	 (03CR) 10RobH: [C: 032] adding sni.wikimedia.org crt to repo [puppet] - 10https://gerrit.wikimedia.org/r/170805 (owner: 10RobH)
[21:22:36] <YuviPanda>	 matanya: :D
[21:23:22] <grrrit-wm>	 (03PS2) 10Krinkle: contint: Fix allow/deny rules for Apache 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/170792 
[21:23:46] <grrrit-wm>	 (03PS6) 10Krinkle: [WIP] contint: Apply contint::qunit_localhost to labs slaves [puppet] - 10https://gerrit.wikimedia.org/r/168631 
[21:31:56] <grrrit-wm>	 (03PS3) 10Yuvipanda: shinken: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/168133 
[21:43:47] <grrrit-wm>	 (03CR) 10Dzahn: "Hi Christopher, this seems to need further discussion because of concerns Chase brought up because this is tying into the overall phab upg" [puppet] - 10https://gerrit.wikimedia.org/r/166406 (owner: 10Christopher Johnson (WMDE))
[21:48:33] <grrrit-wm>	 (03PS1) 10Cscott: Reconfigure OCG to use bunyan logging instead of winston. [puppet] - 10https://gerrit.wikimedia.org/r/170828 
[21:52:08] <grrrit-wm>	 (03PS2) 10Dzahn: icinga - lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170475 
[21:55:04] <grrrit-wm>	 (03PS2) 10Yuvipanda: shinken: Add missing shinkengen.yaml config file [puppet] - 10https://gerrit.wikimedia.org/r/168134 
[21:55:58] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] icinga - lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170475 (owner: 10Dzahn)
[22:02:35] <grrrit-wm>	 (03PS3) 10Yuvipanda: shinken: Add missing shinkengen.yaml config file [puppet] - 10https://gerrit.wikimedia.org/r/168134 
[22:02:42] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] shinken: Add missing shinkengen.yaml config file [puppet] - 10https://gerrit.wikimedia.org/r/168134 (owner: 10Yuvipanda)
[22:04:06] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] wikimetrics - lint fixes [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/170478 (owner: 10Dzahn)
[22:07:01] <mutante>	 ottomata: what to do after merging in wikimetrics submodule? nothing?
[22:07:48] <ottomata>	 in operations/puppet, do the submoudle dance:
[22:08:17] <ottomata>	 if you edited modules/wikimetrics in your clone that you usually submit from, you can probably see that the submodule's sha has changed
[22:08:28] <ottomata>	 git status or git diff should show a change for modules/wikimetrics
[22:08:29] <ottomata>	 does it?
[22:09:55] <mutante>	 ottomata: no, i mean what to do on palladium 
[22:11:30] <jgage>	 normal puppet-merge will show the submodule diff
[22:12:41] <mutante>	 jgage: https://gerrit.wikimedia.org/r/#/c/170478/  merged though
[22:12:43] <ottomata>	 oh
[22:12:51] <mutante>	 git log doenst show it on palladiumm
[22:12:52] <ottomata>	 if you puppet-merge on palladium, that's it
[22:12:56] <ottomata>	 analytics devs take it from there
[22:13:06] <ottomata>	 their labs instances run self hosted puppet masters
[22:13:14] <ottomata>	 so they need to do git pull and rebase on those
[22:13:24] <mutante>	 ok, cool, so i just wanna see it on palladium. then great
[22:13:29] <ottomata>	 yup
[22:13:33] <mutante>	 yet i don't yet
[22:14:23] <grrrit-wm>	 (03PS1) 10BBlack: switch r::c::ssl::sni (cp1008 only) to new wm.o cert [puppet] - 10https://gerrit.wikimedia.org/r/170839 
[22:15:41] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] switch r::c::ssl::sni (cp1008 only) to new wm.o cert [puppet] - 10https://gerrit.wikimedia.org/r/170839 (owner: 10BBlack)
[22:16:42] <cscott>	 hello, ops!
[22:16:52] <grrrit-wm>	 (03PS2) 10Cscott: Reconfigure OCG to use bunyan logging instead of winston. [puppet] - 10https://gerrit.wikimedia.org/r/170828 
[22:17:14] <cscott>	 jgage: could i get a review & deploy of a puppet patch so i can finish my ocg deploy for today? https://gerrit.wikimedia.org/r/170828
[22:17:40] * jgage looks
[22:17:46] <cscott>	 thanks.
[22:18:25] <jgage>	 cscott: i have no familiarity with ocg. is there someone who does, who can review this?
[22:18:38] <jgage>	 i see you tagged andrewbogott
[22:19:08] <cscott>	 i can get arlolra to review the OCG part of it, but he can only +1
[22:19:18] <andrewbogott>	 cscott: I think Jeff_Green is the expert
[22:19:20] <andrewbogott>	 ...maybe
[22:19:49] <jgage>	 i can +2 but i'm uneasy about doing that until we get a review from someone who knows this code
[22:20:10] <cscott>	 jgage: sure, understandable.
[22:21:29] <grrrit-wm>	 (03PS1) 10Dzahn: wikimetrics - update submodule [puppet] - 10https://gerrit.wikimedia.org/r/170842 
[22:22:08] <Jeff_Green>	 cscott: i've got to run in about 10 mins, so I'm probably not a good candidate either. I can do it in the AM as soon as you're ready though?
[22:23:58] <cscott>	 Jeff_Green: well, ocg is in a bit of an inconsistent state right now, since i deployed the OCG-side changes on labs for testing, but they are not logging to logstash any more because this puppet part of the reconfig isn't yet pushed.  but it's not too bad, I'd just need to write a mini-essay to !log if we leave things like this overnight. ;)
[22:24:30] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "this is Change-Id: I752db14673f139e129" [puppet] - 10https://gerrit.wikimedia.org/r/170842 (owner: 10Dzahn)
[22:25:17] <Jeff_Green>	 cscott: i see. well I can roll it but I can't undo/modify. I'm supposed to be somewhere at 5:30
[22:27:18] <grrrit-wm>	 (03CR) 10Arlolra: [C: 031] Reconfigure OCG to use bunyan logging instead of winston. [puppet] - 10https://gerrit.wikimedia.org/r/170828 (owner: 10Cscott)
[22:28:17] <cscott>	 Jeff_Green: maybe let jgage do the roll out and (hopefully unnecessary) revert.
[22:28:21] <Jeff_Green>	 k
[22:28:45] <cscott>	 jgage: arlolra gave the +1 and i verified that w/ this change applied locally to deployment-pdf02, log packets are getting to logstash-beta.wmflabs.org as expected.
[22:31:45] <jgage>	 cscott: ok, thanks. i'll merge it now. hopefully you can help me notice if it goes awry :)
[22:32:06] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] releases: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170464 (owner: 10Dzahn)
[22:32:08] <cscott>	 yep, i'll do the testing
[22:32:51] <cscott>	 !log updated OCG to version 5834af97ae80382f3368dc61b9d119cef0fe129b
[22:32:56] <grrrit-wm>	 (03CR) 10Gage: [C: 032] "I don't know this code, but I discussed with cscott on IRC and it has +1. Merging." [puppet] - 10https://gerrit.wikimedia.org/r/170828 (owner: 10Cscott)
[22:32:57] <morebots>	 Logged the message, Master
[22:35:08] <jgage>	 cscott ok, it should be live within 20 mins
[22:35:33] <cscott>	 jgage: thanks.
[22:35:41] <jgage>	 sure :)
[22:36:20] <cscott>	 jgage: any way i can force a puppet run on ocg1001 so i can test things out?
[22:37:07] <jgage>	 probably not you, but i'll do it. one sec.
[22:37:08] <grrrit-wm>	 (03PS1) 10Dzahn: ssl: sni.wm.org, add CA name for correct chain [puppet] - 10https://gerrit.wikimedia.org/r/170847 
[22:38:12] <jgage>	 cscott: puppet run complete on ocg1001
[22:38:26] <cscott>	 great, i see the new config file.  testing it.
[22:38:31] <jgage>	 cool
[22:46:12] <grrrit-wm>	 (03CR) 10Matanya: puppetmaster - lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170471 (owner: 10Dzahn)
[22:46:16] <Krenair>	 mutante, so what's this sni.wm.o stuff about? :)
[22:46:49] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "tiny, just a quoted filemode and order of parameters" [puppet] - 10https://gerrit.wikimedia.org/r/170480 (owner: 10Dzahn)
[22:52:36] <grrrit-wm>	 (03CR) 10Matanya: mysql_wmf - autoload layout and lint fixes (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/170479 (owner: 10Dzahn)
[22:53:25] <mutante>	 Krenair: https://gerrit.wikimedia.org/r/#/c/161193/11/manifests/role/cache.pp   https://gerrit.wikimedia.org/r/#/q/owner:bblack+status:merged+message:SNI,n,z
[22:54:01] <matanya>	 Krenair: are t
[22:54:22] <matanya>	 *you asking why we need it?
[22:54:52] <Krenair>	 I'm not exactly sure what I'm asking.
[22:56:32] <Krenair>	 matanya, actually yes, what is it/why do we need it?
[22:56:42] <mutante>	 Krenair: now, if you look into files/ssl/ in puppet, each project has a separate star. cert, that would be to unify them all into one
[22:56:53] <mutante>	 using sni.wikimedia.org.crt
[22:57:19] <mutante>	 sni for http://en.wikipedia.org/wiki/Server_Name_Indication
[22:57:24] <matanya>	 Krenair: reading SNI article in en.wiki would answer 99 of the q
[22:59:25] <grrrit-wm>	 (03PS2) 10Dzahn: mysql_wmf - autoload layout and lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170479 
[22:59:27] <grrrit-wm>	 (03CR) 10Dzahn: mysql_wmf - autoload layout and lint fixes (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/170479 (owner: 10Dzahn)
[22:59:42] <cscott>	 jgage: looks like it working.  thanks.
[23:00:08] <cscott>	 jgage: i might need to make another tweak tomorrow, since the elasticsearch fields for the logs aren't quite what i want, but that's not urgent.
[23:00:26] <cscott>	 (and that might be a code change, not a puppet/config file change.)
[23:00:37] <jgage>	 cscott ok, cool. just ping me if needed.
[23:01:24] <cscott>	 !log reconfigured OCG logstash path to use bunyan. The _type field is currently missing (used to be "OfflineContentGenerator").  Will fix tomorrow.
[23:01:29] <morebots>	 Logged the message, Master
[23:02:05] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] apparmor: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170469 (owner: 10John F. Lewis)
[23:02:22] <Krenair>	 matanya, mutante: I was under the impression that we already had one cert with all the domains in Subject Alternative Name field?
[23:03:02] <matanya>	 no. we had wildcard one
[23:03:02] <cscott>	 bd808: can you remind me where the logstash gelf input munging configuration file is found?  in puppet somewhere?
[23:03:18] <matanya>	 yes cscott 
[23:03:33] <bd808>	 cscott: Yeah. https://github.com/wikimedia/operations-puppet/blob/production/files/logstash/filter-gelf.conf
[23:03:45] <cscott>	  bd808.  ah, thanks.
[23:04:02] <grrrit-wm>	 (03CR) 10Matanya: mysql - lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170486 (owner: 10Dzahn)
[23:04:46] <bd808>	 cscott: Looks like type is copied from the "facility" field in the GELF message
[23:05:02] <cscott>	 ah, ok.
[23:05:40] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] mariadb - lint fixes [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/170491 (owner: 10Dzahn)
[23:06:21] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] extdist: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170497 (owner: 10John F. Lewis)
[23:08:55] <cscott>	 bd808: ah.  the _type was OfflineContentGenerator before, now it's mw-ocg-service.  Is it easy to change the query on the OCG dashboard?
[23:09:02] <cscott>	 https://logstash.wikimedia.org/#/dashboard/elasticsearch/OCG%20Backend ?
[23:09:27] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] base: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170477 (owner: 10John F. Lewis)
[23:09:39] <bd808>	 cscott: Yeah. Want me to fix it?
[23:10:21] <bd808>	 cscott: Should be fixed now.
[23:10:57] <grrrit-wm>	 (03CR) 10Dzahn: elasticsearch: lint fixes (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/170496 (owner: 10John F. Lewis)
[23:11:21] <grrrit-wm>	 (03Abandoned) 10Legoktm: Set $wgCentralAuthAutoMigrate = false on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167730 (owner: 10Legoktm)
[23:17:10] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] ceph: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170488 (owner: 10John F. Lewis)
[23:19:19] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] bacula: lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/170476 (owner: 10John F. Lewis)
[23:20:15] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] apache: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170466 (owner: 10John F. Lewis)
[23:25:39] <Nemo_bis>	 whee bblack revived it.m.wikipedia.org :) https://it.wikipedia.org/w/index.php?title=Speciale:UltimeModifiche&days=30&from=&hideliu=1&limit=500&tagfilter=mobile+web+edit
[23:25:43] <Nemo_bis>	 Thanks again
[23:31:57] <MaxSem>	 Nemo_bis, yay! https://it.wikipedia.org/w/index.php?title=Longobardi&curid=713739&diff=68981497&oldid=67692794 :P
[23:34:46] <bd808>	 !log Changed elasticsearch template for logstash to use "doc_values" for raw fields. http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/
[23:34:52] <morebots>	 Logged the message, Master
[23:36:10] <Nemo_bis>	 MaxSem: rolled back faster than ever thanks to HHVM ;)
[23:51:26] <icinga-wm>	 PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: puppet fail  
[23:58:29] <grrrit-wm>	 (03PS1) 10BBlack: Set correct CA for sni.wm.o cert [puppet] - 10https://gerrit.wikimedia.org/r/170862 
[23:59:46] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] Set correct CA for sni.wm.o cert [puppet] - 10https://gerrit.wikimedia.org/r/170862 (owner: 10BBlack)