[00:58:59] <grrrit-wm>	 (03CR) 10Ori.livneh: "Petrb: <https://gist.github.com/atdt/258f0994f988870a3bd5> is a working example that subscribes to the rcstream feed and writes each incom" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (owner: 10Ori.livneh)
[01:12:29] <grrrit-wm>	 (03PS7) 10Ori.livneh: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 
[01:17:16] <grrrit-wm>	 (03PS8) 10Ori.livneh: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/16599) 
[02:13:01] <icinga-wm>	 PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3792 MB (3% inode=99%):  
[02:14:22] <logmsgbot>	 !log LocalisationUpdate completed (1.24wmf2) at 2014-05-05 02:13:18+00:00
[02:14:31] <morebots>	 Logged the message, Master
[02:22:01] <icinga-wm>	 PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3434 MB (3% inode=99%):  
[02:26:26] <grrrit-wm>	 (03PS9) 10Ori.livneh: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/16599) 
[02:26:37] <logmsgbot>	 !log LocalisationUpdate completed (1.24wmf3) at 2014-05-05 02:25:34+00:00
[02:26:43] <morebots>	 Logged the message, Master
[02:32:22] <^demon|away>	 !log antimony: ran very very aggressive repacking on mediawiki/core, operations/puppet, mediawiki/extensions/{UploadWizard,CentralAuth,CentralNotice,DonationInterface,FlaggedRevs,AbuseFilter,BlueSpiceExtensions,Translate,WikimediaMessages,EducationProgram,UniversalLanguageSelector,Wikibase}, pywikibot/{core,compat}, operations/dumps/tests. Basically anything taking up >90MB on disk. Probably not the cause of gitb
[02:32:22] <^demon|away>	 lit's wonkiness but they're certainly not helping matters.
[02:32:29] <morebots>	 Logged the message, Master
[02:32:50] <^demon|away>	 !log [gitb]lit's wonkiness but they're certainly not helping matters.
[02:32:56] <morebots>	 Logged the message, Master
[02:40:19] <grrrit-wm>	 (03PS6) 10Springle: MariaDB multi-source replication monitoring. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131015 
[02:41:42] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[02:42:41] <grrrit-wm>	 (03CR) 10Springle: [C: 032] MariaDB multi-source replication monitoring. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131015 (owner: 10Springle)
[02:47:31] <grrrit-wm>	 (03PS10) 10Krinkle: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/16599) (owner: 10Ori.livneh)
[02:48:53] <grrrit-wm>	 (03PS11) 10Krinkle: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/16599) (owner: 10Ori.livneh)
[02:50:36] <grrrit-wm>	 (03PS12) 10Krinkle: Add 'rcstream' module for broadcasting recent changes over WebSockets [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/14045) (owner: 10Ori.livneh)
[02:51:27] <grrrit-wm>	 (03CR) 10Krinkle: "Referring to bug 14045 instead of bug 16599 (tracker bug), and removed duplicate RFC (one is marked as superseded by the other and already" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131040 (https://bugzilla.wikimedia.org/14045) (owner: 10Ori.livneh)
[02:56:44] <grrrit-wm>	 (03PS1) 10Springle: Use correct replication channel name s/m1/m2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131414 
[02:58:30] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Use correct replication channel name s/m1/m2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131414 (owner: 10Springle)
[03:01:00] <icinga-wm>	 RECOVERY - Disk space on virt0 is OK: DISK OK  
[03:12:22] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Mon May  5 03:11:15 UTC 2014 (duration 11m 14s)
[03:12:28] <morebots>	 Logged the message, Master
[03:34:23] <grrrit-wm>	 (03PS1) 10Springle: Remove db68 from site.pp. To be rebuilt. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131416 
[03:36:09] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Remove db68 from site.pp. To be rebuilt. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131416 (owner: 10Springle)
[04:10:55] <grrrit-wm>	 (03PS1) 10Springle: Use m2-master CNAME to make DB rotations neater. This allows a master switch to be a DNS change plus a simple port 3306 tcp redirect with socat until TTL. Should also help if we switch to a haproxy configuration in the future. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131418 
[04:13:22] <grrrit-wm>	 (03CR) 10Springle: "Is this likely to work with consumers after TTL and at least one DB reconnect, or are they going to need an explicit restart if CNAME chan" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131418 (owner: 10Springle)
[04:19:59] <grrrit-wm>	 (03PS1) 10Springle: Use m2-master CNAME to make DB rotations neater. This allows a master switch to be a DNS change plus a simple port 3306 tcp redirect with socat until TTL. Should also help if we switch to a haproxy configuration in the future. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131419 
[04:20:36] <grrrit-wm>	 (03CR) 10Springle: "Is this likely to work with gerrit after TTL and at least one DB reconnect, or will it need an explicit restart if CNAME changes?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131419 (owner: 10Springle)
[04:25:50] <grrrit-wm>	 (03PS1) 10Springle: Use m2-master CNAME to make DB master rotations neater. This allows a master switch to be a DNS change plus a simple port 3306 tcp redirect with socat until TTL. Should also help if we switch to a haproxy configuration in the future. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131420 
[04:28:32] <grrrit-wm>	 (03CR) 10Springle: "Does this sound ok? The scholarships app doesn't appear to use persistent connections (great!) so I'm expecting the DNS/socat process woul" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131420 (owner: 10Springle)
[04:37:24] <grrrit-wm>	 (03PS1) 10Springle: Useful to have one slave CNAME for m1, m2, and x1 shards [operations/dns] - 10https://gerrit.wikimedia.org/r/131421 
[04:43:10] <grrrit-wm>	 (03CR) 10Springle: "Want these to put in puppet in a few places rather than actual db host names. Trying to think of any catches that could arise some day?" [operations/dns] - 10https://gerrit.wikimedia.org/r/131421 (owner: 10Springle)
[05:42:43] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[06:29:23] <icinga-wm>	 PROBLEM - Disk space on vanadium is CRITICAL: DISK CRITICAL - free space: / 4018 MB (3% inode=95%):  
[06:30:23] <icinga-wm>	 RECOVERY - Disk space on vanadium is OK: DISK OK  
[07:52:23] <grrrit-wm>	 (03PS1) 10Hashar: Retab role/gerrit.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/131432 
[07:52:26] <grrrit-wm>	 (03PS1) 10Hashar: Remove a level of indentation from role/gerrit.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/131433 
[07:52:27] <grrrit-wm>	 (03PS1) 10Hashar: role::gerrit::labs::ci for CI project on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131434 
[07:54:13] <_joe_>	  /win 27
[07:54:19] <_joe_>	 oh I hate this
[07:54:24] <twkozlowski>	 you won!
[07:58:58] <grrrit-wm>	 (03Abandoned) 10Hashar: role::gerrit::labs::ci for CI project on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131434 (owner: 10Hashar)
[08:43:43] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[09:13:46] <grrrit-wm>	 (03PS1) 10Hashar: Adjust role::zuul::labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131438 
[09:20:13] <grrrit-wm>	 (03PS5) 10Hashar: contint: split Zuul server and merger (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 
[09:20:31] <grrrit-wm>	 (03CR) 10Hashar: "Pass $git_dir to zuul::merger since it needs it for git repack." [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 (owner: 10Hashar)
[09:29:10] <grrrit-wm>	 (03PS6) 10Hashar: contint: split Zuul server and merger (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 
[09:37:28] <grrrit-wm>	 (03PS2) 10Hashar: Adjust role::zuul::labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131438 
[10:56:45] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: Move cluster definition to the node level. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130591 
[11:44:35] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[11:45:07] <grrrit-wm>	 (03PS1) 10Gage: initial debianization [operations/debs/python-statsd] - 10https://gerrit.wikimedia.org/r/131449 
[12:25:20] <grrrit-wm>	 (03CR) 10Gage: "* Uses git-buildpackage and pristine-tar" [operations/debs/python-statsd] - 10https://gerrit.wikimedia.org/r/131449 (owner: 10Gage)
[12:39:52] <grrrit-wm>	 (03PS3) 10Nemo bis: We have APC, let's use it in $wgMainCacheType [operations/puppet] - 10https://gerrit.wikimedia.org/r/119102 
[12:43:25] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: WIP: Remove various sdtpa leftover DNS records [operations/dns] - 10https://gerrit.wikimedia.org/r/131453 
[12:45:19] <akosiaris>	 !log removing various sdtpa devices from LibreNMS
[12:45:26] <morebots>	 Logged the message, Master
[12:49:17] <apergos>	 akosiaris: there are now three such patchsets
[12:50:24] <akosiaris>	 :-(
[12:50:48] <apergos>	 the other two: https://gerrit.wikimedia.org/r/#/c/130573/   and https://gerrit.wikimedia.org/r/#/c/131348/
[12:50:49] <apergos>	 oh and
[12:51:06] <paravoid>	 lol
[12:51:07] <apergos>	 https://gerrit.wikimedia.org/r/#/c/131347/
[12:51:20] <apergos>	 choose any, don't care, let's jut merge before there's a fourth :-D
[12:52:24] <apergos>	 in other news ff29 is weirding me out a bit (where's the ___ button?  oh it's morphed into _that_ thing...)
[12:53:30] <Gloria>	 Yeah, I heard they did major redesign.
[12:54:11] <paravoid>	 I installed the "classic theme restorer" extension
[12:54:15] <paravoid>	 and all is well
[12:54:26] <apergos>	 I"m going to try to stick it out for a little while
[13:06:52] <paravoid>	 I couldn't stand the tabs on the top
[13:09:09] <_joe_>	 I love it
[13:09:11] <_joe_>	 :)
[13:27:00] <andrewbogott>	 manybubbles|away: I'm wondering if it's now safe to chance your UID or if you still have running jobs… ping me when you get in?
[13:27:56] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131462 
[13:27:58] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131462 (owner: 10Andrew Bogott)
[13:34:36] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131463 
[13:48:59] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Remove cr1-sdtpa/csw1-sdtpa, fix crosslinks DNS [operations/dns] - 10https://gerrit.wikimedia.org/r/131347 (owner: 10Faidon Liambotis)
[13:49:03] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Kill DNS for sdtpa mgmt hosts (asw, cam, ps, scs) [operations/dns] - 10https://gerrit.wikimedia.org/r/131348 (owner: 10Faidon Liambotis)
[13:49:21] <paravoid>	 apergos: merged mine, as it was fixing (and not removing) crosslinks
[13:49:41] <apergos>	 ok, off to abandon
[13:50:39] <grrrit-wm>	 (03Abandoned) 10ArielGlenn: sdtpa is gone, remove all entries to devices, links, etc [operations/dns] - 10https://gerrit.wikimedia.org/r/130573 (owner: 10ArielGlenn)
[13:51:04] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: WIP: Remove various sdtpa leftover DNS records [operations/dns] - 10https://gerrit.wikimedia.org/r/131453 (owner: 10Alexandros Kosiaris)
[13:53:15] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Rename aaharoni to amire80 to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131464 
[13:53:37] <grrrit-wm>	 (03Abandoned) 10Andrew Bogott: Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131462 (owner: 10Andrew Bogott)
[13:56:41] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Rename aaharoni to amire80 to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131464 (owner: 10Andrew Bogott)
[13:56:49] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Rename gdubuc to 'gilles' to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131465 
[13:57:42] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131463 
[13:59:12] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Rename gdubuc to 'gilles' to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131465 (owner: 10Andrew Bogott)
[13:59:46] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131463 
[14:01:44] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Update dartar's UID to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131463 (owner: 10Andrew Bogott)
[14:30:09] <manybubbles>	 andrewbogott_afk: I'm now totally logged out - sorry, wasn't paying attention to pings for a while there
[14:30:50] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: torrus: csw2-esams in accessswitches, not corerouters [operations/puppet] - 10https://gerrit.wikimedia.org/r/131473 
[14:36:02] <anomie>	 aharoni: For the SWAT, please prepare gerrit changes that actually backport the changes to the relevant deployment branches (wmf/1.24wmf2 and/or wmf/1.24wmf3), and link them from the Deployments page. The "Cherry Pick To" button in Gerrit is helpful for this, although note for extensions you'll additionally need to prepare a revision updating the submodule in the appropriate mediawiki/core branch. Thanks.
[14:43:19] <matanya>	 anomie: is https://bugzilla.wikimedia.org/show_bug.cgi?id=64727 going in this SWAT ?
[14:43:50] <anomie>	 matanya: Planning on it, why?
[14:44:21] <matanya>	 my bot is not running for a week now, and people are complaining 
[14:45:12] <matanya>	 a week == 3 days 
[14:45:36] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[14:45:50] <matanya>	 anomie: so, if this gets in it would help me a lot. thanks
[14:47:06] <anomie>	 matanya: Ok. (: I thought you were asking because you saw some problem with it from an ops perspective.
[14:47:33] <matanya>	 well, i'm not an op, so no :)
[14:54:14] <anomie>	 manybubbles: I'll do the SWAT today
[14:55:06] <aharoni>	 hi anomie 
[14:55:17] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] "Scholarships does actually use persistent connections (see https://github.com/wikimedia/wikimedia-wikimania-scholarships/blob/master/src/W" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131420 (owner: 10Springle)
[14:55:27] <anomie>	 hi aharoni 
[14:56:00] <aharoni>	 looking at the SWAT stuff
[14:56:46] <aharoni>	 back in a couple of minutes
[15:00:29] * anomie  is beginning the SWAT deploy, with his own backport

[15:00:41] <manybubbles>	 anomie: sweet
[15:01:02] <grrrit-wm>	 (03PS3) 10BBlack: Use whole subnets in squid.php list for XFF acceptance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130634 
[15:01:45] <bblack>	 I guess I should wait for SWAT before trying to merge in the above
[15:01:56] <bblack>	 (or if it's easy to include it, feel free)
[15:03:44] <aharoni>	 anomie: back
[15:03:56] <aharoni>	 had to reconnect
[15:04:22] <aharoni>	 so, do I have to do the "Cherry pick to" thing?
[15:05:43] <aharoni>	 to wmf/1.24wmf3 ?
[15:09:31] <aharoni>	 anomie: ^
[15:09:40] <andrewbogott>	 manybubbles: ok if I move you right now?
[15:10:23] <anomie>	 aharoni: It'd be helpful, yes. I can do it for you in a few minutes, but for the future it saves time to do it yourself so I don't have to ask which version you're wanting to backport to.
[15:10:38] <logmsgbot>	 !log anomie synchronized php-1.24wmf2/includes/api/ApiLogin.php  'SWAT: Backport change 131056 to 1.24wmf2 to fix bug 64727'
[15:10:43] <aharoni>	 anomie: I should learn.
[15:10:45] <morebots>	 Logged the message, Master
[15:10:52] <aharoni>	 only wmf/1.24wmf3 , or also wmf/1.24wmf2 /
[15:10:54] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Move Nik Everett/manybubbles to UID 3304 to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131477 
[15:10:55] <aharoni>	 ?
[15:11:33] <anomie>	 aharoni: It depends if it needs to go to all the wikis or just the ones on 1.24wmf3 already.
[15:11:59] <logmsgbot>	 !log anomie synchronized php-1.24wmf3/includes/api/ApiLogin.php  'SWAT: Backport change 131056 to 1.24wmf3 to fix bug 64727'
[15:12:05] <morebots>	 Logged the message, Master
[15:12:07] <anomie>	 matanya: Bug fix is deployed now, hopefully that makes your bot work
[15:12:33] * anomie  has verified that the bug fix fixed the bug it was supposed to fix

[15:12:53] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Move Nik Everett/manybubbles to UID 3304 to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131477 
[15:13:01] <aharoni>	 my guess is that only wmf3
[15:13:25] <anomie>	 aharoni: I'm ready to deploy as soon as you have the patches ready
[15:15:16] <thedj>	 one day... if I have another lifetime left in me, i'm rewriting half the plugins of jenkins... 
[15:16:07] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Move Nik Everett/manybubbles to UID 3304 to match labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131477 (owner: 10Andrew Bogott)
[15:16:41] <aharoni>	 OK, preparing
[15:17:36] <aharoni>	 anomie: does this look OK - https://gerrit.wikimedia.org/r/#/c/131479/ ?
[15:18:06] * anomie  +2s

[15:18:19] <aharoni>	 anomie - the other one: https://gerrit.wikimedia.org/r/#/c/131480/
[15:18:42] <greg-g>	 aharoni: if the bug is only present in one wmf branch, you only need one backport
[15:19:03] <greg-g>	 that was my question in my email on that thread, in fact :) (if this was only wmf3)
[15:19:47] <andrewbogott>	 mutante: you around?  Can you give me a hand with an RT access question?
[15:19:58] <anomie>	 aharoni: The second one looks good too, although the backport also needs the instructions at https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule followed. I can do it, but if you want to learn how you're welcome to.
[15:20:25] <aharoni>	 anomie: I'll look. I should know such things at least a bit.
[15:24:23] <logmsgbot>	 !log anomie synchronized php-1.24wmf3/includes/specials/SpecialAllmessages.php  'SWAT: Backport change 131041 to 1.24wmf3 to fix bug in Special:AllMessages'
[15:24:28] <anomie>	 aharoni: The Special:AllMessages change is deployed now, please verify that things are now working
[15:24:29] <morebots>	 Logged the message, Master
[15:24:38] <aharoni>	 thanks
[15:24:55] <aharoni>	 anomie: checking.
[15:25:02] <aharoni>	 in the meantime, `git submodule update --init --recursive` takes some time...
[15:26:05] <greg-g>	 :) a bit
[15:26:30] <aharoni>	 anomie: Special:Allmessages doesn't seem to work.
[15:26:43] <aharoni>	 Caching issue, or something broken?
[15:26:54] <anomie>	 aharoni: First, which wiki are you testing on?
[15:27:22] <aharoni>	 English, Hebrew
[15:27:25] <aharoni>	 should it be test?
[15:27:52] <anomie>	 aharoni: IIRC, only mediawiki.org and the testwikis are on wmf3. Check Special:Version.
[15:28:14] <anomie>	 If you need it backported to wmf2 as well, we have time to do that.
[15:29:18] <anomie>	 Yes, only mediawiki.org and the testwikis are on wmf3. Non-Wikipedias go tomorrow, and Wikipedias on Thursday.
[15:29:58] <aharoni>	 anomie: it's broken on enwiki.
[15:30:03] <aharoni>	 so wmf2?
[15:30:25] <aharoni>	 yes, looks like it in Special:Version
[15:30:32] <aharoni>	 (I know so little about deployment.)
[15:30:34] <anomie>	 aharoni: Yes, we'll need to do enwiki as well. Same cherry-pick process, just to the 1.24wmf2 branch
[15:31:13] <aharoni>	 OK
[15:31:42] <aharoni>	 anomie: https://gerrit.wikimedia.org/r/#/c/131481/
[15:32:06] <aharoni>	 anomie: https://gerrit.wikimedia.org/r/#/c/131482/
[15:32:07] <anomie>	 aharoni: +2ed
[15:32:35] <aharoni>	 (and git submodule is still running for that MofileFrontend thing on wmf3)
[15:36:13] <aharoni>	 git submodule done, going on...
[15:37:47] <logmsgbot>	 !log anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php  'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
[15:37:54] <morebots>	 Logged the message, Master
[15:37:59] <aharoni>	 anomie: about the MobileFrontend change: 
[15:38:01] <anomie>	 aharoni: enwiki has the Special:AllMessages fix now
[15:38:17] <aharoni>	 on wmf3 the Git procedure didn't seem to do anything.
[15:38:31] <aharoni>	 git review said error: failed to push some refs to 'ssh://amire80@gerrit.wikimedia.org:29418/mediawiki/core.git'
[15:39:49] <anomie>	 aharoni: No details on why it failed?
[15:40:21] <aharoni>	 pretty straightforward
[15:40:25] <aharoni>	 remote: Processing changes: refs: 1, done    
[15:40:27] <aharoni>	 To ssh://amire80@gerrit.wikimedia.org:29418/mediawiki/core.git
[15:40:28] <aharoni>	  ! [remote rejected] HEAD -> refs/publish/wmf/1.24wmf3 (no new changes)
[15:40:30] <aharoni>	 error: failed to push some refs to 'ssh://amire80@gerrit.wikimedia.org:29418/mediawiki/core.git'
[15:41:15] <anomie>	 aharoni: There you go, "no new changes". Did you commit your change to the extension before trying to update the submodule ref in core?
[15:42:06] <aharoni>	 git commit -a -m "Update MyCoolExtension"
[15:42:20] <aharoni>	 do I have to run `git commit -a -m "Update MyCoolExtension"` in the core directory or in the extension directory?
[15:42:36] <anomie>	 aharoni: I see that https://gerrit.wikimedia.org/r/#/c/131480/ isn't merged, so the git fetch and git checkout in the extension directory didn't do anything.
[15:43:36] <aharoni>	 right.
[15:43:50] <aharoni>	 are you supposed to merge it first?
[15:44:07] <anomie>	 You need to for the submodule update to work right
[15:45:06] <aharoni>	 I'm not sure that I understand.
[15:45:15] <aharoni>	 I cannot merge https://gerrit.wikimedia.org/r/#/c/131480/ .
[15:45:17] <aharoni>	 no +2 right
[15:45:28] <anomie>	 aharoni: Ah, ok. I'll take care of that then.
[15:45:41] <aharoni>	 thanks
[15:46:29] <aharoni>	 and Special:Allmessages is still broken on English and Hebrew Wikipedia.
[15:48:51] <anomie>	 aharoni: Ugh. That would be my fault, I forgot one step in the process. :(
[15:49:52] <logmsgbot>	 !log anomie synchronized php-1.24wmf2/includes/specials/SpecialAllmessages.php  'SWAT: Backport change 131041 to 1.24wmf2 to fix bug in Special:AllMessages'
[15:49:57] <aharoni>	 \o/ seems to work
[15:50:06] <anomie>	 Good!
[15:50:22] <anomie>	 Now, we've got about 10 minutes to deploy and verify your other backport.
[15:50:31] <aharoni>	 OK, is there anything more I can do to help with it?
[15:50:36] <aharoni>	 (I'm here, in any case.)
[15:50:56] <anomie>	 Well, are you still wanting to try preparing the submodule update for wmf3? Or should I just do it?
[15:51:32] * andre__  also has a question for anomie but this looks more urgent, so better not interrupting here

[15:51:42] <greg-g>	 uno momento por favor
[15:51:48] <aharoni>	 anomie: I will have the same problem with +2
[15:52:10] <anomie>	 aharoni: I'd +2 it anyway, I was just wondering if you still wanted to put the change in Gerrit
[15:52:21] <aharoni>	 Oh.
[15:52:23] * anomie  will do the backport to wmf2 in the mean time

[15:52:24] <aharoni>	 a sec
[15:54:27] <aharoni>	 now it seems to work
[15:55:17] <aharoni>	 anomie: does this make sense? - https://gerrit.wikimedia.org/r/#/c/131486/
[15:56:13] <anomie>	 aharoni: Yes
[15:58:22] <anomie>	 aharoni: Doing the wmf2 one now, please check as soon as the bot says it's done
[15:58:28] <anomie>	 Then I'll do the wmf3 version
[15:59:03] <logmsgbot>	 !log anomie synchronized php-1.24wmf2/extensions/MobileFrontend/  'SWAT: Backport change 131237 to 1.24wmf2 to fix bug in MobileFrontend'
[15:59:10] <morebots>	 Logged the message, Master
[15:59:25] * anomie  is doing the wmf3 MobileFrontend sync now

[16:00:25] <logmsgbot>	 !log anomie synchronized php-1.24wmf3/extensions/MobileFrontend/  'SWAT: Backport change 131237 to 1.24wmf3 to fix bug in MobileFrontend'
[16:00:31] <morebots>	 Logged the message, Master
[16:00:53] * anomie  is done with this morning's SWAT

[16:01:10] <greg-g>	 aharoni: let us know when you've confirmed the fix
[16:03:20] <twkozlowski>	 Sorry! This site is experiencing technical difficulties.
[16:03:24] <twkozlowski>	 Try waiting a few minutes and reloading.
[16:03:26] <twkozlowski>	 (Cannot contact the database server: Too many connections (10.64.16.10))
[16:04:03] <greg-g>	 twkozlowski: ?
[16:04:05] <greg-g>	 which wiki?
[16:04:08] <aude>	 wfm
[16:04:18] * aude  assumes wikidata

[16:04:28] <aharoni>	 greg-g: the MobileFrontend change looks good
[16:04:36] <greg-g>	 aharoni: great, thanks
[16:04:37] <twkozlowski>	 Wikidata
[16:04:48] <aude>	 twkozlowski: any particular page?
[16:04:57] <twkozlowski>	 https://www.wikidata.org/wiki/Q9381500#sitelinks-wikipedia
[16:05:12] <twkozlowski>	 Too many connections, it seems.
[16:05:13] <twkozlowski>	 :-D
[16:05:17] <greg-g>	 wfm
[16:08:26] <twkozlowski>	 wfm2, only saying cause you like to know when things break
[16:08:52] <greg-g>	 twkozlowski: I do I do :)
[16:09:03] <aude>	 if it happens again, do let us know
[16:10:13] <greg-g>	 oh right, gold coast for them this week
[16:11:59] <hoo>	 damn, I missed it again :/
[16:34:24] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: Adding ability to compute change-based diffs. [operations/software] - 10https://gerrit.wikimedia.org/r/131495 
[16:57:28] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Move torrus to netmon1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/131499 
[17:00:13] <grrrit-wm>	 (03PS1) 10BryanDavis: Configure git submodules to rebase on update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131500 (https://bugzilla.wikimedia.org/61562) 
[17:01:21] <^d>	 <3
[17:01:26] <^d>	 bd808: ^
[17:01:45] * bd808  basks in ^d's adoration :)

[17:02:29] <grrrit-wm>	 (03CR) 10BryanDavis: Configure git submodules to rebase on update (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131500 (https://bugzilla.wikimedia.org/61562) (owner: 10BryanDavis)
[17:02:31] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Configure git submodules to rebase on update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131500 (https://bugzilla.wikimedia.org/61562) (owner: 10BryanDavis)
[17:02:38] <grrrit-wm>	 (03Merged) 10jenkins-bot: Configure git submodules to rebase on update [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131500 (https://bugzilla.wikimedia.org/61562) (owner: 10BryanDavis)
[17:02:48] <grrrit-wm>	 (03CR) 10Aaron Schulz: [C: 032] Revert "Increased htmlCacheUpdate throttle" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131408 (owner: 10Aaron Schulz)
[17:02:57] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Increased htmlCacheUpdate throttle" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131408 (owner: 10Aaron Schulz)
[17:03:09] <^d>	 bd808: We'll test it this week's deploy.
[17:03:29] <^d>	 In theory, could test it on currently deployed branches too.
[17:03:31] <^d>	 Let's do that.
[17:03:44] <bd808>	 Oh yeah. You're deploying this week
[17:04:28] <^d>	 Can we clean up 1.23wmf*s now?
[17:04:58] <bd808>	 On tin or in gerrit?
[17:05:14] <logmsgbot>	 !log aaron synchronized wmf-config/CommonSettings.php  'Revert "Increased htmlCacheUpdate throttle"'
[17:05:21] <morebots>	 Logged the message, Master
[17:05:34] <^d>	 The latter definitely, can turn them to tags.
[17:05:36] <bd808>	 On tin we need to keep branches for ~30 days after they are rolled off the 'pedias to support varnish caches
[17:05:56] * bd808  goes to dig up the email where he analyzed that

[17:06:06] <^d>	 Is that automated yet?
[17:06:09] <^d>	 Cleaning up old ones?
[17:06:45] <bd808>	 There's a cleanup script, but it's not hooked into anything. I was doing it manually during the Tuesday deploy window
[17:09:51] <^d>	 Ah ok.
[17:11:07] <bd808>	 ^d: I wrote some emails about the process in early March. They went to the old core-l, ops-l and engineering.
[17:13:20] <^d>	 bd808: Updated currently deployed branches wmf[23]
[17:13:35] <^d>	 With the submodule rebase thingie.
[17:13:44] <bd808>	 oh cool
[17:45:49] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[18:02:51] <AaronSchulz>	 ori: is gdash broken?
[18:21:43] <grrrit-wm>	 (03CR) 10Umherirrender: [C: 031] Create 'noratelimit' user group on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130809 (https://bugzilla.wikimedia.org/57819) (owner: 10Withoutaname)
[18:41:45] <grrrit-wm>	 (03PS1) 10Chad: Remove $wmgCirrusIsBuilding. Cirrus is always on everywhere now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131513 
[18:57:40] <grrrit-wm>	 (03PS4) 10BBlack: Use whole subnets in squid.php list for XFF acceptance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130634 
[18:58:06] <bblack>	 ^ going to push this through tin in a minute, if no objections
[18:58:21] <bblack>	 (to the timing of the deploy, I mean!)
[18:59:25] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] Use whole subnets in squid.php list for XFF acceptance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130634 (owner: 10BBlack)
[18:59:30] <paravoid>	 :)
[18:59:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1585.733276  
[18:59:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1427.366699  
[19:00:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1336.033325  
[19:00:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1084.266724  
[19:01:03] <bblack>	 ?? %
[19:01:06] <bblack>	 ^ even
[19:01:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1315.866699  
[19:01:45] <paravoid>	 bblack: have you done a mw-config deploy before?
[19:01:47] <logmsgbot>	 !log bblack updated /a/common to {{Gerrit|I5a2d86ef0}}: Use whole subnets in squid.php list for XFF acceptance
[19:01:48] <grrrit-wm>	 (03PS5) 10Dzahn: full sudo ALL for jkrauska on pmacct (rhenium) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 
[19:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1172.400024  
[19:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1231.133301  
[19:01:54] <morebots>	 Logged the message, Master
[19:01:55] <bblack>	 paravoid: kinda, I'm following the wiki!
[19:02:06] <paravoid>	 :)
[19:02:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1163.766724  
[19:02:53] <bblack>	 what's with the kafka stuff?
[19:02:59] <paravoid>	 should be pretty acurate, ask if you have any questions
[19:03:00] <paravoid>	 who knows :)
[19:03:42] <bblack>	 I basically followed this: https://wikitech.wikimedia.org/wiki/How_to_do_a_configuration_change#In_your_own_repo_via_gerrit
[19:03:45] <logmsgbot>	 !log bblack synchronized wmf-config/squid.php  'Update wgSquidServersNoPurge to use whole subnets for XFF checking'
[19:03:49] <bblack>	 seems to be working fine so far
[19:03:51] <morebots>	 Logged the message, Master
[19:04:03] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] full sudo ALL for jkrauska on pmacct (rhenium) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 (owner: 10Dzahn)
[19:04:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:04:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:04:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:04:48] <paravoid>	 bblack: nope!
[19:04:58] <paravoid>	 bblack: revert
[19:05:10] <paravoid>	 http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=cpu_report is bad enough
[19:05:16] <paravoid>	 but http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=API+application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=cpu_report is really bad
[19:05:26] <mark>	 yep, revert
[19:05:36] <grrrit-wm>	 (03PS1) 10BBlack: Revert "Use whole subnets in squid.php list for XFF acceptance" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131519 
[19:05:43] <paravoid>	 heh :/
[19:05:50] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] Revert "Use whole subnets in squid.php list for XFF acceptance" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131519 (owner: 10BBlack)
[19:05:57] <grrrit-wm>	 (03CR) 10Dzahn: "Admins::Pmacct/Sudo_user[jkrauska]/File[/etc/sudoers.d/jkrauska]/ensure...." [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 (owner: 10Dzahn)
[19:05:59] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Use whole subnets in squid.php list for XFF acceptance" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131519 (owner: 10BBlack)
[19:05:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:06:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:06:24] <paravoid>	 bd808: ^^^ fwiw :)
[19:06:27] <bblack>	 yeah I saw the CPU too
[19:06:39] <bd808>	 blerg
[19:06:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:07:14] <Eloquence>	 fyi - http://blog.wikimedia.org/2014/05/05/wikimedia-foundation-selects-cyrusone-in-dallas-as-new-data-center/
[19:07:16] <logmsgbot>	 !log bblack updated /a/common to {{Gerrit|Iaf4d57d54}}: Revert "Use whole subnets in squid.php list for XFF acceptance"
[19:07:21] <bblack>	 the merge step takes forever :P
[19:07:23] <morebots>	 Logged the message, Master
[19:07:35] <bd808>	 paravoid: So the existing CIDR handling in core it too slow to use this way I guess
[19:07:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:07:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:08:00] <bblack>	 I could trim the CIDR list too, there are several currently-unused subnets in it
[19:08:01] <paravoid>	 Eloquence: \o/
[19:08:13] <bblack>	 but... checking an IP against a few CIDR ranges shouldn't be hard to do efficiently :P
[19:08:31] <paravoid>	 bblack: even if you cut it by half it'd still be pretty bad
[19:08:33] <bblack>	 (modulo php being php and all that)
[19:08:39] <paravoid>	 yeah you would know wouldn't you :P
[19:08:46] <logmsgbot>	 !log bblack synchronized wmf-config/squid.php  'REVERT: Update wgSquidServersNoPurge to use whole subnets for XFF checking'
[19:08:52] <morebots>	 Logged the message, Master
[19:09:01] <bd808>	 bblack: Agreed. But the code in core that I was leveraging is ... chock full of regex madness
[19:09:26] <paravoid>	 php5-gdnsd
[19:09:46] <bblack>	 it's kind of awesome when someone engineers this brilliant binary tree address space for efficient routing and filtering implementations, and then someone uses regexes on them :)
[19:09:47] <RobH>	 !log disabled puppet on cp40XX, ssl10XX, and ssl30XX
[19:09:53] <morebots>	 Logged the message, RobH
[19:10:17] <RobH>	 paravoid: I'm going to commit the known good key to the private repo and then merge my other patchset (after removing mark as reviewer since we fixed that issue)
[19:10:18] <bblack>	 php5-libgdmaps!
[19:10:40] <RobH>	 figured may as well review it in public channel, its not a security issue =]
[19:10:50] <paravoid>	 Eloquence, guillom: the link to the second photo on the dallas blog post is broken
[19:11:04] <guillom>	 paravoid: damn; thanks; fixing now
[19:11:06] <bblack>	 so, basically a 20% cpu jump: http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=cpu_report
[19:11:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1052 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1543.900024  
[19:11:27] <paravoid>	 also see the api appservers, that was way worse
[19:11:54] <guillom>	 paravoid: Fixed!
[19:11:57] <bblack>	 wow, like 40% there
[19:12:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1569.633301  
[19:12:22] <paravoid>	 :)
[19:13:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1340.0  
[19:13:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1068 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1625.966675  
[19:14:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1499.133301  
[19:14:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1491.233276  
[19:14:35] <bblack>	 bd808: do you have a quick pointer to where the php code for the network-checking is?
[19:14:40] <bd808>	 bblack, paravoid: I updated the bug I opened for the original CPU spike. This is fixable by either changing IP::isInRange() or by adding a more performant alternate code path.
[19:14:47] <grrrit-wm>	 (03CR) 10RobH: [C: 032] "I've addressed the private key concern with Mark already. He was right, and the key had been accidentally updated by applying the newest " [operations/puppet] - 10https://gerrit.wikimedia.org/r/130797 (owner: 10RobH)
[19:14:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1093.93335  
[19:15:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1495.666626  
[19:15:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq60 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1517.266724  
[19:15:42] <RobH>	 paravoid: Ok, I've merged my key and cert update.  any preference on where its updated first?
[19:15:49] <paravoid>	 nope
[19:15:59] <paravoid>	 is ottomata in zurich yet?
[19:16:04] <dr0ptp4kt>	 bblack, you around?
[19:16:08] <bd808>	 bblack: https://github.com/wikimedia/mediawiki-core/blob/master/includes/utils/IP.php#L709
[19:16:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1202.266724  
[19:16:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1341.833374  
[19:16:14] <paravoid>	 maybe we should page him for all the varnishkafka errors
[19:16:18] <paravoid>	 I'll do that
[19:16:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1052 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:17:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1363.0  
[19:17:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1643.93335  
[19:17:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1304.233276  
[19:17:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:17:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1216.466675  
[19:17:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq48 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1556.666626  
[19:18:05] <RobH>	 !log depooling ssl1001 to test new certs live on system
[19:18:12] <morebots>	 Logged the message, RobH
[19:18:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1780.199951  
[19:18:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:18:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1531.633301  
[19:18:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1127.5  
[19:18:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1068 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:18:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:18:52] <bblack>	 bd808: maybe we could do something like the first user-comment here? http://www.php.net/manual/en/ref.network.php
[19:19:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1447.43335  
[19:19:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq61 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1332.699951  
[19:19:10] <bblack>	 (but with v6 support)
[19:19:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:19:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:19:23] <paravoid>	 paged otto
[19:20:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:20:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq60 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:20:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1465.666626  
[19:20:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1650.366699  
[19:21:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:21:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:21:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:21:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:21:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq48 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:21:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1249.0  
[19:22:05] <RobH>	 arggghhhh goddamn newlines
[19:22:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:22:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:22:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 327.866669  
[19:22:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1060 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 101.633331  
[19:23:09] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:23:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 360.0  
[19:23:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq61 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:23:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:23:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1802.533325  
[19:23:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1567.099976  
[19:23:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:23:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1138.56665  
[19:23:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:23:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1121.06665  
[19:24:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 334.799988  
[19:24:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 350.166656  
[19:24:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1688.400024  
[19:24:22] <grrrit-wm>	 (03PS1) 10RobH: missing newline on unified cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/131528 
[19:24:27] <bblack>	 bd808: I'm going to play with the PHP code offline and do some benchmarking and see if I can replace all that with something faster and more efficient, it may take a little while
[19:24:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:24:40] <grrrit-wm>	 (03CR) 10RobH: [C: 032 V: 032] missing newline on unified cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/131528 (owner: 10RobH)
[19:24:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:24:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1059 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 107.833336  
[19:24:50] <bd808>	 bblack: Cool. Let me know if I can help.
[19:25:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 41.933334  
[19:25:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4003 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 376.333344  
[19:25:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1290.43335  
[19:25:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1489.866699  
[19:25:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1046 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 107.900002  
[19:25:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 890.799988  
[19:25:56] <grrrit-wm>	 (03PS1) 10Dzahn: add shell account for Monte Hurd [operations/puppet] - 10https://gerrit.wikimedia.org/r/131529 
[19:26:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1158.366699  
[19:26:52] <bd808>	 bblack: The check that is going crazy is https://github.com/wikimedia/mediawiki-core/blob/master/includes/GlobalFunctions.php#L4167 . If changing IP::isInRange() seems too scary/invasive a new check just for this usage could be added.
[19:26:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:27:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4004 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 318.399994  
[19:27:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:28:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:28:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:28:24] <bblack>	 bd808: I think my basic plan is to "precompile" the configured list into some prepped integer stuff (for IPs, CIDRs, explicit ranges), and then have a much faster replacement for isInRange() that operates on that prepped data
[19:28:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4001 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 370.933319  
[19:28:53] <bblack>	 probably not a real tree or anything, just something that doesn't involve a ton of text parsing every check
[19:28:58] <RobH>	 paravoid: would you be so kind as to check ssl1001.wikimedia.org and let me know if it passes your chain test
[19:29:15] <RobH>	 i get the same results on it i get on the unpatched ssl1002, so i'd think its ok, but perhaps i need to test further items
[19:29:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 400.766663  
[19:29:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:29:35] <RobH>	 im just reviewing the openssl s_client connect
[19:29:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:30:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:30:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 465.600006  
[19:30:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:30:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 357.633331  
[19:31:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 51.900002  
[19:32:26] <paravoid>	 RobH: that's the old cert
[19:32:39] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 1 below the confidence bounds  
[19:32:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:32:51] <RobH>	 ...eh?
[19:33:19] <RobH>	 i merged it and saw it update, checking
[19:33:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1497.833374  
[19:33:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1052 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1418.033325  
[19:33:25] <paravoid>	 you haven't restarted nginx though
[19:33:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:33:40] <bd808>	 bblack: Just caching the output of IP::parseRange() in the php process might make a big difference.
[19:33:44] <RobH>	 i thought it did so automatically at the time of replacement of main file
[19:33:45] <RobH>	 lemme do so
[19:33:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1068 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1567.900024  
[19:33:50] <paravoid>	 no, on purpose :)
[19:33:54] <mutante>	 greg-g: Reedy "dologmsg" doesn't work anymore on tin?
[19:33:57] <RobH>	 my bad =P
[19:34:00] <paravoid>	 so that when it breaks, it doesn't break spectacularly ;)
[19:34:04] <logmsgbot>	 mutante tested dologmsg script on tin
[19:34:09] <mutante>	 oh, heh, it does
[19:34:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 47.866665  
[19:34:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1411.5  
[19:34:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1275.533325  
[19:34:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 45.366665  
[19:34:26] <paravoid>	 jgage: ping?
[19:34:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1363.266724  
[19:34:57] <paravoid>	 is gage at the office?
[19:35:00] <RobH>	 paravoid: yes, but NOW its the new cert
[19:35:02] <RobH>	 he wasnt no
[19:35:05] <mutante>	 paravoid: no
[19:35:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1365.833374  
[19:35:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1549.733276  
[19:35:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1212.400024  
[19:35:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq60 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1265.93335  
[19:35:29] <RobH>	 paravoid: and also seems to be working
[19:35:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1430.333374  
[19:35:39] <icinga-wm>	 PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 993.790875712  
[19:35:53] <RobH>	 if you confirm i'll start repooling and depooling them in ones or twos and rolling the update
[19:36:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1213.0  
[19:36:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1526.0  
[19:36:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1229.93335  
[19:36:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1129.099976  
[19:36:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1503.599976  
[19:36:35] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "there are no more "home-mounted Apaches" and dologmsg works fine on tin" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130600 (owner: 10Dzahn)
[19:36:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq48 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1450.866699  
[19:37:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1357.333374  
[19:37:12] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq61 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1513.266724  
[19:37:12] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1439.366699  
[19:37:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1372.06665  
[19:37:40] <paravoid>	 RobH: looks good
[19:37:48] <RobH>	 huzzzzzzaaaaahhhhhhhh
[19:38:06] <RobH>	 i'll repool ssl1001 and then depool the rest of eqiad in pairs (seems ok to you?)
[19:38:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1052 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:38:22] <RobH>	 except esams, thats one by one
[19:38:25] <RobH>	 there arent many there
[19:38:29] <paravoid>	 yup, sounds fine
[19:38:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1266.400024  
[19:38:41] <RobH>	 cool thanks
[19:38:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1068 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:38:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1360.033325  
[19:39:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:39:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:39:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:39:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:39:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:40:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:40:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:40:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:40:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:40:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1243.333374  
[19:41:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1344.233276  
[19:41:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1458.466675  
[19:41:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:41:20] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1570.466675  
[19:41:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq60 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:41:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq48 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:42:38] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq61 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:42:38] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:42:38] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:42:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:43:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:43:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1540.133301  
[19:43:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1647.466675  
[19:43:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:44:41] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:45:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:45:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:46:00] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:46:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:46:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1179.93335  
[19:47:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:47:54] <paravoid>	 can someone page jgage about all that
[19:48:06] <paravoid>	 otto hasn't responded
[19:48:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4003 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:48:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:48:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1034.366699  
[19:49:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:50:12] <cajoel>	 paravoid: only XFPs here
[19:50:31] <cajoel>	 paravoid: I thought we were going to connect to a switch in ulsfo (not mx)
[19:50:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:50:40] <paravoid>	 yes
[19:50:42] <paravoid>	 that's correct
[19:50:56] <cajoel>	 switch doesn't have 10/100/1000 ports?
[19:51:13] <cajoel>	 fiber problem...
[19:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4004 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:51:27] <kaldari>	 Would any ops people be available to meet with Ray King at the office today at 3pm? He's the guy that owns the .wiki TLD.
[19:51:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 830.833313  
[19:51:52] <paravoid>	 have an SFP fiber embedded into the chassis? :)
[19:52:12] <cajoel>	 media converter a bad idea?
[19:52:18] <cajoel>	 what's the switch in ulsfo?
[19:52:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:52:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:52:23] <kaldari>	 mutante: ^ ?
[19:52:25] <cajoel>	 it has SFP ports?
[19:52:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4001 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:53:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1692.466675  
[19:53:22] <paravoid>	 we have an EX4500 and an EX4550
[19:53:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1432.56665  
[19:53:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1080.300049  
[19:54:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1496.43335  
[19:54:16] <mutante>	 kaldari: ?
[19:54:18] <cajoel>	 do you want a juniper brand SFP or do you use third party?
[19:54:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:54:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1052 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1392.233276  
[19:54:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1068 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1348.5  
[19:55:01] <mutante>	 kaldari: if it's about the Varnishkafka i believe Ottomata has already been paged
[19:55:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1160.56665  
[19:55:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1011.666687  
[19:55:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:55:57] <paravoid>	 cajoel: third parties are fine, do you have spares?
[19:56:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1151.466675  
[19:56:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq61 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1212.033325  
[19:56:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1320.56665  
[19:56:11] <cajoel>	 paravoid: only found XFPs in the boxes here.
[19:56:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1354.599976  
[19:56:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 700.400024  
[19:56:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1532.133301  
[19:56:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq60 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1266.599976  
[19:56:22] <paravoid>	 ok, so we'll make an order
[19:56:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:56:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1359.266724  
[19:56:48] <paravoid>	 I'll file an RT
[19:56:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1082.333374  
[19:56:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq48 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1203.199951  
[19:57:09] <paravoid>	 but before I do that, can you confirm with reliance that they'll terminate in a 1Gbps SFP
[19:57:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1444.333374  
[19:57:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1220.866699  
[19:57:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1014.56665  
[19:57:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:57:21] <paravoid>	 and whether we'll use one or two? we can order two of course, they're going to be cheap
[19:57:34] <cajoel>	 should be <$100
[19:57:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:57:43] <paravoid>	 yup
[19:57:48] <cajoel>	 I'd order 2, just so we have a backup
[19:57:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1164.133301  
[19:57:52] <cajoel>	 I'll call reliance right now..
[19:58:08] <cajoel>	 double check that the 4500 doesn't have any SFPs in it already?
[19:58:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1065.733276  
[19:58:13] <RobH>	 !log ssl1001 back in service, ssl1002-1003 set to disabled in pybal
[19:58:18] <morebots>	 Logged the message, RobH
[19:58:47] <paravoid>	 faidon@asw-ulsfo> show chassis hardware | match SFP | except SFP\+  Xcvr 16      REV 01   740-013113   BYJOHNSJ221987C   SFP-T Xcvr 27      REV 01   740-013113   BYJOHNSJ221987D   SFP-T Xcvr 18      REV 01   740-013113   BYJOHNSJ221987A   SFP-T Xcvr 20      REV 01   740-013113   BYJOHNSJ221987B   SFP-T
[19:58:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:58:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:58:54] <paravoid>	 it's full of XFPs :)
[19:58:58] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] add shell account for Monte Hurd [operations/puppet] - 10https://gerrit.wikimedia.org/r/131529 (owner: 10Dzahn)
[19:59:02] <paravoid>	 (ironically)
[19:59:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1367.866699  
[19:59:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1268.766724  
[19:59:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1237.633301  
[20:00:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:00:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1468.633301  
[20:00:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1438.266724  
[20:00:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1052 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:29] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:39] <grrrit-wm>	 (03PS1) 10Dzahn: add account mhurd to stat1003 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131537 
[20:01:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:40] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1695.133301  
[20:01:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq48 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1068 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:01:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq61 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:20] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1558.366699  
[20:02:21] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq60 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:02:32] <subbu>	 greg-g, getting ready to deploy parsoid. .. any reason to wait?
[20:02:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:03:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:03:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:03:19] <grrrit-wm>	 (03PS2) 10Dzahn: add account mhurd to stat1003 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131537 
[20:03:35] <greg-g>	 subbu: don't think so, I've grown to ignore varnichkhafka errors unfortunately :/
[20:03:42] <subbu>	 ok. 
[20:03:52] * subbu  logs on to bastion

[20:04:15] <paravoid>	 cajoel: 4500/4550 doesn't support 100BASE SFPs
[20:04:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:04:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:04:41] <paravoid>	 cajoel: only the 4200 does; so please convince them to do GigE
[20:04:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:05:04] <cajoel>	 paravoid: well nasty --- so get a 1G no matter what, and if we can't get GigE, we'll have to do some sort of mediuaconverte/cheapo switch anyway (ugly)
[20:05:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:05:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:05:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:06:03] <paravoid>	 cajoel: they said okay for a single 1gbps port in your initial mail, so I see no reason why not
[20:06:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:07:24] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "had approval, waiting period over" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131537 (owner: 10Dzahn)
[20:07:46] <paravoid>	 cajoel, RobH: filed procurement request RT #7416
[20:08:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1047 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 119.833336  
[20:08:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:08:19] <kaldari>	 mutante: WOould you be available to meet with Ray King at the office today at 3pm? He's the guy that owns the .wiki TLD.
[20:08:44] <kaldari>	 would be good to have someone from ops at the meeting
[20:09:16] <grrrit-wm>	 (03PS1) 10Dr0ptp4kt: Test support for Nokia proxy fronting assigned cellular IP for one subdomain. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131540 
[20:09:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1709.966675  
[20:09:27] <mutante>	 kaldari: oh.. i had no idea he was coming to office.. if you can tell me what the plan is, sure
[20:09:36] <mutante>	 kaldari: can we meet at office before?
[20:10:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 225.833328  
[20:10:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1211.300049  
[20:10:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 248.53334  
[20:10:28] <kaldari>	 mutante: sure, I can meet with you now or whenever. Meeting with Ray is scheduled for 3-4pm. I'll add you to the invite.
[20:10:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1434.266724  
[20:11:04] <grrrit-wm>	 (03CR) 10Dzahn: "Mhurd/Unixaccount[Monte Hurd]/User[mhurd]/ensure: created.." [operations/puppet] - 10https://gerrit.wikimedia.org/r/131537 (owner: 10Dzahn)
[20:11:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4003 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 195.666672  
[20:11:58] <paravoid>	 RobH: how is it going?
[20:13:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4004 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 222.833328  
[20:13:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 69.599998  
[20:13:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4001 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 211.300003  
[20:14:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:14:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:14:43] <RobH>	 sorry got a phone call, im updating ssl1002-1003 now
[20:14:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 93.099998  
[20:15:17] <subbu>	 !log deployed parsoid f2f1f1d7 (with deploy sha 71072f8a)
[20:15:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 184.199997  
[20:15:23] <morebots>	 Logged the message, Master
[20:15:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1384.43335  
[20:15:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:15:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1009.966675  
[20:16:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1371.666626  
[20:17:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq61 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1302.266724  
[20:17:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1538.166626  
[20:17:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1792.533325  
[20:17:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:17:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1316.133301  
[20:17:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1052 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1144.866699  
[20:17:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 31.9  
[20:17:40] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 200.666672  
[20:17:41] <paravoid>	 mutante/robh: can you call gage and tell him about all this varnishkafka stuff?
[20:17:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq48 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1430.466675  
[20:17:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1282.099976  
[20:18:08] <RobH>	 !log putting ssl1002/3 back into service
[20:18:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1287.866699  
[20:18:15] <morebots>	 Logged the message, RobH
[20:18:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1095.866699  
[20:18:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1054.366699  
[20:18:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1439.599976  
[20:18:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1182.533325  
[20:18:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 973.733337  
[20:18:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1068 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1264.766724  
[20:18:53] <RobH>	 paravoid: i just texted him
[20:18:59] <paravoid>	 thx
[20:18:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1383.866699  
[20:19:00] <RobH>	 if he doesnt reply to text shortly i'll call
[20:19:00] <mutante>	 paravoid: calling
[20:19:04] <mutante>	 oops
[20:19:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1081.633301  
[20:19:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1184.5  
[20:19:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1150.900024  
[20:19:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1365.5  
[20:19:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 143.766663  
[20:19:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1464.666626  
[20:19:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq60 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1165.866699  
[20:19:21] <RobH>	 i had him in my recent texts from him texting me about his dropping connection in meeting
[20:19:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:19:55] <RobH>	 !log ssl1004/5 disabled for update
[20:20:02] <morebots>	 Logged the message, RobH
[20:20:06] <mutante>	 paravoid: he'll look, talked to him
[20:20:08] <RobH>	 bah, depooled, whatevs
[20:20:14] <jgage>	 hi
[20:20:30] <jgage>	 weird rob, i just got your text now
[20:20:39] <^d>	 mutante: Sooo, silver as the formey ldap admin box replacement. I can't ssh to it because my account isn't on it.
[20:20:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 903.233337  
[20:20:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1470.800049  
[20:20:49] <^d>	 (we tried to fix this friday but broke)
[20:21:40] <RobH>	 !log puppet agent has been re-enabled on ssl1001-1003
[20:21:47] <morebots>	 Logged the message, RobH
[20:21:53] <RobH>	 jgage: yay multi-carrier text delays =]
[20:22:10] <jgage>	 yeah
[20:22:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1269.233276  
[20:22:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:22:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:22:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1333.766724  
[20:22:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1622.56665  
[20:22:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 980.633362  
[20:23:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq61 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:23:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:23:47] <greg-g>	 so, about the varnichkhafka spam....
[20:23:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq48 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:23:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:23:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:00] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:00] <greg-g>	 ahh, I see you're pinging gage
[20:24:03] <jgage>	 sorry, cause not yet identified
[20:24:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:18] <jgage>	 can we make icinga shut up temporarily?
[20:24:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:20] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1052 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1068 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:24:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq60 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:27] <RobH>	 !log ssl1004/5 returned to service (and puppet agents enabled)
[20:25:34] <morebots>	 Logged the message, RobH
[20:25:39] <RobH>	 !log depooled ssl1006/7 for update
[20:25:47] <morebots>	 Logged the message, RobH
[20:25:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:25:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:26:16] <RobH>	 paravoid: Bahhhhh
[20:26:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1551.233276  
[20:26:24] <paravoid>	 what?
[20:26:25] <RobH>	 so somehow the puppet agent disable systems called in
[20:26:37] <RobH>	 they have the udpated unified cert (but not chained since i didnt get to them yet)
[20:26:42] <paravoid>	 you don't have to disable puppet anymore
[20:26:43] <RobH>	 it doesnt restart nginx, so we should be ok
[20:26:47] <paravoid>	 exactly
[20:27:11] <RobH>	 well, then im just going to salt remove the chain on them all and force puppet runs, since nginx doesnt restart there isnt a good reason for me to do one by one anymore right?
[20:27:18] <RobH>	 (not doing that until you confirm mind you)
[20:27:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:27:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:28:23] <RobH>	 the only thing i need to manually do is restart nginx after depooling them and draining traffic to each
[20:28:27] <RobH>	 seems to me.
[20:28:30] <RobH>	 well, one by one.
[20:28:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1464.5  
[20:29:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:29:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:30:15] * AaronSchulz  wonders why Template:WPBannerMeta is so slow

[20:30:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:30:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:31:20] <grrrit-wm>	 (03CR) 10Hashar: [C: 031 V: 032] "Deployed on contint puppetmaster and proven to be working :]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131438 (owner: 10Hashar)
[20:31:24] <RobH>	 well, now its not all, its just ssl1006-1009, so just rm'd the old chain file and running puppet on them.  1006/7 are already depooled so once thats done i'll restart their nginx and test them, repool
[20:32:00] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "Seems to be working on labs :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/129292 (owner: 10Hashar)
[20:32:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1413.866699  
[20:33:05] <Nemo_bis>	 AaronSchulz: is that the template for CN magic (disabling banners on multiple sites and the like)?
[20:33:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:34:26] <RobH>	 hrmm, ssl1006 is being bitchy.
[20:34:37] <RobH>	 i'll flag it and come back since ssl1007 is fine with same exact steps
[20:34:50] <RobH>	 !log ssl1007 going back into service, ssl1008 depooling
[20:34:57] <morebots>	 Logged the message, RobH
[20:35:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1813.733276  
[20:35:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1179.800049  
[20:36:05] <jgage>	 gah
[20:36:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1261.56665  
[20:37:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1392.5  
[20:38:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq61 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1333.966675  
[20:38:15] <jgage>	 !log forced kafka broker reelection
[20:38:22] <morebots>	 Logged the message, Master
[20:38:33] <grrrit-wm>	 (03PS1) 10Dzahn: add mhurd to bast1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131592 
[20:38:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:38:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1060 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:38:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq48 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1347.733276  
[20:38:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2547.100098  
[20:39:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2819.866699  
[20:39:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3146.133301  
[20:39:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:39:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2131.399902  
[20:39:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3275.333252  
[20:39:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3136.266602  
[20:39:39] <icinga-wm>	 RECOVERY - Kafka Broker Messages In on analytics1021 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 1925.8109877  
[20:39:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1059 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:39:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1046 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:39:55] <RobH>	 !log ssl1008 back into service, ssl1009 already depooled
[20:40:00] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:40:02] <morebots>	 Logged the message, RobH
[20:40:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2584.93335  
[20:40:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2216.533447  
[20:40:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:40:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1759.599976  
[20:40:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1762.866699  
[20:41:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2375.399902  
[20:41:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2004.599976  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2196.666748  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1633.533325  
[20:41:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3310.5  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq60 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2147.199951  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3060.56665  
[20:41:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1052 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1912.233276  
[20:41:19] <RobH>	 interesting, ssl1009 has same refusal error as ssl1006 (but wasnt pooled before)
[20:41:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2714.699951  
[20:41:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:41:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2013.599976  
[20:41:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1068 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2204.300049  
[20:41:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:42:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq61 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:42:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:42:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:42:22] <mutante>	 ^d: re: LDAP access. i don't think you had that access before.. just robla,reedy,sumanah. https://gerrit.wikimedia.org/r/#/c/125721/8/manifests/site.pp
[20:42:36] <mutante>	 ^d: when it was on formey i mean
[20:42:46] <mutante>	 ^d: oh, i see, because you had root on formey for gerrit?
[20:42:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq48 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:00] <RobH>	 !log ssl1009 was refusing connections both before and after my ssl cert update.  ssl1006 is presently refusing connections post update.  they are set to disabled in pubal
[20:43:03] <^d>	 mutante: Yeah I did because it was formey.
[20:43:04] <RobH>	 !log pybal
[20:43:06] <morebots>	 Logged the message, RobH
[20:43:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:11] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1651.366699  
[20:43:11] <morebots>	 Logged the message, RobH
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:43:22] <mutante>	 ^d: looking ..
[20:43:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1005.06665  
[20:44:11] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:44:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:44:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:44:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1428.166626  
[20:44:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq60 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:44:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:44:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2076.899902  
[20:44:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:45:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:45:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:45:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1052 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:45:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:45:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1068 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:46:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4004 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:46:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2113.899902  
[20:46:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:46:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 989.06665  
[20:46:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:46:49] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[20:47:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4003 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:47:39] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected  
[20:47:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:48:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:48:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:48:50] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4001 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:48:54] <jgage>	 sorry for the icinga spam folks, i'd turn it off if i knew how
[20:48:59] <jgage>	 restarting kafka brokers now
[20:49:00] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1046 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 37.166668  
[20:49:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:49:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1059 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 42.333332  
[20:49:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:50:16] <grrrit-wm>	 (03PS1) 10Dzahn: include accounts in LDAP admin role, add demon [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 
[20:50:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:50:39] <RobH>	 !log ssl1006 and ssl1009 are responsive to nginx and back in service
[20:50:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:50:46] <morebots>	 Logged the message, RobH
[20:50:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1060 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6076.366699  
[20:50:59] <grrrit-wm>	 (03PS2) 10Dzahn: add mhurd to bast1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/131592 
[20:51:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1047 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1060 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1059 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:51:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1046 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:52:05] <grrrit-wm>	 (03PS2) 10Manybubbles: WIP: Add some analyzers to labs [operations/software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/130969 
[20:52:19] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on analytics1021 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 44.0  
[20:52:50] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "..or he can't get to stat1003" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131592 (owner: 10Dzahn)
[20:52:56] <jgage>	 hrm i can't get it to reelect analytics1022
[20:54:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 307.866669  
[20:54:22] <RobH>	 !log cp4001-4020 unified cert and nginx service reloaded, back in service 
[20:54:24] <grrrit-wm>	 (03CR) 10Chad: "I have no idea how gwtorm behaves here." [operations/puppet] - 10https://gerrit.wikimedia.org/r/131419 (owner: 10Springle)
[20:54:29] <morebots>	 Logged the message, RobH
[20:55:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:55:47] <grrrit-wm>	 (03PS3) 10Manybubbles: WIP: Add some analyzers to labs [operations/software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/130969 
[20:56:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 268.799988  
[20:58:10] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[20:58:19] <icinga-wm>	 PROBLEM - Kafka Broker Messages In on analytics1022 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 5.56206809584  
[20:58:19] <jgage>	 !log both kafka brokers back in service
[20:58:21] <RobH>	 !log ssl1001-1003 now have updated unified cert in service
[20:58:26] <morebots>	 Logged the message, Master
[20:58:32] <morebots>	 Logged the message, RobH
[20:58:45] <jgage>	 how come it calls you robh and me master?
[20:59:07] <^d>	 People hack the script to make it say funny things.
[20:59:11] <RobH>	 yep
[20:59:32] <RobH>	 leslie and notpeter have custom messages
[20:59:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 646.799988  
[20:59:47] <se4598>	 RobH: wikipedia now has new certs with old values but new fingerprint only?
[20:59:58] <RobH>	 just added zero to SANs
[21:00:05] <jgage>	 ok well i have confirmed that the kafka brokers are both acting as expected, time to drill down..
[21:00:08] <RobH>	 for *.zero.wikipedia.org support
[21:00:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4004 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 435.966675  
[21:00:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 428.399994  
[21:00:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1046 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 504.0  
[21:00:50] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 92.400002  
[21:01:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 512.400024  
[21:01:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4003 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 411.600006  
[21:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4001 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 319.200012  
[21:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1059 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 445.200012  
[21:01:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 646.799988  
[21:02:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1901.400024  
[21:02:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 352.799988  
[21:02:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 546.0  
[21:02:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 554.400024  
[21:02:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1060 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 504.0  
[21:03:10] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 520.799988  
[21:03:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 781.200012  
[21:03:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 596.400024  
[21:04:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1047 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 392.933319  
[21:04:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 495.600006  
[21:04:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:04:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1503.599976  
[21:04:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:06:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:06:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:06:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:06:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:06:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1069 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 714.0  
[21:06:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 184.800003  
[21:07:19] <icinga-wm>	 RECOVERY - Kafka Broker Messages In on analytics1022 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 1592.54232073  
[21:07:19] <icinga-wm>	 RECOVERY - Kafka Broker Under Replicated Partitions on analytics1021 is OK: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value OKAY: 0.0  
[21:07:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:07:35] <grrrit-wm>	 (03PS1) 10EBernhardson: Drop wgFlowCacheKey from CommonSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131609 
[21:07:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 277.200012  
[21:07:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:07:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:07:40] <jgage>	 !log trying on analytics1022: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Recovering_a_laggy_broker_replica
[21:07:48] <morebots>	 Logged the message, Master
[21:07:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1046 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:07:52] <grrrit-wm>	 (03CR) 10Chad: "Very similar to what we tried friday: https://gerrit.wikimedia.org/r/#/c/131010/ which didn't work." [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 (owner: 10Dzahn)
[21:07:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4004 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4003 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:29] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:08:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4001 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:09:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:10:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 113.933334  
[21:10:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1059 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:10:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1060 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:10:59] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1015.533325  
[21:11:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 257.600006  
[21:11:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 860.866638  
[21:11:39] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 782.733337  
[21:12:09] <grrrit-wm>	 (03CR) 10Dzahn: "should be the missing $gid = 500 and "wikidev" group, i just had that same issue with admins::pmacct" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 (owner: 10Dzahn)
[21:12:19] <icinga-wm>	 PROBLEM - Kafka Broker Messages In on analytics1022 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 0.0  
[21:12:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1047 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:12:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1321.266724  
[21:12:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 412.700012  
[21:12:49] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 229.899994  
[21:13:11] <grrrit-wm>	 (03PS2) 10Dzahn: include accounts in LDAP admin role, add demon [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 
[21:13:19] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 239.100006  
[21:14:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:14:19] <icinga-wm>	 RECOVERY - Kafka Broker Messages In on analytics1022 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 2039.08236657  
[21:14:59] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:15:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:15:29] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:15:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:15:39] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:16:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:16:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:16:49] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1069 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:17:02] <jgage>	  now stay that way, dammit
[21:17:40] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:19:21] <greg-g>	 :)
[21:19:35] <mwalker>	 I created an instance in betalabs; and then realized I don't actually need it. Is there a 'production'esque way to remove the puppet and salt keys from the master? or is it just dropping the box off the face of the planet and having puppet/salt clean themselves automatically?
[21:20:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:20:59] <mwalker>	 ah; yes; found it on wikitech
[21:21:05] <hashar>	 \O/
[21:23:19] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[21:27:52] * jgage  works on varnishkafka incident report, while keeping an eye on it

[21:28:30] <greg-g>	 thanks much jgage 
[21:50:44] <renoirb_>	 	 Hi all, I'm wondering how you manage deployment of the wmf/1.NNwmfNN in relation to submodules
[21:51:34] <renoirb_>	 At webplatform.org we are running wmf/1.20 (ish)  and I'd like to make a better way to upgrade versions
[21:51:57] <renoirb_>	 (without re-engineering what's been figured out, or already in place)
[21:57:03] <^d>	 renoirb_: Lots of hang-wringing. We've got a script called make-wmf-branch which handles creating new wmf branches with their appropriate submodules.
[21:57:42] <^d>	 Then we'll update any submodules as needed during a cycle when the branch is live. Rather manual and painful.
[22:01:09] <renoirb_>	 ^d so, how do you deploy then. You have a builder VM and once its done, you rsync files?
[22:02:39] <^d>	 We've got a deployment box that we pull the repos to then we sync the files, yeah.
[22:14:41] <renoirb_>	 ^d can you give me a link to that script make-wmf-branch?
[22:14:58] <renoirb_>	 and/or any useful utility that are available for me to understand how you handle deployment, pls :)
[22:15:50] <^d>	 renoirb_: https://git.wikimedia.org/tree/mediawiki%2Ftools%2Frelease.git/HEAD/make-wmf-branch
[22:15:54] <^d>	 https://git.wikimedia.org/tree/mediawiki%2Ftools%2Frelease.git/HEAD/make-wmf-branch
[22:15:56] <^d>	 renoirb: ^
[22:16:38] <renoirb>	 oh, sweet! ^
[22:16:39] <renoirb>	 ^d
[22:20:35] <grrrit-wm>	 (03PS3) 10Chad: include accounts in LDAP admin role, add demon [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 (owner: 10Dzahn)
[22:22:27] <grrrit-wm>	 (03CR) 10Rush: "hey man this seems...awesome but I'm not sure how to review it, mainly because I'm not sure how exactly to use it? Can you share with me " [operations/software] - 10https://gerrit.wikimedia.org/r/131495 (owner: 10Giuseppe Lavagetto)
[22:32:07] <greg-g>	 ori: mwalker, fyi, I'll be afk during the beginning of swat. do your thing.
[22:42:18] <ori>	 mwalker: i can do it if you like
[22:42:38] <mwalker>	 ori, yes please; I'm stuck in a sprint review
[22:47:25] <renoirb>	 If I may ask for confirmation, If I get it correctly ^d, I could use make-wmf-branch config's default.conf to set what extension my site uses, right?
[22:50:36] <^d>	 renoirb: Should be able to :)
[22:56:03] <renoirb>	 I do not see anything to prepare a version for deployment though ^d 
[22:56:27] <renoirb>	 I guess i'd have to change the default behavior, isnt' it?
[22:56:33] <^d>	 Ah, that's another script that's 100% wmf-specific.
[22:56:49] <^d>	 checkoutMediaWiki. There's no way that'll work as-is for you.
[22:58:51] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Rename mflaschen to mattflaschen and change UID to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131624 
[23:00:28] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Rename mflaschen to mattflaschen and change UID to match labs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131624 (owner: 10Andrew Bogott)
[23:04:52] <dr0ptp4kt>	 bblack, when you have a minute, would you please review and, if appropriate, +2 https://gerrit.wikimedia.org/r/#/c/131540/ 
[23:04:54] <dr0ptp4kt>	 ?
[23:07:35] <grrrit-wm>	 (03PS2) 10BBlack: Test support for Nokia proxy fronting assigned cellular IP for one subdomain. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131540 (owner: 10Dr0ptp4kt)
[23:07:45] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] Test support for Nokia proxy fronting assigned cellular IP for one subdomain. [operations/puppet] - 10https://gerrit.wikimedia.org/r/131540 (owner: 10Dr0ptp4kt)
[23:08:07] <dr0ptp4kt>	 BBlack, thx
[23:08:11] <bblack>	 np
[23:08:40] <grrrit-wm>	 (03CR) 10Rush: [C: 04-1] "Just a few minor things, I end up with a package named something like: python-statsd_0.1.10-chasemp20140227-1_all.deb. Which based on the" [operations/debs/python-statsd] - 10https://gerrit.wikimedia.org/r/131449 (owner: 10Gage)
[23:14:06] <RoanKattouw>	 ori: You're doing SWAT today then?
[23:15:33] <grrrit-wm>	 (03PS4) 10Dzahn: include accounts in LDAP admin role, add demon [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 
[23:17:23] <ori>	 RoanKattouw: yes
[23:17:33] <ori>	 waiting on jenkins
[23:17:33] <RoanKattouw>	 Sweet
[23:17:37] <ori>	 doing ve next, i think
[23:18:53] <grrrit-wm>	 (03PS1) 10Dzahn: add w.wiki, link to wikipedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/131627 
[23:19:54] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "not actually adding new permissions, just finishing RT #6134 people had this on formey before" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 (owner: 10Dzahn)
[23:20:50] <grrrit-wm>	 (03CR) 10Krinkle: [C: 031] "LGTM. Makes no effective change (resulting setting remains the same)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131513 (owner: 10Chad)
[23:21:44] <grrrit-wm>	 (03CR) 10Chad: "I should probably have Nik +1 this as well. Only reason I see us needing it is some emergency fallback but we're almost past that point." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131513 (owner: 10Chad)
[23:22:11] <grrrit-wm>	 (03CR) 10Krinkle: "Plus, the original config setting is still there in CommonSettings.php if needed." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131513 (owner: 10Chad)
[23:22:25] <Krinkle>	 Or is that an anti-pattern?
[23:22:36] <Krinkle>	 wmg should only be needed per wiki I suppose.
[23:22:53] <^d>	 Hmm.
[23:23:29] <logmsgbot>	 !log ori synchronized php-1.24wmf2/extensions/EventLogging  'Update EventLogging for Id23b37fbe for SWAT.'
[23:23:31] <grrrit-wm>	 (03CR) 10Dzahn: "also see: https://gerrit.wikimedia.org/r/#/c/125721/8/manifests/site.pp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131597 (owner: 10Dzahn)
[23:23:34] <morebots>	 Logged the message, Master
[23:26:20] <logmsgbot>	 !log ori synchronized php-1.24wmf3/extensions/EventLogging  'Update EventLogging for Id23b37fbe for SWAT.'
[23:26:26] <morebots>	 Logged the message, Master
[23:44:05] <logmsgbot>	 !log ori Started scap: SWAT deploy for VisualEditor and Flow cherry-picks
[23:44:11] <morebots>	 Logged the message, Master
[23:47:49] <icinga-wm>	 PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Sat May  3 05:29:08 2014  
[23:48:49] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Mon May  5 20:48:34 2014  
[23:54:01] <logmsgbot>	 !log ori Finished scap: SWAT deploy for VisualEditor and Flow cherry-picks (duration: 09m 55s)
[23:54:08] <morebots>	 Logged the message, Master
[23:54:20] <ori>	 ^ RoanKattouw, ebernhardson
[23:54:25] <ori>	 ebernhardson: I still have to sync the config change
[23:54:26] <ori>	 doing that now
[23:54:38] <ebernhardson>	 ok, perfect
[23:54:49] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Drop wgFlowCacheKey from CommonSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131609 (owner: 10EBernhardson)
[23:55:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Drop wgFlowCacheKey from CommonSettings.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131609 (owner: 10EBernhardson)
[23:57:18] <logmsgbot>	 !log ori updated /a/common to {{Gerrit|Id1f2e0acf}}: Drop wgFlowCacheKey from CommonSettings.php
[23:57:25] <morebots>	 Logged the message, Master