[00:06:19] * Josve05a Krinkle: Might have been https://chrome.google.com/webstore/detail/ppmagkdpdaaabpmdppckemgkngkpchdl?utm_source=chrome-app-launcher-info-dialog
[00:06:32] <Josve05a>	 Was recomended by someone here along time ago I thing
[00:06:34] <Josve05a>	 think*
[00:09:57] <Josve05a>	 Nope, wasn't it
[00:10:37] <Josve05a>	 Krinkle: But it's only affecting FLow-pages on svwp, while I'm logged in
[00:11:15] <Krinkle>	 Josve05a: So youre saying opening https://sv.wikipedia.org/wiki/Wikipediadiskussion:Flow in Chrome while logged-in makes the page freeze and spin forever?
[00:11:57] <Josve05a>	 not freeze, I can still scroll, and however over thing and click links, but not type in Flow-message-fields or edit
[00:12:09] <Josve05a>	 hover* hower* ?
[00:12:19] <icinga-wm>	 PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds
[00:13:19] <RoanKattouw>	 I'm looking into it
[00:13:19] <Josve05a>	 https://www.irccloud.com/pastebin/3S2xGLxJ/
[00:13:51] <RoanKattouw>	 "loadingSpinner_kix-oribi-audio" 
[00:13:53] <RoanKattouw>	 WTF
[00:14:02] <RoanKattouw>	 Josve05a: Does that sound familiar? A browser plugin perhaps, or a gadget or user script?
[00:14:15] <Josve05a>	 ...
[00:14:18] <Josve05a>	 hmm...
[00:14:21] <Josve05a>	 GOD DAMN IT
[00:14:38] <Josve05a>	 something my school has installed on my chromebook, which I can not deactivate
[00:14:42] <Josve05a>	 sorry guys
[00:14:48] <RoanKattouw>	 :S
[00:14:50] <ori>	 haha
[00:14:59] <ori>	 I'm glad you got down to the bottom of it
[00:16:07] <greg-g>	 That sucks, Josve05a.
[00:16:21] <Josve05a>	 however, I've had it for a few months...
[00:16:32] <Josve05a>	 unsure why it's causing a hastle now
[00:16:39] <Josve05a>	 hassle* (sp?)
[00:16:45] <ori>	 hassle :)
[00:18:29] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1516845 (10thcipriani) > I hope to be wrong about this, but my fear is that things that aren't explicitly c...
[00:18:48] <icinga-wm>	 PROBLEM - Restbase endpoints health on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[00:18:58] <ori>	 gwicke: ^
[00:20:47] <icinga-wm>	 RECOVERY - Restbase endpoints health on xenon is OK: All endpoints are healthy
[00:24:08] <icinga-wm>	 RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 61446 bytes in 1.841 second response time
[00:30:29] <icinga-wm>	 PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds
[00:38:43] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1516868 (10GWicke) >  a general way to have a puppet run be part of the deploy process  As well as disablin...
[00:55:47] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Add apiserver class + init script [puppet] - 10https://gerrit.wikimedia.org/r/230019 
[00:56:56] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] k8s: Add apiserver class + init script [puppet] - 10https://gerrit.wikimedia.org/r/230019 (owner: 10Yuvipanda)
[00:57:11] <ori>	 k8s?
[00:57:57] <ori>	 oh, apparently it's some hipster word for kubernetes
[00:59:19] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Rename initscript to match expectation [puppet] - 10https://gerrit.wikimedia.org/r/230021 
[00:59:26] <logmsgbot>	 !log ori Synchronized php-1.26wmf17/includes/OutputPage.php: c0ca5700c6: resourceloader: Restore anticipated loader states for hardcoded module requests (duration: 00m 12s)
[00:59:33] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:59:49] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Rename initscript to match expectation [puppet] - 10https://gerrit.wikimedia.org/r/230021 (owner: 10Yuvipanda)
[01:00:28] <gwicke>	 ori: remember C*?
[01:00:41] <ori>	 yes, I am clearly getting old
[01:01:48] <gwicke>	 with golang it's the other way around
[01:02:01] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Remove useless $master_ip ipresolve [puppet] - 10https://gerrit.wikimedia.org/r/230022 
[01:02:17] <YuviPanda>	 ori: I keep typoing it
[01:02:27] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Remove useless $master_ip ipresolve [puppet] - 10https://gerrit.wikimedia.org/r/230022 (owner: 10Yuvipanda)
[01:06:59] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Don't use quotes around values [puppet] - 10https://gerrit.wikimedia.org/r/230024 
[01:07:30] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Don't use quotes around values [puppet] - 10https://gerrit.wikimedia.org/r/230024 (owner: 10Yuvipanda)
[01:07:49] <ori>	 gwicke: heh, true!
[01:08:42] <logmsgbot>	 !log ori Synchronized php-1.26wmf17/extensions/Gadgets: I2089b21fc: Updated mediawiki/core Project: mediawiki/extensions/Gadgets (duration: 00m 13s)
[01:08:49] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[01:10:27] <icinga-wm>	 PROBLEM - Restbase endpoints health on praseodymium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[01:14:27] <icinga-wm>	 RECOVERY - Restbase endpoints health on praseodymium is OK: All endpoints are healthy
[01:15:03] <wikibugs>	 6operations, 10Deployment-Systems, 6Release-Engineering, 6Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1516944 (10GWicke)
[01:16:08] <icinga-wm>	 PROBLEM - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out
[01:18:07] <icinga-wm>	 RECOVERY - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 496 bytes in 0.089 second response time
[01:19:33] <YuviPanda>	 stupid ipv6
[01:28:47] <icinga-wm>	 PROBLEM - Restbase endpoints health on cerium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[01:30:57] <icinga-wm>	 RECOVERY - Restbase endpoints health on cerium is OK: All endpoints are healthy
[01:45:07] <icinga-wm>	 PROBLEM - Restbase endpoints health on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[01:46:58] <icinga-wm>	 RECOVERY - Restbase endpoints health on xenon is OK: All endpoints are healthy
[01:48:19] <icinga-wm>	 PROBLEM - Restbase endpoints health on praseodymium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[01:49:44] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Set master_ip explicitly for the API server [puppet] - 10https://gerrit.wikimedia.org/r/230028 
[01:50:05] <YuviPanda>	 gwicke: do you know what's up with these alerts?
[01:50:13] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Set master_ip explicitly for the API server [puppet] - 10https://gerrit.wikimedia.org/r/230028 (owner: 10Yuvipanda)
[01:50:18] <icinga-wm>	 RECOVERY - Restbase endpoints health on praseodymium is OK: All endpoints are healthy
[01:50:31] <gwicke>	 YuviPanda: yes, it's RB being slow during load testing on the staging cluster
[01:50:53] <gwicke>	 b-black set up a downtime in icinga earlier, but I think that has elapsed already
[01:51:11] <gwicke>	 the test hasn't completed yet
[01:51:23] <gwicke>	 hosts are xenon, cerium and praseodymium
[01:52:42] <gwicke>	 I don't have icinga access, so can't do much about the alerts myself
[01:53:30] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Pass master_host to the api class [puppet] - 10https://gerrit.wikimedia.org/r/230029 
[01:53:51] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Pass master_host to the api class [puppet] - 10https://gerrit.wikimedia.org/r/230029 (owner: 10Yuvipanda)
[01:54:56] <logmsgbot>	 !log krinkle Started scap: Rebuild l10n for Gadgets after I2089b21fc
[01:55:03] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[01:57:53] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Fix parameter name to match realtiy [puppet] - 10https://gerrit.wikimedia.org/r/230030 
[01:58:23] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Fix parameter name to match realtiy [puppet] - 10https://gerrit.wikimedia.org/r/230030 (owner: 10Yuvipanda)
[02:02:29] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Specify explicitly insecure port for kube-apiserver [puppet] - 10https://gerrit.wikimedia.org/r/230031 
[02:02:45] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Specify explicitly insecure port for kube-apiserver [puppet] - 10https://gerrit.wikimedia.org/r/230031 (owner: 10Yuvipanda)
[02:15:17] <grrrit-wm>	 (03PS1) 10BBlack: remove conftool data for cache_bits [puppet] - 10https://gerrit.wikimedia.org/r/230032 
[02:15:19] <grrrit-wm>	 (03PS1) 10BBlack: add conftool/hiera data for new eqiad misc boxes [puppet] - 10https://gerrit.wikimedia.org/r/230033 
[02:15:33] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] remove conftool data for cache_bits [puppet] - 10https://gerrit.wikimedia.org/r/230032 (owner: 10BBlack)
[02:16:11] <grrrit-wm>	 (03PS2) 10BBlack: add conftool/hiera data for new eqiad misc boxes [puppet] - 10https://gerrit.wikimedia.org/r/230033 (https://phabricator.wikimedia.org/T86718) 
[02:16:42] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] add conftool/hiera data for new eqiad misc boxes [puppet] - 10https://gerrit.wikimedia.org/r/230033 (https://phabricator.wikimedia.org/T86718) (owner: 10BBlack)
[02:19:25] <logmsgbot>	 !log krinkle Finished scap: Rebuild l10n for Gadgets after I2089b21fc (duration: 24m 28s)
[02:19:32] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:22:40] <bblack>	 !log pooled cp105[67],cp1069,cp1070 into eqiad misc caches
[02:22:49] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:26:20] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review: Upgrade eqiad-misc varnish cluster from 2 to 4 systems. - https://phabricator.wikimedia.org/T86718#1516964 (10BBlack) The 4x previous eqiad-bits machines (cp1056, cp1057, cp1069, cp1070) are functioning in the eqiad misc cluster now, alongside the old.  Will give th...
[02:27:31] <legoktm>	 Krinkle: um, wtf? why did you deploy that now?
[02:28:24] <Krinkle>	 legoktm: Because they were all broken.
[02:28:55] <legoktm>	 What do you mean?
[02:29:01] <legoktm>	 Why didn't you fix them then?
[02:29:08] <Krinkle>	 <script src=Gadget-foo.js&action=raw> doesn't work well because 1) it's unconditional (so default gadgets cause exceptions in old browsers) and 2) since async, they can execute after jQuery so it really doesn't work
[02:29:11] <legoktm>	 (fix whatever was breaking them)
[02:29:25] <Krinkle>	 Ori and I tried to come up with different solutions but nothing seemed feasible
[02:29:32] <legoktm>	 so...revert?
[02:29:38] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 14.29% of data above the critical threshold [500.0]
[02:29:39] <legoktm>	 that seems pretty easy to me
[02:30:10] <ori>	 no
[02:30:57] <ori>	 we're not going back
[02:33:33] <ori>	 the breakage was a half-second performance penalty levied on 18 billion page views every month
[02:33:49] <legoktm>	 that's a pretty one-sided view.
[02:34:46] <legoktm>	 I'm fairly sure if you undeploy half the extensions we currently use, you could make the site even faster.
[02:35:12] <ori>	 it is tempting!
[02:35:57] <Krinkle>	 https://www.mediawiki.org/wiki/ResourceLoader/Migration_guide_(users)#MediaWiki_1.26
[02:35:59] <ori>	 but seriously: the suggestion to revert is one-sided. this is a hard problem, not a glitch, and the solution is to let users know with a deprecation warning and provide a migration path.
[02:37:26] <ori>	 if there are high-value gadgets that need to be fixed ASAP, I'm happy to help with the migration
[02:37:29] <legoktm>	 Great, now we have two sides!
[02:38:20] <Helder>	 I think AntiVandal-Tool had problems when I tried to enable ResourceLoader for it on ptwiki some years ago
[02:39:17] <legoktm>	 you can probably start with nav-popups
[02:39:28] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[02:40:10] <ori>	 (what does nav-popups do that enabling the hovercards beta feature doesn't?)
[02:40:18] <icinga-wm>	 PROBLEM - Restbase endpoints health on cerium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[02:40:34] <logmsgbot>	 !log l10nupdate Synchronized php-1.26wmf17/cache/l10n: l10nupdate for 1.26wmf17 (duration: 06m 15s)
[02:40:39] <legoktm>	 are you asking that seriously?
[02:40:42] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:40:54] <ori>	 yes
[02:41:06] <Krinkle>	 HoverCards is like 0.1% of popups feature sets
[02:41:15] <legoktm>	 ^
[02:41:19] <legoktm>	 it only shows article preview
[02:41:21] <RoanKattouw>	 NavPopups is a behemoth that took years to develop / grow
[02:41:25] <YuviPanda>	 and nobody's working on HoverCards and it's explicitly not supposed to have all of navpopup's features
[02:41:27] <legoktm>	 no diffs, editing, fast reverts, etc.
[02:41:30] <RoanKattouw>	 Its a pretty giant code base
[02:41:34] <Krinkle>	 but.. it doesn't do anything that is conceptually impossible without document.write or global eval
[02:41:47] <legoktm>	 info about users, edit count, groups, global groups, block status, lock status, etc.
[02:41:57] <Krinkle>	 Yeah, we're not defending hover cards.
[02:42:00] <Krinkle>	 (I'm not anyway :P)
[02:42:11] <ori>	 OK, I'll take a shot at those two
[02:42:17] <icinga-wm>	 RECOVERY - Restbase endpoints health on cerium is OK: All endpoints are healthy
[02:42:23] <Krinkle>	 However it's been 4 years since ResourceLoader, and they've still not managed to upgrade popups. But yeah, we can help.
[02:42:34] <Josve05a>	 On sv.wp. lol
[02:42:34] <Josve05a>	 Gadget "editssincelast" was not loaded. Please migrate it to use ResourceLoader.  See <https://sv.wikipedia.org/wiki/Special:Finesser>.
[02:42:34] <Josve05a>	 Gadget "stats" was not loaded. Please migrate it to use ResourceLoader.  See <https://sv.wikipedia.org/wiki/Special:Finesser>.
[02:42:35] <ori>	 https://www.mediawiki.org/wiki/ResourceLoader/Migration_guide_(users)#MediaWiki_1.26 is significant
[02:42:37] <Krinkle>	 Roan and I have just triaged commons, nlwiki, cawiki and svwiki
[02:42:42] <ori>	 "Gadgets are now required to use ResourceLoader"
[02:43:15] <RoanKattouw>	 We did a full triage of cawiki and fixed almost all of them, but on other wikis we only looked at default gadgets
[02:43:33] <ori>	 the expectation that code in the mediawiki / gadget / user namespaces will work forever without periodic maintenance to keep it up to date is IMO completely insensible
[02:43:35] <logmsgbot>	 !log l10nupdate@tin LocalisationUpdate completed (1.26wmf17) at 2015-08-07 02:43:35+00:00
[02:43:42] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:44:33] <legoktm>	 ori: no one is saying that. But they do need notification *beforehand*
[02:45:12] <ori>	 https://www.mediawiki.org/wiki/ResourceLoader/Migration_guide_(users)#MediaWiki_1.26 dates to 2012
[02:45:39] <legoktm>	 you mean the page? that's not really the point.
[02:46:09] <RoanKattouw>	 ori: Actually that section of it dates to today https://www.mediawiki.org/w/index.php?title=ResourceLoader/Migration_guide_%28users%29&diff=1818340&oldid=1816928
[02:46:11] <Krinkle>	 I agree it was suboptimal to do it unscheduled, but there have been many notifications. All of the last 3 years worth of migrations are within the RL era. At this point non-RL is not even being discussed anymore.
[02:46:42] <Krinkle>	 I don't believe another round of announcement would've made it go any faster. At some point the only ears who haven't heard it yet are those that don't know JavaScript or those that forgot they had it enabled.
[02:46:57] <ori>	 yes, that
[02:52:10] <ori>	 MediaWiki:Gadget-popups.js should be fairly straightforward
[02:53:08] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 10.00% of data above the critical threshold [500.0]
[02:55:29] <ori>	 legoktm: is it actually broken? it is marked as RL-ready
[02:57:07] <RoanKattouw>	 It allegedly works on Commons but not enwiki
[02:57:20] <legoktm>	 ori: yes, I'm not seeing any popups.
[02:57:28] <icinga-wm>	 PROBLEM - OCG health on ocg1003 is CRITICAL ocg_job_status 555963 msg: ocg_render_job_queue 6775 msg (=3000 critical)
[02:58:38] <icinga-wm>	 PROBLEM - OCG health on ocg1001 is CRITICAL ocg_job_status 559402 msg: ocg_render_job_queue 9080 msg (=3000 critical)
[02:58:57] <icinga-wm>	 PROBLEM - OCG health on ocg1002 is CRITICAL ocg_job_status 559953 msg: ocg_render_job_queue 9407 msg (=3000 critical)
[03:00:27] <icinga-wm>	 PROBLEM - puppet last run on cp3007 is CRITICAL puppet fail
[03:00:37] <RoanKattouw>	 ori: Note that some gadgets are really just wrappers that mw.loader.load() a script from another domain, which without proper dependency management will probably end badly
[03:03:09] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[03:14:57] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.14% of data above the critical threshold [500.0]
[03:20:58] <ori>	 legoktm: ok, I'm working on it
[03:21:34] <legoktm>	 Thanks
[03:24:47] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[03:25:18] <icinga-wm>	 RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 61475 bytes in 0.287 second response time
[03:27:57] <icinga-wm>	 RECOVERY - puppet last run on cp3007 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:40:32] <ori>	 legoktm: https://en.wikipedia.org/wiki/User:Nakon beat me to it
[03:41:12] <legoktm>	 Oh fantastic
[03:43:16] <ori>	 legoktm: sorry if I sounded dismissive earlier. are there any other ones that you think are vital? I'll take a shot at the one Helder mentioned above next
[03:46:48] <icinga-wm>	 RECOVERY - OCG health on ocg1003 is OK ocg_job_status 593845 msg: ocg_render_job_queue 270 msg
[03:47:17] <legoktm>	 ori: I'm on my phone, but hotcat would probably be the next one if it's still broken (someone said it was in -tech)
[03:47:59] <icinga-wm>	 RECOVERY - OCG health on ocg1001 is OK ocg_job_status 593990 msg: ocg_render_job_queue 0 msg
[03:48:17] <icinga-wm>	 RECOVERY - OCG health on ocg1002 is OK ocg_job_status 594018 msg: ocg_render_job_queue 0 msg
[04:36:36] <grrrit-wm>	 (03PS1) 10BBlack: remove cp104[34] from eqiad misc [puppet] - 10https://gerrit.wikimedia.org/r/230038 
[04:36:38] <grrrit-wm>	 (03PS1) 10BBlack: remove cache::misc role from cp104[34] [puppet] - 10https://gerrit.wikimedia.org/r/230039 
[04:37:06] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] remove cp104[34] from eqiad misc [puppet] - 10https://gerrit.wikimedia.org/r/230038 (owner: 10BBlack)
[04:37:33] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] remove cache::misc role from cp104[34] [puppet] - 10https://gerrit.wikimedia.org/r/230039 (owner: 10BBlack)
[04:38:42] <wikibugs>	 6operations, 10Traffic: Expand misc cluster into cache PoPs - https://phabricator.wikimedia.org/T101339#1517057 (10BBlack)
[04:38:44] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review: Upgrade eqiad-misc varnish cluster from 2 to 4 systems. - https://phabricator.wikimedia.org/T86718#1517054 (10BBlack) 5Open>3Resolved a:3BBlack Old hosts removed.  eqiad now has 4 hosts (previous bits cluster hosts).
[04:39:48] <wikibugs>	 6operations, 10ops-eqiad, 10Traffic: decom/reclaim cp104[34] - https://phabricator.wikimedia.org/T108281#1517059 (10BBlack) 3NEW
[04:51:46] <grrrit-wm>	 (03PS1) 10BBlack: HTTP/2 alpha patch [software/nginx] (wmf-1.9.3-1-h2) - 10https://gerrit.wikimedia.org/r/230040 
[05:02:50] <grrrit-wm>	 (03PS2) 10BBlack: HTTP/2 alpha patch [software/nginx] (wmf-1.9.3-1-h2) - 10https://gerrit.wikimedia.org/r/230040 
[05:24:28] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1517098 (10mmodell) >>! In T107532#1516868, @GWicke wrote: >> a general way to have a puppet run be part of...
[06:30:28] <icinga-wm>	 PROBLEM - puppet last run on cp1054 is CRITICAL Puppet has 1 failures
[06:30:39] <icinga-wm>	 PROBLEM - puppet last run on mw1112 is CRITICAL Puppet has 1 failures
[06:31:08] <icinga-wm>	 PROBLEM - puppet last run on db2056 is CRITICAL puppet fail
[06:31:27] <icinga-wm>	 PROBLEM - puppet last run on cp3017 is CRITICAL puppet fail
[06:31:37] <icinga-wm>	 PROBLEM - puppet last run on db2064 is CRITICAL Puppet has 1 failures
[06:31:39] <wikibugs>	 6operations, 6Performance-Team, 6Release-Engineering, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1517180 (10mmodell)
[06:31:57] <icinga-wm>	 PROBLEM - puppet last run on mw1086 is CRITICAL Puppet has 1 failures
[06:31:58] <icinga-wm>	 PROBLEM - puppet last run on db1059 is CRITICAL Puppet has 1 failures
[06:32:17] <icinga-wm>	 PROBLEM - puppet last run on mw2126 is CRITICAL Puppet has 2 failures
[06:32:18] <icinga-wm>	 PROBLEM - puppet last run on wtp2017 is CRITICAL Puppet has 1 failures
[06:32:18] <icinga-wm>	 PROBLEM - puppet last run on mw1135 is CRITICAL Puppet has 3 failures
[06:32:38] <icinga-wm>	 PROBLEM - puppet last run on mw2207 is CRITICAL Puppet has 1 failures
[06:32:39] <icinga-wm>	 PROBLEM - puppet last run on mw2050 is CRITICAL Puppet has 1 failures
[06:32:48] <icinga-wm>	 PROBLEM - puppet last run on wtp2008 is CRITICAL Puppet has 1 failures
[06:34:17] <jynus>	 ^This is my wake up alarm, no need for a clock!
[06:42:07] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517189 (10JeroenDeDauw) So using PHP 5.4 in a MediaWiki extension used by WMF will still cause problems? :(
[06:43:38] <icinga-wm>	 RECOVERY - Host db1035 is UPING OK - Packet loss = 0%, RTA = 1.10 ms
[06:55:19] <icinga-wm>	 RECOVERY - puppet last run on mw1086 is OK Puppet is currently enabled, last run 10 seconds ago with 0 failures
[06:55:58] <icinga-wm>	 RECOVERY - puppet last run on cp1054 is OK Puppet is currently enabled, last run 12 seconds ago with 0 failures
[06:56:17] <icinga-wm>	 RECOVERY - puppet last run on mw1112 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures
[06:56:27] <icinga-wm>	 RECOVERY - puppet last run on wtp2008 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures
[06:56:58] <icinga-wm>	 RECOVERY - puppet last run on cp3017 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures
[06:57:07] <icinga-wm>	 RECOVERY - puppet last run on db2064 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:19] <icinga-wm>	 RECOVERY - puppet last run on db1059 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:47] <icinga-wm>	 RECOVERY - puppet last run on mw2126 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:48] <icinga-wm>	 RECOVERY - puppet last run on mw1135 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:48] <icinga-wm>	 RECOVERY - puppet last run on wtp2017 is OK Puppet is currently enabled, last run 2 minutes ago with 0 failures
[06:58:18] <icinga-wm>	 RECOVERY - puppet last run on mw2207 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:18] <icinga-wm>	 RECOVERY - puppet last run on mw2050 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:47] <icinga-wm>	 RECOVERY - puppet last run on db2056 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:09:37] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected
[07:34:34] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug  7 07:34:34 UTC 2015 (duration 34m 33s)
[07:34:41] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[07:48:42] <grrrit-wm>	 (03CR) 10Nikerabbit: Start CirrusSearch AB test on suggestion confidence (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/229462 (https://phabricator.wikimedia.org/T108103) (owner: 10EBernhardson)
[07:57:37] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517308 (10Joe) >>! In T86081#1514006, @Krenair wrote: >>>! In T86081#1513499, @JeroenDeDauw wrote: >> Are these production machines that run MediaWiki? I'd lik...
[08:11:05] <godog>	 !log upgrade cassandra test cluster to openjdk 8u66-b01-1~bpo8+1
[08:11:12] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[08:11:12] <grrrit-wm>	 (03PS2) 10Yuvipanda: Tools: Fix mail address for webservice jobs [puppet] - 10https://gerrit.wikimedia.org/r/226311 (https://phabricator.wikimedia.org/T106462) (owner: 10Tim Landscheidt)
[08:11:20] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] Tools: Fix mail address for webservice jobs [puppet] - 10https://gerrit.wikimedia.org/r/226311 (https://phabricator.wikimedia.org/T106462) (owner: 10Tim Landscheidt)
[08:12:22] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 04-1] "This logging actually helped me find some issues with the hanging daemon, so leaving it in place for now." [puppet] - 10https://gerrit.wikimedia.org/r/223075 (owner: 10Yuvipanda)
[08:19:25] <wikibugs>	 6operations, 10RESTBase-Cassandra, 7Blocked-on-Services: Test JDK8 with Cassandra - https://phabricator.wikimedia.org/T104888#1517341 (10fgiunchedi) the cassandra test cluster has been upgraded to latest openjdk-8 `8u66-b01-1~bpo8+1`
[08:26:25] <godog>	 !log restart cassandra on test cluster
[08:26:32] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[08:30:37] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[08:30:48] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[08:34:33] <wikibugs>	 6operations, 10vm-requests: eqiad: 1 VM request for mailman - https://phabricator.wikimedia.org/T108065#1517352 (10akosiaris) >>! In T108065#1514315, @JohnLewis wrote: > @akosiaris yes, misc-web to view the interface is what I meant, perhaps not the best wording in my part :) >  > The service is fully puppetis...
[08:34:49] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge.
[08:36:38] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge.
[08:36:43] <wikibugs>	 6operations, 10Mathoid, 10RESTBase, 6Services: Document and hook up public mathoid end point in RB - https://phabricator.wikimedia.org/T102030#1517362 (10Physikerwelt)
[08:41:15] <wikibugs>	 6operations, 10RESTBase-Cassandra: upgrade RESTBase cluster to Cassandra 2.1.8 - https://phabricator.wikimedia.org/T107949#1517393 (10fgiunchedi) a:3fgiunchedi the test cluster has been upgraded to cassandra 2.1.8 (+latest openjdk in T104888)
[08:44:29] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517407 (10akosiaris)  > ``` > #!/bin/sh > # removes all iptables rules > # https://en.wikipedia.org/wiki/Tear_down_this_wall! > echo "flushing all iptables rules.." > iptables -F > iptables -X > iptables -t...
[08:45:42] <wikibugs>	 6operations, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1517417 (10Tau) Which is the easiest way to restore the files `includes/HttpFunctions.php` and `includes/filerepo/ForeignAPIR...
[08:45:48] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517418 (10akosiaris) Sigh, so NOTRACK failed due to being applied on the wrong table and then disaster ensued.
[08:48:41] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517436 (10akosiaris) Is it me or this incident documentation is not in https://wikitech.wikimedia.org/wiki/Incident_documentation ? I seem to be unable to find it ...
[08:52:56] <moritzm>	 !log restarted HHVM on API apaches in codfw for libtidy/PCRE security updates
[08:53:03] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[08:54:19] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Temporarily disable helium's poolcounter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230049 (https://phabricator.wikimedia.org/T104996) 
[08:59:24] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Temporarily disable helium's poolcounter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230049 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[08:59:30] <grrrit-wm>	 (03Merged) 10jenkins-bot: Temporarily disable helium's poolcounter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230049 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[09:01:15] <logmsgbot>	 !log akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
[09:01:21] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[09:02:24] <akosiaris>	 !log disabled helium as a poolcounter temporarily while applying base::firewall again
[09:02:31] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[09:06:52] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: ferm: NOTRACK needs to be applied on raw table [puppet] - 10https://gerrit.wikimedia.org/r/230052 (https://phabricator.wikimedia.org/T104996) 
[09:08:21] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Revert "Revert "bacula: enable firewall on helium"" [puppet] - 10https://gerrit.wikimedia.org/r/230053 (https://phabricator.wikimedia.org/T104996) 
[09:09:53] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: ferm: NOTRACK needs to be applied on raw table [puppet] - 10https://gerrit.wikimedia.org/r/230052 (https://phabricator.wikimedia.org/T104996) 
[09:10:34] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] ferm: NOTRACK needs to be applied on raw table [puppet] - 10https://gerrit.wikimedia.org/r/230052 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[09:11:15] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Revert "Revert "bacula: enable firewall on helium"" [puppet] - 10https://gerrit.wikimedia.org/r/230053 (https://phabricator.wikimedia.org/T104996) 
[09:13:15] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Revert "Temporarily disable helium's poolcounter" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230054 (https://phabricator.wikimedia.org/T104996) 
[09:13:55] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Revert "Revert "bacula: enable firewall on helium"" [puppet] - 10https://gerrit.wikimedia.org/r/230053 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[09:17:49] <icinga-wm>	 PROBLEM - Certificate expiration on nembus is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:IO::Socket::SSL: connect: Connection refused
[09:17:53] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517540 (10akosiaris) After fixing the bug with NOTRACK with https://gerrit.wikimedia.org/r/230052 and re-enabling firewall with https://gerrit.wikimedia.org/r/230053 on helium, things seem to be OK. I 'll r...
[09:19:13] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Add scheduler class [puppet] - 10https://gerrit.wikimedia.org/r/230055 
[09:19:58] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Add scheduler class [puppet] - 10https://gerrit.wikimedia.org/r/230055 
[09:20:09] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Add scheduler class [puppet] - 10https://gerrit.wikimedia.org/r/230055 (owner: 10Yuvipanda)
[09:20:24] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Revert "Temporarily disable helium's poolcounter" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230054 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[09:20:30] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Temporarily disable helium's poolcounter" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230054 (https://phabricator.wikimedia.org/T104996) (owner: 10Alexandros Kosiaris)
[09:24:02] <logmsgbot>	 !log akosiaris Synchronized wmf-config/PoolCounterSettings-eqiad.php: (no message) (duration: 00m 12s)
[09:34:19] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517546 (10akosiaris) 5Open>3Resolved After repooling helium as a poolcounter backend, things are working just fine. Resolving this
[09:36:25] <icinga-wm>	 PROBLEM - Check size of conntrack table on helium is CRITICAL nf_conntrack is 100 % full
[09:44:49] <grrrit-wm>	 (03PS2) 10Muehlenhoff: Add ferm rules for Hadoop worker nodes [puppet] - 10https://gerrit.wikimedia.org/r/229707 (https://phabricator.wikimedia.org/T83597) 
[09:46:18] <wikibugs>	 6operations, 7Database: New hardware for production core mysql cluster - https://phabricator.wikimedia.org/T106847#1517562 (10jcrespo)
[09:48:30] <godog>	 akosiaris: helium conntrack full? 
[09:50:27] <grrrit-wm>	 (03PS2) 10Muehlenhoff: Add ferm rules for spark/jmxtrans [puppet] - 10https://gerrit.wikimedia.org/r/229716 
[09:51:37] <akosiaris>	 godog: hmm why though ?
[09:51:46] <akosiaris>	 it seems to not cause problems for poolcounter this time though
[09:52:14] <akosiaris>	 and the kernel is not really complaining
[09:52:22] <akosiaris>	 not dropping packets or something
[09:52:40] <akosiaris>	 I think I know why.. lemme test something
[09:52:44] <godog>	 yeah that's odd
[09:55:48] <akosiaris>	 sysctl net/netfilter/nf_conntrack_max net/netfilter/nf_conntrack_count
[09:55:49] <akosiaris>	 net.netfilter.nf_conntrack_max = 65536
[09:55:49] <akosiaris>	 net.netfilter.nf_conntrack_count = 65536
[09:55:53] <akosiaris>	 ok so that's true
[09:57:41] <moritzm>	 I think we also need notrack for the output chain
[09:57:56] <moritzm>	 it's currently only marked for prerouting
[09:58:38] <moritzm>	 and conntrack -L lists many outgoing connections
[09:59:38] <akosiaris>	 all in time_wait
[09:59:51] <akosiaris>	 moritzm: I only see incoming though
[10:00:01] <akosiaris>	 dst=10.64.0.179
[10:02:17] <akosiaris>	 moritzm: added manually the NOTRACK for output too
[10:02:25] <akosiaris>	 let's see if it gets better
[10:02:40] <akosiaris>	 it is indeed
[10:03:10] <akosiaris>	 dropping at a rate of 500 c/s 
[10:03:19] <akosiaris>	 moritzm: ok, I am adding it
[10:03:42] <moritzm>	 sounds good, I tried filtering with "conntrack -L -s 1064.0.179" and that gave 59243 about 30s ago
[10:03:56] <akosiaris>	 net.netfilter.nf_conntrack_count = 8644
[10:03:56] <akosiaris>	 net.netfilter.nf_conntrack_max = 65536
[10:04:35] <icinga-wm>	 RECOVERY - Check size of conntrack table on helium is OK nf_conntrack is 0 % full
[10:04:37] <akosiaris>	 ok we are doing to <300 now
[10:05:56] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 13.33% of data above the critical threshold [500.0]
[10:06:33] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: ferm: Also NOTRACK in OUTPUT raw table chain [puppet] - 10https://gerrit.wikimedia.org/r/230061 
[10:07:17] <godog>	 \o/
[10:07:42] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 031] ferm: Also NOTRACK in OUTPUT raw table chain [puppet] - 10https://gerrit.wikimedia.org/r/230061 (owner: 10Alexandros Kosiaris)
[10:11:32] <godog>	 also it could be that the kernel wasn't complaining because they were dropped in output? specifically early_drop vs drop in conntrack -S 
[10:12:15] <jem>	 Hi, Javascript accesories don't seem to work for at least eswiki and cawiki, is this known?
[10:15:28] <valhallasw`cloud>	 jem: http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical.ambassadors/1256/focus=83521 ? 
[10:16:04] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] ferm: Also NOTRACK in OUTPUT raw table chain [puppet] - 10https://gerrit.wikimedia.org/r/230061 (owner: 10Alexandros Kosiaris)
[10:16:38] <wikibugs>	 6operations: Monitor failing ferm restarts - https://phabricator.wikimedia.org/T108303#1517580 (10MoritzMuehlenhoff) 3NEW
[10:18:36] <jem>	 valhallasw`cloud: Well, it would be quite surprising, and there are no blank pages, just pages without the JS extras
[10:19:16] <jem>	 And eswiki isn't listed in https://phabricator.wikimedia.org/P1832
[10:19:40] <jem>	 Nor cawiki
[10:19:55] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[10:21:11] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517608 (10Krenair) >>! In T86081#1517308, @Joe wrote: >>>! In T86081#1514006, @Krenair wrote: >>>>! In T86081#1513499, @JeroenDeDauw wrote: >>> Are these produ...
[10:22:34] <icinga-wm>	 PROBLEM - puppet last run on lvs4001 is CRITICAL puppet fail
[10:22:53] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: GitIgnore JetBrain's IDEA setting files [puppet] - 10https://gerrit.wikimedia.org/r/229730 (owner: 10Yurik)
[10:23:00] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] GitIgnore JetBrain's IDEA setting files [puppet] - 10https://gerrit.wikimedia.org/r/229730 (owner: 10Yurik)
[10:23:34] <wikibugs>	 6operations, 5Patch-For-Review: Ferm rules for backup roles - https://phabricator.wikimedia.org/T104996#1517618 (10MoritzMuehlenhoff) > == Actionables == > - T105378 > - gerrit:228137 > - ?  I've created https://phabricator.wikimedia.org/T108303 to detect failing ferm restarts
[10:24:18] <YuviPanda>	 akosiaris: mmm you're coming over to the dark side? :)
[10:24:45] <YuviPanda>	 (re IDEA)
[10:24:52] <YuviPanda>	 oh, I guess thats just yurik
[10:24:53] <YuviPanda>	 alright then
[10:26:20] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517619 (10Joe) Do we really want to use hack in mediawiki-config? Who decided to vendor-lock ourselves into HHVM forever?  Also, Zend tests are needed for thir...
[10:28:01] <akosiaris>	 YuviPanda: it's not the dark side.. it's the murky side
[10:28:09] <YuviPanda>	 I quite enjoy IDEA products
[10:28:11] <akosiaris>	 I 'd be willing to do dark, but not murky
[10:28:15] <YuviPanda>	 I wrote puppet in it for a long time
[10:28:19] <wikibugs>	 6operations, 6Release-Engineering: Try out hack (<?hh) for mediawiki-config - https://phabricator.wikimedia.org/T91590#1517624 (10Joe) Do we really want to vendor-lock us into HHVM? For what gain?  Do we really, really have a compelling reason to use hack in mediawiki-config? A performance gain?
[10:28:28] <YuviPanda>	 back to vim now ofc - Unite plugin fixed my problems :)
[10:30:39] <wikibugs>	 6operations, 6Labs, 10wikitech.wikimedia.org, 7HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1517626 (10Joe) This is not a blocker to drop php 5.3, it is not a blocker to drop zend tests (or are we testing OSM? we might keep tests just for it)  Converting silver could be a...
[10:31:50] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Add kube-proxy class [puppet] - 10https://gerrit.wikimedia.org/r/230065 
[10:31:55] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] k8s: Add kube-proxy class [puppet] - 10https://gerrit.wikimedia.org/r/230065 (owner: 10Yuvipanda)
[10:32:18] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Add kube-proxy class [puppet] - 10https://gerrit.wikimedia.org/r/230065 
[10:32:26] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Add kube-proxy class [puppet] - 10https://gerrit.wikimedia.org/r/230065 (owner: 10Yuvipanda)
[10:32:39] <wikibugs>	 6operations: better cassandra process checks - https://phabricator.wikimedia.org/T108306#1517627 (10fgiunchedi) 3NEW a:3fgiunchedi
[10:33:31] <grrrit-wm>	 (03PS1) 10Filippo Giunchedi: cassandra: switch to nrpe::monitor_systemd_unit_state [puppet] - 10https://gerrit.wikimedia.org/r/230066 (https://phabricator.wikimedia.org/T108306) 
[10:34:38] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Fix extra paren [puppet] - 10https://gerrit.wikimedia.org/r/230067 
[10:34:53] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Fix extra paren [puppet] - 10https://gerrit.wikimedia.org/r/230067 (owner: 10Yuvipanda)
[10:37:26] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517641 (10JeroenDeDauw) > Also, Zend tests are needed for third-party, for mediawiki  Yeah, it'd be unfortunate to no longer verify MediaWiki works with Zend P...
[10:39:44] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1517643 (10Joe) @JeroenDeDauw I think it's referring to zend tests on wmf-only branches, to be fair. My point then would be we don't need those for silver alone.
[10:44:49] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: service::node: Ensure saner permissions for logdir [puppet] - 10https://gerrit.wikimedia.org/r/230071 
[10:48:35] <icinga-wm>	 RECOVERY - puppet last run on lvs4001 is OK Puppet is currently enabled, last run 5 seconds ago with 0 failures
[10:50:25] <icinga-wm>	 PROBLEM - puppet last run on mw1010 is CRITICAL Puppet has 1 failures
[10:51:16] <valhallasw`cloud>	 jem: the empty pages bug was fixed, so it should just not work anymore.
[10:51:42] <valhallasw`cloud>	 jem: in any case, check your console, it might show you what's going on
[10:54:45] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: osm: various fixes [puppet] - 10https://gerrit.wikimedia.org/r/230072 
[10:57:25] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Provision worker nodes properly [puppet] - 10https://gerrit.wikimedia.org/r/230073 
[10:57:47] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Provision worker nodes properly [puppet] - 10https://gerrit.wikimedia.org/r/230073 (owner: 10Yuvipanda)
[11:00:08] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/230075 
[11:00:24] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/230075 (owner: 10Yuvipanda)
[11:12:53] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, I think we're currently storing dashboards in elasticsearch so grafana is effectively stateless" [puppet] - 10https://gerrit.wikimedia.org/r/229737 (owner: 10Dzahn)
[11:13:31] <wikibugs>	 6operations, 10RESTBase-Cassandra, 5Patch-For-Review: better cassandra process checks - https://phabricator.wikimedia.org/T108306#1517697 (10mobrovac)
[11:14:26] <icinga-wm>	 RECOVERY - puppet last run on mw1010 is OK Puppet is currently enabled, last run 34 seconds ago with 0 failures
[11:16:38] <wikibugs>	 6operations, 10RESTBase-Cassandra, 5Patch-For-Review: better cassandra process checks - https://phabricator.wikimedia.org/T108306#1517699 (10mobrovac) Wouldn't it be prudent to switch to a real SystemD unit conf file first?
[11:17:06] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 031] labs: new role::logstash::stashbot class [puppet] - 10https://gerrit.wikimedia.org/r/227175 (owner: 10BryanDavis)
[11:24:13] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] osm: various fixes [puppet] - 10https://gerrit.wikimedia.org/r/230072 (owner: 10Alexandros Kosiaris)
[11:24:17] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: osm: various fixes [puppet] - 10https://gerrit.wikimedia.org/r/230072 
[11:24:28] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [V: 032] osm: various fixes [puppet] - 10https://gerrit.wikimedia.org/r/230072 (owner: 10Alexandros Kosiaris)
[11:29:19] <jem>	 valhallasw`cloud: Ok, it seems we have to migrate to ResourceLoader (but I don't remember any warn about this in the Tech News)
[11:40:24] <jem>	 Well, popups are now working almost Ok but not the rest, we'll have to see what else we have to adjust
[11:40:39] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 04-1] "pending a cassandra systemd unit file" [puppet] - 10https://gerrit.wikimedia.org/r/230066 (https://phabricator.wikimedia.org/T108306) (owner: 10Filippo Giunchedi)
[11:40:45] <jem>	 Thanks for the help, valhallasw`cloud 
[11:42:14] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 031] "LGTM as the owner is still able to create files (needed by services)." [puppet] - 10https://gerrit.wikimedia.org/r/230071 (owner: 10Alexandros Kosiaris)
[11:42:21] <wikibugs>	 6operations, 10RESTBase-Cassandra, 5Patch-For-Review: better cassandra process checks - https://phabricator.wikimedia.org/T108306#1517720 (10fgiunchedi) more than prudent, it is a requirement.. for some reason I thought we already did that!
[11:45:48] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: osm: more fixes [puppet] - 10https://gerrit.wikimedia.org/r/230079 
[11:47:18] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: service::node: Ensure saner permissions for logdir [puppet] - 10https://gerrit.wikimedia.org/r/230071 
[11:47:41] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: service::node: Ensure saner permissions for logdir [puppet] - 10https://gerrit.wikimedia.org/r/230071 
[11:47:52] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] service::node: Ensure saner permissions for logdir [puppet] - 10https://gerrit.wikimedia.org/r/230071 (owner: 10Alexandros Kosiaris)
[11:51:05] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: osm: more fixes [puppet] - 10https://gerrit.wikimedia.org/r/230079 
[11:51:13] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] osm: more fixes [puppet] - 10https://gerrit.wikimedia.org/r/230079 (owner: 10Alexandros Kosiaris)
[11:56:16] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1517731 (10brion) Patch https://gerrit.wikimedia.org/r/#/c/230078/ switches MW-Vagrant's TMH from...
[12:01:35] <icinga-wm>	 PROBLEM - puppet last run on mw1158 is CRITICAL Puppet has 1 failures
[12:04:54] <grrrit-wm>	 (03CR) 10Tobias Gritschacher: [C: 031] "So, according to Jan and Dzahn, the patch could go in just as it is, right? Or do we have to change anything?" [puppet] - 10https://gerrit.wikimedia.org/r/227466 (https://phabricator.wikimedia.org/T101235) (owner: 10WMDE-leszek)
[12:07:18] <moritzm>	 !log restarted HHVM on jobrunners/imagescalers in eqiad/codfw for libtidy/PCRE security updates
[12:07:25] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[12:27:25] <icinga-wm>	 RECOVERY - puppet last run on mw1158 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:32:30] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1517796 (10brion) I'm also unable to produce VP9 WebM output with this ffmpeg build...  TMH patch...
[12:38:06] <grrrit-wm>	 (03PS2) 10Muehlenhoff: Enable base::firewall on labsdb1003 [puppet] - 10https://gerrit.wikimedia.org/r/227945 
[12:38:19] <wikibugs>	 6operations, 10Continuous-Integration-Infrastructure, 6Multimedia, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1517801 (10MoritzMuehlenhoff) I had a quick look at the PPA and the build embeds a local copy of l...
[12:47:19] <JohnFLewis>	 paravoid: hi! Around?
[12:50:23] <wikibugs>	 6operations, 7Database: huge (140GB) decrease on available disk space on db1042 - https://phabricator.wikimedia.org/T108316#1517831 (10jcrespo) 3NEW a:3jcrespo
[12:51:11] <grrrit-wm>	 (03CR) 10Muehlenhoff: ferm rules for nutcracker (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/223844 (https://phabricator.wikimedia.org/T104970) (owner: 10Dzahn)
[12:51:27] <JohnFLewis>	 (or mark, which ever has a few seconds/minutes - mailman related sort of)
[12:55:45] <grrrit-wm>	 (03CR) 10Jcrespo: [C: 031] Enable base::firewall on labsdb1003 [puppet] - 10https://gerrit.wikimedia.org/r/227945 (owner: 10Muehlenhoff)
[12:58:23] <JohnFLewis>	 paravoid / mark: basically; as a plain shell user I can't view logs I need on fermium (mailman, Apache + syslog). Due to my holiday next week, and the root request for Monday ops meeting - viewing the logs today would be a great help to support Daniel and Rob today. Otherwise it might delay progress for another week in worst case scenario. Thanks!
[13:10:11] <grrrit-wm>	 (03PS1) 10BBlack: Add benefactors.wm.o A, MX, SPF [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) 
[13:10:56] <grrrit-wm>	 (03PS2) 10BBlack: Add benefactors.wm.o A, MX, SPF [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) 
[13:12:37] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Introduce scb100{1,2}.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/229700 (https://phabricator.wikimedia.org/T107287) 
[13:13:34] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Introduce scb100{1,2}.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/229700 (https://phabricator.wikimedia.org/T107287) (owner: 10Alexandros Kosiaris)
[13:14:06] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: Introduce pybal-test200{1,2,3}.codfw.wmnet [dns] - 10https://gerrit.wikimedia.org/r/229696 (https://phabricator.wikimedia.org/T107901) 
[13:14:21] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Introduce pybal-test200{1,2,3}.codfw.wmnet [dns] - 10https://gerrit.wikimedia.org/r/229696 (https://phabricator.wikimedia.org/T107901) (owner: 10Alexandros Kosiaris)
[13:15:19] <grrrit-wm>	 (03PS3) 10BBlack: Add benefactors.wm.o A, MX, SPF [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) 
[13:15:32] <grrrit-wm>	 (03PS4) 10BBlack: Add benefactors.wm.o A, MX, SPF, DKIM [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) 
[13:16:36] <wikibugs>	 6operations, 10Traffic, 10Wikimedia-DNS, 5Patch-For-Review: DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) - https://phabricator.wikimedia.org/T107940#1517866 (10BBlack) Patch ref'd above has the basics.  It doesn't yet have the DKIM record, which would be at `benefactors._domaink...
[13:17:44] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Introduce scb100{1,2}.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/229710 (https://phabricator.wikimedia.org/T107287) 
[13:27:26] <grrrit-wm>	 (03CR) 10Ottomata: Add ferm rules for Hadoop worker nodes (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/229707 (https://phabricator.wikimedia.org/T83597) (owner: 10Muehlenhoff)
[13:29:13] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Introduce pybal-test200{1,2,3}.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/230091 
[13:30:27] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Introduce scb100{1,2}.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/229710 (https://phabricator.wikimedia.org/T107287) (owner: 10Alexandros Kosiaris)
[13:31:24] <Betacommand>	 who broke greasemonkey scripts for WMF wikis?
[13:31:26] <grrrit-wm>	 (03PS5) 10BBlack: Add benefactors.wm.o A, MX, SPF, DKIM [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) 
[13:32:01] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] Add benefactors.wm.o A, MX, SPF, DKIM [dns] - 10https://gerrit.wikimedia.org/r/230090 (https://phabricator.wikimedia.org/T107940) (owner: 10BBlack)
[13:33:06] <_joe_>	 Betacommand: I suppose no one on purpose, can you specify what broke? So that we might think of a solution. First of all, since when are they broken? What error do you see exactly?
[13:33:28] <grrrit-wm>	 (03CR) 10Muehlenhoff: Add ferm rules for Hadoop worker nodes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/229707 (https://phabricator.wikimedia.org/T83597) (owner: 10Muehlenhoff)
[13:33:33] <Betacommand>	 _joe_: Noticed them today
[13:33:37] <wikibugs>	 6operations, 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1517904 (10akosiaris) I am removing the operations project, as I am not seeing anything the ops can help with/have to do. Fee free to readd
[13:33:58] <_joe_>	 also, I suspect ops are not the best bet for breaking javascript (one could never know, but...)
[13:33:58] <grrrit-wm>	 (03PS3) 10Muehlenhoff: Add ferm rules for Hadoop worker nodes [puppet] - 10https://gerrit.wikimedia.org/r/229707 (https://phabricator.wikimedia.org/T83597) 
[13:34:18] <Betacommand>	 _joe_: http://tools.wmflabs.org/betacommand-dev/greasemonkey/wikipedia.nfcc.user.js
[13:34:22] <grrrit-wm>	 (03PS1) 10BBlack: benefactors SPF: s/jangosmtp/jangomail/ [dns] - 10https://gerrit.wikimedia.org/r/230095 (https://phabricator.wikimedia.org/T107940) 
[13:34:40] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] benefactors SPF: s/jangosmtp/jangomail/ [dns] - 10https://gerrit.wikimedia.org/r/230095 (https://phabricator.wikimedia.org/T107940) (owner: 10BBlack)
[13:35:19] <Betacommand>	 _joe_: the actual code hasnt changed, and works reliably, something on the server side caused them to no longer run 
[13:35:42] <_joe_>	 Betacommand: does this operate on wiki pages, right?
[13:35:55] <Betacommand>	 It should
[13:36:06] <_joe_>	 can you tell me what error do you see in the javascript console?
[13:36:07] <Betacommand>	 but doesnt any more
[13:36:17] <Betacommand>	 _joe_: Im not getting any errors
[13:36:36] <Betacommand>	 Im just not seeing the tabs that should be there
[13:37:06] <wikibugs>	 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 7Performance: HTTPS performance tuning - https://phabricator.wikimedia.org/T86666#1517917 (10akosiaris) p:5High>3Normal
[13:37:41] <urandom>	 !log starting nodetool cleanup on restbase1002 (see: T108083)
[13:37:49] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:37:53] <_joe_>	 Betacommand: I'm strongly suspecting this has to do with http://www.gossamer-threads.com/lists/wiki/wikitech/616215
[13:38:20] <Betacommand>	 _joe_: none of my scripts use that
[13:38:35] <_joe_>	 Betacommand: well, I've just seen a document.write in the script
[13:39:01] <_joe_>	 but I don't think it's used that way, no
[13:39:02] <Betacommand>	 _joe_: where is there that line in http://tools.wmflabs.org/betacommand-dev/greasemonkey/wikipedia.nfcc.user.js
[13:39:27] <wikibugs>	 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 7Performance: HTTPS performance tuning - https://phabricator.wikimedia.org/T86666#1517925 (10BBlack)
[13:39:58] <urandom>	 !log starting nodetool cleanup on restbase1004 (see: T108083)
[13:40:05] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:40:33] <moritzm>	 !log install pcre security updates on elastic*, analytics*, wtp*, db* and es*
[13:40:39] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[13:41:44] <_joe_>	 Betacommand: uhm sorry, you're right
[13:42:05] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Introduce pybal-test200{1,2,3}.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/230091 
[13:42:12] <Betacommand>	 _joe_: When I saw that email I double checked my code
[13:42:24] <_joe_>	 Betacommand: then I have no idea what can have changed, I can look at deployments but I'm not sure I can help easily :(
[13:42:24] <wikibugs>	 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 7Performance: HTTPS performance tuning - https://phabricator.wikimedia.org/T86666#1517939 (10BBlack) 5Open>3Resolved a:3BBlack Basically everything we talked about here is already well-handled and/or we moved in a different direction and this ticke...
[13:42:45] <_joe_>	 the fact you get no error is just weird, too.
[13:43:03] <Betacommand>	 _joe_: its a greasemonkey thing
[13:43:49] <wikibugs>	 6operations, 7HTTPS, 5HTTPS-by-default: HTTPS RFC5077 session tickets encryption key rollovers - https://phabricator.wikimedia.org/T86671#1517945 (10BBlack) 5Resolved>3Open Re-opening this.  Current thinking is we will eventually do this, and it's do-able with our current software stack.  Just requires s...
[13:43:59] <_joe_>	 Betacommand: I'm looking at your code and it seems very very simple, lemme test one thing
[13:44:06] <wikibugs>	 6operations, 10Traffic, 7HTTPS: HTTPS RFC5077 session tickets encryption key rollovers - https://phabricator.wikimedia.org/T86671#1517948 (10BBlack) p:5High>3Normal
[13:45:03] <_joe_>	 Betacommand: in my browser's js console I get
[13:45:05] <_joe_>	 mw.config.get('wgNamespaceNumber')
[13:45:07] <_joe_>	 null
[13:45:29] <_joe_>	 for a random enwiki page
[13:45:38] <Betacommand>	 WTF?
[13:46:02] <_joe_>	 sorry, didn't log in, now I get 0
[13:46:45] <_joe_>	 Betacommand: where is the tab supposed to be?
[13:47:07] <Betacommand>	 _joe_: In monobook up after the normal edit tabs
[13:47:47] <Betacommand>	 After the watch tab
[13:48:35] <wikibugs>	 6operations, 10ContentTranslation-cxserver, 6Language-Engineering, 10MediaWiki-extensions-ContentTranslation, and 2 others: Apertium leaves a ton of stale processes, consumes all the available - https://phabricator.wikimedia.org/T107270#1517975 (10Unhammer) I explained some of this on IRC, but repeating he...
[13:48:47] <_joe_>	 Betacommand: running the script by hand in my browser's console it works
[13:49:39] <Betacommand>	 _joe_: Then why did it stop working for me?
[13:50:00] <kart_>	 Need emergency fix to deploy. Anyone around to deploy? https://gerrit.wikimedia.org/r/#/c/230101/
[13:50:02] <_joe_>	 Betacommand: no idea, but your js code is a) correct b) works with monobook
[13:51:42] <_joe_>	 Betacommand: I have never used greasemonkey, so I'm not able to help further I guess
[13:52:51] <grrrit-wm>	 (03PS1) 10BBlack: add benefactors.wm.o to wikimedia_domains list for exim [puppet] - 10https://gerrit.wikimedia.org/r/230102 (https://phabricator.wikimedia.org/T107940) 
[13:53:19] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] add benefactors.wm.o to wikimedia_domains list for exim [puppet] - 10https://gerrit.wikimedia.org/r/230102 (https://phabricator.wikimedia.org/T107940) (owner: 10BBlack)
[13:54:59] <wikibugs>	 6operations, 10ops-eqiad, 10Traffic, 5Patch-For-Review: eqiad: investigate thermal issues with some cp10xx machines - https://phabricator.wikimedia.org/T103226#1518006 (10BBlack) 5Open>3Resolved Temps all still look good today, closing this up.  Thanks @cmjohnson!
[14:00:12] <wikibugs>	 6operations, 7Mail: mail alias for benefactorevents - https://phabricator.wikimedia.org/T107977#1518031 (10BBlack)
[14:00:14] <wikibugs>	 6operations, 10Traffic, 10Wikimedia-DNS, 5Patch-For-Review: DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) - https://phabricator.wikimedia.org/T107940#1518032 (10BBlack)
[14:01:08] <wikibugs>	 6operations, 10Traffic, 10Wikimedia-DNS, 5Patch-For-Review: DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) - https://phabricator.wikimedia.org/T107940#1507931 (10BBlack) [merged in the other task re aliases, as it will be easier to discuss it all in one ticket here]  I've deployed...
[14:02:27] <wikibugs>	 6operations, 10Traffic, 10Wikimedia-DNS, 5Patch-For-Review: DNS request for wikimedia.org (let 3rd party send mail as wikimedia.org) - https://phabricator.wikimedia.org/T107940#1518036 (10BBlack) (edited comment above, alias info was incorrect)
[14:03:11] <aharoni>	 yo kart_ 
[14:04:10] <kart_>	 aharoni: yo. It seems no one around to deploy fix.
[14:04:47] <akosiaris>	 kart_: how many languages pairs do we got ? 
[14:05:05] <akosiaris>	 just off the top of your head, no need to go around counting or something
[14:05:29] <akosiaris>	 I am just trying to do some very simple arithmetics based on https://phabricator.wikimedia.org/T107270#1517975
[14:06:12] <_joe_>	 my arithmetics is "restart every midnight just in case" is laughing stock :)
[14:06:33] <_joe_>	 akosiaris: -j1 -m 100 :P
[14:07:10] <akosiaris>	 yeah I am thinking -j1 -m 15-30 tops
[14:07:40] <_joe_>	 akosiaris: '-m' appears to be working... in testing!
[14:08:16] <akosiaris>	 well, if it is not we are gonna be reporting a bu
[14:08:18] <akosiaris>	 bug*
[14:08:22] <akosiaris>	 but anyway
[14:08:28] <kart_>	 akosiaris: 20+
[14:08:34] <kart_>	 27 :)
[14:08:46] <akosiaris>	 only 27 ?
[14:08:56] <akosiaris>	 I feared worse
[14:11:18] <kart_>	 akosiaris: oops.
[14:11:22] <kart_>	 58
[14:11:35] <kart_>	 We added simple and other recently.
[14:12:04] <_joe_>	 so 58 languages means 116 processes started right away
[14:12:09] <kart_>	 _joe_: -m, is good idea then.
[14:12:24] <_joe_>	 since apertium seems not to use all that cpu for now, I'd go with -j 2
[14:12:50] <akosiaris>	 no, not 116. its 58*7*32
[14:12:54] <akosiaris>	 give or take
[14:12:56] <_joe_>	 which would still mean 232 processes, but we might be able to live with that
[14:13:04] <_joe_>	 akosiaris: uh?
[14:13:16] <akosiaris>	 7 processes per language pair, give or take
[14:13:21] <_joe_>	 oh right 7 processes per language pair
[14:13:38] <_joe_>	 so if we have 58 languages, how many pairs?
[14:13:57] <_joe_>	 each language can be crossed over with every other one?
[14:14:07] <_joe_>	 I hope not
[14:14:19] <_joe_>	 58*7 is already too much, wtf
[14:14:27] <akosiaris>	 12992 processes
[14:14:39] <akosiaris>	 if the simple 32(cores)*7*58 is correct
[14:14:45] <_joe_>	 akosiaris: 58 is the number of language pairs, then?
[14:14:53] <_joe_>	 not the number of languages
[14:15:09] <kart_>	 Yes. Pairs. Not languages.
[14:15:09] <akosiaris>	 yes
[14:15:25] <_joe_>	 ok then 58*7 processes is already more than enough
[14:15:35] <_joe_>	 -j 1 -m 300
[14:15:41] <kart_>	 https://cxserver.wikimedia.org/list/mt
[14:15:44] <_joe_>	 not more than this, I'd say
[14:16:19] <kart_>	 OK. Meanwhile, I need emergency fix for ContentTranslation ext.
[14:16:35] <_joe_>	 for that, wait for someone from the releasers
[14:16:47] <kart_>	 yeah.
[14:16:59] <kart_>	 Wake up people. I have dinner to do :D
[14:17:17] <_joe_>	 kart_: I'm sorry but I don't feel confident with that change
[14:17:25] <_joe_>	 I can't control its consequences
[14:21:22] <kart_>	 _joe_: which change?
[14:21:36] <kart_>	 -m ?
[14:21:53] <kart_>	 or CX ext
[14:37:59] <wikibugs>	 6operations, 10ContentTranslation-cxserver, 6Language-Engineering, 10MediaWiki-extensions-ContentTranslation, and 2 others: Apertium leaves a ton of stale processes, consumes all the available - https://phabricator.wikimedia.org/T107270#1518112 (10akosiaris) Hello,  It's very good that you decided to repea...
[14:38:28] <wikibugs>	 6operations, 7Mail, 5Patch-For-Review: Ferm rules for MX mail servers - https://phabricator.wikimedia.org/T104979#1518113 (10MoritzMuehlenhoff) 5Open>3Resolved This is already resolved for a while, the rules are active on polonium and lead.
[14:43:03] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Introduce pybal-test200{1,2,3}.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/230091 (owner: 10Alexandros Kosiaris)
[14:43:08] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: Introduce pybal-test200{1,2,3}.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/230091 
[14:43:21] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Introduce pybal-test200{1,2,3}.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/230091 (owner: 10Alexandros Kosiaris)
[14:44:05] <yurik>	 about to sync to map-test
[14:44:34] <grrrit-wm>	 (03CR) 10Rush: "This isn't what I understood when we merged the module and I don't think it is the best representation of the idea. Sure, we can overloa" [puppet] - 10https://gerrit.wikimedia.org/r/227466 (https://phabricator.wikimedia.org/T101235) (owner: 10WMDE-leszek)
[14:46:23] <wikibugs>	 6operations, 7Monitoring: revamp apaches ganglia grouping - https://phabricator.wikimedia.org/T79947#1518153 (10Dzahn) per gage's comments above and looking at a report like f.e.  http://ganglia.wikimedia.org/latest/?c=API%20application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2  the "Apache 8...
[14:53:36] <wikibugs>	 7Blocked-on-Operations, 6operations, 6Discovery, 10Maps, 3Discovery-Maps-Sprint: Grant access to "sudo systemctl daemon-reload" on maps - https://phabricator.wikimedia.org/T108331#1518173 (10Yurik) 3NEW a:3akosiaris
[14:53:52] <yurik>	 akosiaris, ^ ?
[14:54:06] <akosiaris>	 yurik: niah
[14:54:11] <akosiaris>	 it's once off
[14:54:14] <akosiaris>	 I 'll do it for ya
[14:54:17] <yurik>	 thx
[14:54:27] <yurik>	 but i don't want to bug you guys if i can avoid with with the rights
[14:54:44] <yurik>	 one offs are bad :)
[14:54:44] <akosiaris>	 yeah, which is why I said it's once off
[14:54:52] <akosiaris>	 it will probably never happen again
[14:55:11] * yurik writes down the last famous quote by akosiaris 
[14:55:13] <akosiaris>	 wait
[14:55:24] <akosiaris>	 no ignore I was right
[14:56:09] <yurik>	 akosiaris, btw, that service is fully rewritten, so will probably crash at first
[14:56:24] <kart_>	 akosiaris: trying to fix -j/-m, still WIP.
[14:57:05] <wikibugs>	 7Blocked-on-Operations, 6operations, 6Discovery, 10Maps, 3Discovery-Maps-Sprint: Grant access to "sudo systemctl daemon-reload" on maps - https://phabricator.wikimedia.org/T108331#1518191 (10akosiaris) 5Open>3declined From IRC ``` akosiaris: yurik: niah akosiaris: it's once off akosiaris: I 'll do it...
[14:57:16] <akosiaris>	 yurik: done
[14:57:20] <akosiaris>	 wanna retry ?
[14:57:27] <akosiaris>	 kart_: oh, you too ?
[14:57:45] <yurik>	 akosiaris, restarted, didn't give me a warning
[14:57:49] <akosiaris>	 ok, I 'm redirected then to other things, put me as a reviewer thanks
[14:57:57] <akosiaris>	 yurik: works fine then. thanks!
[14:58:20] <kart_>	 oh :)
[14:58:37] <kart_>	 I'm sure my puppet is you-know-how-it-is
[15:00:13] <grrrit-wm>	 (03PS1) 10KartikMistry: Apertium: Add -j -m and parameters [puppet] - 10https://gerrit.wikimedia.org/r/230108 
[15:02:27] <kart_>	 akosiaris: ^
[15:03:45] <icinga-wm>	 PROBLEM - puppet last run on cerium is CRITICAL Puppet last ran 19 hours ago
[15:04:54] <moritzm>	 headsup: gerrit will be restarted 16 UTC (i.e. in approx. 55 mins for a Java security update)
[15:05:19] <ottomata>	 moritzm: hiya, why 'hadoop-hdfs-ha-journalnode' and 'hadoop-hdfs-journalnode-jmx'
[15:05:25] <ottomata>	 is the -ha- part something different?
[15:05:45] <icinga-wm>	 RECOVERY - puppet last run on cerium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:06:05] <icinga-wm>	 RECOVERY - Cassandra database on xenon is OK: PROCS OK: 1 process with UID = 111 (cassandra), command name java, args CassandraDaemon
[15:06:24] <icinga-wm>	 RECOVERY - Cassandra database on praseodymium is OK: PROCS OK: 1 process with UID = 111 (cassandra), command name java, args CassandraDaemon
[15:09:03] <ostriches>	 moritzm: Also pinged #-dev
[15:09:14] <icinga-wm>	 RECOVERY - Cassandra database on cerium is OK: PROCS OK: 1 process with UID = 111 (cassandra), command name java, args CassandraDaemon
[15:10:14] <gwicke>	 !log switched cassandra staging cluster back to G1GC / default puppet
[15:10:21] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:10:26] <yurik>	 !log synced kartotherian to maps-test* servers
[15:10:27] <moritzm>	 ottomata: the former (port 8485) is what the cloudera docs describe as configured through dfs.namenode.shared.edits.dir, I took the name "hadoop-hdfs-ha-journalnode" as it was named that way in the wikitech page
[15:10:33] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:10:37] <moritzm>	 ostriches: thanks!
[15:10:56] <ottomata>	 oh ok, hm.  
[15:11:03] <ottomata>	 interesting, ok!  ith ikn that's fine then.
[15:11:12] <ottomata>	 moritzm: one more nit you missed there: jmxtrans-jmx
[15:11:24] <ottomata>	 not hadoop_jmxtrans
[15:11:40] <ottomata>	 2101 is the jmx port on the jmxtrans jvm process, right?
[15:12:18] <moritzm>	 yeah, I'll fix the name of the service
[15:14:28] <grrrit-wm>	 (03PS1) 10Jcrespo: Repool db1035, config cleanup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230111 
[15:16:20] <grrrit-wm>	 (03PS2) 10Jcrespo: Repool db1035, config cleanup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230111 
[15:17:44] <icinga-wm>	 PROBLEM - BGP status on cr2-eqiad is CRITICAL host 208.80.154.197, sessions up: 73, down: 3, shutdown: 0BRPeering with AS1273 not established - CWBRPeering with AS8218 not established - NEO-ASNBRPeering with AS62651 not established - BR
[15:18:41] <wikibugs>	 6operations, 6Release-Engineering: [Spike] Try out hack (<?hh) for mediawiki-config - https://phabricator.wikimedia.org/T91590#1518237 (10greg)
[15:20:20] <wikibugs>	 6operations, 6Release-Engineering: [Spike] Try out hack (<?hh) for mediawiki-config - https://phabricator.wikimedia.org/T91590#1090671 (10greg) >>! In T91590#1517624, @Joe wrote: > Do we really want to vendor-lock us into HHVM? For what gain? >  > Do we really, really have a compelling reason to use hack in me...
[15:26:11] <wikibugs>	 6operations, 10RESTBase-Cassandra: Update JDK 8 package in backports repo - https://phabricator.wikimedia.org/T104887#1518255 (10GWicke) @MoritzMuehlenhoff, @fgiunchedi: Many thanks, indeed!
[15:27:32] <andrewbogott>	 mark: I was hoping to work on that nova-network thing this morning but instead I am sitting with a friend at the emergency room :(  Want to make an actual official appointment to work together on that next week?  It sounds like we have most of a plan and should go ahead and try it.
[15:29:32] <grrrit-wm>	 (03PS1) 10EBernhardson: Rsync cirrus user testing logs to stat1002 [puppet] - 10https://gerrit.wikimedia.org/r/230113 
[15:30:13] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Rsync cirrus user testing logs to stat1002 [puppet] - 10https://gerrit.wikimedia.org/r/230113 (owner: 10EBernhardson)
[15:31:12] <grrrit-wm>	 (03PS2) 10EBernhardson: Rsync cirrus user testing logs to stat1002 [puppet] - 10https://gerrit.wikimedia.org/r/230113 
[15:36:14] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1518303 (10mmodell) >>! In T107532#1517098, @mmodell wrote: > One approach that I've been thinking about a...
[15:37:04] <grrrit-wm>	 (03CR) 10Jcrespo: [C: 032] Repool db1035, config cleanup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230111 (owner: 10Jcrespo)
[15:40:51] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1518323 (10GWicke) @mmodell, would this get along with a running puppet agent for general system configurat...
[15:42:18] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1518331 (10mmodell) @gwicke: yes it would, as long as the rules applied by `puppet apply` don't try to chan...
[15:50:29] <grrrit-wm>	 (03CR) 10Ottomata: [C: 031] "I can merge whenever if you are ready." [puppet] - 10https://gerrit.wikimedia.org/r/230113 (owner: 10EBernhardson)
[15:51:13] <grrrit-wm>	 (03Abandoned) 10Ottomata: Allow labstore1003 to rsync from stat servers [puppet] - 10https://gerrit.wikimedia.org/r/228251 (https://phabricator.wikimedia.org/T107576) (owner: 10Ottomata)
[15:53:35] <aharoni>	 kart_: now is good?
[15:54:46] <logmsgbot>	 !log thcipriani Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.images.js: FIX: Use .attr() to set the resource attribute of image, while adapting [[gerrit:230101]] (duration: 00m 11s)
[15:54:54] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:54:54] <thcipriani>	 ^ kart_ quick fix deployed
[15:55:04] <kart_>	 aharoni: ^^
[15:55:09] <kart_>	 thcipriani: thanks!
[15:55:15] <jynus>	 ?
[15:57:07] <moritzm>	 headsup: gerrit restart in 5 minutes
[15:59:30] <jynus>	 did the bot logged a deploy 1 minute before the thcipriani one?
[15:59:47] <greg-g>	 jynus: not that I see
[15:59:51] <jynus>	 I'm confused
[15:59:59] <kart_>	 thcipriani: we're good. Thanks!
[16:00:09] <kart_>	 jynus: really?
[16:00:34] <jynus>	 !log repool db1035 database with low traffic
[16:00:41] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[16:00:44] <icinga-wm>	 PROBLEM - BGP status on cr2-eqiad is CRITICAL host 208.80.154.197, sessions up: 73, down: 3, shutdown: 0BRPeering with AS1273 not established - CWBRPeering with AS8218 not established - NEO-ASNBRPeering with AS62651 not established - BR
[16:01:17] <greg-g>	 ??? ^
[16:02:06] <moritzm>	 gerrit restart completed
[16:02:16] <ostriches>	 Looks good
[16:06:15] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Samsung GT-S3650 can't connect to Wikipedia - https://phabricator.wikimedia.org/T108298#1518379 (10BBlack) p:5Triage>3Normal
[16:07:03] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Samsung GT-S3650 can't connect to Wikipedia - https://phabricator.wikimedia.org/T108298#1518388 (10BBlack) Can we get details about the software being used?  As in: operations system type/revision, browser or app used and its revision, etc?  Carrier network / country could be...
[16:07:28] <wikibugs>	 6operations, 10Mathoid, 10RESTBase, 6Services: Document and hook up public mathoid end point in RB - https://phabricator.wikimedia.org/T102030#1518394 (10mobrovac) [PR #303](https://github.com/wikimedia/restbase/pull/303) adds the Mathoid public API as per T103811
[16:07:34] <wikibugs>	 6operations, 10Traffic, 7Browser-Support-Internet-Explorer, 7HTTPS: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#1518397 (10BBlack)
[16:08:03] <wikibugs>	 6operations, 10Traffic, 7HTTPS: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#1518398 (10BBlack)
[16:08:28] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Preload HSTS - https://phabricator.wikimedia.org/T104244#1518401 (10BBlack)
[16:11:21] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Samsung GT-S3650 can't connect to Wikipedia - https://phabricator.wikimedia.org/T108298#1518426 (10BBlack) google translate says the error message is "Certificate issuer not found" in Italian.
[16:11:55] <wikibugs>	 6operations, 10Traffic, 7HTTPS: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#1518431 (10BBlack)
[16:12:35] <icinga-wm>	 PROBLEM - RAID on ms-be2006 is CRITICAL 1 failed LD(s) (Offline)
[16:14:04] <icinga-wm>	 PROBLEM - Disk space on ms-be2006 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sdg1 is not accessible: Input/output error
[16:15:14] <icinga-wm>	 PROBLEM - BGP status on cr2-eqiad is CRITICAL host 208.80.154.197, sessions up: 73, down: 3, shutdown: 0BRPeering with AS1273 not established - CWBRPeering with AS8218 not established - NEO-ASNBRPeering with AS62651 not established - BR
[16:20:35] <icinga-wm>	 PROBLEM - puppet last run on ms-be2006 is CRITICAL Puppet has 1 failures
[16:27:25] <wikibugs>	 6operations, 10ops-codfw: ms-be2006 failed disk - https://phabricator.wikimedia.org/T108340#1518480 (10jcrespo) 3NEW
[16:41:00] <wikibugs>	 7Blocked-on-Operations, 6operations, 5Continuous-Integration-Isolation: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1518585 (10Andrew) p:5Triage>3Normal
[16:43:05] <wikibugs>	 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1518590 (10Krenair) >>! In T86081#1517619, @Joe wrote: > I would go as far as to suggest we either: >  > a) Drop OSM  I think that was already in the works with...
[16:47:52] <mutante>	 grrrit-wm: hey there, speak up
[16:48:55] * valhallasw`cloud prods grrrit-wm
[16:49:57] <mutante>	 thanks valhalla
[16:53:22] * valhallasw`cloud prods grrrit-wm1
[16:54:03] <ori>	 thanks valhalla1
[16:54:14] <icinga-wm>	 PROBLEM - BGP status on cr2-eqiad is CRITICAL host 208.80.154.197, sessions up: 73, down: 3, shutdown: 0BRPeering with AS1273 not established - CWBRPeering with AS8218 not established - NEO-ASNBRPeering with AS62651 not established - BR
[16:55:30] <valhallasw`cloud>	 ok, alive again
[16:55:32] <grrrit-wm>	 (03CR) 10Alex Monk: "Glaisher, see I9c018193" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/227879 (https://phabricator.wikimedia.org/T106724) (owner: 10John F. Lewis)
[16:55:47] <grrrit-wm>	 (03CR) 10Alex Monk: "Also I9c018193 for the messages" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/224420 (https://phabricator.wikimedia.org/T105676) (owner: 10Alex Monk)
[16:56:24] <ori>	 this is the most german video ever: https://www.youtube.com/watch?v=gYpMz63WAjM
[16:56:51] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review, 7Pybal: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#1518640 (10BBlack)
[16:57:23] <wikibugs>	 6operations, 10Traffic, 7HTTPS: HTTPS performance & UA adoption metrics - https://phabricator.wikimedia.org/T86664#1518642 (10BBlack)
[16:59:07] <wikibugs>	 6operations: Create an offboarding workflow with HR & Operations - https://phabricator.wikimedia.org/T108131#1518649 (10RobH)
[16:59:22] <wikibugs>	 6operations, 10Traffic, 7HTTPS: HTTPS performance & UA adoption metrics - https://phabricator.wikimedia.org/T86664#1518651 (10BBlack) 5Open>3Resolved a:3BBlack We're not exactly flying blind anymore.  Analytics has done a bunch of independent analysis post-switch.  We've got cipher/protocol/sessionid m...
[16:59:24] <wikibugs>	 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default: Force all Wikimedia cluster traffic to be over SSL for all users (logged-in and anon) - https://phabricator.wikimedia.org/T49832#1518654 (10BBlack)
[16:59:29] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM - https://phabricator.wikimedia.org/T105756#1518657 (10JohnLewis)
[16:59:31] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: give John Lewis shell access on the mailman staging VM - https://phabricator.wikimedia.org/T108082#1518655 (10JohnLewis) 5Resolved>3Open Re-open for sudo access. Patch coming for group.
[17:00:40] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Enable HSTS on Wikimedia sites - https://phabricator.wikimedia.org/T40516#1518665 (10BBlack)
[17:01:14] <wikibugs>	 6operations, 10Traffic, 10Wikimedia-Apache-configuration, 7HTTPS: HTTP->HTTPS redirects need to unconditional send Vary header - https://phabricator.wikimedia.org/T98990#1518672 (10BBlack)
[17:02:10] <grrrit-wm>	 (03PS1) 10John F. Lewis: admin: add mailman-admins group [puppet] - 10https://gerrit.wikimedia.org/r/230133 
[17:02:12] <grrrit-wm>	 (03PS1) 10John F. Lewis: admin: add johnflewis to mailman-admins [puppet] - 10https://gerrit.wikimedia.org/r/230134 (https://phabricator.wikimedia.org/T108082) 
[17:02:40] <grrrit-wm>	 (03PS2) 10John F. Lewis: admin: add mailman-admins group [puppet] - 10https://gerrit.wikimedia.org/r/230133 
[17:02:52] <grrrit-wm>	 (03PS2) 10John F. Lewis: admin: add johnflewis to mailman-admins [puppet] - 10https://gerrit.wikimedia.org/r/230134 (https://phabricator.wikimedia.org/T108082) 
[17:03:14] <icinga-wm>	 RECOVERY - Disk space on ms-be2006 is OK: DISK OK
[17:05:25] <grrrit-wm>	 (03CR) 10John F. Lewis: "This would replace https://gerrit.wikimedia.org/r/#/c/229585/ as the best option. This could also apply to production while the other patc" [puppet] - 10https://gerrit.wikimedia.org/r/230133 (owner: 10John F. Lewis)
[17:05:45] <grrrit-wm>	 (03CR) 10John F. Lewis: [C: 04-1] "https://gerrit.wikimedia.org/r/#/c/230133/ is preferred." [puppet] - 10https://gerrit.wikimedia.org/r/229585 (https://phabricator.wikimedia.org/T108082) (owner: 10Dzahn)
[17:09:11] <yurik>	 !log synced kartotherian to maps-test* servers again, restarted the service
[17:09:17] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[17:09:46] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 031] admin: add mailman-admins group [puppet] - 10https://gerrit.wikimedia.org/r/230133 (owner: 10John F. Lewis)
[17:10:03] <grrrit-wm>	 (03Abandoned) 10Andrew Bogott: admin: add mailman root group and add john [puppet] - 10https://gerrit.wikimedia.org/r/229585 (https://phabricator.wikimedia.org/T108082) (owner: 10Dzahn)
[17:10:48] <JohnFLewis>	 andrewbogott: do you want to move https://phabricator.wikimedia.org/T108082 to the ops meeting review column of the access-requests board?
[17:11:37] <valhallasw`cloud>	 ori: that was amazing
[17:11:53] <ori>	 ikr
[17:12:37] <wikibugs>	 6operations, 10Traffic, 7HTTPS: Samsung GT-S3650 can't connect to Wikipedia - https://phabricator.wikimedia.org/T108298#1518696 (10BBlack) Digging around https://en.wikipedia.org/wiki/Samsung_Corby and various Google results, apparently this is a 2009 phone that used Touchwiz 1.0, which is probably a UI hack...
[17:16:29] <andrewbogott>	 JohnFLewis: yes please :)
[17:16:42] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review, 7Pybal: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#1518732 (10BBlack) p:5High>3Low a:5BBlack>3None
[17:16:43] <andrewbogott>	 The main trick will be me remembering to bring it up :)
[17:17:12] <JohnFLewis>	 andrewbogott: well "You do not have permission to edit this object." :)
[17:17:19] <andrewbogott>	 ok, I’ll do it
[17:17:45] <andrewbogott>	 actually this needs a subtask anyway
[17:17:55] <JohnFLewis>	 andrewbogott: don't worry - mutante will remember, if not - I'll ensure its added in the meeting after because thats when I get off holiday :)
[17:18:17] <JohnFLewis>	 [the meeting, the week after not after the meeting of course]
[17:18:53] <wikibugs>	 6operations, 10Traffic, 5Patch-For-Review, 7Pybal: Add support for setting weight=0 when depooling - https://phabricator.wikimedia.org/T86650#1518736 (10BBlack) I think at this point, this has fallen down to the nice-to-have category, but may not be worth the effort.  The majority of our sessions are fresh...
[17:21:25] <wikibugs>	 10Ops-Access-Reviews: John Lewis sudo as 'list' on mailman staging VM - https://phabricator.wikimedia.org/T108349#1518752 (10Andrew) 3NEW a:3Dzahn
[17:22:58] <wikibugs>	 6operations, 6Labs, 5Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1518770 (10Andrew)
[17:23:00] <wikibugs>	 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Replace jdouglas's production ssh key - it matched labs key - https://phabricator.wikimedia.org/T108111#1518769 (10Andrew) 5Open>3Resolved
[17:34:17] <wikibugs>	 10Ops-Access-Reviews: Analytics-users membership for csteipp - https://phabricator.wikimedia.org/T108351#1518828 (10Andrew) 3NEW
[17:35:23] <wikibugs>	 10Ops-Access-Requests, 6operations: Access to stat1002 for csteipp - https://phabricator.wikimedia.org/T108227#1518842 (10Andrew) Chris --  This should be fine.  Please get your manager (if you have one :p ) to add a note of approval to this ticket and review the attached patch.  We can merge three work days a...
[17:35:27] <wikibugs>	 6operations, 10ContentTranslation-cxserver, 6Language-Engineering, 10MediaWiki-extensions-ContentTranslation, and 2 others: Apertium leaves a ton of stale processes, consumes all the available - https://phabricator.wikimedia.org/T107270#1518840 (10Unhammer) > I am not sure of the performance gains of not s...
[17:49:22] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Add Chris Steipp to analytics-users [puppet] - 10https://gerrit.wikimedia.org/r/230142 (https://phabricator.wikimedia.org/T108227) 
[17:50:34] <wikibugs>	 6operations, 10Wikimedia-DNS: Set up compat redirect stats.wikipedia.org -> stats.wikimedia.org - https://phabricator.wikimedia.org/T21353#1518900 (10Glaisher) Google "site:stats.wikipedia.org" gives nothing. [[ https://wikipediatools.appspot.com/linksearch.jsp?set=top40&link=stats.wikipedia.org | Top 40 Wikip...
[17:51:28] <wikibugs>	 6operations, 10Wikimedia-DNS: Set up compat redirect stats.wikipedia.org -> stats.wikimedia.org - https://phabricator.wikimedia.org/T21353#1518905 (10Glaisher)
[17:59:02] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-1] "Doesn't actually set wmgVisualEditorAvailableNamespaces anywhere" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228040 (owner: 10Jforrester)
[17:59:10] <grrrit-wm>	 (03CR) 10Alex Monk: "sorry, *use" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228040 (owner: 10Jforrester)
[18:02:27] <James_F>	 Krenair: It doesn't need to, does it? The magic of wmg -> wg…
[18:02:58] <Krenair>	 I don't think it's automagic?
[18:03:31] <James_F>	 Oh, hmm.
[18:03:31] <Krenair>	 But let's stick it on tin and find out :p
[18:04:22] <James_F>	 Krenair: That's one way to check. :-)
[18:04:49] <James_F>	 Krenair: Also, if you're willing, maybe we should move it from the 16:00 to the 08:00 slot so that you're around?
[18:05:06] <icinga-wm>	 PROBLEM - Cassandra database on restbase1001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (cassandra), command name java, args CassandraDaemon
[18:06:38] <Krenair>	 James_F, yep, doesn't work
[18:07:09] <James_F>	 Boo.
[18:07:10] <gwicke>	 restbase1001 is me
[18:07:25] <Krenair>	 https://phabricator.wikimedia.org/P1854
[18:07:45] <Krenair>	 of course, I had to modify php-1.26wmf17/LocalSettings.php to point to the staging config
[18:08:27] <gwicke>	 !log switched restbase1001 to CMS temporarily, to gather metrics; will switch back to G1GC tonight
[18:08:34] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[18:09:15] <James_F>	 Krenair: So just `$wgVisualEditorAvailableNamespaces = $wmgVisualEditorAvailableNamespaces;`?
[18:09:29] <James_F>	 Krenair: (Eww.)
[18:09:49] <greg-g>	 bblack: dumb question, any caching things change recently that would cause this to happen: Beta Cluster main page loads fine, but any other random article doesn't http://en.wikipedia.beta.wmflabs.org/wiki/UserMergeqvkpbd
[18:11:59] <Krenair>	 James_F, no, I think you have to merge it in somehow
[18:12:27] <James_F>	 Krenair: Into what? The whole point is we want to replace DefaultSettings, not in any way merge into it.
[18:12:31] <Krenair>	 probably $wgVisualEditorAvailableNamespaces += $wmgVisualEditorAvailableNamespaces;
[18:12:45] <Krenair>	 Ah.
[18:12:50] <James_F>	 Krenair: That defeats part of the purpose of the change, though.
[18:12:58] <James_F>	 (To be able to e.g. disable NS0 or NS2.)
[18:13:08] <Krenair>	 Okay, $wgVisualEditorAvailableNamespaces = $wmgVisualEditorAvailableNamespaces; then
[18:13:17] <James_F>	 Lovely.
[18:13:20] * James_F modifies.
[18:13:46] <Krenair>	 It might even work to just set 'wgVisualEditorAvailableNamespaces' straight in InitialiseSettings
[18:14:04] <James_F>	 I thought that was a no-no?
[18:14:14] <Krenair>	 I don't remember why
[18:14:25] <Krenair>	 But I vaguely recall something along those lines
[18:14:47] <grrrit-wm>	 (03PS3) 10Jforrester: VisualEditor: Switch from …Namespaces to …AvailableNamespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228040 
[18:14:52] <James_F>	 Test that?
[18:15:25] <wikibugs>	 6operations, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: Support VP9 in TMH (Unable to decode) - https://phabricator.wikimedia.org/T55863#1519023 (10Paladox) When will vp9 most likely be switched on in Wikimedia since this https://gerrit.wikimedia.org/r/#/c/230078/ patch was merged and switched to...
[18:16:09] <grrrit-wm>	 (03PS4) 10Jforrester: Enable VisualEditor on NS_PROJECT for meta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/228041 (https://phabricator.wikimedia.org/T107003) 
[18:17:30] <Krenair>	 James_F, yep, looks good
[18:17:37] <Krenair>	 works as you'd expect
[18:17:56] <James_F>	 Nice.
[18:18:02] <James_F>	 Maybe that line can be improved, though.
[18:18:13] <James_F>	 Folded into the if block below.
[18:18:38] <Krenair>	 You should be able to remove the !isset( $wgVisualEditorAvailableNamespaces ) check now
[18:19:15] <bblack>	 greg-g: probably!
[18:20:08] <bblack>	 greg-g: (give me a few to switch tracks and I'll poke around at things)
[18:20:45] <James_F>	 Krenair: Does the config magically instantiate based on 'default'? I guess it should.
[18:21:18] <Krenair>	 yes
[18:25:55] <bblack>	    18 FetchError   c Junk after gzip data
[18:25:59] <bblack>	 ^ is kiilling pages on beta
[18:26:51] <bblack>	 I don't see anything cherrypicked to deployment that would cause that obviously, though
[18:31:49] <bblack>	 is there some change to apache/hhvm on beta vs prod that would affect this? gzip stuff
[18:32:44] * greg-g shrugs
[18:33:00] <greg-g>	 bblack: thcipriani is diagnosing as well
[18:33:04] <legoktm>	 Krenair, James_F: with extension registration, you no longer need wmg -> wg. Setting wg should just work.
[18:33:42] <Krenair>	 how is extension registration connected with wikimedia's config?
[18:34:18] <James_F>	 Lazy evaluation or something?
[18:34:40] <thcipriani>	 bblack: doesn't seem to be any recent changes to beta's apache/hhvm
[18:34:47] <legoktm>	 it's due to load order.
[18:35:46] <legoktm>	 InitialiseSettings goes first, exports to globals. Then you require_once the PHP entry point, which sets defaults, wiping out anything from InitialiseSettings (hence wg = wmg hack). But with wfLoadExtension(), we load later on, and instead merge into what is already set instead of setting the defaults
[18:35:59] <wikibugs>	 6operations, 6Labs, 10wikitech.wikimedia.org, 7HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1519166 (10Andrew) I have no idea whether hhvm will interact badly with OSM.  It's pretty different from other extensions so it wouldn't shock me.  A day long wikitech outage isn't...
[18:38:13] <Krenair>	 legoktm, so if we just relied on setting wg in InitialiseSettings, we still wouldn't be able to prevent the defaults
[18:38:26] <Krenair>	 but our own extra configured namespaces should work?
[18:38:41] <legoktm>	 uh, you would prevent the defaults by setting them to explicit false
[18:38:48] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Add controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230148 
[18:39:00] <Krenair>	 and then it wouldn't overwrite them with the defaults?
[18:39:03] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Add controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230148 
[18:39:15] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Add controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230148 (owner: 10Yuvipanda)
[18:40:16] <wikibugs>	 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Access to stat1002 for csteipp - https://phabricator.wikimedia.org/T108227#1519197 (10Andrew) Terry has responded to me via email approving this request.  I guess he doesn't have a phab account...
[18:41:59] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Specify master port explicitly for controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230150 
[18:42:05] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] k8s: Specify master port explicitly for controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230150 (owner: 10Yuvipanda)
[18:42:18] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Specify master port explicitly for controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230150 
[18:43:16] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] k8s: Specify master port explicitly for controller manager [puppet] - 10https://gerrit.wikimedia.org/r/230150 (owner: 10Yuvipanda)
[18:54:01] <hoo>	 greg-g: Look, a three headed monkey... and a friday deploy, but don't look that way :D
[18:54:20] <hoo>	 I need to deploy a JS fix for Wikidata
[18:56:09] <greg-g>	 hoo: does it work/do as expected on beta cluster?
[18:56:14] <greg-g>	 (is this fixing that issue?)
[18:56:31] <hoo>	 greg-g: No, the beta cluster issue is separate
[18:56:39] <hoo>	 The beta cluster issue just waits for a +2
[18:57:33] <greg-g>	 hoo: who should review it? (I assume https://gerrit.wikimedia.org/r/#/c/230152/ )
[18:57:39] <greg-g>	 hoo: what's the prod issue?
[18:57:57] <greg-g>	 https://phabricator.wikimedia.org/T108003 ?
[18:58:01] <hoo>	 greg-g: Prod issue is that we assume that $ (jQuery) is there on page load, which it's not (as of this week)
[18:58:09] <hoo>	 yeah
[18:58:17] <greg-g>	 hoo: yeah, dothat
[18:58:44] <hoo>	 I don't think any developers from our team are there, so probably no one to CR from us for now
[18:58:46] <hoo>	 maybe jzerebecki 
[18:59:37] <greg-g>	 hoo: for it breaking beta, let's just self-merge and you all can post-merge-review later?
[18:59:54] <hoo>	 ok, fair enough
[19:00:06] <greg-g>	 ty
[19:00:24] <greg-g>	 thcipriani: ^ should be good soon
[19:00:47] <thcipriani>	 ack. thanks hoo!
[19:01:23] <mutante>	 jzerebecki: ^
[19:01:39] <mutante>	 i know he's in this timezone at least :)
[19:02:30] <greg-g>	 hoo: I'm goign to go find lunch, thanks for the quick response on both those things
[19:03:36] <hoo>	 Thanks for letting me know :)
[19:11:46] <jzerebecki>	 looks
[19:15:42] <jzerebecki>	 looks sane
[19:17:22] <wikibugs>	 6operations, 10ops-eqiad: Decom and wipe cisco virt servers virt1001-1009 then remove from racks - https://phabricator.wikimedia.org/T107159#1519328 (10Cmjohnson) 5Open>3Resolved All disks are wiped and servers have been removed from racks and added to decom rack.
[19:17:28] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] grafana: add role to krypton (VM) [puppet] - 10https://gerrit.wikimedia.org/r/229737 (owner: 10Dzahn)
[19:17:55] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Don't have kubelet confibure cbr0 [puppet] - 10https://gerrit.wikimedia.org/r/230161 
[19:17:58] <wikibugs>	 6operations, 6Discovery, 10Maps, 10Traffic, 3Discovery-Maps-Sprint: Set up standard HTTPS Termination -> 2layer caching -> LVS -> servicehosts for maps service - https://phabricator.wikimedia.org/T105076#1519330 (10BBlack)
[19:18:08] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Don't have kubelet confibure cbr0 [puppet] - 10https://gerrit.wikimedia.org/r/230161 
[19:18:15] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Don't have kubelet confibure cbr0 [puppet] - 10https://gerrit.wikimedia.org/r/230161 (owner: 10Yuvipanda)
[19:23:28] <hoo>	 mh... beta is still broken
[19:23:48] <hoo>	 I don't see any new error on its fluorine, though
[19:24:34] <wikibugs>	 6operations, 6Discovery, 10Maps, 10Traffic, 3Discovery-Maps-Sprint: Set up standard HTTPS Termination -> 2layer caching -> LVS -> servicehosts for maps service - https://phabricator.wikimedia.org/T105076#1519339 (10BBlack)
[19:26:09] <hoo>	 $ grep -c 'Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `mediawikiwiki`.' fatal.log 
[19:26:09] <hoo>	 1994053
[19:26:15] <hoo>	 On beta
[19:27:04] <Krenair>	 hoo, I think I was complaining about that a few days ago
[19:27:10] <Krenair>	 all from videoscaler01, right?
[19:27:13] <hoo>	 yeah
[19:27:30] <Krenair>	 if you turn off the jobrunner there it goes away
[19:29:09] <logmsgbot>	 !log hoo Synchronized php-1.26wmf17/extensions/Wikidata/: Update Wikibase: Fix WB spinner, UnresolvedRedirectException handling on client (duration: 00m 21s)
[19:29:16] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:35:01] <grrrit-wm>	 (03PS1) 10Yuvipanda: k8s: Don't start the flannel service before the file exists [puppet] - 10https://gerrit.wikimedia.org/r/230164 
[19:35:13] <grrrit-wm>	 (03PS2) 10Yuvipanda: k8s: Don't start the flannel service before the file exists [puppet] - 10https://gerrit.wikimedia.org/r/230164 
[19:35:19] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] k8s: Don't start the flannel service before the file exists [puppet] - 10https://gerrit.wikimedia.org/r/230164 (owner: 10Yuvipanda)
[19:36:01] <wikibugs>	 6operations, 10ops-eqiad: apply wdqs100x labels to wmf3543/44 - https://phabricator.wikimedia.org/T108367#1519357 (10Cmjohnson) 3NEW a:3Cmjohnson
[19:37:19] <YuviPanda>	 Krinkle_: I put a meeting on your calendar at 2:30 to talk about cvn, hope that's ok
[19:39:18] <grrrit-wm>	 (03PS1) 10Hoo man: Bump wgCacheEpoch for Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230172 
[19:39:46] <grrrit-wm>	 (03CR) 10Hoo man: [C: 032] Bump wgCacheEpoch for Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230172 (owner: 10Hoo man)
[19:39:52] <grrrit-wm>	 (03Merged) 10jenkins-bot: Bump wgCacheEpoch for Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230172 (owner: 10Hoo man)
[19:40:47] <logmsgbot>	 !log hoo Synchronized wmf-config/: Bump wgCacheEpoch for Wikidata (duration: 00m 13s)
[19:40:54] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:43:34] <grrrit-wm>	 (03PS1) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 
[19:43:55] <grrrit-wm>	 (03PS1) 10Ori.livneh: dotfiles: make `field` read from either stdin or a file [puppet] - 10https://gerrit.wikimedia.org/r/230174 
[19:45:33] <grrrit-wm>	 (03PS2) 10Ori.livneh: dotfiles: make `field` read from either stdin or a file [puppet] - 10https://gerrit.wikimedia.org/r/230174 
[19:45:51] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] dotfiles: make `field` read from either stdin or a file [puppet] - 10https://gerrit.wikimedia.org/r/230174 (owner: 10Ori.livneh)
[19:54:58] <wikibugs>	 6operations, 6Discovery, 10Maps, 10Traffic, 3Discovery-Maps-Sprint: Set up standard HTTPS Termination -> 2layer caching for maps service - https://phabricator.wikimedia.org/T105076#1519408 (10BBlack)
[19:56:09] <bblack>	 looks like betalabs error went away?
[19:56:15] <bblack>	 err deployment-cluster?
[19:56:20] <bblack>	 whatever it is!
[19:59:08] <greg-g>	 bblack: yep, it was an issue in wikibase
[20:00:09] <grrrit-wm>	 (03PS2) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 
[20:01:08] <bblack>	 yay!
[20:01:16] <bblack>	 well yay for it not being my fault I guess :P
[20:03:38] <greg-g>	 bblack: it's the small wins
[20:04:26] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[20:04:31] <Krenair>	 hoo|away, greg-g do we have a task for the fatal.log spam on beta?
[20:04:46] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).
[20:06:28] <greg-g>	 Krenair: not that I know of?
[20:06:45] <Krenair>	 ok
[20:07:00] <grrrit-wm>	 (03PS3) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 
[20:26:28] <grrrit-wm>	 (03CR) 10Ottomata: "Bunch of discussion here about this:" [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (owner: 10Ottomata)
[20:27:51] <grrrit-wm>	 (03CR) 10Ottomata: "bblack, BTW, one of the motivations for this is the consolidation of cache types. We need varnishkafka to be smarter about the topics it " [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (owner: 10Ottomata)
[20:43:48] <grrrit-wm>	 (03PS2) 10coren: Add PTR record for mailrelay-02.tools.wmflabs.org [dns] - 10https://gerrit.wikimedia.org/r/222362 (https://phabricator.wikimedia.org/T97574) (owner: 10Merlijn van Deen)
[20:43:52] <grrrit-wm>	 (03CR) 10Edenhill: "See per file comments." (039 comments) [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (owner: 10Ottomata)
[20:44:52] <grrrit-wm>	 (03PS4) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) 
[20:44:59] <grrrit-wm>	 (03CR) 10coren: [C: 032] "We need to find a fix for labs addresses leaking to prod DNS this way, but this is not the time to solve the general problem." [dns] - 10https://gerrit.wikimedia.org/r/222362 (https://phabricator.wikimedia.org/T97574) (owner: 10Merlijn van Deen)
[20:45:06] <grrrit-wm>	 (03PS3) 10coren: [tools] New host: tools-mailrelay-02 [puppet] - 10https://gerrit.wikimedia.org/r/222358 (https://phabricator.wikimedia.org/T97574) (owner: 10Merlijn van Deen)
[20:46:26] <Krinkle>	 YuviPanda: OK
[20:47:08] <grrrit-wm>	 (03CR) 10coren: [C: 032] [tools] New host: tools-mailrelay-02 [puppet] - 10https://gerrit.wikimedia.org/r/222358 (https://phabricator.wikimedia.org/T97574) (owner: 10Merlijn van Deen)
[20:48:04] <Coren>	 ori: Ack!  Sorry, you slipped in a changeset while I was merging and I noticed *just* as I hit enter
[20:48:25] <ori>	 Coren: np, thanks
[20:48:36] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge.
[20:49:04] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge.
[20:54:06] <grrrit-wm>	 (03PS5) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) 
[20:57:38] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: export config and archive data from sodium - https://phabricator.wikimedia.org/T108071#1519623 (10Dzahn) copied just the config of list "test" from sodium and put it in place on fermium  first there was a mailman bug page but the issues were just file system permissions...
[20:58:04] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: test importing of mailing list configs and archives on staging VM - https://phabricator.wikimedia.org/T108073#1519624 (10Dzahn) copied just the config of list "test" from sodium and put it in place on fermium  first there was a mailman bug page but the issues were just f...
[20:58:13] <grrrit-wm>	 (03PS1) 10Alex Monk: Don't try to create accounts on non-existent beta mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230227 (https://phabricator.wikimedia.org/T108375) 
[20:59:13] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 032] Don't try to create accounts on non-existent beta mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230227 (https://phabricator.wikimedia.org/T108375) (owner: 10Alex Monk)
[20:59:18] <grrrit-wm>	 (03Merged) 10jenkins-bot: Don't try to create accounts on non-existent beta mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230227 (https://phabricator.wikimedia.org/T108375) (owner: 10Alex Monk)
[21:00:32] <logmsgbot>	 !log krenair Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/230227/ - should be a noop for prod (duration: 00m 12s)
[21:00:39] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[21:02:29] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: create basic mailman setup on fermium (jessie) for testing import - https://phabricator.wikimedia.org/T108383#1519633 (10Dzahn) 3NEW a:3Dzahn
[21:03:05] <grrrit-wm>	 (03CR) 10Ottomata: [WIP] Add format.topic configuration (034 comments) [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) (owner: 10Ottomata)
[21:03:33] <grrrit-wm>	 (03CR) 10Ottomata: "OH! Ok cool!" [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) (owner: 10Ottomata)
[21:08:51] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: create basic mailman setup on fermium (jessie) for testing import - https://phabricator.wikimedia.org/T108383#1519649 (10Dzahn) this is done, it's a manual install of mailman on jessie from distro package (2.1.18-2)   with Apache config from sodium that we adjusted for i...
[21:10:03] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM - https://phabricator.wikimedia.org/T105756#1519659 (10Dzahn)
[21:10:05] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: create basic mailman setup on fermium (jessie) for testing import - https://phabricator.wikimedia.org/T108383#1519658 (10Dzahn) 5Open>3Resolved
[21:11:58] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: give John Lewis shell access on the mailman staging VM - https://phabricator.wikimedia.org/T108082#1519663 (10Dzahn) since regular shell access is already done, and there is T108349 for the privilege escalation, we can resolv...
[21:12:05] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: give John Lewis shell access on the mailman staging VM - https://phabricator.wikimedia.org/T108082#1519666 (10Dzahn) 5Open>3Resolved
[21:12:07] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM - https://phabricator.wikimedia.org/T105756#1519667 (10Dzahn)
[21:12:26] <wikibugs>	 10Ops-Access-Reviews: John Lewis sudo as 'list' on mailman staging VM - https://phabricator.wikimedia.org/T108349#1518752 (10Dzahn)
[21:12:27] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: give John Lewis shell access on the mailman staging VM - https://phabricator.wikimedia.org/T108082#1511883 (10Dzahn)
[21:13:44] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: give John Lewis shell access on the mailman staging VM - https://phabricator.wikimedia.org/T108082#1511883 (10Dzahn)
[21:13:46] <wikibugs>	 10Ops-Access-Reviews: John Lewis sudo as 'list' on mailman staging VM - https://phabricator.wikimedia.org/T108349#1518752 (10Dzahn)
[21:14:03] <grrrit-wm>	 (03PS3) 10Dzahn: admin: add mailman-admins group [puppet] - 10https://gerrit.wikimedia.org/r/230133 (https://phabricator.wikimedia.org/T108349) (owner: 10John F. Lewis)
[21:14:06] <JohnFLewis>	 mutante: that's the review task though
[21:14:39] <mutante>	 oh? 
[21:15:03] <mutante>	 i reversed "blocked" and "blocked by", they were the wrong way around
[21:15:04] <JohnFLewis>	 yeah, you closed the request in favour of the review task :)
[21:15:19] <mutante>	 arr
[21:15:50] <mutante>	 but the review tasks get created automatically
[21:15:54] <mutante>	 this was manual 
[21:16:03] <mutante>	 there is probably another one :p
[21:18:02] <JohnFLewis>	 I don't think the ticket has the correct policy?
[21:18:03] <mutante>	 JohnFLewis: i don't think it is the review task, actually
[21:18:10] <mutante>	 it still looks right to me
[21:18:14] <mutante>	 sudo is blocked by shell
[21:18:18] <JohnFLewis>	 andrewbogott: ^ is that the review task? :P
[21:18:28] <JohnFLewis>	 the whole situation is complex
[21:18:30] <mutante>	 whichever we use for in the meeting 
[21:19:01] <JohnFLewis>	 mutante: just, pick a task, cc the people needed and make sure it is the meeting Monday. Beyond that - I don't care :)
[21:19:10] <mutante>	 shell access was done, sudo was not
[21:19:15] <mutante>	 exactly
[21:19:33] <mutante>	 the one i linked in gerrit now :)
[21:20:04] <JohnFLewis>	 and that reminds me https://phabricator.wikimedia.org/T89781 still exists, open with zero outcome whatever
[21:20:20] <JohnFLewis>	 mutante: ^ you can follow that up next week for me ;)
[21:22:57] <grrrit-wm>	 (03PS1) 10Jdlrobson: Add WikidataPageBanner extension to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230232 
[21:24:23] <andrewbogott>	 Eh, if review tasks are automatic now that's news to me :)
[21:25:22] <mutante>	 JohnFLewis: eh, so that ticket is probably a bit stuck because it was so specific to the Bugzilla migration. to revive it needs to be more obvious just a phabricator permission request nowadays with a rationale
[21:25:35] <Krenair>	 andrewbogott, if you create it properly, yes
[21:25:38] <mutante>	 andrewbogott: so there is some kind of phab magic for it
[21:25:45] <Krenair>	 You set security=Access request
[21:25:54] <Krenair>	 But only when *creating* the ticket
[21:25:57] <mutante>	 when creating an access request it makes a second ticket 
[21:26:11] <andrewbogott>	 Ok, sorry if I clouded the issue
[21:26:13] <mutante>	 that being said, i yet have to see it really being used much
[21:26:16] <Krenair>	 Yes
[21:26:24] <Krenair>	 I've seen a bunch of people just ignore it
[21:26:38] <JohnFLewis>	 mutante: you created it so close/deal with it as you see fit. No use cases exist beyond what you know with me anyway so - enjoy the ticket :)
[21:29:01] <grrrit-wm>	 (03CR) 10Kaldari: [C: 032] Add WikidataPageBanner extension to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230232 (owner: 10Jdlrobson)
[21:29:06] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add WikidataPageBanner extension to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230232 (owner: 10Jdlrobson)
[21:29:49] <grrrit-wm>	 (03PS1) 10BryanDavis: logstash: Count MediaWiki log events with statsd [puppet] - 10https://gerrit.wikimedia.org/r/230233 (https://phabricator.wikimedia.org/T100735) 
[21:30:32] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] logstash: Count MediaWiki log events with statsd [puppet] - 10https://gerrit.wikimedia.org/r/230233 (https://phabricator.wikimedia.org/T100735) (owner: 10BryanDavis)
[21:32:01] <grrrit-wm>	 (03PS2) 10BryanDavis: logstash: Count MediaWiki log events with statsd [puppet] - 10https://gerrit.wikimedia.org/r/230233 (https://phabricator.wikimedia.org/T100735) 
[21:35:43] <wikibugs>	 6operations, 10Traffic, 10fundraising-tech-ops, 5Patch-For-Review: Decide what to do with *.donate.wikimedia.org subdomain + TLS - https://phabricator.wikimedia.org/T102827#1519748 (10CCogdill_WMF) Confirming that IBM updated our links so that all Silveprop links will be https from now on. Can we close thi...
[21:39:22] <grrrit-wm>	 (03CR) 10Ottomata: [WIP] Add format.topic configuration (036 comments) [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) (owner: 10Ottomata)
[21:40:06] <Krinkle>	 YuviPanda: halfak: https://www.mediawiki.org/wiki/User:Krinkle/Patrolling
[21:41:16] <grrrit-wm>	 (03PS6) 10Ottomata: [WIP] Add format.topic configuration [software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/230173 (https://phabricator.wikimedia.org/T108379) 
[21:56:06] <Krinkle>	 YuviPanda: halfak: https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
[22:03:01] <kaldari>	 hey greg-g, I'm trying to deploy the new WikidataPageBanner extension to beta labs. I've updated the labs config files, the new beta config build has completed. I double-checked that the extension is in the mediawiki/extension repo. But it still isn't showing up on Special:Version. Anything else I need to check?
[22:03:53] <hoo>	 Did someone from the Wikidata team review it?
[22:04:30] <greg-g>	 kaldari: come on over to -releng and ask there
[22:04:41] <kaldari>	 greg-g: nevermind, it finally showed up :P
[22:04:47] <greg-g>	 sweet
[22:13:12] <grrrit-wm>	 (03PS1) 10Dzahn: pybal-test: include admin [puppet] - 10https://gerrit.wikimedia.org/r/230238 
[22:13:47] <Krinkle>	 YuviPanda: halfak: https://etherpad.wikimedia.org/p/ores
[22:15:56] <ori>	 mutante: but ::standard includes admin already?
[22:17:36] <mutante>	 ori: yes, that's right, that patch is not needed, it's just not done installing yet
[22:17:59] <mutante>	 they are all 3 signed now on puppetmaster but the user accounts are not there yet
[22:18:09] <mutante>	 i see it installing standard packages now on 2001
[22:18:14] <ori>	 oh cooooool
[22:18:33] <ori>	 that's awesome :) i was hoping to tinker around with them on the weekend and it looks like that will be possible
[22:18:35] <ori>	 thanks again
[22:18:38] <mutante>	 after the puppet run is finished we should also have request on the salt master
[22:18:42] <mutante>	 and can sign the keys for that 
[22:19:06] <ori>	 i can do that part, if you like -- i have done it before
[22:19:57] <mutante>	 ok, i'll let you know when normal ssh should work
[22:20:11] * ori nods
[22:21:01] <grrrit-wm>	 (03Abandoned) 10Dzahn: pybal-test: include admin [puppet] - 10https://gerrit.wikimedia.org/r/230238 (owner: 10Dzahn)
[22:21:59] <grrrit-wm>	 (03PS1) 10John F. Lewis: fermium: add mapped ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/230239 
[22:22:59] <wikibugs>	 6operations, 7Graphite: scale statsd reporting/aggregation (plan) - https://phabricator.wikimedia.org/T89857#1519939 (10Tgr) https://gerrit.wikimedia.org/r/#/c/226639/6 adds a (somewhat awkward) workaround via StatsD request sampling (in MediaWiki). Per IRC discussion with Ori, use of this workaround should be...
[22:23:07] <mutante>	 i wonder if that will work the same on ganeti VMs ^
[22:29:18] <wikibugs>	 6operations, 10Wikimedia-Mailing-lists: test importing of mailing list configs and archives on staging VM - https://phabricator.wikimedia.org/T108073#1519957 (10Dzahn) tested import of archives for list test (private) archives. there are 2 directories ./private and public/ , all actual files are in private and...
[22:29:52] <wikibugs>	 6operations, 10ops-eqiad, 10Traffic: decom/reclaim cp104[34] - https://phabricator.wikimedia.org/T108281#1519958 (10BBlack) 5Open>3declined a:3BBlack Actually, I'm re-purposing these immediately...
[22:34:40] <icinga-wm>	 PROBLEM - salt-minion processes on pybal-test2001 is CRITICAL: Connection refused by host
[22:35:00] <icinga-wm>	 PROBLEM - DPKG on pybal-test2002 is CRITICAL: Connection refused by host
[22:35:01] <icinga-wm>	 PROBLEM - configured eth on pybal-test2003 is CRITICAL: Connection refused by host
[22:35:19] <icinga-wm>	 PROBLEM - Disk space on pybal-test2002 is CRITICAL: Connection refused by host
[22:35:20] <icinga-wm>	 PROBLEM - dhclient process on pybal-test2003 is CRITICAL: Connection refused by host
[22:35:40] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2003 is CRITICAL: Connection refused by host
[22:35:51] <icinga-wm>	 PROBLEM - RAID on pybal-test2002 is CRITICAL: Connection refused by host
[22:35:51] <icinga-wm>	 PROBLEM - salt-minion processes on pybal-test2003 is CRITICAL: Connection refused by host
[22:36:10] <icinga-wm>	 PROBLEM - configured eth on pybal-test2002 is CRITICAL: Connection refused by host
[22:36:10] <icinga-wm>	 PROBLEM - DPKG on pybal-test2001 is CRITICAL: Connection refused by host
[22:36:21] <icinga-wm>	 PROBLEM - dhclient process on pybal-test2002 is CRITICAL: Connection refused by host
[22:36:21] <icinga-wm>	 PROBLEM - Disk space on pybal-test2001 is CRITICAL: Connection refused by host
[22:36:21] <ori>	 ^ ignore, that's initial provisioning
[22:36:39] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2002 is CRITICAL: Connection refused by host
[22:36:49] <icinga-wm>	 PROBLEM - salt-minion processes on pybal-test2002 is CRITICAL: Connection refused by host
[22:36:50] <icinga-wm>	 PROBLEM - RAID on pybal-test2001 is CRITICAL: Connection refused by host
[22:37:00] <icinga-wm>	 PROBLEM - DPKG on pybal-test2003 is CRITICAL: Connection refused by host
[22:37:11] <mutante>	 :) indeed, and it tells us icinga has been added but still in the middle of installing 
[22:37:19] <icinga-wm>	 PROBLEM - Disk space on pybal-test2003 is CRITICAL: Connection refused by host
[22:37:19] <icinga-wm>	 PROBLEM - configured eth on pybal-test2001 is CRITICAL: Connection refused by host
[22:37:29] <icinga-wm>	 PROBLEM - dhclient process on pybal-test2001 is CRITICAL: Connection refused by host
[22:37:40] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2001 is CRITICAL: Connection refused by host
[22:37:40] <icinga-wm>	 PROBLEM - RAID on pybal-test2003 is CRITICAL: Connection refused by host
[22:38:32] <grrrit-wm>	 (03PS1) 10John F. Lewis: fermium: override role default IPs [puppet] - 10https://gerrit.wikimedia.org/r/230240 
[22:39:39] <grrrit-wm>	 (03CR) 10John F. Lewis: [C: 04-1] "Don't merge until new service IPs in public1-eqiad-c have been given (or out of subnet IPs)." [puppet] - 10https://gerrit.wikimedia.org/r/230240 (owner: 10John F. Lewis)
[22:40:53] <grrrit-wm>	 (03PS1) 10BBlack: remove unused revdns for osm-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/230241 
[22:40:55] <grrrit-wm>	 (03PS1) 10BBlack: create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 
[22:40:57] <grrrit-wm>	 (03PS1) 10BBlack: add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 
[22:41:12] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 (owner: 10BBlack)
[22:41:19] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 (owner: 10BBlack)
[22:41:27] <wikibugs>	 6operations, 6Discovery, 10Maps, 10Traffic, 3Discovery-Maps-Sprint: Set up standard HTTPS Termination -> 2layer caching for maps service - https://phabricator.wikimedia.org/T105076#1519982 (10BBlack) a:5jcrespo>3BBlack
[22:41:42] <grrrit-wm>	 (03PS2) 10BBlack: add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 (https://phabricator.wikimedia.org/T105076) 
[22:41:44] <grrrit-wm>	 (03PS2) 10BBlack: create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 (https://phabricator.wikimedia.org/T105076) 
[22:41:46] <grrrit-wm>	 (03PS2) 10BBlack: remove unused revdns for osm-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/230241 (https://phabricator.wikimedia.org/T105076) 
[22:41:48] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: [toollabs] Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) 
[22:41:51] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[22:41:59] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[22:42:22] <grrrit-wm>	 (03PS2) 10Yuvipanda: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:42:27] <YuviPanda>	 valhallasw`cloud: want me to merge?
[22:42:48] <valhallasw`cloud>	 zomg tags
[22:43:03] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:43:26] <valhallasw`cloud>	 I got the puppet syntax wrong again ~_~
[22:43:35] <YuviPanda>	 heh
[22:43:41] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: [toollabs] Allow HBA login to all hosts [puppet] - 10https://gerrit.wikimedia.org/r/230245 (https://phabricator.wikimedia.org/T104613) 
[22:43:58] <YuviPanda>	 valhallasw`cloud: tch tch, toollabs: over [toollabs] :P
[22:43:58] <grrrit-wm>	 (03PS1) 10BBlack: Add cache_maps and related LVS config, using cp104[34] [puppet] - 10https://gerrit.wikimedia.org/r/230246 (https://phabricator.wikimedia.org/T105076) 
[22:45:04] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: toollabs: Allow HBA login to all hosts [puppet] - 10https://gerrit.wikimedia.org/r/230245 (https://phabricator.wikimedia.org/T104613) 
[22:46:14] <grrrit-wm>	 (03PS3) 10BBlack: add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 (https://phabricator.wikimedia.org/T105076) 
[22:46:16] <grrrit-wm>	 (03PS3) 10BBlack: create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 (https://phabricator.wikimedia.org/T105076) 
[22:48:33] <grrrit-wm>	 (03PS3) 10Merlijn van Deen: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) 
[22:48:48] <valhallasw`cloud>	 YuviPanda: ^ should be ok now...
[22:49:08] <YuviPanda>	 valhallasw`cloud: looking
[22:49:15] <grrrit-wm>	 (03PS1) 10Legoktm: Isolate wikidata.org cookies and CORS policies [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230247 (https://phabricator.wikimedia.org/T108101) 
[22:50:30] <icinga-wm>	 RECOVERY - salt-minion processes on pybal-test2002 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[22:50:33] <YuviPanda>	 valhallasw`cloud: where is $sshrsakey coming from?
[22:50:50] <icinga-wm>	 RECOVERY - DPKG on pybal-test2002 is OK: All packages OK
[22:50:59] <valhallasw`cloud>	 YuviPanda: facter
[22:51:01] <icinga-wm>	 RECOVERY - Disk space on pybal-test2002 is OK: DISK OK
[22:51:15] <YuviPanda>	 aaah
[22:51:15] <YuviPanda>	 ok
[22:51:21] <grrrit-wm>	 (03PS4) 10Yuvipanda: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:51:29] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230244 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:51:30] <icinga-wm>	 RECOVERY - RAID on pybal-test2003 is OK no RAID installed
[22:51:40] <icinga-wm>	 RECOVERY - salt-minion processes on pybal-test2003 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[22:51:40] <icinga-wm>	 RECOVERY - RAID on pybal-test2002 is OK no RAID installed
[22:51:48] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] remove unused revdns for osm-lb.esams [dns] - 10https://gerrit.wikimedia.org/r/230241 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[22:51:58] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4 [dns] - 10https://gerrit.wikimedia.org/r/230242 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[22:51:59] <icinga-wm>	 RECOVERY - configured eth on pybal-test2002 is OK - interfaces up
[22:52:08] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] add maps.wikimedia.org (-> maps-lb.eqiad) [dns] - 10https://gerrit.wikimedia.org/r/230243 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[22:52:08] <YuviPanda>	 valhallasw`cloud: merged!
[22:52:09] <icinga-wm>	 RECOVERY - dhclient process on pybal-test2002 is OK: PROCS OK: 0 processes with command name dhclient
[22:52:17] <valhallasw`cloud>	 YuviPanda: \o/
[22:52:20] <icinga-wm>	 RECOVERY - puppet last run on pybal-test2002 is OK Puppet is currently enabled, last run 46 seconds ago with 0 failures
[22:52:31] <YuviPanda>	 valhallasw`cloud: there was another one enabling HBA wasn't there?
[22:52:36] <valhallasw`cloud>	 yes
[22:52:49] <icinga-wm>	 RECOVERY - configured eth on pybal-test2003 is OK - interfaces up
[22:52:50] <icinga-wm>	 RECOVERY - DPKG on pybal-test2003 is OK: All packages OK
[22:52:58] <valhallasw`cloud>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: left operand of + is not a number at /etc/puppet/modules/toollabs/manifests/init.pp:66 on node tools-bastion-01.tools.eqiad.wmflabs
[22:52:59] <valhallasw`cloud>	 GAH
[22:53:00] <icinga-wm>	 RECOVERY - Disk space on pybal-test2003 is OK: DISK OK
[22:53:00] <icinga-wm>	 RECOVERY - dhclient process on pybal-test2003 is OK: PROCS OK: 0 processes with command name dhclient
[22:53:13] <ori>	 mutante: woot, I got shell
[22:53:19] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Revert "toollabs: Add ecdsa keys to ssh_known_hosts" [puppet] - 10https://gerrit.wikimedia.org/r/230248 
[22:53:25] <valhallasw`cloud>	 YuviPanda: ^
[22:53:29] <YuviPanda>	 valhallasw`cloud: hahahahaha :D
[22:53:30] <icinga-wm>	 RECOVERY - puppet last run on pybal-test2003 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:53:48] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] Revert "toollabs: Add ecdsa keys to ssh_known_hosts" [puppet] - 10https://gerrit.wikimedia.org/r/230248 (owner: 10Merlijn van Deen)
[22:57:06] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230249 (https://phabricator.wikimedia.org/T103999) 
[22:57:37] <valhallasw`cloud>	 YuviPanda: and we try again but grrrit-wm broke: https://gerrit.wikimedia.org/r/230249
[22:57:48] <grrrit-wm>	 (03PS2) 10Merlijn van Deen: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230249 (https://phabricator.wikimedia.org/T103999) 
[22:57:50] <icinga-wm>	 RECOVERY - DPKG on pybal-test2001 is OK: All packages OK
[22:57:58] <valhallasw`cloud>	 ah there we go
[22:58:01] <icinga-wm>	 RECOVERY - Disk space on pybal-test2001 is OK: DISK OK
[22:58:11] <grrrit-wm>	 (03PS3) 10Yuvipanda: toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230249 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:58:19] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] toollabs: Add ecdsa keys to ssh_known_hosts [puppet] - 10https://gerrit.wikimedia.org/r/230249 (https://phabricator.wikimedia.org/T103999) (owner: 10Merlijn van Deen)
[22:58:20] <icinga-wm>	 RECOVERY - salt-minion processes on pybal-test2001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[22:58:30] <icinga-wm>	 RECOVERY - RAID on pybal-test2001 is OK no RAID installed
[22:58:43] <YuviPanda>	 valhallasw`cloud: done
[22:59:00] <icinga-wm>	 RECOVERY - configured eth on pybal-test2001 is OK - interfaces up
[22:59:11] <icinga-wm>	 RECOVERY - dhclient process on pybal-test2001 is OK: PROCS OK: 0 processes with command name dhclient
[22:59:30] <icinga-wm>	 RECOVERY - puppet last run on pybal-test2001 is OK Puppet is currently enabled, last run 34 seconds ago with 0 failures
[22:59:46] <valhallasw`cloud>	 applying...
[22:59:58] <grrrit-wm>	 (03PS1) 10BryanDavis: logstash: Enable doc_values in template mapping [puppet] - 10https://gerrit.wikimedia.org/r/230250 
[23:02:01] <grrrit-wm>	 (03CR) 10BryanDavis: "Cherry-picked to beta cluster. Graphite metrics are being created as expected." [puppet] - 10https://gerrit.wikimedia.org/r/230233 (https://phabricator.wikimedia.org/T100735) (owner: 10BryanDavis)
[23:02:34] <YuviPanda>	 bd808: informal 'puppet swat' - changes you want me to merge?
[23:03:16] <bd808>	 YuviPanda: awesome offer. Let me look...
[23:05:07] <bd808>	 YuviPanda: this one is trivial -- https://gerrit.wikimedia.org/r/#/c/223301/
[23:05:43] <grrrit-wm>	 (03PS2) 10Yuvipanda: Logstash: Cleanup exclusion of API continuation logging [puppet] - 10https://gerrit.wikimedia.org/r/223301 (owner: 10Anomie)
[23:06:07] <mutante>	 YuviPanda: https://gerrit.wikimedia.org/r/#/c/194796/ 
[23:06:32] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Logstash: Cleanup exclusion of API continuation logging [puppet] - 10https://gerrit.wikimedia.org/r/223301 (owner: 10Anomie)
[23:06:46] <YuviPanda>	 bd808: any babysitting needed?
[23:07:14] <bd808>	 YuviPanda: naw. it will restart logstash when puppet runs on those hosts
[23:07:22] <YuviPanda>	 mutante https://tools.wmflabs.org/watroles/role/misc::labsdebrepo is still non-empty
[23:07:59] <Krenair>	 YuviPanda, what about https://gerrit.wikimedia.org/r/#/c/223365/ ?
[23:08:14] <bd808>	 YuviPanda: There's this one that has been rotting on the cherry-pick stack in beta too -- https://gerrit.wikimedia.org/r/#/c/210618 -- seems to have a comment from godog though wondering if it is "really" needed
[23:09:16] <valhallasw`cloud>	 YuviPanda: ok, ecdsa works.
[23:09:20] <valhallasw`cloud>	 can we also do hba?
[23:09:30] <YuviPanda>	 Krenair: I can merge it if hoo +1s it (since he -1'd it earlier)
[23:09:40] <YuviPanda>	 valhallasw`cloud: yup, let me just merge bd808's stuff for a minute
[23:09:42] <hoo>	 huh?
[23:09:45] <valhallasw`cloud>	 ok!
[23:10:21] <mutante>	 YuviPanda: i will remove it from the one that is mine
[23:10:59] <YuviPanda>	 bd808: I guess nobody wants to spend the time to make it a wrapper now...
[23:11:13] <YuviPanda>	 bd808: I can merge it if you think that patch isn't going to get a response anytime soon
[23:11:23] <bd808>	 I really doubt it
[23:11:33] <Krenair>	 hoo, trying to get https://gerrit.wikimedia.org/r/#/c/223365/ done
[23:11:33] <bd808>	 CI folks are hip deep in bigger problems
[23:11:54] <YuviPanda>	 bd808: yeah, I guess so
[23:11:58] <YuviPanda>	 bd808: i'll merge it shortly
[23:12:06] <bd808>	 awesome
[23:12:18] <hoo>	 Krenair: Did you test --write
[23:12:18] <bd808>	 and hurray for Puppet SWAT
[23:12:28] <YuviPanda>	 valhallasw`cloud: so once we merge it we must test to make sure this doesn't allow normal usrs to ssh into places like the proxy and redis
[23:12:32] <YuviPanda>	 but gonna merge it now
[23:12:38] <YuviPanda>	 it shouldn't because of the access.conf but good to test
[23:12:43] <grrrit-wm>	 (03PS3) 10Yuvipanda: toollabs: Allow HBA login to all hosts [puppet] - 10https://gerrit.wikimedia.org/r/230245 (https://phabricator.wikimedia.org/T104613) (owner: 10Merlijn van Deen)
[23:12:45] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] sql command: use slave server unless '--write' provided as an option before DB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/223365 (https://phabricator.wikimedia.org/T105046) (owner: 10Alex Monk)
[23:12:51] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] toollabs: Allow HBA login to all hosts [puppet] - 10https://gerrit.wikimedia.org/r/230245 (https://phabricator.wikimedia.org/T104613) (owner: 10Merlijn van Deen)
[23:12:58] <valhallasw`cloud>	 YuviPanda: if they can do it with hba they can already do it with pubkey
[23:13:03] <valhallasw`cloud>	 but I'll check tool users
[23:13:10] <valhallasw`cloud>	 er
[23:13:11] <valhallasw`cloud>	 tools
[23:13:13] <YuviPanda>	 valhallasw`cloud: yeah I agree. just wanted to check, I guess
[23:13:40] <grrrit-wm>	 (03PS9) 10Yuvipanda: beta: Add script from Jenkins beta-update-databases [puppet] - 10https://gerrit.wikimedia.org/r/210618 (https://phabricator.wikimedia.org/T96199) (owner: 10Thcipriani)
[23:14:16] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] "Considering this is in limbo for a while and it looks like CI folks are way deep in stuff to attempt to redo this (at least according to @" [puppet] - 10https://gerrit.wikimedia.org/r/210618 (https://phabricator.wikimedia.org/T96199) (owner: 10Thcipriani)
[23:14:38] <YuviPanda>	 bd808: I guess that's done now
[23:14:49] <Krenair>	 hoo, yes
[23:14:58] <YuviPanda>	 I think having a puppet SWAT window once a day or more run exactly like the MW SWAT window would be great
[23:15:07] <YuviPanda>	 I can push for it when I come back from vacation if people find it useful
[23:15:50] <bd808>	 I think it would be awesome
[23:16:03] <bd808>	 even once a week would be awesome
[23:16:06] <YuviPanda>	 hmm
[23:16:10] <YuviPanda>	 that might be an easier sell :D
[23:16:14] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "f.e. on analytics1004 confirmed with netstat, though i dont really see it /etc/spark where MASTER_PORT, MASTER_WEBUI_PORT and others are d" [puppet] - 10https://gerrit.wikimedia.org/r/229716 (owner: 10Muehlenhoff)
[23:16:18] <YuviPanda>	 mutante ok, let me find and do the rest actually
[23:16:36] <YuviPanda>	 mutante I guess wikistats-petcow is you
[23:16:51] <bd808>	 heh. a cattle pet
[23:17:35] <valhallasw`cloud>	 YuviPanda: ok, redis-01 closes connection for tool but I can login
[23:18:21] <mutante>	 YuviPanda: yes, petcow is mine. i need to just eh.. get wikitech access really quick :)
[23:18:30] <YuviPanda>	 heh
[23:18:30] <Krenair>	 hoo, does it not work for you?
[23:18:37] <hoo>	 I didn't test it
[23:18:41] <valhallasw`cloud>	 YuviPanda: bastion-02 for both. ok...
[23:19:04] <valhallasw`cloud>	 YuviPanda: so looks OK?
[23:19:07] <Krenair>	 hoo, what's wrong then?
[23:19:18] <hoo>	 how is that supposed to work, even
[23:19:34] <mutante>	 bd808: yea, reference:)
[23:19:45] <YuviPanda>	 valhallasw`cloud: \o/
[23:19:54] <YuviPanda>	 mutante I also left a comment there earlier
[23:20:10] <icinga-wm>	 PROBLEM - puppet last run on mw1101 is CRITICAL puppet fail
[23:20:40] <SPF|Cloud>	 :o
[23:21:10] <hoo>	 ah, there's a shift below the initial $db
[23:21:19] <grrrit-wm>	 (03PS2) 10BBlack: Add cache_maps and related LVS config, using cp104[34] [puppet] - 10https://gerrit.wikimedia.org/r/230246 (https://phabricator.wikimedia.org/T105076) 
[23:21:43] <hoo>	 that script is just so awry
[23:21:54] <Krenair>	 yes
[23:21:56] <grrrit-wm>	 (03CR) 10Yurik: [C: 031] "bblack is awesome!!!!!!!!!!!!!!! It seems good to me so far, but not an expert in puppets... yet" [puppet] - 10https://gerrit.wikimedia.org/r/230246 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[23:22:00] <grrrit-wm>	 (03CR) 10Hoo man: "nvm" [puppet] - 10https://gerrit.wikimedia.org/r/223365 (https://phabricator.wikimedia.org/T105046) (owner: 10Alex Monk)
[23:26:34] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] "..." [puppet] - 10https://gerrit.wikimedia.org/r/230246 (https://phabricator.wikimedia.org/T105076) (owner: 10BBlack)
[23:26:36] <mutante>	 YuviPanda: the instance is configured now to not use it anymore. sorry for the delay, had to reset 2factor auth and re-enable it, it's really enforced too ,heh
[23:26:52] <mutante>	 YuviPanda: comment where
[23:27:02] <YuviPanda>	 mutante :) https://gerrit.wikimedia.org/r/#/c/194796/7/modules/labs_debrepo/manifests/init.pp
[23:31:16] <grrrit-wm>	 (03CR) 10Dzahn: move misc/labsdebrepo out of misc to module (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/194796 (owner: 10Dzahn)
[23:31:20] <grrrit-wm>	 (03PS8) 10Dzahn: move misc/labsdebrepo out of misc to module [puppet] - 10https://gerrit.wikimedia.org/r/194796 
[23:31:22] <mutante>	 YuviPanda: amended
[23:32:59] <mutante>	 ah, one more
[23:34:23] <icinga-wm>	 PROBLEM - puppet last run on cp1044 is CRITICAL puppet fail
[23:35:07] <bblack>	 ignore puppetfail on cp104[34] for now, sorry!
[23:35:31] <grrrit-wm>	 (03PS9) 10Dzahn: move misc/labsdebrepo out of misc to module [puppet] - 10https://gerrit.wikimedia.org/r/194796 
[23:35:52] <icinga-wm>	 PROBLEM - puppet last run on cp1043 is CRITICAL puppet fail
[23:36:20] <grrrit-wm>	 (03PS1) 10BBlack: bugfix for 15bdf16c: bad filename [puppet] - 10https://gerrit.wikimedia.org/r/230254 
[23:36:28] <grrrit-wm>	 (03CR) 10Dzahn: "amended for the name of the define, and removed old class from the petcow instance" [puppet] - 10https://gerrit.wikimedia.org/r/194796 (owner: 10Dzahn)
[23:36:45] <grrrit-wm>	 (03CR) 10BBlack: [C: 032 V: 032] bugfix for 15bdf16c: bad filename [puppet] - 10https://gerrit.wikimedia.org/r/230254 (owner: 10BBlack)
[23:39:02] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "yea, this is what he had talked about on IRC, journalctl for log access as is done for other services and all commands as the service user" [puppet] - 10https://gerrit.wikimedia.org/r/230133 (https://phabricator.wikimedia.org/T108349) (owner: 10John F. Lewis)
[23:39:18] <grrrit-wm>	 (03PS4) 10Dzahn: admin: add mailman-admins group [puppet] - 10https://gerrit.wikimedia.org/r/230133 (https://phabricator.wikimedia.org/T108349) (owner: 10John F. Lewis)
[23:39:32] <icinga-wm>	 RECOVERY - puppet last run on mw1101 is OK Puppet is currently enabled, last run 50 seconds ago with 0 failures
[23:40:05] <grrrit-wm>	 (03CR) 10CSteipp: [C: 031] "I like the concept, and it looks right to me. Haven't tested." [puppet] - 10https://gerrit.wikimedia.org/r/223365 (https://phabricator.wikimedia.org/T105046) (owner: 10Alex Monk)
[23:40:33] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "going ahead - this only adds an empty group - adding user to group will wait" [puppet] - 10https://gerrit.wikimedia.org/r/230133 (https://phabricator.wikimedia.org/T108349) (owner: 10John F. Lewis)
[23:41:46] <grrrit-wm>	 (03PS3) 10BryanDavis: logstash: Count MediaWiki log events with statsd [puppet] - 10https://gerrit.wikimedia.org/r/230233 (https://phabricator.wikimedia.org/T100735) 
[23:44:10] <grrrit-wm>	 (03CR) 10CSteipp: "Do we need to handle test.wikidata.org for SUL? It will still work if they visit, just no autologin-- not sure if that's a significant dif" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230247 (https://phabricator.wikimedia.org/T108101) (owner: 10Legoktm)
[23:46:35] <grrrit-wm>	 (03CR) 10Legoktm: "I'm not sure...I assumed that it would be easy to communicate to developers that they need to visit the wiki first before they can test th" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230247 (https://phabricator.wikimedia.org/T108101) (owner: 10Legoktm)
[23:49:37] <wikibugs>	 6operations, 10Deployment-Systems, 10RESTBase, 6Services, 5Patch-For-Review: [Discussion] Move restbase config to Ansible (or $deploy_system in general)? - https://phabricator.wikimedia.org/T107532#1520271 (10GWicke)
[23:52:30] <mutante>	 petcow fits even better now when i reinstall with jessie - Jessie is the "yodeling cowgirl" in toystory
[23:52:50] <grrrit-wm>	 (03PS16) 10BryanDavis: Update configuration for logstash 1.5.3 [puppet] - 10https://gerrit.wikimedia.org/r/226991 (https://phabricator.wikimedia.org/T99735) 
[23:53:16] <grrrit-wm>	 (03PS2) 10BryanDavis: logstash: Enable doc_values in template mapping [puppet] - 10https://gerrit.wikimedia.org/r/230250 (https://phabricator.wikimedia.org/T74930) 
[23:54:56] <grrrit-wm>	 (03CR) 10CSteipp: "hoo, does that seem reasonable? Or is there someone else we should ask about the wikidata testing workflow?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230247 (https://phabricator.wikimedia.org/T108101) (owner: 10Legoktm)
[23:56:44] <grrrit-wm>	 (03CR) 10Hoo man: "This will probably break client linkItem functionality testing (which uses CORS to testwikidata), on top of that, this sounds ok (I guess " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230247 (https://phabricator.wikimedia.org/T108101) (owner: 10Legoktm)