[00:54:16] 10serviceops, 10Gerrit, 10Icinga, 10Operations, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) The new check "Gerrit JSON" works now: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi... [00:56:20] 10serviceops, 10Gerrit, 10Icinga, 10Operations, and 2 others: improve Gerrit monitoring (was: Investigate why icinga did not report high cpu/load for gerrit) - https://phabricator.wikimedia.org/T215033 (10Dzahn) 05Open→03Resolved a:03Dzahn [03:37:34] 10serviceops, 10Language-Team, 10MediaWiki-Language-converter, 10Parsing-Team, and 5 others: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice? - https://phabricator.wikimedia.org/T213345 (10Krinkle) [03:40:32] 10serviceops, 10Language-Team, 10MediaWiki-Language-converter, 10Parsing-Team, and 5 others: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice? - https://phabricator.wikimedia.org/T213345 (10Krinkle) Re-tagging "RFC"-like task on the TechCom workboard as actual RFC. Moving t... [06:13:00] 10serviceops, 10Operations, 10vm-requests, 10Patch-For-Review, 10User-fsero: eqiad: 1-2 VM requests for docker-registry-beta.wikimedia.org - https://phabricator.wikimedia.org/T212212 (10fsero) [06:14:06] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Make swift containers for docker registry cross replicated. - https://phabricator.wikimedia.org/T214289 (10fsero) [06:14:55] 10serviceops, 10Citoid, 10Operations, 10Patch-For-Review, and 2 others: allow zotero container nodejs server to define the amount of heap used instead of the fixed limit of 1.7Gi - https://phabricator.wikimedia.org/T213414 (10fsero) [06:15:10] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, and 2 others: improve docker registry architecture - https://phabricator.wikimedia.org/T209271 (10fsero) [06:15:31] arg [06:15:33] too noisy [09:24:24] 10serviceops, 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, and 3 others: Document and possibly fine-tune how Proton interacts with Varnish - https://phabricator.wikimedia.org/T213371 (10akosiaris) >>! In T213371#4932956, @pmiazga wrote: > @Tgr I assume you're still waiting for answers fro... [09:24:41] 10serviceops, 10Gerrit, 10Icinga, 10Operations, and 2 others: gerrit: Add a icinga check that uses the healthcheck endpoint - https://phabricator.wikimedia.org/T215457 (10hashar) [09:33:30] <_joe_> fsero, akosiaris let's migrate to https://github.com/ibuildthecloud/k3s [09:36:31] 10serviceops, 10Operations, 10vm-requests, 10Release-Engineering-Team (Watching / External): Increase mwdebugXXXX hosts CPU and memory(?) - https://phabricator.wikimedia.org/T212955 (10hashar) I think @fsero / @akosiaris should be able to bump the number of CPUs on those Ganeti instances :-] We can try wit... [09:46:04] 10serviceops, 10Operations, 10vm-requests, 10Release-Engineering-Team (Watching / External): Increase mwdebugXXXX hosts CPU - https://phabricator.wikimedia.org/T212955 (10akosiaris) [09:49:07] 10serviceops, 10Operations, 10vm-requests, 10Release-Engineering-Team (Watching / External): Increase mwdebugXXXX hosts CPU - https://phabricator.wikimedia.org/T212955 (10akosiaris) 05Open→03Resolved a:03akosiaris I 've removed the memory part cause https://grafana.wikimedia.org/d/000000377/host-over... [09:56:01] akosiaris: updated master as well https://gerrit.wikimedia.org/g/operations/debs/helm/+/refs/heads/master [09:56:13] sadly gbp created two commits from the merge from upstream [09:56:20] and i wasnt able to submit it to gerrit for review [09:56:29] however the changed part is already reviewed [09:56:34] so i hope is ok this way [10:05:28] fsero: that's fine [10:05:59] going to update deploy* and contint* is only the client part and it should talk with 2.8 tiller [10:06:06] tiller upgrade should be done afterwards [10:06:11] cool [11:07:52] 10serviceops, 10Operations, 10User-jijiki: Fix spamassassin's "warn: netset: cannot include " warning - https://phabricator.wikimedia.org/T215496 (10jijiki) p:05Triage→03Normal [11:34:07] _joe_: akosiaris @anyone i want to rebuild the tiller docker image. It seems there is no CI job that clones the repo, executes docker-pkg and publish it under the registry, so i guess i should run it manually somewhere "where" is the place i should do it ¿boron? it doesnt have docker-pkg installed [11:34:30] <_joe_> boron has docker-pkg installed [11:34:33] <_joe_> to a virtualenv [11:34:38] <_joe_> but you don't need that [11:35:24] <_joe_> sudo /usr/local/bin/build-production-images [11:35:40] ook [11:35:43] <_joe_> DTRT once you've updated the clone under /srv/images/production-images [11:36:10] <_joe_> I agree that we should set up a CI job to do that, but it requires a few things IMHO [11:36:29] <_joe_> starting from a dedicated CI slave (maybe on ganeti? I dunno) [11:36:42] <_joe_> I wouldn't want to add it to contint1001 [11:37:07] please reserve some of your scarce free time to discuss about that [11:37:08] <_joe_> the plan has always been to do CD of images, we just never got there [11:37:15] i would really like to have more thing into CI [12:14:29] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Joe) [13:24:16] 10serviceops, 10Operations, 10Wikimedia-General-or-Unknown, 10PHP 7.2 support, 10User-jijiki: mwscript dies on mwmaint with PHP=php7.2 due to php-redis missing - https://phabricator.wikimedia.org/T215376 (10Krenair) >>! In T215376#4932704, @Dzahn wrote: >>>! In T215376#4932577, @Reedy wrote: >> In `modul... [13:51:35] "Server HA: Just don't right now :) It's currently broken" [14:25:15] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10akosiaris) How is the data going to make it from Hadoop, which resides in the analytics cluster and is firewalled at the router level... [14:29:00] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Ottomata) > How is the data going to make it from Hadoop, which resides in the analytics cluster and is firewalled at the router level... [14:33:43] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10bmansurov) >>! In T213566#4934832, @akosiaris wrote: > Is it just a `LOAD DATA INFILE "something.tsv"` or is it something more complex... [14:41:16] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10akosiaris) >>! In T213566#4934835, @Ottomata wrote: >> How is the data going to make it from Hadoop, which resides in the analytics cl... [14:44:31] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10bmansurov) > That does look simple enough and not resource expensive on mwmaint1002. I guess it can fit in there as well? But a VM is... [14:56:43] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Ottomata) > they will also not allow them to send the SYN/ACK packet required for the second (of the three) phase of the TCP handshake... [15:07:28] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 2 others: Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10EvanProdromou) So, re-reading https://phabricator.wik... [15:17:59] 10serviceops, 10Operations, 10Wikimedia-General-or-Unknown, 10PHP 7.2 support, 10User-jijiki: mwscript dies on mwmaint with PHP=php7.2 due to php-redis missing - https://phabricator.wikimedia.org/T215376 (10Dzahn) If we use "present" (and not a specific version or "latest" either) we would get whatever t... [17:17:07] btw, the breakfast thing is on youtube..where you can go back in time to see the beginning [17:18:15] I'm watching the youtube stream indeed, as always [17:52:22] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 2 others: Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10EvanProdromou) One thing we'd need to make sure of is... [20:36:17] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Nuria) Ideally I would prefer that stats machines are completely out of the workflow of pushing data to machines like mwmaint1002.eqia... [20:43:16] 10serviceops, 10Analytics, 10Operations, 10Research, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Nuria) As @Ottomata pointed out more generic discussion about this topic can be found here: https://phabricator.wikimedia.org/T213976 [21:12:00] 10serviceops, 10CirrusSearch, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review: Find an alternative to HHVM curl connection pooling for PHP 7 - https://phabricator.wikimedia.org/T210717 (10debt) Moving to #discovery-search-sprint waiting column to see if there is anything else we need...