[07:22:46] <_joe_> I doubt kustomize can work with helm [07:23:03] <_joe_> or well if it can, I can't imagine the level of indirection you get to :D [12:09:44] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Deploy version 1.10.0 of wikidiff2 to production - https://phabricator.wikimedia.org/T236963 (10jijiki) Version 1.10.0 has been deployed to the canaries, we can roll out to production on Monday [14:09:24] 10serviceops, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): [Bug] Chromium binary missing in proton's production docker image - https://phabricator.wikimedia.org/T238890 (10akosiaris) ` docker run --rm --entrypoint /bin/bash -it docker-registry.wikimedia.org/wikimedia/mediawiki-services... [14:12:54] 10serviceops: Build and Supply nodejs (nodejs-slim and nodejs-devel) container images - https://phabricator.wikimedia.org/T238927 (10akosiaris) [14:13:00] 10serviceops: Build and Supply nodejs (nodejs-slim and nodejs-devel) container images - https://phabricator.wikimedia.org/T238927 (10akosiaris) p:05Triage→03Normal [14:14:11] 10serviceops: Build and Supply nodejs (nodejs-slim and nodejs-devel) container images - https://phabricator.wikimedia.org/T238927 (10akosiaris) This was mentioned in T238890 (of which it is not a blocker) but we will anyway needs this for all services. [17:51:19] 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn) That's true, just had one last little todo here for the one in codfw. Doing that now. [18:03:47] 10serviceops, 10Operations: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10jijiki) [18:46:09] 10serviceops, 10Operations, 10Traffic: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10Dzahn) [19:30:14] 10serviceops, 10Operations, 10Traffic: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10CDanis) At ~18:36 there was another spike in long-tail latency, but then, latency seemed to return to 'normal': https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red... [19:41:14] 10serviceops, 10Operations: dropped packets to phab1003 22280/tcp - https://phabricator.wikimedia.org/T238781 (10ayounsi) 05Open→03Resolved Confirmed! [19:41:23] 10serviceops, 10Operations, 10Phabricator, 10Traffic: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10ayounsi) [19:48:51] 10serviceops: Build and supply buster nodejs (nodejs-slim and nodejs-devel) container images - https://phabricator.wikimedia.org/T238927 (10Jdforrester-WMF) [20:58:01] 10serviceops, 10Operations, 10ops-eqiad: (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10wiki_willy) [21:02:13] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10ssastry) @Joe @Dzahn can that memory bump patch ( https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/548944 ) be deployed next week? Or, are we waiting... [22:04:05] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) [22:04:26] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) [22:04:33] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn) [22:06:33] 10serviceops, 10Release-Engineering-Team: decom phab1003 - https://phabricator.wikimedia.org/T238957 (10Dzahn) [22:07:18] 10serviceops, 10Operations, 10ops-eqiad: (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10jijiki) @wiki_willy I will provide racking instructions on Monday for you, sorry we have delayed you this much. [22:07:53] 10serviceops, 10Release-Engineering-Team: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Dzahn) 05Open→03Stalled p:05Triage→03Normal [22:07:56] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) [22:08:08] 10serviceops, 10Operations, 10ops-eqiad: (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10wiki_willy) Thanks @jijiki , much appreciated [22:09:39] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) p:05Triage→03High @20after4 Let's find a date for this. Preferably relatively soon before the holidays and not during the holidays as we initially planned. [22:10:07] 10serviceops, 10Operations, 10Traffic: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10jijiki) [22:10:49] 10serviceops, 10Operations, 10Traffic: Increased latency in appservers - 22 Nov 2019 - https://phabricator.wikimedia.org/T238939 (10jijiki) [22:11:05] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn) [22:11:08] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) [22:11:16] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Open→03Resolved [22:11:24] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn) [22:11:46] 10serviceops, 10Operations, 10hardware-requests: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10Dzahn) We will give this back in T238957. [22:16:22] 10serviceops, 10Operations, 10Phabricator, 10hardware-requests, 10Release-Engineering-Team (Development services): The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10Dzahn) After further discussion with Mukunda a... [22:16:35] 10serviceops, 10Operations, 10Phabricator, 10hardware-requests, 10Release-Engineering-Team (Development services): The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10Dzahn) 05Open→03Declined [22:16:40] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10Dzahn) [23:24:45] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Dzahn) For my part it is blocked on first merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/546448 because it uses that.