[17:29:28] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) @jijiki, my tests suggest that this upgrade will go smoothly. If you judge MW to be mostly ready for Buster then I'll go ahead and upgrade things this week; if n... [17:39:39] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubernetes tmpfiles references path below legacy directory /var/run/ - https://phabricator.wikimedia.org/T270298 (10JMeybohm) 05Open→03Resolved [17:39:43] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [17:39:45] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Restart kubernetes services on package upgrades - https://phabricator.wikimedia.org/T270302 (10JMeybohm) 05Open→03Resolved [17:39:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [18:20:56] 10serviceops, 10LibUp, 10Security: Ensure that all deployed nodejs services have regular dependency security scanning - https://phabricator.wikimedia.org/T271132 (10Legoktm) [18:45:21] 10serviceops, 10LibUp, 10Security: Ensure that all deployed nodejs services have regular dependency security scanning - https://phabricator.wikimedia.org/T271132 (10Legoktm) I think the missing repos appear to be: * wikibase/termbox * eventgate-wikimedia * research/mwaddlink (written in Python) * blubber (g... [18:55:31] 10serviceops, 10SRE-tools: Some Service Operations clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271142 (10crusnov) [19:15:28] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Legoktm) Specifically regarding pygmentize, we should be using the version that's currently bundled... [19:42:08] 10serviceops, 10SRE-tools, 10IPv6: Some Service Operations clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271142 (10Aklapper) [20:03:27] I am going to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/654231 and then fix the php-readline situation on mwmaint* [21:22:19] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [21:22:28] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [21:22:47] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [21:45:47] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [21:45:58] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) Etherpad created: https://etherpad.wikimedia.org/p/appserver-buster-upgrade [22:11:32] 10serviceops, 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Legoktm) [22:11:36] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [22:27:36] mutante: what's the practical difference between metal and ganeti, is it just performance? [22:30:12] also cost and setup time [22:33:58] legoktm: I feel like the 2 of you already listed the main difference. Maybe flexbility too. it's easier to add a second disk to a VM when needed [22:35:36] everything is easier with virtualization (as long as you are not in charge of the virt layer) [22:35:45] hardware is the worst [22:40:14] from a skim, most of the cron jobs seem db query intensive rather than on the host's CPU/memory [22:57:44] 10serviceops, 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Legoktm) [23:16:04] 10serviceops, 10Operations, 10Parsoid-Tests, 10Parsoid (Tracking), 10Patch-For-Review: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10Dzahn) [23:18:38] 10serviceops, 10Operations, 10Parsoid-Tests, 10Parsoid (Tracking), 10Patch-For-Review: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10Dzahn) A generic mariadb server has now been installed by puppet on testreduce1001. (no change on scandium which... [23:22:00] 10serviceops, 10Operations, 10Parsoid-Tests, 10Parsoid (Tracking), 10Patch-For-Review: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10ssastry) [23:26:59] 10serviceops, 10Operations, 10Parsoid-Tests, 10Parsoid (Tracking), 10Patch-For-Review: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10Dzahn) Next we need to re-create the DNS entries (parsoid-rt-tests, parsoid-vd-tests) before we can point them t... [23:53:00] 10serviceops, 10Operations, 10Parsoid-Tests, 10Parsoid (Tracking), 10Patch-For-Review: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10ssastry) Couple observations: 1. We don't need parsoid-vd-tests on testreduce1001 anymore since there are no imm...