[10:18:56] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [10:19:42] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [10:26:18] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10jijiki) 05Open→03Resolved a:03jijiki [10:27:57] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki) [10:28:04] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10jijiki) [10:28:07] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki) 05Open→03Resolved [10:38:04] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) Looking at mc1035's behaviour after reimaging, it appears that its hit ratio reached 0.95 after ~10h, and it reached 28Mil... [10:40:13] https://gerrit.wikimedia.org/r/646967 [10:40:18] ^ elukey akosiaris [10:42:14] -3, it's clear it needs to be bullseye already :P [10:42:59] your timemachine is functioning nicely [10:43:05] mine is stuck in the present :p [10:43:16] I 'll defer to elukey btw, not sure about that shard16 stuff [10:43:39] sure, the tldr is that our total redis cluster size is ~2G [10:43:51] and by not sure, I mean I am actually not very familiar with that [10:43:55] so we can easily remove some shards without affecting much [10:44:29] the impact of removing a shard is within reason, we can lose an mc* host any given time [10:45:02] there are 3 special redis shards that are confugured in mediawiki, but shard16 is not one of them [10:45:26] nutcracker starts resharding a shard's data as soon as it becomes unavailable [10:46:15] and we did the same thing ~15 hours ago, it was fine [10:46:31] I will wait for luca too then [10:46:53] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10MoritzMuehlenhoff) [10:55:57] effie: o/ I am off today, but I +1ed. I am a little concerned about all the keys in a shard vanishing all at once, buuut if this is ok let's reimage :) [10:56:12] (in a redis shard I mena) [10:56:16] *mean [11:03:30] for example, IIRC echo uses the main stash on redis to store notifications for users.. if we nuke a shard all new notifications will be re-sharded, but not the current ones [11:03:56] (it might not be the correct picture but it should be not too far from what happens) [11:06:14] On the other hand, a while ago we realized that redis was constantly evicting things due to the cache space being full, and we didn't get any user report so.. [11:06:32] (the eviction rate was fixed since then) [11:07:33] anyway, +1 from me to proceed, but we should keep a slow pace and check user reports (maybe reaching out to some community liaison could be good) [12:08:08] https://groups.google.com/g/kubernetes-dev/c/BrYFA9Fbko0/m/wQ7Q3byGAAAJ . Seems like we are moving off from docker after all. Been some time coming [12:09:01] elukey: thank you! [12:13:20] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [13:50:46] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2034.codfw.wmnet ` The log can be... [14:16:24] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2034.codfw.wmnet'] ` and were **ALL** successful. [14:23:43] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [14:43:10] Apertium migration done :-) [14:44:08] akosiaris: \o/ \o/ \o/ [14:49:11] great akosiaris! [15:07:34] 10serviceops: [EPIC] Docker deprecation as a container runtime enginer for kubernetes. - https://phabricator.wikimedia.org/T269684 (10akosiaris) [15:08:02] jayme :-) [15:08:11] But I got us a gift it seems as well: https://phabricator.wikimedia.org/T269684 [15:08:26] Anyway, we got some time, they did give us a pretty early warning [15:10:28] yep...I already thought it would maybe smart to take a look with the migration to buster as we need to upgrade docker then as well... [15:18:26] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10hashar) [15:18:30] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10hashar) 05Resolved→03Open **Summary:** buster-wikimedia components/php72 provides php-wikidiff2 1.8.1 instead of 1.10.0 I have compared the Strech versus Buster images... [15:31:00] akosiaris: thanks for that email about the docker CRE deprecation, I'd been meaning to ask what that meant for us [15:31:49] yw. Thankfully not much (I hope). [15:34:56] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10jijiki) 05Resolved→03Open @Urbanecm reported that scap is throwing some warnings: ` 15:27:45 WARNING - Duplicate plugin named CheckoutMediaWiki, skipping.... [15:35:29] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10jijiki) p:05Triage→03High [15:36:02] 10serviceops, 10Discovery-Search, 10Maps, 10Product-Infrastructure-Team-Backlog: [OSM] Backport imposm3 to the debian channel - https://phabricator.wikimedia.org/T238753 (10Jgiannelos) I wrapped up (partially out of curiosity) the packaging of imposm3 with deb build dependencies, in case we prefer to avoid... [15:49:22] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1034.eqiad.wmnet ` The log can be... [15:51:40] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10MoritzMuehlenhoff) >>! In T250515#6676402, @hashar wrote: > **Summary:** buster-wikimedia components/php72 provides php-wikidiff2 1.8.1 instead of 1.10.0 > > > I have compa... [16:04:18] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10LarsWirzenius) p:05High→03Medium This is due to {T248490}, and it's harmless. mediawiki-config contains a bunch of plugins for Scap, and they've been includ... [16:15:28] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1034.eqiad.wmnet'] ` and were **ALL** successful. [16:57:20] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10Jdforrester-WMF) >>! In T250515#6676486, @MoritzMuehlenhoff wrote: >>>! In T250515#6676402, @hashar wrote: >> **Summary:** buster-wikimedia components/php72 provides php-wiki... [17:03:53] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10LarsWirzenius) 05Open→03Resolved mediawiki-config has dropped tthe plugins. Issue should be resolvedd. [17:17:27] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) a:03Andrew [17:18:31] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) p:05Triage→03Medium [17:25:50] 10serviceops, 10Release-Engineering-Team, 10Security, 10cloud-services-team (Kanban): Implement SSH CA (certificate authority) for host keys? - https://phabricator.wikimedia.org/T268344 (10Andrew) p:05Triage→03Low Patches welcome! This isn't something we're likely to have time for anytime soon. [18:08:23] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10dduvall) [19:05:12] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) It's most useful if effort is directed towards completing T237773, which will render this issue moot. In theory the DBAs are going to work on the first step of t...