[07:34:03] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1023.eqiad.wmnet ` The log can be... [07:34:15] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2023.codfw.wmnet ` The log can be... [07:49:36] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [08:12:10] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1023.eqiad.wmnet'] ` and were **ALL** successful. [08:34:30] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2023.codfw.wmnet'] ` and were **ALL** successful. [09:58:57] 10serviceops, 10Operations, 10User-jijiki: Upgrade memcached to version 1.6.x - https://phabricator.wikimedia.org/T270315 (10Volans) p:05Triage→03Medium [10:00:02] 10serviceops, 10Dumps-Generation, 10Operations, 10Platform Engineering: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) Preliminaries: - build mwbzutils package for buster and make sure it passes all tests [16:30:47] Wkandek: Hi, I'm in a car and can't get the laptop to start and don't have Google login on phone. Can't join the meeting if it's starting now. Sorry [16:31:24] Rlazarus: see above please, using webchat to send a message ;) [16:31:36] ack thanks [16:31:54] Thanks. I will send an email later. [16:54:59] 10serviceops, 10Operations, 10Performance-Team, 10Traffic, and 2 others: Very long response time on frwiki main page - https://phabricator.wikimedia.org/T266865 (10Aklapper) And we might face this again, see T270631 [16:55:36] 10serviceops, 10Operations, 10Traffic, 10Performance Issue: When logged in, loading the frwiki homepage takes a very long time - https://phabricator.wikimedia.org/T270631 (10Aklapper) [17:02:03] _joe_: are you ready for me to merge the shellbox pipeline patches? [17:02:11] <_joe_> yes [17:02:44] <_joe_> but I tweaked them further, because a new blubber version came out making us able to avoid that ugly shell script \o/ [17:03:23] <_joe_> so I'd like one further review from you, and I also have to talk to moritz at some point about lillypond for buster [17:03:52] ok [17:04:40] <_joe_> and btw, I also have one integration/config patch up to remove the other tests and just run them via the pipeline [17:05:10] based on https://security-tracker.debian.org/tracker/source-package/lilypond (and the fact that Moritz uploaded the most recent version to buster) I'd think the one in Debian is fine for now [17:06:07] <_joe_> I think so too [17:22:50] 10serviceops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) I think I'm ready to do this whenever -- I upgraded our test host and it looks just fine. [17:29:59] 10serviceops, 10Operations, 10Traffic, 10Performance Issue: When logged in, loading the frwiki homepage takes a very long time - https://phabricator.wikimedia.org/T270631 (10Legoktm) Yeah, this looks like the exact same issue as {T266865}. `Memcached error for key "WANCache:frwiki:featured-feeds:1:fr|#|v"... [17:40:33] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1024.eqiad.wmnet ` The log can be... [17:40:46] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2024.codfw.wmnet ` The log can be... [18:01:40] 10serviceops, 10Operations, 10Performance-Team, 10Traffic, and 2 others: Very long response time on frwiki main page - https://phabricator.wikimedia.org/T266865 (10Legoktm) [18:06:44] 10serviceops, 10Operations, 10Traffic, 10Performance Issue: When logged in, loading the frwiki homepage takes a very long time - https://phabricator.wikimedia.org/T270631 (10Legoktm) >>! In T270631#6706192, @Legoktm wrote: > one of the frwiki feeds is too big to fit in memcache. Either we need to figure ou... [18:18:43] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2024.codfw.wmnet'] ` and were **ALL** successful. [18:22:48] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1024.eqiad.wmnet'] ` and were **ALL** successful. [18:45:35] 10serviceops, 10Operations, 10Traffic, 10Performance Issue: When logged in, loading the frwiki homepage takes a very long time - https://phabricator.wikimedia.org/T270631 (10Legoktm) p:05Triage→03High [21:55:22] 10serviceops, 10Operations: Add python3.8 to buster-wikimedia pyall component - https://phabricator.wikimedia.org/T241195 (10Legoktm) 05Open→03Resolved a:05MoritzMuehlenhoff→03Legoktm During the import I ran into: {P13619} Moritz said this has happened with Jenkins before, and it seems like something... [21:58:34] 10serviceops, 10MW-on-K8s, 10Operations, 10Shellbox: Decide on logging in k8s for ShellBox - https://phabricator.wikimedia.org/T263545 (10Legoktm) [21:58:43] 10serviceops, 10MW-on-K8s, 10Operations, 10Shellbox, and 3 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10Legoktm) [21:58:46] 10serviceops, 10MW-on-K8s, 10Operations, 10Shellbox, 10Platform Team Workboards (Purple): Make Shellbox actually do streaming - https://phabricator.wikimedia.org/T268427 (10Legoktm) [23:04:27] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Legoktm) >>! In T261369#6695002, @akosiaris wrote: >>>! In T261369#6693707, @Krinkle wrote: >> Some... [23:05:51] 10serviceops, 10Shellbox, 10Continuous-Integration-Config: Set up PHP security scanning for Shellbox - https://phabricator.wikimedia.org/T270660 (10Legoktm) [23:22:20] 10serviceops, 10Operations, 10Scap, 10Wikimedia-General-or-Unknown, and 2 others: "Currently active MediaWiki versions:" broken on noc/conf - https://phabricator.wikimedia.org/T235338 (10Legoktm) 05Open→03Resolved a:05jijiki→03Legoktm