[00:05:51] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: eqiad: Physical moves for MediaWiki servers - https://phabricator.wikimedia.org/T266164 (10Dzahn) 05Open→03Resolved per Willy the remaining ones are also listed on T267065 which is a wider task about the same thing. Suggesting to... [02:53:13] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM - https://phabricator.wikimedia.org/T247653 (10Dzahn) [02:53:22] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM - https://phabricator.wikimedia.org/T247653 (10Dzahn) [02:53:33] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM - https://phabricator.wikimedia.org/T247653 (10Dzahn) a:03Dzahn [02:54:01] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) [11:17:54] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10Pginer-WMF) 05Open→03Resolved [13:15:22] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1021.eqiad.wmnet ` The log can be... [13:15:41] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2021.codfw.wmnet ` The log can be... [13:41:04] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1021.eqiad.wmnet'] ` and were **ALL** successful. [13:56:11] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [13:56:16] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor calico deploy strategy - https://phabricator.wikimedia.org/T267653 (10JMeybohm) 05Open→03Resolved This is done with calico deployed now via puppet (CNI plugins and calicoctl) as well as helm3 (helmfile.d/admin_n... [13:59:35] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) We packaged and deployed calico 3.16 to staging-codfw but currently we lack full IPv6 support due to the fact that we now require kubernetes to dual-stack/IPv... [14:03:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) As of https://gerrit.wikimedia.org/r/c/operations/puppet/+/648356 we're now running staging-codfw with docker 18.06.3... [14:05:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver flag --admission-control has been deprecated - https://phabricator.wikimedia.org/T270063 (10JMeybohm) [14:10:04] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2021.codfw.wmnet'] ` and were **ALL** successful. [14:17:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubernetes tmpfiles references path below legacy directory /var/run/ - https://phabricator.wikimedia.org/T270298 (10JMeybohm) a:03JMeybohm I don't see those directories being used anywhere so I suggest we just remove the tmpfile registration from master and no... [14:28:27] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) 05Open→03Resolved >>! In T270270#6699809, @ema wrote: > I think we can revert the no-s... [15:07:37] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10hashar) Thank you @ema for the full explanation (and for the fix of course)! [15:51:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Restart kubernetes services on package upgrades - https://phabricator.wikimedia.org/T270302 (10JMeybohm) a:03JMeybohm [15:55:13] I'll leave for the "holidays" now...see you all next year (latest) or when we all get bored ;-) [15:58:18] so long jayme :) [16:03:45] have a great break! [16:07:27] <_joe_> jayme: have fun [18:37:17] 10serviceops, 10Operations: Investigate opcache hit rate on Buster appserver - https://phabricator.wikimedia.org/T270517 (10RLazarus) p:05Triage→03Medium [18:38:09] 10serviceops, 10Operations: Investigate opcache hit rate on Buster appserver - https://phabricator.wikimedia.org/T270517 (10RLazarus) [18:38:14] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10RLazarus) [20:59:10] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) Thanks to @CDanis for digging into this with me. There are a couple of different things going on. # The thumbor service has a lot of trouble producing a 2880-pixel version... [21:14:00] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) [21:29:54] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) [21:30:31] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10AntiCompositeNumber) > When I fetched it directly, the request hung for a minute or so before coming back as a 503, likely because of a timeout somewhere in the stack. Yup, Thumbor g... [21:34:02] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) Ah, cheers @AntiCompositeNumber, I thought it sounded familiar. So it sounds like this will get better with T216815, which I think I've heard mutterings about doing Soon™. [21:45:30] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) So, since the root cause here is a known issue, I'm duping this ticket over to the SVG rendering issue, but thanks @RoySmith for filing -- the subtasks are still open and wi... [21:46:04] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) [21:54:04] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10AntiCompositeNumber) [23:35:23] 10serviceops, 10Operations, 10Thumbor: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) Doh, thanks @AntiCompositeNumber. :)