[00:58:50] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) [01:04:31] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) Update: 2 new buster VMs in eqiad are running, and I depooled the 2 stretch ones, will delete them on Monday if no other problems arise. In codfw 1 buster VM is running alongside the 2... [08:07:38] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10JMeybohm) >>! In T272550#6884932, @Legoktm wrote: > In codfw 1 buster VM is running alongside the 2 stretch ones, except I accidentally created the second new VM with the wrong name (`registry20... [08:13:10] jayme: https://phabricator.wikimedia.org/T276381#6884923 [08:13:31] oh, nice [08:14:25] maybe it gets confused because of the number/dc missmatch :) [08:14:30] I proposed https://gerrit.wikimedia.org/r/#/c/668572 but I think we need a interim workaround [08:14:41] Well I didn't finish the creation of the VM [08:14:52] so it's not in puppetdb, etc. [08:15:12] ah, okay. I see. That was not clear to me in first place [08:15:20] Jenkins complained when I tried to do https://gerrit.wikimedia.org/r/#/c/668571 [08:16:09] Also see -sre [08:16:58] ack [09:18:29] FYI we're termbox deploying again :) [09:56:08] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [09:56:34] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10JMeybohm) 05Open→03Resolved Switched back to staging-eqiad with no surprises. Docs at https://wikitech.wikimedia.org/wiki/Kubernet... [10:22:31] 10serviceops, 10SRE, 10User-jijiki: Modernise memcached systemd unit / sync to current buster setup - https://phabricator.wikimedia.org/T273950 (10Joe) `systemd-memcached-wrapper` is a perl script, an evolution of the old wrapper script debian always used and that caused me more headaches than it solved. I'd... [10:31:15] starting termbox update stuff again [10:37:16] jayme: so when we are about to `helmfile apply` to production it is obviously also applying all the extra egress network stuff there. Is that "ok"? [10:42:20] 10serviceops, 10MW-on-K8s, 10Release Pipeline: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10akosiaris) >>! In T264209#6876015, @Legoktm wrote: >>>! In T264209#6549716, @akosiaris wrote: >> That being said, with compression on the client before the push takin... [10:57:31] tarrow: yeah, sorry. That's totally fine [10:57:48] great! We thought as much [11:10:58] 10serviceops, 10MW-on-K8s, 10Release Pipeline: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10Joe) For the record, we're now building the actual multiversion images of mediawiki, it would be interesting to do all testing using those. In particular it's interes... [11:31:08] jayme: we're all done and hands off again :). I think we probably should have some level of alerting to tell us that staging/test are broken so that we catch stuff early; does that sound sensible? [11:32:19] 👍 [11:42:12] 10serviceops, 10Wikidata, 10Wikidata-Termbox: Missing alerts for Termbox staging and test services - https://phabricator.wikimedia.org/T276550 (10Jakob_WMDE) [11:43:48] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech-focus: Missing alerts for Termbox staging and test services - https://phabricator.wikimedia.org/T276550 (10Addshore) [11:55:57] <_joe_> tarrow: I think that should be included in a sort of release framework that's still to be built [13:21:35] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10akosiaris) List of services in staging and happiness by service checker Through some automation (see P14643) and some manual followup... [13:35:50] tarrow: yeah. I will also have to figure out why we did not catch that on during tests on the new cluster. IIUC did get a proper response (and helm rolled back) during deployment [14:44:42] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10JMeybohm) helm test annotations changed a bit: > Note that until Helm v3, the job definition needed to contain one of these helm test hook annotations: `helm.sh/hook: test-success` or `... [21:30:20] 10serviceops, 10SRE: upgrade mwmaint servers to buster - https://phabricator.wikimedia.org/T267607 (10Dzahn) [21:31:03] 10serviceops, 10SRE: upgrade mwmaint servers to buster - https://phabricator.wikimedia.org/T267607 (10Dzahn) [21:32:03] 10serviceops, 10SRE: upgrade mwmaint servers to buster - https://phabricator.wikimedia.org/T267607 (10Dzahn) [21:34:37] 10serviceops, 10SRE: upgrade mwmaint servers to buster - https://phabricator.wikimedia.org/T267607 (10Dzahn) [21:43:34] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) [23:18:28] 10serviceops, 10SyntaxHighlight: Package latest python3-pygments for apt.wikimedia.org - https://phabricator.wikimedia.org/T276298 (10Legoktm) 05Open→03Resolved Packaged and uploaded, can be installed on appservers/elsewhere once SyntaxHighlight is ready :) Source at https://gerrit.wikimedia.org/g/operati... [23:30:59] 10serviceops, 10SyntaxHighlight: Package latest python3-pygments for apt.wikimedia.org - https://phabricator.wikimedia.org/T276298 (10Legoktm) Also filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984625 asking for 2.8.0 in Debian proper.