[03:01:04] 10serviceops, 10Operations, 10Datacenter-Switchover: Updates to warmup script - https://phabricator.wikimedia.org/T269179 (10Krinkle) >>! In T269179#6662243, @Volans wrote: >> you were thinking about rewriting the warmup script in Python -- you were kind enough to let me talk you out of doing that right befo... [03:01:33] 10serviceops, 10Operations, 10Datacenter-Switchover: Updates to warmup script - https://phabricator.wikimedia.org/T269179 (10Krinkle) [08:08:12] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10MoritzMuehlenhoff) I checked the journalctl of ferm on parse2001 and there are no errors logged, so it's not a case of Ferm failing to start (via syntax errors, failing DNS lookup or whatev... [09:39:35] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10akosiaris) [10:39:18] I would welcome +1s for https://gerrit.wikimedia.org/r/c/operations/puppet/+/643234 [10:39:27] and https://gerrit.wikimedia.org/r/c/operations/puppet/+/643235 [10:39:34] this has been rolled to app and api [10:39:50] in theory, it is safe to roll out to the rest [10:44:05] jayme: akosiaris wink wink [10:46:01] 10serviceops, 10Operations, 10conftool, 10Datacenter-Switchover: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10Volans) I'm not familiar on what's the usual execution time of all the scripts, but I'm wondering if we also need to stop all the systemd unit triggered b... [12:28:08] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [12:30:34] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [13:10:43] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [13:25:47] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [13:26:33] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10akosiaris) [13:56:54] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [13:58:19] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [14:15:27] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor calico deploy strategy - https://phabricator.wikimedia.org/T267653 (10JMeybohm) a:03JMeybohm [15:48:38] 10serviceops, 10Operations, 10Datacenter-Switchover: Updates to warmup script - https://phabricator.wikimedia.org/T269179 (10RLazarus) [17:03:18] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [17:59:32] 10serviceops, 10Graphoid, 10Operations, 10Patch-For-Review, 10Platform Engineering (Icebox): Undeploy graphoid for phase 3 wiki's - https://phabricator.wikimedia.org/T259207 (10Jseddon) [18:03:21] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) >>! In T268524#6665888, @MoritzMuehlenhoff wrote: > let's maybe reimage also parse2002 to show whether it's a reproducible error and if it happens again, keep it in the broken state... [18:09:33] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10jijiki) >>! In T268524#6667353, @Dzahn wrote: >>>! In T268524#6665888, @MoritzMuehlenhoff wrote: >> let's maybe reimage also parse2002 to show whether it's a reproducible error and if it h... [18:10:41] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [18:10:45] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) Alright, will do that and start in a couple minutes. [18:22:52] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10MoritzMuehlenhoff) >>! In T268524#6667353, @Dzahn wrote: >>>! In T268524#6665888, @MoritzMuehlenhoff wrote: >> let's maybe reimage also parse2002 to show whether it's a reproducible error... [18:23:00] 10serviceops, 10Operations, 10Platform Engineering: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10jijiki) [18:23:31] 10serviceops, 10Operations, 10Platform Engineering: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10jijiki) [18:23:37] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [19:21:10] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` parse2001.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020120319... [20:10:20] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2001.codfw.wmnet'] ` and were **ALL** successful. [21:27:27] 10serviceops, 10Discovery-Search, 10Maps, 10Product-Infrastructure-Team-Backlog: [OSM] Backport imposm3 to the debian channel - https://phabricator.wikimedia.org/T238753 (10akosiaris) >>! In T238753#6630924, @MSantos wrote: > @hnowlan this can be a good resource for this task https://github.com/omniscale/i... [21:45:48] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) What can I say.. it did not happen this time. Reimaged, ran puppet.. waited a bit, checked Icinga.. all green with the following exceptions: - Ensure local MW versions match expect... [23:00:05] 10serviceops, 10Operations, 10conftool, 10Datacenter-Switchover: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10RLazarus) Oh, that's a good idea! We'd set `maintenance_host` to the empty string before the switchover, so that no new jobs would start anywhere, then se...