[05:50:48] can someone purge varnish for releases.wikimedia.org urls ... i released a new deb and apt-get update doesn't yet see it because of cached info. [06:32:53] subbu: since we can both see the new verson on releases.wikimedia.org, something else might be up [06:33:08] we'll have a look later [06:33:28] ok [08:17:24] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10elukey) I just acked two icinga LVS alerts for restrouter in icinga, please let me know if they were something different :) [09:10:44] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, 10Patch-For-Review: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10MoritzMuehlenhoff) p:05Triage→03Normal [09:43:10] 10serviceops, 10Operations, 10ops-eqiad: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet - https://phabricator.wikimedia.org/T241850 (10jijiki) @Jclark-ctr Can you provide a date that is convenient for you for racking these? Thank you! [10:27:57] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10MoritzMuehlenhoff) p:05Triage→03Normal [11:09:25] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10jijiki) @Krinkle, I am fine to merge any time, as soon as you have tested it:) [11:31:29] 10serviceops, 10Operations, 10ops-eqiad: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10elukey) Hi @Jclark-ctr, any timeline for these hosts to be racked? [14:08:19] FYI, I'll merge https://gerrit.wikimedia.org/r/#/c/operations/dns/+/564588/ to point url downloaders in codfw to the new buster hosts; it's mostly zotero, and citoid accessing these and my tests with manual proxy accesses were all fine [14:29:08] <_joe_> moritzm: ack thanks [15:06:19] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10ssastry) [16:42:34] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, 10Patch-For-Review: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10thcipriani) [17:07:18] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) Now blocked by {T242770}. [17:21:48] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) [17:22:04] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) [17:26:01] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) 05Open→03Stalled p:05Triage→03Normal [17:47:37] 10serviceops, 10Operations, 10Release-Engineering-Team: Hundreds of tags for `wikimedia/mediawiki-core` image - https://phabricator.wikimedia.org/T242775 (10Joe) p:05Triage→03High [17:54:11] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) [18:00:59] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10Dzahn) Did it use releases.wikimedia.org as an APT repo before the upload of the new version as well? Or did it use apt.wikimedia.org as the repo in the past? [18:04:55] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10Dzahn) https://releases.wikimedia.org/parsoid/ does not show the 0.11 version to me, only all older versions. But https://releases.wikimedia.org/debian/pool/main/p/parsoid/ shows only the new versio... [18:07:30] 10serviceops, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move old transparency report pages to historical URLs and setup redirect - https://phabricator.wikimedia.org/T230638 (10Jdforrester-WMF) 05Open→03Resolved a:03BBlack This looks fully done. Thank you! [18:11:04] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10Dzahn) I can't reproduce the error. When i add it as a fresh repo and import the key per https://wikitech.wikimedia.org/wiki/Releases.wikimedia.org and then run apt-get update it shows no errors (504... [18:12:16] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10Dzahn) I'll assume it was literally the "Mirror sync in progress" (103.102.166.224 is eqsin). [19:43:35] 10serviceops, 10Performance-Team, 10Patch-For-Review: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) [20:41:49] 10serviceops, 10Operations, 10ops-eqiad: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10Cmjohnson) [23:17:29] 10serviceops: Decide on librdkafka deployment model for k8s services - https://phabricator.wikimedia.org/T242810 (10holger.knust)