[00:10:26] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) changes to add the new backends to scap... [00:16:34] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle also see T149924#6699330 and... [00:17:28] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#6814030, @Krinkle wrote:... [00:59:42] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) There is definitely only doc1001 in the... [01:19:05] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) ` Got error 'PHP message: PHP Parse er... [01:25:12] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) This is a stretch server with PHP 7.0.... [01:27:58] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) >>! In T247653#6817226, @Dzahn wrote:... [01:57:43] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) after the changes above and rebooting th... [09:20:59] jayme: thanks for the reviews :) [09:21:15] legoktm: thanks for all the patches! :D [09:41:56] <_joe_> jayme / akosiaris which one you hate more? https://phabricator.wikimedia.org/P14291 [09:43:08] the latter is horrible, the former is close to maybe be readable [09:43:57] and really, just add an else and put the zero setting there [09:44:18] when I see if clauses full of else ifs without an else I get triggered [09:48:25] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Gilles) 05Resolved→03Open Didn't test this enough, it's actually blocked by {T245552}. As soon as you pick a debug server from the extension and relo... [09:48:58] <_joe_> akosiaris: well the problem is [09:49:07] <_joe_> I want to initialize the value [09:49:36] <_joe_> this is actually absolutely the same thing :) [09:50:37] functionally it's the same, but reading it always makes you wonder (oh wait, what about the cases not caught here? [09:50:56] it's about readability, not functionality [09:51:00] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Joe) I am tempted to close this as invalid, as with the upcoming move to kubernetes we will have to rethink the debug extension deeply, and a list of se... [09:51:16] <_joe_> akosiaris: so you prefer to have the variable redefined inside every else if [09:51:28] <_joe_> so $has_volumes := 1 [09:51:34] <_joe_> instead of modifying the value [09:52:20] you can initialize it outside the if clause if that makes it more readable for you. [09:52:47] it's makes the else redundant of course functionally but somewhat more readable [09:53:20] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Gilles) The code is already there. We are still getting requests to update that list in the current reality: {T274026} [09:58:21] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Joe) >>! In T268167#6817859, @Gilles wrote: > The code is already there. We are still getting requests to update that list in the current reality: {T2740... [09:59:25] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Joe) >>! In T274023#6809729, @MoritzMuehlenhoff wrote: > Why create new ones, this only creates needl... [09:59:46] _joe_: I would go for the former as well, assuming you are going to abstract that aqay into a namend template somehow [10:00:42] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10Joe) [10:00:56] <_joe_> jayme: you can't really do that [10:01:10] <_joe_> unless we want to extract the whole volumes stanza out [10:01:35] <_joe_> a variable you define has scope local to the current template [10:04:53] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Gilles) The new version of the WikimediaDebug extension is good to do, we just have to tag and push a new version which takes a few minutes. It's just T2... [10:05:57] hmm, true [10:07:38] and with the volume stanza out, we would not really win anything (apart from deployments.yaml being a bit more readable) [10:15:21] <_joe_> maybe it's worth it, dunno [11:35:25] <_joe_> jayme, akosiaris I'm thinking of moving create_new_service.sh to be a rake task [11:36:02] <_joe_> so that I can not only use a semi-decent programming language, but also add CI for scaffolding seamlessly [11:39:58] fine with me if we're okay with adding ruby/rake or docker :) as dependency to create a new service [11:44:26] <_joe_> yeah I think it's worth it [11:45:04] <_joe_> also it's time to split that huge rakefile [15:42:49] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10hashar) I can not tell why the homepage of doc.... [15:48:34] 10serviceops, 10Prod-Kubernetes: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10JMeybohm) p:05Triage→03High [15:56:40] <_joe_> jayme: you made some refactoring to rakefile that results in new charts created from scaffolding to not pass validation :P [15:57:20] _joe_: maybe I was right? :-) [15:58:27] I remeber there where a bunch of issues with scaffolding uncovered and I tried to include a test for that in the rake task [15:58:36] *in a rake task [15:58:49] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/638025 [16:00:26] <_joe_> jayme: heh I just re-wrote scaffolding as a ruby class just to do that correctly :) [16:01:24] yeah, cool. We came to the conclusion already that my approach was not good. Abandoning now [16:02:01] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (CI & Testing services): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#6819019, @hashar wrote: >... [16:23:33] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:24:29] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:26:13] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:27:40] jayme: is there a way we can have the hiera in just one common place? I'd assume that means we'd have to have the password in two places in the real private repo as well? [17:29:34] legoktm: I think there is not (but I'm no reference at all) and AFAIK we have that pattern all over the place in hiera [17:30:02] Okay, if that's already a pattern then... [17:30:04] <_joe_> uhm wait [17:30:10] <_joe_> what are you trying to do? [17:30:32] <_joe_> the hiera hierarchy has a "common" part that is searched for every server [17:30:48] <_joe_> but if you need to have the same user just for two roles, then yes, duplicate it [17:31:15] <_joe_> we determined that a small amount of duplication is better than the complexities that the further abstractions would creep into the puppet codebase [17:31:34] it's transporting a secret from the place where it is defined (docker_registry role) to the place where it is used (k8s workers) [17:42:26] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1295.eqiad.wmnet'] ` an... [17:51:21] jayme: should I give the k8s hiera key a different name then? [17:51:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:52:42] legoktm: I don't know if we have a convention for that. I personally like the fact that greping for the key would reveal both places [17:52:55] (I'm greping a lot in puppet repo :D) [17:52:59] Me too [17:55:19] we could name the second part the same [17:55:44] so profile::kubernetes::node::restricted_read_password and if you grep "restricted_read_password" it'll match [17:59:20] you can do hiera inside hiera [17:59:30] I know because I am currently replacing those with lookup too [17:59:43] so you can refer to one place from the other place [18:00:08] you would still have 2 keys but the actual value in a single place [18:00:10] https://gerrit.wikimedia.org/r/c/operations/puppet/+/663064/1/modules/profile/manifests/kubernetes/node.pp#22 is what we're discussing right now [18:00:30] how do you do that? [18:01:34] that sounds a bit evil :-) ... "profile::kubernetes::node::restricted_read_password" I do not like very much as it does not say that the password is for the docker registry [18:02:03] 10serviceops, 10Prod-Kubernetes: Check/Rebuild all dpcker-pkg build docker images running on kubernetes - https://phabricator.wikimedia.org/T274254 (10JMeybohm) [18:02:27] 10serviceops, 10Prod-Kubernetes: Check/Rebuild all docker-pkg build docker images running on kubernetes - https://phabricator.wikimedia.org/T274254 (10JMeybohm) p:05Triage→03High [18:02:47] docker_registry_restricted_read_password? :p [18:03:07] eheh, yeah [18:03:13] lookup('docker_registry_ha::restricted_read_password'), < this should be lookup('profile::docker_registry_ha::registry::restricted_read_password') ? [18:03:46] ideally only profiles have parameters and all lookups are profile::something [18:04:33] https://gerrit.wikimedia.org/r/c/operations/puppet/+/663064/1/modules/profile/manifests/kubernetes/node.pp the naming is a bit inconsistent [18:04:46] I'll put profile:: in front [18:05:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1371.eqiad.wmnet'] ` Of... [18:06:44] I'll need to go for today but will happily read backlog here tomorrow :-D [18:06:55] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1379.eqiad.wmnet'] ` Of... [18:07:00] o/ cya [18:13:26] this kind of thing is not really right: String $username = lookup('docker_registry_ha::username'), [18:13:49] because docker_registry_ha::username is not a profile and actually not even a class at all [18:14:17] the reason jenkins is stricter now is probably because John added it [18:14:23] to prevent those [18:15:37] mutante: ack, I think I fixed it if you could take a look (also added you as a reviewer on both changes) [18:22:24] legoktm: looked. for both changes I say +1 for the hiera/lookup part discussed above, just not speaking for the rest of it [18:22:32] but yea, that looks better now [18:22:33] thank you :) [18:22:51] * legoktm updates labs/private with new key names [18:24:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [18:24:22] I was about to say, dont forget the labs/private. I just did a compiler run on "*" again for something else and noticed there are as usual a handful of servers that always fail [18:24:36] because people added secrets but not fake secrets [18:25:12] would be nice though if there weren't any special ones [18:25:51] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [18:32:26] 18:30:02 | Splaying the start of the next reimage by 2 minutes [18:32:28] this is new :o [18:48:19] <_joe_> sorry I was afk, but for conventions and hiera key names and placement, please review https://wikitech.wikimedia.org/wiki/Puppet_coding#Organization [18:51:54] <_joe_> legoktm: read that page, it should clarify some of your doubts (and hopefully open up not too many more) [18:52:07] thanks [18:58:02] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:01:03] another special case. one appserver not coming back from reboot and DRAC says "Unable to perform requested operation" if you try to powercycle it. that's an unusual one [19:06:17] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:06:40] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1294.eqiad.wmnet'] ` an... [19:17:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [19:19:57] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:33:32] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [19:34:21] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) a:03Robert-RtC3V [19:36:44] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10RhinosF1) @Krinkle: Robert doesn't seem to have been active for a few years nor involved in this task. Did you mean to assign it to him? [19:37:53] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) a:05Robert-RtC3V→03None It wasn't me, it was . [19:38:52] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10RhinosF1) >>! In T245183#6819999, @Krinkle wrote: > It wasn't me, it was . Can we get some... [19:40:21] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Ladsgroup) I'm admin and I can't disable it.... [19:42:00] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10RhinosF1) >>! In T245183#6820029, @Ladsgroup wrote: > I'm admin and I can't disable it.... Phabricator is good at that. I guess it'll have to be... [19:45:53] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Dzahn) >>! In T245183#6820029, @Ladsgroup wrote: > I'm admin and I can't disable it.... You should ask the members of the "phabricator-admin" sh... [19:53:01] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1321.eqiad.wmnet', 'mw13... [20:26:56] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1370.eqiad.wmnet'] ` an... [20:27:46] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1378.eqiad.wmnet'] ` an... [20:36:06] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1293.eqiad.wmnet'] ` an... [20:45:04] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:46:04] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:52:51] 10serviceops, 10dev-images, 10docker-pkg, 10Release-Engineering-Team (Local Dev), 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10b... [21:00:37] 10serviceops, 10dev-images, 10docker-pkg, 10Release-Engineering-Team (Local Dev), 10User-brennen: docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001 - https://phabricator.wikimedia.org/T274306 (10b... [21:07:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [22:10:52] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1369.eqiad.wmnet'] ` an... [22:12:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1377.eqiad.wmnet'] ` an... [22:35:16] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1325.eqiad.wmnet', 'mw13... [23:29:06] 10serviceops, 10Performance-Team, 10Platform Engineering, 10Wikimedia-Rdbms, and 3 others: Determine and implement multi-dc strategy for ChronologyProtector - https://phabricator.wikimedia.org/T254634 (10aaron) The only thing that currently updates the replication positions on HTTP GET, that is not an easi...