[10:29:23] traffic switch looks quite smooth so far: https://grafana.wikimedia.org/d/000000066/resourceloader?orgId=1&from=1598819577873&to=1598947824621 [14:26:09] navtiming journal on webperf1001.eqiad: [14:26:15] > Sep 01 14:03:15 webperf1001 python[1357]: 2020-09-01 14:03:15,213 [INFO] (is_master:232) eqiad is not the master datacenter, disabling consumer [14:26:15] > Sep 01 14:03:15 webperf1001 python[1357]: 2020-09-01 14:03:15,214 [INFO] (run:755) No longer running in the master datacenter [14:26:15] > Sep 01 14:03:15 webperf1001 python[1357]: 2020-09-01 14:03:15,214 [INFO] (run:756) Stopping consuming [14:26:15] > Sep 01 14:03:26 webperf1001 python[1357]: 2020-09-01 14:03:26,226 [INFO] (run:714) Checking whether datacenter has been promoted [14:26:38] navtiming journal on webperf2001.codfw: [14:26:43] > Sep 01 14:03:12 webperf2001 python[1471]: 2020-09-01 14:03:12,314 [INFO] (run:714) Checking whether datacenter has been promoted [14:26:43] > Sep 01 14:03:12 webperf2001 python[1471]: 2020-09-01 14:03:12,316 [INFO] (is_master:228) codfw is the master datacenter, going live [14:26:43] > Sep 01 14:03:12 webperf2001 python[1471]: 2020-09-01 14:03:12,316 [INFO] (run:717) Starting Kafka connection to brokers [14:27:36] that's the first time our services are following the switchover and looks to be working fine. [14:28:06] Mostly thanks to what Ian implemented (overwrite and deterministic buffer, so that they don't need to be perfectly aligned) [14:30:20] neat [14:31:09] automatically based on polling the MW config for etcd, which is not perfect, but works for now (it's not related to MW in any way)