[00:00:52] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2242.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280000_dzahn_213079_mw22... [00:01:55] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2218.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280001_dzahn_213234_mw22... [00:02:00] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2218.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2218.codfw.wmnet'] ` [00:14:35] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2244.codfw.wmnet'] ` and were **ALL** successful. [00:57:55] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2236.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280057_dzahn_224625_mw22... [01:05:56] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2227.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280105_dzahn_228153_mw22... [01:07:30] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2242.codfw.wmnet'] ` and were **ALL** successful. [01:12:40] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2218.codfw.wmnet'] ` and were **ALL** successful. [01:15:32] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2219.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280115_dzahn_230669_mw22... [01:16:04] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw2241.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201911280115_dzahn_230758_mw22... [01:18:24] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10Dzahn) [01:55:49] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Deploy version 1.10.0 of wikidiff2 to production - https://phabricator.wikimedia.org/T236963 (10tstarling) >>! In T236963#5696069, @Legoktm wrote: > @tstarling is the gpg key that you used to sign that release available anywhere?... [02:05:59] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Deploy version 1.10.0 of wikidiff2 to production - https://phabricator.wikimedia.org/T236963 (10tstarling) keys.txt only has my 2008 and 2009 keys, since that's when I was doing MediaWiki releases. [02:09:54] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2236.codfw.wmnet'] ` and were **ALL** successful. [02:17:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2227.codfw.wmnet'] ` and were **ALL** successful. [02:23:57] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2219.codfw.wmnet'] ` and were **ALL** successful. [02:27:28] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2241.codfw.wmnet'] ` and were **ALL** successful. [02:33:10] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10Dzahn) [04:17:38] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Krinkle) Ye... [07:12:14] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Pybal, 10Traffic: Proposal: simplify set up of a new load-balanced service on kubernetes - https://phabricator.wikimedia.org/T238909 (10Joe) Also to clarify further: Pybal does **none** of the meaningful load-balancing. Load-balancing between pods is done... [07:35:48] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Pybal, 10Traffic: Proposal: simplify set up of a new load-balanced service on kubernetes - https://phabricator.wikimedia.org/T238909 (10Joe) [08:01:00] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1345.eqiad.wmnet', 'mw1335.eqiad.wmnet', 'mw1325.eqiad.wmnet'] ` The log can be found in `/var/log/... [08:23:07] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw2280.codfw.wmnet', 'mw2281.codfw.wmnet', 'mw2282.codfw.wmnet'] ` The log can be found in `/var/log/... [09:18:15] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1266.eqiad.wmnet', 'mw1276.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [09:56:36] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10Joe) [09:56:50] <_joe_> addshore: just slightly late https://phabricator.wikimedia.org/T239392 [09:59:21] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10Joe) [10:06:49] thanks _joe_ [10:13:55] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2280.codfw.wmnet', 'mw2281.codfw.wmnet', 'mw2282.codfw.wmnet'] ` and were **ALL** successful. [10:29:29] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10ema) p:05Triage→03Normal [10:36:05] <_joe_> akosiaris: bikeshedding opportunity [10:36:34] <_joe_> right now, if I want to add something coming from secret() in puppet to my helm deployment values [10:36:46] <_joe_> I can't, I have to copy/pasta values in a yaml [10:36:51] <_joe_> which is available via hiera [10:37:08] <_joe_> now this is both horrible and less secure than using secret() [10:37:18] <_joe_> my idea is to allow people to put in hiera [10:37:36] <_joe_> mykey: secret::path_to_secret [10:37:54] <_joe_> and to fetch the secret ourselves in those cases and write them to the final yaml [10:38:06] <_joe_> now bikeshedding [10:38:15] <_joe_> mykey: secret::path_to_secret seems ok to you? [10:38:24] <_joe_> also effie apergos if you're interested :P [10:40:29] _joe_: can you give an example? [10:40:39] <_joe_> ok [10:41:08] <_joe_> tls: certs: cert: secret::secret/certificates/blubberoid.pem [10:42:11] ok just reached a point where i can stop for a bit, reading the backscroll [10:43:11] no opinion, please continue your bikeshedding uninterrupted by me :-) [10:47:23] just to hit the discussion, you could have: [10:47:32] mykey: secret(path_to_secret) :-P [10:47:39] *heat [10:48:09] <_joe_> meh [10:48:32] <_joe_> parens are not special in yaml, right? [10:53:19] those will be only for helm deployment values ? [10:55:15] <_joe_> yes [10:55:47] <_joe_> it will mean we can move most of the hiera out of the private repo too, which is good [10:56:13] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw2279.codfw.wmnet', 'mw2278.codfw.wmnet', 'mw2277.codfw.wmnet', 'mw2276.codfw.wmnet', 'mw2275.codfw.... [11:00:26] my only thinking is [11:00:36] if it should have a prefix like helm: [11:02:00] helm: certs: secret::secret/certificates/blubberoid.pe [11:03:08] <_joe_> I was describing the structure of the yaml [11:08:46] sigh [11:09:18] I am missing context, I give up :p [11:09:25] I go back to my reimaging [11:33:14] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1345.eqiad.wmnet', 'mw1335.eqiad.wmnet', 'mw1325.eqiad.wmnet'] ` and were **ALL** successful. [11:37:16] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1344.eqiad.wmnet', 'mw1334.eqiad.wmnet', 'mw1324.eqiad.wmnet'] ` The log can be found in `/var/log/... [11:39:34] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1266.eqiad.wmnet', 'mw1276.eqiad.wmnet'] ` and were **ALL** successful. [11:48:27] _joe_: just saw this, lemme find an example in /srv/private to figure out what you refer to [11:52:22] <_joe_> akosiaris: profile::kubernetes::deployment_server_secrets::services [11:53:10] <_joe_> wherever you see a cert, we could have instead a pointer to a secrets file [11:54:46] <_joe_> reducing the steps I need to do for adding tls to a service on k8s [11:55:31] <_joe_> I can just run cergen and refer to secret:: in the hiera file, instead of having to paste the cert in hiera directly [11:55:44] <_joe_> which is btw an antipattern we should avoid going forward [11:59:43] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1267.eqiad.wmnet', 'mw1277.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [12:30:29] so americans are out today [12:30:34] do we want to have the regular meeting today? [12:30:48] <_joe_> we do have stuff to discuss [12:31:07] <_joe_> but I think not having 2 people would make discussing things suboptimal [12:31:26] <_joe_> I'm ok with not having the meeting today [12:32:08] thu next week then? [12:32:16] fine by me too [12:34:33] works for me [12:38:22] _joe_: wouldn't %{secret("blabhblah")} just do it? hiera 3.x allows interpolations using functions IIRC [12:38:38] works for me as well [12:39:41] ah no, it's not arbitrary functions. It's just alias(), literal(), lookup() and hiera() that are the supported functions [12:39:54] that being said, maybe we could add secret() ? [12:40:08] meh.. messing with the parser... [12:42:31] <_joe_> akosiaris: I'm basically doing that in post-production with a wrapper function [12:42:33] <_joe_> :P [12:43:25] <_joe_> so, what syntax would make you happier? [12:43:50] perl sigils! [12:44:00] put as many of @#^$#@^&!%$& as you can in there [12:44:04] :P [12:48:22] <_joe_> akosiaris: so something like <|%{@secret(myunquotedsecret)%}|> ? [12:48:27] <_joe_> I can make this work [12:57:55] * effie read perl girls [12:58:17] and everything stopped making sense [14:12:10] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2279.codfw.wmnet', 'mw2278.codfw.wmnet', 'mw2277.codfw.wmnet', 'mw2276.codfw.wmnet', 'mw2275.codfw.wmnet'] ` and were **ALL** successful. [14:23:52] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1267.eqiad.wmnet', 'mw1277.eqiad.wmnet'] ` and were **ALL** successful. [14:29:37] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1343.eqiad.wmnet', 'mw1342.eqiad.wmnet', 'mw1341.eqiad.wmnet'] ` The log can be found in `/var/log/... [15:05:33] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1333.eqiad.wmnet', 'mw1332.eqiad.wmnet', 'mw1331.eqiad.wmnet'] ` The log can be found in `/var/log/... [15:09:36] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1344.eqiad.wmnet', 'mw1334.eqiad.wmnet', 'mw1324.eqiad.wmnet'] ` and were **ALL** successful. [15:55:03] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10ema) We could also think of writing a sort of HTTP router that returns a list of PyBal AP... [15:58:58] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1311.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911281558_jiji_182152.log`. [16:01:16] 10serviceops, 10Operations, 10Pybal, 10SRE-tools, 10Traffic: Applications and scripts need to be able to understand the pooled status of servers in our load balancers. - https://phabricator.wikimedia.org/T239392 (10Joe) >>! In T239392#5700283, @ema wrote: > We could also think of writing a sort of HTTP r... [16:10:06] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1333.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911281608_jiji_185447.log`. [17:04:54] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1311.eqiad.wmnet'] ` and were **ALL** successful. [17:20:48] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1340.eqiad.wmnet', 'mw1339.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [17:22:37] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1333.eqiad.wmnet'] ` and were **ALL** successful. [17:27:28] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [17:28:23] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [17:30:01] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1332.eqiad.wmnet', 'mw1331.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [18:03:11] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1343.eqiad.wmnet', 'mw1342.eqiad.wmnet', 'mw1341.eqiad.wmnet'] ` and were **ALL** successful. [18:26:40] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1332.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911281825_jiji_214589.log`. [18:30:03] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1319.eqiad.wmnet', 'mw1318.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [19:38:00] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1332.eqiad.wmnet'] ` and were **ALL** successful. [19:45:15] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1340.eqiad.wmnet', 'mw1339.eqiad.wmnet'] ` and were **ALL** successful. [19:48:56] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1331.eqiad.wmnet', 'mw1330.eqiad.wmnet', 'mw1310.eqiad.wmnet'] ` The log can be found in `/var/log/... [20:04:14] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1313.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911282004_jiji_236588.log`. [20:42:41] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1319.eqiad.wmnet', 'mw1318.eqiad.wmnet'] ` and were **ALL** successful. [21:11:40] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1316.eqiad.wmnet', 'mw1315.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [21:16:54] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1313.eqiad.wmnet'] ` and were **ALL** successful. [21:20:18] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1313.eqiad.wmnet', 'mw1309.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20191... [23:14:58] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1331.eqiad.wmnet', 'mw1330.eqiad.wmnet', 'mw1310.eqiad.wmnet'] ` and were **ALL** successful. [23:21:51] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mw1329.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201911282321_jiji_18834.log`. [23:38:47] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1313.eqiad.wmnet', 'mw1309.eqiad.wmnet'] ` and were **ALL** successful. [23:42:32] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1316.eqiad.wmnet', 'mw1315.eqiad.wmnet'] ` and were **ALL** successful.