[00:00:18] 10serviceops, 10Continuous-Integration-Infrastructure, 10Developer-Wishlist (2017), 10Patch-For-Review, and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Jdforrester-WMF) [01:56:18] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2350.codfw.wmnet ` The log can be fou... [02:05:52] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2351.codfw.wmnet ` The log can be fou... [02:18:13] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2350.codfw.wmnet'] ` and were **ALL** successful. [02:20:19] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2352.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [02:28:04] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2351.codfw.wmnet'] ` and were **ALL** successful. [02:28:32] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2353.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [02:42:21] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2352.codfw.wmnet'] ` and were **ALL** successful. [02:42:52] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2354.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [02:51:49] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2353.codfw.wmnet'] ` and were **ALL** successful. [02:53:14] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2355.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [03:07:13] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2354.codfw.wmnet'] ` and were **ALL** successful. [03:14:44] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) |Servers Rack D3| Ready for service| |mw2350|Yes| |mw2351|Yes| |mw2352|Yes| |mw2353|Yes| |mw2354|Yes| |mw2355|Yes| |mw2356|| |mw2357|| |mw2358| |mw2359|| |mw2... [03:15:29] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2355.codfw.wmnet'] ` and were **ALL** successful. [09:04:28] <_joe_> mutante: found the bug https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/577186 [11:06:45] 10serviceops, 10Release-Engineering-Team, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Enable phpdbg on mwdebug* servers - https://phabricator.wikimedia.org/T244549 (10hnowlan) 05Open→03Resolved [13:09:24] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2356.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [13:09:48] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2357.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [13:32:41] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2356.codfw.wmnet'] ` and were **ALL** successful. [13:34:26] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2358.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [13:55:10] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) a:03Papaul [13:56:34] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2358.codfw.wmnet'] ` and were **ALL** successful. [13:59:07] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2357.codfw.wmnet'] ` and were **ALL** successful. [14:06:09] 10serviceops, 10Operations, 10observability, 10vm-requests: Provision grafana VM in codfw - https://phabricator.wikimedia.org/T244357 (10akosiaris) Is there anything left to do in this task? [14:10:16] hi akosiaris ! Thanks for merging the intake-analytics stuff [14:10:40] am I missing a piece? it isn't working yet [14:10:43] curl https://intake-analytics.wikimedia.org/_info [14:16:38] ottomata: hmm interesting. I think I forgot to test this [14:16:53] interestingly the discovery curl works from cp hosts [14:20:13] 20200305.14h19m10s CONNECT: could not connect to 10.2.2.52 for 'https://eventgate-analytics-external.discovery.wmnet:4692/_info' (setting last failure time) connect_result=5 [14:21:01] eh? [14:21:13] my reaction exactly [14:21:45] from the exact same host I can do a curl just fine [14:22:45] ema: your awesome ATS debugging skills are required :-) [14:32:56] akosiaris: ehlos! [14:33:48] ema: hey. Context is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/573369/ [14:33:54] merged by me, forgot to test it [14:33:59] but it doesn't work? [14:34:04] cp3058 logs [14:34:26] 20200305.14h32m33s CONNECT:[0] could not connect [CONNECTION_ERROR] to 10.2.2.52 for 'https://eventgate-analytics-external.discovery.wmnet:4692/_info' [14:34:38] but a curl from that same host for that exact url works fine [14:34:48] is intake-analytics.wikimedia.org in the certificate SAN? [14:35:33] should be let me chheck [14:35:58] ema: isn't it the wildcard one? [14:36:06] its not. [14:36:09] *.wikimedia.org, *.wikipedia.org etc [14:36:15] whoopsie [14:36:18] ok fixing that [14:36:23] ottomata: ? [14:36:32] for logging external [14:36:33] alt_names: ['intake-logging.wikimedia.org', 'eventgate-logging-external.discovery.wmnet', 'eventgate-logging-external.svc.codfw.wmnet', 'eventgate-logging-external.svc.eqiad.wmnet'] [14:36:36] for analyticsi extenral [14:36:40] alt_names: ['eventgate-analytics-external.discovery.wmnet', 'eventgate-analytics-external.svc.codfw.wmnet', 'eventgate-analytics-external.svc.eqiad.wmnet'] [14:36:45] aaah the discovery one ? [14:36:57] no that si thte cn [14:37:07] I see this in SAN: DNS:eventgate-analytics-external.discovery.wmnet, DNS:eventgate-analytics-external.svc.codfw.wmnet, DNS:eventgate-analytics-external.svc.eqiad.wmnet [14:37:13] sorry, too many TLSes [14:37:20] ema: good catch [14:37:22] gotta add the public name [14:37:24] thanks [14:37:53] np! [15:03:44] <_joe_> rlazarus: so turns out envoy --mode validate opens the access log files configured [15:04:51] hmmm that sounds vaguely familiar [15:06:44] is it causing problems? I'm still reading back [15:07:07] (I heard there was something with envoy startup on new machines but I haven't seen details yet) [15:07:13] rlazarus: I assume this is re: https://gerrit.wikimedia.org/r/577186 [15:07:41] aha thanks [15:14:30] it's been a little while so my memory's hazy, but I think that was intentional -- the idea was that you should run --mode validate and --mode serve as the same user, so that the validation would *catch* that kind of permission error, rather than causing it :) [15:14:37] sorry for the trouble though [15:29:25] fixed! curl https://intake-analytics.wikimedia.org/_info [15:32:29] _joe_: thanks! re: bug fix. i'll continue with getting stuff pooled today [15:50:14] akosiaris: how's this sound? [15:50:15] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/577264 [15:50:22] for eventstreams limits + replicas [15:56:32] ottomata: memory+cpu sound fine, replicas == 8 for the staging cluster? not so much [15:56:39] but 8 sounds great for production clusters [16:20:03] oops you right [16:20:11] forgot i had to make those other helmfiles :p [16:20:15] but yes, for prod, not staging [16:37:49] akosiaris: patched https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/577264, ok if I deploy those? [16:40:24] ottomata: +1ed, go for it [16:40:53] ty! [16:54:28] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2359.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [17:00:06] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2360.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [17:18:39] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2359.codfw.wmnet'] ` and were **ALL** successful. [17:23:00] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2360.codfw.wmnet'] ` and were **ALL** successful. [17:23:32] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2361.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [17:26:38] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2362.codfw.wmnet ` The log can be found in `/var/log/wmf-au... [17:27:11] we have agreed with dcops now on a slight change in the workflow and where we handover new servers [17:28:14] dcops will add servers to site.pp now with spare::role. and thats the demarcation point where the dcops ticket can be closed and we take it (on a new ticket probably) as service owners [17:28:49] that way we don't have servers with OS that are not in puppet but also dcops don't apply any roles except the spare one [17:29:10] a side-effect is we can downtime hosts before applying the actual role and avoid icinga spam [17:29:29] and they have the full firewall while waiting to be taken into prod [17:30:21] and we don't necessarily have the racking tickets open for so long just because the last checkbox "service implementation" has not been done yet, which is good for the dcops workboard [17:46:50] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2361.codfw.wmnet'] ` and were **ALL** successful. [17:46:53] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [17:47:31] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2363.codfw.wmnet ` The log can be fou... [17:48:59] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2362.codfw.wmnet'] ` and were **ALL** successful. [17:54:28] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2364.codfw.wmnet ` The log can be fou... [18:11:37] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2363.codfw.wmnet'] ` and were **ALL** successful. [18:11:41] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2365.codfw.wmnet ` The log can be fou... [18:12:32] we need to to decom of old appservers in c3 [18:12:50] c3 specifically first because that blocks dcops from racking remaining new ones in codfw [18:13:05] no decom ticket yet, i'll make one [18:18:19] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2364.codfw.wmnet'] ` and were **ALL** successful. [18:32:36] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2365.codfw.wmnet'] ` and were **ALL** successful. [18:50:43] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) For a Total of 86 mw servers , 71 are done and 15 left. The 15 left are waiting for space in row C in rack C3 [18:55:59] 10serviceops, 10Release-Engineering-Team, 10Core Platform Team Workboards (Clinic Duty Team): Enable phpdbg on mwdebug* servers - https://phabricator.wikimedia.org/T244549 (10EBernhardson) I'm sure this could be further improved, but in general it works! ` ebernhardson@mwdebug1002:~$ cat test.php 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [19:20:04] 10serviceops: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Dzahn) [19:23:42] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) 05Open→03Stalled thanks Papaul for all the new servers. We'll continue with racking right after eqiad is done. We can do that on... [19:24:03] 10serviceops, 10Operations, 10ops-codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Papaul) [19:26:22] we need to move at least 15 old servers out of C3 in codfw before we can rack the last 15 new ones there [19:29:31] 10serviceops, 10Operations: move all 86 new codfw appservers into production - https://phabricator.wikimedia.org/T247021 (10Dzahn) [19:30:56] 10serviceops, 10Operations: move all 86 new codfw appservers into production - https://phabricator.wikimedia.org/T247021 (10Dzahn) Only 71 of 86 servers can be done until T247018 is resolved first because there is not enough rack space for them. [19:40:09] 10serviceops, 10Operations, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 8 host(s) and their services with reason: new_install ` mw[1385-1392].eqi... [20:19:02] 10serviceops, 10Operations, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 3:00:00 set by dzahn@cumin1001 on 8 host(s) and their services with reason: new_install ` mw[1385-1392].eqi... [20:21:40] 10serviceops, 10Operations, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 8 host(s) and their services with reason: new_install ` mw[1385-1392].eqi... [21:14:16] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Nuria) [21:14:21] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Nuria) 05Open→03Resolved [21:41:05] 10serviceops, 10Operations, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by rzl@cumin1001 on 9 host(s) and their services with reason: new install ` mw[1405-1413].eqiad... [22:10:48] 10serviceops, 10Operations: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by rzl@cumin1001 on 9 host(s) and their services with reason: new install ` mw[1405-1413].eqiad.wmnet ` [22:40:09] 10serviceops, 10Operations: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 8 host(s) and their services with reason: new_install ` mw[1385-1392].eqiad.wmnet ` [22:55:42] 10serviceops, 10Operations: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Dzahn) ` {"mw1385.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"} {"mw1385.eqiad.wmnet": {"weight": 30, "pooled": "yes... [22:56:34] the second batch of new eqiad appservers is also pooled now (except one for testing purposes) [22:56:59] we could probably see another improvement in latency [22:58:15] 10serviceops, 10Operations: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Dzahn) [23:00:20] 10serviceops, 10Operations: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Dzahn) 05Open→03Resolved The one exception (1413 at the bottom) is currently being used for a test. Everything else is pooled with weight 30 and alternating between... [23:12:33] shaved off 10ms? https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-6h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=appserver&var-method=GET [23:36:29] mutante: Hmm, yeah, wmf.22 seems faster than wmf.21. Not sure I was expecting that. [23:38:56] James_F: oh, heh, i did not meen MW version, i meant because we added more appservers [23:39:20] https://phabricator.wikimedia.org/T241849#5946671 [23:39:43] was just talking to Krinkle about that too [23:41:16] Ah, that seems a more reasonable cause. :-) [23:43:58] added the timestamps in UTC to that last ticket comment [23:44:59] https://grafana.wikimedia.org/d/000000066/resourceloader?orgId=1&fullscreen&panelId=45 [23:57:05] 10serviceops, 10Operations, 10Performance-Team, 10Performance-Team-publish: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Krinkle) I've added a Grafana annotation for this event, and looks like we've got some nice perf wins here across the...