[08:40:49] <dcaro>	 Morning!
[11:10:35] <arturo>	 Majavah: :-)
[11:10:42] <arturo>	 <3
[11:45:59] <dcaro>	 !log toolsbeta The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix
[11:46:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[12:43:14] <Vort>	 Hi. Do anyone knows why toolforge ssh access may be lagging now?
[12:44:12] <Vort>	 Even simple commands as "ls" may execute for 10-20 seconds
[12:46:40] <arturo>	 Vort: I think you are right. I don't know why
[12:47:21] <arturo>	 we have very high load avg
[12:47:39] <arturo>	 https://grafana-labs.wikimedia.org/d/7fTGpvsWz/toolforge-vm-details?orgId=1&var-VM=tools-sgebastion-07
[12:50:01] <Vort>	 also traffic is high
[12:50:30] <Vort>	 I mean eth0 RX/TX
[12:50:43] <arturo>	 there is apparently a big rsync going on
[12:52:03] <arturo>	 could it be tools.checkpersondata?
[12:52:38] <Vort>	 I can only say that it is not my bot :)
[12:53:22] <arturo>	 I just killed the rsync
[12:53:39] <arturo>	 !log tools.checkpersondata killed rsync proc that was consuming all network IO on the toolforge bastion
[12:53:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.checkpersondata/SAL
[12:54:13] <arturo>	 Vort: should be better now?
[12:54:27] <Vort>	 yes, thanks
[12:54:56] <proc>	 y'know, it's really discouraging to wake up every other morning to be pinged letting me know that i've been killed :(
[14:39:18] <arturo>	 proc: LOL
[15:15:59] <arturo>	 !log toolsbeta dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
[15:16:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:18:05] <arturo>	 !log toolsbeta dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
[15:18:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:21:10] <dcaro>	 !log toolsbeta creating new proxy instance toolsbeta-proxy-03
[15:21:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:42:06] <dcaro>	 !log toolsbeta re-creating the toolsbeta-proxy-03, used wrong image on the first try (T267140)
[15:42:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:42:09] <stashbot>	 T267140: [toolsbeta] Rebuild servers to learn how to take down the services without downtime - https://phabricator.wikimedia.org/T267140
[16:43:57] <wm-bot>	 !log tools.bridgebot <bd808> Restart to upgrade to matterbridge 1.19.0 release
[16:44:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[16:46:10] * bd808 would enjoy it if matterbrige started up a bit faster
[16:46:59] * bd808 waves to wm-bb
[16:48:46] <bd808>	 how about now wm-bb? Are you going to stay running?
[16:55:29] <bd808>	 how about now wm-bb? Is the third try going to work?
[16:55:53] <wm-bb>	 [telegram] <bd808> The bot is at least not crashing yet...
[16:57:23] <wm-bot>	 !log tools.bridgebot <bd808> Rolled back to 1.18.3 and filed T267239
[16:57:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL