[14:49:40] <wikibugs>	 07Zuul, 06collaboration-services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Increase memory for Zuul VMs - https://phabricator.wikimedia.org/T425679#11899128 (10Dzahn) status quo:   ` zuul1001     memory: default (8192) zuul1002     memory: default (4096) zuul1003     memory: def...
[15:53:02] <dduvall>	 bd808: currently looking into the zuul -> wmcs k8s issue, and i see that the current health status of the cluster is `UNKNOWN`. horizon errors when i try to get the cluster config as well. any thoughts?
[16:46:01] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Test new zuul test VMs - https://phabricator.wikimedia.org/T405120#11899699 (10dduvall) The primary error seems due to a bug in the upstream kubernetes driver. I've submitted a [[ https://review.opendev.org/c/...
[16:52:10] <bd808>	 dduvall: did you manage to get into the cluster? It needs to be rebuilt with the higher IOPS container settings, so I guess if it’s busted you could delete it all and start again. 
[16:53:10] <bd808>	 I’m out sick today so probably not a great thought partner 
[18:35:35] <dduvall>	 bd808: ah, feel better. i did manage to get into it after setting up `.env` and using `bin/kubeconfig.py`. nice setup!
[18:36:08] <dduvall>	 the issue isn't the currently provisioned role but a bug in upstream https://review.opendev.org/c/zuul/zuul/+/987725
[18:53:38] <wikibugs>	 07Zuul, 06collaboration-services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Increase memory for Zuul VMs - https://phabricator.wikimedia.org/T425679#11900245 (10Dzahn) It seems like all 3 types (in eqiad) are only using half of their RAM though within the last 2 days:  zuul1001 (...