[03:36:13] Hi [03:37:18] I need help [03:54:53] :| [12:49:14] !log paws set hiera `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-17'` at project level (T268669) [12:49:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:49:17] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [12:49:47] !log paws disable puppet in all k8s nodes to prepare for the upgrade (T268669) [12:49:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:28:13] [telegram] Fountain tool (used in Wikimedia Asia Month to log new articles) broke: error code 500. [15:49:40] !log draining paws-k8s-control-1 for upgrade T268669 [15:49:40] bstorm: Unknown project "draining" [15:49:41] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [15:49:47] oops [15:49:56] !log paws draining paws-k8s-control-1 for upgrade T268669 [15:49:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:53:17] !log paws proceeding with upgrade to 1.17 on paws-k8s-control-1 T268669 [15:53:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:17:20] !log paws starting upgrade on paws-k8s-control-2 T268669 (first kubectl drain paws-k8s-control-2 --ignore-daemonsets) [16:17:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:17:24] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [16:31:44] !log paws upgrading pods on paws-k8s-control-3 T268669 [16:31:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:31:48] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [17:08:59] !log paws delete orphaned jupyter server pod `kubectl -n prod delete pod jupyter--45volutionoftheuniverse`. Respective server not running in jupyter admin UI. [17:09:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [17:14:05] !log paws updated the calico-kube-controllers deployment to use our internal registry to deal with docker-hub rate-limiting T268669 T269016 [17:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [17:14:09] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [17:14:10] T269016: Cloud: workaround new docker hub ratelimits - https://phabricator.wikimedia.org/T269016 [17:25:25] !log paws upgrading the worker nodes (this will likely kill services briefly when some pods are rescheduled) T268669 [17:25:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [17:25:29] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [18:12:18] !log admin removing all osds from cloudcephosd1015 in order to investigate T268746 [18:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:12:21] T268746: [ceph] cloudcephosd1004-1015 think that their hard drives are HDD when they are SSD - https://phabricator.wikimedia.org/T268746 [18:22:00] !log paws 1.17 upgrade for kubernetes complete T268669 [18:22:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [18:22:03] T268669: Upgrade PAWS k8s to 1.17 - https://phabricator.wikimedia.org/T268669 [20:49:05] legoktm: should I delete the 'butterfly' project? You and mz are the current admins. [20:50:42] andrewbogott: ummm let me text him, it's really his and I just help oit [20:50:45] Out* [20:50:53] ok. I also sent a nag email just now [21:19:57] andrewbogott: you can delete butterfly [21:20:32] legoktm: can you mark that on the purge page? I'm having a hard time keeping track of irc & email responses [21:20:38] ack [21:21:49] done [21:22:17] legoktm: can you mark Sccache one way or the other too please? [21:26:28] andrewbogott: did that one too :) [21:26:39] thanks [21:40:51] !log butterfly deleting project as per communication here with legoktm [21:40:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Butterfly/SAL [21:42:26] !log sccache deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge#DELETE_sccache [21:42:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Sccache/SAL [21:44:57] !log grantreview deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge#grantreview [21:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Grantreview/SAL [21:46:46] !log techblog deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge#techblog [21:46:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Techblog/SAL [21:55:52] !log orch deleting project; clearly unused [21:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Orch/SAL [21:57:47] !log wdqs-scaling deleting project; clearly unused [21:57:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wdqs-scaling/SAL