[10:16:49] !log tools upgrading oldstable/main packages (clinic duty) [10:16:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:53:58] !log tools upgrade kubectl (clinic duty) [10:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:58:03] !log tools upgrade kubectl done (clinic duty) [10:58:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:04:17] !log tools upgrade oldstable/main packages (clinic duty) [11:04:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:08:14] !log tools upgrade stretch-wikimedia/component/php72 (minor upgrades) (clinic duty) [11:08:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:20:48] !log tools upgrading stretch-wikimedia/thirdparty/mono-project-stretch (clinic duty) [11:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:27:15] !log tools upgrading stretch-wikimedia/main (clinic duty) [11:27:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:35:13] !log tools uncordon tools-k8s-worker-71 and tools-k8s-worker-55, they weren't uncordoned yesterday for whatever reasons (T263284) [11:35:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:35:17] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [11:58:16] !log disabled all the repos blocking upgrades on tools-package-builder-02 (duplicated, other releases...) [11:58:16] dcaro: Unknown project "disabled" [11:58:22] !log tools disabled all the repos blocking upgrades on tools-package-builder-02 (duplicated, other releases...) [11:58:24] xd [11:58:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:01:24] !log tools upgrading stretch-backports/main, mainly libuv (clinic duty) [12:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:03:32] !log tools upgrading stable-updates/main, mainly cacertificates (clinic duty) [12:03:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:12:25] !log tools upgrading buster-wikimedia/main (clinic duty) [12:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:14:15] !log tools upgrading stable/main (clinic duty) [12:14:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:47:56] musikanimal: congrats on the xtools win in the Coolest Tool Award :) [17:48:14] hehe thank you!!! [18:04:32] musikanimal: I wanted to click "award token" but Phab is missing that button on member pages:) https://phabricator.wikimedia.org/project/members/803/ [18:05:47] :) [18:10:40] Hey, I might be missing something obvious but webservice documentation says it supports lighttpd for k8s backend but when I try it, it fails [18:10:42] https://www.irccloud.com/pastebin/TrDBfKmy/ [18:10:53] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#Overview [18:11:52] of course, I should just go with "php7.3" [18:14:31] Amir1: yes, php7.3 is a lighttpd backend for Kubernetes [18:14:40] I'll check on the docs :) [18:14:59] but it fails now [18:15:03] https://www.irccloud.com/pastebin/m2xlfnpK/ [18:15:13] for lack of kube config [18:15:29] even when I made an empty config file, it still failed with the second error [18:17:03] Amir1: is that a brand new tool? [18:17:10] yeah [18:17:54] I wonder if the credential provisioning is messed up following the Kubernetes version upgrade yesterday... let me take a lookl [18:24:57] Amir1: there is some bug with provisioning the kubernetes credentials for that tool. I'm investigating [18:25:14] Thanks [18:25:31] I'm on grid for now, will change to solar once it's fixed [18:29:33] !log tools certificatesigningrequest.certificates.k8s.io "tool-production-error-tasks-metrics" deleted to stop maintain-kubeusers issues [18:29:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:30:49] Amir1: ^ I think bstorm fixed it for you. We see "Provisioned creds for user production-error-tasks-metrics" in the logs now [18:31:20] oh thanks [23:31:01] !log toolsbeta increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts [23:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL