[06:28:36] hello - reposting in here as well -- just found out about this channel: [06:29:14] Hello - I am wondering if there are any tasks or projects that I could help out on -- My skills and background are in automation, microservices and cloud-native "transformation" (k8s/gitops/IaC) -- oh, and I am a golang/python coder [06:32:44] as well, I have a *lot* of baremetal experience, but more with hyperconverged type stacks (UCS, etc...) [06:33:09] hmu [07:07:05] epsas: sounds cool! thanks for offering. have you seen the bug tracker yet? https://phabricator.wikimedia.org/ [07:59:29] i have! there are quite a few repos -- I have also been going through the wikimedia dev pages [08:01:13] epsas: maybe search for tickets that mention k8s [08:03:10] +1 [08:04:01] ahh, lots of helm - unfortunate - lol. [08:11:50] epsas: fyi, there are 2 separate k8s cluster at WMF. one in cloud and one in production. [08:11:59] and they are probably quite different [08:16:48] yeah, I am reading kosiaris' presentation on bare-metal k8s at wikimedia now [08:17:58] I see references to the toolforge cluster and the PAWS cluster [08:18:07] and PAWS is not yet under puppet [08:20:24] !log tools.zppixbot-test updating everything and deploying stuff [08:20:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:22:06] !log tools.zppixbot-test tools.zppixbot-test@tools-sgebastion-07:~$ /data/project/zppixbot-test/zppixbottest/bin//pip3 freeze > /data/project/zppixbot-test/26maypip3.txt [08:22:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:30:55] !log tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ git pull && tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ /data/project/zppixbot-test/zppixbottest/bin//pip3 install -U -r /data/project/zppixbot-test/26maypip3.txt && restart pods [08:30:57] RhinosF1: Unknown project "tools.zppixbot-test@tools-sgebastion-07:~/.sopel$" [08:31:07] !log tools.zppixbot tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ git pull && tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ /data/project/zppixbot-test/zppixbottest/bin//pip3 install -U -r /data/project/zppixbot-test/26maypip3.txt && restart pods [08:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [08:31:19] !log tools.zppixbot-test tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ git pull && tools.zppixbot-test@tools-sgebastion-07:~/.sopel$ /data/project/zppixbot-test/zppixbottest/bin//pip3 install -U -r /data/project/zppixbot-test/26maypip3.txt && restart pods [08:31:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:34:05] !log tools.zppixbot-test Successfully installed PyYAML-5.3.1 SQLAlchemy-1.3.17 certifi-2020.4.5.1 dnspython-1.16.0 geoip2-3.0.0 jedi-0.17.0 maxminddb-1.5.4 multidict-4.7.6 parso-0.7.0 praw-7.0.0 prawcore-1.3.0 pytz-2020.1 six-1.15.0 sopel-7.0.4 update-checker-0.17 urllib3-1.25.9 wcwidth-0.1.9 websocket-client-0.57.0 yarl-1.4.2 [08:34:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:46:03] !log tools-zppixbot.test added rm sopel-default.pid to starter.sh as otherwise the bot gets very angry and refuses to start [08:46:04] RhinosF1: Unknown project "tools-zppixbot.test" [08:46:14] !log tools.zppixbot-test added rm sopel-default.pid to starter.sh as otherwise the bot gets very angry and refuses to start [08:46:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [08:46:20] * RhinosF1 apologies for spam [08:49:07] nevar complain about ircOps [08:49:30] morning epsas [08:49:56] epsas we are working on PAWS puppet now [08:50:08] arturo - coolness [08:50:22] I am reading the puppet guidelines now actually [09:25:38] !log toolsbeta `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again [09:25:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:30:45] !log toolsbeta set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying T246122 [09:30:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:30:48] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [09:56:02] !log toolsbeta installing kubectl/kubeadm 1.16.9 on k8s control nodes (T246122) [09:56:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:56:05] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [09:57:24] !log toolsbeta installing kubectl/kubeadm 1.16.9 on k8s worker nodes (T246122) [09:57:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:39:14] Majavah: is the landing page you use for your tools open source? [11:39:40] RhinosF1: do you mean the basic bootstrap page that I use everywhere [11:39:57] Majavah: yes [11:40:19] RhinosF1: https://github.com/wikimedia/labs-tools-github-pr-closer/blob/master/templates/index.html [11:40:27] credit appreciated but not required [11:41:01] Majavah: I’ll credit of course [11:41:19] thanks! what are you working on? :D [11:44:02] Majavah: https://tools.wmflabs.org/phabsearchemail [11:44:07] 503 isnt so pretty [11:44:27] 504 for me [11:47:21] also while I'm here, would an oauth admin mind taking a look and approving https://meta.wikimedia.org/w/index.php?title=Special:OAuthListConsumers/view/fd9195d17bb55765bd11e275a94cd5e9 please [11:47:46] Ah ye [11:47:54] It’s just a few crowns [11:48:49] RhinosF1: Toolforge is open source only :-) [11:53:01] arturo: ah cool, that’s good [11:53:17] * RhinosF1 only has the config private [11:53:50] * Majavah only has admin-like credentials to WM GitHub organization private [11:57:16] That sounds like a good idea [11:57:28] The config is a list of emails and the query to send to them [11:58:06] I mean my creds only have read-write access to PRs and can check if a file named ".gitreview" exists but still :D [11:58:23] :) [11:58:41] * RhinosF1 needs to deploy some changes to the tool and make it use the git version [11:59:05] Currently I just copy the code onto the tool [11:59:20] Because the git repo didn’t exist when I created the tool. [12:03:00] why does https://tools.wmflabs.org/admin/tool/phabsearchemail say unknown [12:29:23] Majavah: trying it now [12:30:59] RhinosF1: atleast https://phabsearchemail.toolforge.org/ gives me a "no webservice" error [12:31:20] Majavah: approved [12:31:31] tgr: thanks, appreciate it [12:31:43] AmandaNP: ^ [12:32:25] Majavah: the webservice won't start [12:38:35] why does tools.phabsearchemail@tools-sgebastion-07:~/phabsearchemail$ ln -s $HOME/phabsearchemail/public_html/ $HOME/public_html/ not work [12:39:10] what error do you get? [12:39:17] ln: target '/data/project/phabsearchemail/public_html/' is not a directory: No such file or directory [12:39:52] despite the fact I can see sftp://login.toolforge.org/mnt/nfs/labstore-secondary-tools-project/phabsearchemail/phabsearchemail/public_html in the files [12:46:35] Majavah: ^ [12:47:37] uhhh [12:48:54] what does `echo $HOME` return [12:49:19] Majavah: /data/project/phabsearchemail [12:50:02] can you try `ln -s /data/project/phabsearchemail/phabsearchemail/public_html /data/project/phabsearchemail/public_html` [12:50:26] Majavah: might have worked [12:50:36] !log tools.phabsearchemail ln -s /data/project/phabsearchemail/phabsearchemail/public_html /data/project/phabsearchemail/public_html [12:50:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [12:51:10] !log tools.phabsearchemail start webservice [12:51:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [12:51:25] it says "github-pr-closer" :D [12:51:47] Majavah: yeah, see where I got the code from :) [12:53:28] Majavah: how did you get {{ revision }} to woork [12:53:59] https://gerrit.wikimedia.org/r/#/c/labs/tools/phabsearchemail/+/598732/2/public_html/index.html should fix the github-pr-closer bit [12:54:05] RhinosF1: it's via flask https://github.com/wikimedia/labs-tools-github-pr-closer/blob/master/app.py#L22-L26 [12:55:27] Majavah: works [13:06:55] !log tools.phabsearchemail fixed reception's email and made cron use git repo [13:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [13:14:24] !log tools.phabsearchemail crons are now '0 14 * * 1 /usr/bin/jsub -N cron-29 -once -quiet python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/weekly.csv' && [13:14:24] '0 9 1 * * /usr/bin/jsub -N cron-30 -once -quiet python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/monthly.csv' [13:14:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [13:29:26] zhuyifei1999_: Finally, I was able to deploy a newer version -- for some reason I could make a production build on my laptop but not on toolforge. [13:29:36] Well, now it works! [13:29:43] ok [13:30:15] I just built it locally and pushed that to git repo and later cloned it. [13:30:29] You can now save timezones. [13:30:49] As you mentioned you have to regularly convert between two -- you can now save that. [13:41:13] acagastya: ok thanks [14:52:07] !log tooslbeta bump installed version of kubeadm and kubectl to 1.16.10 (T246122) [14:52:08] arturo: Unknown project "tooslbeta" [14:52:09] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [14:54:08] !log toolsbeta bump installed version of kubeadm and kubectl to 1.16.10 (T246122) [14:54:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:54:18] !log toolsbeta `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` (T246122) [14:54:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:02:04] !log toolsbeta first k8s upgrade failed for yet-to-be-known reasons (T246122) [15:02:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:02:08] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [15:27:57] !log paws updated profile::wmcs::kubeadm::kubernetes_version to 1.16.10 for cluster init T211096 [15:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:28:00] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [16:17:41] !log toolsbeta fix incorrect volume name in kubeadm-config T246122 [16:17:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:17:45] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [16:20:24] !log tools fix incorrect volume name in kubeadm-config configmap T246122 [16:20:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:29:00] !log paws bootstrapped the new control plane nodes T211096 [18:29:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [18:29:03] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [18:45:36] !log tools upgrading maintain-kubeusers to match what is in toolsbeta T246059 T211096 [18:45:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:45:41] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [18:45:41] T246059: Add admin account creation to maintain-kubeusers - https://phabricator.wikimedia.org/T246059 [22:03:53] !log paws created paws.admin group and kubernetes admin accounts on the new k8s cluster T211096 T246059 [22:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [22:03:57] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [22:03:57] T246059: Add admin account creation to maintain-kubeusers - https://phabricator.wikimedia.org/T246059 [22:04:35] !log paws created paws-focused PodSecurityPolicies and the prod namespace in the new cluster T211096 [22:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [22:05:51] !log paws temporarily deleted the deployment for maintain-kubeusers pending patch to fix context creation for new admin accounts T211096 T246059 [22:05:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [22:34:39] !log paws restored the deployment for maintain-kubeusers so anyone added to the paws.admin group will have admin on the cluster now that the bug is fixed T211096 T246059 [22:34:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [22:34:43] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [22:34:43] T246059: Add admin account creation to maintain-kubeusers - https://phabricator.wikimedia.org/T246059