[00:25:52] !log tools `renice -n 15 -p 28865` (`tar cvzf` of `tools.giftbot`) on tools-bastion-02, been hogging the NFS IO for a few hours [00:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:41:41] !log toolsbeta rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd [16:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:13:16] zhuyifei1999_: Hi! I wanted to know how can I use toolsbeta to change the codebase of webservice. [17:14:00] hi, could you wait like 30 mins? I’m having lunch right now [17:14:18] zhuyifei1999_: Sure :) [17:23:59] vim ~/.pwds.gpg [17:24:05] nope! [17:24:58] for the curious: I'm using gnupg.vim from vim-scripts [17:30:14] Neha16: do you use putty or openssh? [17:32:32] zhuyifei1999_: I use openssh. [17:33:27] ok, can you follow https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_(recommended) to ssh into toolsbeta-bastion-01.toolsbeta.eqiad.wmflabs? [17:35:26] I have done that part and sshed into toolsbeta-bastion-01 [17:37:20] ok, so there are a few 'tools' available, like 'admin', 'test', 'test2' [17:38:23] unfortunately we don't have a mechanism to add people to the groups now, but since you have sudo you can `$ sudo become test` etc. into the tools [17:40:24] webservice command (in particular, the k8s part) is still half-broken, and I'm working on the setup [17:41:24] It says that I am not allowed to run sudo on toolsbeta-bastion-01. [17:41:57] hmm [17:42:07] * zhuyifei1999_ checks [17:43:30] Neha16: can you re-login and try again? [17:43:49] zhuyifei1999_: Sure. Give me a minute [17:43:50] I added you to 'Sudoer Policy' on horizon [17:45:11] zhuyifei1999_: Yes. Now, I am able to log in. [17:45:25] ok [17:49:57] zhuyifei1999_: So, after logging in I can modify the source code of this tool? [17:50:06] yeah [17:50:53] zhuyifei1999_: I suppose I cannot log in to webservice right now? [17:51:01] you can try [17:51:12] it might work, or it might not [17:51:30] * zhuyifei1999_ is continuing to fix everything... [17:51:54] the webservice code are in /usr/lib/python2.7/dist-packages/toollabs/ [17:52:01] you can edit that with sudo [17:52:15] but I'll install a new version within a few hours [17:52:48] zhuyifei1999_: Which tool should I log into? [17:53:28] I think the 'admin' is most ready [17:53:39] idk about the other tools [17:54:01] I'm experimenting with the admin tool [17:55:49] zhuyifei1999_: I have logged into admin but I cannot find the source code of webservice here [17:56:02] I think it's public_html [17:56:09] it's a php + lighttpd tool [17:56:27] it serves tools-beta.wmflabs.org [17:56:43] ^ was the old design [17:58:12] zhuyifei1999_: I can't find it in public_html :( [17:58:27] !log toolsbeta cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 | (while read dir; do id toolsbeta.`basename $dir` 2> /dev/null || sudo rm -rfv $dir; done)` [17:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:58:57] * zhuyifei1999_ looks [17:59:27] Neha16: toolsbeta.admin@toolsbeta-bastion-01:~$ ls public_html [17:59:27] apple-touch-icon.png content favicon.ico htmlpurifier img index.php libs robots.txt style.css Tool_Labs_logo_thumb.png [17:59:43] what do you mean by you can't find it? [18:00:29] argh did I kill paws again... [18:00:55] zhuyifei1999_: Actually, I meant this source code https://phabricator.wikimedia.org/diffusion/OSTW/. [18:01:04] PAWS production? [18:01:12] paws beta [18:01:25] the /data/project/paws directory [18:01:45] Neha16: /usr/lib/python2.7/dist-packages/toollabs/ [18:02:03] As long as the directory exists paws-beta os fine [18:02:20] s/os/is [18:02:37] chicocvenancio: I wonder if someone can create the toolsbeta.paws service group so /data/project/paws is legit [18:02:53] You might have nuked userhomes, but that is not a problem [18:03:48] zhuyifei1999_: Ah! Found it. Thanks a lot :) [18:03:58] chicocvenancio: k [18:04:09] Neha16: np [18:04:26] The db is now living in toolsdb [18:05:00] Should be possible to create the service group manually, but I'm not sure [18:07:28] !log admin stopped the toolhistory job because it is totally broken and fills /tmp. [18:07:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:41:21] !log toolsbeta rebuilding toolsbeta-k8s-etcd-01 [18:41:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [19:01:04] !log toolsbeta install kubernetes-client on toolsbeta-worker-1001 to debug stuffs [19:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [20:48:50] !log tools building, signing, and publishing toollabs-webservice 0.39 T190893 [20:49:03] stashbot: hello? [20:49:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:49:34] T190893: Setup the webservice-related instances in toolsbeta - https://phabricator.wikimedia.org/T190893 [20:49:34] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [21:02:04] !log tools re-building all docker images T190893 [21:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:02:07] T190893: Setup the webservice-related instances in toolsbeta - https://phabricator.wikimedia.org/T190893 [21:28:48] Is there major replication lag on the replica database? [21:36:04] ragesoss: looks like yes :( https://tools.wmflabs.org/replag/ [21:37:00] framawiki: thanks! [21:37:08] * ragesoss notes url for future reference. [21:42:23] something going on? [22:40:11] !log quarry deployed 24038e3 to quarry-main-01 [22:40:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL