[00:01:59] !log tools.zppixbot-test restarting for config change [00:02:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [00:20:05] !log tools.zppixbot-test restarting for config change [00:20:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [01:18:47] !log tools.zppixbot-test restarting for config change [01:18:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [02:41:45] !log library-upgrader deleted upgrader-05 instace, which has been shutoff for a while now [02:41:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [02:52:24] !log library-upgrader created libup-diff01 instance [02:52:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [09:38:08] !log toolsbeta live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251 [09:38:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:43:15] !log toolsbeta restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert [09:43:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:52:13] !log toolsbeta created MX record pointing to mail.toolsbeta.wmflabs.org [10:52:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:55:42] !log toolsbeta allow ingress smtp/smtps traffic in the MTA security group [10:55:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:02:14] !log toolsbeta setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix [11:02:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:36:06] !log toolsbeta removing `benapetr` and adding myself to the test tool [11:36:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:10:19] !log toolsbeta added herron to the test tool for email testing [13:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:24:57] !log tools.zppixbot-test rm -rf credentials.txt (superseded by default.cfg) [14:24:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [14:37:34] !log tools-zppixbot restarting for config/code changes [14:37:37] MacFan4000: Unknown project "tools-zppixbot" [14:37:56] !log tools.zppixbot restarting for config/code changes [14:37:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [14:39:32] !log tools.zppixbot restarting for config change [14:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [14:48:59] !log tools.zppixbot rm -rf credentials.txt (superseded by default.cfg) [14:49:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [16:08:40] !log tools created acme-chief cert `tools_mail` in the prefix hiera [16:08:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:47:16] !log tools.zppixbot restarting for config change [16:47:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [16:55:15] !log tools.zppixbot-test restarting for config change [16:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [17:05:54] !log deployment-prep restarted puppetdb.service on deployment-puppetdb03 [17:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:09:06] !log deployment-prep restarted postgresql on deployment-puppetdb03 [17:09:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:12:57] !log tools.zppixbot-test restarting for config change [17:12:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [17:15:20] !log deployment-prep to fix puppet on several hosts, setting profile::java::hardened_tls: false in project puppet on horizon [17:15:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:35:06] Hi. I can't connect to toolforge - it took forever to ssh and its now stuck on `Last login: ...` [17:35:45] 1m load average on 07 is at 50 [17:35:57] i got in by ssh-ing directly to instance-tools-sgebastion-08.tools.wmflabs.org [17:36:05] -07 is the host that's slow [17:36:25] yeah, looks like someone is running a bot or other NFS-intensive task on the bastion [17:36:40] so `ssh dannys712@instance-tools...`? [17:36:42] nfs latency is into the minute range [17:36:47] * Reedy tries to login [17:37:01] 3-5 minute range, that is [17:37:02] i would do ssh macfan4000@nstance-tools-sgebastion-08.tools.wmflabs.org [17:37:28] * Reedy hands MacFan4000 an i [17:37:51] aka dev.toolforge.org [17:38:06] so much faster [17:39:17] if there aren't alerts on tools-sgebastion-07 load average and NFS latency, there should be [17:39:36] * arturo looking [17:39:44] or better yet, a better resource limiter/killer [17:40:30] we have lots of resource limits, people just still find ways to exhaust the underlying system [17:40:50] load avg 33. let's see what the hog is [17:41:20] https://www.irccloud.com/pastebin/QBmVWBin/ [17:41:25] a pwb script of course [17:42:41] * Reedy is still waiting for a prompt [17:43:33] mmm [17:43:34] Reedy: we need to get your root key in. That's really the only way to connect when things are laggedout [17:43:40] also I see many OOM messages [17:43:44] aye [17:43:47] * bd808 will try to do that today [17:43:50] <3 [17:47:17] I killed those procs, not sure how to log it [17:47:21] !log tools.zppixbot-test restarting or config change [17:47:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [17:47:30] was just about to ask about toolforge bastion slowness, looks like it's already under control [17:48:02] users really really really need to stop running their pywikibot scripts there :/ [17:50:06] naming and shaming on the cloud list would be my suggestion, but that probably wouldn't help [17:50:58] AntiComposite: do you really think that they read that list? :P [17:51:43] one of the top hits now is someone deleting a bunch of files :) [17:55:31] !log tools killed procs for users `hamishz` and `msyn` which apparently were tools that should be running in the grid / kubernetes instead [17:55:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:01:22] Majavah and AntiComposite: apparently that was WMCS doing a thing. It's done now. Sorry [18:30:56] rsync, rm -r, sftp, find, tar, sha.*sum, basically all those IO-bound commands can hammer NFS without realizing :/ [19:11:10] how do I add someone as a maintainer to my tool? their username isn't showing up for some reason [19:11:15] under "manage maintainers" [19:16:27] ningu: you should be able to do it from https://toolsadmin.wikimedia.org/tools/id/,your tool name here>/maintainers/ [19:16:48] bd808: that's where I am [19:16:55] but the person I want to add's username isn't being found [19:17:02] they recently created their wikitech account, not sure if that's relevant [19:17:27] they will need to have been granted toolforge project membership [19:17:29] ningu: do you know for sure that they have been added to Toolforge already? [19:17:34] no [19:17:48] I can ask [19:18:04] I can double check for you if you know their Developer account name [19:18:27] he is doing it now [19:19:21] btw you can host static files in toolforge, right? I'm looking for the docs on that [19:20:34] ningu: yes. there are 2 ways to do that: run a webservice as your tool or use tools-static.wikimedia.org [19:20:42] ok yes, I see [19:20:46] i was looking for the static one [19:20:53] tools-static is -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server [19:21:10] might not be possible though, unless there has been any work done on a ticket I filed a few months ago [19:21:55] https://phabricator.wikimedia.org/T245474 [19:22:27] looks like you can't currently use oauth from client-side-only js from tools-static [19:23:40] bd808: btw his username is Iwsfutcmd, I guess he applied for membership a minute ago [19:24:43] ningu: {{done}} -- https://toolsadmin.wikimedia.org/tools/membership/status/825 [19:24:47] thanks! [19:31:31] hrm... what are best practices for git repos and tools, with multiple users? [19:32:17] if you develop on ux-wikimedia0.fnf.archive.org and become the tool, I guess you wouldn't want multiple people editing the $HOME/www/js dir and pushing? [19:37:27] ningu: I would stay the bet practice is to use a git repo, either a repo you create through toolsadmin or one somewhere like gitlab, write code locally, and pull the code to the tool [19:37:46] ok, yeah, that's kinda what I figured, thanks [19:38:38] so I should add the toolforge account's ssh key as a deploy key I guess [19:39:14] whoops I gave the wrong hostname there heh, I meant tools-login.wmflabs.org [19:39:56] oh and deploy key is irrelevant if it's public so nevermind :) [21:37:40] it's best not to commit private keys tu public repos ;) [21:39:59] Platonides: eh? [21:40:06] I meant as a deploy key on the github repo [21:40:12] in its settings [21:40:28] but, if it's a public repo then you don't need special permission to pull from it [23:41:54] !screen [23:41:54] $ script /dev/null (https://wikitech.wikimedia.org/wiki/Screen#Troubleshooting)