[00:58:03] <TheSandDoctor>	 !help I have previously used tools a fair bit, but it has been a while since I have installed python and it appears that python 3 needs reinstalling since trusty was deprecated/removed. I followed instructions for installing python and packages found on wikitech https://wikitech.wikimedia.org/wiki/Help:Toolforge/FAQ#My_Tool_requires_a_package_that_is_not_currently_installed_in_Toolforge._How_can_I_add_it? , however, only succeeded
[00:58:03] <TheSandDoctor>	 in installing python 2. Any advice would be greatly appreciated. I would love to get my bots up and running again.
[00:58:03] <wm-bot>	 TheSandDoctor: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[01:36:35] <bstorm_>	 TheSandDoctor: it sounds like you need tp rebuild a virtualenv https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation scroll to "solutions to common problems"
[01:37:19] <bstorm_>	 *Need to. Sorry for typos. I am typing on my phone on am airplane waiting for it to take off. 
[16:49:20] <Wurgl>	 tools-sgebastion-07 is pretty slow, is NFS-load to high?
[16:51:17] <Wurgl>	 "time ls -l" says real    0m29.954s (0.1 seconds is expected)
[16:51:18] <Krenair>	 bstorm_, ^
[16:51:46] <Krenair>	 https://grafana.wikimedia.org/d/000000568/labstore1004-1005?orgId=1
[16:51:49] <bstorm_>	 Wurgl: that can just be because the cgroup restrictions on there
[16:52:11] <bstorm_>	 It will restrict everything your shell does after some things...that said, it might not be at all
[16:52:28] <bstorm_>	 NFS load is quite low, actually
[16:52:31] <Krenair>	 I just logged it and it was behaving glacially slowly bstorm_ 
[16:52:35] <bstorm_>	 surprisingly low :)
[16:52:42] <bstorm_>	 So it's something else
[16:53:17] <bstorm_>	 LOL, I'm still waiting to log in.  It's quite slow there right now
[16:53:20] <Krenair>	  16:50:48 up 16 days,  3:35, 23 users,  load average: 2.63, 4.29, 3.91
[16:53:22] <bstorm_>	 I'll try to hunt it down
[16:53:52] <zhuyifei1999_>	 bstorm_: root login is fast
[16:53:58] <bstorm_>	 The NFS server's load is only "high" when it is over 64, which isn't uncommon with it's recent kernel
[16:54:25] <bstorm_>	 zhuyifei1999_: that suggests ldap/cgroups/sudo/who-knows possibilities, but that helps, thanks
[16:54:26] <zhuyifei1999_>	 someone is scp-ing
[16:54:31] <bstorm_>	 Good find!
[16:54:35] <bstorm_>	 That'll do it
[16:54:37] <bstorm_>	 if it's really big
[16:54:56] <bstorm_>	 Logging in as root to get some more context
[16:55:20] <zhuyifei1999_>	 hmm doesn't seen to be relevant. the scp seems gone but ny normal login is still D-state
[16:55:44] <Wurgl>	 Now I had for "ls -l" a line like "real    3m45.858s" 
[16:56:27] <Wurgl>	 just ~300 files in that directory, so not really huge
[16:58:20] <bstorm_>	 There's an scp process hanging around
[16:58:27] <bstorm_>	 the /home mount might be screwed up as well
[16:58:52] <zhuyifei1999_>	 my 'ls' without -l isn't finishing
[16:59:07] <bstorm_>	 That scp is on a file that is currently 12G
[17:00:50] <zhuyifei1999_>	 hmm. it's in S state right now
[17:02:04] <zhuyifei1999_>	 it seems just doing poll() and write() on fd1 and fd1 is a pipe that idk where it goes to
[17:02:53] <bstorm_>	 Everything gets slow when I hit the NFS.  The server seems ok, but the client is screwed
[17:03:12] <zhuyifei1999_>	 and it occasionally reads from fd3 which is the 12G file
[17:03:18] <bstorm_>	 yeah
[17:03:32] <bstorm_>	 That's still writing and may be filling the pipe so to speak
[17:03:49] <bstorm_>	 I'll kill it and reach out to the owner to say sorry/explain.  
[17:04:08] <zhuyifei1999_>	 k, yeah good idea
[17:04:12] <bstorm_>	 And ls seems fine now
[17:04:26] <zhuyifei1999_>	 yep, thanks
[17:04:31] <Wurgl>	 Fine
[17:04:43] <bstorm_>	 Looks good :)
[17:30:09] <bstorm_>	 I'm seeing the issue again...
[17:30:40] <bstorm_>	 Ah, same user, using rsync now instead lol
[17:33:36] <bstorm_>	 I'm suggesting the use of the scratch mount instead since that *might* not affect everyone the same way
[17:35:30] <bstorm_>	 Killed and am in contact with the user.
[18:13:20] <anomie>	 bstorm_: Ping regarding https://phabricator.wikimedia.org/T212972#5057586
[18:14:16] <bstorm_>	 Thanks!  Weird!?!  I totally missed that.
[18:18:40] <bstorm_>	 Will take a look
[18:55:24] <mutante>	 i noticed on labweb (wikitech) there is a cron "runJobs" that runs duplicate.. once as traditional cron and once as systemd timer  https://phabricator.wikimedia.org/T222900
[18:55:42] <mutante>	 it looks like the cron should just be removed.. but it's on both labweb1001/1002
[18:56:28] <mutante>	 already tested that puppet does not recreate it after removing it.. but also it did not absent it as the code says it was supposed to
[18:57:05] <mutante>	 if you agree i can just remove it and do the cleanup in puppet
[19:49:01] <andrewbogott>	 mutante: yep, just remove the unpuppetized one if your confident that they're redundant.
[19:49:03] <andrewbogott>	 thanks!
[22:55:23] <mutante>	 andrewbogott: ACK, done just now! updating ticket and resolving
[23:24:39] <lucaswerkmeister>	 !log tools.quickcategories deployed cf208a54e9 (track query execution times)
[23:24:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL