[09:47:30] <arturo>	 !log tools run `apt-get clean` to wipe 4GB of unused .deb packages, usage on / (root) was > 90%
[09:47:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:47:46] <arturo>	 !log tools run `apt-get clean` to wipe 4GB of unused .deb packages, usage on / (root) was > 90% (on tools-sgebastion-08)
[09:47:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:39:37] <xSavitar>	 !log tools.awmd-stats Deployed 426f7f6 and restarted the webservice
[10:39:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL
[10:48:06] <xSavitar>	 !log tools.awmd-stats Tool up and running again after restart
[10:48:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL
[16:10:45] <arturo>	 Lucas_WMDE: you were using mosh to connect to toolforge?
[16:11:51] <Lucas_WMDE>	 arturo: yes
[16:12:13] <arturo>	 could you share your setup? :-P
[16:12:24] <arturo>	 specifically, all the bastion/proxy dance
[16:13:20] <arturo>	 Lucas_WMDE: I was expecting a drop-in replacement, but apparently doesn't work that way
[16:13:51] <Lucas_WMDE>	 it does for me, I have nothing mosh-specific in my config…
[16:13:57] <Lucas_WMDE>	 I think those are the most important snippets from my SSH config: https://gist.github.com/lucaswerkmeister/53abe8d26e4f4833bedfa09c164f289a
[16:15:14] <arturo>	 and the you just `mosh server.example.com`?
[16:15:21] <Lucas_WMDE>	 `mosh toolforge`
[16:15:49] <Lucas_WMDE>	 the ~/.ssh/config.d/toolforge file maps Host toolforge to HostName login-stretch.tools.wmflabs.org
[16:15:57] <Lucas_WMDE>	 (which reminds me, I should remove the -stretch from there now ^^)
[16:16:55] <arturo>	 oh, you don't use a bastion to redirect the connection, you just connect to the server -_-
[16:17:28] <Lucas_WMDE>	 oh yeah
[16:17:35] <Lucas_WMDE>	 connecting via mosh to any other host can’t work, they’re firewalled
[16:17:42] <Lucas_WMDE>	 only the bastion is reachable
[16:18:39] <Lucas_WMDE>	 I guess the bastion-cloud config in that gist is actually unrelated, nevermind that
[16:18:44] <Lucas_WMDE>	 got confused myself :D
[21:49:14] <lucaswerkmeister>	 can someone with root rights on the tools grid find out what PID 7551 on tools-sgeexec-0942.tools.eqiad.wmflabs is doing?
[21:49:22] <lucaswerkmeister>	 it’s from my tool (wd-shex-infer), but I’m not allowed to strace it
[21:51:18] <Krenair>	 zhuyifei1999_?
[21:51:53] * zhuyifei1999_ looks
[21:54:15] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/sLC5alqz/
[21:54:22] <zhuyifei1999_>	 it's CPU-bound
[21:54:34] <zhuyifei1999_>	 lucaswerkmeister: ^
[21:54:41] <lucaswerkmeister>	 hm
[21:55:44] <lucaswerkmeister>	 it’s running this thing https://github.com/lucaswerkmeister/RDF2Graph/blob/b4f57f57752b372b342873f97265adc30d3649fb/Makefile#L14-L19 but that’s supposed to have a `sleep 5` between iterations
[21:55:59] <lucaswerkmeister>	 I’ll try to stare at it a bit longer and see where it could be burning all that CPU
[21:56:12] <lucaswerkmeister>	 but otherwise I’d better kill that job, it’s clearly not doing anything useful
[21:56:37] <lucaswerkmeister>	 thanks anyways
[21:57:59] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/22kvPX7K/
[21:58:17] <zhuyifei1999_>	 if that helps
[21:58:41] <lucaswerkmeister>	 that looks like there should be some clone()s or fork()s in the strace output…
[22:02:23] <lucaswerkmeister>	 I’ve stopped the job
[22:02:40] <lucaswerkmeister>	 zhuyifei1999_: can you exit gdb so the zombie can be reaped? :)
[22:02:58] <zhuyifei1999_>	 ok
[22:05:20] <lucaswerkmeister>	 !log tools.wd-shex-infer killed rogue job and disabled job submission via /data/scratch/wd-shex-infer/broken-2019-05-28
[22:05:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL
[22:10:09] <gabrielchihongle>	 https://usercontent.irccloud-cdn.com/file/bT4evsns/image.png
[22:11:07] <gabrielchihongle>	 wrong channel, sorry 😂
[22:11:15] <lucaswerkmeister>	 :D