[18:25:01] <multichill>	 How long is the command "qacct -j 2793552" supposed to take? :-)
[18:26:30] <multichill>	 Hmm, even crontab -l takes ages to complete
[18:26:44] <multichill>	 NFS problems on tools-bastion-02?
[18:29:00] <multichill>	 bd808: ^
[18:30:02] <Zppix>	 Hes at an conference, i dont think hes around atm multichill
[18:31:41] <multichill>	 Ah I see. He seems to be in the topic :-) Maybe !help works?
[18:31:50] <multichill>	 !help
[18:31:50] <wm-bot>	 multichill: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[18:31:57] <multichill>	 lol
[18:34:07] <madhuvishy>	 (We are all at kubecon/offsite)
[18:34:15] <madhuvishy>	 load and io looks fine on that server
[18:34:27] <madhuvishy>	 it may be that the accounting file is too large
[18:34:53] <multichill>	 Ah, nice, I noticed Yuvi commenting about it
[18:35:11] <multichill>	 All commands are extremely slow (crontab -l took more than 20 seconds).
[18:35:31] <madhuvishy>	 yeah
[18:35:32] <multichill>	 System is 90%+ idle so that's why I mentioned NFS
[18:35:42] <madhuvishy>	 become takes forever too
[18:37:10] <multichill>	 Are job id's supposed to be unique? The command returned a job ran in Mar 2017 by another user.... (still running)
[18:38:08] <multichill>	 A job seqfaulted several times and I think it's because of low mem so I want to check the returned result
[18:40:43] <madhuvishy>	 there's a user who seems to be running multiple tmux/mosh connections
[18:58:13] <zhuyifei1999_>	 multichill: don't run qacct on the bastions
[18:59:33] <zhuyifei1999_>	 and which host is slow? neither bastion-03 or -02 seems slow to me
[19:01:10] <Hauskatze>	 I was having issues with qstat this morning as well
[19:01:42] <Hauskatze>	 and I find toolforge slower than usual
[19:02:12] <multichill>	 zhuyifei1999_: Where else should I run it?
[19:03:32] <multichill>	 That's what they call intermittent problems. Sometimes it's fast, sometimes it's extremely slow. Or madhuvishy just fixed it ;-)
[19:04:04] <zhuyifei1999_>	 eg. find an less used grid node
[19:04:11] <zhuyifei1999_>	 or bastion-05
[19:04:45] <zhuyifei1999_>	 I'll file a ticket regarding making an optimized qacct that don't hog NFS too much
[19:05:09] <multichill>	 Any other to just get the status of a crashed job?
[19:05:43] <multichill>	 Can't we just rotate all junk older than a month into /dev/null?
[19:09:11] * zhuyifei1999_ dunno
[19:22:50] <zhuyifei1999_>	 multichill: https://phabricator.wikimedia.org/T182451
[20:03:32] * zhuyifei1999_ is scp-ing the accounting logs through the spare bastion-05 to an instance-local storage so that I can do a through analysis on them (eg. if the job ids always increases)
[22:05:23] <wikibugs>	 (03Draft1) 10MarcoAurelio: Add mediawiki/tools/phan/SecurityCheckPlugin [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/396533
[22:05:26] <wikibugs>	 (03PS2) 10MarcoAurelio: Add mediawiki/tools/phan/SecurityCheckPlugin [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/396533
[22:42:29] <wikibugs>	 (03PS1) 10Slashme: Use correct font name [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396540
[22:51:57] <wikibugs>	 (03PS1) 10Slashme: Correct color of Independent seats in US diagram [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396542
[22:52:11] <wikibugs>	 (03CR) 10Kaldari: [C: 032] Use correct font name [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396540 (owner: 10Slashme)
[22:55:03] <Sagan>	 !log rcm Xenon: Updating packages
[22:55:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[22:55:18] <Sagan>	 !log rcm Oxygen: Updating packages
[22:55:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[22:55:31] <Sagan>	 !log rcm CAC: Updating packages
[22:55:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[22:56:15] <Sagan>	 !log rcm Tin: Updating packages
[22:56:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:00:00] <Sagan>	 !log rcm Neon: Updating packages
[23:00:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:01:48] <Sagan>	 !log rcm CAC: Updating vagrant (git pull of vagrant dir)
[23:01:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:02:05] <Sagan>	 !log rcm CAC: Updating vagrant (vagrant git-update)
[23:02:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:04:11] <Sagan>	 !log rcm Xenon: Updating Phabricator
[23:04:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:08:00] <Sagan>	 !log rcm CAC: Updating composer
[23:08:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:08:42] <Sagan>	 !log rcm CAC: Rerunning update.php foreachwiki after fixing config error
[23:08:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[23:41:22] <wikibugs>	 (03CR) 10Slashme: [V: 032] Use correct font name [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396540 (owner: 10Slashme)
[23:51:50] <wikibugs>	 (03CR) 10Slashme: [V: 031 C: 032] Correct color of Independent seats in US diagram [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396542 (owner: 10Slashme)
[23:52:08] <wikibugs>	 (03CR) 10Slashme: [V: 032 C: 032] Correct color of Independent seats in US diagram [labs/tools/parliamentdiagram] - 10https://gerrit.wikimedia.org/r/396542 (owner: 10Slashme)