[09:35:05] at toolforge how do i check the uptime of my currently running job? [09:35:11] i.e. when it was started [09:35:56] i see 'qstat' returns empty output; i'd like to check whether or not the job is running, for starters :) [09:36:13] `webservice status` should tell you that, at leas [09:36:38] if the tool is running on kubernetes, `kubectl get pods` should report an age, too [11:43:44] !log tools.pbbot Webservice down since 20201014, deploy cc97af8 [11:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [13:53:17] !log tools.pbbot Deploy e0d6437: fix URL extraction in pretty-ref and NKJP-generator [13:53:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [13:56:16] !log admin [codfw1dev] drop neutron l3 agent hacks in cloudnet2002/2003-dev (T261724) [13:56:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:56:19] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [15:02:03] gry: normally if qstat has no data for your tool then there are no grid jobs running. There can be some exceptional circumstances where the job gets "lost" by the accounting process, but that is fairly rare. [15:17:30] !log admin [codfw1dev] try cleaning up anything related to address scopes in the neutron database (T261724) [15:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:17:34] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [16:44:41] !log tools depooling tools-k8s-worker-35, 37, 45 [16:44:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:27:50] !log tools uncordoning tools-k8s-worker-35, 37, 45 [17:27:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:28:26] !log tools depooling tools-k8s-worker-53, 54, 55, 59 [17:28:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:23:41] !log tools uncordoning tools-k8s-worker-53, 54, 55, 59 [18:23:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:36:53] !log paws draining paws-k8s-worker-1 for move to ceph [18:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [18:48:20] !log paws draining paws-k8s-worker-2 for move to ceph [18:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [19:09:04] bd808: ok thanks [19:12:54] !log paws uncordoned paws-k8s-worker-1 and -2 [19:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [20:46:28] so wikibugs phab part died quite a while back [20:47:54] it's working in -operations [20:48:04] er, no, that's gerrit [20:49:01] yeah [20:49:17] !log tools.wikibugs $ python3 manage.py start_job wb2-phab [20:49:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [20:55:36] mutante: for future reference, how did you find https://phabricator.wikimedia.org/T265568#6547888 ? [20:59:49] legoktm: [lists1001:~] $ grep potd /var/log/exim4/mainlog [21:00:10] ahh, well thank you :) [21:00:29] np [21:00:41] so regarding the WMF mailservers [21:00:48] they add a header with a spam score [21:00:54] but they don't drop it based on that [21:01:21] what happens based on the spam score is on the mailman side.. and can be different config for each list [21:01:44] got it, I feel like we should be able to just disable the spam score thing since we only have one address approved to send out stuff [21:01:45] well..maybe there is a global spam score where exim rejects it too.. but that would be much higher [21:02:12] sounds like it, yea [22:00:00] !log tools manually removing nscd from tools-sgebastion-08 and running puppet [22:00:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:01:00] bstorm: confirmed noop on clouddb1004 fwiw [22:01:14] Thanks for making sure :)