[00:28:49] !log tools broke etcd trying to fix it and then restored it [00:32:08] oh yeah, that won't work right now đŸ˜… [01:32:05] yay! everything seems to be back. Thanks everyone! [01:33:05] musikanimal: sssshhh don't jinx us :) [01:33:15] hehe okay [01:33:25] not calling the all clear yet, but things are looking much better [01:34:43] musikanimal: you jinxed it [01:34:53] but we have a line on the proper fix [01:35:00] oh no! whoopsies [03:14:01] * AntiComposite had to read the email about the k8s disruption 3-4 times before he got the whole thing [03:17:08] I know what all the words mean and it still was a bit turboencabulator-y [09:46:21] job 6568688 (tools.stewardbots) is not being deleted for few minutes, looks like something abnormal? (in the past it has been deleted within minutes) [11:35:44] !log tools.pbbot Install wiki-java fork @ d1def2e (2019/09/07) [11:35:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [12:11:08] Could someone on toolforge please kill job 8307356? I've been trying to delete it for hours, with -f, no luck [12:18:14] hm, that’s at least two people requesting job kills, might be a general problem with the grid? [12:19:00] I don’t think I have the rights to kill them, but I can at least do… [12:19:15] !help revi and magnus91 need jobs killed/deleted (not working for some reason, see logs) [12:19:15] Lucas_WMDE: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [12:20:33] I commented on https://phabricator.wikimedia.org/T232536 [12:28:32] hm, I would be surprised if that’s related… aren’t jobs a gridengine thing? [12:29:09] though I suppose the grid could also be affected by the puppet/x509 mistake [13:20:30] revi: I'm looking into this [13:21:07] it doesn't appear to be related to the certificate problems, I think we have a SGE host misbehaving [13:30:57] !log tools restart tools-sgeexec-0912 [13:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:38:28] magnus91: 8307356 is deleted, the node running that job needed to be restarted. [14:00:59] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @amir1 & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:58] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:08:39] !log redirects Fixed broken /etc/puppet/puppet.conf on redirects-nginx01.redirects.eqiad.wmflabs [16:08:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL [16:26:31] Off-Topic – I start to like that DDoS from the last days … https://twitter.com/Wikimedia/status/1171476639064055808 Nice! [16:28:31] Wurgl: :) I can pretty much guarantee that Craig was talking to us for a long time about that grant [16:28:39] but it is super nice of him! [16:31:42] bd808: So, Wikimedia foundation hired some guys for that DDoS to force Craig's check? [16:32:17] that's not even a very plausible one Wurgl. You can do better ;) [16:32:39] *g* [18:02:52] !log tools.stashbot Updated to 07e7e73 (Fix nested dict iteration for py3) [18:02:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL