[08:09:27] !log admin downtiming labstore1004 and stopping puppet for mysql restart (T279657) [08:09:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:09:31] T279657: Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 [08:40:33] !log admin enabling puppet on labstore1004 after mysql restart (T279657) [08:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:40:37] T279657: Upgrade mysql on db1128 (m5 db master) - https://phabricator.wikimedia.org/T279657 [10:51:22] !log tools setting prometheus data source in grafana to 'server' to avoid CORS issues [10:51:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:53:13] !log tools reverting setting prometheus data source in grafana to 'server', can't connect, [10:53:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:23:32] !log tools.lexeme-forms deployed 4effbc2a36 (l10n updates) [20:23:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:26:53] !log tools.lexeme-forms deployed 2f8f589a62 (Swedish proper nouns) [20:26:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [23:23:46] Anyone around who can kill a zombie grid job? 3477038 should be deleted but isnt going away [23:30:20] !log tools.betacommand-dev Force deleted job 3477038 per irc request [23:30:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.betacommand-dev/SAL [23:30:28] Betacommand: ^ [23:31:07] thanks, I tried killing it and it wouldnt die [23:33:31] submission time was "Fri Feb 26 20:00:35 2021". I wouldn't be surprised if the job was actually already dead and the grid scheduler just didn't notice [23:33:50] we had a few rough days with the grid in the last month :/