[01:36:47] <wm-bot>	 !log tools.bridgebot <bd808> Testing a Freenode<->Discord bridge
[01:36:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[01:41:13] <AntiComposite>	 o no
[09:22:29] <godog>	 hi, re: T259465 I see ack'd WMCS alerts from Aug 10th and 7th, are those ok to resolve before I turn on "retrigger ack'd alerts after 24h" ?
[09:22:30] <stashbot>	 T259465: VictorOps behavior on long-ack'd incidents - https://phabricator.wikimedia.org/T259465
[14:50:43] <bstorm>	 godog: I don't see any right now in VO?
[15:06:52] <godog>	 bstorm: I'm looking at https://portal.victorops.com/ui/wikimedia/incidents and e.g. https://portal.victorops.com/ui/wikimedia/incident/378 shows up
[15:10:56] <bstorm>	 Those are all resolved from what I see. Also I cannot see incident 378
[15:11:34] <bstorm>	 https://usercontent.irccloud-cdn.com/file/NDzzBEq8/Screen%20Shot%202020-08-17%20at%208.11.05%20AM.png
[15:11:40] <bstorm>	 That's really interesting...
[15:11:48] <bstorm>	 Like there might be some I cannot see
[15:12:39] <godog>	 interesting indeed! I'm digging further
[15:14:28] <godog>	 ah yeah so the alerts I'm seeing have no team (!)
[15:16:07] <godog>	 ok so those are generated from the icinga 'acknowledgment' emails, not sure why they wouldn't have a team though
[15:16:31] <CurbSafeCharmer>	 @bstorm I'm afraid refill is dead again!
[15:18:09] <godog>	 bstorm: thanks for your help! I've resolved the incidents
[15:18:34] <godog>	 still not 100% sure what's up with those, but we'll figure it out
[15:43:36] <andrewbogott>	 !log shinken deleting all VMS and the project, as per T236547
[15:43:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL
[15:43:39] <stashbot>	 T236547: "shinken" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236547
[15:44:50] <CurbSafeCharmer>	 !help
[15:44:50] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[15:45:07] <bd808>	 CurbSafeCharmer: do you have a question?
[15:45:45] <CurbSafeCharmer>	 Hi bd808 - I am not sure if Brooke is around at the moment - in the past she has helped when ReFill has got stuck. Generally she has restarted the pod to fix it
[15:46:22] <bd808>	 CurbSafeCharmer: ack. I see your ping about that here now.
[16:29:27] <CurbSafeCharmer>	 @bd808 shall I log it on phabricator?
[16:30:29] <bd808>	 bstorm: are you able to do a refill restart? Or maybe even better, point me to a how to on what you have been doing to get that horrible, broken tool to run again when it falls over?
[16:30:53] * bd808 knows that folks love refill on-wiki, but it is a pile of broken
[16:31:18] <bstorm>	 I can. I just delete the pod over at refill-api. On it!
[16:32:27] <bstorm>	 Done. It should be respawning
[16:32:42] <bstorm>	 Hopefully that fixes it
[16:41:53] <CurbSafeCharmer>	 @bstorm no change (yet)
[16:42:59] <bstorm>	 It has restarted, I am able to confirm
[16:43:11] <bstorm>	 So this could be something entirely different
[16:43:51] <bstorm>	 Hrm. The logs look like:
[16:43:52] <bstorm>	 https://www.irccloud.com/pastebin/qX6rrz9B/
[16:44:31] <bstorm>	 I wonder if tools-redis is having some kind of issue?
[16:45:18] <bstorm>	 It was clearly working and then stopped `[2020-08-17 16:43:24,797: WARNING/ForkPoolWorker-67] took 14.913414239883423`
[16:46:51] <CurbSafeCharmer>	 I think it just started working
[16:47:56] <CurbSafeCharmer>	 yup
[16:49:31] <bstorm>	 Weird
[16:49:37] <bstorm>	 No evidence of resource crashing https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?orgId=1&refresh=5m&var-namespace=tool-refill-api
[16:50:41] <bstorm>	 This is what I'd expect to cause the failure https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?panelId=2&fullscreen&orgId=1&refresh=5m&var-namespace=tool-refill-api
[16:51:46] <bstorm>	 I might double check the monitor to make sure it's all accurate
[20:16:15] <wm-bot>	 !log tools.bridgebot <bd808> Enabling ukwiki Telegram<->Discord bridge (T260502)
[20:16:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[20:21:03] <hauskatze>	 https://thegoodplace.wmflabs.org/wiki/Special:RecentChanges doesn't look so good ;-)
[22:19:31] <wm-bot>	 !log tools.bridgebot <bd808> Really enabling ukwiki Telegram<->Discord bridge this time. (T260502)
[22:19:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[22:29:33] <wm-bot>	 !log tools.bridgebot <bd808> Update Discord server id for ukwiki Telegram<->Discord bridge. (T260502)
[22:29:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[22:31:06] <wm-bb>	 [telegram] <Thecladis> Bd808 ping
[22:31:24] <bd808>	 Thecladis: what's up?
[22:31:49] <wm-bb>	 [telegram] <Thecladis> I see you struggling with the bot, but you do not seem to be in tg where I wrote you :)
[22:32:56] <bd808>	 I mostly Telegram on my phone when not working. I see your messages there now that I look. :)
[22:33:38] <bd808>	 that account I joined into the Discord is the "owner" of the bot. I needed to lookup the Discord server id (task had the channel id in it)
[22:34:13] <wm-bb>	 [telegram] <Thecladis> Oh, indeed, that was channel id
[22:34:31] <wm-bb>	 [telegram] <Thecladis> Relatable :) (re @wmtelegram_bot: [irc] <bd808> I mostly Telegram on my phone when not working. I see your messages there now that I look. :))
[22:45:51] <wm-bot>	 !log tools.bridgebot <bd808> Update Telegram channel id for ukwiki Telegram<->Discord bridge. (T260502)
[22:45:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL