[00:35:35] !log admin moving cloudvirt1023 back into maintenance because T269467 continues to puzzle [00:35:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:35:39] T269467: Upgrade nic firmware on cloudvirt1023 - https://phabricator.wikimedia.org/T269467 [00:42:31] !log paws `kubectl delete po renderer-794886b9cd-9nc6c -n prod` after seeing lots of listen queue full errors in the pod logs. [00:42:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [14:44:49] Hi, I have a couple of bots on toolforge which make use of the recent changes stream. But for many hours, they do not seem to be working. [14:45:36] I have not tried restarting them, though I could, but I wonder if they are no longer allowed to use the stream or what? [15:12:12] isn't that what logs are for? [15:12:34] "RCStream" has been offline for a few years, so I'm presuming you don't mean https://wikitech.wikimedia.org/wiki/Obsolete:RCStream [15:12:49] So presumably you are using https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams [15:15:20] Yes, stream.wikimedia.org -- I want to know what are some of the reasons the stream would stop -- could it be TMI causes a lag? [16:52:41] acagastya: I have noticed the same phenomenon on my side too [16:53:15] pintoch: were you able to fix it? [16:53:23] no… which library do you use to consume the stream? [16:53:42] Let's see. [16:54:21] npm.im/eventsource [16:54:48] and I use https://pypi.org/project/sseclient/ [16:55:40] I have had the project of trying to play with timeouts to detect when no events are sent, and then restart the stream (ideally at the right position), but I haven't got round to it yet [16:56:07] it's obviously not so easy to debug as the stream only stops after a few days in my experience [16:58:25] Few days? Which stream are you using? [16:59:29] Is it an IRC bot? [17:00:03] I consume the recent changes feed from various tools [17:00:33] (so, https://stream.wikimedia.org/v2/stream/recentchange) [17:00:43] for instance for https://editgroups.toolforge.org/ [17:02:42] It is not an IRC bot, right? [17:04:24] Any chance you could connect it to IRC? [17:04:38] huh? [17:04:55] I am using this trick -- if the tool is aborted, it is restarted by toolforge. [17:06:33] So, if my bot is not functioning, instead of restarting the tool after sshing, I just added this: if the bot receives a PM "KILL" from a particular user it executes `process.abort()` and the tool restarts. [17:06:55] Now, my bot being an IRC bot and me lurking on IRC all the time means I can do it quickly. [17:10:52] But if you don't have a way to signal abort, you should think of a way to make it easy.