[03:40:26] <wm-bot>	 !log tools.phab-ban <bd808> App busted due to attempted python3.7 upgrade being interruped by a power outage at my house.
[03:40:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phab-ban/SAL
[15:38:33] <wm-bot>	 !log tools.phab-ban <bd808> Upgraded to Python 3.7 and --canonical
[15:38:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phab-ban/SAL
[21:16:14] <Texas>	 !log tools.zppixbot kubectl delete pods --all
[21:16:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[21:55:24] <RhinosF1>	 !log tools.zppixbot-test invesigating incident
[21:55:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[22:09:19] <RhinosF1>	 I hate this bot at time
[22:12:36] <RhinosF1>	 !log tools.zppixbot-test no known reason for current increase in timeouts
[22:12:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[22:16:43] <RhinosF1>	 !help is there any reason that yesterday and today (on the toolforge side) there's been a massive surge in the bot timing out. The timings don't seem to correlate with it being my rolled back python update
[22:16:44] <wm-bot>	 Sorry, you are not authorized to perform this
[22:17:19] * RhinosF1 is lost
[22:17:47] <RhinosF1>	 I can't see anything in our logs that would indicate why it would suddenly surge
[22:59:01] <andrewbogott>	 RhinosF1: There's not anything I know of that would be causing trouble.  Are you seeing that across multiple bots, or just yours?  And, what is it trying to do when it times out?
[22:59:16] <andrewbogott>	 (Also, no idea why wm-bot scolded you, bang-help should work just fine here)
[22:59:35] <RhinosF1>	 andrewbogott: just the -test instance
[22:59:52] <RhinosF1>	 which is what I tried and rolled back a python update on
[23:00:14] <RhinosF1>	 but it's nearly two days after that that the occurences of the timeout bug surged
[23:00:30] <Texas>	 andrewbogott: wm-bot thought he was trying to set an infobot key
[23:00:34] <Texas>	 !help
[23:00:34] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[23:00:38] <Texas>	 !help is
[23:00:38] <wm-bot>	 Sorry, you are not authorized to perform this
[23:00:40] <Texas>	 yep
[23:00:55] <andrewbogott>	 huh
[23:00:57] <andrewbogott>	 seems broken :)
[23:01:10] <andrewbogott>	 RhinosF1: I don't understand what you mean by 'just the -test instance'
[23:01:23] <Texas>	 andrewbogott: He means the ZppixBot-test bot
[23:01:50] <RhinosF1>	 andrewbogott: zppixbot-test tool
[23:02:28] <andrewbogott>	 ok — if it's just the one tool then it's unlikely to be an infrastructure thing.  You could try restarting it just in case rescheduling gets it out of its rut.
[23:03:29] <andrewbogott>	 If you're able to get specific logs about what's getting throttled or failing we can try to investigate.  Probably best to open a ticket for that though.
[23:03:37] <RhinosF1>	 I tried deleting the pod once happened again but I've just restarted it for another issue
[23:04:20] <RhinosF1>	 andrewbogott: I just find it weird that there's a huge surge in a bug, it's probably not an infra thing, just baffaling me.
[23:04:58] <andrewbogott>	 sorry I can't help; I don't know anything about that particular bot
[23:05:37] * RhinosF1 might just try re-creating the venv if it goes weird again