[01:38:46] <Krenair>	 !log tools Cordoned tools-worker-1012 and deleted pods associated with dplbot and dewikigreetbot as well as my own testing one, host seems to be under heavy load - T241523
[01:38:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[01:38:50] <stashbot>	 T241523: tools-worker-1012 unhealthy? - https://phabricator.wikimedia.org/T241523
[02:06:46] <bd808>	 !log tools.fourohfour Migrating from Python 3.5 to Python 3.7
[02:06:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[02:11:04] <bd808>	 !log tools.fourohfour Getting weird results from py3.7 deployment. Reverting to py3.5
[02:11:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[02:18:24] <bd808>	 !log tools.fourohfour Weird behavior exists with py3.5 as well. Will open task and investigate further.
[02:18:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[03:42:32] <bd808>	 !log tools.replag Stop/Start cycle of webservice to see if it changes the active nginx ingress rules (T241525)
[03:42:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.replag/SAL
[03:42:34] <stashbot>	 T241525: fourohfour tool returning "replag" as the assigned tool for all 404 urls - https://phabricator.wikimedia.org/T241525
[04:01:55] <bd808>	 !log tools.stashbot Stop/Start for debugging T241525
[04:01:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[04:01:58] <stashbot>	 T241525: fourohfour tool returning "replag" as the assigned tool for all 404 urls - https://phabricator.wikimedia.org/T241525
[04:10:24] <bd808>	 !log tools.bash Stop/Start for debugging T241525
[04:10:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL
[04:10:27] <stashbot>	 T241525: fourohfour tool returning "replag" as the assigned tool for all 404 urls - https://phabricator.wikimedia.org/T241525
[04:37:45] <bd808>	 !log tools.fourohfour Creating custom ingress to try to fix T241525
[04:37:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[04:37:48] <stashbot>	 T241525: fourohfour tool returning "replag" as the assigned tool for all 404 urls - https://phabricator.wikimedia.org/T241525
[05:01:32] <leloiandudu>	 bd808: thank you, I will try to adjust connection pool settings in my tool
[05:22:51] <bd808>	 !log tools.fourohfour Migrated to py37 after figuring out that errors were unrelated to python version change
[05:22:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[15:04:29] <Cyberpower678>	 andrewbogott: ever since you moved the DB VM, CPU usage is absurdly high, and the writes to the DB are very slow.  My script is running at a snail's pace.
[15:51:51] <andrewbogott>	 Cyberpower678: that's interesting, I think at the moment you have that hardware almost to yourself.  Let me check.
[15:53:09] <andrewbogott>	 hm, maybe not
[15:59:15] <andrewbogott>	 Cyberpower678: the host it's on isn't under very high load but I can try moving you someplace else if you don't mind the downtime.
[16:02:41] <Cyberpower678>	 andrewbogott: Well right now, I'm running a script that's writing large amounts of data to it.
[16:02:49] <Cyberpower678>	 Let's wait for that to finish.
[16:03:31] <andrewbogott>	 reasonable :)
[16:09:13] <Cyberpower678>	 andrewbogott: it'll probably finish in 2 days based on the rate though. :-(
[16:09:22] <Cyberpower678>	 It used to only take 14 hours
[16:10:47] <andrewbogott>	 Cyberpower678: ping me when you're ready to try a migration, I'll be around
[16:10:57] <Cyberpower678>	 Will do.