[05:21:57] <[1997kB]`>	 balloons, yes I followed https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_from_a_Servlet_in_Tomcat
[09:30:59] <[1997kB]>	 frequent 502 Bad gateway..
[09:32:10] <arturo>	 !log admin rebooting cloudvirt1012 to investigate linuxbridge agent issues
[09:32:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:24:42] <arturo>	 !log tools.sal tool webservice detected to be misbehaving, several uncaught exceptions in the source code
[11:24:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL
[11:26:30] <arturo>	 !log tools.sal trying a simple `webservice restart`
[11:26:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL
[11:27:04] <arturo>	 !log puppet-diffs syncing facts from puppetmaster1001
[11:27:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL
[11:33:12] <arturo>	 !log admin disabling puppet and downtiming every virt/net server in the fleet in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 (T262979)
[11:33:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:33:15] <stashbot>	 T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) - https://phabricator.wikimedia.org/T262979
[11:36:28] <arturo>	 !log admin [codfw1dev] rebooting cloudnet2003-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
[11:36:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:38:23] <arturo>	 !log admin [codfw1dev] rebooting cloudnet2002-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
[11:38:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:40:36] <arturo>	 !log admin rebooting cloudnet1004 (standby) to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 (T262979)
[11:40:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:40:39] <stashbot>	 T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) - https://phabricator.wikimedia.org/T262979
[11:46:57] <arturo>	 !log admin rebooting cloudvirt1012
[11:46:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:49:01] <arturo>	 !log admin rebooting cloudvirt1039
[11:49:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:24:12] <arturo>	 !log cloudvirt-canary created canary1012-01 VM in cloudvirt1012
[12:30:28] <arturo>	 !log admin move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
[12:31:13] <arturo>	 stashbot_: ?
[12:32:55] <arturo>	 !log tools.stashbot restarting with `webservice restart`
[12:36:22] <arturo>	 !log admin move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
[12:36:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:36:38] <arturo>	 !log admin rebooted cloudnet1003 (active) a couple of minutes ago
[12:36:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:36:57] <arturo>	 !log tools.stashbot restarted with `bin/stashbot.sh restart`
[12:36:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[12:39:21] <arturo>	 !log admin root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1031
[12:39:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:40:44] <arturo>	 !lo admin root@cloudcontrol1005:~# wmcs-drain-hypervisor cloudvirt1031
[12:51:12] <arturo>	 !log admin rebooting cloudvirt1013 and moving it to the ceph host aggregate
[12:51:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:55:15] <arturo>	 !log admin rebooting cloudvirt1014 and moving it to the ceph host aggregate
[12:55:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:56:48] <arturo>	 !log cloudvirt-canary creating VM canary1013-01
[12:56:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL
[12:58:55] <arturo>	 !log cloudvirt-canary creating VM canary1014-01
[12:58:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL
[13:02:13] <arturo>	 !log admin rebooting cloudvirt1016 and moving it to the ceph host aggregate
[13:02:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:06:46] <arturo>	 !log cloudvirt-canary creating VM canary1016-01
[13:06:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL
[13:15:15] <arturo>	 !log admin `aborrero@cloudcontrol1003:~$ sudo nova-manage placement sync_aggregates` after reading a hint in nova-api.log
[13:15:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:25:07] <arturo>	 !log cloudvirt-canary deleted a bunch of failed VMs
[13:25:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL
[13:27:30] <arturo>	 !log admin extend icinga downtimes for another 120 mins
[13:27:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:28:49] <arturo>	 !log admin enable puppet, reboot and pool back cloudvirt1031
[13:28:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:48:43] <anomie>	 bd808: All AnomieBOT's processes seem to be blocked somehow since about 16:25 UTC yesterday. Do you (generic) want to look at it at all, or should I go ahead and try killing the jobs and such?
[15:49:43] <wm-bot>	 !log tools.wikiloves <jeanfred> Rebuilding missing datasets
[15:49:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL
[15:49:47] <bd808>	 anomie: could you get us a dump of your qstat job status to look at?
[15:50:44] <anomie>	 Just `qstat`, or should some args be used?
[15:50:52] * bd808 was briefly confused about whey `sudo become anomiebot` didn't work... on his local laptop :/
[15:51:23] <bd808>	 anomie: trying to remember now. I think we need the xml dump to see interesting details
[15:52:13] <bd808>	 anomie: I got a dump. Start fixing your things as needed
[15:53:32] * anomie starts by trying to jstop all the jobs
[15:59:07] <bd808>	 anomie: looks like they are stuck in 'd' state. Should I sudo kill them for you?
[15:59:32] <anomie>	 bd808: Go ahead
[15:59:59] <bd808>	 !log tools.anomiebot sudo qdel -f 2097160
[16:00:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL
[16:00:48] <bd808>	 !log tools.anomiebot sudo qdel -f 2097161 2097166 2097167 336318 417408
[16:00:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL
[16:01:02] <bd808>	 anomie: ready to start them back up I think
[16:03:10] <anomie>	 Thanks!
[16:03:19] <bd808>	 I'm running startbot.sh now
[16:03:21] <wm-bot>	 !log tools.anomiebot <root> ./startbot.sh
[16:03:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL
[16:03:57] <anomie>	 I think I beat you to it (:
[16:04:18] <anomie>	 But the script should be safe, the second attempt will just report "is already running"
[16:08:27] <bd808>	 anomie: "scheduling info: job dropped because of user limitations" -- does that sound normal? % of the jobs are in that state ("qw")
[16:08:34] <anomie>	 ... And, looks like I spoke too soon. We did manage to have double jobs running.
[16:09:13] * anomie qdels everything again
[16:09:44] <bd808>	 crud. Yeah 15 are running and the other 5 are queued. I think 15 is the default concurrent job limit these days
[16:11:39] <Waggie>	 bd808: I have Freenode staff asking about wm-bb and some sort of flood/reconnect issue..
[16:12:04] <bd808>	 Waggie: ugh. Did they say what channel?
[16:12:17] <Waggie>	 Well, flood/reconnect would be any channel that it's in.
[16:12:39] <bd808>	 let me see if the bot's logs tell me anything
[16:13:06] <anomie>	 Ok, qdeled everything and restarted just once this time ;)
[16:14:13] <wm-bot>	 !log tools.bridgebot <bd808> Restarting. IRC join failures
[16:14:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[16:15:13] <bd808>	 Waggie: trying a full restart to see if that fixes the bot. I think it gets into strange states when it netsplits
[16:15:28] <bd808>	 looks like it at least joined properly here
[16:15:54] <bd808>	 Waggie: bot's logs look good now. Thank you for the poke
[16:17:22] <Waggie>	 Ok, thanks
[16:47:32] <andrewbogott>	 !log admin rebooting cloudvir1032, 1033, 1034 for T262979
[16:47:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[16:47:36] <stashbot>	 T262979: cloudvirts: the rocky/buster combo has iptables/ebtables issues, producing errors when launching VMs (and probably other stuff) - https://phabricator.wikimedia.org/T262979
[16:53:54] <wm-bot>	 !log tools.bridgebot <bd808> Upgrade to 1.18.3; Add RejoinDelay=5 for IRC channels (T264212)
[16:53:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[18:25:36] <bstorm>	 !log tools.listeria restarting the webservice because it's not receiving requests from what I can tell T264219
[18:25:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.listeria/SAL
[18:25:39] <stashbot>	 T264219: Toolforge nginx/openresty timeout is too short - https://phabricator.wikimedia.org/T264219
[18:29:16] <andrewbogott>	 !log tools depooling tools-sgeexec-0918 so I can reboot cloudvirt1036
[18:29:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:34:19] <andrewbogott>	 !log tools repooling tools-sgeexec-0918
[18:34:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:11:29] <wm-bot>	 !log tools.refill-api <root> Restarting for T264233
[20:11:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL
[21:12:34] <wm-bot>	 !log tools.bridgebot <bd808> Another restart for irc flood crash (T264212)
[21:12:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[22:14:35] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> installed cldr because [[User:Jforrester|Jforrester]] said so :)
[22:14:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[23:32:32] <lucaswerkmeister>	 not sure if anyone’s online at this time but I thought I’d try a quick kubernetes question
[23:32:53] <lucaswerkmeister>	 I wrote a deployment.yaml for a continuous job, based on the stashbot example on wikitech
[23:33:03] * bd808 listens
[23:33:07] <lucaswerkmeister>	 and after a few syntax errors, `kubectl create -f deployment.yaml` seemed to succeed
[23:33:12] <lucaswerkmeister>	 but it’s just not been created for half an hour now
[23:33:28] <lucaswerkmeister>	 only one pod in `kubectl get pods`, presumably the webservice
[23:33:40] <lucaswerkmeister>	 `kubectl get deployments` shows it as ready 0/1, up-to-date 0, available 0
[23:33:52] <lucaswerkmeister>	 is that just the consequence for asking for a lot of memory? (2Gi)
[23:34:10] <lucaswerkmeister>	 (notwikilambda tool, fwiw)
[23:35:52] <bd808>	 lucaswerkmeister: run `kubectl get deployment.apps/notwikilambda.update -o yaml` and cehck out the errors it is showing
[23:36:11] <lucaswerkmeister>	 ah
[23:36:16] <lucaswerkmeister>	 “Invalid value: "2Gi": must be less than or equal to memory limit'”
[23:36:19] <lucaswerkmeister>	 I thought the limit was 4Gi?
[23:36:41] <lucaswerkmeister>	 or do I need to specify a non-default “limits” as well, if I’m specifying a “requests”?
[23:36:42] <bd808>	 I think that's the limit for the entire namespace...
[23:38:28] <lucaswerkmeister>	 addding limits: memory: "2048Mi" seems to have fixed it
[23:38:31] <lucaswerkmeister>	 thanks a lot!
[23:38:45] <bd808>	 heh. sounds like a parse issue :)
[23:39:29] <lucaswerkmeister>	 huh, I didn’t even notice that the error message said 2Gi while my yaml file says 2048Mi ^^
[23:39:34] <lucaswerkmeister>	 anyways, seems to be working
[23:39:38] <lucaswerkmeister>	 and will hopefully be more stable than a grid job
[23:40:25] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> migrated `cron` from jstart to k8s, see deployment.yaml
[23:40:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL