[01:24:29] <Zoranzoki21>	 !help Hello, I installed Windows I forgot which bastion I should use so I can connect to my Cloud VPS instances
[01:24:29] <wm-bot>	 Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[01:25:46] <Zoranzoki21>	 Oh, it's username@primary.bastion.wmflabs.org I found now. Works, thanks in each case
[01:42:33] <bstorm_>	 Glad to hear!
[01:49:38] <Zoranzoki21>	 bstorm_: Oh, you are here :). Fine. I run sudo apt-get dist-upgrade on my discordwiki.srwiki-dev.eqiad.wmflabs instance, and this is output: https://pastebin.com/5F5kpVUP
[01:50:06] <Zoranzoki21>	 I'm confused a bit, as there is error. (Read from line 154)
[01:51:25] <bstorm_>	 It's trying to install VM software on a VM?
[01:52:23] <Zoranzoki21>	 VM software on VM?
[01:53:18] <bstorm_>	 Hrm.
[01:54:07] <bstorm_>	 It's failing on a libvirt upgrade. 
[01:54:16] <Zoranzoki21>	 I do not know, it is essential that the site works.
[01:54:42] <bstorm_>	 Anyway, as silly as it sounds, does it fail if you try it again?
[01:55:53] <Zoranzoki21>	 Yes, https://pastebin.com/iCKYdR3w
[01:56:12] <bstorm_>	 What does 'sudo journalctl -xe' say?
[01:57:17] <Zoranzoki21>	 https://pastebin.com/AFqhpQ9j
[01:58:10] <bstorm_>	 Apparently, you need virtlogd to be enabled in the latest version of libvirt. 
[01:58:35] <bstorm_>	 So try 'sudo systemctl enable virtlogd' then try
[01:59:39] <Zoranzoki21>	 https://pastebin.com/aAxCfPVM
[01:59:43] <bstorm_>	 It seems this is an odd systemd dependency thing, reading up on it. 
[02:01:11] <Zoranzoki21>	 journalctl -xe now: https://pastebin.com/GBgz4QLn
[02:01:45] <bstorm_>	 Hrm. 
[02:01:58] <bstorm_>	 Can you reboot it?
[02:02:04] <Zoranzoki21>	 Oh I just wanted to ask
[02:02:06] <Zoranzoki21>	 Sure
[02:03:32] <bstorm_>	 Apparently this issues shows up during a certain set of version upgrades, but I haven't hit it myself. Kind of hoping systemd will sort it out with reboots. 
[02:04:00] <bstorm_>	 Sorry for the typos. Typing on my phone. 
[02:04:49] <Zoranzoki21>	 Reboot done
[02:04:53] <Zoranzoki21>	 I tried again and works
[02:05:02] <bstorm_>	 Lol. Nice
[02:05:18] <bstorm_>	 Well. At least that was easy.
[02:05:37] <bstorm_>	 Glad it works. Sorry I didn't know a better way. 
[02:06:12] <Zoranzoki21>	 This is even the easiest, no problem
[02:06:45] <Zoranzoki21>	 Thank you for help as always (and bd808 also)
[04:28:11] <gifti>	 can we have a jrestart or something to that effect on toolforge? that would help with stopping it, waiting for it to go down and then starting it up again
[06:09:04] <DSquirrelGM>	 what exactly do you mean by jrestart?
[06:40:09] <legoktm>	 gifti: IIRC qmod -rj should do that
[06:40:18] <legoktm>	 specifically, qmod -rj $jobname
[06:42:27] <gifti>	 that does not work for me (denied: host "tools-sgebastion-07.tools.eqiad.wmflabs" is not an admin host)
[08:02:47] <legoktm>	 hm
[08:27:52] <zhuyifei1999_>	 yeah qmod -rj doesn't work on the new sge
[08:28:06] * zhuyifei1999_ support such a tool
[08:28:12] <zhuyifei1999_>	 jrestart
[08:28:42] <zhuyifei1999_>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Restarting_jobs_(Deprecated)
[09:44:26] <arturo>	 !log admin deleted DNS zone wmcloud.org and try re-creating it
[09:44:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:50:04] <arturo>	 !log tools temporarily delete DNS zone tools.wmcloud.org to try re-creating it
[09:50:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:42:36] <wmf2020>	 Hi!! I can install Hyper-V to create many virtual machines to provision puppet for MediaWiki???
[13:47:44] <wmf2020>	 Hi!! I can install Hyper-V to create many virtual machines to provision puppet for MediaWiki???
[13:53:38] <wmf2020>	 Hi!! I can install Hyper-V to create many virtual machines to provision puppet for MediaWiki??? davidwbarratt
[14:10:34] <wmf2020>	 Hi!! I can install Hyper-V to create many virtual machines to provision puppet for MediaWiki???
[14:14:15] <Lucas_WMDE>	 wmf2020: please stop spamming the same question
[14:14:40] <wmf2020>	 Lucas_WMDE I can provision Puppet???
[14:48:46] <Zppix>	 arturo:  are you able to restart the copyvio tool again for me please?
[15:12:32] <andrewbogott>	 I just reset all keystone sessions for T243418.  Please let me know if anyone encounters new, bad behavior
[15:12:32] <stashbot>	 T243418: Convert keystone from uuid tokens to fernet tokens - https://phabricator.wikimedia.org/T243418
[17:00:52] <arturo>	 !log tools.copyvio restarting webservice per Zppix request on IRC
[17:00:53] <stashbot>	 arturo: Unknown project "tools.copyvio"
[17:00:58] <arturo>	 !log tools.copyvios restarting webservice per Zppix request on IRC
[17:00:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.copyvios/SAL
[19:30:37] <ningu>	 I assume it's no problem to combine static file serving on tools-static.wmflabs.org with (say) a nodejs webservice for the same tool?
[19:33:11] <ningu>	 but it seems like it wouldn't work to run, say, a nodejs service at the same time as a python service?
[19:35:17] <DSquirrelGM>	 Why wouldn't you just have the "static" files under the webservice in the first place?
[19:35:51] <ningu>	 because the webservice is simply a proxy to get around CORS restrictions that haven't been updated yet, and I hope it won't be needed forever
[19:36:33] <ningu>	 I want to send api requests from tools.wmflabs.org to, say, wikisource.org/w/rest.php/oauth2/access_token
[19:37:05] <ningu>	 but I can't get an access token that way currently without hosting a little server on tools.wmflabs.org to proxy the request
[19:39:34] <ningu>	 https://phabricator.wikimedia.org/T245474
[19:53:23] <Gopa>	 can we use RabbitMQ on toolforge ?
[19:54:52] <valhallasw`cloud>	 Gopa: I don't think there is a shared RabbitMQ server available. We do have a Redis server iirc
[19:55:37] <Gopa>	 valhallasw`cloud: okay
[19:56:07] <valhallasw`cloud>	 ningu: static & dynamic services are completely separated so you can use both any way you want (well, as long as the static one serves static files ;-)).
[19:56:50] <valhallasw`cloud>	 ningu: nodejs + python is also possible in theory but would probably require a custom lighttpd webservice. I would suggest to just make a second tool if you really need two types of webservices running at the same time
[19:57:13] <Gopa>	 valhallasw`cloud: any wiki link for the redis server iirc ?
[19:57:47] <Gopa>	 documentation*
[19:57:48] <valhallasw`cloud>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge
[19:58:00] <Gopa>	 thanks valhallasw`cloud :)
[20:06:58] <ningu>	 valhallasw`cloud: yeah really I was just curious if it was technically possible. I don't need it. I do need static + nodejs probably
[20:07:01] <ningu>	 thanks!
[21:02:24] <andrewbogott>	 !log admin adding .eqiad1.wikimedia.cloud records to all existing eqiad1 VMs, updating all eqiad1 internal pointer records to reference the new eqiad1.wikimedia.cloud fqdns.
[21:02:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[22:16:12] <andrewbogott>	 !log admin moved wmcloud.org dns domain to the cloud-infra project
[22:16:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[22:19:33] <andrewbogott>	 !log admin transferred the tools.wmcloud.org. to the tools project
[22:19:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[22:22:55] <Krinkle>	 some cron failed and e-mailed me:
[22:22:57] <Krinkle>	 > error: commlib error: access denied (client IP resolved to host name "tools-sgecron-01.tools.eqiad1.wikimedia.cloud". This is not identical to clients host name "tools-sgecron-01.tools.eqiad.wmflabs")
[22:23:04] <Krinkle>	 seems to be infra related, not related to my cron
[22:23:31] <Krinkle>	 "Cron <tools.krinklebot@tools-sgecron-01> /usr/bin/jsub  …"
[22:24:13] <Krenair>	 Krinkle, cloud team are aware, it relates to the new names, being worked on
[22:24:14] <Zppix>	 Krinkle: i also got an email
[22:24:34] <Zppix>	 Krinkle: but mine is different
[22:25:10] <Zppix>	 Incase this may be useful for you guys heres the cron error email i got https://www.irccloud.com/pastebin/y268GDMf
[22:25:29] <Krinkle>	 looks the same to me
[22:25:37] <Krinkle>	 I only pasted the first line
[22:25:39] <bd808>	 yeah, that's functionally the same problem
[22:25:50] <Zppix>	 Oh
[22:25:58] <Zppix>	 I guess i should not skim over
[22:26:04] <bd808>	 grid does host based auth and the change of the PTR records made it sad
[22:26:15] <Zppix>	 Dont you love DNS bryan?
[22:26:37] <bd808>	 change is reverted and should get things working again as soon as DNS caches expire
[22:26:54] <bd808>	 computers are the worst ;)
[22:27:19] <Zppix>	 Heh
[22:27:22] <Krenair>	 Zppix, this feels like it's caused by the way SGE works, not sure we can place the blame entirely on DNS
[22:27:43] <Zppix>	 Krenair: true
[22:28:34] * Zppix is trying to find ways to keep procrastinating setting up the k8s for zppixbot-test
[22:38:18] <bd808>	 grid submission is working again
[22:38:37] <Krenair>	 Krinkle, Zppix ^
[22:40:11] <Zppix>	 Krenair: thanks for the ping luckily cron doesnt affect me much as i use it just as a simple git pull
[23:20:20] <bstorm_>	 !log toolsbeta added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists
[23:20:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[23:26:51] <bstorm_>	 !log tools added tools-sgegrid-master.tools.eqiad1.wikimedia.cloud and tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud to gridengine admin host lists
[23:26:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL