[08:49:51] !log wikidata-dev wb-reconcile: created instance [08:49:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [09:56:07] are there any known DNS issues in Cloud VPS at the moment? [09:56:16] I can’t SSH to wb-reconcile.eqiad.wmflabs or wb-scratch.eqiad.wmflabs via the normal method [09:56:36] and when I try to ping them from the primary.bastion.wmflabs.org, I get a “name and service not known” error [09:56:52] if I get the IP directly from Horizon and SSH to that manually (still with ProxyJump bastion-cloud), then it works [09:57:12] other instances in wikidata-dev, such as queripulator or wb-product-testing, seem to work normally [09:57:33] (s/name and service/name or service/ three messages up) [09:59:13] Lucas_WMDE: try `ssh wb-reconcile.wikidata-dev.eqiad1.wikimedia.cloud` [09:59:35] new VMs only get the newer domain [09:59:51] the old `eqiad.wmflabs` was left for compat reasons only on active VMs [10:00:08] ah, I see [10:00:16] so I was wrong about “the normal method” ^^ [10:00:22] one moment [10:01:02] yup, works \o/ [10:01:05] thank you! [10:01:09] 🎉! [10:01:14] any time :-) [10:03:01] do the new domains also work for old VMs? [10:09:14] yes! [10:09:34] ok, then I can try to get used to the new domains and maybe delete the old ones from my SSH config ^^ [14:45:12] hm, one more SSH question – hostnames without the project name are not supported under wikimedia.cloud? [14:45:42] Lucas_WMDE: no :-P [14:45:53] ok ._. [14:46:18] I guess I’ll just write a few “Host myalias \n HostName instance.project.eqiad1.wikimedia.cloud” files to make my life easier :P [14:47:31] at some point I experimented with having some kind of shortname for the domain, to be able to do `ssh myvm.project.w.c` but didn't work out at first and then I lost interest [14:54:20] Lucas_WMDE: if you do Host lines like this: Host doc1002.devtools wikistats-dancing-goat.wikistats wikistats-wild-tiger.wikistats deployment-mediawiki-parsoid10 [14:54:28] then you can ssh to host.project [14:55:44] ProxyCommand ssh -W %h.eqiad1.wikimedia.cloud:%p dzahn@restricted.bastion.wmcloud.org [14:56:14] * bd808 likes his ssh known_hosts setup to allow tab completion of hostnames [15:34:10] !log wikidata-dev wb-reconcile: rebuild instance (ansible setup worked reasonably well but installed a query service which required lots of RAM and was presumably the cause for kswapd0 using 99.9% CPU; try again without blazegraph) [15:34:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [15:42:52] !log wikidata-dev wb-reconcile: deleted and recreated instance, I couldn’t ssh into the rebuilt one [15:42:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [15:45:43] !log wikidata-dev wb-reconcile: deleted instance, waiting a bit before recreating it again [15:45:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [16:07:36] !log wikidata-dev wb-reconcile: recreated instance [16:07:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [16:08:11] Lucas_WMDE: beware, recreating the instance with the same name might need some time for the DNS to update itself [16:08:38] yup, I waited until `ping` and `dig` stopped resolving the hostname before recreating this time [16:08:44] (`ping` and `dig` on the bastion, that is) [16:08:46] cool [16:09:09] also, rebuild was apparently supposed to be disabled long ago (T140259), which might explain why I had issues with it [16:09:09] T140259: Investigate (and probably disable) 'rebuild instance' option - https://phabricator.wikimedia.org/T140259 [16:09:41] idk if it was fixed and intentionally reenabled in Horizon, or just became enabled in Horizon accidentally but still doesn’t work [16:10:04] no idea, never used it [16:10:24] when I want an instance recreated I manually delete it and create it again with the same name [16:19:18] now I can’t ssh in and also horizon is “unable to get log for instance "[guid]"” [16:19:29] I think I’ll just delete the instance one more time and try again tomorrow [16:19:41] and that’ll hopefully be enough time to clear whatever caches might be causing confusing [16:19:53] !log wikidata-dev wb-reconcile: deleted instance again [16:19:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [16:26:38] Lucas_WMDE: is reusing the instance name important for other things in the project? [16:28:07] no, I just couldn’t be bothered to think of an alternative one [16:28:25] and don’t want to end up with a final name like wb-reconcile-2-final-realfinal ;) [16:29:18] wb-- :) [16:29:44] like wb-reconcile-01, -02, -03, -04, .... [16:31:04] I’ll try that tomorrow if the first wb-reconcile doesn’t work out ^^ [16:48:33] 🧐 what happened there, @bd808 ? (re @wmtelegram_bot: no, I just couldnâ€™t be bothered to think of an alternative one) [16:50:46] @jhsoby: Lucas did a thing we say may or may not work and it didn't work mostly (delete instance named "foo" and then create new instance named "foo" right after). Race conditions in tearing down and setting up state in the OpenStack control plane are the likely culprit. [16:52:56] Sorry, i was being cryptic. On the telegram side, Lucas's message has mojibake: no, I just couldnâ€™t be bothered to think of an alternative one [16:55:01] @jhsoby: oh.. no idea. I do see his messages with smart quotes in my irc client. I'd guess that means that his client is sending some encoding that is not utf-8 and bridgebot is not trying to guess the encoding [16:56:21] So how does my surname turn out on irc? Søby [16:56:31] "Søby" [17:10:26] D'oh, i can't tell if it's right or wrong on the irc side. But it's wrong on telegram at least. Did something change? Because I'm pretty sure I've seen the bridge bot carry over emojis (which also ought to be affected) without problem before... [17:13:00] it's got your weird o/ [17:13:05] :P [17:15:40] 😂 [17:33:22] o/l as in beer :D [17:48:41] @jhsoby: does ø look correct? How about 🦄🎉? [17:49:30] (if the :unicorn_face: emoji passes through things should be utf-8 clean) [17:54:24] bd808: everything looks good. Thanks, good job 😁 [17:55:57] @jhsoby: my test was sending utf-8 codepoints from my irc client. So I think any round trip weirdness is probably the client on one side or the other of the bridge *not* using utf-8 encoding. [19:48:56] bd808: Ah, I see. So what you're saying is that we need to chastise Lucas_WMDE (into submission and conformity)?