[10:42:56] <Majavah>	 I just created deployment-urldownloader03, it's refusing to let me in with "debug1: channel 0: FORCE input drain" "Connection closed by UNKNOWN port 65535", already tried rebooting. Anything else I should try?
[11:04:14] <Majavah>	 it vies a "Permission denied (publickey)." when trying with an invalid key, so my key is accepted but for some other reason not letting me log in
[11:23:58] <Krenair>	 hi Majavah 
[11:24:10] <Krenair>	 I get: channel 0: open failed: administratively prohibited: open failed
[11:24:12] <Krenair>	 stdio forwarding failed
[11:24:14] <Krenair>	 ssh_exchange_identification: Connection closed by remote host
[11:26:56] <Majavah>	 Krenair: that sounds like a wrong hostname
[11:29:54] <Majavah>	 because I think "administratively prohibited: open failed" means that the bastion can't open a connection to the vm itself
[11:30:20] <Majavah>	 Krenair: deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud is the full hostname I'm using, make sure you're not using .wmflabs or something similar
[11:33:09] <Krenair>	 ooh my SSH config might be doing that by default
[11:33:17] <Krenair>	 alright yeah I get the same as you
[11:34:02] <Krenair>	 ok I'm in as root
[11:34:19] <Krenair>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::pki::client::auth_key' (file: /etc/puppet/modules/profile/manifests/pki/client.pp, line: 7) on node deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud
[11:34:49] <Majavah>	 ahhh ...
[11:35:02] <Majavah>	 that hiera value is set in puppetmaster04 private
[11:35:56] <Krenair>	  /etc/security/access.conf.d/99-labs_restrict_to_project contained `-:ALL EXCEPT (project-admin) root:ALL`
[11:36:03] <Majavah>	 but that profile::pki::client was set via horizon to all instances
[11:36:23] <Majavah>	 it does not let me log in since puppet can't change the access rule
[11:38:25] <Krenair>	 I've updated the access file
[11:38:35] <Majavah>	 thanks
[11:38:49] <Majavah>	 I'm not sure what's the best way to fix that
[11:39:01] <Krenair>	 though that still doesn't let me in under my user
[11:40:25] <Krenair>	 oh right it needs /etc/security/access.conf
[11:40:45] <Krenair>	 ok now I'm in as me, and you should be able to get in too
[11:43:05] <Majavah>	 yep, got in
[11:45:23] <Krenair>	 I wonder if we could make that access fix part of the bootstrap script
[11:46:24] <Krenair>	 you may want to put your key on the deployment-prep root list btw
[11:46:43] <Majavah>	 I think we're trying to get rid of that script
[11:46:59] <Majavah>	 and do everything with puppet and cloud-init
[11:49:42] <Krenair>	 well in cloud-init then
[11:52:22] <Majavah>	 can you run puppet on that vm? looks like the rules for that are missing too and I just set that hiera key on horizon to a temp value for now, will do a more permanent solution later today
[11:54:34] <Krenair>	 oh, no ability to sudo. sigh
[11:55:44] <Krenair>	 Majavah: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Cfssl::Config[client-cfssl]: parameter 'auth_keys' entry 'default_auth' entry 'key' expects a value of type Sensitive[String] or Pattern[/^[a-fA-F0-9]{16}$/], got String (file: /etc/puppet/modules/cfssl/manifests/client.pp, line: 26) on node deployment-urldownloader03.deployment-prep.eqi
[11:55:44] <Krenair>	 ad1.wikimedia.cloud
[11:56:57] <Majavah>	 what about now?
[11:58:10] <Krenair>	 Notice: Applied catalog in 29.07 seconds
[11:58:29] <Krenair>	 now it does the project-puppetmaster SSL issue :)
[11:58:32] <Majavah>	 thanks, works now
[11:58:50] <Majavah>	 I'll fix this mess in a moment
[14:07:19] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 61744950f0 (l10n updates)
[14:07:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[14:41:05] <MacFan4000>	 !help I’m trying to run composer update but the process gets killed every time
[14:41:05] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[14:41:15] <MacFan4000>	 https://www.irccloud.com/pastebin/VtIwfN3b
[14:57:23] <MacFan4000>	 If I run it not in the php7.3 cli then it tells me that I need php7.3 but in the php7.3 cli it’s getting killed every time I run it
[15:16:51] <Krenair>	 hmm
[15:17:07] <Krenair>	 skins-65cf875d79-vtpqv
[15:17:13] <Krenair>	 this is a k8s container in toolforge MacFan4000 ?
[15:17:31] <MacFan4000>	 Yes Krenair
[15:18:04] <Majavah>	 in my experience composer can get angry if it doesn't have tons of memory allocated to it, but usually it gives an error message suggesting that instead of just "Killed"
[15:19:47] <Krenair>	 krenair@tools-k8s-control-1:~$ kubectl get -n tool-skins pod skins-65cf875d79-vtpqv -o json | jq .spec.containers[].resources.limits.memory
[15:19:48] <Krenair>	 "512Mi"
[15:21:45] <Majavah>	 you can pass "-m 1Gi" or something similar to `webservice shell` to allocate more memory to it, up to the total namespace quotas
[15:22:16] <MacFan4000>	 To be clear, composer install works fine
[15:25:10] <MacFan4000>	 Using -m 1Gi worked, thanks
[18:40:28] <Leaderboard>	 Majavah: to clarify, for a MW instance SSL certs will be done automatically? Didn't quite catch your last comment
[18:43:09] <Majavah>	 Leaderboard: yes, unless you run something as complex as the beta cluster
[18:52:01] <wm-bb>	 <Alex> Beta cluster does public facing SSL certs automatically, like prod
[18:53:06] <Leaderboard>	 who is "wm-bb"
[18:57:08] <Majavah>	 bridge bot relaying telegram to here