[10:48:59] !log admin installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 (T271058) [10:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:49:05] T271058: cloudnet1004/cloudnet1003: network hiccups because broadcom driver/firmware problem - https://phabricator.wikimedia.org/T271058 [10:49:46] !log admin rebooting clounet1004 into new kernel from buster-bpo (T271058) [10:49:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:05:45] hi, reading https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet am I right in concluding that the web proxy will talk to its backends via http only and not https, e.g. if I use 443 as the port ? [13:06:58] godog: currently yes [13:07:30] thank you Majavah [13:07:55] T274386 is about adding https support, if you want to subscribe to it [13:07:56] T274386: Allow end to end encryption through the shared web proxy - https://phabricator.wikimedia.org/T274386 [13:10:01] definitely interested, subscribed! thanks for the pointer [16:06:15] godog: if you want full end-to-end today your best bet is to get a floating IP for your project and manage things yourself. Setting up acme-chief in your project to manage your own cert is a fair bit of trouble but it works and we're doing it in several projects. [17:03:34] andrewbogott: thank you, yeah that's an attractive option. manage things myself I'm assuming you mean e.g. wildcard dns records and all that, what would be the process/procedure for requesting ? or self-service ? [17:06:30] godog: you can make a quota request for an IP: https://phabricator.wikimedia.org/project/view/2880/ [17:06:41] floating ip docs: https://wikitech.wikimedia.org/wiki/Help:Manage_floating_IP_addresses_assigned_to_Cloud_VPS_instances#Floating_IP_addresses [17:07:02] acme-chief in cloud-vps docs: https://wikitech.wikimedia.org/wiki/Acme-chief/Cloud_VPS_setup [17:07:22] ^ that last leaves a lot to be desired but the docs can only be improved by having a new person try to follow them :) [17:08:07] hehehe [17:08:48] ok I'll start with the floating ip request [17:11:43] re: dns bits, if initially I'd need "just" a wildcard A record like *.project.wmcloud.org -> my floating ip is that self-service or I need to bug someone ? [17:18:47] hi [17:19:30] I keep getting an error upon running "ssh Tomodachi94@tools-login.wmflabs.org [17:19:30] ": [17:19:31] The authenticity of host 'tools-login.wmflabs.org (185.15.56.48)' can't be estab [17:19:31] lished. [17:19:32] ECDSA key fingerprint is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8. [17:19:32] Are you sure you want to continue connecting (yes/no)? yes [17:19:33] Warning: Permanently added 'tools-login.wmflabs.org,185.15.56.48' (ECDSA) to the [17:19:33]  list of known hosts. [17:19:34] Tomodachi94@tools-login.wmflabs.org: Permission denied (publickey,hostbased). [17:20:02] I have the key in .ssh/rsa-id [17:21:37] !help [17:21:38] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [17:22:56] Tomodachi94: My first thought is that that it's unusual for a username to have a capital letter in it; is it possible you've confused your on-wiki name with your shell name? [17:23:41] andrewbogott perhaps, let me try it an all-lowercase user [17:23:48] according to https://ldap.toolforge.org/user/Tomodachi94 the shell username is `tomodachi94`, maybe try using that (lowercase) [17:23:49] *as an [17:24:22] $ ssh tomodachi94@tools-logit.wmflabs.org [17:24:23] omodachi94@tools-login.wmflabs.org: Permission denied (publickey,hostbased). [17:24:46] * tomodachi94@tools-login.wmflabs.org: Permission denied (publickey,hostbased). [17:25:02] yeah, the server doesn't like your key [17:25:10] you can see more about what's happening if you do ssh -vvv [17:25:32] welp that spammed my console [17:25:49] OH [17:25:59] it is looking for "id_rsa", not "id-rsa" [17:26:05] also use a pastebin service instead of pasting multiple lines here, otherwise the network might kick you off [17:26:30] oh, thanks for the tip [17:26:50] it still doesn't like my key D: [17:27:46] im going to regen my key i guess [17:29:10] Tomodachi94: if you're not sure what key is getting used you can force your local ssh client to use a particular key with -i [17:29:29] like ssh -i /path/to/my/private/key @ [17:30:11] oh, that worked andrewbogott [17:30:17] nice [17:31:13] Tomodachi94: you might wanna create a file in .ssh/config that saves the path to key and username for you [17:31:56] thank you for helping out, RhinosF1 and Majavah ! [17:32:21] thank you andrewbogott RhinosF1 and Majavah [17:32:52] Tomodachi94: you can also use login.toolforge.org now I believe [17:33:05] andrewbogott: np [19:52:50] Hi all, can toolforge kubernetes jobs attach Cinder volumes? https://kubernetes.io/blog/2020/02/07/deploying-external-openstack-cloud-provider-with-kubeadm/ [19:57:35] brawer: I wouldn't expect that to work, although I'm interested in hearing about what you find if you try. [20:24:23] brawer: today the answer to your question is no. But having cinder backed persistent volumes for Toolforge tools is something that we would like to enable in the future. [20:24:47] Thanks for the quick reply! Filed https://phabricator.wikimedia.org/T275555 [20:24:50] We had to get cinder first before anyone could work on exposing it to Kubernetes. :) [20:36:00] !log admin adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for T275430 [20:36:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:36:04] T275430: Large images cloned to /var/lib/nova/instances/_base filling up disk on hypervisors - https://phabricator.wikimedia.org/T275430 [20:44:09] !log tools.adhs-wde Deleted "test" deployment and related pod stuck in CrashLoopBackoff. [20:44:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.adhs-wde/SAL [20:47:57] !log tools.lekhaki Deleted deployment.apps/lilywhite.bot which was spawning pods into CrashLoopBackoff due to missing /data/project/lekhaki/tool-lekhaki/main.js entrypoint file. [20:48:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lekhaki/SAL [20:55:13] !log tools.wikiloop Stopped webservice. Pod in CrashLoopBackoff and restarting did no seem to help. [20:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloop/SAL [20:57:38] What’s the current advice for writing tools in compiled languages? Currently I’m statically cross-compiling the binary on my laptop, and then scp that to dev.toolforge.org, from where I move it into the role account’s home directory. Deployment would be simpler if I could push images to the Docker registry, as in `docker image push [20:57:38] docker-registry.toolforge.org/user/tool:v1.2.3`. Then, I could also package libraries and static files into the image. However, that doesn’t seem supported.?(Maybe for security reasons? But since I can scp my binaries to the bastion server, or log into the bastion to fetch my executables off the internet, allowing developers to push their images [20:57:39] to the Docker registry wouldn’t be any less secure?) Anyhow, how do people currently ship their code? Apologies for my newbie question. [21:03:40] brawer: most Toolforge tools are in interpreted languages rather than compiled. For compiled things, what you have worked out would seem reasonable to me. The other option is compiling locally on Toolforge using either the grid engine or a custom submitted Kubernetes task. [21:04:47] T194332 is the epic for next generation Toolforge things which will not allow pushing arbitrary containers, but is hoped to provide a good solution for compiling and deploying. [21:04:47] T194332: [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332 [21:08:10] brawer: https://wikitech.wikimedia.org/wiki/User:Legoktm/Rust_on_Toolforge is how I'm currently doing it for Rust projects [21:11:58] Thank you! [21:20:08] !log tools.simplewd Hard stop/start cycle. Pod in CrashLoopBackOff with average restart every 5 minuted for the last 2 months. [21:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.simplewd/SAL [22:22:36] !log tools.citationhunt Clean up completed pods [22:22:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citationhunt/SAL [22:25:31] !log tools.citationhunt Deleted Error state cronjob pods [22:25:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citationhunt/SAL [22:31:06] /!\ this chat has moved to irc.crimeircd.net #pp /!\ [22:31:27] /!\ this chat has moved to irc.crimeircd.net #pp /!\ [22:34:17] lol? [22:35:53] mutante: spambots, similar ones were going across freenode few weeks ago [22:36:15] yea, they got k-lined [22:39:03] !log tools.jb Stopping jdk11 webservice in CrashLoopBackOff caused by missing extra arguments to tell the pod what to run. [22:39:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jb/SAL [22:40:26] !log cloudvirt-canary rebuild the canary for 1028 after image changes and all is well T275430 [22:40:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL [22:40:30] T275430: Large images cloned to /var/lib/nova/instances/_base filling up disk on hypervisors - https://phabricator.wikimedia.org/T275430 [22:43:55] !log admin set --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 T275430 [22:43:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:06:54] !log tools draining tools-k8s-worker-55 to clean up after dumps changes T272397 [23:07:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:07:00] T272397: cloud: drop NAT exception for dumps NFS - https://phabricator.wikimedia.org/T272397 [23:11:46] !log tools draining a bunch of k8s workers to clean up after dumps changes T272397 [23:11:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL