[00:23:17] alas, the images did not build due to a bug in kokkuri but i have a fix. i will retry the build tomorrow morning (UTC-7) [00:28:04] I'm making progress on T396936 too. I now have a tiny cluster, a bastion, and a web proxy setup. I need to figure out how to put credentials for the Kubernetes cluster on the bastion next. [00:28:04] T396936: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936 [12:38:07] SUCCESS I had a first quibble build passing (qunit): https://zuul-dev.wmcloud.org/t/wikimedia/build/2e13672fcdfd4d4ba68ec123eceac5dc/console#1/0/7/quibble [12:40:12] fun thing the web interface tries to fetch a non existing `job-output.json.gz` [13:27:51] yeah, we generally try to use gzip content-encoding when uploading to log storage now, but there's some code in there to handle explicitly compressed logs if they exist [13:31:24] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_s3_upload.py#L196 is where we silently gzip text files when uploading logs. so you shouldn't need to change anything, you should already be benefitting from that. [16:45:27] I added details about setting up the containers in https://phabricator.wikimedia.org/T395938 -- the command line is easy; the extra info is mostly about setting up bind mounts and port mappings [16:52:14] dduvall: some quibble playbook is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1159468 and the dumb for gerrit-ping is https://gerrit.wikimedia.org/r/c/test/gerrit-ping/+/1152814 [16:52:29] dduvall: commenting "retest" should get Zuul to reenqueue them in the `check` pipeline [16:58:44] zuul images based on upstream Dockerfile + wmf base have been built and published, i.e. docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-executor:wmf-12.0.0-4 [16:59:17] i just reconfigured zuul-1001.zuul3.eqiad1.wikimedia.cloud to use them [16:59:44] so far nothing catastrophic in the logs following a restart [17:02:11] oh joy I found a bug in firefox inspector :] [17:02:47] corvus: let me know what you think of https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/commit/e319cfb42901e24932ec345f1d68a3e2e9223a49 and its potential for acceptance upstream. i can simplify it further to only introduce a base stage for the final zuul targets (`zuul-base`) if you think that's best, since the only stage we're overriding in our `wmf-bake.hcl` is `zuul-base` [17:02:54] i just released 12.1.0 upstream, btw. i don't think there's any urgent need to use it... maybe just bump up to it whenever it's convenient. [17:03:06] will do [17:04:14] the object storage URL does not show any icons cause we have a CSP policy on it, but looking at it in the firefox inspector does show the icons, I guess cause it does not honor the CSP [17:04:14] * hashar https://phabricator.wikimedia.org/F62379119 [17:07:14] dduvall: the only reason i could think not to accept that would be if it has some kind of performance degradation, and i don't expect it to. honestly, i think it looks even better than current because it's sort of like an import or include block in a program where we put all the dependencies together. i'd +2 that. :) [17:07:52] nice! [17:19:33] hashar: looks like the job for my gerrit-ping test is waiting on a node https://zuul-dev.wmcloud.org/t/wikimedia/status/pipeline/check [17:19:42] perhaps the newly restarted launcher is having trouble [17:23:02] the "nodes" are at https://zuul-dev.wmcloud.org/t/wikimedia/nodes [17:23:27] which really are containers in pod in the microk8s I have setup [17:23:29] corvus: perhaps a dumb question, but what is the difference between `nodepool-launcher` and `zuul-launcher`? the new image i built has the latter as the entrypoint but the current `docker-compose.yaml` executes the former [17:23:32] (well setup, I just started it) [17:26:57] * dduvall reads https://zuul-ci.org/docs/zuul/latest/developer/specs/nodepool-in-zuul.html#introduction [17:29:26] dduvall: pretend zuul-launcher doesn't exist yet. it's a work in progress, and not ready for production use. [17:29:35] (that is the correct place to learn about it though) [17:30:04] got it :) [17:30:07] once it's ready for use, we'll write documentation for it and recommend that people start using it instead of nodepool-launcher [17:30:50] it's designed to operate along side nodepool-launcher, so there will be a trasition period where you can run both (so you don't have to do a hard cutover if you don't want; or you can if you do want) [17:31:06] currently opendev is experimentally running it for some tenants -- zuul and opendev included. but not openstack. [17:31:33] we're getting pretty close; but having said that, i did just shut done of our two zuul-launchers off because of an error [17:33:05] * dduvall nods [17:33:19] that illustrates why we're developing it like this; we want it to be really solid in opendev before anyone else uses it [17:33:38] (also, it currently only has aws and openstack drivers; we haven't ported over k8s or static yet) [17:35:23] looks like we'll probably need to build a wmf-based nodepool image as well. maybe i'll focus on submitting the upstream Dockerfile changes. that way we won't really have to maintain forks of the zuul/nodepool git repos. we can just have a single image configuration repo for both projects [17:35:25] reading back, i have no idea what it means to "shut done off" but i'm glad you figured it out. :) [17:36:05] i'm blind to typos these days (see my typos) :) [17:36:35] yeah, sorry it's not ready in time for you to skip nodepool; that would have been nice. [17:40:32] I am off for dinnner! [17:41:05] the index.html does not show up icons cause of a content security policy ( https://phabricator.wikimedia.org/T397351 ) . Easy fix :) [17:41:18] then most people will just look at the Zuul web interface [17:42:03] yeah, we're trying to make the web interface the primary (one day, only) way of interacting with the logs [18:25:53] well the icons are showing now https://object.eqiad1.wikimediacloud.org/swift/v1/AUTH_a3598983742448b3b056b5fcb228faa9/artifacts/2e1/wikimedia/2e13672fcdfd4d4ba68ec123eceac5dc/index.html [18:26:11] but that system puppet manifest has a documentation stating it is not intended for direct human consumption [18:31:10] corvus: submitted https://review.opendev.org/c/zuul/nodepool/+/952859 and https://review.opendev.org/c/zuul/zuul/+/952854 for your consideration :) [18:36:43] dduvall: thanks; easy +2s, but i left a nitpick-level comment on the zuul one [18:59:52] corvus: fixed it up. thanks! [19:00:20] * dduvall lunches [20:46:04] thinking out loud, when looking at a build inventory we got the list of hosts and the label they come from [20:46:32] eg https://zuul-dev.wmcloud.org/t/wikimedia/build/356ff7ea96274745921830f473ceff15/log/zuul-info/inventory.yaml has: [20:46:32] all.hosts.nodepool.label: quibble-bullseye-php81 [20:47:22] and I wonder whether it could get the image that was used ( https://zuul-ci.org/docs/nodepool/latest/kubernetes.html#attr-providers.[kubernetes].pools.labels.image ) [20:51:07] hashar: in theory, but i think the right way to do that would be by sending structured data from nodepool to zuul, and i'm reluctant to suggest doing that since the mechanism for that in zuul-launcher is so different. so i'd suggest trying to work with what's there for now (maybe use regexes or lookup tables, etc), and then once zuul-launcher is further along, we can bring more of that information [20:51:13] over. [20:51:21] (with zuul-launcher, the labels will be defined in zuul as well, so it will already have all that information) [20:51:34] that is the nodepool-in-zuul work isn't it ? [20:52:34] yep [20:52:47] nice :]