[08:32:56] * volans back
[08:44:28] <godog>	 greetings, and happy 2026
[09:11:15] <dhinus>	 happy new year! :)
[09:11:51] <volans>	 indeed
[09:19:01] <taavi>	 hello :P
[09:24:36] <volans>	 taavi: I might mis-remeber, so please correct me if I'm wrong. Am I in clinic duty now and you've covered a bit the last couple of days?
[09:28:37] <taavi>	 volans: sounds correct to me. I think I've cleared all the backlogs except T413097 which is a bit interesting
[09:28:37] <stashbot>	 T413097: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097
[09:29:14] <volans>	 <3 thanks for the backlog!
[09:29:46] <volans>	 I'm not too familiar with our standard limits but 256 seems a bit high
[09:30:52] <godog>	 indeed, also I'm curious about the general context/strategy for the task/project, even with a 256gb instance then what's the plan?
[12:24:34] <taavi>	 I also have a bunch of small cookbook patches from yesterday's toolsbeta upgrade starting from https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1223633
[12:44:15] <godog>	 looking
[12:53:17] <taavi>	 thanks
[12:53:37] <godog>	 sure np
[12:56:16] <godog>	 phew almost done with the puppet/phab backlog
[13:10:43] <godog>	 andrewbogott: re: new cloud hw coming in, e.g. in T412568 any reason not to go with uefi as opposed to bios ?
[13:11:53] <andrewbogott>	 uefi should be fine if there are already uefi partman configs for them
[13:12:09] <godog>	 yes standard recipes come with uefi support
[13:12:23] <volans>	 see also the email from luca: [sre] UEFI boot is now the default for new servers
[13:12:23] <godog>	 i.e. the -efi equivalent
[13:12:32] <andrewbogott>	 oh, we don't have to specify a uefi recipe anymore?
[13:12:33] <andrewbogott>	 oh yeah
[13:12:59] <andrewbogott>	 I have a gut reaction against splitting up a group of servers but that's going to happen eventually.
[13:14:11] <godog>	 indeed might as well take the hit now while we're at it
[13:14:31] <andrewbogott>	 yeah, that's fine with me
[13:17:09] <godog>	 ok! I'll do the switch
[13:18:28] <andrewbogott>	 thanks!
[13:19:13] * volans lunch
[13:21:32] <godog>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1224070
[13:23:15] <andrewbogott>	 lgtm. We also need to change the 'Boot Method' on the related tickets
[13:24:48] <godog>	 indeed, would you mind doing that? I'll reimage the hosts
[13:27:03] <andrewbogott>	 done
[13:28:33] <godog>	 thank you, I didn't realize the hosts are not installed yet heh
[13:28:39] <godog>	 even easier
[13:29:33] <andrewbogott>	 yeah, dcops has been asking for racking info earlier and earlier in the process :)
[13:32:37] <andrewbogott>	 A few years ago we tried to order some nvme cephosd nodes to see if they were fast enough to host etcd... the project stalled because Dell couldn't build us an nvme host with bios boot. Probably we should give that another go next hardware budget cycle.
[13:34:35] <godog>	 SGTM
[13:50:27] <taavi>	 any opposition to me scheduling the tools k8s 1.31 upgrade for next Wednesday?
[13:51:38] <godog>	 +1
[13:54:25] <dhinus>	 +1
[13:55:07] <taavi>	 https://etherpad.wikimedia.org/p/toolforge-k8s-1.31
[13:57:16] <dhinus>	 I removed a double "cluster" on the first line, the rest looks good. maybe add a link to the phab?
[13:59:06] <taavi>	 done
[14:07:10] <taavi>	 dhinus: any more concerns before I send it?
[14:13:07] <dhinus>	 go for it!
[14:14:40] <taavi>	 sent
[16:00:54] <paladox>	 oh wow only just noticed that arturo is gone.
[16:10:03] <andrewbogott>	 yeah, it's been maybe six months now?
[16:10:38] <andrewbogott>	 taavi: I'm trying to reproduce your missing images with my CSP but unsure I'm doing things right.  Does the thumbnail on https://wikitech-static.wikimedia.org/wiki/CDN.html display for you?
[16:10:49] <andrewbogott>	 captioned "This diagram is for the "text" cache cluster. The "upload" cluster is similar. "
[16:12:41] <taavi>	 andrewbogott: nope, fails with a 404
[16:13:28] <andrewbogott>	 what am I doing wrong here?
[16:13:30] <andrewbogott>	 https://www.irccloud.com/pastebin/IXQBIHmx/
[16:14:18] <taavi>	 honestly, I'm not sure if that's a CSP issue, the images seem to just straight up be missing from the container image
[16:15:13] <andrewbogott>	 I would like to reproduce the issue you're seeing though...
[16:15:20] <andrewbogott>	 https://wtsbuild.wmcloud.org/wiki/CDN.html  <- thumbnail displays
[16:15:39] <andrewbogott>	 I'm sure you're right that it's missing, I just want to see what you're seeing
[16:19:47] <taavi>	 that works for me as well
[16:21:13] <taavi>	 andrewbogott: the source of https://wikitech-static.wikimedia.org/wiki/CDN.html has `src="../../upload.wikimedia.org/wikipedia/commons/thumb/7/76/WMF_Inbound_Text_Traffic_Diagram.svg/250px-WMF_Inbound_Text_Traffic_Diagram.svg.png"
[16:21:13] <taavi>	 srcset="//upload.wikimedia.org/wikipedia/commons/thumb/7/76/WMF_Inbound_Text_Traffic_Diagram.svg/500px-WMF_Inbound_Text_Traffic_Diagram.svg.png 1.5x"`, meaning it'll try to use the local file, except if the browser supports high-res versions it'll try to use upload.wikimedia.org
[16:21:22] <taavi>	 which probably explains why it works for you but not me
[16:23:34] <andrewbogott>	 oh, that makes sense.
[16:24:12] <taavi>	 but it doesn't explain why https://wtsbuild.wmcloud.org/upload.wikimedia.org/wikipedia/commons/thumb/7/76/WMF_Inbound_Text_Traffic_Diagram.svg/250px-WMF_Inbound_Text_Traffic_Diagram.svg.png exists but
[16:24:12] <taavi>	 https://wikitech-static.wikimedia.org/upload.wikimedia.org/wikipedia/commons/thumb/7/76/WMF_Inbound_Text_Traffic_Diagram.svg/250px-WMF_Inbound_Text_Traffic_Diagram.svg.png does not
[16:25:07] <andrewbogott>	 oh, that I can explain, wtsbuild is a different scrape.
[16:25:35] <andrewbogott>	 Which fixed copying over the things under upload.wikimedia.org
[16:26:35] <andrewbogott>	 But since the image works for me on wikitech-static-wikimedia.org the two sites behave the same for me. You're saying that wtsbuild is an improvement, from your POV?
[16:27:53] <taavi>	 try this: https://wikitech-static.majava.org/wiki/CDN.html (with the username/password I'll send you in PM), that's the exact same content as wikitech-static.wikimedia.org has but with all direct HTTP connectivity to wikimedia sites blocked with a CSP
[16:29:20] <taavi>	 you should see no thumbnail, because the CSP is blocking the request to pull the image from upload.wikimedia.org
[16:29:59] <andrewbogott>	 indeed I do not
[16:30:06] <taavi>	 see the problem now?
[16:30:24] <andrewbogott>	 yes.
[16:30:31] <taavi>	 i.e. it only works on wikitech-static.wm.o for you because it still has config to pull it from upload.wikimedia.org, which we don't want during an outage
[16:31:07] <andrewbogott>	 Yep, I didn't doubt you. I just want to reproduce the CSP on my test/dev so I can easily see when the issue is fixed.
[16:31:19] <andrewbogott>	 How are you adding the CSP to your site?
[16:32:19] <taavi>	 this is the CSP I'm using: `Content-Security-Policy "default-src 'none'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' 'unsafe-eval'"`
[16:32:45] * andrewbogott tries
[16:33:33] <taavi>	 so the trick is that I'm not actually using the published container, instead I have a script that extracts the files from the container to local disk and then serves them directly with plain apache2. but you could just as well add the CSP with a reverse proxy, or by modifying the nginx config running in the container
[16:34:09] <andrewbogott>	 "modifying the nginx config running in the container" is the paste that I started this conversation with :)
[16:35:09] <andrewbogott>	 there we go!  https://wtsbuild.wmcloud.org/wiki/CDN.html
[16:39:01] <andrewbogott>	 I think I don't know how relative urls show up in nginx after a click.  Will $uri = ../../upload.wikimedia.org... during the image load, or is it normalized somehow?
[16:39:10] <andrewbogott>	 It has to be normalized, because otherwise... relative to what?
[17:06:09] * dhinus off