[15:37:08] phuedx: change is deployed, do you need me to run it on any physical machine? [15:38:03] that'll need a helmfile apply following https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Deploying_periodic_jobs [15:38:14] phuedx: jynus: for that patch to take effect (1) puppet needs to run on the active deployment host and (2) mw-cron needs deployed. [15:38:22] ^ that [15:38:32] I will do the deploy1003 run [15:38:48] not there [15:38:50] on codfw [15:39:32] ah, thanks jynus! I can take the mw-cron deployment when you're done. [15:39:43] ok [15:39:53] ongoing on deploy2003 [15:40:06] I am a bit rusty with mw since the k8s conversion [15:40:25] we were not told of those changes in the training [15:40:38] done on deploy2003 [15:40:54] (the puppet run) [15:41:58] deploy2003 will run rather quickly, since it's insetup. the active deployment host is deploy2002, where a puppet run will take 5-6 minutes [15:42:07] oh [15:42:58] doing that instead, I didn't look at the roles as I just thought the logo will tell me I was on the right one [15:43:16] I was wondering why I didn't see any applied change [15:44:10] swfrench-wmf: would you mind guiding me on the help deploy instead? [15:44:14] we were not told of those changes in the training < the training was specifically not about mw-on-k8s [15:44:29] I wasn't complaining :-D [15:44:37] jynus: I'm happy to take care of it - I've got a shell open anyway :) [15:44:43] swfrench-wmf: I don't plan to do many of those [15:44:55] but would love to learn in case of an emergency [15:45:18] jynus: https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Deploying_periodic_jobs [15:45:21] disabling a mw job run is something that I think everyone should know [15:45:27] but with codfw instead of eqiad [15:46:31] while deploying some new stuff is usually not done under stress [15:47:56] jynus: so, the docs claime referenced should cover this. if you want to share your commands with me here before you run them, I'm happy to review them. alternatively, I can take it from here :) [15:48:29] and to calm claime, I don't intend to apply to a serviceops position any time now 0:-) [15:48:46] Did I give the impression I was angry ? :D [15:49:04] no, maybe just worried :-) [15:49:28] swfrench-wmf: puppet finished [15:50:07] swfrench-wmf: screen -S deploy_mw_job ; cd /srv/deployment-charts/helmfile.d/services/mw-cron [15:52:07] then: helmfile -e codfw -i apply --context 5 [15:52:14] can I do it as my user? [15:52:19] yes [15:52:32] -operations looks clear [15:52:46] how long will it take? [15:52:55] jynus: so, that should be pretty quick. [15:53:04] you will be presented with a diff showing yaml being removed [15:53:04] ok to go then? [15:53:08] ok [15:53:39] if should say experimentationlab-updateconfigs and have type CronJob (up near the top) [15:53:44] *it should [15:53:55] ok, will confirm after prompt [15:54:13] still waiting for your ok to proceed [15:54:29] jynus: is that what you see? ^ (what I described above0 [15:54:58] havent enter'ed yet, waiting for a clear "yes, please execute the above command" [15:55:02] as instructed [15:55:26] ah, got sorry - yes, good to go on `helmfile -e codfw -i apply --context 5` [15:56:12] that will present you with the diff I described, and present you with a confirmation [15:56:16] (that's the `-i` part) [15:56:26] it is presenting the full tempalte for cronjob: "experimentationlab-updateconfigs", as a deletion [15:57:03] and the only addition is an empty line [15:57:16] jynus: perfect, that's exactly what I would expect from that patch [15:57:20] afected releases: main (wmf-stable/mediawiki) UPDATED [15:57:41] change looks expected to me [15:58:12] ok to y? swfrench-wmf [15:58:37] great, yes as long as that's the only object removed, you should be good to 'y' :) [15:59:21] yes, just name: experimentationlab-updateconfigs [15:59:24] (aside: I've confirmed in parallel that `helmfile -e codfw diff --context 5` shows only the expected diff) [15:59:27] hitting y [15:59:31] * swfrench-wmf thumbs up [16:00:12] no errors [16:00:14] main mw-cron 1139 2026-03-10 15:59:38.126072162 +0000 UTC deployed mediawiki-0.12.1 [16:00:29] any other thing to check? [16:00:54] aside from log checks from phuedx, on the admin side? [16:01:12] You can do `kube-env mw-cron codfw; kubectl get cronjob experimentationlab-updateconfigs` [16:01:16] Should find nothing [16:01:25] jynus: Not on my side I don't think :) [16:01:26] claime: thanks, that's exactly what I wanted [16:01:26] ty [16:01:28] * swfrench-wmf is slower than claime at typing [16:02:22] mmm, not sure I want to be part of the restricted stuff [16:03:40] wdym restricted? [16:04:04] which kube-env returns 404 [16:04:13] I supposed it required restricted right [16:04:34] Huuuuh [16:04:42] kube-env is a bash function [16:04:44] It doesn't [16:04:58] that's how it manipulates your environment [16:05:03] Oh, you did `which kube-env` [16:05:10] Yeah that won't work as swfrench-wmf said [16:05:23] still, the original "bash: kube-env: command not found" [16:05:26] Do you have a weird shell setup? [16:05:32] it could be [16:05:47] jynus: your shell hasn't sourced /etc/profile or /etc/profile.d (forget which) [16:06:08] interesting [16:06:10] source /etc/profile.d/kube-env.sh and it will work [16:06:39] sh, I know why [16:06:44] *ah [16:07:02] it should work normally I think [16:07:38] yep, it was my fault (manually) not my bashrc profile [16:08:09] Error from server (NotFound): cronjobs.batch "experimentationlab-updateconfigs" not found [16:08:31] in my time we edited mw cronjobs manually :-D, much easier LUL [16:09:03] again, don't worry, I won't be touching this much, I just wanted to try in a controlled env [16:09:10] just in case there was an emergency [16:09:18] thanks for giving it a try! :) [16:09:42] thanks swfrench-wmf claime cdanis and phuedx for the patience to deal with an old man like me [16:10:05] feel free to ask me about editing the live mw database next time 0:-D [16:10:32] mm, there was a job error on -operations [16:10:55] ah, that is a prometheus one, I think [16:11:15] unrelated, I got worried for a second [16:12:04] * swfrench-wmf hopefully does not find opportunity to have to live-edit databases [16:12:15] indeed, that alert is unrelated [16:13:23] I have to say, you serviceops have really nice docs