[09:30:02] Raine: are you around for the deployment? I'm doing a final pass on the patches... [09:42:37] jayme: would you be able to help me test the liftwing ratelimit stuff? I have a patch, but I don't know how to make sure it's doing the right thing. There's no traffic on the liftwing endpoints on the rest-gateway yet, right? so can I just deploy the limits, and you test the endpoints before re-raouting traffic? [09:42:48] The patch is here 8rebase coming in a few minutes): https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1272765/12 [09:48:52] i can likely help with that, if i understand correctly, c.laime included testing instructions in T422804 [09:54:31] ah, cool. thanks bjensen. lmk if you need help [09:57:27] bjensen: excellent, thank you! is this testable in staging, or do you want me to folly deploy first? I have a few other things to push out first, I should get there in about an hour or so. [09:58:05] thank you bjensen <3 I'm around too [09:58:05] I think it should be testable in staging, I'm happy to attempt that, anyway :) [09:59:18] ok, i'll ping you when I'm about to merge that patch. [10:00:03] sounds good, thanks [11:09:35] bjensen: I'm getting ready to deploy the liftwing limits to staging. wrt testing, I'm particularly interested to make sure I didn't break the edit-check:predict path. I had to move it to a separate endpoint, since it *shouldn't* get the relaxed limits. [11:12:16] alright, thanks for the heads up [11:16:15] just ran all the tests (all succeeded, no surprises), so we have a baseline [11:19:01] bjensen: ok, i have deployed to staging, if you want to test there. [11:19:21] duesen: testing now [11:19:29] ty [11:20:19] looks good, including edit-check:predict [11:20:35] oh wait sorry, one moment [11:26:48] i think i failed to point to staging correctly, need to verify the port [11:27:44] bjensen: should be https://staging.svc.eqiad.wmnet:4113 [11:27:55] that's what I use for my system tests when deploying [11:31:37] hm, then i'm not sure this is working... would it be possible to revert staging for a moment? [11:34:12] bjensen: want me to merge a revert, or just revert the deployment? [11:34:37] I can do the latter by checking out the previous commit in my home dir, and applyoing to staging from there. [11:34:46] duesen: just revert the deployment, i think [11:36:26] ok done. [11:36:32] prod is now out of sync with master [11:37:13] in both circumstances, i'm seeing a 404 when curling against staging, so i'm not sure i'm testing this correctly [11:39:54] maybe it just doesn't work on staging. quite a few backend services are not configured right there. [11:40:11] we could deploy to codfw and test there. if it doesn't work, do a proper revert. [11:40:50] that's a fair point [11:41:39] ah, that sounds reasonable to me as well [11:42:35] let's proceed that way [11:42:50] ok, dpeloying to codfw [11:44:20] ok done [11:45:33] I have to take my dog outside. I'll be back soon though, the weather is terrible [11:45:36] okay, now we're good, i think [11:46:15] then I'm quickly running for coffee before the weather gets terrible here too :D [11:51:34] duesen: tests are complete in codfw, and things look good [11:52:00] did we bring staging back in sync? [12:01:30] ah, excellent. [12:01:47] staging and eqiad are out of sync with master. i'll deploy to staging first, then eqiad [12:02:30] thanks! [12:08:04] staging done [12:12:15] bjensen: i applied to staging as well. can you double check that it's working? [12:13:04] duesen: doing so now [12:14:17] duesen: behaviour seems the same before and after the patch [12:16:05] excellent! [12:16:16] Does that mean you will start to re-route traffic soon? [12:17:06] indeed! i'd like to get one endpoint done tomorrow, and do the majority of the migrations next week [12:19:02] sweet! [12:21:52] awesome sauce