[00:31:28] 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10ssastry) >>! In T249705#6045160, @Dvorapa wrote: > Occasionally also fails on mediawiki.org... [00:52:34] 10serviceops, 10MediaWiki-Page-derived-data, 10Performance-Team: Post-send DeferredUpdates unreliable due to fpm timeout before MW (missing edits in Watchlist and Recentchanges) - https://phabricator.wikimedia.org/T248564 (10Krinkle) [07:45:30] <_joe_> akosiaris: I was discussing with rlazarus what needs to happen to fix at least a bit our helmfile.d layout. AFAICT helm 3 fixed all the problems we had [07:46:20] <_joe_> so it would make possible to deduplicate those completely. I don't know how much work that would be [07:46:27] <_joe_> (the helm 3migration) [07:46:52] <_joe_> jayme: do you have experience with doing that transition by any chance? [07:47:53] <_joe_> for one I guess we'd have to re-do the RBAC setup we have [07:53:06] <_joe_> but in the spirit of not letting the perfect be the enemy of the good, I think I can spend a couple days trying to fix the current structure a bit. Make it at least suck less [07:53:28] <_joe_> so that we have services/// [07:53:53] <_joe_> and that we define most values under services//common_values.yaml [08:02:29] while I know people are here and paying attention, in case you didn't check the backscroll, the lowdown from greg on deployments is: "no train that week and no deploys on those days off. swats open for the non-off days though." [08:02:53] the deployment calendar will sometime soon be updated to reflect that. [08:09:51] <_joe_> apergos: oh great [08:10:49] yep, thanks apergos! [09:32:38] 10serviceops, 10Operations: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10Dzahn) 05Open→03Stalled stalled by T247018 [09:32:41] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) [11:07:51] _joe_: unfortunately not :/ Initially we wanted to wait some releases for things to settle, then the new owner took over and focus was on cutting costs [11:08:16] <_joe_> jayme: oh thanks, also I just remembered you're off today [11:08:19] <_joe_> sorry for the ping [11:08:38] np [11:11:34] "helm 3 fixed all the problems we had" sounds like a very familiar sentence btw. :) [11:12:04] really curious to see if that turns out to be true for anybody [11:56:37] <_joe_> well the bugs like helm-diff not honouring --kubeconfig AIUI at least :P [11:56:51] <_joe_> I'm sure it will create plentiful other issues [12:02:23] 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10Dvorapa) @Xqt Could we skip the whole flow_tests.py somehow? Or is there an option to skip t... [12:08:24] 10serviceops, 10Operations, 10Patch-For-Review: upgrade people.wikimedia.org backend to buster - https://phabricator.wikimedia.org/T247649 (10Dzahn) [12:18:50] 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10Xqt) The whole script can be skipped by adding code like the following in flow_tests.py: `... [14:11:58] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Mholloway) I'm trying to interpret what this task being closed as declined means. Does it mean that t... [14:25:02] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10WDoranWMF) hey @Mholloway, we are not porting Restbase to k8s so this becomes irrelevant. Restbase wil... [14:30:37] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Mholloway) Ah, I see. My interest is specifically in service-runner as my understanding is that it wi... [14:36:04] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Pchelolo) `service-runner` itself is not going anywhere. DHT-based rate limiting however is likely to... [14:38:24] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Eevans) >>! In T235437#6046252, @Mholloway wrote: > Ah, I see. My interest is specifically in service... [14:41:24] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Joe) >>! In T235437#6046252, @Mholloway wrote: > Ah, I see. My interest is specifically in service-ru... [14:45:40] 10serviceops, 10Operations, 10Services, 10service-runner, and 3 others: Move service-runner legacy rate limiter into hyperswitch - https://phabricator.wikimedia.org/T249919 (10Pchelolo) [14:46:07] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Pchelolo) Ok, to avoid further confusion, I will do T249919 [14:48:27] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10Mholloway) Thanks, everyone. The repo I'm working with is https://github.com/mdholloway/pushd, and ba... [14:54:12] 10serviceops, 10Prod-Kubernetes: Adjust our helm charts to support kubernetes 1.16 - https://phabricator.wikimedia.org/T249920 (10akosiaris) [14:54:21] 10serviceops, 10Prod-Kubernetes: Adjust our helm charts to support kubernetes 1.16 - https://phabricator.wikimedia.org/T249920 (10akosiaris) p:05Triage→03Medium [15:08:26] 10serviceops, 10Operations: dropped packets to echostore.svc.eqiad 8082/tcp - https://phabricator.wikimedia.org/T238789 (10akosiaris) [15:11:39] 10serviceops, 10Operations, 10Services, 10service-runner, and 3 others: Move service-runner legacy rate limiter into hyperswitch - https://phabricator.wikimedia.org/T249919 (10Pchelolo) p:05Triage→03Medium [15:24:10] 10serviceops, 10Operations, 10Kubernetes, 10User-fsero, 10User-jijiki: Support e - https://phabricator.wikimedia.org/T249927 (10akosiaris) [15:24:31] 10serviceops, 10Operations, 10Kubernetes, 10User-fsero, 10User-jijiki: Support kubernetes Egress networkpolicies in our helm charts - https://phabricator.wikimedia.org/T249927 (10akosiaris) p:05Triage→03Medium [15:26:02] 10serviceops, 10Patch-For-Review: restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10Pchelolo) Nothing to do here for CPT, untagging. [15:45:41] 10serviceops: Integrate kube-metrics-server into our infrastructure - https://phabricator.wikimedia.org/T249929 (10akosiaris) [15:48:51] 10serviceops: Integrate kube-metrics-server into our infrastructure - https://phabricator.wikimedia.org/T249929 (10akosiaris) [16:07:19] 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: Intermittent internal API errors with Flow - https://phabricator.wikimedia.org/T249705 (10cscott) [16:39:32] 10serviceops, 10Operations, 10Services, 10service-runner, and 3 others: Move service-runner legacy rate limiter into hyperswitch - https://phabricator.wikimedia.org/T249919 (10Pchelolo) 05Open→03Declined Actually, after reviewing the code once more, this doesn't seem to be feasible. rate limiter in ser... [21:54:30] 10serviceops, 10Operations, 10Parsing-Team, 10Performance-Team, and 4 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10aaron) >>! In T244058#6041722, @daniel wrote: > Another though from the TechCom meeting: we could...