[00:42:43] this looks relevant: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/citoid/+/refs/heads/master/.pipeline/blubber.yaml#27 [00:43:33] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/zotero/+/refs/heads/master/.pipeline/blubber.yaml#25 [00:58:40] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/zotero/+/013b6cf2eae47ff1bf6b87ead16180abf9193a53%5E%21/#F0 [00:58:54] which was after the timestamp on 20181019165254-production [00:59:09] but there is no other production build at https://tools.wmflabs.org/dockerregistry/wikimedia/mediawiki-services-zotero/tags/ [00:59:17] so what is prod running? [01:31:45] picked the latest candidate image instead for zotero, it now works but citoid is still not happy with it. not sure why [01:36:35] that seems to be because it is trying to talk to zotero over localhost by the built-in config.yaml instead of our one [01:37:10] just gonna file everything as bugs [01:40:52] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Citoid container: Our config.yaml provided via Docker is unused? - https://phabricator.wikimedia.org/T223344 (10Krenair) [01:43:53] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Zotero container: Latest production container lacking ca-certificates package - https://phabricator.wikimedia.org/T223345 (10Krenair) [01:49:55] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) I've opened a couple of subtasks and T223346. Considering the Citoid service on sca01 is probably already br... [02:20:12] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Krenair) May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal world, the teams developing new services would... [02:36:02] 10serviceops, 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) (Citoid migration commit is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510290/) [02:37:36] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Cherry-picked the above commit on deployment-puppetmaster, VE-Citoid integration appears to work again (thou... [02:43:24] 10serviceops, 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) Removed citoid role from them, and now: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find clas... [03:33:27] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Citoid container: Our config.yaml provided via Docker is unused? - https://phabricator.wikimedia.org/T223344 (10Krenair) Maybe I can set override_cmd to handle this [04:40:23] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Citoid container: Our config.yaml provided via Docker is unused? - https://phabricator.wikimedia.org/T223344 (10Krenair) yep... Do wonder if service::docker should handle this somehow though. [04:46:25] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) `krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker run -p 8080:8080 -v /etc/mediawiki-services-cx... [05:16:02] _joe_: good morning :] We have a request to add a new namespace to the docker registry 'dev' [05:16:22] that is intended to publish images for the local development tooling project (a replacement of mediawiki/vagrant) [05:16:55] with images genearted by docker-pkg from releng/dev-images.git [05:16:56] https://phabricator.wikimedia.org/T223329 [05:17:14] seems we need those images to be made available lfor the hackathon [05:19:20] <_joe_> hashar: hi, I would guess you'd need new separated credentials, right [05:19:31] possibly yes [05:19:34] <_joe_> let's wait for fsero to be around [05:20:25] sounds good, i will poke it after I had breakfast with family etc ;) [06:03:13] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Zotero container: Latest production container lacking ca-certificates package - https://phabricator.wikimedia.org/T223345 (10Joe) It's working in production because we connect to external UR... [06:53:37] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10WMDE-leszek) >>! In T220402#5183351, @Krenair wrote: > May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal... [07:24:28] 10serviceops, 10Operations, 10RESTBase, 10RESTBase-API, and 4 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac) [07:25:05] 10serviceops, 10Operations, 10RESTBase, 10RESTBase-API, and 4 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac) [07:49:05] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Joe) A net effect of this patch is now all logged-in users are back to HHVM. I think we need to backport the patch above to the runni... [08:14:14] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Zotero container: Latest production container lacking ca-certificates package - https://phabricator.wikimedia.org/T223345 (10akosiaris) >>! In T223345#5183583, @Joe wrote: > It's working in... [08:18:53] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10akosiaris) >>! In T220235#5183451, @Krenair wrote: > `krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker ru... [08:44:47] fsero: good morning :] [08:45:15] eventually we have the use for a new namespace in the Docker registry and associated credentials [08:45:22] Morning hashar [08:45:34] the aim is to publish some docker images intended to replace mediawiki/vagrant and thus provide a development environment to mediawiki developers [08:45:36] https://phabricator.wikimedia.org/T223329 ;D [08:46:08] our workaround is to have the docker-pkg definitions to be copied to integration/config and the image published under the /releng/ registry namespace, but that is ugly. They are really two different projects with different sub team owrking on each [08:46:15] hence the request for a new namespace ;] [08:46:19] There is any authorization scheme on our registry right now so if we create new credentials there is nothing that avoid that credentials to publish in any namespace [08:46:33] ohhhhh [08:46:41] Said that is a good idea to have another namespace [08:46:47] but we still need the namespace to be created beforeheand don't we? [08:46:56] Err no [08:47:08] You just need to tag the image to include the namespace [08:47:10] so any credential can create any namespace. Good [08:47:25] (well I cant tell whether it is good really, but that is nice for my use case :] ) [08:47:34] Is horrible [08:48:10] We can create a new set of credentials indeed but I don't see the upside without authorization [08:50:26] eventually brennen created a puppet change that craft a docker-pkg config file [08:50:26] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ [08:51:43] but maybe we can just reuse the same credentials [08:52:42] <_joe_> fsero: we can do path-based auth in... nginx! [08:52:55] <_joe_> I just never got to do it [08:54:20] currently on contint1001 we have a user "uploader" for the CI images on /releng/ [08:54:32] so I guess we can just reuse the same credentials, but publish to a /dev/ namespace instead [08:55:13] <_joe_> for now, yes [08:56:35] good, I am going to hijack the puppet patch [09:02:13] _joe_: absolutely but If we invest time on it id prefer to do something more solid [09:02:30] hashar: also what are the policies to create new namespaces? [09:02:41] Namespace per team namespace per topic? [09:02:46] <_joe_> as in? have a split between public and private registries? [09:02:53] <_joe_> fsero: topic, up to now [09:02:58] I have no idea of a policy really :-( [09:03:06] and nothing is enforced anyway right now? [09:03:20] for the CI images we went with /releng/ out of the blue [09:03:31] not now but if we want to enforce it we need a policy [09:03:43] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Nohtr15) What was the point remove PHP7 beta feature? Now all logged-in users are forced to use HHVM. There is a huge difference betw... [09:03:48] then experience has shown that naming things based on team name is usually a bad idea. Teams do not last long and releng might well be rename in the short future [09:05:10] is creating that policy something that belongs to releng? [09:06:23] no clue? ;D [09:08:33] <_joe_> no, to us all [09:08:40] <_joe_> next pipeline meeting [09:08:44] +1 [09:09:43] meanwhile I could use a docker-pkg config to push those new images from contint1001 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ [09:10:04] I edited the patch to reuse the existing credential (from hiera docker::registry::username and docker::registry::password ) [09:11:12] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Joe) >>! In T219128#5183956, @Nohtr15 wrote: > What was the point remove PHP7 beta feature? Now all logged-in users are forced to use... [09:18:14] hashar: yep [09:23:38] fsero: can you possibly +2 the docker-pkg config that reuses the same credentials and publish to a /dev/ namespace? :) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ [09:23:46] we need those images publishfor the hackathon this week [09:23:56] week-end [09:24:58] sure, done, might be better to think about this before the next hackathon and no rush into decisions :) [09:26:09] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Nohtr15) HHVM feels very slow after using PHP7, so it doesn't make any sense to use HHVM. I'm very sad because I cannot choose PHP7 a... [09:30:25] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Reedy) >>! In T219128#5183998, @Nohtr15 wrote: > HHVM feels very slow after using PHP7, so it doesn't make any sense to use HHVM. I'm... [09:33:13] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Joe) >>! In T219128#5183998, @Nohtr15 wrote: > HHVM feels very slow after using PHP7, so it doesn't make any sense to use HHVM. I'm v... [09:36:32] fsero: agreed ;) [09:43:39] fsero: for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ you also need to submit it? then i guess that requires a rebase now :((( [09:45:20] hashar: i thought you only needed a +1 you also need a merge from me and a puppet run? [09:46:31] fsero: yeah puppet.git is special [09:46:44] CI does vote Verified +2 on test [09:46:55] but a human has to CR+2 and submit the change to have it merged [09:46:59] (CI does not merge for you) [09:47:07] which really should be fixed but ... [09:49:29] hashar: could you run a PCC on contint* please? [09:49:39] i need to be afk for some time now, i'll merge it later [09:49:52] <_joe_> hashar: I think fsero wasn't aware you can't merge a change on ops/puppet [09:50:01] <_joe_> he's aware of puppet-merge :P [09:50:13] yeah [09:50:22] well eventually I will finally escalate that to get puppet rights [09:50:34] it has been such a pain point for close to a decade now [09:50:59] yeah i wasnt aware of that [09:51:15] +1 on running the ppc [09:51:20] I alway sforget about itt [09:53:28] <_joe_> hashar: I think the problem is we can't delegate +2 even for portions of the infrastructure [09:53:36] <_joe_> frankly as it is, +2 == root [09:53:45] yeah [09:54:08] <_joe_> so I would be extremely happy if we had a way to give you people +2 on just the code that affects the CI servers [09:54:18] <_joe_> where you have root [09:54:42] You can using prolog [09:54:45] <_joe_> it's very very hard to do so with puppet and gerrit in the mix, and no one ever had the time to try to engineer something in that direction [09:54:47] In gerrit [09:55:11] <_joe_> paladox: they can't run the merge step afterwards, it's not that simple sadly [09:55:23] Ah ok [09:55:31] paladox: yeah there is an extra step that requires access to the puppetmaster which is global for the whole infra [09:55:40] and having access to the puppetmaster === cluster wide root access [09:55:43] which really [09:55:50] <_joe_> as I said, it's complex to do it well, and I fear it would require splitting repositories in some clever way [09:55:51] is sensible ;] [09:55:55] <_joe_> yes [09:56:13] anyway that is an odd situation and releng suffers from it over and over [09:56:19] but thta is not a new topic :] [09:56:46] fsero: puppet compiler https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ works :] [09:56:48] I am off [09:56:57] gotta get the kids back from school + lunch break etc [09:56:58] :) [09:57:00] bbl [09:57:03] Great hashar thank you [09:57:19] feel free to merge anytime :] [09:57:21] ++ [11:05:15] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ArielGlenn) The blocking ticket is closed; what else is needed for this to move forward? [11:05:23] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ArielGlenn) [11:09:36] tarrow: got an incoming for you https://gerrit.wikimedia.org/r/#/c/wikibase/termbox/+/510470/. Seems like I messed up openapi.json on the first try. I 've tested this one though and seems fine [11:54:22] 10serviceops, 10Operations, 10observability, 10PHP 7.2 support, and 2 others: [Regression] Varnish is replacing the detailed HTTP 500 page from PHP 7 with "503 Service Temporarily Unavailable" - https://phabricator.wikimedia.org/T223336 (10Joe) [11:55:55] 10serviceops, 10Operations, 10observability, 10PHP 7.2 support, and 2 others: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm. - https://phabricator.wikimedia.org/T223336 (10Joe) [11:57:37] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban), and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10ArielGlenn) [11:59:06] 10serviceops, 10Operations, 10observability, 10PHP 7.2 support, and 2 others: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm. - https://phabricator.wikimedia.org/T223336 (10Joe) I changed the title of the task to reflect myt findings, and changed the associated tags acc... [12:06:50] Hey there? [12:07:11] any idea why this: https://gerrit.wikimedia.org/r/c/maps/kartotherian/package/+/510456 is not being submitted to jenkins [12:07:23] fsero: ^ [12:23:13] <_joe_> onimisionipe: wrong channel, I think [12:23:41] Oh.. releng [12:23:43] sorry [12:42:41] fsero: so I ran this morning patch through the puppet compiler and it seems to work just fine ;) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ [12:42:57] i know hashar we are dealing with something right now [12:43:07] i'll merge when things are clearer [12:54:44] fsero: cool ;) [12:54:54] I am off myself, will check later tonight [13:28:08] 10serviceops, 10Operations, 10Performance-Team, 10observability, and 2 others: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm. - https://phabricator.wikimedia.org/T223336 (10Krinkle) [15:56:30] <_joe_> fsero: I'll look at the mcrouter patch later or tomorrow morning [15:56:50] thanks, i dont know why jenkins is downvoting me [15:57:01] there is no clear errors [16:00:11] <_joe_> have you checked jenkins logs? [16:00:51] yes https://integration.wikimedia.org/ci/blue/organizations/jenkins/operations-puppet-tests-stretch-docker/detail/operations-puppet-tests-stretch-docker/13287/pipeline/ [16:01:20] there is Build step 'Execute shell' marked build as failure but dont see any obvious error [16:13:16] fsero: I replied on the change ;) [16:13:22] (not sure how i ended up being a review on it [16:15:35] thank you hashar [16:15:52] 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10Joe) [16:15:59] 10serviceops, 10Operations, 10Beta-Feature, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: Remove php7 beta feature - https://phabricator.wikimedia.org/T219128 (10Joe) 05Open→03Resolved [16:16:58] fsero: jbond42 should be able to fix that setup_python_extension method ;D [16:17:12] fsero: and I can still use a merge of the docker-pkg config at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510249/ ;) [16:17:18] the puppet catalog works there [16:17:30] hashar: just rebased it [16:17:41] \o/ [16:19:45] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) I fiddled around with that and now the container exits after 15s without printing anything :| [16:23:46] 10serviceops, 10dev-images, 10Patch-For-Review: Create credentials and add a puppet secret for publishing dev-images with docker-pkg in /dev/ namespace - https://phabricator.wikimedia.org/T223329 (10hashar) 05Open→03Resolved a:03hashar After discussion with SRE, we went reusing the credentials used to... [16:24:00] fsero: looks good on contint1001. Thank you! [16:27:55] hashar: thanks will catch up in after this meeting [16:39:15] fsero: the images have been published just fine. Hurrah and thank you very much [16:39:21] that unlock folks for the Hackathon! [17:24:04] 10serviceops, 10Operations, 10PHP 7.2 support: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10Dzahn) [17:49:31] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container runs silently for 15 seconds and then stops - https://phabricator.wikimedia.org/T223395 (10Krenair) [17:52:24] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container runs silently for 13-14 seconds and then stops - https://phabricator.wikimedia.org/T223395 (10Krenair) [17:53:01] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container runs silently for 13-14 seconds and then stops - https://phabricator.wikimedia.org/T223395 (10Krenair) Looks like after I fiddle with the permissions it does log errors to /etc/mediawiki-services-cxse... [17:53:07] 10serviceops, 10Operations, 10PHP 7.2 support: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10Dzahn) We now have both PHP 7.0 and PHP 7.2 packages installed but we are still using HHVM. https://wikitech.wikimedia.org/w/info.php [18:05:02] 10serviceops, 10CX-cxserver, 10Beta-Cluster-reproducible, 10Services (next): Cxserver container: Container does not send fatal errors to docker logs via stdout? - https://phabricator.wikimedia.org/T223395 (10Krenair) [18:13:56] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Thank you to whoever has been following my progress and updating the *-beta.wmflabs.org proxies. I haven't... [18:51:31] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) cxserver runs but does not seem to know about the basics: ` krenair@deployment-docker-cxserver01:~$ curl loc... [18:58:25] 10serviceops, 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) 05Open→03Resolved a:03Krenair Cxserver migration commits are https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510588/ and https://gerrit... [19:09:38] 10serviceops, 10Operations: SRE FY2019 Q4 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Jdforrester-WMF) [19:09:40] 10serviceops, 10Operations, 10Patch-For-Review: Use PHP7 to run maintenance scripts - https://phabricator.wikimedia.org/T219135 (10Jdforrester-WMF) [19:12:20] 10serviceops, 10Operations: SRE FY2019 Q4 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Jdforrester-WMF) [19:12:26] 10serviceops, 10Operations, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10Dzahn) [19:12:30] 10serviceops, 10Operations: SRE FY2019 Q4 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Dzahn) [19:12:39] 10serviceops, 10Operations: SRE FY2019 Q4 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Jdforrester-WMF) [19:13:28] 10serviceops, 10Operations, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10Dzahn) oops, duplicate of both T219185 and T218615 [19:13:44] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 3 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki) [19:26:00] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Jdforrester-WMF) [20:04:22] anyone seen docker failing to stop containers before? [20:23:40] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Niklas found the solution to that one: set `mw_host: en.wikipedia.beta.wmflabs.org` in conf - otherwise it d...