[00:44:13] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, 10Patch-For-Review: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) Note that Shellbox depends on monolog and core will depend on shellbox, so this will make core indirectly depend on... [02:19:24] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, 10Patch-For-Review: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) Regarding restrictions again. I'm not a big fan of Firejail after my recent code review and bug reports, so I'm loo... [06:16:57] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Traffic, 10Wikimedia-production-error: [Bug] Page content service is deployed with localhost links to the CSS and JS, breaking all pages that have been edited recently - https://phabricator.wikimedia.org/T262437 (10Joe) A short-form inc... [08:12:48] jayme: before merging the dns removal make sure to run the decom cookbook on those hosts [08:13:52] volans: yeah, thanks. Won't do this until next week [09:43:26] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki) [09:49:24] _joe_, akosiaris I send you guys an invite for today to talk about the ingress/lamdaoid/serviceproxy thing - if you find the time [09:56:34] <_joe_> can we do 4 pm our timezone? [10:04:31] I got a meeting for mathoid deployment at 16:30, probably [10:04:35] probably not* [10:22:04] could resched for Tue or Monday then ? [10:28:48] 10serviceops, 10Analytics, 10Release-Engineering-Team, 10observability, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10LarsWirzenius) @jijiki I haven't ever looked at the parts of Scap that would be affected by this, but I can't imagine it's t... [10:52:09] <_joe_> effie: no, let's do the original time. it's ok [10:52:52] I am joing just to listen in, whatever works for the rest of you [11:15:09] 10serviceops, 10Scap, 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10User-jijiki: Deploy Scap version 3.15.0-1 - https://phabricator.wikimedia.org/T261234 (10jijiki) Built successful and docs updated, I changed `debian/source/format` to `1.0` and it built just fine. I will roll out to c... [11:15:12] 10serviceops, 10Scap, 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10User-jijiki: Deploy Scap version 3.15.0-1 - https://phabricator.wikimedia.org/T261234 (10jijiki) p:05Triage→03High [11:27:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Define the plan for the upgrade of kubernetes cluster to a security supported release - https://phabricator.wikimedia.org/T241076 (10JMeybohm) [11:28:38] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Draft a plan for upgrading kubernetes machines to buster - https://phabricator.wikimedia.org/T245272 (10JMeybohm) [11:29:07] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10JMeybohm) [11:29:20] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade production kubernetes clusters to a security supported version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [11:32:38] * effie bbl [11:40:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10JMeybohm) p:05Triage→03Medium [11:46:43] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10MoritzMuehlenhoff) We should probably create some linux419 base profile (or similar) which * pulls in linux-image-4.19-amd64 (the meta package for 4.19 on stretch) * instal... [11:59:55] 10serviceops, 10DBA, 10SRE-tools, 10conftool, and 2 others: Alerting spam and wrong state of primary dc source info on databases while switching dc from eqiad -> codfw - https://phabricator.wikimedia.org/T261767 (10Marostegui) @RLazarus what do you want to do with this task? is this something that needs fi... [12:28:10] start the meeting and I will jump in [13:26:27] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10akosiaris) >>! In T262527#6450137, @MoritzMuehlenhoff wrote: > We should probably create some linux419 base profile (or similar) which > * pulls in linux-image-4.19-amd64 (... [13:27:50] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) a:03Papaul [14:44:12] <_joe_> I did a cursory analysis of the packages we will need to rebuild for buster, and the list is really short [14:44:43] <_joe_> it's basically the php packages (and we knew it) plus a few prometheus exporters that are probably go and thus can just be copied over [14:45:02] <_joe_> so moving mediawiki to buster might be a faster endeavour than it used to be [15:18:31] prometheus-apache-exporter is even natively in buster, no need to rebuild [15:19:33] and the mcrouter one is already build for buster as well (used on the IDPs), so should just be nutcracker and FPM left [15:21:59] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10observability: illegal_argument_exception - https://phabricator.wikimedia.org/T262429 (10fgiunchedi) Translated in english, the message `[detail] is defined as an object in mapping [fcm_send_failed] but this... [15:34:15] I offer to build some if needed [15:34:45] are we going tohave 1 person do packaging or we are going to split it? [15:35:00] also I was under the impression that 7.2 is default in buster, but I was clearly wrong [15:39:20] it's 7.3 [15:39:30] yeah, I read [15:41:35] <_joe_> so packaging is a minor concern this time, compared to the other transitions [15:41:48] <_joe_> partly because there is no multimedia or latex stuff around [15:41:59] <_joe_> I think we might need to adapt puppet a bit [15:42:17] <_joe_> but apart from that, it should be much smoother than before [15:42:39] well, not quite: ICU is much more in use in core libs compared to jessie->stretch, since Boost uses ICU [15:42:58] as we'll need to rebuild the ICU reverse deps for the intermediate versions [15:43:08] <_joe_> moritzm: oh but that's something you offered to take care of :D [15:43:14] I will :-) [15:43:17] <_joe_> smoother for *me*, I meant [15:43:26] sure, sure :-) [15:43:49] <_joe_> also I'm planning of not doing this myself, if at all possible [15:43:50] one other issue is that libxml uses Boost, which also radiates pain in other packages, we'll see [15:44:46] moritzm: I can help if you share [15:45:38] I will just echo "1.0" > debian/source/format [15:45:43] and everything will build nicely [15:45:57] yeah, we can probably split some of it, but first need a good overview of what's needed in detail [15:46:07] sure [15:46:37] the video scalers are simple this time, we currently use a backport of ffmpeg/libvpx, which will simply be provided by the buster packages by default [15:47:15] or job runners these days [15:47:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10JMeybohm) a:03JMeybohm [15:50:57] 10serviceops, 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10Milimetric) x-wikimedia-debug shouldn't affect anything we do [16:17:55] 10serviceops, 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10jijiki) @Milimetric my question is, if I want to do a performance test and fire 700k requests towards our debug server... [17:25:13] 10serviceops, 10Scap, 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10User-jijiki: Deploy Scap version 3.15.0-1 - https://phabricator.wikimedia.org/T261234 (10LarsWirzenius) Excellent, thank you! [19:40:05] 10serviceops, 10Analytics-Radar, 10Release-Engineering-Team, 10observability, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10Milimetric) >>! In T262202#6451136, @jijiki wrote: > @Milimetric my question is, if I want to do a performance test fr... [19:41:43] 10serviceops, 10Operations, 10Recommendation-API: recommendation-api alerting and api errors - https://phabricator.wikimedia.org/T262587 (10jijiki) [20:36:33] 10serviceops, 10Operations, 10Recommendation-API: recommendation-api alerting and api errors - https://phabricator.wikimedia.org/T262587 (10Mholloway) Looks like 503s from the API appservers are back down following today's MW train deployment. The root cause appears to have been T262575 but it's not clear t... [21:57:33] 10serviceops, 10MW-on-K8s, 10Operations, 10Patch-For-Review, and 2 others: Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10daniel) a:05tstarling→03None [21:57:52] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10daniel) [22:06:20] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10jeena) [22:11:36] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10jeena) I added php support to blubber. An example what would be added to the variant in your blubber... [22:17:12] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10jeena) [23:00:06] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10jeena)