[00:28:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2248.codfw.wmnet'] ` an... [00:28:24] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2249.codfw.wmnet'] ` an... [01:00:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2283.codfw.wmnet', 'mw22... [01:15:52] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [02:37:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2251.codfw.wmnet', 'mw22... [10:08:20] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10akosiaris) @hnowlan, hi. So, TL;DR this task is about exposing an instance of the new linkrecommendation service publicy. We... [10:30:26] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10hnowlan) This seems very well suited to the API gateway. All that really needs to be configured is a external path and an in... [10:41:51] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10Joe) If the usage is mostly for developers not having to run their own instance with their own data, I would suggest we expo... [10:59:51] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10hnowlan) We already have some support for x-mediawiki-debug for appservers in the gateway - given that we'll need to do some... [12:20:30] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10akosiaris) >>! In T269581#6786608, @Joe wrote: > If the usage is mostly for developers not having to run their own instance... [12:27:48] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10akosiaris) @kostajh I think the path described above (that is having the external linkrecommendation service exposed under e... [12:28:18] _joe_: where did the numbers in your email come from? [12:30:39] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) >>! In T269581#6786792, @akosiaris wrote: > @kostajh I think the path described above (that is having the external... [12:31:24] <_joe_> legoktm: go to sleep! [12:31:38] <_joe_> legoktm: from the "envoy telemetry" dashboard [12:33:17] <_joe_> I just selected "local port 80" as destination, and the single instances as "origin" [12:35:39] it's neat how envoy can be used for this use case as a side effect [12:36:12] I'll respond to the email but it would be nice if we could get split flamegraphs (https://performance.wikimedia.org/php-profiling/) to see stretch vs buster [14:20:40] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) Tests of xml/sql dumps in buster instance in deployment-prep look good. Next step: reimage the snapshot testbed instance in prod... [14:43:13] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` snapshot1007.eqiad.wmnet ` The log c... [15:11:13] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10hnowlan) >>! In T269581#6786793, @kostajh wrote: >>>! In T269581#6786792, @akosiaris wrote: >> @kostajh I think the path des... [15:16:54] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['snapshot1007.eqiad.wmnet'] ` and were **ALL** successful. [15:40:41] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) >>! In T269581#6787204, @hnowlan wrote: >>>! In T269581#6786793, @kostajh wrote: >>>>! In T269581#6786792, @akosiar... [16:10:28] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Create a yaml structure for defining apache virtualhosts for mediawiki, that can be used both in puppet and in helm charts. - https://phabricator.wikimedia.org/T272305 (10Joe) p:05Triage→03Medium [16:11:09] <_joe_> I would really love to get a feedback on https://gerrit.wikimedia.org/r/c/operations/puppet/+/657139 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/659941 [16:11:39] <_joe_> the tldr is: most of the work to adapt our apache configs to work in helm charts seamlessly and in sync is done [16:12:20] <_joe_> now to change the apache configs one needs to change a yaml in puppet, and that will reflect both in the apache vhosts on the appservers, and in a data structure I'll derive the apache configs from in helm [17:56:32] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) Test run of elwikiquote on reimaged testbed server running buster looks good, but I should do a prefetch run tomorrow morning ju... [18:42:53] mutante: I can't find the task for the buster server perf investigation [18:43:23] :/ [18:44:12] effie: there isn't a separate task for that yet, just the normal upgrade task and the wikitech page [18:44:28] I think we should create a task then [18:44:45] ok, i'll do that [18:48:06] and 1405, 1403 have the same hardware right? [18:49:21] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: performance degradation on appservers after buster upgrade? - https://phabricator.wikimedia.org/T273312 (10Dzahn) [18:49:39] great, thank you [18:51:26] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: performance degradation on appservers after buster upgrade? - https://phabricator.wikimedia.org/T273312 (10Dzahn) [18:52:31] effie: https://phabricator.wikimedia.org/T273312 yes, 1403 and 1405 confirmed in netbox, from the same procurement ticket [18:52:59] great thank you ! [18:53:04] thank you too [19:19:55] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10akosiaris) >>! In T269581#6787322, @kostajh wrote: >>>! In T269581#6787204, @hnowlan wrote: >>>>! In T269581#6786793, @kosta... [19:38:24] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: performance degradation on appservers after buster upgrade? - https://phabricator.wikimedia.org/T273312 (10Reedy) [20:01:03] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10jijiki) [20:01:21] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10jijiki) [20:23:51] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10jijiki) The problem is more evident on the API, [[ https://g... [20:28:37] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10jijiki) The problem is more evident on the API, [[ https://g... [21:01:09] 10serviceops, 10SRE, 10Performance Issue, 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10CDanis) Here are overall, whole-cluster mean latency millise... [21:06:53] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Krinkle) [21:46:37] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10jijiki) > > Something I haven't looked at yet, but w... [23:15:28] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) p:05Triage→03High [23:39:41] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) From `#wikimedia-perf`: ` 04:45:33