[14:29:52] <_joe_> ottomata, I think we need a meeting [14:30:47] <_joe_> with marostegui and kormat as well, at least [14:30:59] <_joe_> (re: T120242) [14:38:44] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) 05Open→03Resolved This works \o/ https://api.wikimedia.org/service/linkrecommendation/v0/linkrecommendati... [14:38:46] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-1.36-notes (1.36.0-wmf.34; 2021-03-09), 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) [14:44:33] 10serviceops, 10Add-Link, 10Growth-Team, 10Platform Team Initiatives (API Gateway): 504 timeout and 503 errors when accessing linkrecommendation service - https://phabricator.wikimedia.org/T277297 (10kostajh) @akosiaris maybe the [keepalive](https://docs.gunicorn.org/en/stable/settings.html#keepalive) valu... [14:44:50] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Platform Team Initiatives (API Gateway): 504 timeout and 503 errors when accessing linkrecommendation service - https://phabricator.wikimedia.org/T277297 (10kostajh) [14:50:02] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-1.36-notes (1.36.0-wmf.34; 2021-03-09), 10Patch-For-Review: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) 05Open→03Resolved Done, thank you so much to everyone who helped... [15:34:22] _joe_: indeed! [16:34:06] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Platform Team Initiatives (API Gateway): 504 timeout and 503 errors when accessing linkrecommendation service - https://phabricator.wikimedia.org/T277297 (10Tgr) >>! In T277297#6908496, @kostajh wrote: > For the 504s, yes. For the 503s you'll need... [16:37:55] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10Dzahn) a:03Dzahn [16:55:50] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2224.codfw.wmnet` - mw2224.codfw.wmnet (**PASS**) - Downti... [16:56:42] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2220.codfw.wmnet` - mw2220.codfw.wmnet (**PASS**) - Downti... [17:28:41] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2221.codfw.wmnet` - mw2221.codfw.wmnet (**PASS**) - Downti... [17:55:15] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2222.codfw.wmnet` - mw2222.codfw.wmnet (**PASS**) - Downti... [18:04:40] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10Dzahn) [18:07:50] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2223.codfw.wmnet` - mw2223.codfw.wmnet (**PASS**) - Downti... [18:18:02] 10serviceops, 10Performance-Team, 10SRE, 10Patch-For-Review, 10User-jijiki: Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe - https://phabricator.wikimedia.org/T264604 (10Krinkle) Hm.. so the above poses a bit of a paradox. Implementing the `onhostRoutingPrefix` option is dependent on tom... [19:06:38] can someone verify for me that this output from `helmfile diff` is OK? - https://phabricator.wikimedia.org/P14861 [19:07:01] the egress stuff, basically [19:07:30] I'm guessing it was added by someone in serviceops (maybe a while ago), and just wasn't deployed [19:10:09] oh yeah, `git blame` says that was _joe_ in September of last year [19:11:00] it looks like it's meant to allow access to the Cassandra cluster (though that seems to be possible anyway) [19:15:02] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Patch-For-Review, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) [19:49:52] 10serviceops, 10Analytics-Radar, 10Cassandra, 10ContentTranslation, and 10 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10Eevans) [19:53:19] 10serviceops, 10Platform Engineering, 10SRE, 10Performance-Team (Radar): Get rid of nutcracker for connecting to redis - https://phabricator.wikimedia.org/T277183 (10Gilles) [20:23:44] 10serviceops, 10Patch-For-Review: decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242) - https://phabricator.wikimedia.org/T277119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2225.codfw.wmnet` - mw2225.codfw.wmnet (**PASS**) - Downti... [21:04:12] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) 05Resolved→03Open One remaining issue, the k8s nodes can't pull restricte... [21:27:25] legoktm: [no hurry], when you have a second, could you set the priority to T277263 please? (/me with clinic duty hat) [21:27:37] * legoktm hurries [21:28:08] done :) [21:32:33] thx, I have T276686 too if you're in the right mood :) [21:33:05] aaaand T276632 (last one for today, I promise) [21:35:09] {{done}} [21:35:41] <3 [23:10:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [23:10:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10Jclark-ctr) Handing over to Chris to finish the 2 we have received mc1037 r,A7 u41 P5 ID5352 mc1038 r,A7 u42 P19 ID5353