[00:06:00] 10serviceops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson Finished cables handing off to chris for remaining steps name rack_name position switch p... [00:06:16] 10serviceops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Jclark-ctr) [01:12:30] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10ssastry) [07:31:50] 10serviceops, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Move CORE REST API to be served from the MW API Cluster - https://phabricator.wikimedia.org/T246002 (10jijiki) Hey Will, Can you please provide more details so we can help you? Please link related tasks as well... [07:39:44] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10jijiki) >>! In T245841#5919699, @Joe wrote: > > What would having all scap proxies also be mcrouter proxies change in terms of the scenario you described above? > This w... [09:42:08] 10serviceops, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Move CORE REST API to be served from the MW API Cluster - https://phabricator.wikimedia.org/T246002 (10Joe) >>! In T246002#5922550, @jijiki wrote: > Hey Will, > > Can you please provide more details so we can h... [10:14:38] 10serviceops, 10MediaWiki-General, 10Operations, 10observability: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10Joe) [11:57:41] 10serviceops, 10Operations, 10ops-eqiad: mw1280 crashed logging correctable memory errors - https://phabricator.wikimedia.org/T240187 (10Volans) The host has been down a week, hence it has been removed from PuppetDB and the Netbox report catched it. Updated Netbox setting it's state to Failed. Please follow... [12:07:12] 10serviceops: a few appservers at a time suffer mcrouter backlogs, leading to high latency - https://phabricator.wikimedia.org/T240409 (10Joe) 05Open→03Resolved We've changed the niceness of mcrouter to be on par with the niceness of php-fpm; since then, we didn't see such huge lags anymore. Marking this as... [15:06:22] 10serviceops, 10ChangeProp, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 3 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10hnowlan) I've committed tokens on the private repo. Stubs have been merged to labs-private. [15:52:01] <_joe_> rlazarus, akosiaris https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/575268 [15:52:13] <_joe_> and followups [15:52:22] <_joe_> how do you feel about switching those on monday? [15:52:39] <_joe_> I'll prepare further patches, starting from https urls [15:53:16] <_joe_> also akosiaris do you think it's worth the effort installing it on scb? [15:54:08] _joe_: envoy? [15:54:13] I sincerely doubt it [16:29:43] I will be 5' late [17:25:47] 10serviceops, 10Release-Engineering-Team, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Enable phpdbg on mwdebug* servers - https://phabricator.wikimedia.org/T244549 (10hnowlan) [19:10:08] actually akosiaris dunno if you are still thre but i'm having a bit of trouble deploying eventstreams in staging. [19:10:44] I had destroyed the releases there because of this trouble, and now am trying to apply again, but the command just hangs [19:10:47] no pods are created [19:10:53] i see the Service is created... [19:11:12] the helm release seems stuck in [19:11:13] STATUS: PENDING_INSTALL [19:46:09] AH [19:46:14] it works if i lower memory limits [19:46:15] HMMMM [19:46:21] maybe staging cluster is full? [19:50:20] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10greg) Approved from my end. [20:03:05] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforre... [20:13:27] 10serviceops, 10Parsoid-PHP, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Create /srv/mediawiki/parsoid-vendor on production MW appservers, a check out of vendor.git's parsoid branch - https://phabricator.wikimedia.org/T245886 (10Jdforrester-WMF) 05Open→03Declined For now, we won't do this. [20:13:32] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforre... [21:03:10] 10serviceops, 10MediaWiki-General, 10Operations, 10observability: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10colewhite) One alternative is to adopt a sidecar in the form of statsd_exporter and have it do the heavy lifting of translating MediaWiki and MW Extension metrics in... [21:42:22] 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10Core Platform Team Workboards (Clinic Duty Team): Allow MW REST API to be called on job runners and video scalers - https://phabricator.wikimedia.org/T246389 (10Pchelolo) [22:32:25] hello folks. long time listener, first time caller [22:34:20] it appears that scandium isn't setting SERVERGROUP to parsoid, like it should. can someone help me with that? [22:34:33] we're not convinced that our beta machines are setting SERVERGROUP either as well [22:34:51] but maybe that's a question for #wikimedia-releng? I can't keep all these channels straight. [22:35:00] context: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/575336/2/wmf-config/CommonSettings.php ... applying that on scandium causes failures because that check fails. [22:45:43] cscott: I can just make the patch be || scandium || realm = labs [22:46:30] that adds checks on the critical path in production [22:46:44] it would work though [22:47:11] Yes, so not optimal.:-) [22:48:38] i've got to turn into a pumpkin to pick up my son from school, so i think we're not going to SWAT this tonight regardless