[06:06:23] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) Taking a quick dump of the `$_SERVER` variable shows the content of `REQUEST_URI` are - as expected - une... [06:17:15] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) No, MediaWiki [prefers `REQUEST_URI`](https://github.com/wikimedia/mediawiki/blob/a03feb8ab05c29ecdf... [06:22:46] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) If I set `AllowEncodedSlashes` to `NoDecode` it works as intended, but: * It must be set at the vhost le... [06:46:44] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) After further examination, given MediaWiki indeed uses `REQUEST_URI`, and given that is unencoded in all... [07:20:16] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) 05Open→03Invalid Just confirmed the problem is at the varnish layer, not at the mediawiki one. I modi... [08:06:48] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) [08:37:32] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) >>! In T213089#5023501, @MoritzMuehlenhoff wrote: >>>! In T213089#5023411, @elukey wrote: >> Leaving here also a referen... [08:40:56] 10serviceops, 10Arc-Lamp, 10Operations, 10Performance-Team, 10Patch-For-Review: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) Immediate issue is resolved. But this is expected to alert again in 5-10 days unless a few days' worth are delete... [08:41:18] 10serviceops, 10Arc-Lamp, 10Operations, 10Performance-Team, 10Patch-For-Review: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) a:03Krinkle [08:41:39] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) I have been discussing how to proceed with @jijiki recently, and T208934 will be prioritized otherwise each reimage of a... [08:52:57] 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10Krinkle) [09:15:00] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) >>! In T213089#5574371, @elukey wrote: > I have been discussing how to proceed with @jijiki recently, and T208934 will b... [09:24:35] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) Thank you @Joe for taking the time to investigate. I created {T235478} to follow up with the traffic... [09:50:11] 10serviceops, 10Arc-Lamp, 10Performance-Team: Backups for arclamp application data - https://phabricator.wikimedia.org/T235481 (10Krinkle) [10:31:10] 10serviceops, 10Operations: Jobrunners: allow to check that they are in sync with the etcd data - https://phabricator.wikimedia.org/T235488 (10Volans) [10:34:54] 10serviceops, 10Operations: Jobrunners: allow to check that they are in sync with the etcd data - https://phabricator.wikimedia.org/T235488 (10Joe) I think the best way is probably writing a small endpoint in operations/mediawiki-config that just exposes that. [10:37:22] 10serviceops, 10Operations: Jobrunners: allow to check that they are in sync with the etcd data - https://phabricator.wikimedia.org/T235488 (10Krinkle) [10:43:34] 10serviceops, 10Scap, 10observability: logstash_checker should be able to check for error for any php version - https://phabricator.wikimedia.org/T235490 (10jijiki) [10:43:56] 10serviceops, 10Scap, 10observability: logstash_checker should be able to check for error for any php version - https://phabricator.wikimedia.org/T235490 (10jijiki) [10:43:58] 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10jijiki) [11:15:49] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) Settings that we might want to use with memcached 1.5.6 (Buster's version): * `-o modern`: alias for the following sett... [11:59:31] 10serviceops, 10Operations, 10HHVM, 10Patch-For-Review, 10Performance-Team (Radar): Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10jijiki) [12:13:02] 10serviceops, 10Operations, 10service-runner, 10CPT Initiatives (RESTBase Split (CDP2)), and 5 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10mobrovac) [15:18:04] 10serviceops, 10Growth-Team, 10Notifications, 10Operations, and 3 others: Echostore service endpoints - https://phabricator.wikimedia.org/T234464 (10Joe) I've done all the puppet/dns prep work. You can now proceed to prepare this new kask deployment in `operations/deployment-charts`. Data you will need:... [15:23:51] 10serviceops, 10Growth-Team, 10Notifications, 10Operations, and 3 others: Echostore service endpoints - https://phabricator.wikimedia.org/T234464 (10Eevans) >>! In T234464#5575683, @Joe wrote: > I've done all the puppet/dns prep work. You can now proceed to prepare this new kask deployment in `operations/... [16:00:38] 10serviceops, 10Operations, 10Kubernetes: Upgrade the envoyproxy package to its latest version. - https://phabricator.wikimedia.org/T235412 (10Joe) [18:16:13] 10serviceops, 10Operations: Jobrunners: allow to check that they are in sync with the etcd data - https://phabricator.wikimedia.org/T235488 (10crusnov) p:05Triage→03Normal [18:33:25] 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10Krinkle) [19:58:16] 10serviceops, 10Operations, 10media-storage, 10Patch-For-Review, and 2 others: Swift object servers become briefly unresponsive on a regular basis - https://phabricator.wikimedia.org/T226373 (10Gilles) [20:26:23] 10serviceops, 10Core Platform Team, 10Performance-Team (Radar): Reconsider memcached connection method for MW in PHP7 world - https://phabricator.wikimedia.org/T235216 (10daniel) What's the ask for CPT here? [20:47:23] 10serviceops, 10Growth-Team, 10Notifications, 10Operations, 10CPT Initiatives (Multi-DC Echo Notification Storage): Dashboards for monitoring of echostore - https://phabricator.wikimedia.org/T235558 (10Eevans) [20:48:01] 10serviceops, 10Growth-Team, 10Notifications, 10Operations, 10CPT Initiatives (Multi-DC Echo Notification Storage): Dashboards for monitoring of echostore - https://phabricator.wikimedia.org/T235558 (10Eevans) p:05Triage→03Normal [21:05:39] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Gilles) Additional possibility: handling the compression at the OS level Eg. https://btrfs.wiki.kernel.org/index.php/Compression [21:09:56] 10serviceops, 10Core Platform Team, 10Performance-Team (Radar): Reconsider memcached connection method for MW in PHP7 world - https://phabricator.wikimedia.org/T235216 (10Joe) As far as I remember, the php memcached extension didn't allow using a unix socket to connect to memcached. According to https://www.... [21:22:33] 10serviceops, 10Arc-Lamp, 10Operations, 10Performance-Team, 10Patch-For-Review: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) 05Open→03Resolved [21:24:29] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Krinkle) a:03Krinkle Short-term decision based on today's meeting: Temporarily reduce retention from 90 days to 45 days. Hopefully only for a few weeks. Once done... [21:38:33] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Krinkle) Review of overall approach on using YAML perf restrictions/flexibilities (e.g. what should def be cached... [22:15:01] 10serviceops, 10Machine vision, 10Operations, 10Product-Infrastructure-Team-Backlog, and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Mholloway) 05Open→03Resolved The work encompassed by this task (updating the open_nsfw service) is com... [23:35:33] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn)