[06:40:10] 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10observability: job queue insert rate metrics gone from Grafana - https://phabricator.wikimedia.org/T238296 (10jcrespo) [07:42:22] 10serviceops, 10Community-Tech, 10Operations, 10wikidiff2, 10Patch-For-Review: Deploy new version of wikidiff2 package - https://phabricator.wikimedia.org/T140443 (10jijiki) [07:42:51] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Deploy latest version of wikidiff2 to production - https://phabricator.wikimedia.org/T236963 (10jijiki) [08:30:41] 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10observability: job queue insert rate metrics gone from Grafana - https://phabricator.wikimedia.org/T238296 (10elukey) Just "fixed" the https://grafana.wikimedia.org/d/000000400/jobqueue-eventbus removing `cluster='eventbus'` from the broken metrics. As... [08:36:46] 10serviceops, 10Core Platform Team, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10observability: job queue insert rate metrics gone from Grafana - https://phabricator.wikimedia.org/T238296 (10jcrespo) Both in cases, be it deprecated or not, probably we will want better discoverability (tags) on the new dashboa... [09:55:05] 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10observability, 10Core Platform Team Workboards (Clinic Duty Team): job queue insert rate metrics gone from Grafana - https://phabricator.wikimedia.org/T238296 (10mobrovac) Yes yes, we need it! All the JobQueue stats are there, so please don't remove it. [10:00:43] 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10observability, 10Core Platform Team Workboards (Clinic Duty Team): job queue insert rate metrics gone from Grafana - https://phabricator.wikimedia.org/T238296 (10jcrespo) Just to be clear, I wasn't suggesting removing it- mostly it was fixing the miss... [12:40:41] 10serviceops, 10Operations, 10PHP 7.2 support: Mysterious, coordinated slowdowns every ~ 25 minutes on API servers - https://phabricator.wikimedia.org/T231011 (10mobrovac) @Ladsgroup might be on the correct track here. The spikes neatly coincide with the [WD bot updates](https://eu.wikipedia.org/wiki/Berezi:... [12:42:49] 10serviceops, 10Operations, 10PHP 7.2 support: Mysterious, coordinated slowdowns every ~ 25 minutes on API servers - https://phabricator.wikimedia.org/T231011 (10Ladsgroup) >>! In T231011#5663421, @mobrovac wrote: > @Ladsgroup might be on the correct track here. The spikes neatly coincide with the [WD bot up... [15:45:11] 10serviceops, 10Operations, 10Wikimedia-Apache-configuration, 10Patch-For-Review: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10crusnov) We use `scap` to deploy Netbox, a possible use-case would be to run httpbb as the last step to verify that apache is configured... [15:47:55] 10serviceops, 10Operations, 10Wikimedia-Apache-configuration, 10Patch-For-Review: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10crusnov) >>! In T236699#5664011, @crusnov wrote: > We use `scap` to deploy Netbox, a possible use-case would be to run httpbb as the las... [16:07:20] 10serviceops, 10Parsoid-PHP, 10CPT Initiatives (Parsoid REST API in PHP (CDP2)): Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing - https://phabricator.wikimedia.org/T228069 (10Dzahn) @ssastry fyi, the PHP minor version on scandium got upgraded. [18:39:59] 10serviceops, 10Operations: Upgrade to PHP 7.2.24 - https://phabricator.wikimedia.org/T237239 (10Dzahn) 05Open→03Resolved @jijiki Done! @MoritzMuehlenhoff I see no more yellow or red in debmonitor for PHP 7.2 packages, only for PHP 7.0 packages. [21:26:53] rzl@cumin1001:~$ httpbb --host mwdebug1001 --insecure /srv/deployment/httpbb-tests/baseurls.yaml [21:26:53] Sending to mwdebug1001... [21:26:53] PASS: 18 requests sent, all assertions passed. [21:27:04] _joe_: ^ :D [21:30:07] (well, that plus a "hey, you turned off cert verification, are you sure???" warning that I thought I'd disabled, but) [21:32:15] <_joe_> :D [21:32:23] <_joe_> rlazarus: great [21:34:09] <_joe_> rlazarus: we shouldn't need to turn it off though [21:34:26] <_joe_> the https cert we have on the appservers includes all the public domains [21:36:53] hmmm, I get "SSLError: hostname 'mwdebug1001' doesn't match either of '*.wikipedia.org', [snip]" [21:37:10] but of course it's weird that requests is trying to check the certs for _that_ hostname [21:37:12] let me take another look [21:37:58] <_joe_> yeah it's something tricky about requests IIRC [21:58:02] 10serviceops, 10Operations, 10PHP 7.2 support: Mysterious, coordinated slowdowns every ~ 25 minutes on API servers - https://phabricator.wikimedia.org/T231011 (10Joe) @Theklan might be interested in watching this task too. [22:30:15] ahh looks like the answer is https://toolbelt.readthedocs.io/en/latest/adapters.html#requests_toolbelt.adapters.host_header_ssl.HostHeaderSSLAdapter [22:30:39] I'll give this a shot in the morning, have a nice evening all!