[00:46:07] 10serviceops, 10Operations, 10PHP 7.2 support, 10PHP 7.3 support: PHP 7.2 is very slow on an allocation-intensive benchmark - https://phabricator.wikimedia.org/T230861 (10tstarling) Fun fact: the "gc address" of an object is packed into 14 bits, so increasing GC_ROOT_BUFFER_MAX_ENTRIES beyond 16383 causes... [03:49:33] 10serviceops, 10Operations, 10PHP 7.2 support, 10PHP 7.3 support: PHP 7.2 is very slow on an allocation-intensive benchmark - https://phabricator.wikimedia.org/T230861 (10tstarling) PHP 7.3 rearranges the bitfield so that the address gets 20 bits. It's a binary compatibility break, and possibly a source co... [06:53:05] 10serviceops, 10Operations, 10PHP 7.2 support, 10PHP 7.3 support: PHP 7.2 is very slow on an allocation-intensive benchmark - https://phabricator.wikimedia.org/T230861 (10ssastry) >>! In T230861#5433119, @tstarling wrote: > PHP 7.3 rearranges the bitfield so that the address gets 20 bits. It's a binary com... [09:02:53] 10serviceops, 10Operations, 10PHP 7.2 support, 10PHP 7.3 support: PHP 7.2 is very slow on an allocation-intensive benchmark - https://phabricator.wikimedia.org/T230861 (10tstarling) Tracing of TagTk::__destruct() shows that tokens are freed as it goes, they're not the problem. Disabling the GC causes the r... [13:35:27] 10serviceops, 10Operations, 10PHP 7.2 support: Mysterious, coordinated slowdowns every ~ 25 minutes on mw1347,mw1348 (php7 api servers) - https://phabricator.wikimedia.org/T231011 (10Joe) First smoking gun is in all the intervals I controlled the offender was parsoid-batch with quite large requests. I'm tryi... [13:36:24] _joe_: in my usual way I'm going to ask you a very naive question [13:36:55] when MW wants to call out to another HTTP(S) API as part of handling a request, is there one common code path for that? [13:38:00] <_joe_> yes [13:38:13] <_joe_> cdanis: can I ask you what's the context? [13:38:39] another thing it would be useful to have a per-request distribution of [13:38:54] was thinking of it as one of the things possibly related to T231011 [13:39:01] but yes we should just get real distributed tracing [13:39:10] <_joe_> no I don't think that's it [13:39:17] <_joe_> but yeah once envoy does the proxying [13:39:20] <_joe_> :) [13:39:35] <_joe_> I'm pretty sure it's some code path that's particularly bad for php7-fpm [13:39:48] <_joe_> I'm tempted to disable GC on one of those nodes [13:40:21] I have been looking for excuses to attempt using eBPF userspace code tracing [13:40:33] this could be one of them [13:41:51] <_joe_> yeah in the case of mw1270 I intend to just attach perf top once it's in a difficult situation, as it persists. But with transient behaviour it's probably the best way to try to go blackbox debugging [13:41:59] <_joe_> I am trying to go the other way around [13:42:17] <_joe_> find what kind of requests cause this behaviour every ~ 25 minutes [13:44:16] yeah [15:10:04] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10Jdforrester-WMF)