[07:10:57] dpifke: FYI: one problem I had one I tried to run WPT in Docker was that Firefox stopped to work after a while. See https://phabricator.wikimedia.org/T192050#5112398. it turned out at least at that time no one else was using the Dockerized version or if they did they only used Chrome. It was really annoying because it started to fail after X amount of hours. [07:58:14] Krinkle: About AWS and stability in metrics: I did some try on the Mac mini I rented a while ago and for the iOS simulator (without the overhead of running ffmpeg) the stability was better than AWS. For Desktop tests it was higher though. The Mozilla team tried out using capture cards instead of ffmpeg to get rid of the overhead. The thing is that when gilles and me tried out on a bare metal server, we had a 1 CPU server. [07:59:05] And I think with dpifke and a better server we can do better :) I think that's the only way to take control over the noisy neighbour problem. [07:59:21] Even on AWS stability changes over time. [07:59:39] And is different on different instances. [08:25:04] doing an hdmi capture for the screen recording for desktop might be interesting. devices that do that for video game streaming are pretty inexpensive. I have one laying around [08:25:40] it'd be interesting to see how much extra stability we get that way, by doing the video recording on a secondary machine [08:26:27] even looping back to the same machine might somehow work, since the video encoding is done by the dongle [10:34:17] https://github.com/dpalmeiro/browsertime-capturecard but it was a couple of years ago, also seems for Windows. I don't think the Mozilla team measured stability, they checked overhead. When I tried on the Mac mini the simulator (using the video stream built into the simulator) had 1% vs desktop 20-25%. But it was on Mac Stadium and I'm unsure of stable internet connection each Mac has. [10:41:19] hmm yeah the Elgato dongle with hardware encoder is Windows-only, maybe there are competing options [10:45:57] BlackMagic has one that should work [10:46:25] https://www.amazon.com/Blackmagic-Design-Recorder-Distributes-Websites/dp/B005ABJ0H8 [19:35:48] dpifke: I think I may have an answer regarding the reqId issue... [19:35:54] dpifke: there is a static cache inside that method [19:36:07] and... we first call that method *before* local settings are loaded [19:36:20] thus $wgAllowExternalReqID hasn't become true yet (in prod) [19:36:28] from Setup.php via HeaderCallback [19:36:40] literally one statement before loading LocalSettings (and has to be, unfortunately) [19:37:55] will likely fiddle a bit with HEaderCallback so as to not need it yet, since we do want to defend against that before loading all of LcoalSttings (and wmf-config) [19:38:02] bit of a paradox as it stands [19:42:05] Good catch!