[09:20:30] phedenskog: there's no way to know what extra JS the browser is running before and after, is there? [09:20:35] like a flame graph of sorts [09:42:33] I think we could get that with Firefox gecko profiler but we don't run that today, but we should then. I added a task for adding Firefox but haven't got around to do it yet [09:43:02] Have you looked at the Chrome trace log in devtools, I haven't checked that yet. [10:12:05] not yet [10:12:24] phedenskog: is it possible to access WPT test runs older than 7 days? I'm getting a 403 when I try [10:13:51] From Grafana with "Show each test" but zoom in first to that area: https://grafana.wikimedia.org/d/000000057/webpagetest-drilldown?orgId=1 [10:15:14] WPT has a default only 7 days in a PHP file (its not configurable if I remember correct) I used to hack it but then the automatic git pull doesn't work, so I guess I must have missed doing it now. I can add task for finding a better solution. [10:15:26] Making it configurable upstream [10:29:27] This is cool: https://twitter.com/FirefoxPreview/status/1309425312258719744 [10:55:18] I'm going to wait until the MobileFrontend folks chime in before doing more research, seems like the best lead so far [10:56:09] something's definitely up with our code, regardless of the source [11:45:47] last RUM graph from mobile Safari looks really bad [11:46:08] I have no idea how such a small HTML change can have cause so much harm [11:46:27] hopefully the people who made the change have a better idea [12:21:53] I created dashboard for the data that we collect from the Moto G5 at Kobition: https://grafana.wikimedia.org/d/Tw-s6iKMz/mobile-webpagereplay?orgId=1 [12:22:29] Look at the increase in time spent in long tasks. [13:35:00] What is "warp"? I can't find anything about it.. [13:35:12] (RE: Firefox) [16:48:13] phedenskog: is wpt.wmftest.org dead for you too? [16:48:23] I'm getting a 500 trying to get to the home page [16:52:48] dpifke: you might want to look into this ^ [17:08:54] gilles: Looks like it ran out of disk space, investigating why. [17:20:50] Most (~22G) of the 30G / partition is /var/www/webpagetest/www/results. There's a nearly-empty 30G /data partition seemingly for that purpose. If noone has any immediate objections, I'm going to move the results folder there and create a bind mount. [17:26:34] dpifke: sounds good to me, thanks [17:29:03] It's copying ~20MB/s, so will take ~15 min or so. Currently ~6G in. [17:29:41] Also: /var/www/webpagetest/www/tmp is a ~500M RAM disk that's also full. Once I free up space on /, I'm going to convert it back to a regular directory. [17:55:55] gilles: Lucas_WMDE in #-operations was looking for some help with grafana dashboards re: T263832, if you're still around [17:55:55] T263832: Major performance regression on mobile site associated with 1.36.0-wmf.10 - https://phabricator.wikimedia.org/T263832 [17:56:55] cdanis: I’m on my phone for the rest of the evening, I’ll see I I can help [17:57:14] if you remember offhand the name of the dashboard you were looking at for diagnosis, that'd be enough, I think [17:57:28] just to verify if the fix was successful [18:04:07] It’ll be the webpagetest drilldown next time the WPT tests run [18:04:25] I’ve just purged the enwiki articles for Facebook and Sweden which are among those tested [18:04:38] Done moving files around, wpt host looks healthy. [18:05:20] If the fix is successful in the drilldown we will see the metrics go back to their levels of a couple of days back, in particular the layout metrics in the chrome internals [18:05:35] dpifke: thanks! [18:06:26] I don’t know at what frequency the tests for this pages run but I imagine we’ll have a new run in the next couple of hours at the most [18:06:54] In the wepagetest drilldown dropdowns you want to select the emulated mobile option in the first drop down [18:07:00] The issue only affects the mobile site [18:07:13] Pick that, enwiki and then Facebook or Sweden [18:07:49] If you look back at least a couple of days ago you’ll see the issue appear soon after the next train deploy. Almost right away for the Facebook article [18:08:19] Sweden and Barack Obama got the issue later as it took longer for them to get an edit/be purged somehow after the train [18:08:40] Anyway I can look at all that later on [18:08:57] thanks, I think I got it – https://w.wiki/ddz shows the increase, at least [18:26:24] Lucas_WMDE: yep, exactly, and if this patch fixed the issue, it should go back down in the next test run [18:27:14] well, I think I’m going home before the next test run, but I’ll take another quick look later this evening :) see you! [19:08:32] metrics seem to have recovered :) [19:12:58] Lucas_WMDE: thanks! [19:58:16] These are the long tasks from the new mobile setup, it seems fixed but maybe not down to what it was before: https://grafana.wikimedia.org/d/Tw-s6iKMz/mobile-webpagereplay?viewPanel=3&orgId=1