[08:15:49] apergos: Setting stuff for profiling in LocalSettings.php is the "right" way indeed. I've optimised away any difference or distance between loading StartProfiler and LocalSettings in 1.31, and removed in 1.32 to save a file op on every req. [08:16:20] ah Krinkle! great, and now I only have two questions (or maybe it's three) left [08:16:30] apergos: In addition to LocalSettings being at the same point where StartProfiler used to be, one may want to migrate to a php auto-prepend file instead, so that you capture the stuff before LocalSettings loads as well. [08:16:58] good point; for my purposes it doesn't matter because it's all about long-running maintenance scripts [08:17:03] k :) [08:17:05] but it would be more 'proper' [08:17:30] the wgProfiler config is also not the same as "starting the profiler", maintenance scripts still dont start it by default. [08:17:55] so my question s, do you have it working to display output (or write json to a file, that's ok too, maybe better) for profilng *maintenance scripts*? because I had to do some crappy hacks, so surely I'm doing it wrong [08:18:10] However if your StartProfiler.php script explicitly calls xhprof_enable the yes, it will enable everywhere. If like before you only declare which Profiler to use, then it starts on web req only, with the --profile option, like before, being the way to have maint enable it. [08:18:12] right, you have to pass --profiler=text or something [08:18:43] yeah, I don't remember the exact option format, but something like that indeed, and hasn't (shouldn't have) changed [08:18:46] so let's say I want it to write serialized php (I guess that's what it is) to a file in some dir [08:18:59] how do I make that happen, for profiling a maintenance script, any notion? [08:19:32] (I couldn't get it to just write plain text output at the end of the script either, but it turns out that's less useful so I don't care now) [08:19:49] I'm not familiar with any php-serialised related output format for MW profiling. [08:19:49] s/how do I/how do I properly/ :-P [08:19:54] ok well uh [08:20:02] Did that used to exist? [08:20:04] it's some sort of looks kind of like json but not really output [08:20:32] The ProfilerOutput classes can be used to format the data captured by xhprof [08:20:49] a:2237:{s:72:"Wikimedia\Rdbms\Database::selectField==>Wikimedia\Rdbms\Database::select";a:2:{s:2:"ct";i:9;s:2:"wt";i:10217;}s:104:"Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}==>Wikimedi [08:20:49] a\Rdbms\LBFactory::getChronologyProtector";a:2:{s:2:"ct";i:2;s:2:"wt";i:27;}s:35:"SqlBagOStuff::getMaxDateTime==>time";a:2:{s:2:"ct";i:2;s:2:"wt";i:0;}s:55 [08:20:50] etc [08:20:51] The default is basically a bullet list sorted by percentage of inclusive self time [08:21:28] that's the beginning of one such pile of output and that's fine, I'm just sure that what I had to do (hacking around in includes/libs/Xhprof.php) can't be right [08:21:29] apergos: OK, I guess that is ProfilerOutputDump [08:21:34] I didn't know it before today, but looks like that would do it [08:21:37] gotta go [08:21:48] ok, well thanks for your help [08:22:38] I'll look at how those classes are used later, hopefully that will clear it up [10:00:43] anarcat: TIL there is the --show-legacy option in the new facter that shows you the old ones ;) [13:00:27] volans: ah yes of course :) [13:08:39] hey guys, why am I seeing you are being logged out when I logged out in zhwp? seems a bug, shouldn't it be in zh? just wondering only [13:47:30] On a contributions page, a user with rollback will only see rollback links if they can, in fact, rollback the page: most recent contribution, no editing restrictions, more than one author. Is that information exposed in the user interface? [13:50:38] McJohn: what do you mean by “exposed in the user interface”? [13:50:45] I would say it’s exposed via the presence or absence of the link :) [13:50:55] but it’s not exposed anywhere else (e. g. an API) as far as I’m aware [13:51:06] Well, not if you don’t have rollback… [13:51:49] And querying for info&inprop=intestactions is inefficient [13:51:55] you mean, you want the information whether a rollbacker could rollback, even if the current user is not a rollbacker? [13:52:14] Whether the user can edit the page [13:56:49] Not sure if it’s just rollback or if the interface knows IsProbablyEditable (rather quickly) for every item Special:Contributions [20:42:34] Anything going on? I gets errors running pywikibot against nowiki [20:43:10] jebald, what error? [20:43:18] jeblad, ^ [20:44:31] raise ConnectionError(e) [20:44:33] requests.exceptions.ConnectionError: HTTPSConnectionPool(host='no.wikipedia.org', port=443): Read timed out. [20:44:49] socket.timeout: The read operation timed out [20:44:53] jebald, accessing what page? loads fine for me [20:44:59] jeblad: ^ [20:45:11] jeblad: pywiki locally? Is it working in your web browser? [20:45:18] Yes, it loads here too [20:45:24] Loads fine [20:45:52] can you curl no.wikipedia.org successfully? [20:46:03] Could be flaky line, there have been a lot of thunderstorms today in Norway [20:46:46] are you saying it's a one-off or a recurring problem? [20:47:05] I'll let the bot run for a while and see if it gets worse [20:47:44] About ten errors wile pywikibot loads pages for processing [20:48:47] Wild guess the bot loaded about 500 pages before starting processing? [21:03:07] Seems like pywikibot is processing without interruption, it was the initial loading that got interrupted [22:15:03] I've created a banner which works well, except on plwiki it gives me a "Content Security Policy" warning. When I use my banner on plwiki, I notice the request that site sends to: https://pl.wikipedia.org/w/load.php?debug=false&lang=pl&modules=WDsearch . Why would this be? [22:18:57] notconfusing, how do I trigger this banner? [22:19:12] https://pl.wikipedia.org/w/index.php?title=Wikipedia:Strona_g%C5%82%C3%B3wna&banner=CS_Test5&force=1 [22:19:31] Content Security Policy violation detected! Tried to load something from https://en.wikipedia.org/w/index.php?title=MediaWiki:Wdsearch.js&action=raw&ctype=text/javascript. [22:19:46] Interesting, it seems semi-surprising that plwiki would block loading from enwiki via CSP [22:20:09] Exactly. It actually looks like it's a bug with all CentralNotice banners on plwiki, as far as I can tell. Even random banners that aren't mine trigger this warning. [22:20:52] default-src *.wikimedia.org *.wikipedia.org *.wiktionary.org *.wikisource.org *.wikibooks.org *.wikiversity.org *.wikiquote.org *.wikinews.org www.mediawiki.org www.wikidata.org *.wikivoyage.org data: blob: 'self'; script-src *.wikimedia.org 'unsafe-inline' 'unsafe-eval' 'self'; style-src *.wikimedia.org data: 'unsafe-inline' 'self'; [22:21:11] Reedy, ^ [22:21:20] I'm not up to speed on CSP policies around here [22:21:38] me neither, but i thought I should report it. [22:21:44] possibly [22:21:58] Is CSP actually blocking it? [22:22:00] Or just complaining? [22:22:07] Krenair: I get even more warnings due to my global.js on that page. They're all enwp scripts though. [22:22:40] Content Security Policy: The page's settings blocked the loading of a resource at https://en.wikipedia.org/w/index.php?title=MediaWiki:Wdsearch.js&action=raw&ctype=text/javascript ("script-src"). [22:22:41] in console [22:22:46] when following notconfusing's link [22:23:45] which suggests to me it actually got blocked ? [22:24:45] rather than just being a report thing [22:24:51] there's a great big alert when you follow that link too [22:26:23] Yeah, did we accidentally switch CSP on in enforcing mode? [22:29:16] When I throw the CVN overlay script on, that still only produces report-only CSP warnings [22:29:36] I'm not seeing CSP logs on logstash, only csp-report-only. [22:31:13] Interesting...When I go to the link with the banner, I get the same blocked CSP messages, a report-only message from the CVN overlay, and a block message for the overlay in the console [22:32:51] And CSP reports haven't spiked recently, they're about normal. [22:37:49] wgCSPHeader is false on plwiki (as expected). The policy as set by nginx (?) looks sane. [22:37:55] (Did we used to set a default policy?) [22:43:08] Shouldn't the csp policy be: default-src 'self' blob: data: *.wikimedia.org *.wikipedia.org *.wiktionary.org *.wikisource.org *.wikibooks.org *.wikiversity.org *.wikiquote.org *.wikinews.org www.mediawiki.org www.wikidata.org *.wikivoyage.org data: blob: 'self'; script-src *.wikimedia.org 'unsafe-inline' 'unsafe-eval' 'self'; style-src *.wikimedia.org data: 'unsafe-inline' 'self'; [22:43:09] ? [22:43:10] Somewhere an extra header is being set http://dpaste.com/2A3Y8CD [22:44:43] And only with the &banner=CS_Test5&force=1 in the URL, no matter which wiki [22:45:27] Going to https://en.wikipedia.org/wiki/Main_Page?banner=CS_Test5&force=1 throws a bunch of CSP alert()s from various user scripts [22:45:41] oh [22:45:43] that's why [22:45:44] script-src *.wikimedia.org [22:45:47] that's in the header [22:46:17] so only a script on a wikimedia site can executed. [22:46:18] https://csp-evaluator.withgoogle.com [22:47:03] Weird, as AntiComposite points out the CSP header isn't there if you don't have the ?banner= parameter in the query string [22:47:23] RoanKattouw: Set by JS? [22:47:38] Oh! Set by CN? [22:47:45] I know it has some stuff in it. [22:48:02] I think that's where it must be coming from [22:48:06] A banner written to require some stuff might work fine on meta but break outwith *.wikimedia.org. [22:48:53] *.wikipedia.org [22:48:53] se.wikipedia.org is known to host JSONP endpoints which allow to bypass this CSP. [22:48:55] > var_dump($wgCentralNoticeContentSecurityPolicy); [22:48:55] string(335) "default-src *.wikimedia.org *.wikipedia.org *.wiktionary.org *.wikisource.org *.wikibooks.org *.wikiversity.org *.wikiquote.org *.wikinews.org www.mediawiki.org www.wikidata.org *.wikivoyage.org data: blob: 'self'; script-src *.wikimedia.org 'unsafe-inline' 'unsafe-eval' 'self'; style-src *.wikimedia.org data: 'unsafe-inline' 'self';" [22:49:17] paladox: You mean se.wikimedia.org? [22:49:24] In any case, yes, *.wikimedia.org is problematic [22:49:29] nope, https://csp-evaluator.withgoogle.com [22:49:39] says that [22:49:46] if you whitelist *.wikipedia.org [22:55:33] RoanKattouw: You filing a task? Should I? [22:55:44] I'm not, you should [22:55:52] I'll continue digging into when/why this changed [22:56:57] // FIXME: as soon as I80f6f469ba4c0b60 is available in core, get rid [22:56:57] // of $wgCentralNoticeContentSecurityPolicy and use their stuff. [22:57:01] Written in March 2018 [22:57:04] * James_F sighs. [22:57:14] Did we change something in CN recently? [22:57:39] (It was merged in May) [22:58:24] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/CentralNotice/+/wmf_deploy landed last Monday [22:58:33] Everything I've found so far (creating the config var and using it) dates to Apri 2018 [22:59:47] https://phabricator.wikimedia.org/T225261 [23:00:08] hello [23:00:27] i have updated a query that used rev_user_text to use the new actor table, but now it's a lot slower [23:00:53] what are the best practices to make it faster? [23:03:08] Depends what you're doing [23:03:14] SigmaWP: Can you pastebin / link to the query before and after the actor table change? [23:03:43] https://pastebin.com/FszvjZtq is the new query [23:03:55] James_F: I can find no reason why this wouldn't have been working this way since April 2018, but it's conditional on the banner parameter being present in the URL. It doesn't appear to be used for regular page views, even regular page views with banners on them [23:04:05] i basically added a join and did s/rev_user_text/actor_name/ [23:04:29] hello mark [23:05:53] RoanKattouw: Ooh. Interesting. [23:05:57] SigmaWP: Do you have the old query handy, for convenience? [23:12:15] RoanKattouw: https://pastebin.com/BAHHabfS [23:22:57] SigmaWP: looks like you are filtering by one table in the JOIN and ORDERing by another which is usually a perf issue - you should eliminate the join. Seems like you are only ever filtering for a single user, so you should first resolve that user name to an actor ID in a separate query, then use this actor ID to filter by rev_actor. [23:23:34] Oh. [23:38:30] mszabo-wikia: doing queries to get data ahead of time isn't very compatible with my current program flow [23:38:35] are you sure there's not another way [23:39:52] you could just make it right before executing this revision query [23:40:39] do you have a link to the program? [23:41:45] i'm constructing the query before i actually have an sql session [23:42:19] so i'm basically assuming that i already have all the constant values i need before i make an sql connection [23:42:44] would a sub query work instead? [23:42:59] probably, but that would be executed for every single row [23:43:17] may be worth a try here [23:43:51] if it's a constant where... it'll probably just cache the result [23:43:53] if that doesn't work out you'll probably have to rearchitect your SQL query building approach [23:44:13] Though, if you're always running it against the same wiki... [23:44:19] Just get the actor id once and profit [23:44:27] tru [23:53:05] nice [23:54:19] is that a thing? caching subqueries if the where clause is constant? [23:55:22] well, it caches results for sure [23:55:26] Have a look at it in EXPLAIN [23:56:11] ERROR 1345 (HY000): ANALYZE/EXPLAIN/SHOW can not be issued; lacking privileges for underlying table [23:56:29] https://tools.wmflabs.org/sql-optimizer [23:56:33] nice [23:57:02] Query error: ANALYZE/EXPLAIN/SHOW can not be issued; lacking privileges for underlying table [23:57:03] lmao [23:57:14] oh, because i put explain in it