[13:28:50] \o [13:28:56] nice and quiet in here [13:28:58] o/ [13:29:08] OK so I'm going through the query history on both servers to try to prove that they received contradictory insertions [13:29:13] But it's slow going [13:29:21] I'm looking through the usages of DB SLAVE / MASTER in abusefilter and out new revstore stuff now [13:30:10] $dbw = $this->getDBConnection( DB_REPLICA ); [13:30:11] There is not much in the beta SAL so I can't tell if db04 was legitimately the master at some point [13:30:12] DanielK_WMDE: RoanKattouw ^^ [13:30:14] LOL [13:30:16] Wel... [13:30:22] That would do it [13:30:22] Think I might have found it.... [13:30:34] Do we not have a mechanism in MW to prevent writes to DB_REPLICA connections?! [13:30:41] RoanKattouw: apparently not [13:30:45] it just does ->insert [13:31:13] https://github.com/wikimedia/mediawiki/blob/master/includes/Storage/SqlBlobStore.php#L231 [13:31:15] RoanKattouw: looking into fixing that at least for unit tests [13:31:22] ideally for everything. [13:31:50] there should be a NoWrite implementation of the db interface [13:31:56] Well that would do it [13:31:58] that refuses to accept calls to any writ methods [13:32:06] That's exactly the place where we would expect this, too [13:32:11] or infact... 2 separate interfaces.. [13:32:14] OK I'm going to file a couple tasks here [13:32:20] RoanKattouw: agreed [13:32:25] 1) We need all replica DBs to have writes disabled, in labs as well as production [13:32:31] 2) [13:32:45] 2) Trying to write to a DB_REPLICA handle needs to throw an exception [13:32:46] If we disable writes how could masters write to replicas? [13:32:51] O.o you can write to replicas?? [13:32:58] apergos: Yuuup [13:32:59] apergos: on the beta cluster yes [13:33:02] owie [13:33:05] Zppix: Replication is exempt from the no-writes flag [13:33:06] and within mediawiki yes, [13:33:11] Ah ok [13:33:20] 3) deployment-db04 is hosed and needs to become un-hosed [13:33:29] (I am not to smart when it comes to db) [13:33:55] (I wondered the same thing actually, so I looked up the documentation and it said that replication and admins were exempt) [13:34:33] I'm going to begin with depooling deployment-db04 so as to unbreak the beta wikis [13:34:37] what the heck is getting a db replica handle and trying to write to it anyways? [13:34:43] Then we probably need to reprovision it or something [13:34:48] apergos: The new MCR code :) [13:34:54] https://github.com/wikimedia/mediawiki/blob/master/includes/Storage/SqlBlobStore.php#L231 [13:34:56] oh dear... [13:35:03] Yeah... [13:35:13] So the text table's IDs diverged on the master and replica DBs [13:35:21] I know it's not really funny but I am lmao here [13:35:24] there could perhaps be a very simply phpcs sniff for that too [13:35:26] Then when the master tried to replicate its writes, replication died [13:35:30] of course [13:35:33] apergos: my bad. whoops. we need more eyes on this code... [13:35:52] well the good thing is it gets discovered in labs [13:35:54] apergos: If you're in #wikimedia-dev already, see my pastes in the backscroll there, you'll see the two servers disagreeing on the contents of the text and revision tables [13:36:03] apergos: RoanKattouw the great news is this issue is much simpler to fix than some of the other possible reasons for breaking that we were thinking about. [13:36:07] We also had revision IDs with different meanings on the two servers [13:36:11] Yes [13:36:12] eewwww [13:36:20] And it's not a chronology protection issue [13:36:20] RoanKattouw, addshore: i'm introducing allowWrite into the datbase base class. I'm also introducing a wrapper so writing to a replica connection will fail even if there is only one db and one connection. [13:36:22] yes I suppose it would all go pear-shaped pretty soon after [13:36:33] DanielK_WMDE: sounds good [13:36:34] doing the backread in the other channel [13:36:49] DanielK_WMDE: Yes that's good, and necessary to catch these things in dev. I'll file that task first so you can claim it [13:36:53] DanielK_WMDE: shall I fix the line or you? [13:37:15] RoanKattouw, addshore: thank you for helping with this! [13:37:18] addshore: i'll fix it. [13:37:52] Im glad i mentioned directly writing to replicas :P [13:38:15] Zppix: thank you, indeed :)+ [13:38:39] Zppix: I would have expected that to just fail HARD, not silently corrupt the database [13:38:51] I wonder if this possibly in prod as well [13:38:55] Possible* [13:39:06] RoanKattouw: will you also file a ticket to make sure replicates are not writable in production? This kind of thing can SERIOUSLY mess up the database :( [13:39:15] Zppix: yea, scary [13:39:33] Yes [13:39:38] See my 1-2-3 above [13:41:53] that was a fun little read [13:42:08] not easy to track down either [13:42:29] yep... several wrong leads [13:43:12] T183242 [13:43:13] T183242: DB handles obtained with DB_REPLICA should not allow writes - https://phabricator.wikimedia.org/T183242 [13:43:16] (DanielK_WMDE) [13:44:54] DanielK_WMDE: i can go thru the logs and add them to a paste and then put them into the task if you'd like [13:45:22] Zppix: that would be awesome, thank you very much!" [13:45:49] DanielK_WMDE: you dont happen to have timeframes do you on the chat? [13:47:22] Zppix: timeframe of our chat or of the issue? [13:48:26] Chat [13:48:42] 13:28 till now in here [13:48:49] Ok [13:49:20] 12:29 -> 1:28 in -dev [13:49:24] *13:28 [13:49:30] ...that's UTC [13:50:47] Yes! [13:50:57] UTC is the ONLY timezone :D Also, I am currently in UTC apparently [13:51:16] addshore: so, what do you think? do we still want try to merge this before christmas? I'd at least want some feedback from legoktm. But other than that... [13:53:41] Yeh, I'm all set to merge it again over the christmas period [13:54:00] I think we want to try and get the db thing for not allowing writes to replicas merged first [13:54:04] and the fix of course [13:54:19] but I am very keen to have it on beta for as long as possible [13:55:24] addshore: the fix is already up [13:55:41] i'll submit a patch for preventing writes to actual replicas today [13:55:55] for unit tests, it's going to be more tricky [13:57:50] hmmm, https://gerrit.wikimedia.org/r/#/c/399174/4/includes/Storage/SqlBlobStore.php I was going to do it in a seperate patch! [13:59:06] OK I can confirm that all production replicas are read-only [13:59:25] RoanKattouw: *phew* thank you :) [13:59:38] (From now on no more discussions in -dev) [13:59:47] addshore: why? [13:59:59] i don't care much. just seemed pointless [14:00:06] * DanielK_WMDE has to go to a meeting ofr an hour now [14:08:04] addshore: RoanKattouw added chat logs [14:15:56] OK I have filed T183252 [14:15:57] T183252: Unbreak replication in beta labs - https://phabricator.wikimedia.org/T183252 [14:16:08] I'm going to first try to put the master in a consistent state [14:16:26] Then I'm going to try depooling the replica, which I hope will bring the beta wikis back up [14:34:30] addshore: Could you help me by reviewing+merging https://gerrit.wikimedia.org/r/399182 ? [14:35:24] RoanKattouw: in a meeting right this second [14:35:39] oh wait, I have meetings from now for the next 2.5 hours :/ [14:35:40] Anyone else around to do that? [14:40:03] RoanKattouw: I can do it now [14:40:07] Thanks [14:41:13] oh, RoanKattouw jenkins gave it a -1 [14:41:18] Ugh really [14:41:53] Pfft, coding style [14:41:54] Fine [14:42:50] Amended, let's see if it likes it now [15:22:30] / [15:28:36] The Parser is brokenised again [15:28:48] Take a look at what should be a simple example [15:29:11] https://en.wikivoyage.org/wiki/Special:ExpandTemplates [15:29:33] And try doing *{{listing-x|content=line1|content2=line2}} [15:30:10] Rather than wrapping the DIV INSIDE the List item as expected it, it decides to terminate the div in the wrong place to close the List item [15:30:23] [15:30:47] It would sometimes be nice if the parser behaved in a logical manner [15:31:55] [15:32:14] its mainly due to the way the parser code is written in the template [15:32:15] subbu: ---^^ [15:32:39] but I still don't know what it is, or how to fix it [15:32:51] Also I shouldn't have to FORCE paragrpahs [15:33:28] I'f use BDI runs directly 'albiet' with blank lines the parser collapse what should be seperated paragraphs into a SINGLE block [15:33:42] which isn't waht was desired either [15:33:42] it would be easy enough to add template support for paragraphs [15:33:57] The problem isn't the template [15:34:09] The problem is that the parser has no EXPLICT break markers [15:34:38] I only ever use {{Clear}} and
tags myself [15:34:43] meaning things that shouldn't have to be forced, have to be done in raw HTML [15:34:57] which breaks things on a spectacular scale [15:35:06] because the parser thinks it knows better [15:35:27] I am convinced the breakage is realted to the fact this template uses bdi tags [15:35:31] ShakespeareFan00, RoanKattouw Dysklyver ya .. in line-based-wikitext constructs lists, it processes things on a line-by-line basis without regards to html structure. [15:35:43] subbu [15:35:55] So you are saying the parser is too stupid to cope? [15:35:56] so had to add https://www.mediawiki.org/wiki/Help:Extension:Linter/multiline-html-table-in-list recently [15:36:22] "parser is too stupid to cope" = i woundlt say it was smart [15:36:37] it does a very limited function well [15:36:52] when Parsoid becomes the default parser, we can make more structural changes .. but, we have to tread carefully there as well ot make sure we don't break existing pages. [15:37:02] https://www.mediawiki.org/wiki/Help:Extension:Linter/multiline-html-table-in-list#Why_does_this_happen? [15:37:33] That doesn't help :( [15:37:55] It gives an explanation but doesn't explain how to fix it. [15:38:15] https://www.mediawiki.org/wiki/Help:Extension:Linter/multiline-html-table-in-list#Solution for that particular case [15:38:59] your example is not a multiline-html-table .. but a multiline-div-construct .. so, something similar should apply, i think. [15:39:05] subbu: That solution did not work in the particular instance [15:39:26] Perhaps you could look at the relevant template and produce a fully compliant version? [15:39:53] "we are right now in a deployment freeze" yes its very cold [15:40:10] https://en.wikivoyage.org/wiki/Template:Listing-x being the relevant template [15:40:38] The reason it was under development was to replace the Listing template which can't cope with the multiple paragraphs [15:41:23] because it's using bdi tags which can't easily cope with implied P tags inserted by the Parser or Tidy [15:41:49] Personally, I am very inclined to just ask someone to enable the new parser anyway [15:42:00] and wait for the screams [15:42:06] when EVERYTHIGN breaks [15:42:22] ! [15:42:23] yes i was meaning to ask, why bdi tags? why not span [15:42:26] right now, there isn't a wikitext-native solution that lets you embed multiline-structures in a list item .. you would have to use html list item in those case as far as i can tell .. but, maybe consult with tpl editors on enwiki? [15:42:43] they may have some suggestions for you. [15:42:58] enwiki does not use these tags (not that i have ever seen anyway) [15:43:16] subbu: When I moved the positining of the closing LI UL pair... [15:43:24] It did what was expected. [15:43:30] So I call this as a parser bug [15:43:53] Perhaps you could test it against Parsoid vs Tidy? [15:43:56] what do you mean by move the positioning of the closing li/ul pair? [15:44:31] subbu: Have you been able to generate the raw HTML from Special:ExpandTemplate [15:44:50] yes, i have it. [15:45:13] https://gist.githubusercontent.com/subbuss/01476a4556d2539c3867dbd5793c1c6a/raw/62f32a46d8e6793e7df3d6af1bb1d4712df994b7/gistfile1.txt [15:46:08] Try expanding - *{{Listing-x|content1=Block1|content2=Block2}} [15:46:48] The parser tries to close the list item and list BEFORE it opens the Block2 paragraph [15:47:09] Ignoring the fact that it's (a) inside a div at that point... [15:47:28] right, i know .. it is because the parser doesn't understand html5 structures. [15:47:57] is the parser techincally outdated then? [15:47:58] Okay [15:48:11] That's why it's being replaced :) [15:48:17] makes sense [15:48:26] And a lot of old clever stuff will explodeify in [15:48:37] maybe anomie knows some tricks to get the templating you want to do? [15:49:07] anomie knows everything :) [15:49:12] Stuff will break because the new parser is a LOT tighter on what it will accept [15:49:13] ShakespeareFan00, Dysklyver note that the new parser will not magically fix things becaues there are millions of pages that depend on the current parser's functioning.. [15:49:26] subbu: That I know [15:49:30] subbu, yeah that will be fun [15:49:53] alright, i have to prepare for a meeting in 10 mins ... sorry i don't have anything more constructive for you right now. [15:50:00] 4000 or more reasons here - https://en.wikivoyage.org/wiki/Special:LintErrors [15:50:02] XD [15:50:20] Also the new parser will break pages on something really simple [15:50:29] like mismatched '' '' and ''' ''' formatting [15:50:34] ROFL [15:50:50] well thats just awesome [15:50:51] i don't know what you mean by that .. [15:51:15] subbu: '' '' and ''' ''' translate to tags [15:51:29] If they are misnested, boom :( [15:51:38] we can have fun fixing over a decade of formatting to make it html5 able, just before html 6 breaks it all again xD [15:52:16] Part of the problem is that a lot of this tightning up should have been done years ago [15:52:33] But no-one seems to care about fixing real bugs at times [15:52:39] ShakespeareFan00, no more boom than what goes boom now .. do you have an example where that is not the case? [15:53:18] https://species.wikimedia.org/w/index.php?title=Special:LintErrors/missing-end-tag&dir=prev&namespace=0 [15:53:40] Most of these entries are ENTIRLY due to mismatched '' '' or ''' ''' [15:53:56] that is just reporting errors in case you want to fix it .. it won't break a page if yo udon't fix it. [15:54:04] that is why it is in the low-priority category. [15:54:08] Okay [15:54:24] https://www.mediawiki.org/wiki/Help:Extension:Linter#Goal:_Clean_markup [15:55:12] subbu: https://species.wikimedia.org/wiki/Special:LintErrors/multiple-unclosed-formatting-tags [15:55:23] That says it has unclosed tags... [15:55:26] I can't see any [15:56:31] it has one page listed. it probably has a foo and some long text instead of foo and some long text [15:56:38] https://species.wikimedia.org/w/index.php?title=Wikispecies:Village_Pump/Archive_41&action=parsermigration-edit&lintid=29333 will show you the net result. [15:57:38] you can see that the error is in the section titled "Speedy creation of new species articles" .. [15:57:42] anyway, i gotta run now. [16:08:37] Question what was the tool subbu was using? [16:11:35] ShakespeareFan02: You mean Special:LintErrors, or action=parsermigration-edit ? [16:11:46] Actually I guess they're part of the same tool [16:11:48] Didn't know that was a thing [16:11:57] but not linked from the LintErrors report ;) [16:12:15] There are some docs at https://www.mediawiki.org/wiki/Help:Extension:Linter [16:12:26] Which is linked from the "Help" link on the main Special:LintErrors page [16:13:01] On individual report pages (e.g. https://species.wikimedia.org/wiki/Special:LintErrors/html5-misnesting ) the Help link points to documentation about that particular type of error [16:13:43] RoanKattouw: https://gerrit.wikimedia.org/r/#/c/399208/ [16:14:46] ShakespeareFan02: There's also https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/FAQ which has a bunch of info about both LintErrors/ParserMigration and the background for why we have them (the upcoming Tidy->Remex migration) [16:16:09] RoanKattouw: the wrapper for the master connection will be a lot more tricky. for one thing, I'm not sure where it should go in LoadBalancer->mConns. [16:16:23] RoanKattouw: also, that wrapper would look much like DBConnRef. We'll probably want a DBConnProxy base class. [16:29:18] come back Phab, I miss you [16:29:56] looks like it promptly complied [16:31:12] * Elitre hugs the screen [16:49:52] This is apparently misnested - https://species.wikimedia.org/wiki/Wojciech_Niedba%C5%82a [16:49:56] I can't see it [16:49:59] Can you? [16:56:20] It would be HELPFUl if LIntErrors gave the PRECISE line where it chokes [17:24:14] And I really don't see HOW this https://species.wikimedia.org/w/index.php?title=Template:User_Autopatrolled/it&oldid=3557098 [17:24:18] this should have worked [17:24:32] given that small and span CAN be embedded in each other per [17:24:42] https://html.spec.whatwg.org/multipage/dom.html#phrasing-content-2 [17:45:01] ShakespeareFan02, reg. Wojciech page .. that one was hard to spot .. but * {{aut|Olszanowski Z., [[Wojciech Niedbala|Niedbała, W.]] Checklist of oribatid mites (Acari, Oribatida) of Poland. Data basis FAUNA EUROPAEA (http://www.faunaeur.org) ed. W. Bogdanowicz, Polish Academy of Sciences, Warszawa (ca 600 records) ????? .. [17:45:15] that transclusion is unclosed and throws everything for a loop after that. [17:45:55] We need a bot [17:48:16] subbu: I can't seem to find the releveant line? [17:48:22] Precise link please? [17:48:25] If possible [17:49:28] Thanks [17:49:39] subbu: [17:49:44] And apologies [17:54:45] no worries .. it is hard to spot. [17:56:16] 3 more to go [17:56:37] https://species.wikimedia.org/w/index.php?title=Template:TakeNote&oldid=3104351 [17:56:49] Not sure why this saying there's misnesting... [17:57:01] There's ONLY one pair of tags... [17:57:50] Unless is considered misnesting. [18:19:18] sbbu: Is there a way to see what the new parser thinks it's generating? [18:19:39] subbu: Special:ExpandTemplates is presumably working on the current template? [18:19:45] *current one [18:20:15] sorry in a meeting .. will take a look later. [18:20:46] Thanks [18:46:10] ShakespeareFan00, use ?action=parsermigration-edit on the page [18:46:22] it will show you old (Tidy) output and new (RemexHTMl) output side by side [18:46:29] Yes.. [18:46:32] I did that [18:46:33] you can make edits and update the preview [18:46:41] And didn't see anything different in rendering [18:46:55] I am wanting to see the RAW html generated on either parser [18:47:06] then, there are no problems ... linter is being pedantic (since it doesn't know that things won't always affect rendering). [18:47:14] ah .. raw html. [18:47:30] https://species.wikimedia.org/wiki/Wikispecies:Village_Pump/Archive_42 [18:47:31] what is an example page where you are finding this? [18:47:32] ok. [18:47:47] I am not seeing an obvious rendering error [18:47:57] https://species.wikimedia.org/wiki/Template:TakeNote [18:48:01] Also [18:48:10] and I'm not seeing ANY tag mismatching [18:48:14] on the latter [18:48:21] Unless it's buried inside a module [18:50:14] the {{archive top}} at the top of the page that leaves an open
tag is throwing off the linting code and it is just saying that there is an error somewhere on the page :) [18:50:45] but, one option is to remove that temporarily and lint it .. i can do that locally .. hold on. [18:52:46] The remainder are here - https://species.wikimedia.org/wiki/Special:LintErrors [18:52:54] I'm reaching the end of my comptence [18:52:57] XD [18:53:09] But will continue to match '' '' tags [18:53:24] ShakespeareFan00, https://species.wikimedia.org/w/index.php?title=Wikispecies:Village_Pump/Archive_42&action=edit§ion=26 .. has the error .. looks like the langswitch template? [18:53:42] That's where I am seeing some issues as well [18:53:45] on other pages [18:53:51] Langswitch calls a module [18:54:08] so I am not sure whats gone wrong [18:54:58] https://species.wikimedia.org/wiki/Template:Inconclusive has simmilar problems with Langswitch and others [18:55:08] the problem isthe newline between "Hello everyone," and the following paragraph. Looks like the tpl wraps that content in a tag [18:55:15] but the newline introduces a paragraph and breaks the [18:55:29] that is what linter is complaining about. [18:56:41] https://species.wikimedia.org/w/index.php?title=Template:LangSwitch&action=edit .. has the wrapper around content. [18:57:15] but it doesn't know what kind of content can be generated there .. so, paragraphs will break the [18:57:24] i don't know what the solution there is. [18:57:57] alright, got to run to another meeting now :) [18:58:07] one of those days today. [18:58:41] removing all line-feeds helps [18:58:57] Don't know why it would be THAT simple but it works [18:59:58] removing line feeds eliminates the paragraphs and so the bdi tag doesn't get broken up anymore. [19:00:06] Quite [19:00:26] This is why the parser.wikitext needs explicitly specfied break markers IMO [19:12:56] It seems like some kind of annotation tool is running in Firefox on Wikipedia. Sometimes it work, sometimes it doesn't. Anyone that has any knowledge on this? [19:13:29] I thought I had a plain Firefox version, and not a developer thingy, but now I start to wonder… [19:13:31] hard without steps to reproduce [19:13:51] like which "Wikipedia" and how to trigger it :) [19:15:12] Actually, I have no clue! It sometimes emerge and I have absolute no idea what I do… [19:16:54] There are no weird looking extension, but there are some notes that annotation now is part of Firefox. How to access it is not described anywhere. [19:17:14] Its a firefox thing then is what it sounds like [19:17:34] Nah, its a W3 standard [19:17:39] But it is new [19:19:44] jeblad: firefox 57 I assume? is it related to https://getpocket.com/ ? [19:20:44] or do you mean the annote gadget on commons? https://commons.wikimedia.org/wiki/Help:Gadget-ImageAnnotator [19:20:45] That is nothing I have enabled as far as I know [19:21:22] Nope, not commons. [19:22:33] When it emerge it looks like the article popup, but a one-liner. One linked item is "annotate". [19:23:10] Usually it shows up on either Mediawiki or Enwiki [19:24:17] I guess I'm pushing some key combo that trigger it, and it could be something made just for testing. [20:52:09] ShakespeareFan00, you should change the to a
in https://species.wikimedia.org/w/index.php?title=Template:Distinguished_author_2017-05&action=edit&lintid=26263 .. if something wraps multiple paragraphs to change formatting, that ought to be a div, not a span. [21:00:41] subbu: Noted, but the template concerned is protected at present. [21:00:53] And I have made a simmilar change elsewhere [21:01:50] k [21:02:22] If you have the ability to edit protected page, or leave a talk page note feel free [21:02:40] because there are only 23 pages to go... [21:02:55] on the high priorty list... and one of them is effectively a talk page