[10:00:36] I'm getting an expected error on testwiki [10:00:50] Your edit has been rejected because your client mangled the punctuation characters in the edit token. The edit has been rejected to prevent corruption of the page text. This sometimes happens when you are using a buggy web-based anonymous proxy service. [10:01:17] I've got fairly aggressive JavaScript blocking, but don't see any JS blocked for testwiki [10:01:47] I'm trying to enter a heading ending in the equals sign [10:02:29] Josh_Parris: could you file a bug with the relevant details? [10:02:41] I'd love to help but it's 2 AM here [10:02:41] sure [10:02:48] thanks :) [10:02:50] go to bed then [10:03:03] you can't make me! [10:37:40] https://bugzilla.wikimedia.org/show_bug.cgi?id=57411 [13:45:45] my second factor token is considered wrong :( [13:52:38] fale: dive? [13:52:46] *dove [13:52:50] Nemo_bis: wikitech [13:53:05] hai attivato il 2f? [13:53:39] perché generale si può ignorare quel campo [13:56:15] Nemo_bis: lo avevo attivato tempo fa, perchè mi era stato richiesto per qualcosa :/. Adesso dopo il mass-problem degli username e password ho cambiato la pass ma non ho toccato il token e adesso mi dice che è necessario [16:19:27] aude: anything needed for https://bugzilla.wikimedia.org/show_bug.cgi?id=57410 (unable to login on chechen wiki due to the slashes in translated special pages) [16:26:56] greg-g: i think localisation update? [16:27:14] * aude is quite confused [16:30:46] has this really been broken for months? if so, people logging in elsewhere via central auth? [16:30:50] aude: just checking if anything more than that is needed [16:30:52] no idea [16:31:00] I'll ask chris when he gets in [16:31:27] i don't know how /if to do localisation update on demand, but it should run in a few hours [16:32:04] then, question... can translate extension disallow slashes in special page names? or jenkins block such thing? [16:32:18] or is this a bug? [16:32:19] aude: that'd be a good follow up [16:32:28] yeah [16:32:42] yes, I think "translations breaking login" is pretty bad and shouldn't be allowed [16:32:51] yeah [16:35:05] aude: so, apparently this is a known thing, robla just said from like 2011 or so when it last happened :) [16:35:31] he *thinks* it's not something that can be broken/fixed from *just* a translation update on translatewiki [16:35:37] that a code change is needed [16:35:44] but, he's without laptop right now ;) [16:36:30] mmm [16:37:09] it's the spcial page aliases [16:37:10] greg-g: translating special pages names is definitely possible on TWN [16:37:15] i don't know how they get updated [16:37:17] greg-g: what's not possible is translating namespace names [16:37:25] but they are translated in translatewiki [16:37:30] alright then [16:37:33] as aliases [16:37:44] I'll take your word for it, since you all are on a computer where as robla isn't :) [16:37:45] and were introduced with 1.22wmf10 [16:38:12] login work with ce worked on wmf9 but not wmf10 [16:38:27] * greg-g nods [16:38:46] someone alreaady fixed the translation, so i hope it's just automatic :) [16:39:42] * greg-g is asking reedy about it in -operations [16:39:47] guess I could have here, oh well [16:40:47] so, we have to wait until the l18n update that happens overnight [16:40:48] :/ [17:07:09] greg-g: The deployments page says there's a lightning deploy today? I thought we had no deploys on Friday. [17:07:18] * anomie would like to fix https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Section_edit_links_are_appearing_on_diffs [17:08:03] anomie: might be related? https://bugzilla.wikimedia.org/show_bug.cgi?id=57402 [17:08:25] likely [17:08:30] MaxSem: ^ [17:08:43] YuviPanda: Could be. See if https://gerrit.wikimedia.org/r/#/c/96999/ fixes it [17:09:33] YuviPanda, if current app is unaffected it doesn't need to be deployed? [17:09:39] anomie: see the time [17:09:49] anomie: stupid UTC/PST/Daylight savings [17:10:11] greg-g: Oh, it's showing Friday based on UTC rather than PST? [17:10:20] anomie: yeah [17:10:34] MaxSem: the current app is unaffected, and I don't know why. That's kinda scary. Checking out why [17:10:38] * anomie should pay more attention to the column headers [17:11:12] anomie: it's dumb though, regardless [17:11:25] it should be pinned to pacific daylight/standard [17:15:01] anomie: does that mean this will be broken until the next train roll out? [17:15:54] YuviPanda: Well, we could fix it Monday. Unless greg-g approves an earlier deploy. [17:16:52] hrmmm [17:17:41] it should've broken the current app, and it hasn't and I've no idea why. I'm investigating. [17:17:50] * YuviPanda also looks at greg-g with puppy dog eyes [17:17:56] you guys suck [17:18:21] it stops development of the new wikipedia app I'm on until it gets fixed tho :( [17:18:34] right, so, the fix is a 'simple' revert, right? [17:18:58] ie: no new code is needed to fix this if we do the revert today (and wait for any new code until Dec 2nd) [17:19:27] yup [17:19:44] anomie: concur? :) [17:20:14] greg-g: yup, this is just a revert [17:20:16] greg-g: Yes, it's just a revert. MatmaRex or someone might at some point try to redo the reverted patch in some other way, but this just reverts [17:20:18] * MatmaRex broke the sites agaaaain [17:20:29] MatmaRex: dammit, I haven't done that even once! [17:20:50] k [17:21:00] MatmaRex, it's not broken uless icinga tells you otherwise [17:21:01] well, I'm ok with that [17:21:24] reverting that bit so that yuvi can work and people stop complaining [17:21:34] who'll deploy it? [17:21:57] I can [17:22:01] MaxSem: heh, i think i'll make that my motto [17:22:53] MaxSem: thanks, one sec, lemme check with reedy on timing [17:22:58] * greg-g goes to -operations [17:26:28] greg-g: Re dates on the deployments page, what would you say to having the module insert (Thu) before the SF time if the UTC day is different? [17:29:46] ?? [17:29:58] Reedy: https://gerrit.wikimedia.org/r/#/c/96999/ [17:30:11] to fix https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Section_edit_links_are_appearing_on_diffs and https://bugzilla.wikimedia.org/show_bug.cgi?id=57402 [17:30:51] 4 or 5? [17:31:08] Reedy: both [17:31:24] yeah, looks like the thing it's reverting is in both [17:31:45] * Reedy glares at MatmaRex [17:31:51] * MatmaRex hides [17:32:18] :) [17:32:44] Won't merge automagically to 4... [17:32:52] Reedy: agh [17:32:55] Reedy: stupid release notes [17:33:06] they were different for a while [17:33:21] want me to submit a patch to wmf4? [17:33:34] Sure [17:33:39] k, one sec [17:38:58] Reedy: https://gerrit.wikimedia.org/r/97056 [17:39:46] anomie|meeting: MaxSem MatmaRex greg-g ugh, that revert doesn't actually seem to fix the mobileview bug [17:39:57] weee [17:40:17] YuviPanda: that sounds unlikely [17:40:27] MatmaRex: not fixing the bug? [17:40:34] yes [17:40:43] MatmaRex: talking about https://bugzilla.wikimedia.org/show_bug.cgi?id=57402 [17:40:46] i'd push it anyway, it fixes at least one other bug [17:41:00] MatmaRex: I'm on latest mediawiki master, and I still get editsection and such [17:42:51] ugh, what, you promised [17:43:39] MaxSem: MatmaRex am I hitting some sort of possible cache locally? [17:43:53] you probably are [17:44:00] let me create a new page [17:44:18] meetings, meetings for everyone! [17:44:30] YuviPanda: no idea how those mobile thingies work, most likely [17:45:16] let's have YuviPanda confirm it fixes that before we dpeloy [17:46:15] upading local MF to master and creating a new page [17:46:53] greg-g: it fixes at least one other bug, which was the actual cause of the revert afaik [17:47:19] so regardless of MF, i'd say deploy, worst-case we'll have one fewer problem to investigate [17:47:21] still occurs on latest mediawiki core master, with a brand new page. [17:47:38] YuviPanda: run update.php to clear parser caches [17:47:51] then purge pages and try again [17:48:36] MatmaRex: I don't see how that should affect a brand new page that I just created, but I just did that anyway and same results [17:49:14] YuviPanda: so we're still unsure what causes your issue? [17:49:18] greg-g: yup [17:49:31] we are sure that the revert does *not* fix my issue [17:49:45] D: [17:49:46] but it does fix the other one [17:49:56] alright, fine, fix the other one [17:49:57] * MatmaRex installs MobileFrontend [17:50:34] * Vito takes the helmet [17:51:35] * Reedy uninstalls YuviPanda [17:51:56] * YuviPanda puts Reedy in the sun [17:52:09] Please [17:52:11] It's dark here [17:52:24] Hey, isn't you card still on my amazon account? [17:52:26] ;) [17:52:52] Reedy: it is, yeah :P [17:53:26] Reedy: you should have a wishlist and paste it after deploys [17:54:27] hah [17:57:06] MatmaRex: https://dpaste.de/nVMj is my output, and you can see the sectionediting links [17:57:16] including ve and regular links [17:57:23] I do actually have lists on both Amazon UK and 2 items on my US one :P [17:57:33] 2nd result in the Amazon list apparently ;) [17:57:33] * MatmaRex is still installing MF [18:08:47] YuviPanda: well, poop, you're right. [18:09:01] i know :( [18:09:13] let me back it up, see I can find out where it broke [18:09:45] `gt bisect` :) [18:09:47] git* [18:10:14] yeah :) [18:18:50] greg-g: If you don't like what I did to the module for the deployments page, https://wikitech.wikimedia.org/w/index.php?diff=90277&oldid=87372 is what you'd want to revert. [18:20:25] hah [18:20:47] anomie: I like, thankya much [18:23:47] <^d> I have one suggestion for the deployments page, while we're on the subject. [18:24:46] <^d> Actually, nevermind, it's not really important and would be hard probably :p [18:25:33] haha, wishlist man, wishlist [18:25:37] ^d: You may as well throw it out there, in case it grabs someone's interest [18:25:53] <^d> I was wondering if we could find a way to color code the table a bit. [18:25:56] oooo [18:26:06] <^d> So you can quickly glance and see "Already done / In progress / Not done yet" [18:26:17] that's a good one [18:26:22] it'd have to be manual for now :? [18:26:22] :/ [18:26:54] ^d: see also: https://www.mediawiki.org/wiki/Wikimedia_Release_%26_QA_Team/Wishlist#Code_Deploy_Dashboard [18:26:57] I'd think that would be best done with JS, otherwise it'd quickly get outdated just because of parser caching. [18:27:01] MatmaRex: wat, I see the bug even at 36542339b56d002c8816576d3dc760c4146d0d22 [18:27:06] anomie: good point [18:27:10] <^d> anomie: My thought. [18:27:10] MatmaRex: which is the commit before your original commit was merged [18:27:20] lulz [18:27:34] YuviPanda: are you sure you cleared various caches really well? [18:27:44] MatmaRex: i ran update.php again [18:27:48] YuviPanda: frontend cache, parser cache, MF caches the results of that query again [18:27:48] MatmaRex: hmm let me purge [18:28:00] and who knows what else is cached [18:28:15] do you have memcached, redis, whatever [18:28:37] redis [18:28:39] this is vagrant [18:28:48] I just turned it off [18:28:57] and now everything takes forever [18:29:02] :P [18:29:05] heh [18:29:17] hey ^d, about git-review https://bugzilla.wikimedia.org/show_bug.cgi?id=57399#c4 – do you think the solution from comment 4 would break things very badly? [18:29:35] happen to know what git-review does when its remote exists, but it's different than what it wants? (e.g. clone via anonymous https) [18:29:38] cloned* [18:30:00] <^d> Not a clue. [18:30:35] eh. [18:30:48] okay, let's dig into that puke-mountain of code [18:31:14] <^d> Where let's is you, right? I hope that wasn't a collective "let's all" :p [18:31:56] it's currently just me, yes. :( [18:32:21] <^d> I've looked at git-review's code before. I'd prefer to not again. [18:32:37] MatmaRex: mediawiki is a clusterfuck [18:33:20] <^d> lolol, I figured the easy way to fix the broken clones of mediawiki! [18:33:26] <^d> Just delete them and reclone, duh. [18:34:00] YuviPanda: you must be new here. [18:34:53] blergh. so from my cursory review of that code, i think that when the default remote is not authentiated ssh, git-review will shit itself. :D [18:35:43] how hard is it to get the git-review guys to accept a patch and push a new version? [18:35:55] or in other words, can it be done within a few days? [18:36:40] MatmaRex: where's the parsercache at? I don't see anything related to that in redis [18:37:05] YuviPanda: by default in the database. no idea what happens if you have other caches set up [18:38:33] * YuviPanda waves at MaxSem [18:39:02] MaxSem: is a cache somewhere getting in the way of me being able to berify this? [18:39:20] YuviPanda, you can bypass it by editing that page [18:41:17] MaxSem: yeah done that a couple of times, no changes, even if I'm on a commit before the commit that MatmaRex made about action=render [18:41:31] MaxSem: can you check that locally on your machine? [18:41:39] sure [18:43:28] YuviPanda, I'm puzzled - the URL you provided has noheadings whch doesn't permit anything from headings in [18:43:35] YuviPanda: maybe, just maybe, it's MF that's broken? [18:43:37] MaxSem: indeed [18:43:49] to see the edit links you need to call it w/o noheadings [18:44:20] MaxSem: that is how it *should* be, but it isn't [18:44:39] MaxSem: I think MF isn't perhaps detecting the headings - that will explain why there is no 'line' property being set even when I ask for it [18:45:22] YuviPanda, https://en.wikipedia.org/w/api.php?action=mobileview&format=jsonfm&page=India§ions=1-§ionprop=line&noheadings= [18:45:40] where are secton links? [18:45:56] that's.. weird [18:46:11] https://dpaste.de/nVMj has them, for example [18:46:54] MaxSem: http://en.wikipedia.org/w/api.php?sectionprop=toclevel%7Cline%7Canchor§ions=1-&noheadings=true&page=India&action=mobileview&prop=text&format=json gives me sectionlinks [18:47:35] pastebinning [18:48:03] grrrr [18:48:10] something's fucked up [18:48:15] yeah [18:48:38] large blobby paste https://dpaste.de/fFK7 [18:49:34] ok, what'd I miss? :) [18:50:07] greg-g: lots of annoyed debugging and flailing arms [18:50:10] on an unrelated note [18:50:17] who is a list admin for mediawiki-l? [18:50:27] some dude has been posting spam to it for the last few days [18:50:35] about time he got banner [18:50:38] banned* [18:50:47] MatmaRex: not sure, the list page will tell you [18:50:59] MatmaRex: y innocentkiller at gmail.com, platonides+mediawiki-admin at gmail.com, aklapper at wikimedia.org [18:51:09] MediaWiki-l list run by innocentkiller at gmail.com, platonides+mediawiki-admin at gmail.com, aklapper at wikimedia.org [18:51:12] bah! [18:51:21] ^d: ^ [18:51:21] mutante: curse you [18:51:41] or simply: mediawiki-l-owner@lists.wikimedia.org [18:51:45] works with any list [18:51:47] -owner [18:51:53] * greg-g nods [18:51:56] greg-g: flailing arms indeed so far. MaxSem confirms "something's fucked up", and now we are trying to figure out what exactly is [18:52:01] <^d> Pointless pings are pointless. [18:52:26] YuviPanda: good? [18:52:34] <^d> Booted him from the list since he doesn't seem to have gotten bored. [18:53:08] <^d> MatmaRex: Unrelated, but http://git.wikimedia.org/summary/mediawiki%2Fcore.git is fixed. [18:53:54] ^d: yay & yay [18:53:58] you're the bestest [18:54:01] YuviPanda: Compare https://en.wikipedia.org/w/api.php?action=mobileview&format=jsonfm&page=India§ions=1-§ionprop=line&noheadings=&prop=text vs https://en.wikipedia.org/w/api.php?action=mobileview&format=jsonfm&page=India§ions=1-§ionprop=line&noheadings=&prop=text|sections [18:54:01] <^d> I know. [18:54:41] anomie: hmm, so specifying 'sections' or not seems to be the difference? [18:55:16] Looks like it, yes. The code for your API module does different things depending on the presence of sections in prop. [18:55:34] MaxSem: ^ [18:56:05] I now dunno if that is a bug or a feature :| [18:56:12] MatmaRex: looks like it is MF, not core [18:56:55] :0 [18:57:24] greg-g: hm, was that revert deployed in the end? [18:58:00] MatmaRex: not yet, MaxSem was delayed. Max? [18:58:26] greg-g, investigating [18:58:29] k [18:58:32] let's ot rush:) [18:58:33] no prob [18:58:35] *not [18:58:37] zactly [19:53:11] ori-l, greg-g: did you see this? http://lists.wikimedia.org/pipermail/analytics/2013-November/001289.html [19:53:28] any chance we could get https://gerrit.wikimedia.org/r/#/c/96901/ deployed to reenable the logs? [19:53:36] greg-g: we've found the bug, and also a workaround (for now). Can definitely wait for monday LD [19:53:45] MaxSem: ^ [19:53:57] (without waiting for a deployment window, its a single change in the config file) [19:54:10] +1 on waiting [19:58:56] MaxSem: https://bugzilla.wikimedia.org/show_bug.cgi?id=57440 filed [19:59:05] thx [20:00:00] DarTar: OK [20:06:42] ori-l: thank you [20:22:59] YuviPanda: awesome, I like waiting on fridays :) [20:23:33] greg-g: :) [20:24:05] DarTar: awesome, mind if I copy it into the Incident documetation page(s) on wikitech? [20:24:10] wikitech.wikimedia.org, that is [20:24:20] go for it [20:24:23] thanks guys [20:27:45] It seems that the personal appeal is broken: https://twitter.com/jilliancyork/status/403982152863199232 [20:29:47] OsamaK: to me, it seems like somebody donated to Wikipedia some time ago using that person's e-mail and first name Robert. [20:30:11] K4-713: do i remember correctly that you are the person to ask about fundraising things? [20:33:12] MatmaRex: Mostly. I can at least rout you to the right person. What's up? [20:33:38] Oh, that. [20:34:06] So, somebody donated with that email address, and the name Robert, within the last year. [20:34:35] OsamaK: ^ [20:34:47] We are in the process of reaching out to her. [20:39:19] MatmaRex, K4-713, thanks! [22:22:43] someone can help me in filling a bug? I found a possible issue but I'd need some feedback in order to write down some clear lines [22:23:12] Vito: Hi! What's the issue you're seeing? [22:23:21] Vito: are you saying it should maybe not be public? [22:23:48] nope mutante, doens't seem to be a security issue [22:24:00] k,good, making sure [22:24:05] basically marktraceur I have a ff ext filtring scripts [22:24:13] (namely noscript) [22:25:22] since...yesterday it intercepts "potential XSS attempt" when opening commons from another wiki [22:25:58] also when opening an image in high res [22:26:26] noscript hasn't be changed since at least a week so I'm wondering if there's something wrong with Commons [23:23:13] ori-l: If MaxSem is right about the apache2.log files that's what I use as my early warning indicators [23:23:21] via fatalmonitor [23:23:30] (modified in my home dir for extra excludes) [23:23:38] pwn3d!:P [23:23:56] ori-l: ++ for getting it out of pmtpa [23:24:10] ori-l: IIRC one of the NFS hosts is doing the collection of them currently [23:24:15] So should be switched off too :) [23:24:20] i have no idea what is populating that [23:24:28] the one on fenari, that s [23:24:30] *is [23:26:16] Have a loook on nfs1 [23:26:39] I'm fairly sure I've seen it int he apache confs somewhere [23:26:51] ori-l: grep4lyf [23:26:53] files/syslog-ng/syslog-ng.conf.nfs: file("/home/wikipedia/syslog/apache.log" group(500)); [23:27:59] file("/home/wikipedia/syslog/syslog" group(500)); [23:27:59] file("/home/wikipedia/syslog/apache.log" group(500)); [23:27:59] file("/home/wikipedia/syslog/swift" group(500)); [23:28:11] I wonder which else are still live/being used [23:28:49] swift has gotta be dead [23:29:21] Nope [23:29:25] All actively being written to [23:29:26] -rw-r----- 1 root wikidev 14685472 Nov 22 23:28 apache.log [23:29:26] -rw-r----- 1 root wikidev 47655253953 Nov 22 23:28 swift [23:29:26] -rw-r----- 1 root wikidev 2504316295 Nov 22 23:28 syslog [23:32:40] https://github.com/wikimedia/operations-puppet/blob/production/files/syslog-ng/syslog-ng.conf.nfs [23:39:30] mutante: ^ loooook :D [23:41:33] Reedy: w..t..h [23:41:50] 45G [23:43:04] Overly verbose logging I think [23:43:11] paravoid might have an idea or 3 [23:44:05] mutante: Luckily we only seem to keep a week in archives [23:44:23] 85G in archive... [23:44:53] paravoid parsoid paranoid :D therms are becoming weirder day by day) [23:44:56] well.. 420G available [23:46:02] The original ping was that ori has started to move it to eqiad too [23:46:15] Where is a list of all sites on wmf servers? Especialy of non mediawiki based ones [23:46:24] Reedy: ooh, you mean apache deployment? [23:46:38] apache syslogs aggregation [23:46:41] Reedy: i thought it was about "why is it logging at all " :p [23:47:03] Base-w: apache config are available in the noc, i think [23:47:05] !noc [23:47:05] All configuration is done in LocalSettings.php (near the end of the file). Editing other files means modifying the software. Default settings are not in LocalSettings.php, you can look in DefaultSettings.php. See , , , and [23:47:05] so you're moving it away from fenari currently? [23:47:06] configs* [23:47:15] the logs [23:47:19] Base-w: http://noc.wikimedia.org/conf is the relevant link [23:47:23] or operations/apache.git [23:47:54] mutante: Yeah, ori-l didn't know that they were aggregated to nfs1 so made them aggregate to fluorine (where we want logs to aggregate!) [23:48:31] so there's syslog/syslog and syslog/swift to also move [23:48:36] and cut off syslog/apache [23:48:41] Reedy: the apache.log is just 15M though [23:49:13] but swift.. that seems to be a LOT [23:49:31] gotcha [23:49:33] tail -f it and watch the scroll [23:49:34] :D [23:49:38] matmarex, thanks i'll try to learn it and dig listlike thing [23:49:49] i actually did and it didnt seem that fast [23:50:04] really? [23:50:12] it took my console a while to catch up after ^C [23:50:21] wait, this is the old one then [23:50:30] what's the new location then [23:50:37] on fluorine [23:51:16] fluorine:/a/mw-log/apache2.log [23:51:36] yes, that's more :) [23:52:05] how do those happen [23:52:14] /usr/local/apache/common/docroot/wikipedia.org/es.wiktionary.org [23:52:46] isnst that one of the things paravoid included in a recent report [23:54:09] uhhh [23:54:39] reedy@tin:/a/common/docroot$ cd wikipedia.org/es.wiktionary.org [23:54:39] -bash: cd: wikipedia.org/es.wiktionary.org: No such file or directory [23:59:55] Reedy: yea, i know, but i meant how it happens that they're requested sometimes