[00:33:21] ori-l: you around? [00:33:25] I guess it's not going out today? [00:34:01] After today's outages I think Greg has just declared that all deployments shall cease until Monday [00:34:09] ori-l: actually; I'll just ask -operations [00:34:31] <^d> RoanKattouw: Actually, we're going to deploy master versions of everything around midnight UTC tonight. Just for kicks :) [00:34:32] RoanKattouw: ah. Has a report been released on what went wrong today? [00:35:19] Yeah post-mortems were sent to the ops list, not sure if they've been written up on-wiki yet [00:36:26] k [00:36:36] i'm reading -operations scrollback right now [00:37:31] legoktm: I can give you the basic overview with the benefit of hindsight [00:37:47] sure [00:37:49] :) [00:37:54] So Faidon noticed that for the past couple days we've had weird spikes in 5xx responses [00:38:11] But because all of the other issues and outages that have plagued us it was hard to diagnose [00:38:41] So since things looked calm this morning, Faidon investigated and found that one of the new text Varnishes in eqiad was falling over every now and then [00:39:15] Turned out it had a couple of very popular URLs mapped to it, two related to Special:CentralAutoLogin and one related to the ULS API [00:40:34] So Mark put in a hack to work around the CentralAutoLogin by disabling hash mapping for those requests (instead distributing them uniformly), then Ori emergency-deployed a fix for the ULS issue which had already landed in wmf4 [00:41:18] However, the ULS fix shifted requests from the API URL to a bits.wm.o/static-current/... URL, and the static-current symlink was a broken symlink because of a bug in a recent change to the script that updates that symlink [00:42:51] So all of those ULS asset requests started 403ing, and 403s are only cached in Varnish for 1 minute, so every minute there would be a spike in cache misses, which caused the number of active HTTP sessions on the bits caches to rise up to the configured limit, and so bits started 503ing [00:44:36] heh [00:45:08] thanks, that was really clear [00:45:19] you should just copy/paste that and send it to the lists :P [00:46:39] ori-l: Did you or paravoid write up an on-wiki outage report for today's bits outage? [00:49:51] RoanKattouw: no, not yet [00:50:28] greg-g may have time to help consolidate it, especially if there's nothing sensitive in it [00:50:57] I like Roan's summary; would you be OK with posting it? (Or I could post it and attribute to you, up to you.) [00:51:22] Or is it something that should properly be left up to Greg? [00:51:27] I hereby release the above summary under CC-BY-SA, go bonkers with it [00:51:30] :) [00:51:41] I'm happy for someone who was personally involved in the outage to post a summary [00:51:54] I was just distilling information from the ops-l posts written by Faidon and you [00:52:00] Eloquence: RoanKattouw ori-l I was just pining those two via email about that.... [00:52:05] ha [00:52:11] :) [00:52:40] anyone can do it, I'm not the arbitrator of incident reports [00:52:53] I'm glad at least both outages were related to us fixing things :) [00:53:01] I'll review and ping you for action items, but posting the text isn't a one-man gig :) [02:15:25] csteipp: ping [02:15:25] Any BZ admins around? [02:24:10] THO got it [03:08:23] legoktm: Do you have a minute? [03:08:30] Possibly, what's up? [03:08:47] Can you try making a new account with MobileFrontend and editing a page? [03:09:26] The pencil icon just seems to blink at me. [03:09:52] sure, lemme try it on my test wiki [03:10:24] I was hoping you'd try en.m.wikipedia.org [03:10:33] Because that's where I'm having my issue. [03:12:08] I wonder if it's a setting in my phone browser. [03:13:07] I block cookies from third parties. [03:13:21] Though I should still be able to edit locally... [03:13:23] > Congratulations, you are now a devwiki editor! [03:13:24] I dunno. [03:13:35] I should probably try on my phone [03:13:42] Try with en.m.wikipedia.org. [03:13:51] Do you have third party cookies disabled? [03:14:10] nope [03:14:13] i probably should [03:15:41] I think there's some way to see the dev tools with this. [03:16:15] do you see it when viewing or in the edit screen? [03:17:14] Elsie: ^ [03:17:39] See what? [03:17:45] I click the pencil icon and nothing happens. [03:17:51] The pencil icon just flickers a little. [03:17:54] oh, i thought you said it was blinking [03:17:54] And the page stays the same. [03:17:55] hrm [03:18:03] Not blinking, just nothing happens. [03:18:20] Does the edit form load for you? [03:18:27] yes [03:18:28] i made an edit [03:18:30] I registered "I don't understand catnip". [03:18:38] i just changed my cookie settings, let me try now [03:18:41] https://en.wikipedia.org/w/index.php?title=Wujek_Ridge&diff=prev&oldid=581718335 [03:18:50] You may have to log out and log back in. [03:18:52] To test. [03:18:55] I did [03:19:00] You go girl. [03:19:10] It may just be my phone. [03:19:13] Or me. [03:19:21] don't we have the same phone? [03:19:26] Probably. [03:19:29] I have an iPhone 5. [03:19:37] oh [03:19:41] I'm on a 4S [03:20:05] Probably same iOS though. [03:20:06] heh [03:20:10] found a bug [03:20:19] Mine? [03:20:21] Or another? [03:20:25] different one [03:22:15] Hrmmmmmm. [03:23:22] is the topic still current? (JS/CSS loading issues?) [03:24:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=57095 [03:29:57] RoanKattouw_away: thx:) [03:30:00] aways [07:39:24] hurray, 5xx back to normal levels :) https://gdash.wikimedia.org/dashboards/reqerror/ [07:45:47] Nemo_bis: was there an outage today Nemo_bis ? [08:05:47] matanya: yes [08:05:54] thanks [08:05:59] well, "today" pacific time [08:06:33] http://lists.wikimedia.org/pipermail/wikitech-l/2013-November/073112.html [08:12:38] hi, anybody on iPad ? [14:30:44] is this error already reported? see the links of global usage and delinker log; https://commons.wikimedia.org/w/index.php?title=File:Charisma_Carpenter_Buffy_reunion2.JPG&action=edit&redlink=1 [14:30:47] the file is double used [14:31:24] right now the global usage link of this deleted file points to https://commons.wikimedia.org/wiki/Special:GlobalUsage?target=Charisma_Carpenter_Buffy_reunion2.JPGFile%3ACharisma_Carpenter_Buffy_reunion.JPG [14:31:34] same for the delinker log [14:31:41] which is clearly wrong [15:04:37] ?? [15:41:26] no reply [15:41:30] then I'll file a bug [15:42:15] don't file quite yet [15:42:21] when did it start? [15:42:36] no idea [15:42:39] I saw it today [15:42:49] you see it too, right? [15:43:12] yes... [15:43:20] maybe it's because I typed the other file name in the deletion reason [15:43:34] because that's the one that's mentioned after the other [15:43:44] uhhh, let's see other examples then [15:43:47] I thought it was the same file name twice, but that's not true [15:44:02] not doubled on https://commons.wikimedia.org/w/index.php?title=File:Charisma_Carpenter_Buffy_reunion.JPG&action=edit&redlink=1 [15:44:31] anyway, hold on a min :) [15:46:54] ok [15:47:12] I said, maybe it's because I mentioned that other file name in the deletion log [15:48:05] right, and i said hold on :) [17:28:37] Trijnstel: ping [17:31:19] var $match = $( this ).find('a[title^="File:"]'); [17:31:20] var $match = $( this ).find('a[title^="File:"]').first(); [17:31:55] re the last 2 lines here: change the first line as I did to make the second line. https://commons.wikimedia.org/wiki/MediaWiki:Common.js [17:32:49] test cases in browser console: [17:32:53] deletions = $('li.mw-logline-delete').map(function() { return $( this ).find('a[title^="File:"]').text().substring(5).replace(/ /g,'_');}).get() [17:32:59] deletions = $('li.mw-logline-delete').map(function() { return $( this ).find('a[title^="File:"]').first().text().substring(5).replace(/ /g,'_');}).get() [17:33:35] but doesn't work if you're using the popups script. so maybe you need to try in a private browsing window [17:33:49] ehm [17:34:08] yes? :) [17:34:17] * Trijnstel looks [17:36:10] maybe I'm looking wrongly, but I can't find those lines [17:38:55] oh, hmm [17:46:50] ah, found it [17:49:01] https://commons.wikimedia.org/w/index.php?title=MediaWiki%3ACommon.js&diff=109737618&oldid=109600502 jeremyb <- [17:49:03] and it works! :D [17:49:09] thank you very much [17:50:07] You've added more leading whitespace! [17:50:21] yeah, bad Trijnstel! [17:50:29] Trijnstel: otherwise lgtm :) [17:52:16] Reedy: oh, is that wrong? [17:52:22] don't know why that happened btw [17:52:26] Not syntactically [17:52:36] It just looks wrong due to the mixed indentation [17:53:23] each level of nest should correspond with a level of indent [17:53:45] Reedy: https://commons.wikimedia.org/w/index.php?title=MediaWiki:Common.js&diff=109738199&oldid=109737618 ;) [17:53:53] RL won't care ofc [17:54:13] Trijnstel: now fix the .first() line too :) [17:54:21] jeremyb: I did.... [17:54:42] https://commons.wikimedia.org/w/index.php?title=MediaWiki:Common.js&diff=109738199&oldid=109600502 [17:54:44] I thought [17:55:15] huh. i thought i saw it messed up [17:55:29] the only thing I did now was adding your suggestion [17:55:29] nothing more nothing less [17:56:29] yeah, i'm just seeing things [17:57:08] ok, all done :) [18:34:57] !helper Hello all-- curious about a text-rendering problem I've noticed lately on Wikimedia sites. [18:38:05] NickServ identify [18:38:17] Anyone out there? [18:41:13] MrPique: Ask your question/state your problem! [18:44:00] Lately some text in the Wikimedia interface has been rendering blurry for me (FF and Chrome). For example, in the navbar to left, the language links are blurry, but the rest above are fine. Same happens with the language-switching link on Commons. Any idea how those links differ from others? Trying to troubleshoot this. [18:44:27] Can you post a screenshot? [18:45:19] Yes, but can't see how to. Clueless with IRC. I'm on ChatZilla. [18:47:00] Did that work? [18:51:10] Ok, Reedy, think I sent screenshot to you now. [18:51:25] hah [18:51:26] Nope [18:51:27] DCC? [18:51:46] MrPique: upload one to imgur.com [18:51:50] Possibly. Nothing appeared though [18:52:39] Reedy: I right-clicked on your name and chose "send file" [18:53:35] MrPique: well don't do that. upload it to imgur.com; paste the URL to the image in here [18:54:00] http://imgur.com/5EKMYip [18:54:15] Thanks. Light dawns on Marblehead... [18:55:22] https://bugzilla.wikimedia.org/56346 [18:57:36] Ok, see that. Looks a little different from what I'm seeing on mine. Is autonym the name for a certain class of link in Wikimedia? [18:58:21] i think it's the name of a Font [18:58:29] i don't know too much about the issue [18:59:39] That's what I thought at first, but now not sure. The image on "Autonym glyphs are missing pixels" matches best what I'm seeing. [19:00:22] https://www.mediawiki.org/wiki/UniversalLanguageSelector/AutonymFont [19:01:01] MrPique: it's definitely the same issue [19:01:19] Ah, got something: https://blog.wikimedia.org/2013/10/28/the-autonym-font-for-language-names/ [19:01:54] MatmaRex: Thanks, do you know if it's user-fixable or something on the WM software end? [19:02:26] MrPique: not really user-fixable, but you could try overriding the font in your user CSS if you really want to [19:03:16] unfortunately the developers of that don't really care when it breaks :( [19:05:18] Ok, thanks, all. Looks like it's being discussed here: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Interlanguage_links_in_different_font [19:06:25] greg-g: Is Echo deploy to dewiki and itwiki on Nov 19 or Nov 20? I only know 19th, but [[Deployments]] lists it on the 20th [19:07:04] Thanks for putting up with my clumsiness here, all. Gotta go back to work now. Bye. [19:07:07] se4598: you can trust [[Deployments]] [19:07:25] well, as much as you can trust me, at least ;) [23:48:05] greg-g: I drafted two different replies to your comment on bug 54985. [23:48:09] But meh. [23:48:24] You seem to be willingly uninformed. [23:48:40] You could try reading the previous bugs. [23:48:51] And the suggestion that anyone copy specific people is an unsustainable approach to take. [23:49:00] Given staff (and role) turnover. [23:49:03] But w/e. [23:49:48] not willingly [23:50:03] sure, I'm tryign to add myself to defautl cc [23:50:28] but: assuming that something will happen with a request without anyone replying "ack" (or similar) for > month: not helpful. [23:50:31] For site requests? That requires a Bugzilla admin, if you're not one. [23:50:39] right, site requests [23:51:16] I did try reading the previous bugs, but there was no conclusion from the first one https://bugzilla.wikimedia.org/show_bug.cgi?id=42447 (nothing helpful) [23:51:34] well, reedy's comments are, but opaque/just cli output :) [23:52:10] and then the other bug he added as see also was the lastest election I was involved with, and with that one, the questions I asked on this latest bug were not answered until near the end [23:52:11] greg-g: to be added to default cc, ask andre or file a bug in Wikimedia → Bugzilla [23:53:00] but, whatever, will be taken care of [23:53:03] * Reedy looks blankly [23:53:15] was just looking forward to getting the other things done I wanted to get done this week in the last couple hours, oh well [23:53:22] Reedy: so the normal? [23:53:47] I've no idea [23:54:19] Reedy: hi there :) [23:54:39] The real issue is that SecurePoll requires shell access still. [23:54:40] Reedy: what questions do you need answered assuming you're the one to do this (again)? :) [23:54:44] Elsie: totally. [23:55:10] How much of a bonus do I get? [23:55:17] the usual [23:55:32] I've already got too many plastic mice [23:55:39] :-) [23:55:40] * ^d gives Reedy the SecurePoll barnstar [23:56:13] https://bugzilla.wikimedia.org/show_bug.cgi?id=42464 [23:56:42] cc'd [23:57:41] https://bugzilla.wikimedia.org/show_bug.cgi?id=26960 [23:59:26] ah, commented too soon, probably