[10:17:51] Hi. [10:17:58] Does this open for you (via http)? http://commons.wikimedia.org/wiki/File:Anki_0.9.6_usstates_text_a.png [10:23:10] nm, it's me [11:20:00] OK what did you break on Wikipedia? [11:20:16] Sometimes I am viewing images and the additonal tag for TWINKLE does NOT appear [11:20:22] It's only started doing this the morning [11:20:37] Also when viewing the preferences tag, I don't see a tag for 'Gadgets' [11:21:00] What has been tweaked to cause this annoyance? [12:00:09] mark: what I notice is that gadgets are mostly offline; haven't noticed anything else though [12:00:32] api cluster is being DoSed by parsoid, it seems like [12:00:40] oh [12:03:40] * AzaToth wonders what nutcracker is... [12:08:50] * T13|needsCoffee points AzaToth to his girlfriend... [12:24:06] not only gadgets [12:24:19] https://www.mediawiki.org/wiki/Special:Random/Extension has broken styles [12:24:28] Meta wiki is very unresponsive -especially regarding translations. [12:25:37] notice as well VE doesn't work as well [12:25:48] - one as well [12:25:51] "Up" is undeserved in /topic [12:26:15] apergos (and mark I assume) is looking into the issue [12:26:37] http://pastebin.com/Udtw879e [12:26:39] yes and roan [12:26:49] k [12:34:27] Request: GET http://meta.wikimedia.org/w/api.php?action=tokens&format=json&type=edit, from 208.80.152.87 via sq36.wikimedia.org (squid/2.7.STABLE9) to ()
Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable at Sun, 14 Jul 2013 12:33:52 GMT [12:35:06] was something changed regarding mobile css and/or detection today? [12:35:17] pl wiki main page is mobile for everyone [12:35:47] there are a bunch of issues going on right now, my guess is they are related [12:36:22] I know some are looking into it, obviously if they aren't…. then we should fix that too cause that's somewhat bad … but my guess is they are all related [12:36:28] lazowik_: oh dear. [12:36:57] and probably some gadgets are not working [12:37:14] lazowik_: yep. that's a component of the known bug; sorry ;/ [12:37:31] ok [12:38:00] is it bugzilled? [12:38:21] I believe it is being actively looked at as they try to get the servers working again [12:38:40] yes, we are actively trying to get the site back up right now [12:39:15] pl wiki main page is mobile for everyone? [12:39:32] doesn't look mobile to me [12:40:17] not the whole wiki [12:40:23] but the content of page [12:40:37] normally there are two columns [12:40:43] Missing CSS perhaps? [12:42:00] lazowik_: not some bad edit? [12:42:17] no changes really [12:42:26] lazowik_: linky? [12:42:33] and screeny [12:43:05] err? [12:43:11] I don't understand... [12:43:34] lazowik_: this is how plwiki main page looks like for me: http://i.imgur.com/q1xTn7q.png [12:43:46] yep [12:43:50] and it's not good [12:43:52] I don't know how it should look like [12:44:11] see this: http://pl.wikipedia.org/w/index.php?title=Wikipedia:Strona_g%C5%82%C3%B3wna&diff=36627091&oldid=36614605 [12:45:04] err, not that [12:45:06] one second [12:45:32] http://pl.wikipedia.org/wiki/MediaWiki:Gadget-main-page.css [12:45:41] more or less like that http://pl.wikipedia.org/w/index.php?title=Strona_g%C5%82%C3%B3wna&oldid=36282121 [12:45:52] errm [12:45:58] but after that rev it was changed to relay on templates [12:46:01] "Zagadnienia związane ze stroną główną:"? [12:46:05] yeah, gadgets right now aren't working well because of the server issues [12:46:07] #main-page #main-page-column1 { float: left; } #main-page #main-page-column2 { float: right; } [12:46:12] that's why [12:46:14] rillke: well, if the css is from a gadget, it's highly possible it's due to current fuckup [12:46:28] as no gadgets are really loading [12:46:39] ok, thanks [12:47:20] and I must assume they haven't located the issue yet :( [12:47:50] Confirmed. [12:48:14] After injecting the CSS from the gadget into the page with Firebug, it has 2 columns. [12:48:23] should the topic of the channel include the status? (I'm always afraid to do such things myself in fear of doing it wrong/getting yelled at) [12:48:26] so that's it [12:49:24] MartijnH: we can yell at you perpetually so it makes no difference for you [12:50:00] well, I don't want to give even more reason to do so than usually [12:50:14] heh [12:50:49] ops are still spewing CRITICAL like a madman [12:51:18] RoanKattouw: any status updates? [12:51:47] We've blocked some IPs sending lots of requests to the API [12:51:59] And we're now trying to bring the API backends back up again. About half of them are up [12:52:01] external or internal? [12:52:05] Extenral [12:52:15] The API should be mostly up now, with reduced capacity, as I understand it [12:52:19] oh [12:52:40] RoanKattouw: no gadgets loaded for me [12:57:53] now we need to construct a conspiracy theory whenever the DoS was intentional or not [12:58:19] is said DoS internal or external or...? [12:58:30] external per RoanKattouw [12:58:47] or to be exact: "Extenral" [12:58:55] heh [13:00:45] I had indeed API problems [13:01:23] Interesting [13:02:41] with AWB [13:06:42] Yup, wikidata is practically unusable right now because it depends on the api ;p [13:07:03] I notice [13:07:20] :) [13:07:21] atm there are more OK than CRITICAL on operations at least [13:07:38] erm, no [13:07:43] was too soon [13:09:02] XD [13:09:15] Seems like an even balance :p [13:09:29] why dont we just have someone disable access from their API? [13:09:52] We did [13:10:06] We also stopped the jobs that populate the Parsoid cache which internally use the API [13:10:09] That seems to have helped [13:10:33] I still have no gadgets loaded though [13:10:46] bits should be up [13:12:53] RoanKattouw: I'm getting gadgets now and then, but only like 40% of times [13:13:39] or less [13:13:46] it's pretty scetchy [13:14:57] yeah, it's been loading on-and-off. [13:15:55] I don't see any errors in trying to load something from bits; it's like it's not even trying [13:27:49] Ironholds: troll warning in #wikimedia [13:28:05] T13|needsCoffee: ping an op. I don't have ops in there. [13:28:28] He's after you for banning him I guess. [13:28:52] Figured I'd give you a heads up in case you wanted to avoid him. [13:29:28] I already have him on ignore. [13:29:37] mostly I believe he's after me because he's a lunatic. [13:29:37] :) [13:29:44] Could be. [13:30:18] Just looking out for ya.. now.. back to looking for coffee... [13:43:36] RoanKattouw: seems it works better now [13:43:44] or not... [13:43:52] Yeah it's in better shape now [13:44:03] didn't get gadgets this time [13:44:20] Yes, they sometimes fail [13:44:32] and sometimes not [13:45:06] haven't got any gadgets on 4 cache reloads now... [13:47:58] It's funny -- we have a watchlist notice gadget at commons; if css does not load, the messages are not visible; when js does not load, they aren't grouped [13:48:57] rillke: This is why we should work to build things that are more reliable than gadgets, and communities certainly shouldn't rely on them for important things. :-( [13:49:15] I don't see how the API cluster should affect Gadget loading [13:49:38] RoanKattouw: It's not the same root cause of DDoS? [13:50:00] RoanKattouw: atm it's more of a fact something fucked gadgets up and that the API cluster was ddosed [13:50:18] usually occams razor works [13:50:45] James_F: define "important" [13:51:13] AzaToth: Anything a wiki relies on for telling users what to do and what not to do. :-) [13:51:21] most gadgets are important in their own niche [13:51:39] like the gadget to disable VE? [13:51:44] * AzaToth hides [13:52:24] AzaToth: Anything enabled by default, for example. [13:53:02] AzaToth: Do you see a Gadgets section in your preferences at all? I don't [13:53:07] I don't [13:53:18] RoanKattouw: nope [13:53:30] Gadgets-definition wasn't edited recently [13:53:41] But it's cached in memcached, maybe it was on one of the API machines that fell over [13:53:57] (cur | prev) 2013-07-07T02:23:44‎ TheDJ (talk | contribs | block)‎ . . (3,741 bytes) (+16)‎ . . (include edittop in RL) (rollback: 3 edits | undo | thank) [13:54:16] but why would the gadget tab dissapear? [13:54:45] Error: Unknown dependency: ext.gadget.libAPI [13:54:54] lol [13:54:54] memcached is no longer on apaches [13:57:06] AzaToth: Probably because MW lost track of the gadget list somehow [13:57:15] I see [13:57:24] There, it's back for me now [13:57:36] I made a simple edit to [[MediaWiki:Gadgets-definition]] [13:57:39] not for me [13:58:06] Hrm, startup module grumlbe [13:58:11] * RoanKattouw touches [13:58:20] now I'm getting it [13:58:22] back for me, but it's showing me as only having the defaults [13:58:44] :O [13:58:47] RoanKattouw: now I didn't get it [13:58:52] doing cache reloads [13:59:02] Ironholds: Did you save a change to your preferences while that tab was missing? [13:59:44] Hard-refreshed and Twinkle is back for me [13:59:49] RoanKattouw: I have the gadgets tab under preference, but no gadgets loaded [13:59:56] RoanKattouw: nope. [14:00:02] I'm doing hard refresh all the time [14:00:04] I have my Gadget preferences still [14:00:09] hard refresh doesn't work either [14:00:31] When you look at the Gadgets tab in preferences, does it list the selections you expect? [14:00:34] RoanKattouw: http://i.imgur.com/HMrHB40.png [14:00:37] yes [14:00:48] So yes [14:01:02] Quick check: are you hard-refreshing some other page that isn't your preferences? [14:01:03] RoanKattouw: http://i.imgur.com/1WhAdHa.png [14:01:20] recent changes, watchliost and random article [14:01:25] Hmm [14:01:55] RoanKattouw: I'm also not getting gadgets on hard-refresh; I added a gadget and it's not coming through. [14:02:03] Hrmph [14:02:15] On enwiki or elsewhere? [14:02:23] enwiki. [14:02:24] It's working for me :( [14:02:27] Want me to try another wiki? [14:02:40] RoanKattouw: I'm getting gadgets more and more now [14:03:04] I just enabled UTCLiveClock and it came through right away [14:03:17] James_F: Was just checking, I believe this was an enwiki-specific problem [14:03:26] Hmm. [14:03:38] RoanKattouw: have gotten gadgets now after 20 tries [14:03:51] on http://en.wikipedia.org/wiki/Tolombeh-ye_Shahdai_Haft_Tir [14:04:07] RoanKattouw: I just did the same and don't. [14:04:25] James_F: Ctrl-Shift-R [14:04:45] WFM in both browsers [14:04:53] AzaToth: I'm familiar with how to hard-refresh. :-P [14:04:56] RoanKattouw: seems to work now [14:05:05] AzaToth: What do you think I do all day in VE development? [14:05:16] James_F: sorry, I assumed you where a normal Town Idiot [14:05:31] James_F: Ctrl-Alt-Delete [14:05:59] AzaToth: Doesn't do much on this system. [14:06:29] oh, you are a Apple Fan Boy? [14:06:35] RoanKattouw: seems to work now [14:07:26] Awesome [14:08:33] RoanKattouw: the tab of gadgets is missing in my preferences [14:09:17] on several wiki' [14:09:31] Romaine: on enwiki as well? [14:09:37] Romaine: Make an edit to [[MediaWiki:Gadgets-definition]] or get a sysop to [14:09:41] That fixed it for me on enwiki [14:09:58] en-wiki it is visible [14:11:20] this fixed it on he.wiki too [14:11:23] maybe run some script? [14:12:51] edit on gadgets page worked [14:13:38] tab is back in preferencs [14:15:39] still getting Error: Unknown dependency: ext.gadget.libCommons [14:16:52] oh, now it seems to work [14:22:52] yep, plwiki main page is ok now [14:22:58] thanks! [15:02:07] qchris: ping [15:02:20] dMaggot: pong [15:02:40] dMaggot: Hi. [15:03:02] qchris: hi, I've been trying to contact someone to help me with my gerrit log-in [15:03:06] qchris: and was told that you can help [15:03:26] dMaggot: Let's find out :-) [15:03:38] dMaggot: What's the problem? [15:04:19] qchris: I'm trying to log in with my wikitech account and get an "Invalid username/password" error [15:05:14] dMaggot: Is this a new account, or did it work before? [15:10:09] dMaggot: Is this a new account, or did it work before? [15:10:35] AzaToth: Thanks. I was just about to do the same thing :-) [15:10:41] heh [15:10:58] hi guys [15:11:12] Hi strainu [15:11:12] has any software been upgraded today? [15:11:16] strainu: nope [15:11:19] people on ro.wp are complaining about gadgets dissapearing [15:11:30] http://ro.wikipedia.org/wiki/Special:Gadgeturi shows none active [15:11:35] [10:07:33 AM] qchris: I can, of course, log into wikitech, and I used to be able to log into gerrit [15:11:36] strainu: but there was a big ddos issue which fucked gadgets up [15:11:37] and the tab is not active in the preferences [15:11:43] oh, ok :) [15:11:57] strainu: make a null edit on gadgets-definition [15:12:00] qchris: I gave some more info, it probably got dropped in the connection [15:12:13] thanks AzaToth [15:12:15] dMaggot: Probably :-/ [15:12:46] qchris: well, basically that, the fact that I used to be able to log into gerrit (search owner:DMaggot) [15:13:48] dMaggot: I do not have access to the produciton gerrit's logs. But let's try on gerrit-dev.wmflabs.org [15:13:58] AzaToth: that fixed it, cool! [15:14:10] qchris: do you want me to try login into that? [15:14:32] dMaggot: Just a second. [15:15:53] dMaggot: Yes, could you try logging in on https://gerrit-dev.wmflabs.org/ [15:16:46] dMaggot: Not sure if that arrived at your end ... so: [15:16:52] dMaggot: Yes, could you try logging in on https://gerrit-dev.wmflabs.org/ [15:17:30] qchris: my login works there [15:18:07] dMaggot: You used the same casing, settings, ... etc as on gerrit.wikimedia.org? [15:18:50] qchris: yes, the one that worked was the same casing as in wikitech too [15:19:04] RoanKattouw_away: ping [15:20:07] dMaggot: Mhmm. Logging in works for me on the production gerrit. [15:21:12] dMaggot: I'll need ops support to look on the production server, which probably will not work during weekends [15:21:39] dMaggot: Could you send me your username, so I can try to find ops to help me on Monday? [15:27:18] Have there been any DB errors tonight? [15:27:24] Reedy: ^ [15:27:48] https://bugzilla.wikimedia.org/show_bug.cgi?id=51317 [15:28:08] I suspect this to be caused by the MessageCache::singleton()->disable(); in DatabaseError [15:28:47] We have tonnes of DB errors [15:39:52] meh [16:18:42] hello, gadgets are not loading in wikipedias except en.wikipedia [16:18:57] geraki_: oh wow [16:19:24] bug 37228? [16:19:27] !bug 37228 [16:19:27] https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=37228 [16:19:49] they do work in en, de, don't work in el, it, ru [16:20:45] https://it.wikipedia.org/wiki/Speciale:Preferenze?uselang=en doesn't even show the Gadgets tab [16:22:01] ah, gadgets are loading again on en.wiki? [16:23:01] a null edit on https://el.wikipedia.org/wiki/MediaWiki:Gadgets-definition fixed the problem for this wiki [16:23:34] I verify that HotCat is loading for me on en.wiki [16:23:47] they were broken in enwiki before [16:23:56] roan fixed it [16:24:30] I don't think he debugged it any further, he was woken up to deal with VE issues so it wasn't the best time [16:25:05] 17:07 < RoanKattouw> So I edited the definition page and that was enough to kick it [16:25:20] sounds like what geraki_ did to elwiki [16:25:41] so, we need to make null edits to all wiki's gadget pages (or maybe purge them)? [16:26:00] well, that's a hack [16:26:34] yes [16:26:51] greg-g: ping, in case you are up [16:27:15] (the conversation right now, about bug 37228 and gadgets not displaying on most Wikipedias) [16:27:31] thank you geraki_ for reporting this [16:27:53] :-) [16:28:51] Reedy, MaxSem - maybe one of you could lend a hand? [16:29:24] btw, there's https://bugzilla.wikimedia.org/show_bug.cgi?id=51317 [16:29:29] someone reported this earlier here [16:29:42] it's marked as a dup of 37228 [16:30:04] but it's definitely more wikis experiencing thise [16:30:27] 18:11 < strainu> people on ro.wp are complaining about gadgets dissapearing [16:34:56] A null edit of the definition page has alwayss fixed this in the past [16:36:32] I've just upgraded but 37228 to major and am thinking of taking it higher [16:36:36] liangent: hey, next time you resolve an immediate critical bug as a duplicate, look at the bug it's a dupe of and please raise its priority if that's warranted, thanks [16:36:38] Why? [16:37:07] because it seems to me like people having their gadgets disappear on them is a reasonably bad bug [16:37:52] Null edit the gadget page on the problem wikis [16:38:13] based on past experience, it should disappear for a month or more [16:38:19] if it's every wiki, that seems pretty inelegant [16:38:30] and hard to communicate out [16:39:06] can't we just fix this for everyone, even if it's just a one-time fix? [16:39:13] No [16:39:20] It's not an easily reproducible bug [16:39:25] And hence, is even harder to try and fix [16:39:36] I see gadgets on commons [16:39:43] it's the wikipedias [16:40:01] can someone with relevant powers do some magic to make a null edit to the gadgets definitions page on every wikipedia in one stroke? [16:40:09] hey guys, let me just note that the CSS for main page of pl.wp is loaded via a gadget :P [16:40:25] (and it loads currently, luckily, but i ma told was broken afew hours ago) [16:40:26] * Reedy beats MatmaRex [16:40:45] enwiki has gadgets [16:40:50] sooo yeah, if you need to troubleshoot, just look there. :P [16:41:14] I don't know if that's related, but I asked on #mediawiki why some custom CSS in mediawiki.org wasn't working. p858snake|l said: api is broken atm, which breaks gadgets, which is where most of our css is loaded from [16:41:28] do we know specifically how the API is broken? [16:41:39] it was broken before [16:41:40] I see gadgets on enwiki, dewiki and plwiki [16:41:42] and ifxed [16:41:52] there was an API outage possibly related to parsoid [16:42:04] Reedy: I presume those are wikis where someone has made the relevant null edit in the last few hours [16:42:16] sumanah: there was some earlier discussion here, RoanKattouw was on to some stuff. If there is no logging directly available, I can copypasta some into a pastebin if you want [16:42:17] https://it.wikipedia.org/wiki/Speciale:Preferenze?uselang=en doesn't even show the Gadgets tab [16:42:30] I'll look at http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-tech/20130714.txt thanks MartijnH [16:42:30] Hey sumanah, you are welcome! [16:42:34] Memcached related issue would be the obvious place due to the definitions being cached there [16:43:55] Hoos patch might not fix the problem, but isn't going hinder anything [16:44:06] Rules out the case where for some reason that MW thinks the page is empty [16:44:10] so we're not caching no gadget [16:44:10] s [16:46:51] Actually, the simplest way of probably doing it over all wikis would be to iteratively explicitly purge the memcached key for the gadget definition cache [16:47:09] sumanah: got it but you raised severity and talked about priority [16:47:17] anyway it's a case by case thing [16:47:22] liangent: I should have raised both [16:47:52] sumanah: ok I didn't really check its history and replied based on your lines [16:48:22] pues de 18 a 21 cuando trabajo [16:48:26] oops, wrong chan [16:49:52] liangent: yes, it [16:49:52] it is a case by case thing. In this case I think you should have seen the urgency and raised the priority and/or severity of the underlying issue [16:54:00] pong [16:54:21] the easiest way to purge everywhere is to bump cache version [16:54:44] BUMP ALL OF THE CACHE [16:54:49] DON'T YELL [16:55:00] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA [16:57:27] what's a wiki that doesn't currently show gadgets? [16:57:30] itwiki looks ok to me [16:58:07] ori-l: The tab doesn't appear at all [16:58:37] https://gerrit.wikimedia.org/r/73620 [16:59:12] it does for me [16:59:25] but i don't know how our memcaches are arranged [16:59:33] Yes [16:59:35] I mean [16:59:41] On a wiki where it's not working, it won't appear [16:59:47] itwiki looks to have been "fixed" [17:00:57] OK, is there any wiki on which it is known to be broken? [17:01:04] ori-l: maybe ru.wiki? [17:01:25] ru.wikipedia.org/wiki/Special:Preferences?uselang=en [17:01:26] Yup [17:01:44] ok, hang on a sec [17:03:36] https://ml.wikipedia.org/wiki/Special:Preferences?uselang=en for kicks [17:03:43] http://notes.wikimediadc.org/p/english-lessons [17:03:48] sorry, wrong window [17:04:09] ok, fixed it in RU, i think in can fix it in the rest too [17:04:37] ori-l, if we deply my change it will automatically purge everywhere [17:04:46] With the fix being? [17:05:03] loop through all wikis, check if the key is empty, delete it if so [17:05:22] [17:46:51] Actually, the simplest way of probably doing it over all wikis would be to iteratively explicitly purge the memcached key for the gadget definition cache [17:05:23] ;) [17:05:37] Doesn't the Gadget code already check for it being empty MaxSem? [17:05:58] humm [17:06:03] The extension code itself checks is_array [17:06:03] how can i get a list of all $wgDBnames again? [17:06:05] Cause if it doesn't... [17:06:11] ori-l: all.dblist [17:06:25] and empty is not an array :P [17:06:47] Not even in PHP null is an array [17:07:30] var_dump( is_array( (array)null ) ); [17:07:36] bool(true) [17:07:45] christ, we have a lot of wikis [17:07:59] Casting it to array before checking is_array would be *not so smart* [17:08:06] A whole 877 [17:08:06] (array)null is array(null) [17:08:40] do you want to limit it to the Wikipedias right now since those are the ones with a problem? [17:08:55] sumanah: Wikidata had troubles as well [17:08:58] did a null edit there [17:08:59] oh never mind then [17:09:17] I took geraki_'s original report too literally [17:09:21] thought it was just the pedias [17:09:21] i'm iterating through all wikis [17:09:31] i'm already in the p's [17:09:38] thank you ori [17:10:34] ok, my net connection is terrible so I'm gonna head off, but thanks for working on this [17:10:44] even an empty array is false... php sucks :( [17:10:46] done [17:11:18] https://ml.wikipedia.org/wiki/Special:Preferences?uselang=en is fixed [17:12:29] did what reedy suggested [17:12:50] less invasive than a code change imo [17:13:56] So the get action on memcached in the gadgets code needs reviewing for it what it does for error conditions [17:14:27] https://gist.github.com/atdt/5994974 btw [17:14:38] Reedy: I think the Message and MessageCache classes are rather to blame [17:14:39] there's probably a less idiotic way of running a script on all wikis [17:14:43] but that one's mine [17:15:03] Maybe in some weird edgy edge case they return an empty string for the MediaWiki page and then this gets cached [17:15:09] my 2ct [17:15:29] (cached as an empty gadgets array) [17:17:39] isValidList [17:26:58] what caused an empty array to be generated in the first place? [17:27:07] The latest issue of Tech News is out if anyone's up for some reading [17:27:09]