[02:24:11] !log LocalisationUpdate completed (1.20wmf5) at Sat Jun 23 02:24:11 UTC 2012 [02:24:26] Logged the message, Master [03:58:15] @info db4 [03:58:27] @replag [04:14:34] Quick question: Any way I can get the content back from this page? http://en.wikipedia.org/wiki/Course:Oblivion_%28Star_Trek:_Voyager%29 It was overwritten by the activation of the new education extension. [04:24:50] Hello? [04:25:43] you'll have to ask Reedy [04:26:12] it's not something anyone on-wiki can fix [04:26:24] (besides people like Reedy) [04:27:18] Jasper_Deng: I tried, I think it needs to be rescued from the database, the history doesn't exist. [04:27:37] yeah, only sysadmins can do it [04:27:44] see VPT [04:29:49] Jasper_Deng: Ah, I see it. I wasn't reading close enough. I'll comment there, thanks [05:00:33] Tim-away: Your script did the job (AFAIS) so there's no need for me to run the script. https://commons.wikimedia.org/wiki/Commons:Bots/Work_requests#3_million_null_edits [05:07:03] @info db4 [05:07:03] Krinkle: [db4: s7] 10.0.0.237 [05:07:11] @info s7 [05:07:11] Krinkle: [s7] db37: 10.0.6.47, db56: 10.0.6.66, db58: 10.0.6.68, db26: 10.0.6.36 [05:07:34] @replag db4 [05:07:34] Krinkle: [db4: s7] db4: s [13:20:04] Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes. [13:20:14] for the last couple of minutes, I have started having issues with Commons [13:20:23] same here on no.wiki [13:20:24] Request: POST http://it.wikipedia.org/w/index.php?title=Wikipedia:Pagine_da_cancellare/Rete_neurale_artificiale&action=submit, from 208.80.152.86 via sq76.wikimedia.org (squid/2.7.STABLE9) to () [13:20:24] Error: ERR_CANNOT_FORWARD, errno [No Error] at Sat, 23 Jun 2012 13:19:51 GMT [13:20:30] from EU [13:20:55] petan? reedy? [13:20:57] Yp, same here Request: GET http://nl.wikipedia.org/wiki/Wikipedia:Wikiproject/Erfgoed/Nederlandse_Erfgoed_Inventarisatie/Ongebruikte_foto%27s, from 208.80.152.70 via sq71.wikimedia.org (squid/2.7.STABLE9) to () [13:21:11] hi [13:21:24] If you report this error to the Wikimedia System Administrators, please include the details below. [13:21:24] Request: GET http://ja.wikipedia.org/wiki/%E7%89%B9%E5%88%A5:%E3%82%A6%E3%82%A9%E3%83%83%E3%83%81%E3%83%AA%E3%82%B9%E3%83%88, from 10.64.0.133 via cp1015.eqiad.wmnet (squid/2.7.STABLE9) to () [13:21:24] Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable at Sat, 23 Jun 2012 13:21:06 GMT [13:21:29] server issues [13:22:14] oh god, this is severe [13:22:21] Still ongoing? [13:22:44] multichill: Make that Asia as well :) [13:22:48] Reedy: check -operations [13:22:53] can someone please change the topic? [13:22:57] Is it just europe? [13:23:07] You already did? [13:23:21] no it's not europe [13:23:30] it's apache cluster [13:23:35] they are dying [13:23:45] petan: Everyone? Ok [13:23:57] scary, enwp too :P [13:24:12] what are those apaches doing? [13:24:38] In case this helps, I'm getting [13:24:40] Request: GET http://pt.wikipedia.org/w/index.php?title=MediaWiki_Discuss%C3%A3o:Gadget-fastbuttons.js&action=edit§ion=24, from 10.64.0.142 via cp1014.eqiad.wmnet (squid/2.7.STABLE9) to () [13:24:42] Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable at Sat, 23 Jun 2012 13:23:17 GMT [13:25:00] helderwiki: thanks [13:25:09] * multichill loves http://status.wikimedia.org/ [13:25:19] folks from ops know that, so hopefully it should be fixed soon [13:25:30] Up and down for me [13:25:32] thx petan [13:33:18] thank you, techs. [13:33:48] could be up [13:33:51] woohoo [13:34:05] though it doesn't mean it's fixed :) [13:34:14] touch wood [13:34:20] Judging by the nagios-wm spam in # [13:34:28] wikimedia-operations, it looks fixed to me [13:34:36] silly enter key [13:35:18] petan: what was wrong with it, if you don't mind sharing? [13:36:14] wctaiwan: I don't know details but it was likely some combination of memcache issues and search [13:36:28] uh, OK, over my head :P [13:36:43] apergos is analysing it :) [13:36:53] in other words it had a memory block! :P [13:36:59] cool. Not often that the whole thing goes down. [13:37:04] not so much, I was just hitting the big red panic button [13:38:01] apergos: I hope that you brought one for everyone [13:39:15] sDrewth: no, but he did buy vodka shots for everyone [13:39:18] I sure did [13:39:20] http://commons.wikimedia.org/wiki/File%3APanic_button.jpg [13:39:24] share and share alike! [13:39:51] the thoughtfulness, it is just so hard to measure it [13:39:56] it seems prone for accidental presses. [14:12:35] yes you may want to move it to another part of your keyboard [14:12:39] or even off to the side [14:16:19] nothing helpful from the searchidx2 logs, except that when everything goes out to lucnh they start whingin too [14:16:24] about unreachable apaches [14:17:41] what operations did it begin at 13:10? [14:18:20] or how does it relate to memcached? [14:19:12] nothing [14:19:23] there was nothing in the log until around 13:20 [14:19:28] :-/ [14:19:44] a bunch of stuff from much earlier 12:xx [14:19:49] much earlier, not 12:59 [14:20:35] why did it send those netwrok busts at 13:10 then? :S [14:21:22] don't know [14:21:27] could have been an rsync or something [14:21:31] completely unrelated [14:23:20] it could, but looks very suspicious :) [14:23:46] it does but I can't find an actual correlation [14:23:52] not in the sampeld logs nor anyplace else [14:24:28] so I am officially giving up now as it's been just about an hour [14:24:32] anything in the auth log? [14:32:14] 12 June - srv232; 13 June srv203; 20 June srv268; 21 June srv272; 23 June src288 [15:18:00] hmm, it is gone again [15:18:23] weirdness [15:33:03] can someone confirm that https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Jinantielubuilding.jpg/290px-Jinantielubuilding.jpg is broken while both https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Jinantielubuilding.jpg/291px-Jinantielubuilding.jpg and https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Jinantielubuilding.jpg/289px-Jinantielubuilding.jpg work [15:38:33] liangent: yes [15:39:17] what's wrong with it [17:42:07] 0 bytes [17:45:27] 1 byte [17:45:46] 2 bytes. ah ah ah! [17:46:14] 3rd byte [17:57:25] 0b100 byte [19:47:41] Platonides: https://en.wikipedia.org/w/index.php?title=2012_Arab_Nations_Cup&diff=498870742&oldid=498870712 [19:47:47] doesn't it just drive you crazy? :) [21:23:53] greetings #wikimedia-tech, I'm looking at a third-party tool that's used on English Wikipedia [21:24:00] I've found a potentially problematic security issue [21:24:13] In that tool> [21:24:14] *? [21:24:26] I'm not prepared to say which tool until I know what the security reporting procedure is [21:24:58] Well, if it is not a Wikimedia-maintained tool, then there's probably no Wikimedia procedure [21:25:02] well, the action to take depends on the tool :P [21:25:17] if it's huggle, you should go to #huggle [21:25:37] You could PM me so I direct you the right way [21:26:27] yep, will do [21:27:54] tommorris: bugzilla.wikimedia.org -> new -> "security" -> and the relevent people will be added if possible or you will be directed elsewhere [21:29:10] p858snake|l: the wiki says email security@wikimedia.org I believe [21:29:37] that is also possible, but i believe Tim-away and such prefer the bz method these days [22:07:00] putting in a couple mailing list requests in bugzilla and will be around if anyone in ops has questions about them [22:08:23] varnent, I saw one of them, let me know when you've done the second one then I'll look at both in one go [22:08:48] Thehelpfulone: thank you :) [22:09:03] :) [22:19:42] Thehelpfulone: okay - I'm checked all the mailing lists on my to-do list off [22:20:58] now I'm done..made a typo in Pharos' email [22:21:29] ok, I'll brb [22:40:33] wow, I was able to throw an exception in en.wikipedia.org [22:40:54] on demand? [22:41:09] just visiting a url [22:41:19] https://en.wikipedia.org/wiki/xxxxxxxxx [22:41:30] were you also able to write an empty file? ;) [22:41:41] MediaWiki internal error. [22:41:41] Exception caught inside exception handler. [22:41:54] that's really easy at toolserver nowadays [22:42:00] even if you want them to have content :) [22:42:36] i know ;) [22:42:43] echo | cat [22:42:58] :)