[09:21:40] anyone who can have a look at https://bugzilla.wikimedia.org/show_bug.cgi?id=40736 [09:21:43] Reedy: ^ [09:32:55] yuvipanda, is 50 ppl limit enough? [09:36:44] MaxSem: apparently not [09:36:48] but I got word that it was 'sorted' [09:37:01] so no extra ips needed? [09:38:37] there is a session going on right now in Pune, and they are stuck since IP is now blocked though I had only put a bug https://bugzilla.wikimedia.org/show_bug.cgi?id=40736 [09:38:55] can someone please help me unblock these IPs [09:41:59] Here are the IPs [09:42:05] 14.140.227.65 [09:42:14] 14.140.227.66 [09:42:21] 14.140.227.67 [09:42:25] 14.140.227.68 [09:44:54] Nitika: you told me that it was sorted out? [09:45:11] MaxSem might be able to help, if it isn't [09:45:45] hey yuvipanda [09:46:28] hey Rangilo [09:47:11] some problem with ip cap [09:47:18] in pune right now [09:47:47] hi [09:48:07] yes, but there seems to be a block again [09:48:18] * yuvipanda pokes MaxSem gently [09:48:24] a block or limit reached? [09:48:46] limit reached [09:50:01] ok, do you need extra IPs added? [09:50:26] Rangilo, Nitika ^^ [09:50:49] is the cap removed from the given ip address previously [09:52:41] i am writing a simple webcrawler + robots.txt analyser, which i wanted to demo on the wikipedia-game (all articles lead to Philosophy). when i open http://www.wikipedia.org/robots.txt or http://en.wikipedia.org/robots.txt in a browser, it seems to work fine, but when I open it with my parser, I get an error [09:52:47] Request: GET http://www.wikipedia.org/robots.txt, from 94.193.241.144 via amssq35.esams.wikimedia.org (squid/2.7.STABLE9) to ()
Error: ERR_ACCESS_DENIED, errno [No Error] at Sun, 07 Oct 2012 09:45:51 GMT [09:52:52] ??? please explain in clear words: should I raise the account cap or add exdtra IPs to the 2 already set in the rule? array( '14.140.227.85', '14.140.227.65' ) [09:53:08] Rangilo ^^ [09:53:49] MaxSem: please raise the account cap [09:53:56] does anyone have any experience/advice? cheers. [09:54:01] ok [09:55:22] MaxSem: thanks [09:58:26] Reedy, are the uncommitted changes to throttle.php yours? [10:06:53] @myself: must be some bug in the urllib library of python, with requests it works well [10:07:01] kthxbye [10:08:54] For when labint gets back: He probably forgot to set an agent id so he gets the default agent string and that one is blacklisted [10:11:57] MaxSem: please do get back when you have raised the account cap so that we can start account creation, thanks. [10:13:32] Rangilo, done [13:40:05] MaxSem: nope [13:40:10] Leslies most likely [15:08:13] MaxSem: didn't get a chance to thank you back then, thanks for raising the IP cap immediately on such short notice.... things went out pretty well, collected over 5500+ images from 120 new users. Thanks a lot. Rangilo here. [15:08:32] :) [15:32:29] File moving on commons doesn't work... "Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.0.6.41)" :/ [15:32:47] @dbinfo db31 [15:32:51] errr [15:32:58] i never remember the right incantations [15:33:21] anyway, did you try again? [15:33:39] jeremyb: I'm trying with several files, all fail [15:37:22] wikis are slow [15:37:29] s3 particularly [15:40:13] And uploading also fails for some users [15:40:26] Request: POST http://commons.wikimedia.org/wiki/Special:Upload, from 91.198.174.54 via sq63.wikimedia.org (squid/2.7.STABLE9) to 10.2.1.1 (10.2.1.1) [15:41:27] Mmmm [15:44:26] NFS to ms7 is sllooooow [15:45:47] * jeremyb gives Reedy a can of paint so he can watch it dry [16:02:20] Reedy: and reports about incomplete file deletions... [16:02:33] Indeed [16:17:36] nagios is going mental in -operations [16:20:52] has been for an hour or so [16:25:30] bah 5 minutes to load simple page [16:26:50] Not particularily quick for me, but it's seconds not minutes [16:28:31] It takes ages for me to load the javascript I need the most [16:29:00] Hmm seems to be down completely now [16:29:05] Or at least a lot slower [16:29:41] bits network is down a bit, bu it's still the app servers showing as having quite a bit more load [16:31:13] 504 Gateway Time-out [16:31:13] nginx/0.7.65 [16:31:26] skwikiquote [16:36:15] Request: POST http://sk.wikiquote.org/w/index.php?title=Wikicit%C3%A1ty:Hlavn%C3%A1_str%C3%A1nka&action=submit, from 208.80.152.76 via sq61.wikimedia.org (squid/2.7.STABLE9) to () [16:36:19] Error: ERR_CANNOT_FORWARD, errno [No Error] at Sun, 07 Oct 2012 16:36:01 GMT [16:43:08] Request: POST http://www.mediawiki.org/w/index.php?title=User_talk:Losik34&action=submit, from 208.80.154.9 via cp1013.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.136 (10.64.0.136) [16:43:08] Error: ERR_READ_TIMEOUT, errno [No Error] at Sun, 07 Oct 2012 16:42:25 GMT [16:43:45] Error: errno [No Error]? .. [16:44:05] that's normal [16:44:11] Krenair: It timed out... the Apaches didn't answer the Squid in time [16:44:21] bsod was nevwer useful actually [16:53:35] Why does the error page on mediawiki.org tell you to go to #wikipedia? :/ [16:54:22] there were several discussions about that [16:54:32] in bugzilla, maillists, on meta [16:54:45] And why is the RC channel #mediawiki.wikipedia? :\ [16:54:59] Was it originally mediawiki.wikipedia.org or something? [16:55:16] mediawiki site is in wikipedia family [16:55:25] same eg. wikimanias [16:56:45] is it just me, or the wikis are sluggish? [16:57:00] everything fall down go boom [16:57:12] So no, not just you :p [16:57:32] MatmaRex: :-) [16:57:48] Wiki is down! [16:57:56] I assume someone is already working on the breakage? pl.wp, commons, all sorts of sites are unreachable/give times [16:57:56] ORL?! [16:57:59] *time outs [16:58:04] Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable at Sun, 07 Oct 2012 16:57:29 GMT [16:59:03] multichill, well if there's not then there probably isn't anyone available. There probably is though [16:59:51] Everything is up, if a little slow [17:00:06] Reedy: Bots are getting 504's all over the place [17:00:19] and you cannot save edits, either. [17:00:22] Yeah, there's at least some ops people around now ;) [17:00:45] this is a good sign that's it's time to do some Sunday evening shopping if you cannot edit. [17:00:55] * odder disappears. [17:01:07] this is actually sign of upcoming fundraiser >:) [17:01:45] aha! the outage was done for a purpose! [17:02:00] Yup, we had nothing better to do with our weekends... [17:02:19] it's Monday somewhere, you ignorant :-P [17:03:02] Danny_B|backup: go read a book or something? :-) [17:03:16] guess what i do here? ;-) [17:03:33] it's e- though... [17:03:38] wait till it's fixed, I guess? :-) [17:03:45] Speaking of which, I have a choice between reading, and then refreshing, or refreshing madly for a bit. Which should I be doing? [17:04:03] What's going on? [17:04:04] Jarry1250: you refresh a book? nice [17:04:13] if you have firefox, instal refreshevery [17:04:18] odder :D I wish. [17:04:33] Danny_B: Nooo but then I lose my POST data [17:04:42] it refreshes on behalf of you... sort of tempomat ;-) [17:04:59] Jarry1250: you won't, it keeps them [17:05:10] * Jarry1250 is sceptical [17:05:19] (as long as you already submitted them) [17:05:22] That edit took me like 30 minutes to write. [17:05:48] Well there's a point *extracts POST data* [17:05:59] Danny_B|backup - what do you mean "sign of the fundraiser"? [17:06:09] Do you mean we're going to block access to the site unless you pay? :p [17:06:18] Sounds like an idea :P [17:06:48] There's a test campaign running for all users atm, btw [17:06:50] that's a new campaing, you didn't know? "wiki is not working for you? we need money for servers, ya'know?" [17:07:04] hoo: What is it? [17:07:13] Nathan2055|code: http://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&method=listNoticeDetail¬ice=C12_bitest [17:07:37] loading... [17:07:41] Oh, one of those. [17:08:25] odder: I'm sure Tim would love to be woken at 4am... [17:08:27] hoo: well, it's blank [17:08:37] odder: But still it's loaded [17:08:47] what does it change if I can't see it? [17:09:20] I can read WP now, but I can't run a POST to login. [17:10:26] apaches moved permanently... ;-) [17:10:54] hi LeslieCarr [17:11:06] hello [17:11:12] okay, so what's the situation [17:11:18] just got off a plane [17:11:46] Everything is dead [17:11:49] Wikipedia doesn't work. [17:12:12] Okay, but in all seriousness, people are having issues loading pages and such on all wikis [17:12:22] 504 errors, etc. [17:12:23] Actually, GETs seem to be working now (maybe?) [17:12:38] I managed to squeeze an edit through at 17:01 UTC [17:12:57] * odder goes to do some shopping. [17:13:15] * Sannita just joined to see the show :P [17:13:34] odder: good idea [17:13:40] Hi Sannita [17:14:32] LeslieCarr: So, for me say (Europe), I have good http:// access even on non-cached pages, very slow https:// access, but I can't get an edit through at all on https:// [17:14:48] thanks for the info point Jarry1250 [17:15:04] I think http:// edits are working: http://en.wikipedia.org/wiki/Special:RecentChanges [17:15:45] (maybe; the numbers seem to low perhaps) [17:16:10] sounds like it would be intermittent write access [17:16:11] :-/ [17:16:38] There's definitely lots of edits getting through [17:16:53] Though I'm still struggling to delete a mediawikiwiki page [17:17:22] Krenair: https or http? [17:17:29] (you, I mean) [17:17:31] https of course [17:17:56] Would it work via http? [17:18:05] (Does it work, rather) [17:18:31] if i have a grasp of the problems, i don't believe http versus https should make a difference for deletions/edits [17:18:42] it will be sporadic issues [17:19:08] It's not specific to deletions/edits, all requests I make seem to be broken [17:19:32] https reads work for me on pl.wiki [17:19:36] https writes, nope [17:20:14] yikes [17:25:01] https deletion few mins ago was ok [17:26:56] lol topic [17:32:37] how's the status? it looks like i managed to save and edit via https 15 minutes ago, but gotten an error, in spite of it being saved [17:33:03] MatmaRex: no change [17:39:10] though it seems ok for the moment [17:41:53] base, lol [17:42:08] not so funny [17:42:23] and certainly clueless [17:42:52] i hit the save button, then cancelled, edited again and hit the save button again. the first posted text has been saved as the second one after the corrected one [17:43:39] Danny_B|backup: doesn't sound unpossible [17:44:06] Danny_B|backup: To exlain that: You can edit conflict yourself and the different editing processes can run on different servers taking different time to complete [17:44:13] i think edit conflicts are ignored if they're both the same user [17:44:38] no they aren't, i've edit conflicted myself a few times [17:44:46] huh [17:44:48] idk then [17:45:12] (clicking save twice when commenting in discussion, tildes expanding to different times) [17:56:24] jeremyb: Who is actually working on fixing this right now? [17:56:30] would I be correct in assuming that the ops folks in this channel are aware that the wikis are all slow and throwing error messages? [17:57:15] multichill/Risker yes we are working on it [17:58:11] LeslieCarr: So start communicating! Don't you have a standard procedure in case of major site failure? [17:58:42] From my side of the world, the procedure is "try to fix it" [17:59:00] This isn't major site failure [17:59:24] what's the underlying problem? [17:59:43] this is a major site failure [17:59:50] every other read and all writes fail for me :( [17:59:52] Reedy: For editors it is. http://status.wikimedia.org/ doesn't not agree with ou [17:59:59] Seems to happen every month now [18:00:10] i would say it's a major site failure ... [18:00:10] http://ganglia.wikimedia.org/latest/?r=4hr&cs=&ce=&m=load_report&s=by+name&c=API+application+servers+pmtpa&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [18:00:11] At least you can access this site :p [18:00:13] there's the problem [18:00:17] Went to main page (en.) go the buggy [18:00:20] *got [18:01:02] we'll probably write a blog post or something when we figure out the cause fo this insanity and fix it [18:01:46] Think maybe someone broke into the server boxes and unplugged them? [18:02:05] I got a Guru Meditation error an hour ago [18:02:19] LeslieCarr: I understand that you're not at the office on a Sunday, right? [18:02:36] it would be appreciated if /somebody/ could write a Twitter update or something [18:02:38] please fix this asap because this is getting annoying... :S [18:02:45] no, i actually just got off a plane and got hte pages [18:02:47] odder: I agree [18:02:54] so i am sitting with my laptop right now until it's fixed [18:03:11] Thank you LeslieCarr for jumping in to try to resolve this. [18:03:12] I would like to delete a vandalism page with cyberbullying and I freqently get an error [18:03:30] should I mail Matthew or whoever is responsible for the Twitter profile? [18:03:30] We are having problems [18:03:42] I just woke up [18:03:48] Trijnstel, which page are you trying to delete? [18:03:51] LeslieCarr: But seriously, doesn't operations have standard procedures in case of site outages? How to communicate etc? That helps to get the people of your back so you can actually fix a problem [18:03:54] is no one handling this? [18:04:05] multichill: she's on a plane [18:04:06] Ryan_Lane: asher and leslie are working it [18:04:13] Ryan_Lane: no she's not [18:04:15] Generalcamo: https://nl.wikipedia.org/wiki/Rage_(computerspel) [18:04:18] I got off the plane [18:04:20] (on the dutch wikipedia) [18:04:21] ah [18:04:21] at sfo [18:04:49] standard communication policy is to let comms know, and they tweet, etc [18:04:52] Might want to make a mental note of it, because I can't access that page [18:05:16] 504 gateway timeout [18:05:33] So do the comms post updates on @Wikimedia? [18:05:36] Ryan_Lane: this has been going on for an hour already, a tweet should have been posted 55 minutes ago [18:05:46] odder: it's a sunday dude [18:05:47] Generalcamo: ah, he's finally gone now! [18:05:51] if uploads are disabled, there should be a warning on Special:Upload [18:05:54] Yes, it is [18:05:55] * odder is not on comcom, so he doesn't know about it. [18:06:18] comms has all the twitter/social media passwords [18:06:19] how about we switch channels to a non-voiced channel? [18:06:20] uploads aren't disabled [18:06:22] so stuff is not supposed to break on Sundays? [18:06:23] they keep going on [18:06:26] currently comms doesn't get paged [18:06:28] and you guys stop bothering us during an outage [18:06:38] The wikimedia Blog is still up [18:06:39] okay... [18:06:41] Trijnstel: they will be turned off tomorrow, 11-14:00 UTC [18:06:44] it's not nice to wait 15 mn to upload a file, and then to get a server error [18:06:47] The blog is a completely different system [18:06:55] when the outage is over, you can complain all you want [18:07:14] yes, but they shouldn't take this long imho [18:07:21] but I'll keep my mouth shut [18:07:26] good luck with fixing this [18:07:30] odder: okay [18:08:07] yannf: there's been an idea of setting up a CentralNotice for commons about the disabled uploads [18:08:10] Wikipedia is back up [18:08:28] ok we are seeing signs of improving [18:08:30] odder, that would be good [18:08:37] Apache severs are recovery [18:08:37] not really, i still can't save changes on mediawiki.org. [18:08:42] yeah [18:08:47] yannf: but it's too late now, we can get the banners in about 10 languages only [18:08:57] (I guess.) [18:08:57] Yep Apache Severs and LVS recovering atm [18:08:59] hey, it worked now. [18:09:03] yeah [18:09:25] ok, just for next time then [18:09:54] you don't need translations for shit like that. [18:10:11] yeah, everyone speaks English on this planet. [18:10:12] you need SOME information more than translated information. [18:10:41] odder: there are 4000. we only support 280 anyways. everything is relative [18:11:01] I'm assuming you guys are talking about you enabling a global site notice, right? [18:11:16] Ryan_Lane: yeah, for the upload maintenance. not related to this per se. [18:11:17] Ryan_Lane: no, just about a CentralNotice for Commons [18:11:28] it's probably a good idea for something to be posted, if uploads are disabled, yes [18:11:31] though I get your point --> uploads will be turned off on all projects :-) [18:11:49] They're not disabled, they're just not necesserily going to succeed [18:11:54] Uploads are bring turned off? [18:12:05] Reedy: that's not what Ariel wrote, is it? [18:12:09] Generalcamo: possibly tomorrow for a short while. [18:12:11] Generalcamo: temporarily [18:12:24] but i guess your question reflects the level of communication between tehcs and users.... [18:12:41] Wrote where? [18:12:57] we used to just run a sitenotice. "Due to maintenance, uploads might not succeed" :D [18:13:05] bureacracy has sort of killed stuff like that. [18:13:13] it was on wikitech-ambassadors IIRC [18:13:21] wikitech-l, http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063667.html [18:13:37] users dont read wikitech-l. [18:13:48] I am answering Reedy's question, MatmaRex [18:13:50] Im gonna bookmark that [18:13:54] They're not disabled now [18:13:58] isnt there a global notifications system on meta? [18:14:02] And we didn't disable uploads on friday [18:14:13] * odder sighs. [18:14:23] Reedy: but they also were not stable :D [18:14:27] Indeed [18:14:31] Reedy: I know they weren't, http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063678.html [18:15:08] but there is the possibility they might be turned off tomorrow: http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063704.html [18:15:23] MatmaRex: users are welcome to subscribe to wikitech-ambassadors if they like [18:15:34] should we then try to have a short banner for this purpose tomorrow? [18:15:37] Only the same possibility as friday [18:15:42] This is offtopic for the moment [18:15:55] is there a set topic for this channel? [18:16:10] Well, when there's outages, talking about future maintenance is not really the best idea [18:16:28] it's being fixed as we speak anyway, and the maintenance is going to happen tomorrow. [18:16:34] yes, outages have priority [18:16:36] the outage seems mostly taken care of now? [18:16:40] seems like it [18:16:45] we are recovering at a fast clip [18:16:56] i wouldn't count on it being finished [18:16:57] from 248 critical to 168 [18:17:51] #wikipedia-fr [18:19:07] I just checked twitter, doesn't look like the general public noticed much. Visibility mostly limited to editing community. [18:20:13] yeag' [18:20:15] *yeah [18:21:37] we are continuing to improve [18:21:48] down to 128 critical on Nagios [18:22:34] Bugzilla is reporting performance issues [18:28:56] One successful upload \o/ [18:29:49] yay [18:31:13] omg! [18:31:21] WE ARE BACK [18:31:29] * Damianz pats domas [18:31:29] :P [18:32:09] Screw bugzilla! ;) [18:32:12] I must have broken something [18:32:16] it doesn't make sense [18:32:22] thank you folks for bringing us back [18:32:23] well, duh [18:32:34] domas, it's always your fault. You should know that by now [18:32:37] Request: POST http://commons.wikimedia.org/w/index.php?title=Special:Undelete&action=submit, from 208.80.154.133 via cp1012.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.133 (10.64.0.133) [18:32:37] Error: ERR_READ_TIMEOUT, errno [No Error] at Sun, 07 Oct 2012 18:27:57 GMT [18:32:50] yannf: that was 5 mins ago [18:33:05] reedy: I don't know what I did to fix it [18:33:09] maybe someone else did something [18:33:10] I tried twice [18:33:17] I just broke NFS few times [18:33:19] All of those people spamming F5 [18:33:23] :-) [18:33:26] lol [18:34:14] Nagios holding steady with 125 critical services (down from 248) and 21 warnings [18:34:32] Mobile site has improved but is still reporting performance issues [18:34:50] French Wikipedia is back up [18:34:54] Is that it then? Is it fixed? [18:35:10] not sure, but looking at status.wikimedia [18:35:17] it seeems a lot of the other language Wikipedias have improved [18:35:40] w:uk:Main Page is the only sever still reporting service disruption [18:35:59] http://status.wikimedia.org/ [18:36:20] http://status.wikimedia.org/8777/163394/Wiki-platform-%5B%5Bw:fr:Main-Page%5D%5D-%28s6%29 [18:36:28] Those do have some delay [18:36:37] yes [18:39:51] There are no longer any service disruptions [18:41:31] ukwiki works for me [18:42:53] yeah [18:42:54] it recovered [18:44:38] does ukwiki have separate server? [18:45:06] or it is main wiki for that server? [18:45:08] yeah [18:45:15] UK Wiki has different server [18:45:59] TheAustinMan: please don't guess... [18:46:03] Ahonc: TheAustinMan is wrong [18:46:05] ... what? [18:46:11] oh? [18:46:13] http://status.wikimedia.org/ [18:46:15] I thought it was different [18:46:20] no [18:46:37] * jeremyb doesn't really feel like explaining now ;-( [18:46:57] :/ [18:47:08] food first! [18:47:35] It's just a wiki selected from the s7 database cluster [18:48:14] Ahonc: https://wiki.toolserver.org/view/Wiki_server_assignments [18:48:28] there are a few servers, each running some wikis [18:48:48] uk.wikipedia is on s7 with 11 other wikis [18:49:53] that wiki link isn't guaranteed to be up to date [18:49:57] bbl [18:50:43] it says it's "updated weekly" [18:50:52] and it seems to actually be updated weekly, looking at history [18:51:26] MatmaRex: better to use db.php ... [18:51:46] i prefer hypertext [18:58:34] thanks for fixing it :)