[00:00:49] Pine: I see them both on enwp right now - can you double check that the gadgets are enabled? [00:01:04] marktraceur: I did, and they are enabled. [00:01:07] Hm. [00:01:18] I can try disabling and re-enabling. [00:01:20] Check your browser's JavaScript error console. [00:01:30] * marktraceur defers to Susan's typing ability [00:01:52] Gah [00:01:59] Just got the Wikipedia equivalent of BSOD [00:02:08] That means you've won. [00:02:09] What did someone do to the server now? [00:02:12] You can go home now. [00:02:14] * Pine looks for spilled coffee [00:02:26] Wikipedia runs on one server. [00:02:45] I refreshed, still BSOD. Anyone else? [00:02:53] We just keep adding RAM [00:03:03] Pine: Define "BSOD" - what actually happens? [00:03:04] TimStarling: https://meta.wikimedia.org/wiki/Terms_of_use is erroring. [00:03:11] Nemo_bis: ^^ [00:03:12] PHP fatal error in /usr/local/apache/common-local/php-1.22wmf3/includes/OutputPage.php line 3071: Call to undefined method User::getNewMessageRevisionId() [00:03:12] it's back [00:03:24] On every page I try to load. [00:03:56] ok, I'm back in [00:04:01] Cncmaster: you still getting errors? [00:04:08] Not anymore. [00:04:11] k [00:04:22] anyway, where was I? [00:04:32] something about Java log I think [00:04:47] JavaScript. [00:05:11] ok just a minute [00:05:26] should be fixed [00:05:48] someone is updating some code [00:06:30] Susan: just for fun I tried a different browser. It works in IE but not in FF. [00:06:38] Now I'll look for that JS log [00:08:56] Hm. Susan: do I need Firebug to get a log? [00:10:23] No. [00:13:01] Ok, how do I get a log then? I did run Firebug while loading the page and it displays no errors. [00:14:11] ah, figured it out! [00:14:22] Thanks, problem was due to a NS block. [02:48:38] greaat [02:48:38] https://en.wikipedia.org/w/index.php?title=RT_-_The_Life_and_Music_of_Richard_Thompson&action=history [02:48:45] a page has been missing for a while now [02:48:48] and no one has noticed [02:52:07] legoktm: o.O [02:52:19] I'm running a query right now [02:53:00] You would think the software would check for the existence of a local article /before/ directing to the interwiki... [02:55:16] mysql> select page_id, page_title from page where page_title like "RT:%" limit 10; [02:55:16] Empty set (1 min 33.38 sec) [02:57:20] something is wrong.... [02:58:18] https://en.wikipedia.org/w/index.php?title=Special:Log&page=RT+-+The+Life+and+Music+of+Richard+Thompson [02:58:22] cant revert it [02:58:25] because the page doesnt exist [02:59:39] where did this page go [03:01:12] Did you check the move logs? Maybe it was moved somewhere else? [03:01:58] ... and the log page auto-lowercases IW links, so Special:Log is useless... [03:02:09] im trying [03:02:09] mysql> select page_id, page_title from page where page_title like "%Richard Thompson" limit 10; [03:02:15] maybe the prefix breaks it or something [03:02:20] Huh... [03:02:22] mysql> select page_id, page_title from page where page_title like "%Richard Thompson" limit 10; [03:02:22] Empty set (12.78 sec) [03:02:53] lets try checking the log table.. [03:03:36] ... that was my suggestion :P [03:03:47] *logs into Toolserver* [03:04:05] mysql> select * from logging where log_type="move" and log_action="move" and log_params like "RT:%" limit 10; [03:04:06] 'night, CP678|Zzzzz [03:04:26] legoktm: yes? [03:04:29] Good night, Matthew_. :-) [03:04:37] waiting for results :P [03:04:56] OK. [03:05:29] ... of course, the TS query killer is being *awesome* like usual :P [03:07:47] query killed :/ [03:08:11] You and me both... I was trying the IW links table (out of curiousity) [03:09:07] I just got killed, again. Blarg... [03:09:26] ok, log_title is indexed [03:10:31] legoktm@nightshade:~$ sql enwiki_p < wtf.sql > ~/public_html/wtf.txt [03:10:32] :P [03:10:51] Heh [03:14:13] A whole lotta nothin' [03:15:15] https://en.wikipedia.org/w/index.php?title=Talk:RT_-_The_Life_and_Music_of_Richard_Thompson&action=history [03:16:40] mysql> select page_id, page_title from page where page_title like "Talk:RT:%"; [03:16:46] someone who has access to thr RT system should be able to move it back, right? [03:16:59] we should be able to do it [03:17:02] if we could find it [03:17:04] in the page table [03:17:25] SELECT * FROM `logging` WHERE `log_type`='move' and `log_title` LIKE "RT:" LIMIT 10 [03:17:28] 0 rows [03:18:33] er right [03:18:47] that would be if someone moved the page from RT: blah --> something [03:19:07] Yeah, I realized that. I just tried the reverse, nada [03:22:58] ok [03:23:03] i'm giving up for now and going to eat [03:23:11] :D [03:23:58] OK [03:25:00] lemme file a bug [03:28:08] Yeah, good idea! [03:28:39] Don't use logging if you can avoid it. [03:29:00] Susan: ?? [03:29:00] https://bugzilla.wikimedia.org/show_bug.cgi?id=48196 [03:29:01] And specify a log_namespace if possible. The key is on (log_namespace, log_title). [03:29:10] ah [03:29:18] logging_ts_alternate or whatever. [03:29:36] legoktm: Same with you. [03:29:43] Right [03:29:47] Though I guess you didn't know if you needed to specify page_namespace. [03:30:24] Problem is, IDK which namespace would be stored here... [03:30:29] 0 [03:31:09] Even if it's an InterWiki? [03:31:12] https://en.wikipedia.org/wiki/RT-_The_Life_and_Music_of_Richard_Thompson [03:32:01] o.O [03:32:19] oh what [03:32:21] Search is still broke [03:32:29] Matthew_: What search? [03:32:37] Susan: Try typing it into the search box [03:32:45] Right, RT: is reserved now. [03:32:48] That's expected behavior. [03:33:12] Meh, it'd be awesome to find the page tho... [03:33:28] Find what page? [03:33:38] The one I just linked? [03:33:48] Yeah. [03:33:56] RESOLVED FOUND. [03:34:10] Yeah, but YK, typing it into the search box doesn't work... [03:34:17] * Matthew_ is slightly confused [03:34:25] https://en.wikipedia.org/wiki/de:en:zh:w:n:de:zoo [03:34:48] Matthew_: Type "google:hello" into the search box. [03:34:51] What happens? [03:35:10] You seem a little confused about how interwiki prefixes work. :-) [03:35:20] https://meta.wikimedia.org/wiki/Interwiki_map is the list. [03:35:28] Susan: I'm aware how interwiki links work. This page should be accessable without people getting sent to rt... [03:35:29] https://meta.wikimedia.org/wiki/Special:Interwiki [03:35:43] Matthew_: File a bug? [03:35:59] Though I imagine one already exists. [03:36:05] This happens with other page titles. [03:36:14] Probably. legoktm just did https://bugzilla.wikimedia.org/show_bug.cgi?id=48196 [03:36:26] That's not a bug. [03:37:04] legoktm: No more uncyclopedia interwiki prefix? [03:37:16] How can I link [[UN:N]]? [03:37:22] It was removed :( [03:37:26] Gay. [03:37:28] I've been meaning to propose it's reinstatement [03:37:34] its [03:37:40] UN:N! [03:37:47] It's like you learned those two words backward. [03:38:44] * Matthew_ shrugs. [03:38:47] Possibly [03:38:48] http://en.uncyclopedia.co/wiki/UN:N [03:38:57] Screw it, I'm sure someone else with a lot more expertise probably took care of it. [03:39:07] I'm going to get food now. [03:39:30] Matthew_: You're so lost. :-) [03:40:37] Yes, yes I am [03:51:32] https://en.wikipedia.org/w/index.php?title=RT-_The_Life_and_Music_of_Richard_Thompson&action=history <-- I don't see a move entry that moved it to the new title [03:57:03] superm401: https://bugzilla.wikimedia.org/show_bug.cgi?id=48183 <-- You can always clone bugs too, of course. [03:57:12] Though it's easy to miss that link in the Bugzilla interface. [03:57:38] Susan, yeah, I forgot about that. [03:57:51] :-) [03:58:49] Does it clone the comments? [03:59:52] Just comment 0, bizarrely. [04:00:00] And it doesn't remember product/component. [04:00:03] It's kind of a strange feature. [04:00:08] It's mostly useful for grabbing the CC list. [04:27:09] Susan: file a bug! [06:27:27] !log on marmontel: disabled apache logrotate, stopped apache, disabled apache init script [06:27:34] Logged the message, Master [06:58:16] HaeB: ping [07:21:52] Nemo_bis: it is report by a user he can't access https only http [07:23:21] matanya: is this still the issue being discussed yesterday? [07:23:21] mantanya: where are they located, and which projects? [07:23:40] yes apergos Nemo_bis [07:23:54] located in IL, in project he.wiki [07:24:17] IL - israel, not ilinois [07:26:03] yeah [07:26:09] ok well I had this happen to me today [07:26:38] was there a problem for someone yesterday too? [07:26:54] yes [07:26:58] several [07:31:45] I see it in the backread [07:32:42] mean, users hitting https and timeout? [07:33:07] so for me it's the same behavior [07:33:24] esams, ssl I don't even ever get to send http headers, because there's never a connection established [07:33:28] but http works fine [07:33:38] f course ping and traceroute are just fine :-/ [07:33:49] yes, exactly [08:01:01] mogge [08:35:54] hashar: is it known that interwikis point to production? http://commons.wikimedia.beta.wmflabs.org/wiki/User:Nemo_bis [08:36:19] Nemo_bis: nice catch :-D [08:36:28] I have no idea how to fix it though [08:36:33] the interwiki system is a bit scary [08:37:06] Ariel made all those nice instructions [08:37:45] but maybe someone followed the "lazy" option that doesn't work (reusing CDB file)? [08:38:14] I have no idea honestly [08:51:02] sometime I will have to look at that [08:51:07] (the interwiki issue) [08:51:26] if there is a bug zomeplace assign it to me, but it will be a little while (weeks?) before I get to it most likely [09:35:32] hello, having problem getting to any wikimedia website from zen.co.uk - not affecting other ISPs as far as I can tell. a traceroute finishes fine, seeking ideas on diagnosing what's up. [09:35:51] since last night [09:40:22] davidgerard: https:// or http:// [09:40:33] the former is known about [09:40:50] aha. https. it'll be because i have https everywhere running in my firefox. [09:41:43] although https://en.wikipedia.org/wiki/Test works from my work connection. odd. [09:42:05] so https is slightly broken for europe, the uk, or small portions thereof? [09:42:44] yep, verified home http:// works in chromium [09:43:06] last I heard, we don't actually know what is causing it [09:43:16] apergos might know more, since he is affected iirc [09:43:34] cool, ok thanks :-) i shall use http:// in chromium for how [09:44:53] it appears to be: [09:45:01] folks trying to get ssl through esams [09:45:18] but I don't know that it's everybody, maybe only some routes [09:45:48] and it is at the packet level as far as we know (i.e. we don't even get to send an http header, failure occurs before that) [09:46:03] apergos: so far its you, David (^) and nat-brown (#mediawiki) and a couple from earlier today [09:46:24] I saw the backread from the couple of earlier folks [09:46:35] apergos: can you do a tcptraceroute to port 443? [09:48:19] traceroute -T -p 443 en.wikipedia.org [09:48:25] 1 192.168.1.1 (192.168.1.1) 4.567 ms 4.495 ms * [09:48:33] 2 * * * [09:48:37] so that wasn't very useful [09:48:38] as mentioned earlier, also some folks in the middle east [09:48:58] (not me though) [09:48:58] lemme see if there are some router settings I can monkey with [09:50:30] eqiad works like a charm, not fine from europe [09:50:36] yes [09:51:18] did traceroute with some fellows. people routing eqiad are ok. people routing to amsterdam are bad [09:52:17] can you post such a bad traceroute? [09:52:49] a sec [09:53:02] mark: http://dpaste.com/1117076/ [09:53:10] and then it hangs [09:53:17] here you go :) [09:53:24] the more people the better [09:53:32] so it stops at telia? [09:53:38] yes [09:53:41] but http works fine, just https doesn't? [09:53:47] let's see [09:53:59] yes here [09:54:11] no, http does not [09:54:33] * aude takes web proxy off [09:54:37] ok so http ALSO doesn't work? [09:54:41] seems so [09:55:00] wouldn't notice as i normally proxy via the US [09:55:33] but for matanya http does work? [09:55:37] yes [09:55:44] can you post your traceroute? [09:55:48] i can access wikipedia [09:56:03] let me get the users with issues a sec [09:56:34] http://dpaste.com/1117088/ [09:57:31] i can login via https okay too [09:58:06] aude: so do you just mean that the traceroute doesn't finish, but http/https connectivity actually work fine? [09:58:12] mark: yes [09:58:20] ok that's normal [09:58:24] ok :) [09:58:39] is telia = amsterdam? [09:59:05] yes [09:59:09] ok [09:59:40] the traceroute to eqiad does not hang, but nonetheless no problem here [10:04:56] mark: http://dpaste.com/1117129/ [10:05:01] no https as well [10:05:39] ok [10:06:58] I can't get this crap huawei router to pass the packets, even when I'm configured for dmz. sorry... [10:12:09] or rather I cannot give it to give me any info from intermediate hops. *sigh* anyways [10:26:59] well I'm going to need more traceroutes from people seeing the problem I'm afraid [10:27:12] i'll try to arrane some [10:27:15] thanks [10:27:48] i have like four which aren't having this, would that help? [10:28:20] mark: i mean, hitting esams but have https [10:28:21] probably not at this stage [10:28:27] ok [10:28:35] lots of people are on esams and have working https ;) [10:28:39] including myself [10:29:20] me too [10:43:46] mark: http://pastebin.com/eDT06rxk [10:46:34] thanks [10:46:40] the only common node I see so far is cr2-knams [10:49:34] that assumes the routing is symmetric, doesn't it? :) [10:50:05] no it doesn't [10:50:21] maybe we should ask for people's IPs to run reverse traceroutes [10:50:36] or maybe apergos should try telnet'ing while tcpdump on the three boxes for their IP [10:51:00] to at least see in which direction the problem is [10:51:11] * mark puts paravoid in charge of this problem [10:51:16] haha [10:51:35] unfortunately it works for me [10:52:09] maybe with apergos' help [10:52:37] let me get on the host in esams and try traceroute back to me [10:52:45] I can do the tcpdumps [10:53:08] * mark goes to breakfast [10:53:10] just give me your IP [10:53:22] just let me know where the problem is afterwards ;) [10:54:02] yessir [10:54:05] ;) [10:54:05] symmetric routing is an exception btw [10:54:08] i never assume that [10:54:34] unfortunately i dunno how helpful it's gonna be, since it's not to port 443 going back [10:54:47] and traceroute -T -p 80 does get there from here [10:54:49] just give me your IP and do "telnet en.wikipedia.org 443" [10:55:09] 77.49.92.189 [10:55:34] ok, do the telnet [10:55:55] well I'm using curl [10:56:03] I'll install telnet in just a sec :-/ [10:56:04] works too [10:56:11] curl -I -v https://en.wikipedia.org [10:56:46] again [10:56:54] doing [10:57:26] fun fact is it worked earlier in the day... then stopped. :-D [11:00:19] nice [11:00:34] traceroute from ssl300* to your IP doesn't even get the first hop [11:00:38] csw1-esams [11:00:42] woot [11:01:44] sounds like csw1 is confused [11:03:15] traceroute from the switch works [11:03:58] check arp entry for the gw [11:04:10] It's 00:0c:db:fc:0b:00 [11:04:15] vrrp iirc? [11:04:19] yes [11:04:23] which is interesting as vrrp is disabled [11:04:31] lemme check cr1 [11:04:47] ha [11:04:48] I was trying to see this on the foundry [11:04:52] vrrp is enabled on cr1 [11:04:54] vrrp config [11:05:01] so ssl3001 is getting cr1 as gateway [11:05:04] but foundries are annoying [11:05:29] and why is cr1 not routing properly? [11:05:42] I guess we should be glad it doesn't in this case but still [11:05:49] it's sending it to csw1 [11:06:07] 77.49.0.0/16 *[BGP/170] 1w4d 03:21:22, MED 100, localpref 250, from 91.198.174.246 [11:06:07] AS path: 1241 I [11:06:07] > to 192.168.66.2 via xe-0/0/3.0 [11:06:10] look at that AS path [11:06:12] that looks messed up [11:06:19] csw1 doesn't have an adjacency with that asn [11:06:42] that's apergos' ISP [11:06:48] oh right [11:06:49] I don't think they're in AMSIX at all [11:06:55] csw1 forwards to cr2 [11:06:58] where they may be [11:07:39] yeah it's on ams-ix [11:07:57] oh wow they are [11:08:53] must be new [11:09:38] i'm going to replace that private address with a 91.198 address now we have the space [11:09:58] it's ugly as hell [11:12:17] heh, yeah [11:14:06] perhaps it's all people ending up on a single ssl server [11:14:12] due to sh hashed LVS [11:14:24] and that ssl server may be using the cr1 vrrp gw, whereas the others use csw1 [11:14:28] I thought of that initially and run tests against all three [11:14:49] but all three have the vrrp ip [11:14:56] and from all three traceroute to apergos' fails [11:15:12] alright, i'll look at that in a bit [11:15:23] traceroute to my IP goes "csw1" -> cr2 [11:15:31] don't trust dns [11:15:42] notice the quotes [11:19:34] pings to my IP (which works) go to the vrrp mac address, replies come back from a mac that belongs to cr1 [11:19:48] er [11:19:59] scratch that [11:20:01] cr2 [11:20:06] probably? [11:20:06] yes [11:20:12] it's juniper [11:20:18] let me verify it's cr2 [11:21:34] yes [11:21:35] cr2 [11:22:18] 5c:5e:ab:d8:63:c1 is the mac I'm seeing [11:22:59] ae1's and all subifs of interfaces that belong to ae1 [11:23:51] yeah that makes sense [11:29:37] seems to be the same path from cr1 to me/apergos [11:29:38] how strange [11:33:34] ok, food [12:04:17] mark / paravoid : http://pastebin.com/vXa2PA22 [12:04:27] one more of the compains ^ [12:04:36] *l [12:06:42] hey, that's me [12:08:58] that's more than my traceroute shows [12:09:09] I have * * * all the way down except first hop (router) [12:09:11] yeah, much better one [12:09:52] a bunch of the "* * *" lines appears almost instantly, not sure why there's no wait (immediate failure ?) [12:10:41] I dunno if it means something, but I'm in EU too and get everything fine, but my traceroute goes thru esams instead of ksams [12:17:57] so [12:18:14] i removed VRRP for that vlan everywhere [12:18:21] QueenOfFrance, my working "port 80" traceroute finishes properly at esams [12:18:23] and still all hosts get ARP answers for the VRRP MAC [12:21:43] ah [12:21:49] 00:0c:db is foundry, not VRRP [13:03:23] that foundry is just mightily confused [13:08:46] fixed hehe [13:09:11] !log Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues [13:09:18] Logged the message, Master [13:09:26] apergos: can you check if it works for you now? [13:09:29] and others as well [13:09:38] yay [13:09:53] success! [13:46:14] all hail mark! king of the routing! [13:58:46] where are security issues are reported apergos ? [13:59:04] i .e in 3rd party packages, not mediawiki [13:59:20] um [13:59:29] we have an alias [14:00:07] what version of nginx is used? [14:00:18] if it is a possible exploit for our servers [14:00:30] for example: http://mailman.nginx.org/pipermail/nginx-announce/2013/000112.html [14:00:45] security@wikimedia.org [14:01:50] is there assigned to reading security issues, and i'll just spam, or it is usefull alert? [14:02:49] meh, that list looks like it's more for mw [14:02:53] less ops more dev [14:03:00] so I would just put in an rt ticket [14:03:05] but in this case [14:03:16] we are running apparently 1.1.19-2 [14:04:02] so not affected. thanks apergos [14:04:06] yw [14:04:09] thanks for looking [14:04:16] np [14:05:29] matanya: Bugzilla -> Security is for anything that effects the wikimedia cluster or mediawiki [14:05:44] thanks p858snake|l [14:05:45]