[09:34:24] Anyone having trouble loading https://tools.wmflabs.org/xtools/ ? [09:38:09] jup [09:39:59] Hmm it was working just yesterday [09:50:14] its webservice down? [10:02:13] sDrewth: it's hosted on wnflabs server, which seems to be running [10:02:22] *wmflabs [10:02:26] yeah [10:02:33] but there's subservices that run each project/account [16:18:18] Hello all! Just prepping submodule bump for CentralNotice deploy, and getting a 'fatal: reference is not a tree' for submodule vendor in 1.25wmf2 and 1.25wmf3 [16:19:11] core's got vendor at commit d28bd6a in 1.25wmf3 [16:19:34] but that doesn't seem to exist [16:20:28] https://github.com/wikimedia/mediawiki-vendor/commit/d28bd6abdb931d6d8159073989896d88fc13a6aa [16:20:34] WFM [16:21:01] git fetch --all ? [16:21:35] huh, still not fixing it locally [16:23:10] and trying to create local tracking branch in vendor gives 'requested upstream branch does not exist' [16:23:42] Will try a fresh clone of vendor [16:24:52] still can't find that branch / commit [16:24:59] let's see if it's different over ssh [16:26:58] still nothing. I guess it's only a problem if tin can't see it though. [16:27:40] ejegg: which workflow on the deploy instructions are you doing? [16:27:40] oh [16:27:50] ejegg: Is your vendor submodule pointing at the old repo? [16:27:52] git submodule sync [16:28:05] https://github.com/wikimedia/mediawiki-core-vendor [16:28:19] https://gerrit.wikimedia.org/r/p/mediawiki/core/vendor.git [16:28:36] Right, that's the old one [16:28:42] git submodule sync [16:28:50] Oh! [16:29:51] Yep, that did it. Thanks, Reedy! [16:29:56] yay [16:30:20] woohoo! (from the peanut gallery :) ) [16:30:34] * Reedy wonders when ejegg last tried to deploy [16:30:34] ;) [16:31:30] Reedy: definitely before 1.25! [17:09:53] m.wikisource.org is down [17:10:05] wikisource.org is working fine though [17:13:24] Glaisher, has it ever been up? [17:13:35] I dunno [17:13:43] Glaisher: is that supposed to work? https://en.m.wikisource.org/wiki/Main_Page works [17:14:18] we probably just have never used this particular subdomain [17:14:32] not enwikisource [17:15:09] https://m.wikisource.org/w/index.php?title=Main_Page&mobileaction=toggle_view_mobile [17:15:37] Glaisher: oh, wait, there also is a non-language wikisource [17:15:44] I missed that [17:15:57] aka oldwikisource [17:16:28] Ye Olde WikiSource [17:22:09] I guess it did at once [17:22:10] https://bugzilla.wikimedia.org/show_bug.cgi?id=36002 [17:22:45] and then there's an open bug about the issue https://bugzilla.wikimedia.org/show_bug.cgi?id=69765 [18:04:10] Glaisher, committed fixes, waiting for review [18:04:27] yeah. I saw [18:04:30] thanks :) [19:02:50] just gonna repost here since there's more people here, and probably administrators [19:02:50] [20:00:11] hey guys, is wikimedia being DDoS'd? [19:02:50] [20:00:20] bits.wikimedia.org seems to be having problems [19:02:50] [20:00:31] I'm in the UK if that helps locate the correct server [19:02:50] [20:02:11] seems to have improved over the past 10 minutes, but wikipedia is still running sluggish purely because of bits.wikimedia.org seeming to take forever to load [19:04:39] https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Bits%20caches%20esams&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1413486231&g=network_report&z=large looks ok [19:04:42] Someguy123: You might want to try making a traceroute to bits and pastebin it if it's still slow [19:05:15] oh... [19:05:24] there's packet loss on the intermediay routes once it hits london [19:05:52] https://cdn.mediacru.sh/yV5qXJvyiV9a.png [19:05:55] Reedy, ^ [19:06:12] whatever bb.gin.ntt.net is, is having serious packet loss issues [19:06:19] That's slightly interesting [19:06:27] What ISP are you on? [19:06:35] /What DNS servers? [19:06:38] Hitting eqiad? [19:06:44] Level3 [19:06:45] You're targeting the servers in the USA, not Amsterdam [19:06:53] aka 4.2.2.3/4 [19:07:18] Try traceroute to bits-lb.esams.wikimedia.org ? [19:07:50] the intermediary is now init7 [19:07:55] which also has packet loss surprisingly [19:08:22] https://cdn.mediacru.sh/qV0hrfxjupkA.png [19:08:24] very strange [19:08:42] Krenair, Reedy ^ [19:08:46] What ISP are you using? [19:08:54] EE/Orange [19:09:09] this is only affecting wikipedia, nothing else [19:09:40] What does en.wikipedia.org resolve to? [19:10:05] 208.80.154.224 [19:10:21] (text-lb.eqiad.wikimedia.org) [19:10:47] I guess you're using their provided DNS servers? [19:10:52] nope [19:10:57] my DNS is 4.2.2.3 and 4.2.2.4 [19:10:57] What are you using? [19:10:58] Level3 [19:11:14] didn't know level3 did public dns [19:11:28] it's not necessarily public, but it's there, it's fast, and never goes down [19:11:35] has always been more reliable for me than google DNS [19:11:40] and is just as memorable as 8.8.4.4 [19:11:44] Uhhh [19:11:50] Googling 4.2.2.3 suggests it's Verizon [19:12:10] OrgName: Level 3 Communications, Inc. [19:12:13] whois shows level3 though [19:12:16] ^ [19:12:19] http://whois.domaintools.com/4.2.2.3 [19:12:22] Level3 ^ [19:12:56] It's probably Verizon offloads it to Level3 [19:13:37] Either way, it's resolving based on US geography it seems [19:14:02] It is c.resolvers.level3.net. [19:14:53] Reedy, correct, but even so, this points out there is a problem somewhere in some network... [19:15:01] Probably [19:15:04] no other US-based site is having issues for me [19:15:08] so I find it very odd [19:15:17] But you're not using optimum servers for wikipedia [19:15:26] Reedy, the issue is, UK DNS servers suck. [19:15:31] I know :) [19:15:41] I prefer my nice, unfiltered, and no weird gateway pages level3, or googleDNS [19:15:44] I think I'm still using OpenDNS, which is meh [19:16:01] openDNS always seemed to do slow updates for me [19:16:16] which annoyed me a lot, since I do a lot of web development stuff, including managing peoples domains [19:16:32] only slightly better than my ISP's DNS [19:16:33] You can purge the opendns caches [19:17:17] http://aa.net.uk/kb-broadband-serverlist.html [19:17:24] Reedy, the other issue is lazyness :P the first thing I do on any PC in my house, is change it to either googleDNS, or level3, purely because I know the IP's [19:17:45] There's a lot of routes to the US... There's a fucktonne of datacentres there too [19:17:50] Would expect it to be anycast [19:18:01] So picking up the local one [19:18:26] my browser might... ping however might not [19:18:35] the IP I gave, was just from pinging in CMD [19:18:58] mmm [19:18:59] 11 c.resolvers.level3.net (4.2.2.3) 147.140 ms 147.208 ms 144.909 ms [19:19:11] hop before that is level3 london [19:19:13] Remote Address:208.80.154.224:443 [19:19:13] Request URL:https://en.wikipedia.org/wiki/Main_Page [19:19:15] 10 * ae8.edge4.London.Level3.net (4.68.111.25) 149.168 ms 147.419 ms [19:19:27] guess it is that IP, even in my browser [19:19:35] so chrome isn't changing that [19:22:21] Reedy: Hehe, I just end up in Frankfurt for 4.2.2.3 [19:22:35] multichill: I guess that means the anycast is working at least [19:22:44] Why it's resolving to EQIAD for WP though [19:23:18] Nice, advertised as 4.0.0.0/9 [19:24:11] Reedy, anything else I can do to help here? [19:24:29] it's annoying me having to wait 5 minutes for each page to load because bits.wikimedia is taking 10 years to load [19:24:45] Someguy123: hack your hosts file? :P [19:25:04] uhmmm waht? [19:25:05] C:\Users\Chris>nslookup en.wikipedia.org [19:25:05] Server: c.resolvers.level3.net [19:25:05] Address: 4.2.2.3 [19:25:05] Non-authoritative answer: [19:25:06] Name: en.wikipedia.org [19:25:07] Addresses: 2620:0:861:ed1a::1 [19:25:09] 208.80.154.224 [19:25:13] since when did level3 support IPv6? [19:25:35] especially on an ipv6 DNS server [19:25:40] ipv4* [19:26:28] Someguy123: They've probably been doing ip6 routing for the last 20 years :P [19:26:50] multichill, I know that, but I mean their DNS server [19:27:09] Probably about the same time [19:27:10] not even google public DNS has ever gave me IPv6 stuff [19:27:29] wow, okay. guess even google does now [19:27:39] That's just an AAAA record [19:27:48] also [19:27:48] https://cdn.mediacru.sh/Q5d7ghKN00UF.png [19:28:24] might help ^ [19:28:26] The other records don't really make sense for bits [19:29:42] multichil@bits.wikimedia.org [19:29:54] Reedy, btw even though I like AA, "These 2 servers are caching DNS servers - typically used on your legacy TCP/IP settings." [19:30:07] was about to actually use their DNS servers you linked me to [19:30:21] but the fact that they're caching tells me I'm going to have nightmares the next time I touch a domain [19:43:00] so Reedy multichill what's going to happen? [19:43:11] Not a lot [19:43:16] this has been a rather intermittent problem over the past week, and I really don't know why [19:43:48] Not hitting roundtripping across the atlantic would be of some benefit [19:44:59] Reedy, but, there is still packet loss to 91.198.174.202 [19:45:16] which is the bits.wikimedia.org supposedly nearest to the UK [19:45:19] and it's just as bad [19:45:31] Right, it's in Amsterdam [19:45:40] https://cdn.mediacru.sh/0p4BhgFTlm1V.png [19:46:50] it's not the first time we have issues with level3 is it [19:47:58] well, that packet loss isn't their fault [19:48:00] this time it's my ISP's DNS ^ [19:48:05] that's what my ISP returns [19:48:16] Which is where you do want to be targetting [19:48:35] so apparantly both ntn which is a japanese routes provider, and init7 are having strange packet loss to that server [19:48:38] but nowhere else [19:48:57] Ops *may* be able to do something to help that [19:49:16] at first I thought it was my internet being terrible, but I don't get packet loss to any other server, and video streaming, gaming etc. is all perfect [19:49:37] Ignoring those first 2 with 100% packet loss... [19:49:59] the first two is simply my ISP doesn't respond to ICMP packets [19:50:04] Right [19:50:05] but they DO pass them on [19:51:11] Those further down aren't likely your ISPs fault, other than maybe who they decide to peer with [19:57:26] Reedy, I'm struggling to find any other major site which goes through either NTT or Init7, those that don't, including google, facebook, youtube, freenode etc. seem to have no problems with packet loss at all [19:57:45] so I'm going to assume that there's a fault on both NTN and Init7's routing network, not sure how [19:57:48] NTT are a pretty big transit provider [19:57:56] maybe they both use the same transatlantic cable, and there's a fault there [19:58:37] NTT*