[06:36:04] anyone else not able to access Wikimedia sites or it is just me? [06:36:13] everything else seems to work just fine [06:37:48] ping is giving "From 162.158.237.1 (162.158.237.1) icmp_seq=1 Time to live exceeded" [06:40:08] 162.158.237.1 is a cloudflare IP, and traceroute seems to be hopping between that and other ips in that /24 [06:58:24] managed to file https://phabricator.wikimedia.org/T273949 per https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue [07:04:11] <_joe_> Majavah: hi, still having issues? [07:04:25] _joe_: yes [07:06:02] Majavah: en.wiktionary.org often doesn't resolve for me. DNA? [07:06:49] Or at least I thought it was DNS, didn't check really yet [07:07:20] <_joe_> https://grafana-rw.wikimedia.org/d/43OLwO2Mk/cdanis-hacked-up-nel-stats?viewPanel=4&orgId=1&refresh=15m seems to indicate an increase in errors from FI [07:07:37] <_joe_> well I can assure you that the WMF's dns servers are alive [07:07:42] Nemo_bis: I have internet from DNA but don't use their name servers, my packets are just getting stuck somewhre in cloudflare's network according to my understanding of traceroute [07:07:57] <_joe_> Majavah: yeah that's my reading as well [07:07:58] Majavah: same [07:07:59] DNS seems to be working just fine for me [07:08:03] <_joe_> ok [07:08:23] <_joe_> lemme see if I can wake someone else up, and we'll take action [07:08:44] asked a friend who also has DNA as their ISP, they have the same problem. works just fine with my mobile connectivity from Elisa and my VPSes in DO and UpCloud [07:08:50] thanks _joe_ [07:09:13] <_joe_> please comment on the task, both of you :) [07:09:32] I'll try, having problems with accessing phab [07:09:38] <_joe_> sigh, ofc [07:09:55] <_joe_> Majavah: https://github.com/cdanis/tunnelencabulator might help you :) [07:10:05] I'll take a look, thanks [07:10:13] at least it's not hosted on Gerrit so I can :D [07:10:16] <_joe_> and forgive the shitposty README :D [07:10:45] <_joe_> that's the tool SREs at wikimedia use when they want to tunnel their traffic through a specific DC [07:11:55] that works, thanks [07:13:02] added a comment and cc'd Nemo_bis on that task [07:18:22] https://github.com/cdanis/tunnelencabulator looks nice but for me personally right now it would be enough to route all browser traffic through a ssh tunnel so I don't think I'll try it just yet :) [07:19:57] Funnily my mosh connection to tools-sgebastion-07 is working without hitches [07:20:43] I can't even access toolforge or anything in WMCS [07:23:00] <_joe_> interesting, now I can't reach the sites either [07:23:21] <_joe_> lol? [07:23:23] Finland one hour ahead ^_^ [07:26:42] mtr fwiw https://paste.debian.net/plain/1184113 [07:29:17] <_joe_> Nemo_bis: I have a similar mtr myself [07:29:43] <_joe_> so, in ~ 10 minutes, hopefully, you should see some effect and be able to connect again [07:31:25] thanks! [07:33:14] <_joe_> (lmk if you still have issues in a few minutes, in the meantime someone with a less cargo-culty knowledge of networks than myself has waken up) [07:42:59] _joe_: seems to be working now, thank you! [07:50:42] Hm I think I still have some issues [07:56:15] <_joe_> https://grafana-rw.wikimedia.org/d/43OLwO2Mk/cdanis-hacked-up-nel-stats?viewPanel=4&orgId=1&refresh=15m&from=now-6h&to=now if you select "FI" you can see the effect [07:56:53] of course it became fine again as soon as I launched mtr, hmm [07:58:04] nice to see a "real" traceroute without the Cloudflare blackbox again ;) https://paste.debian.net/plain/1184116 [08:01:19] <_joe_> Nemo_bis: thankfully that's just a network-layer blackbox, so it happens below our encryption [08:48:07] Oh, lists.wikimedia.org still goes through cloudflare [08:48:42] https://paste.debian.net/plain/1184117 [10:42:07] hello, having issues to recover my account, any help where? password reset dont send a message, I am sure its my account but wasnt use it for a long