[00:35:52] 10Operations, 10Security-Team: can we get rid of rsvg security patch? - https://phabricator.wikimedia.org/T104147#3684820 (10MaxSem) [00:35:57] 10Operations, 10Release-Engineering-Team, 10WorkType-NewFunctionality: [Spike] Try out hack ( 10Operations, 10Goal, 10HHVM: Complete the use of HHVM over Zend PHP on the Wikimedia cluster - https://phabricator.wikimedia.org/T86081#3684818 (10MaxSem) 05Open>03declined With HHVM ditching PHP compatibility, we're going in the direction opposite to this task. [00:39:39] 10Operations, 10Gerrit, 10ORES, 10Scoring-platform-team, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684824 (10Paladox) There’s a 2.13 branch https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13 Docs is at https://gerrit.googlesource.com/plugins/lfs/+/st... [00:42:48] 10Operations, 10Dumps-Generation, 10HHVM, 10Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#3684825 (10Jdforrester-WMF) The parent task has been declined, with the view that we've moving to PHP7 instead. Should this task be re-titled to reflect that? [00:43:27] 10Operations, 10Deployments, 10Beta-Cluster-reproducible, 10HHVM, and 3 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#3684826 (10Jdforrester-WMF) [00:57:09] (03PS1) 10Chad: Stop copy+pasting squid/reverse proxy settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384193 (https://phabricator.wikimedia.org/T104148) [00:59:37] 10Operations, 10MediaWiki-Platform-Team, 10Performance-Team, 10HHVM: Convert Wikimedia production HHVM instances to have hhvm.php7.all set true - https://phabricator.wikimedia.org/T173786#3684851 (10MaxSem) So I tried briefly enabling the PHP7 mode on deployment-prep, it just immediately exploded with `PHP... [00:59:52] 10Operations, 10Gerrit, 10ORES, 10Scoring-platform-team, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684852 (10demon) We could possibly do FS-based for now, pending work on supporting Swift. [01:01:58] 10Operations, 10Gerrit, 10ORES, 10Scoring-platform-team, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684856 (10demon) The `/srv/` partition on Gerrit is only 1% full, we've got plenty of space. [03:29:36] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 845.54 seconds [03:52:55] !log manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio) [03:52:58] mutante: ^^ [03:53:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:54:06] 4000 cronspam mails [03:59:16] PROBLEM - Host cp3048 is DOWN: PING CRITICAL - Packet loss = 100% [04:00:36] RECOVERY - Host cp3048 is UP: PING OK - Packet loss = 0%, RTA = 83.75 ms [04:25:56] PROBLEM - IPv4 ping to ulsfo on ripe-atlas-ulsfo is CRITICAL: Traceback (most recent call last) [04:26:37] PROBLEM - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is CRITICAL: Traceback (most recent call last) [04:26:46] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is CRITICAL: Traceback (most recent call last) [04:28:06] PROBLEM - IPv4 ping to codfw on ripe-atlas-codfw is CRITICAL: Traceback (most recent call last) [04:28:26] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is CRITICAL: Traceback (most recent call last) [04:29:17] PROBLEM - IPv4 ping to eqiad on ripe-atlas-eqiad is CRITICAL: Traceback (most recent call last) [04:51:56] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 232.14 seconds [04:58:37] (03PS1) 10BryanDavis: maintain-views: Add log types to logging_whitelist [puppet] - 10https://gerrit.wikimedia.org/r/384201 (https://phabricator.wikimedia.org/T178052) [05:58:35] (03CR) 10Brian Wolff: [C: 031] maintain-views: Add log types to logging_whitelist [puppet] - 10https://gerrit.wikimedia.org/r/384201 (https://phabricator.wikimedia.org/T178052) (owner: 10BryanDavis) [08:03:06] RECOVERY - IPv4 ping to codfw on ripe-atlas-codfw is OK: OK - failed 0 probes of 288 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [08:03:27] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is OK: OK - failed 9 probes of 279 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [08:04:26] RECOVERY - IPv4 ping to eqiad on ripe-atlas-eqiad is OK: OK - failed 0 probes of 289 (alerts on 19) - https://atlas.ripe.net/measurements/1790945/#!map [08:05:57] RECOVERY - IPv4 ping to ulsfo on ripe-atlas-ulsfo is OK: OK - failed 0 probes of 291 (alerts on 19) - https://atlas.ripe.net/measurements/1791307/#!map [08:06:46] RECOVERY - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is OK: OK - failed 8 probes of 278 (alerts on 19) - https://atlas.ripe.net/measurements/1790947/#!map [08:06:47] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is OK: OK - failed 7 probes of 278 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [08:21:26] PROBLEM - HHVM rendering on mw2215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:16] RECOVERY - HHVM rendering on mw2215 is OK: HTTP OK: HTTP/1.1 200 OK - 73574 bytes in 0.292 second response time [08:36:34] 10Operations, 10Goal, 10HHVM: Complete the use of HHVM over Zend PHP on the Wikimedia cluster - https://phabricator.wikimedia.org/T86081#966223 (10HappyDog) >>! In T86081#3684818, @MaxSem wrote: > With HHVM ditching PHP compatibility, we're going in the direction opposite to this task. Reference: http://hhv... [08:40:10] 10Operations, 10Goal, 10HHVM: Complete the use of HHVM over Zend PHP on the Wikimedia cluster - https://phabricator.wikimedia.org/T86081#3685017 (10Reedy) I thought ops still wanted to do this,at least for wikitech to get to a level playing field? [08:42:07] (03CR) 10Zoranzoki21: [C: 031] "Looks good to me, but someone else must approve" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384193 (https://phabricator.wikimedia.org/T104148) (owner: 10Chad) [09:14:51] 10Operations, 10Dumps-Generation, 10HHVM, 10Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#3685020 (10ArielGlenn) Let's leave it for now. I've been following that discussion closely, but if I get to update the snapshot boxes to jessie before stre... [09:38:56] (03PS1) 10Hoo man: Test different batch sizes in dumpwikidatajson.sh [puppet] - 10https://gerrit.wikimedia.org/r/384204 (https://phabricator.wikimedia.org/T177486) [09:39:45] whoops, 0 * 500 = 0 … that doesn't make a good batch size [09:42:35] (03PS2) 10Hoo man: Test different batch sizes in dumpwikidatajson.sh [puppet] - 10https://gerrit.wikimedia.org/r/384204 (https://phabricator.wikimedia.org/T177486) [09:43:18] (03CR) 10Zoranzoki21: [C: 031] "Looks good to me, but someone else must approve" [puppet] - 10https://gerrit.wikimedia.org/r/384204 (https://phabricator.wikimedia.org/T177486) (owner: 10Hoo man) [10:00:58] (03CR) 10Zoranzoki21: "OK" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [10:02:40] 10Operations, 10DBA: Rename be_x_oldwiki database to be_taraskwiki - https://phabricator.wikimedia.org/T127570#3685043 (10Liuxinyu970226) [10:02:42] 10Operations, 10DBA, 10Wikimedia-Site-requests: script & docs to rename wiki databases - https://phabricator.wikimedia.org/T83609#3685041 (10Liuxinyu970226) [10:03:39] 10Operations, 10DBA, 10Wikimedia-Site-requests: script & docs to rename wiki databases - https://phabricator.wikimedia.org/T83609#916322 (10Liuxinyu970226) I bold to mark both as parent tasks, like what happened in T112426 [10:18:15] (03PS5) 10Zoranzoki21: Rename $wmf* to $wmg* in wmf-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) [10:19:27] (03CR) 10jerkins-bot: [V: 04-1] Rename $wmf* to $wmg* in wmf-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [10:19:41] (03PS6) 10Zoranzoki21: Rename $wmf* to $wmg* in wmf-config and tests folder [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) [10:21:03] (03CR) 10jerkins-bot: [V: 04-1] Rename $wmf* to $wmg* in wmf-config and tests folder [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [10:22:43] (03CR) 10Zoranzoki21: "Next?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [11:00:32] 10Operations, 10media-storage, 10User-fgiunchedi: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite"." - https://phabricator.wikimedia.org/T173374#3685106 (10Aklapper) >>! In T173374#3607772, @fgiunchedi wrote: > @nick could you try again to delete... [12:16:18] (03Abandoned) 10Paladox: Gerrit: Increase receive.maxObjectSizeLimit to 200m temporarily [puppet] - 10https://gerrit.wikimedia.org/r/384085 (https://phabricator.wikimedia.org/T178189) (owner: 10Paladox) [12:56:33] (03CR) 10Steinsplitter: [C: 031] maintain-views: Add log types to logging_whitelist [puppet] - 10https://gerrit.wikimedia.org/r/384201 (https://phabricator.wikimedia.org/T178052) (owner: 10BryanDavis) [13:04:29] (03PS1) 10ArielGlenn: make the 'latestlink' job not do cleanups of old files or write status files [dumps] - 10https://gerrit.wikimedia.org/r/384211 [16:41:06] (03PS1) 10Ladsgroup: Enable blocking feature of abuse filter in fawikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384252 [16:43:19] (03PS2) 10Ladsgroup: Enable blocking feature of abuse filter in fawikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384252 (https://phabricator.wikimedia.org/T178227) [16:53:27] (03CR) 10Zoranzoki21: [C: 031] Enable blocking feature of abuse filter in fawikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384252 (https://phabricator.wikimedia.org/T178227) (owner: 10Ladsgroup) [16:53:49] (03CR) 10Andrew Bogott: [C: 031] maintain-views: Add log types to logging_whitelist [puppet] - 10https://gerrit.wikimedia.org/r/384201 (https://phabricator.wikimedia.org/T178052) (owner: 10BryanDavis) [17:46:17] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:11:17] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:33:20] (03CR) 10Huji: "I think it is a good opportunity to also enable range block: `$wgAbuseFilterActions = [ 'rangeblock' => true ];`" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384252 (https://phabricator.wikimedia.org/T178227) (owner: 10Ladsgroup) [20:17:31] (03Draft1) 10MarcoAurelio: Enable AbuseFilter profiler for es.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384265 (https://phabricator.wikimedia.org/T177760) [20:17:37] (03PS2) 10MarcoAurelio: AbuseFilter configuration changes for es.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384265 (https://phabricator.wikimedia.org/T177760) [20:22:01] (03CR) 10MarcoAurelio: [C: 04-1] Enable blocking feature of abuse filter in fawikiquote (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/384252 (https://phabricator.wikimedia.org/T178227) (owner: 10Ladsgroup) [21:26:06] (03PS8) 10Umherirrender: Add ar_content_format and ar_content_model to labs views [puppet] - 10https://gerrit.wikimedia.org/r/363851 (https://phabricator.wikimedia.org/T89741) [21:44:58] (03CR) 10Zoranzoki21: [C: 031] use the logstash LVS endpoint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383355 (https://phabricator.wikimedia.org/T175242) (owner: 10Gehel) [21:47:35] (03CR) 10Zoranzoki21: "What now to fix? I no found nothing" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [21:50:57] (03CR) 10Dereckson: "Do you see how to get the URL of the failing test log?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [21:55:05] (03CR) 10Dereckson: "If not: 1. you go to the last jenkinsbot comment. 2. You'll see the Jenkins jobs done, and the report, you want to check each one with a F" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [21:55:48] (next tip should be grep -r) [21:57:25] (03CR) 10Zoranzoki21: "OK" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [22:05:17] (03CR) 10Zoranzoki21: "I can not work on this patch anymore. I'm leaving this to somebody else." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [22:16:56] (03Abandoned) 10Dereckson: Rename $wmf* to $wmg* in wmf-config and tests folder [mediawiki-config] - 10https://gerrit.wikimedia.org/r/383944 (https://phabricator.wikimedia.org/T45956) (owner: 10Zoranzoki21) [22:18:29] (03Abandoned) 10Dereckson: Configure he.wiki images size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/31580 (https://phabricator.wikimedia.org/T43712) (owner: 10Dereckson) [22:28:14] (03PS4) 10Dereckson: multiversion: Fix PHP notice when no argument is given [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371967 (https://phabricator.wikimedia.org/T173342) [22:40:18] Hi, ZxelA is having problems connecting to Wikimedia services [22:40:36] specifically phabricator which is on misc-web-lb [22:41:44] their text-lb traffic goes to ulsfo, but I think misc-web-lb is slightly weird [22:42:06] phabricator traceroute: https://www.irccloud.com/pastebin/FQxN2aRN/ [22:43:09] that'll be to 208.80.153.248 aka misc-web-lb.ulsfo (but also misc-web-lb.codfw) [22:43:23] bblack, paravoid [22:44:33] XioNoX [22:47:33] Krenair: https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue [22:47:53] (I'm on my phone) [22:48:16] ZxelA, ^ [22:48:22] haven't seen that page before [22:48:52] i can get to my laptop in not too much time if needed [22:49:04] 1 person isn't an outage ;) [22:49:13] Krenair: yeah I wrote it recently :) [22:49:18] it's 1 confirmed person hitting misc services [22:49:19] Krenair: I'd check that out, but it times out for me as well lol [22:49:27] who knows the actual extent of the problem [22:49:35] ZxelA: https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue [22:49:47] hah okay [22:49:53] ^ that's hosted offsite [22:50:20] Reedy: alright that loaded [22:51:49] can you ping phabricator.wikimedia.org? [22:52:34] yeah ping looks fine [22:52:52] um [22:53:14] hm ok [22:53:39] ZxelA, so what happens when you try to connect to phab? [22:55:20] I get a server timed out response currently. [22:55:42] same with wikitech [22:56:00] ZxelA, can you follow the curl instructions on that page? [22:56:03] Sure [22:58:13] curl to phab https://www.irccloud.com/pastebin/QCbPcebv/ [22:58:24] seems like I'm getting something so um.. [22:59:39] well okay do it with https:// ? [23:00:43] can you follow the 301? [23:01:02] curl the "location:" field [23:01:30] that's basically the same thing :D [23:01:50] yup :) [23:02:58] done, though it never actually seems to complete after line 11 https://www.irccloud.com/pastebin/0xwf83mR/ [23:03:02] (03PS6) 10Bearloga: Add profiles/roles for stats/ML on Wikimedia Cloud [puppet] - 10https://gerrit.wikimedia.org/r/383916 (https://phabricator.wikimedia.org/T178096) [23:03:32] right so something is causing TLS issues with your connection [23:03:57] the traceroute thing was a red herring, I checked my own and it breaks in a similar way [23:04:22] hmm alright [23:04:38] glad we narrowed it down at least [23:05:16] Delicious, delicious herring. [23:07:07] ZxelA, what kind of network are you on? [23:07:48] Krenair: It's residential DSL through AT&T [23:08:01] so shouldn't be any deliberate TLS blocking going on [23:10:30] ya, I tried from my phone as well (on the same wireless network) and that's timing out too. So doesn't seem to be anything on the computer that's broken. [23:11:03] Might be some of AT&T's equipment here that's gone bad. [23:11:29] ZxelA, can you try adding a '-4' as an argument? [23:11:41] does that make a difference? [23:12:07] (it shouldn't) [23:12:51] Krenair: with -4 I actually get past the hello [23:12:58] oddly enough.. [23:13:33] what more do you get? [23:14:27] https://www.irccloud.com/pastebin/GSrxeNyb/ [23:15:52] well [23:16:33] that looks like a proper phab response [23:16:44] so you have TLS problems when talking to Phabricator over IPv6 [23:17:43] yeah so it seems [23:18:10] does ipv6 work generally? [23:18:50] one of his previous pastes demonstrated plain unencrypted HTTP working over IPv6: https://www.irccloud.com/pastebin/QCbPcebv/ [23:19:10] let's play about with curl options some more [23:19:16] what if you replace -4 with -3? [23:19:47] Typically it does [23:19:49] ipv3? :P [23:19:54] lol [23:20:26] Reedy, no, -1 through -3 are SSL versions [23:20:57] https://www.irccloud.com/pastebin/BPy0MXQJ/ [23:21:41] that's expected [23:21:53] what about -1? [23:22:38] hm actually that won't tell us anything we don't already know [23:22:41] the command times out at "TLSv1.2 (OUT), TLS handshake, Client hello (1):" [23:23:25] --tlsv1.0 and then --tlsv1.1 ? [23:24:30] 1.0 times out at hello, and 1.1 completes successfully [23:24:55] that's kinda weird [23:25:52] so when connecting to phabricator, TLSv1.0 and TLSv1.2 fail over IPv6 but are fine (TLSv1.2 tested, TLSv1.0 assumed) over IPv4 [23:27:09] can you curl https://en.wikipedia.org ? [23:27:27] uh, with a -v [23:27:54] sure [23:28:28] https://www.ssllabs.com/ssltest/analyze.html?d=phabricator.wikimedia.org&s=2620%3a0%3a860%3aed1a%3a0%3a0%3a3%3ad&latest [23:28:36] TLS 1.3 No [23:28:36] TLS 1.2 Yes [23:28:36] TLS 1.1 Yes [23:28:36] TLS 1.0 Yes [23:29:30] Same for https://www.ssllabs.com/ssltest/analyze.html?d=phabricator.wikimedia.org&s=208.80.153.248 [23:29:36] yeah [23:29:48] Krenair: That times out at hello, and of course I can't load it up in a browser now lol [23:30:18] I think AT&T's 6rd ipv6 is being wonky... [23:30:19] Reedy, I did check that those should succeed locally [23:31:26] ZxelA, wait so you can't even load wikipedia anymore? [23:31:41] nope [23:33:53] I would assume AT&T 6rd, yes [23:34:14] the reason HTTP works but not HTTPS is that the HTTP transaction (GET -> 301) is very small, so it's not tripped MTU-related issues [23:34:52] I just disabled 6rd on the router and everything is loading perfectly now, yup. [23:34:58] I'm at AT&T u-verse vdsl customer as well. every few months I try again to get their 6rd working, and then end up turning it back off in frustration a day or so later. [23:35:18] lol [23:35:18] [00:18:09] does ipv6 work generally? [23:35:22] I guess that's a no then :P [23:35:24] Yep, it's u-verse. [23:35:53] it's mostly that their stupid router doesn't handle the path mtu issues appropriately [23:36:11] bblack, note they were able to successfully use TLS v1.1 [23:36:13] the tunnel is 1472, but it doesn't tell v6 clients that, so they use the 1500 [23:37:01] tlsv1.0/1/2 all have slightly different ways they play out. I can't remember the details off-hand. [23:37:15] I guess the other two versions triggered MTU problems [23:37:52] gotta wonder why this kind of problem would randomly appear where it hasn't before [23:38:34] Zxel: if you want to experiment more, you could try turning 6rd back on (on the AT&T router), and then assuming you don't have any other 3rd party router between you and it: manually set your laptop's ethernet MTU down to something smaller like 1460 [23:38:45] and then it might magically work and confirm some things [23:39:09] Would anyone object to me deploying a one-line workaround for T177825 ? [23:39:10] T177825: No on-site Notifications received for certain type of actions - https://phabricator.wikimedia.org/T177825 [23:39:18] bblack: I could give that a shot [23:39:32] It's a Saturday but this bug breaks all notifications for a couple thousand people [23:40:50] RoanKattouw: Of course, it's also been broken for a while ;) [23:40:55] Yeah since Monday [23:41:13] I thought there were some of the new wikis it's been broken on for ages? [23:41:28] Or is that actually a different issue? [23:41:32] Maybe [23:41:46] T174873 has been since 3rd Sept [23:41:46] T174873: Ping system failed to notify mention on hi.wikiversity and hiwikivoyage - https://phabricator.wikimedia.org/T174873 [23:41:52] Oh right [23:42:22] "'Catrope' == 0 is true" [23:42:26] ah PHP [23:42:37] That was intermittent though [23:42:48] Elena reported that some of her pings did and some did not come through on the Hindi wikis [23:43:10] Krenair: Yeah it has a low opinion of me, doesn't it. Thinks I'm a zero [23:43:26] bblack: yeah, it works fine with the lower mtu with 6rd ipv6 [23:43:34] yeah [23:43:56] the problem the router doesn't advertise that properly, and you probably don't want to be stuck at the odd/smaller mtu for all your v4 traffic either [23:44:07] it's kind of a lose/lose :) [23:44:22] I think I'll just keep ipv6 disabled.. less of a headache [23:44:26] Then again even with the cause I found, people would still be getting notifications from anons [23:44:28] you could set up a custom per-route MTU if it's linux, but it's a PITA [23:44:49] perhaps one day AT&T will figure things out :P [23:45:17] But there are also reports of notifications arriving hours later, so maybe there are two separate issues [23:47:32] Krenair: thanks for helping troubleshoot that issue [23:47:42] maybe email their noc address? [23:49:34] it's the router firmware in the home devices that's the issue [23:49:47] probably their 6rd works fine once you get past that hurdle [23:52:24] ya, just browsing their customer support forums.. it seems the accepted solution is to "turn ipv6 off" lol. [23:52:52] is that owned by atnt? and can they upgrade it remotely? [23:53:06] It is [23:53:54] still gotta fix the firmware first ;)