Netgear R7000, DD-WRT, IPv6 and the lack of a stable gateway

After playing with a few IoT related devices and scenarios, I wanted to finally enable IPv6 in my home network. After looking if my ISP (NET Virtua) was already supporting IPv6, I was happy to find out that they were already claiming that my city (Florianópolis) was covered, so I decided to give that a try (and post my setup/experience).

I’m using Netgear R7000 as my main router, simply because it’s known to be well supported by DD-WRT.

Installing DD-WRT on Netgear R7000 is quite an easy job, and there is also an huge list of links covering several useful tips at http://www.dd-wrt.com/phpBB2/viewtopic.php?t=264152, which is quite handy (overclocking, QoS, installing Tor, etc).

After updating DD-WRT to the latest stable build from Kong (v3.0-r29300M kongac) and enabling bridge mode on my cable modem (from Arris), it was finally time to start playing with IPv6🙂

I first configured my IPv4 network to use DNSMasq for both DHCP and DNS. Being able to use DNSMasq is great because it is quite easy to configure, besides supporting the IPv6 Router Advertisement feature, which is quite handy for my own IPv6 network (no need for radvd).

DD-WRT IPv4 Setup

Then I just enabled IPv6 by going to the Setup->IPv6 tab. I’m using DHCPv6 with prefix delegation since that is supported by my ISP, making it even easier to configure my own network. Since I’m using DNSMasq for DHCP, I can safely disable radvd, but an additional step is required (custom DNSMasq config).

DD-WRT IPv6 Setup

To configure DNSMasq just go to the Services->Services tab, and you will find a field that can be used to store your custom config entries. For a complete overview and description of the supported options, please check http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html. In my case, all I needed was to define define the dhcp-range, dhcp-option, ra-param and enable-ra options.

DNSMasq Configs

Time to reboot and check if everything would indeed work as expected. Once booted, I decided to check the setup over telnet, since it’s easier to debug and understand what is going on🙂

Using telnet is quite easy, and enabled by default in DD-WRT (internal network only):

rsalveti@evapro:~$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.

DD-WRT v3.0-r29300M kongac (c) 2016 NewMedia-NET GmbH
Release: 04/14/16

domosys login: root
Password: 
==========================================================
 
     ___  ___     _      _____  ______       ____  ___ 
    / _ \/ _ \___| | /| / / _ \/_  __/ _  __|_  / / _ \
   / // / // /___/ |/ |/ / , _/ / /   | |/ //_ <_/ // /
  /____/____/    |__/|__/_/|_| /_/    |___/____(_)___/ 
                                                     
                       DD-WRT v3.0
                   http://www.dd-wrt.com
 
==========================================================


BusyBox v1.24.1 (2016-04-14 23:48:45 CEST) built-in shell (ash)

root@domosys:~# 

Great, I’m in🙂 Time to check if the interfaces were configured correctly:

root@domosys:~# ip -6 addr show
1: lo:  mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
4: vlan1@eth0:  mtu 1500 
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
5: vlan2@eth0:  mtu 1500 
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
6: eth1:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
7: eth2:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
10: br0:  mtu 1500 
    inet6 2804:14d:badb:XXXX:XXXX:ff:fe00:0/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

Everything is looking good, br0 got a valid IPv6 address, now to check the default route:

root@domosys:~# ip -6 route
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev vlan1  proto kernel  metric 256 
fe80::/64 dev eth1  proto kernel  metric 256 
fe80::/64 dev eth2  proto kernel  metric 256 
fe80::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev vlan2  proto kernel  metric 256 
default via fe80::217:10ff:fe8b:a78b dev vlan2  proto ra  metric 1024  expires 1783sec hoplimit 64
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

Default route in place, looking correct, so let’s try pinging devices over IPv6, to see if it is indeed functional:

root@domosys:~# ping6 2001:4860:4860::8844
PING 2001:4860:4860::8844 (2001:4860:4860::8844): 56 data bytes
64 bytes from 2001:4860:4860::8844: seq=0 ttl=42 time=80.255 ms
64 bytes from 2001:4860:4860::8844: seq=1 ttl=42 time=135.638 ms
64 bytes from 2001:4860:4860::8844: seq=2 ttl=42 time=93.105 ms
^C
--- 2001:4860:4860::8844 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 80.255/102.999/135.638 ms
root@domosys:~# ping6 google.com
PING google.com (2800:3f0:4004:804::200e): 56 data bytes
64 bytes from 2800:3f0:4004:804::200e: seq=0 ttl=52 time=43.273 ms
64 bytes from 2800:3f0:4004:804::200e: seq=1 ttl=52 time=52.316 ms
64 bytes from 2800:3f0:4004:804::200e: seq=2 ttl=52 time=77.638 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 43.273/57.742/77.638 ms

Awesome, it seems everything is going as planned, time to connect from my notebook and test my IPv6 connection:

rsalveti@evapro:~$ ip -6 addr show wlp3s0
2: wlp3s0:  mtu 1500 state UP qlen 1000
    inet6 2804:14d:badb:XXXX:XXXX:d259:65cf:a8f1/64 scope global temporary dynamic 
       valid_lft 43057sec preferred_lft 43057sec
    inet6 2804:14d:badb:XXXX:XXXX:2de6:6164:8e93/64 scope global mngtmpaddr noprefixroute dynamic 
       valid_lft 43057sec preferred_lft 43057sec
    inet6 fe80::a65e:60ff:fee4:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

rsalveti@evapro:~$ ping6 google.com
PING google.com(2800:3f0:4004:806::200e) 56 data bytes
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=1 ttl=52 time=30.4 ms
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=2 ttl=52 time=35.7 ms
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=3 ttl=52 time=45.8 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 30.414/37.353/45.860/6.404 ms

Looking great. Now before we celebrate, time to check the results from http://test-ipv6.com

test-ipv6

and http://ipv6-test.com

ipv6-test

Nice, easy and working as expected!

Unfortunately life it’s not always that easy and simple, something wrong was happening that made my IPv6 network to die completely after a few minutes, but it was always functional after rebooting my router. Tried the same ping6 commands from both my notebook and router, but all I got was that my IPv6 network was down. Time to get my hands dirty and debug!

root@domosys:~# ip -6 addr show br0
10: br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 
    inet6 2804:14d:badb:XXXX:XXXX:ff:fe00:0/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

root@domosys:~# ip -6 route
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev vlan1  proto kernel  metric 256 
fe80::/64 dev eth1  proto kernel  metric 256 
fe80::/64 dev eth2  proto kernel  metric 256 
fe80::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev vlan2  proto kernel  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

IPv6 address looks correct (same as before), but there is no default route for IPv6! Let’s try to manually add the route and check the network again:

root@domosys:~# ip -6 route add default via fe80::217:10ff:fe8b:a78b dev vlan2 proto ra metric 1024 hoplimit 64

root@domosys:~# ping6 google.com
PING google.com (2800:3f0:4004:804::200e): 56 data bytes
64 bytes from 2800:3f0:4004:804::200e: seq=0 ttl=52 time=35.507 ms
64 bytes from 2800:3f0:4004:804::200e: seq=1 ttl=52 time=40.768 ms
64 bytes from 2800:3f0:4004:804::200e: seq=2 ttl=52 time=30.964 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 30.964/35.746/40.768 ms

Good, that was it, now to understand why my default route disappeared.

IPv6 uses the ICMPv6 Router Solicitation (RS) and Router Advertisement (RA) messages in order to request and announce the default gateway on a network. To investigate if my ISP router is periodically sending the unsolicited RA messages (Cisco routers with ipv6 nd ra suppress?) all we need is some help from tcpdump, which is also available in DD-WRT. To make it easier, let’s check just for the RS and RA messages:

root@domosys:~# tcpdump -vvvv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes

Waited several minutes and nothing, looks like we’re heading to the right direction. As a test, let’s play with the vlan2 interface a bit:

root@domosys:~# ifconfig vlan2 down; ifconfig vlan2 up; tcpdump -vv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::e6f4:c6ff:XXXX:XXXX > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): e4:f4:c6:XX:XX:XX
            0x0000:  e4f4 c6XX XXXX
00:00:00.014184 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::217:10ff:fe8b:a78b > fe80::e6f4:c6ff:XXXX:XXXX: [icmp6 sum ok] ICMP6, router advertisement, length 96
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:17:10:8b:a7:8b
            0x0000:  0017 108b a78b
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc

So the ISP router is sending the required RA message, but only after the initial RS message that is sent by my router when establishing the connection (simulated by bringing the interface down and up). The lifetime interval also gives a hint to why my IPv6 network is only working for a few minutes after booting my router (30 minutes in this case).

Unfortunately there is really not much that can be done from the client side, as my ISP’s router is really not helping much here. One possible workaround could be forcing a default gateway after booting my router, but that is not really a good solution as the address might change after a while.

After investigating a bit more, I decided to try to periodically send RS messages by hand (even if not really recommended by the spec), so I could get the desired RA messages that would be used to update my default gateway route.

Luckily there is already a tool called rdisc6 (from http://www.remlab.net/ndisc6) that can be used to send RS messages from the command line, but unfortunately that is not available by default in the DD-WRT build I used, so time to install the tool by hand.

Once nice thing about using Kong’s DD-WRT builds is that you can use the bootstrap command to enable additional OPKG repositories, allowing the user to extend the image with custom packages. After checking Kong’s OPKG repo, I was able to confirm that the rdisc6 was already available as package, so all I needed to do was to install the extra package, which is great!

Since there is really not much disk space in flash, to install additional packages it’s recommended to first install a USB disk (I used an old 4GB pendrive I had). First clean the disk and create a single ext3 partition, then just plug it in the Netgear N7000 router and reboot. To configure the USB disk just go to the Services->USB tab:

DD-WRT USB

Make sure to mount the disk into /opt, otherwise the bootstrap script will fail to run (first enable core USB and USB storage support, reboot, add the partition UUID and reboot again).

With the disk in place and mounted at the right path, just open a telnet connection and run the bootstrap command (which is available in the DD-WRT image):

root@domosys:~# bootstrap
Bootstrap is checking prerequisites...

USB automounter is enabled.
Found a valid partition: /opt.

Proceed with download and install of opkg? (y/n) [default=n]:
y
Connecting to www.desipro.de (217.160.231.132:80)
opkg.ipk             100% |******| 60071   0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
opkg.ipk.sig         100% |******|   256   0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
functions.sh         100% |******|  7269   0:00:00 ETA
Bootstrap complete. You can now use opkg to install additional packages.

Now install the rdisc6 package by using the opkg tool:

root@domosys:~# opkg install rdisc6

And now to finally test my theory let’s open tcpdump while we manually send the RS message, to check if the default route gets updated once it gets the RA message from my ISP’s router.

Terminal 1:

root@domosys:~# tcpdump -vvvv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes

Terminal 2:

root@domosys:~# ip -6 route
...
fe80::/64 dev vlan2  proto kernel  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

root@domosys:~# /opt/usr/bin/rdisc6 -1 -q vlan2
2804:14d:badb:XXXX::/64
2804:14d:badb:XXXX::/64

root@domosys:~# ip -6 route
...
fe80::/64 dev vlan2  proto kernel  metric 256 
default via fe80::217:10ff:fe8b:a78b dev vlan2  proto ra  metric 1024  expires 1796sec hoplimit 64
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

Great, default route added again, now back to Terminal 1 to check the output from tcpdump:

00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 8) fe80::e6f4:c6ff:XXXX:XXXX > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 8
00:00:00.011702 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::217:10ff:fe8b:a78b > fe80::e6f4:c6ff:XXXX:XXXX: [icmp6 sum ok] ICMP6, router advertisement, length 96
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:17:10:8b:a7:8b
            0x0000:  0017 108b a78b
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc
...

Lovely, RS message sent with help from rdisc6, RA message replied back from the ISP router and default route updated.

Now all that remains is to add the rdisc6 command as a cron job, making sure it gets executed often (e.g. every minute). To add custom cron jobs just go to Administration->Management tab:

DD-WRT Cron Job

With that in place I can safely reboot my router and get a stable IPv6 network setup (until my ISP improves/fixes their infrastructure).

That’s it, now to start doing some real IoT IPv6-based deployments in my local network🙂