Netgear R7000, DD-WRT, IPv6 and the lack of a stable gateway

After playing with a few IoT related devices and scenarios, I wanted to finally enable IPv6 in my home network. After looking if my ISP (NET Virtua) was already supporting IPv6, I was happy to find out that they were already claiming that my city (Florianópolis) was covered, so I decided to give that a try (and post my setup/experience).

I’m using Netgear R7000 as my main router, simply because it’s known to be well supported by DD-WRT.

Installing DD-WRT on Netgear R7000 is quite an easy job, and there is also an huge list of links covering several useful tips at http://www.dd-wrt.com/phpBB2/viewtopic.php?t=264152, which is quite handy (overclocking, QoS, installing Tor, etc).

After updating DD-WRT to the latest stable build from Kong (v3.0-r29300M kongac) and enabling bridge mode on my cable modem (from Arris), it was finally time to start playing with IPv6 🙂

I first configured my IPv4 network to use DNSMasq for both DHCP and DNS. Being able to use DNSMasq is great because it is quite easy to configure, besides supporting the IPv6 Router Advertisement feature, which is quite handy for my own IPv6 network (no need for radvd).

DD-WRT IPv4 Setup

Then I just enabled IPv6 by going to the Setup->IPv6 tab. I’m using DHCPv6 with prefix delegation since that is supported by my ISP, making it even easier to configure my own network. Since I’m using DNSMasq for DHCP, I can safely disable radvd, but an additional step is required (custom DNSMasq config).

DD-WRT IPv6 Setup

To configure DNSMasq just go to the Services->Services tab, and you will find a field that can be used to store your custom config entries. For a complete overview and description of the supported options, please check http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html. In my case, all I needed was to define define the dhcp-range, dhcp-option, ra-param and enable-ra options.

DNSMasq Configs

Time to reboot and check if everything would indeed work as expected. Once booted, I decided to check the setup over telnet, since it’s easier to debug and understand what is going on 🙂

Using telnet is quite easy, and enabled by default in DD-WRT (internal network only):

rsalveti@evapro:~$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.

DD-WRT v3.0-r29300M kongac (c) 2016 NewMedia-NET GmbH
Release: 04/14/16

domosys login: root
Password: 
==========================================================
 
     ___  ___     _      _____  ______       ____  ___ 
    / _ \/ _ \___| | /| / / _ \/_  __/ _  __|_  / / _ \
   / // / // /___/ |/ |/ / , _/ / /   | |/ //_ <_/ // /
  /____/____/    |__/|__/_/|_| /_/    |___/____(_)___/ 
                                                     
                       DD-WRT v3.0
                   http://www.dd-wrt.com
 
==========================================================


BusyBox v1.24.1 (2016-04-14 23:48:45 CEST) built-in shell (ash)

root@domosys:~# 

Great, I’m in 🙂 Time to check if the interfaces were configured correctly:

root@domosys:~# ip -6 addr show
1: lo:  mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
4: vlan1@eth0:  mtu 1500 
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
5: vlan2@eth0:  mtu 1500 
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
6: eth1:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
7: eth2:  mtu 1500 qlen 1000
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
10: br0:  mtu 1500 
    inet6 2804:14d:badb:XXXX:XXXX:ff:fe00:0/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

Everything is looking good, br0 got a valid IPv6 address, now to check the default route:

root@domosys:~# ip -6 route
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev vlan1  proto kernel  metric 256 
fe80::/64 dev eth1  proto kernel  metric 256 
fe80::/64 dev eth2  proto kernel  metric 256 
fe80::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev vlan2  proto kernel  metric 256 
default via fe80::217:10ff:fe8b:a78b dev vlan2  proto ra  metric 1024  expires 1783sec hoplimit 64
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

Default route in place, looking correct, so let’s try pinging devices over IPv6, to see if it is indeed functional:

root@domosys:~# ping6 2001:4860:4860::8844
PING 2001:4860:4860::8844 (2001:4860:4860::8844): 56 data bytes
64 bytes from 2001:4860:4860::8844: seq=0 ttl=42 time=80.255 ms
64 bytes from 2001:4860:4860::8844: seq=1 ttl=42 time=135.638 ms
64 bytes from 2001:4860:4860::8844: seq=2 ttl=42 time=93.105 ms
^C
--- 2001:4860:4860::8844 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 80.255/102.999/135.638 ms
root@domosys:~# ping6 google.com
PING google.com (2800:3f0:4004:804::200e): 56 data bytes
64 bytes from 2800:3f0:4004:804::200e: seq=0 ttl=52 time=43.273 ms
64 bytes from 2800:3f0:4004:804::200e: seq=1 ttl=52 time=52.316 ms
64 bytes from 2800:3f0:4004:804::200e: seq=2 ttl=52 time=77.638 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 43.273/57.742/77.638 ms

Awesome, it seems everything is going as planned, time to connect from my notebook and test my IPv6 connection:

rsalveti@evapro:~$ ip -6 addr show wlp3s0
2: wlp3s0:  mtu 1500 state UP qlen 1000
    inet6 2804:14d:badb:XXXX:XXXX:d259:65cf:a8f1/64 scope global temporary dynamic 
       valid_lft 43057sec preferred_lft 43057sec
    inet6 2804:14d:badb:XXXX:XXXX:2de6:6164:8e93/64 scope global mngtmpaddr noprefixroute dynamic 
       valid_lft 43057sec preferred_lft 43057sec
    inet6 fe80::a65e:60ff:fee4:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

rsalveti@evapro:~$ ping6 google.com
PING google.com(2800:3f0:4004:806::200e) 56 data bytes
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=1 ttl=52 time=30.4 ms
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=2 ttl=52 time=35.7 ms
64 bytes from 2800:3f0:4004:806::200e: icmp_seq=3 ttl=52 time=45.8 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 30.414/37.353/45.860/6.404 ms

Looking great. Now before we celebrate, time to check the results from http://test-ipv6.com

test-ipv6

and http://ipv6-test.com

ipv6-test

Nice, easy and working as expected!

Unfortunately life it’s not always that easy and simple, something wrong was happening that made my IPv6 network to die completely after a few minutes, but it was always functional after rebooting my router. Tried the same ping6 commands from both my notebook and router, but all I got was that my IPv6 network was down. Time to get my hands dirty and debug!

root@domosys:~# ip -6 addr show br0
10: br0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 
    inet6 2804:14d:badb:XXXX:XXXX:ff:fe00:0/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::e6f4:c6ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

root@domosys:~# ip -6 route
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev vlan2  proto kernel  metric 256 
2804:14d:badb:XXXX::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev vlan1  proto kernel  metric 256 
fe80::/64 dev eth1  proto kernel  metric 256 
fe80::/64 dev eth2  proto kernel  metric 256 
fe80::/64 dev br0  proto kernel  metric 256 
fe80::/64 dev vlan2  proto kernel  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

IPv6 address looks correct (same as before), but there is no default route for IPv6! Let’s try to manually add the route and check the network again:

root@domosys:~# ip -6 route add default via fe80::217:10ff:fe8b:a78b dev vlan2 proto ra metric 1024 hoplimit 64

root@domosys:~# ping6 google.com
PING google.com (2800:3f0:4004:804::200e): 56 data bytes
64 bytes from 2800:3f0:4004:804::200e: seq=0 ttl=52 time=35.507 ms
64 bytes from 2800:3f0:4004:804::200e: seq=1 ttl=52 time=40.768 ms
64 bytes from 2800:3f0:4004:804::200e: seq=2 ttl=52 time=30.964 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 30.964/35.746/40.768 ms

Good, that was it, now to understand why my default route disappeared.

IPv6 uses the ICMPv6 Router Solicitation (RS) and Router Advertisement (RA) messages in order to request and announce the default gateway on a network. To investigate if my ISP router is periodically sending the unsolicited RA messages (Cisco routers with ipv6 nd ra suppress?) all we need is some help from tcpdump, which is also available in DD-WRT. To make it easier, let’s check just for the RS and RA messages:

root@domosys:~# tcpdump -vvvv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes

Waited several minutes and nothing, looks like we’re heading to the right direction. As a test, let’s play with the vlan2 interface a bit:

root@domosys:~# ifconfig vlan2 down; ifconfig vlan2 up; tcpdump -vv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::e6f4:c6ff:XXXX:XXXX > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): e4:f4:c6:XX:XX:XX
            0x0000:  e4f4 c6XX XXXX
00:00:00.014184 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::217:10ff:fe8b:a78b > fe80::e6f4:c6ff:XXXX:XXXX: [icmp6 sum ok] ICMP6, router advertisement, length 96
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:17:10:8b:a7:8b
            0x0000:  0017 108b a78b
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc

So the ISP router is sending the required RA message, but only after the initial RS message that is sent by my router when establishing the connection (simulated by bringing the interface down and up). The lifetime interval also gives a hint to why my IPv6 network is only working for a few minutes after booting my router (30 minutes in this case).

Unfortunately there is really not much that can be done from the client side, as my ISP’s router is really not helping much here. One possible workaround could be forcing a default gateway after booting my router, but that is not really a good solution as the address might change after a while.

After investigating a bit more, I decided to try to periodically send RS messages by hand (even if not really recommended by the spec), so I could get the desired RA messages that would be used to update my default gateway route.

Luckily there is already a tool called rdisc6 (from http://www.remlab.net/ndisc6) that can be used to send RS messages from the command line, but unfortunately that is not available by default in the DD-WRT build I used, so time to install the tool by hand.

Once nice thing about using Kong’s DD-WRT builds is that you can use the bootstrap command to enable additional OPKG repositories, allowing the user to extend the image with custom packages. After checking Kong’s OPKG repo, I was able to confirm that the rdisc6 was already available as package, so all I needed to do was to install the extra package, which is great!

Since there is really not much disk space in flash, to install additional packages it’s recommended to first install a USB disk (I used an old 4GB pendrive I had). First clean the disk and create a single ext3 partition, then just plug it in the Netgear N7000 router and reboot. To configure the USB disk just go to the Services->USB tab:

DD-WRT USB

Make sure to mount the disk into /opt, otherwise the bootstrap script will fail to run (first enable core USB and USB storage support, reboot, add the partition UUID and reboot again).

With the disk in place and mounted at the right path, just open a telnet connection and run the bootstrap command (which is available in the DD-WRT image):

root@domosys:~# bootstrap
Bootstrap is checking prerequisites...

USB automounter is enabled.
Found a valid partition: /opt.

Proceed with download and install of opkg? (y/n) [default=n]:
y
Connecting to www.desipro.de (217.160.231.132:80)
opkg.ipk             100% |******| 60071   0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
opkg.ipk.sig         100% |******|   256   0:00:00 ETA
Connecting to www.desipro.de (217.160.231.132:80)
functions.sh         100% |******|  7269   0:00:00 ETA
Bootstrap complete. You can now use opkg to install additional packages.

Now install the rdisc6 package by using the opkg tool:

root@domosys:~# opkg install rdisc6

And now to finally test my theory let’s open tcpdump while we manually send the RS message, to check if the default route gets updated once it gets the RA message from my ISP’s router.

Terminal 1:

root@domosys:~# tcpdump -vvvv -ttt -i vlan2 icmp6 and 'ip6[40] >= 133 && ip6[40] <= 134'
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 65535 bytes

Terminal 2:

root@domosys:~# ip -6 route
...
fe80::/64 dev vlan2  proto kernel  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

root@domosys:~# /opt/usr/bin/rdisc6 -1 -q vlan2
2804:14d:badb:XXXX::/64
2804:14d:badb:XXXX::/64

root@domosys:~# ip -6 route
...
fe80::/64 dev vlan2  proto kernel  metric 256 
default via fe80::217:10ff:fe8b:a78b dev vlan2  proto ra  metric 1024  expires 1796sec hoplimit 64
unreachable default dev lo  proto kernel  metric -1  error -101
ff00::/8 dev eth0  metric 256 
ff00::/8 dev vlan1  metric 256 
ff00::/8 dev eth1  metric 256 
ff00::/8 dev eth2  metric 256 
ff00::/8 dev br0  metric 256 
ff00::/8 dev vlan2  metric 256 
unreachable default dev lo  proto kernel  metric -1  error -101

Great, default route added again, now back to Terminal 1 to check the output from tcpdump:

00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 8) fe80::e6f4:c6ff:XXXX:XXXX > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 8
00:00:00.011702 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::217:10ff:fe8b:a78b > fe80::e6f4:c6ff:XXXX:XXXX: [icmp6 sum ok] ICMP6, router advertisement, length 96
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:17:10:8b:a7:8b
            0x0000:  0017 108b a78b
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc
...

Lovely, RS message sent with help from rdisc6, RA message replied back from the ISP router and default route updated.

Now all that remains is to add the rdisc6 command as a cron job, making sure it gets executed often (e.g. every minute). To add custom cron jobs just go to Administration->Management tab:

DD-WRT Cron Job

With that in place I can safely reboot my router and get a stable IPv6 network setup (until my ISP improves/fixes their infrastructure).

That’s it, now to start doing some real IoT IPv6-based deployments in my local network 🙂

Pre-built images for XBMC (Ubuntu 12.04 based) with hw acceleration finally available at Linaro

Back to the Linaro 12.01 release, we were able to publish a XBMC based image with proper hardware acceleration support for Pandaboard. That release was based on the Ubuntu Oneiric (11.10) release, with a 3.1 based kernel, still using the old multimedia architecture provided by TI (e.g old SGX DDK, Syslink, etc).

Took a while (6 months) for TI to be finally able to rebase the hardware decode/rendering solution to the new architecture, providing quite a few improvements and making the architecture a lot simpler as well. A new SGX DDK was deployed with the functional xf86-video-omap open source X11 driver, kernel and userspace with support for RPMsg, dri2video and a few other fancy technologies that now makes the life of developers a lot easier. We’re still stuck with a few proprietary parts, like the SGX driver itself, but luckly that will also improve over the time, as we’re starting to see more and more reverse-engineer based solutions around (even Rob Clark started one called freedreno).

After TI making all the needed components available for Ubuntu Precise (12.04) during the beginning of this month, I was finally able to get them all integrated and working with the Linaro components and LEB. If you grab the latest LEB provided for Pandaboard, you’ll already be able to enjoy most of the updates available, with a better SGX driver, and a working hardware decode with gstreamer without much effort.

Unfortunatelly our TI Landing Team kernel is still missing a few important multimedia-related bits, so until they are properly integrated from TI’s own kernel tree, you’d need to make sure you’re using the kernel available directly from TI (called linux-image-ti-omap4, also available at our Overlay PPA).

To be able to have a fully functional XBMC as well, we also integrated all the new changes to support a native Gstreamer player (improved a lot by Rob Clark as well, adding EGLImage support), which solves most of the a/v sync issues we had with the Oneiric based release. There are probably a few things missing as well, but at least the patch is not that intrusive anymore, and could probably be accepted by upstream later on.

Want to see how things are looking by yourself? Please check at http://snapshots.linaro.org/precise/pre-built/ti-panda-x11-base/ and download the latest pre-built image available. At the time I’m writing this post I used the one available at http://snapshots.linaro.org/precise/pre-built/ti-panda-x11-base/5/ti-panda-x11-base-precise_linarotv-xbmc_20120729-5.html, and all I needed to do was to extract the img.gz file, and copy it with dd (sudo dd bs=4M if=ti-panda-x11-base-precise_linarotv-xbmc_20120729-5.img of=/dev/sdX).

Boot it up, and you should go directly to XBMC. Give it a try, check how well it’s playing the wide range of different video formats. Bugs are also expected and welcome, so please also make sure to open any bug you might find at https://bugs.launchpad.net/linaro-ubuntu and https://bugs.launchpad.net/ubuntu-omap4-extras-multimedia. We’ll keep improving the support over the next cycles, so also make sure to post which image you’re using when doing your tests.

Enjoy!

Great start at Ubuntu Developer Summit (Q or 12.10) for ARM

This week I’m proudly participating at the Ubuntu Developer Summit to help planning and defining what will the Quantal Quetzal (12.10) release be in the next following months.

As usual I’m wearing not only the Linaro hat, but also my Ubuntu and Canonical ones, interested and participating actively at most topics that are related with ARM in general.

And what can I say after the first 3 days at UDS-Q? Well, busy as never before and with great opportunities to help getting Ubuntu to rock even more at ARM, with current devices/platforms and with the exciting new ones that will be coming in the next few months.

Here are a few highlights from the first days:

Monday – May 7th

  • Introduction and Keynote
    • Great start as usual by Mark, showing the great opportunities for both Canonical and Ubuntu, describing the new target and use cases, and also showing how important Cloud is now for Ubuntu. After that we had, finally, the announcement of a real hardware availability from Calxeda, proving that ARM server are indeed real! (which is a quite important accomplishment)
  • Schedule displays all working with our member’s boards
    • This was the first time that all the schedule displays available at UDS were all covered by the ARM boards provided by Linaro. This time we got Pandaboard, Origen and also Snowball constantly showing the schedule through all the day. Low power and powerful devices all around 🙂
  • Plans for a minimum filesystem for embedded devices
    • Discussion to cover all the possible embedded related use cases for Ubuntu, and trying to understand the real requirements for a minimum filesystem (rootfs) for those devices. While we didn’t decide to generate the smallest-still-apt/dpkg-compatible rootfs for our users (as ubuntu-core is already covering most of the cases), we’ll provide enough tools and documentation on how to easily generate them. At Linaro side the Ubuntu Nano image should probably reflect such suggestions.
  • Identify impact of the switch to pure live images for ARM platforms
    • Here the focus was basically to review and understand if we would really continue providing pre-installed based images instead of just supporting live based ones. Having the images provided only at the SD cards are very useful to make the bootstrap and install quite easy, but it hurts badly the performance. As we’re now getting ARM boards that are very powerful in many ways, the I/O bound shouldn’t limit what the users would be able to get from them. The decision for Quantal is to drop support for the pre-installed images, and provide live based ones at the SD cards (think like the live-sd image as we have with CD on other archs), where the user would install Ubuntu the same way as done with x86, and using USB/Sata based devices as rootfs by default.
  • OpenStack Deployment on ARM Server
    • The focus of this session was basically to better understand what might be the missing pieces for a proper OpenStack support at ARM. Quite a few open questions still, but the missing pkgs enablement, LXC testing and support and KVM for a few platforms will help making sure the support is at least correctly in place. After initial support, continuous test and validation should happen to make sure the ARM platforms keeps well supported over the time (which will be better stressed and tested once MAAS/Juju is also supported properly at ARM).

Tuesday – May 8th

  • Detail and begin the arm64/aarch64 port in Ubuntu
    • Clearly the most important session of the day for ARM. Great discussion on how to prepare and start the ARMv8 port at Ubuntu and Debian, by starting with cross-build support with multiarch and later support with Fast Models and Qemu. A lot is still to be covered once ARM is able to publish the ARMv8 support for Toolchain and Kernel, and session will be reviewed again at Linaro Connect at the end of this month.
  • Ubuntu Kernel Delta Review
    • Usual review of the patches the Ubuntu Kernel team is maintaining at the Ubuntu Kernel tree. At Linaro this is important as we also enable the Ubuntu specific patch-set at the packages provided by the LEB, for proper kernel and user-space support. Luckily this time it seems the delta is really minimum, which should probably also start to be part of Linux Linaro in the following month.
  • Integrate Linaro hwpacks for ARM with the Ubuntu image build infrastructure
    • Usual discussion about trying to avoid replicated work that is strictly related with each ARM board we support at both Ubuntu and Linaro. Decision is to finally sync with the latest flash-kernel available at Debian and try to get the common project/package with the hardware specific bits in place, so it can be used by linaro-image-tools, flash-kernel and debian-cd.

Wednesday – May 9th

  • MAAS Next Steps
    • Session to review and plan what are the next steps for the MAAS project, which is also missing proper ARM support for now. Great discussions on understanding all the requirements, as they will not necessarily match entirely with the usual ARM devices we have at the moment. Here the goal for ARM is to continue improving the PXE support at U-Boot (even with UEFI chainload later), and understanding what might be missing to also have IPMI support (even if not entirely provided by the hardware).
  • System Compositor
    • Great session covering what might be the improvements and development on the graphics side for next release. Goal is to use a system compositor that would be started right at the beginning at the boot, which will then be controlled and used properly once lightdm is up (with X11). This will improve a lot the user experience on normal x86 based desktops, and luckily on ARM we’re also in a quite nice situation with the work done by Linaro helping getting the proper DRM/KMS support for the boards we support, so I hope ARM will be in a great shape here 🙂
  • ARM Server general enhancements (for ARMv7 and perhaps v8)
    • At this session we could cover what seems to be the most recurrent and problematically thing at supporting ARM servers, which is the lack of a single and supported boot method and boot loader. UEFI should be able to help on this front soon, but until then the focus will be to keep checking and making sure the current PXE implementation at u-boot works as expected (chainloading UEFI on u-boot is also another possibility Linaro is investigating). There is also the request for IPMI support, which is still unclear in general how it’ll be done generically speaking.
  • Integration testing for the bootloader
    • As Ubuntu is also moving to the direction of continuous validating and testing all important components available, there’s the need for a proper validation of the bootloader, and the effect at the user experience while booting the system. For ARM it’s also a special case, as U-Boot is still the main bootloader used across the boards. Test case descriptions in place, and discussion will probably continue at Linaro Connect as this is also an area where we also want to help validating/testing.
  • ARM Server Benchmarking and Performance
    • Here the Ubuntu Server Team presented how they are benchmarking and checking performance at the server level at x86, and covering what might still be needed to run and validate the ARM boards the same way. For ARM the plan is to run the same test cases on the available scenarios, and also try to get Linaro involved by making sure this is also part of the continuous validation and testing done with LAVA. Another important topic that will probably be extended at Linaro Connect is finding a way to get the power consumption data when running the test cases/benchmarks, so it can be further optimised later on.
  • Compiz GLES2 Handover
    • Last session of the day, trying to find the missing gaps to finally get the OpenGL ES2.0 support merged at the Compiz and Unity upstream branches used by the entire Ubuntu desktop (across all archs). Following work and actions will basically be to fix the remaining and important plugins after merging the changes, and also getting a few test cases to properly validate the support at Ubuntu. Once all done, it should be merged ASAP.

These are just a few topics which I was able to participate. There are a lot of more exciting work coming on, which can all be found at http://summit.ubuntu.com/uds-q/. Remember that you’re still able to participate in a few of them tomorrow and friday, as remote access is provided for all the sessions we have.

I’m sure a lot of more exciting stuff will be discussed for ARM support until the end of this week, and at Linaro Connect, at the end of the month, we’ll be able to review and get our hands dirty as well 🙂

Exciting times for ARM!

ARM Porting Jam this Friday!

For those following the development of the next Ubuntu release (12.04 – Precise Pangolin), you all know that we’re quite close to the release date already, and to make sure Precise rocks since day 0, we all need to work hard to get most of the bugs sorted out during the next few weeks.

At Linaro, the Linaro Developer Platform team will be organizing an ARM porting Jam this Friday, with the goal of getting all developers interested in fixing and working on bugs and portability issues related with the Ubuntu ARM port (mostly issues with ARMHF at the moment).

The idea of having the Porting Jam at Friday is to have it as a joint effort with Ubuntu’s Fix Friday and Ubuntu Global Jam, so expect quite a few other developers helping improving Ubuntu as well!

It’s quite easy to participate:

Remember that for ARM this release will be a quite huge milestone, as it’ll be the first LTS release supporting ARM, besides delivering support for ARM servers and ARMHF as default, so let’s make sure it rocks!

Looking forward for a great porting Jam!

Happy bug fixing!

Ubuntu TV fully accelerated on a Pandaboard with Ubuntu LEB

As described on my previous post about Ubuntu TV support on a Pandaboard, we were still missing proper support for texture streaming on a Pandaboard, to have the video playback also working and fully accelerated.

This weekend Rob Clark managed to create the first version of the TI’s specific eglImage support at Qtmobility, posting the code at his gitorious account, and for the first time we’re fully able to use Ubuntu TV on a ARM device, using a Pandaboard.

Demo video with the Ubuntu TV UI (accelerated with Qt and OpenGL ES 2.0) and with video decode support of 720p and 1080p:

The code support for TI’s eglImage still needs a few clean-ups, but we hope to be able to push the support at Ubuntu in the following weeks (make it good enough to try at least a package patch).

For people wanting to try it out, a few packages are already available at Linaro’s Overlay PPA, and the remaining ones should be available later today (Qt and Qtmobility), so people can easily run it with our images.

Hope you enjoy, and we’ll make sure we’re always working on keeping and improving the current support, so Ubuntu TV also rocks with ARM 🙂

Cheers!

Ubuntu TV UI at Pandaboard, and next steps

Yesterday Canonical announced the first UI concept for the Ubuntu TV. Together with the announcement, the first code drop was released, so we could read and understand better the technologies used, and how this will behave on an ARM environment, mostly at a Pandaboard (that we already have OpenGL ES 2 and video decode working).

Getting Ubuntu TV to work

If are still using Oneiric, you can just follow the guide presented at https://wiki.ubuntu.com/UbuntuTV/Contributing, where you’ll find all needed steps to try Ubuntu TV at your machine.

As it’s quite close with Unity 2D (similar code base), and also based on Qt, I decided to follow the steps described at wiki page and see if it should work correctly.

First issue we found with Qt, was that it wasn’t rendering at full screen when using with latest PowerVR SGX drivers, so any application you wanted to use with Qt Opengl would just show itself on a small part of the screen. Luckily TI (Nicolas Dechesne and Xavier Boudet) quickly provided me a new release of the driver, fixing this issue (version that should be around later today at the Linaro Overlay), so I could continue my journey 🙂

Next problem was that Qt was enabling brokenTexSubImage and brokenFBOReadBack for the SGX drivers based on the old versions available for Beagle, and seems this is not needed anymore with the current version available at Pandaboard (still to be reviewed with TI, so a proper solution can be forwarded to Qt).

Code removed, patch applied and package built (after many hours), and I was finally able to successfully open the Ubuntu TV interface at my Panda 🙂

UI Navigation on a Pandaboard, with Qt and OpenGL ES2.0

Running Ubuntu TV is quite simple if you’re already running the Unity 2D interface. All you need to do is to make sure you kill all unity-2d components and that you’re running metacity without composite enabled. Other than that you just run ”unity-2d-shell -opengl” and voilà 😉

Here’s a video of the current interface running on my Panda:

As you can see from the video, I didn’t actually play any video, and that’s because currently we’re lacking a generic texture handler for OpenGL ES with Gstreamer at Qtmobility (there’s only one available, but specifically for Meego). Once that’s fixed, the video playback should behave similarly as with XBMC (but with less hacks, as it’s a native GST backend).

Next steps, enabling proper video decode

Looking at what would be needed to finally be able to play the videos, and to make it something useful at your Pandaboard, the first thing is that we need to improve Qtmobility to have a more generic (but unfortunately still specific to Omap) way handle texture streaming with Gstreamer and OpenGL ES. Rob Clark added a similar functionality at XBMC, creating support for ”eglImage”, so we just need to port the work and make sure it works properly with Qtmobility.

Once that’s ported, the video should be streamed as a texture at the video surface, making it also work transparently with QML (the way it’s done with Ubuntu TV).

If you know Qt and Gstreamer, and also want to help getting it to work properly on your panda, here follows a few resources:

As soon video decoding is working properly, a new blog post should be around explaining the details and how to reproduce it at your own Panda with Ubuntu LEB 🙂

Cheers!

HW video decode and XBMC support on a Pandaboard with Ubuntu LEB

Part of the effort we spent during the Linaro 11.12 cycle was to try to enable at Pandaboard not only hardware graphics support (GLES with PVR SGX), but also hardware accelerated video decode, as TI had released all needed userspace to be used at Ubuntu Oneiric (11.10) release.

Unfortunately it didn’t just work with our images because at that time we were using a newer kernel already, based on the 3.1 series that is maintained by the Linaro TI Landing Team. Bug 880840 has all the details.

Luckily Sebastien Jan (from TI) was able to find the root cause of the problem, that was causing so much frame drops that was making the video playback basically unusable. The problem was related with PM support at omap’s hwspinlock implementation, as you can check at this link.

Kernel fix properly integrated and available at the Overlay PPA used by our Linaro Ubuntu Evaluation Build images, and finally able to have a similar user experience as was expected when TI delivered the user space components at their own PPA.

If you want to try it by yourself, just be sure you’re using at least linux-image-3.1.1-6-linaro-lt-omap at your board (all hwpacks >= 20110105 should have it included by default).

Playing videos with HW decode acceleration

Since today you’ll also easily find all the needed packages to enable HW video decode acceleration at our images (Pandaboard only at the moment, more boards coming soon). We just included and copied all needed packages from the TI PPA, so you don’t even need to enable it when installing the additional packages.

Installing the extra packages for video decode at your Pandaboard:

  • Grab the latest Pandaboard hwpack (lt-panda-x11-base-oneiric) and Ubuntu Desktop image from http://snapshots.linaro.org/oneiric (as example I used hwpack_linaro-lt-panda-x11-base_20120106-0_armel_supported.tar.gz and linaro-o-ubuntu-desktop-tar-20120105-0.tar.gz)
  • Create a Ubuntu LEB pandaboard image on a SD card, following the instructions described at https://wiki.linaro.org/Platform/DevPlatform/Ubuntu/ImageInstallation
  • Boot the card and install the ubuntu-omap4-extras-multimedia package: $ sudo apt-get install ubuntu-omap4-extras-multimedia
  • Reboot your pandaboard
  • Play a video with any video player that’s compatible with Gstreamer (e.g. Totem)

In the future we should also have this completely integrated at the hwpack itself, but unfortunately this is not possible at the moment without increasing the image size too much.

XBMC support

Another awesome thing we worked during previous cycle (11.12) was to make an XBMC version available that would use both GLES and Gstreamer, so it could also be used with a Pandaboard. Avik Sil did a great work making it all work with our images, and we were finally able to have XBMC 11 Beta (Eden) available at our Overlay PPA.

For proper support for Gstreamer Rob Clark did an awesome work improving the current patches, and also improving the support quite a bit. At our package you’ll find all latest patches available from Rob, from his current development tree.

To start using XBMC with the Ubuntu LEB image at your Pandaboard, you just need to install the xbmc package, with $ sudo apt-get install xbmc. For best user experience, please use the XBMC session available at LightDM (just log-out the default session and select XBMC instead). This will work a lot better because then there will be no other window manager or compositor taking extra resources from your board.

We also hope to deliver a set-top box image by the end of the current cycle (12.01), that will have XBMC installed by default. Please check the blueprint https://blueprints.launchpad.net/linaro-ubuntu/+spec/create-a-set-top-box-leb-image if you want to follow the progress of it.

Bugs and Issues

Unfortunately not everything is working perfectly at the moment, and issues with the Gstreamer and hw video decode support on Pandaboard are expected. The most annoying one that’s currently affecting XBMC is the issues with seek, as sometimes the video goes faster than the audio, and then it stops for a while until it’s in sync again. We hope to get this fixed soon, but that depends a bit of how much time Rob can spend on it.

In case of any other bug while trying to get video decode to work on your Pandaboard, don’t hesitate to open a bug at https://bugs.launchpad.net/linaro-ubuntu/+filebug or ping aviksil, robclark or rsalveti at #linaro on freenode.

Cheers!

Update: Check bug https://bugs.launchpad.net/linaro-ubuntu/+bug/915456 for the video hanging issue. Without polling XBMC should now play most videos just fine.

Update 2: XBMC-ready image already available at http://snapshots.linaro.org/oneiric/linaro-o-linarotv-xbmc/, just be sure to flash with http://snapshots.linaro.org/oneiric/lt-panda-x11-base-oneiric/.

Update 3: There’s a mem leak at the gst decode codec, check bug https://bugs.launchpad.net/ubuntu-omap4-extras-multimedia/+bug/915768 for progress on that.

UDS-P/Linaro Connect Q4.11 and 11.11 cycle

During the end of October and beginning of November we had the last Linaro Connect for the year. This time we also had it together with the Ubuntu Developer Summit, giving us the opportunity to better discuss the roadmap with both Linaro and the Ubuntu team.

From the Developer Platform team perspective, we had a quite nice week, with demos happening at Monday and Friday (showing people what we’ve been working on), and also sharing some great news with the Ubuntu team, now that Mark Shuttleworth announced that Ubuntu will go to Tablets, TVs and Phones (and ARM for sure will be a huge part of that).

Some nice links and videos of what happened during that week (related with our team):
* Sessions related with the Developer Platform Team (Ubuntu)
* Linaro Demo: Ubuntu Unity with OpenGL ES on Pandaboard
* Linaro Developer Platform Tech Lead Ricardo Salveti Interview at Linaro Connect
* Linaro Connect Q4.11 – Ubuntu LEB tutorial
* Linaro Connect Q4.11 – Interview with Marcin Juszkiewicz

Linaro 11.11 Release

Another quite good achievement for us during November was the 11.11 release.

During this release we had a quite a few great highlights, including some that we were planning for quite a while already:
* Ability to cross build Firefox using Multiarch
* OMAP4 SPL USB Booting, enabling USB boot at Pandaboard
* ARM DS-5 support for the 5.8 release
* CI Builds for Linaro GCC both for cross and native
* And a lot of bug fixes

Now it’s time to get ready to develop the blueprints we’re planning for 11.12, to also make December another great and solid month 🙂 (will do another post about the 11.12 planning later this one).

launchpadlib: creating a changelog for a PPA

As at Linaro we usually work with many PPAs over the releases, there was a need to generate a proper changelog for a PPA, in a way we could know what packages got changed before doing the release.

At first I thought I could just parse the repository metadata (as a PPA is nothing more than a debian repository), but then I realized I could just use the awesome (yes, *awesome*) launchpadlib, if it had a way to get the data I needed.

So I called the launchpadlib master I know (Ursinha), and in 15 minutes we saw that we could use it to parse the “.changes” file, and from there get the data I needed. As Launchpad stores the PPA packages publishing history, it’s quite easy to get all the changes over period of time.

A few minutes later (after also noticing that there’s a python-debian module to parse the changes file), I created the first version of the generate-ppa-changelog.py script, that does exactly what I needed, and with just a few python lines 🙂

Here’s the link: http://bazaar.launchpad.net/~rsalveti/+junk/launchpad/view/head:/generate-ppa-changelog.py.

Usage:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -h
usage: generate-ppa-changelog.py [-h] [-d YYYYMMDD] [-s SERIES] -t TEAM
[-p PPA] [–version]

Print Changelog for a PPA

optional arguments:
-h, –help show this help message and exit
-d YYYYMMDD, –date YYYYMMDD
start date to probe for changes
-s SERIES, –series SERIES
ubuntu series to look for changes (default: natty)
-t TEAM, –team TEAM launchpad team that owns the PPA
-p PPA, –ppa PPA ppa name to probe the changelog (default: first PPA)
–version show program’s version number and exit

If no argument is given, it will probe all the changes for the default series.

Output Example:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -t linaro-maintainers -p overlay -s natty -d 20110701
Changelog for linaro-maintainers’s overlay PPA (series natty) since 2011-07-01 00:00:00

base-files (5.0.0ubuntu28linaro3) natty; urgency=low

* Updating Linaro LEB version to 11.07 (development branch)

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 04:19:40 -0300

libjpeg-turbo (1.1.1-1inaro2) natty; urgency=low

* release
* add timestamp code to cjpeg.c
* default cjpeg.c and djpeg.c timestamp code to off

— Tom Gall Mon, 11 Jul 2011 20:32:23 +0000

linaro-meta (017) natty; urgency=low

* Refreshed dependencies
* Added libjpeg-turbo62 to linaro-alip, linaro-graphical-engineering,
linaro-multimedia-engineering, linaro-ubuntu-desktop

— Tom Gall Wed, 13 Jul 2011 15:41:29 +0000

u-boot-linaro (2011.07.1-0ubuntu1~natty1) natty; urgency=low

* New upstream 2011.07.1 which includes
– PXE FDT fix from previous ubuntu release so patch has been removed
– Added missing PXE env vars (LP: #808815)
– Generated unique usbethaddr (LP: #809015)
– Modify pxe command to look for usbethaddr if ethaddr is NULL

— John Rigby Wed, 13 Jul 2011 23:31:39 -0600

x-loader (1.5.1+git20110715+fca7cd2-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– several fixes for panda and igep
– OMAP3 code cleanup
– Beagle Rev C5 support
– Support for IGEPv3 board

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 05:55:09 -0300

x-loader (1.5.0+git20110714+cdc887b-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– Adding support for IGEPv3 board
* debian/patches/01-Beagle-Rev-C5-support.patch:
– Adding support for the new Beagle C5

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 02:36:58 -0300

And now we can just generate the proper changelog anytime we want to do a call for testing or a Linaro Ubuntu release 🙂

Net booting with TFTP and PXE with Pandaboard

Over the past month I’ve being working with John Rigby to integrate the SMSC95XX and OMAP4 EHCI patches into Linaro U-Boot, so we could deliver the network booting feature for people using Pandaboards.

Those patches are published at the U-Boot mailing list, but still as a working in progress. While we work helping the original developers to get the patches accepted upstream, we also want to deliver the functionality for our users, so all those patches are now integrated at the Linaro U-Boot tree.

You can check the patches by going at http://git.linaro.org/gitweb?p=boot/u-boot-linaro-stable.git;a=shortlog.

Testing with Pandaboard

To make it work properly, besides using Linaro U-Boot you’ll also need to use the upstream X-Loader tree, with one additional patch that’s not yet merged. You can clone the upstream tree from http://gitorious.org/x-loader/x-loader, then just apply the patch http://people.canonical.com/~rsalveti/pxe/0001-omap4-pandaboard-ehci-fref_clkout-per-board-revision.patch and build for the Pandaboard target.

If you just want to test without building your own X-Loader and U-Boot, you can just grab both files from  http://people.canonical.com/~rsalveti:

Building your TFTP + DHCP server for PXE

To build your TFTP + DCHP server just follow the instructions described at https://help.ubuntu.com/community/Desktop/PXE. Don’t worry about the ‘filename “pxelinux.0”;’ line at the dhcpd.conf file, you can remove it.

Then just create your PXE config file at the right place:

$ cat /tftpboot/pxelinux.cfg/0A2A2B0A
default panda-natty
prompt 0
timeout 3

label panda-natty
kernel panda/uImage
append console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd panda/uInitrd

PXE Booting

With the proper X-Loader and U-Boot files in place (at your first SD card partition), and with the TFTP + DHCP server also properly installed, you can just jump and try TFTP/PXE boot.

Stop the U-Boot autoload and call the following commands:

  • setenv pxecfg_ram 0x88000000: location in RAM to load the pxecfg file
  • setenv kernel_ram 0x80000000: location in RAM to load the kernel
  • setenv initrd_ram 0x81600000: location in RAM to load the initrd
  • setenv autoload no: disable autoload while calling bootp (so you can just set up your network without autoboot)
  • usb start: start USB and enables the SMSC95xx ethernet interface
  • bootp: initialize the network, probing the ip address settings from your DHCP server
  • pxecfg get: probe the pxecfg config file
  • pxecfg boot: boot 🙂

You should get a similar output as:

Texas Instruments X-Loader 1.5.0 (Jul 11 2011 – 07:52:49)
Reading boot sector
Loading u-boot.bin from mmc

U-Boot 2011.06 (Jul 11 2011 – 02:49:51)

CPU : OMAP4430
Board: OMAP4 Panda
I2C: ready
DRAM: 1 GiB
MMC: OMAP SD/MMC: 0
Using default environment

In: serial
Out: serial
Err: serial
Net: No ethernet found.
Hit any key to stop autoboot: 0
Panda # setenv pxecfg_ram 0x88000000
Panda # setenv kernel_ram 0x80000000
Panda # setenv initrd_ram 0x81600000
Panda # setenv autoload no
Panda # usb start
(Re)start USB…
USB: Register 1313 NbrPorts 3
USB EHCI 1.00
scanning bus for devices… The request port(2) is not configured
EHCI timed out on TD – token=0x80008c80
The request port(2) is not configured
4 USB Device(s) found
scanning bus for storage devices… 0 Storage Device(s) found
scanning bus for ethernet devices… 1 Ethernet Device(s) found
Panda # bootp
Waiting for Ethernet connection… done.
BOOTP broadcast 1
DHCP client bound to address 10.42.43.10
Panda # pxecfg get
missing environment variable: pxeuuid
missing environment variable: ethaddr
Retreiving file: pxelinux.cfg/0A2A2B0A
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘pxelinux.cfg/0A2A2B0A’.
Load address: 0x88000000
Loading: #
done
Bytes transferred = 239 (ef hex)
Config file found
Panda # pxecfg boot
Hit any key to stop autoboot: 0
Label: panda-natty
kernel: panda/uImage
append: console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd: panda/uInitrd
Retreiving file: panda/uInitrd
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uInitrd’.
Load address: 0x81600000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
############
done
Bytes transferred = 3982715 (3cc57b hex)
Retreiving file: panda/uImage
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uImage’.
Load address: 0x80000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#########################
done
Bytes transferred = 4174480 (3fb290 hex)
## Booting kernel from Legacy Image at 80000000 …
Image Name: Ubuntu Kernel
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4174416 Bytes = 4 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum … OK
## Loading init Ramdisk from Legacy Image at 81600000 …
Image Name: Ubuntu Initrd
Image Type: ARM Linux RAMDisk Image (uncompressed)
Data Size: 3982651 Bytes = 3.8 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum … OK
Loading Kernel Image … OK
OK

Starting kernel …

Uncompressing Linux… done, booting the kernel.
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu

This should be enough for you to get your Pandaboard booting with PXE. You can also script these commands at your boot.scr file that U-Boot loads automatically from your SD card, so you don’t have to call them by hand every time you reboot your board.

In case it doesn’t work for you, just ping me (rsalveti) at #linaro on freenode 🙂