Way of the exploding head

I was at Fosdem over the weekend, there were several interesting talks I attended but the most interesting one for me was a roundtable about the future of Cross distributions.  I was invited to give a 5 minute talk there which I gave, but unfortunately it was right at the end and the people before had overrun, so there was no time to make much of a coherent case.   So I am going to write some articles covering the issues involved here.

Cross as a niche

Cross itself remains absolutely necessary for systems below a certain level of horsepower.  For example, 8051, ARM7, cortex M3 are not really capable to consider native build.  But processors get faster each year, a lot of things we would have used an 8051 on use an ARM7 or cortex M3 now, in a few years it is likely that baseline has moved further up and it’s an ARM9 equivalent.  What I am suggesting then is that over time, the niche where you need cross is shrinking.

All four of the cross distros at FOSDEM target a CPU that’s powerful enough to run Linux, but not powerful enough to build its own binaries.  That is the niche that I believe will shrink to the point that it won’t support all these cross Linux distro projects, possibly none of them in the end.

My background with cross Linux

A few years ago I created an RPM-based cross distro singlehanded, and used it on a product for a customer  This was AT91RM9200-based, a 200MHz ARM9 with 32MBytes of SDRAM.  The amount of effort needed to create a set of cross packages sufficient to create a workable rootfs was huge, it took me many weeks.  Some packages like perl were just so cross-unfriendly that they were basically out of reach (although I later saw other people have done the invasive magic necessary).  It did work well, and I added patches for busybox RPM support that allowed it to do more useful things like erase and keep a package database.  The packaging was valuable in itself but a nice advantage was the source RPMs it generated ensuring GPL compliance.

My background with Openmoko

Subsequently I spent 14 months as (mainly) the kernel maintainer for Openmoko.  Openmoko had an OpenEmbedded basis for it’s rootfs, also a cross system.  I attempted to use it for “hello world” while I was at Openmoko, but it broke because I was on a newly released Fedora.  How it broke was very revealing, the official way to get started with it was to run a huge script that wgetted and locally built 1100 packages.  It died due to some assumption somewhere breaking while it tried to build host dbus libraries.

What I wanted was a cross toolchain that would let me package “hello world”.  What I got was a massive host build action including host dbus libs.  I have perfectly good host dbus libs in my Fedora install, I enquired about it and was told they were the “wrong” libs for the expectation of the rest of the packages, so they had to be rebuilt.

I gave up on trying to use OpenEmbedded, as I guess most of Openmoko’s customers did.

After Openmoko imploded, I designed the software architecture (and influenced the hardware design in some aspects) for the txtr reader device.  On this device, I put into action various lessons I had learned in how not to do things from Openmoko.  I will write further about the other lessons in future articles, but here’s the first one:

Lesson #1: Don’t compile your own rootfs

I was told by a manager at Openmoko that Openmoko had hired most of the main devs of OpenEmbedded and were paying for that accordingly.  This was a pretty big drain on their resources over a long period.

In contrast, nowadays you can head over to http://fedoraproject.org/wiki/Architectures/ARM and download a generic rootfs tarball of prebuilt binaries for ARMv5 and above[1].  It’s made from unpacking prebuilt binary packages.  Once you boot into it, you can install further packages with the usual yum install type action.  You can be up in a high quality rootfs in five minutes flat.

You do not need to go around compiling everything personally when binary packages exist from a reputable distro already.  Normal distros provide -dev and -devel packages for you to link against too, so you do not need to recompile the universe just because you want to build “hello world” either.  That’s how we do things on desktop and server systems, as the processors involved get stronger embedded does not have to be different.

If you want to cross-build specific packages, you install the Fedora ARM Cross Toolchain RPMs on you host via yum and you are ready to go in a couple of minutes.  This is very useful for cooking the kernel on your host both to get started and during development; you can’t native-build the bootstrap stuff needed to boot your platform.  But that’s just a cross compiler and related pieces, it’s not a cross distro.  (The guy from emdebian at this FOSDEM talk also made this point that you do not need to get into making your own toolchain, your distro should have one you can just install).

Fedora ARM’s strategy is native build.  So you install gcc and other dependencies into the actual device, and use standard rpmbuild to build your package there; you can also just configure ; make ; make install for development too down there.  If something’s missing on the rootfs you can yum install it.

(1 To make the comparison fair to openmoko Fedora ARM came along too late for them to choose it from the start, and the GTA02 s3c2442 was not a v5 class processor, they would have been into a distro recook after changing the distro-level compile options.  However my worry is not repeating Openmoko’s errors and today Fedora ARM is available.)

Quality and Quantity

Another major issue is distro quality.  I was so surprised to hear at Fosdem Dr Mickey Lauer of OpenEmbedded boast about the number of devices that managed to use that distro (including the sad shape of the GTA02) and say that unlike the other cross distros, OpenEmbedded focused on “Quantity not Quality”.  From my experience I think he’s right alright about not focusing on quality, and he did go on to explain there are problems with OpenEmbedded they are trying to address.

In the near future, there will be a carcrash between these difficult cross distros that have relatively poor quality and strange requirements to use them and standard, “proper distros” like Fedora ARM, because on higher-end ARMv5s say 400MHz and above, it is already perfectly possible to compare the two worlds on the same device.  I think many devs currently are trained by their experience with buildroot type systems to assume they have to personally build everything Gentoo style.  However as CPUs increase in power at the same price point, the ways of working with these systems efficiently change, and desktop / server “treat it like a PC” lessons like the value of packaging start to really show their traditional advantages over rootfs tarballs.

Like Debian, Fedora has all kinds of rules and requirements about packaging to ensure high quality, there are a huge number of users of these two normal distributions that leads to tested and debugged basic packages and their dependencies.  OpenEmbedded’s boast about number of users is not even a blip in comparison to Fedora or Debian’s consumers and contributors.

Cross distros are locked into local patch hell

A worse problem against their quality even than not many users is the patch load these projects are carrying, I think all of the cross distro projects bemoaned that they were carrying huge patchsets across a large number of packages to get them to build cross at all, and that most upstreams did not care to take them (I assume they don’t want to have to get into testing them).  To uplevel packages, which distros have to daily when they have a large package universe, it can become a nightmare of breakage because of the private patchsets being dragged around.

(BTW I also saw in another presentation that the limo foundation are carrying around more than 80MBytes of diff between their distro and the upstream projects, and these are the guys who sent out a whitepaper explaining the massive cost of delaying sending patches upstream in dollar terms.)

There was proposed a unified crossbuild patch promoting effort, but the effort seemed only to consist of a domain like “sends-patches.org” that you could use when sending patches instead of your own project name, which seems to just be tea and sympathy rather than a solution.

It’s clear that quality will tend to be higher if you are getting packages built with normal distro specfiles and no pile of local patches to get them to build cross (because they were built native).  Combined with higher quality thresholds at the project level and sheer number of users, native Fedora (or Debian) rootfs basis will provide Quantity and Quality if your processor is appropriate.

A couple of hours after the talk I had an interesting conversation with OpenInkpot dev Mikhail Gusarov, who I found also shared my lack of enthusiasm for OpenEmbedded, although he is trapped still in the cross niche generally by the weak processors he targets at the moment.

[update Feb 10 09:00] Mikhail has written his own response, he still likes the speed of cross (and still hates OpenEmbedded).  But there’s some confusion about what Fedora ARM offers, it’s a generic ARMv5 rootfs, it doesn’t care what exact kind of CPU, vendor or peripherals available.  Build farms are less of a requirement when you are no longer building your rootfs but installing it from distro binary packages.  Sheevaplug makes available a 1.2GHz Marvell ARM compatible with 512MBytes of SDRAM that Fedora ARM can work on if you need a native build machine.  Shortly fast dual processor Cortex A9 machines will become available.