Lesson 3: Test is not special
Commonly in embedded work test is the “red-haired stepchild”, nobody wants to take care of it and by common, silent consent it is always left until last. Eventually the need for a test plan becomes overwhelming as the date to go to the factory nears, and the task is assigned to the most junior engineers available, since everybody knows that test is the death knell of your career.
Coming cold to and excluded from being inside an already-existing project, the engineers try to create some kind of test coverage the best way they can. At openmoko two giant test suites were created, DM1 and DM2, written by people who were learning C for the first time. I got the job of modernizing this code so I know from experience the code was already truly terrible and bitrotted at an alarming rate. However I had to admire the guys who wrote it, with everything against them and little experience they did manage to create something that did provide test coverage at the factory, however much it was on life-support.
Totentanz
Similarly, Openmoko used production test jigs, special additional PCBs that formed a kind of custom test environment for the PCB under test. At one version of GTA03 there were so many test points added it was a serious concern that the board would break down under the overall pressure needed to mate the spring-loaded test probes to the test points.
Jigs and test points have an obvious advantage in terms of test throughput, but there are some big disadvantages.
First, you have to design and build the jig, and track changes to the actual device with it. This effort is completely disconnected from moving your actual product on, except that it’s meant to help in production.
Second, test points don’t test your connectors; the test point may be connected OK but not the connector pin the user actually accesses.
Third, you need something else outside the device to assess what is happening on the test points, the code for that also has to be written and maintained against changes in the actual product. It also means that it’s not possible for the tests to be casually performed outside the factory, or maybe by the original engineers if they have access to the ATE gear themselves.
Pain into torture
Additionally the bringup of GTA02 required special versions of U-Boot and kernel which had added “test magic” created by the test guys and unknown to anyone else. These versions were seldom uplevelled.
Since GTA02 had raw NAND, it needed filling up at the factory with the rootfs. The way to do this was via a very fragile OpenOCD using a custom USB – serial based device that was bitbanged. It only worked with certain versions of the usb library needed to talk to it.
All of these quirks and requirements at the factory made production runs difficult and expensive to get right.
I only hurt you because I love you
I spent a lot of time thinking about how to avoid this end result next time I would design something. The mistakes started in having anything special for test I concluded. The jig: special, and so evil. Test kernels or bootloader: special -> evil. Test rootfs -> Evil. test software, like Openmoko’s DM1 and DM2, evil. The device should naturally be able to test itself with the arrangements that already exist inside it to operate at all.
The answer to the problem of “production test” is to completely subsume it into the rest of the design. So it is the responsibility of Linux drivers to provide enough functionality by probe errors, or sysfs features, that one can perform test and diagnosis. The “test suite” should boil down to a bash script that is using features exposed in a normal shipping rootfs and kernel. Bash is ideal because most of the test action will be calling existing commandline tools like ifconfig, ping, l2ping and grepping or looking at their return code, this is what bash is best at. It’s also easily understood and edited by anyone who has worked with Linux for a while.
The bootloader is required for test in only one capacity, it is the only part of the system that is capable to run the SDRAM tests; once you enter Linux you can’t perform a full SDRAM test any more. But even that should be done by the one shipping bootloader image.
In many cases, device interfaces can be tested by external loopback connectors, this proves connectivity through the connectors and it leaves open the possibility of end-users being able to run the same tests on the shipping rootfs.
On the first day of FOSDEM I sat through a presentation on what could be called another “U-Boot derivative”. One of the greatest asspains at Openmoko was the various kinds of Hell caused by the U-Boot bootloader and its philosophy, which can be summed up as “I wanna be Linux when I grow up”.
One of the coolest features of the AT91RM9200 we have been designing with for a couple of years now is the FIQ, or Fast Interrupt Request. This is basically the NMI of the ARM world. It is a bit difficult to get working (Milosch Meriac helped me with the initial version some years ago) because its job is to interrupt WHATEVER is happening and start running your FIQ handler within 1us or so NO EXCUSES. This is the very very hard end of hard realtime, it does what it claims but it does not respect any privacy that Linux may need as an OS and we will see that needs care.
For over a year I have been using an 32MB ARM9-based board I designed with a 1GB USB stick as my mailserver. It is powered from a USB port on my firewall box and takes around 1W. 
After getting on for four months, my mac80211 injection patches have been accepted by the powers-that-be and have made their way into the Linus git repo, the crucible from which vanilla kernel versions are forged, and the upstream from which all major distros are ultimately basing their kernels on. (Edit: they are present in 2.6.23-rc1 also).
The last few months I have been working on the 
I'm