Commonly in embedded work test is the “red-haired stepchild”, nobody wants to take care of it and by common, silent consent it is always left until last.  Eventually the need for a test plan becomes overwhelming as the date to go to the factory nears, and the task is assigned to the most junior engineers available, since everybody knows that test is the death knell of your career.

Coming cold to and excluded from being inside an already-existing project, the engineers try to create some kind of test coverage the best way they can.  At openmoko two giant test suites were created, DM1 and DM2, written by people who were learning C for the first time.  I got the job of modernizing this code so I know from experience the code was already truly terrible and bitrotted at an alarming rate.  However I had to admire the guys who wrote it, with everything against them and little experience they did manage to create something that did provide test coverage at the factory, however much it was on life-support.

Totentanz

Similarly, Openmoko used production test jigs, special additional PCBs that formed a kind of custom test environment for the PCB under test.  At one version of GTA03 there were so many test points added it was a serious concern that the board would break down under the overall pressure needed to mate the spring-loaded test probes to the test points.

Jigs and test points have an obvious advantage in terms of test throughput, but there are some big disadvantages.

First, you have to design and build the jig, and track changes to the actual device with it.  This effort is completely disconnected from moving your actual product on, except that it’s meant to help in production.

Second, test points don’t test your connectors; the test point may be connected OK but not the connector pin the user actually accesses.

Third, you need something else outside the device to assess what is happening on the test points, the code for that also has to be written and maintained against changes in the actual product.  It also means that it’s not possible for the tests to be casually performed outside the factory, or maybe by the original engineers if they have access to the ATE gear themselves.

Pain into torture

Additionally the bringup of GTA02 required special versions of U-Boot and kernel which had added “test magic” created by the test guys and unknown to anyone else.  These versions were seldom uplevelled.

Since GTA02 had raw NAND, it needed filling up at the factory with the rootfs.  The way to do this was via a very fragile OpenOCD using a custom USB – serial based device that was bitbanged.  It only worked with certain versions of the usb library needed to talk to it.

All of these quirks and requirements at the factory made production runs difficult and expensive to get right.

I only hurt you because I love you

I spent a lot of time thinking about how to avoid this end result next time I would design something.  The mistakes started in having anything special for test I concluded.  The jig: special, and so evil.  Test kernels or bootloader: special -> evil.  Test rootfs -> Evil.  test software, like Openmoko’s DM1 and DM2, evil.  The device should naturally be able to test itself with the arrangements that already exist inside it to operate at all.

The answer to the problem of “production test” is to completely subsume it into the rest of the design.  So it is the responsibility of Linux drivers to provide enough functionality by probe errors, or sysfs features, that one can perform test and diagnosis.  The “test suite” should boil down to a bash script that is using features exposed in a normal shipping rootfs and kernel.  Bash is ideal because most of the test action will be calling existing commandline tools like ifconfig, ping, l2ping and grepping or looking at their return code, this is what bash is best at.  It’s also easily understood and edited by anyone who has worked with Linux for a while.

The bootloader is required for test in only one capacity, it is the only part of the system that is capable to run the SDRAM tests; once you enter Linux you can’t perform a full SDRAM test any more.  But even that should be done by the one shipping bootloader image.

In many cases, device interfaces can be tested by external loopback connectors, this proves connectivity through the connectors and it leaves open the possibility of end-users being able to run the same tests on the shipping rootfs.

On the first day of FOSDEM I sat through a presentation on what could be called another “U-Boot derivative”.  One of the greatest asspains at Openmoko was the various kinds of Hell caused by the U-Boot bootloader and its philosophy, which can be summed up as “I wanna be Linux when I grow up”.

Configure system is a bad alternative to good bootloader design

First, it has a config system.  That should be good though, right?  The problem with the config system is that if anything differs from your current config, you must build another incompatible binary with another config and take care of that.  When you have more than a handful of different boards, you are in a maze of incompatible bootloaders.  Openmoko took it one step further, they mandated a different bootloader binary per PCB revision, so left unchecked there would have been a continuous proliferation of incompatible bootloaders, all basically the same.

All persistent bootloader private state is EVIL

Second, U-Boot thinks it’s a good idea to have these environment “scripts”, because it’s “configurable”.  Actually, the job of a bootloader is to Load, then Boot Linux.  You don’t need any configurability for that if the bootloader can figure out what it’s running on and therefore where the memory is and how much there is.  These scripts expose a really deadly trap I call “private bootloader state”.  It means the bootloader stores stuff in nonvolatile memory on the PCB and acts different according to what it hides there.  The end result is that two boards from the same factory may act totally different even with the same rootfs due to “bootloader secrets”.  This is totally needless and ALL private bootloader state can be eliminated by correct design of the bootloader leading to completely deterministic boot action per rootfs.

A good example how that lead you to the path to hell is hardcoding in the U-Boot environment of the amount of kernel image you will copy from somewhere.  People commonly set it to 2MBytes, forget about it and one day they generate a 2.1MB kernel image and wonder why decompress blows up.  Actually, that whole procedure is insane, the kernels are uImages that report their length in a header.  The bootloader should examine the header and compute the length of image to pull.  But that doesn’t fit with this “environment” nonsense.

Do Linux Stuff In Linux

In any of these bloated U-Boot style bootloaders, is there even one feature they do better than the same feature in Linux?  The startup time should be better by a few 100ms.  Other than that, no, every single bloated “I will add it to the bootloader beacuse I can” feature is shittier than you get in Linux.  Every single feature!

If you need some advanced capability or backup / recovery boot action, check for a button held down at boot-time in the bootloader and go fetch a different Linux partition + kernel.  Use standard Linux tools and shells.  In return, get really high quality network stack, proper USB support, NAND access that’s compatible to your main Linux system access in BBT / ECC terms, and all the other advantages of Linux.

Do your peripheral bringup in drivers in Linux

Typically you do not need ANY bringup in the bootloader except SDRAM controller and chip init, since it’s a prerequisite to put Linux in the RAM that it’s initialized.

That’s right, all the megabytes of source spent in U-Boot providing support for so many kinds of peripheral is a waste of time, effort and maintenance.  I am being kind saying “maintenance”, because the drivers in U-Boot are typically “dumbed down” versions of the equivalent Linux driver that were forked irretrievably the moment all the Linux APIs were ripped, so there’s no coherent effort to keep them up to date with the Linux ones .  Lately I saw that they try to ape some Linux APIs there… why not go the whole hog and just load and boot real Linux?  After all, modern CPUs can be running your driver probes in Linux in ~2 seconds from power using a bootloader that doesn’t get in the way.

You typically don’t even need to talk to the PMU in the bootloader, after all, you are running code fine already, right?  Otherwise you wouldn’t be able to run the bootloader code itself.

Fat girl in Ibiza

At least at Openmoko, code quality inside U-Boot was awful bad.  I called U-Boot on the lists there “the fat girl in Ibiza” because you know she’s going to do anything you want.  All kinds of constant-only code, weird new scripting keywords were added for test undocumented, you name it.  Hardware guys felt up to writing such code secretly by themselves once they learned the software engineering marvel that is *((unsigned int *)0x…) = 0x…;

Your bootloader just tests SDRAM

There’s only one test action your bootloader is suited to do, and that is SDRAM test.  Once you are in Linux, it can’t perform a full SDRAM test while it’s running.  But the bootloader is typically starting from on-CPU SRAM, it can actually run a true SRAM test from there.  Otherwise, the bootloader should be completely absent from the test plan.  All other tests should be performed in Linux via standard driver and rootfs tools.

More about board and test and board bringup will feature in another report of a lesson learned.

Qi

While at Openmoko (mainly) I wrote a bootloader that meets these ideals, you can find it in git here One of the nicest things about it is that unlike the bloated bootloaders whose job never finishes trying to become Linux cargo cult style, Qi has been pretty much complete for a few months.  It’s a new job to support a new CPU, a much smaller job to add a new board and it doesn’t want to talk to your peripherals anyway so no problem there.

Qi creates one binary per CPU, that supports all boards with that CPU.  That sounds like a big job but we don’t care about your peripherals so all boards with the same CPU look almost identical.  You have to find something that can detect your particular board at runtime, for example NOR device ID read check.  So there is zero build-time config and Qi generates all CPU support when it’s buit, it takes 3 sec or so typically.

Typical bootloader binary size per CPU is 28-30KBytes.  That supports VFAT, ext2/3/4 typcially the SD controller as well.  The single Qi image also supports being booted from NAND, JTAG or SD Card on processors that support it just by being copied into place and without any changes.

There is zero bootloader private state, however Qi can look in the rootfs and append kernel commandline text from the content of a filesystem file.  This maintains the rule that boot should be completely deterministic per rootfs.

I was at Fosdem over the weekend, there were several interesting talks I attended but the most interesting one for me was a roundtable about the future of Cross distributions.  I was invited to give a 5 minute talk there which I gave, but unfortunately it was right at the end and the people before had overrun, so there was no time to make much of a coherent case.   So I am going to write some articles covering the issues involved here.

Cross as a niche

Cross itself remains absolutely necessary for systems below a certain level of horsepower.  For example, 8051, ARM7, cortex M3 are not really capable to consider native build.  But processors get faster each year, a lot of things we would have used an 8051 on use an ARM7 or cortex M3 now, in a few years it is likely that baseline has moved further up and it’s an ARM9 equivalent.  What I am suggesting then is that over time, the niche where you need cross is shrinking.

All four of the cross distros at FOSDEM target a CPU that’s powerful enough to run Linux, but not powerful enough to build its own binaries.  That is the niche that I believe will shrink to the point that it won’t support all these cross Linux distro projects, possibly none of them in the end.

My background with cross Linux

A few years ago I created an RPM-based cross distro singlehanded, and used it on a product for a customer  This was AT91RM9200-based, a 200MHz ARM9 with 32MBytes of SDRAM.  The amount of effort needed to create a set of cross packages sufficient to create a workable rootfs was huge, it took me many weeks.  Some packages like perl were just so cross-unfriendly that they were basically out of reach (although I later saw other people have done the invasive magic necessary).  It did work well, and I added patches for busybox RPM support that allowed it to do more useful things like erase and keep a package database.  The packaging was valuable in itself but a nice advantage was the source RPMs it generated ensuring GPL compliance.

My background with Openmoko

Subsequently I spent 14 months as (mainly) the kernel maintainer for Openmoko.  Openmoko had an OpenEmbedded basis for it’s rootfs, also a cross system.  I attempted to use it for “hello world” while I was at Openmoko, but it broke because I was on a newly released Fedora.  How it broke was very revealing, the official way to get started with it was to run a huge script that wgetted and locally built 1100 packages.  It died due to some assumption somewhere breaking while it tried to build host dbus libraries.

What I wanted was a cross toolchain that would let me package “hello world”.  What I got was a massive host build action including host dbus libs.  I have perfectly good host dbus libs in my Fedora install, I enquired about it and was told they were the “wrong” libs for the expectation of the rest of the packages, so they had to be rebuilt.

I gave up on trying to use OpenEmbedded, as I guess most of Openmoko’s customers did.

After Openmoko imploded, I designed the software architecture (and influenced the hardware design in some aspects) for the txtr reader device.  On this device, I put into action various lessons I had learned in how not to do things from Openmoko.  I will write further about the other lessons in future articles, but here’s the first one:

Lesson #1: Don’t compile your own rootfs

I was told by a manager at Openmoko that Openmoko had hired most of the main devs of OpenEmbedded and were paying for that accordingly.  This was a pretty big drain on their resources over a long period.

In contrast, nowadays you can head over to http://fedoraproject.org/wiki/Architectures/ARM and download a generic rootfs tarball of prebuilt binaries for ARMv5 and above[1].  It’s made from unpacking prebuilt binary packages.  Once you boot into it, you can install further packages with the usual yum install type action.  You can be up in a high quality rootfs in five minutes flat.

You do not need to go around compiling everything personally when binary packages exist from a reputable distro already.  Normal distros provide -dev and -devel packages for you to link against too, so you do not need to recompile the universe just because you want to build “hello world” either.  That’s how we do things on desktop and server systems, as the processors involved get stronger embedded does not have to be different.

If you want to cross-build specific packages, you install the Fedora ARM Cross Toolchain RPMs on you host via yum and you are ready to go in a couple of minutes.  This is very useful for cooking the kernel on your host both to get started and during development; you can’t native-build the bootstrap stuff needed to boot your platform.  But that’s just a cross compiler and related pieces, it’s not a cross distro.  (The guy from emdebian at this FOSDEM talk also made this point that you do not need to get into making your own toolchain, your distro should have one you can just install).

Fedora ARM’s strategy is native build.  So you install gcc and other dependencies into the actual device, and use standard rpmbuild to build your package there; you can also just configure ; make ; make install for development too down there.  If something’s missing on the rootfs you can yum install it.

(1 To make the comparison fair to openmoko Fedora ARM came along too late for them to choose it from the start, and the GTA02 s3c2442 was not a v5 class processor, they would have been into a distro recook after changing the distro-level compile options.  However my worry is not repeating Openmoko’s errors and today Fedora ARM is available.)

Quality and Quantity

Another major issue is distro quality.  I was so surprised to hear at Fosdem Dr Mickey Lauer of OpenEmbedded boast about the number of devices that managed to use that distro (including the sad shape of the GTA02) and say that unlike the other cross distros, OpenEmbedded focused on “Quantity not Quality”.  From my experience I think he’s right alright about not focusing on quality, and he did go on to explain there are problems with OpenEmbedded they are trying to address.

In the near future, there will be a carcrash between these difficult cross distros that have relatively poor quality and strange requirements to use them and standard, “proper distros” like Fedora ARM, because on higher-end ARMv5s say 400MHz and above, it is already perfectly possible to compare the two worlds on the same device.  I think many devs currently are trained by their experience with buildroot type systems to assume they have to personally build everything Gentoo style.  However as CPUs increase in power at the same price point, the ways of working with these systems efficiently change, and desktop / server “treat it like a PC” lessons like the value of packaging start to really show their traditional advantages over rootfs tarballs.

Like Debian, Fedora has all kinds of rules and requirements about packaging to ensure high quality, there are a huge number of users of these two normal distributions that leads to tested and debugged basic packages and their dependencies.  OpenEmbedded’s boast about number of users is not even a blip in comparison to Fedora or Debian’s consumers and contributors.

Cross distros are locked into local patch hell

A worse problem against their quality even than not many users is the patch load these projects are carrying, I think all of the cross distro projects bemoaned that they were carrying huge patchsets across a large number of packages to get them to build cross at all, and that most upstreams did not care to take them (I assume they don’t want to have to get into testing them).  To uplevel packages, which distros have to daily when they have a large package universe, it can become a nightmare of breakage because of the private patchsets being dragged around.

(BTW I also saw in another presentation that the limo foundation are carrying around more than 80MBytes of diff between their distro and the upstream projects, and these are the guys who sent out a whitepaper explaining the massive cost of delaying sending patches upstream in dollar terms.)

There was proposed a unified crossbuild patch promoting effort, but the effort seemed only to consist of a domain like “sends-patches.org” that you could use when sending patches instead of your own project name, which seems to just be tea and sympathy rather than a solution.

It’s clear that quality will tend to be higher if you are getting packages built with normal distro specfiles and no pile of local patches to get them to build cross (because they were built native).  Combined with higher quality thresholds at the project level and sheer number of users, native Fedora (or Debian) rootfs basis will provide Quantity and Quality if your processor is appropriate.

A couple of hours after the talk I had an interesting conversation with OpenInkpot dev Mikhail Gusarov, who I found also shared my lack of enthusiasm for OpenEmbedded, although he is trapped still in the cross niche generally by the weak processors he targets at the moment.

[update Feb 10 09:00] Mikhail has written his own response, he still likes the speed of cross (and still hates OpenEmbedded).  But there’s some confusion about what Fedora ARM offers, it’s a generic ARMv5 rootfs, it doesn’t care what exact kind of CPU, vendor or peripherals available.  Build farms are less of a requirement when you are no longer building your rootfs but installing it from distro binary packages.  Sheevaplug makes available a 1.2GHz Marvell ARM compatible with 512MBytes of SDRAM that Fedora ARM can work on if you need a native build machine.  Shortly fast dual processor Cortex A9 machines will become available.

I ran the last 10MByte sample against ENT and TestU01… to cut a long story short

$ ./ent ../die.c/dump3
Entropy = 7.999980 bits per byte.

Optimum compression would reduce the size
of this 10002432 byte file by 0 percent.

Chi square distribution for 10002432 samples is 281.26, and randomly
would exceed this value 25.00 percent of the times.

Arithmetic mean value of data bytes is 127.4958 (127.5 = random).
Monte Carlo value for Pi is 3.140111525 (error 0.05 percent).
Serial correlation coefficient is -0.000212 (totally uncorrelated = 0.0).

7.9999 bits of entropy per byte! TestU01 is less turnkey than the other suites — it’s literally a test library with some example code. I amended an example to call the FIPS-140-2 tests:

============== Summary results of FIPS-140-2 ==============

 File:             dump3
 Number of bits:   20000

       Test          s-value        p-value    FIPS Decision
 --------------------------------------------------------
 Monobit               9933           0.83       Pass
 Poker                11.88           0.69       Pass

 0 Runs, length 1:     2482                      Pass
 0 Runs, length 2:     1227                      Pass
 0 Runs, length 3:      630                      Pass
 0 Runs, length 4:      319                      Pass
 0 Runs, length 5:      161                      Pass
 0 Runs, length 6+:     166                      Pass

 1 Runs, length 1:     2466                      Pass
 1 Runs, length 2:     1302                      Pass
 1 Runs, length 3:      620                      Pass
 1 Runs, length 4:      311                      Pass
 1 Runs, length 5:      140                      Pass
 1 Runs, length 6+:     146                      Pass

 Longest run of 0:       16           0.14       Pass
 Longest run of 1:       14           0.46       Pass
 ----------------------------------------------------------
 All values are within the required intervals of FIPS-140-2

So the design’s output is compliant to FIPS-140-2, a requirement for many uses.

Here is a patch against 2.6.20 with the AT91RM9200 patches: it should probably apply okay to later kernels. The patch adds many comments so you probably want to read that and this at the same time. First the things you can’t do, which will save you much pain from finding them out yourself.

Things you can’t do with FIQ

One of the “private times” Linux needs to itself is the virtual memory pagetable management action. It shuts off all interrupts and rewrites the pagetable at intervals, and then goes on as before. That stops driver interrupts coming in and trying to do stuff while the pagetable is empty, or incomplete or just full of garbage.

However FIQ ignores any claim to privacy performed by shutting off normal interrupt response. That means your FIQ ISR code can come in at a “bad time”, if it tries to access memory areas mapped through the pagetable it will instead find nothing or the wrong thing or… the end result is that the FIQ ISR cannot touch any memory mapped by vmalloc.

Unfortunately, when a kernel module is loaded, its various memory footprints including the module code are allocated by… yep, vmalloc. That means your FIQ ISR code cannot live in a module, it has to be part of the monolithic kernel.

Finally all sorts of Linux code also wants “private time” or to guard against multiple access to objects by spinlocks or whatever. FIQ ISRs cannot play those games, it comes it in the middle of anything and has to get out quickly again too. So unless it is a simple macro, you can’t use any Linux APIs in the FIQ ISR.

Things you can do with FIQ

Well reading those constraints, you’re probably wondering if it is still useful. It sure is!

You can touch the memory-mapped IO in the chip using the AT91RM9200 APIs.

FIQ has the super power it will run your ISR within ~1us NO EXCUSES. That means you can rely on the ISR code to act like hardware, you trigger it and 1us later your programmed sequence occurs without fail. In turn that means FIQ is perfect for many hardware interfacing tasks, in particular management of low latency (small buffer) PDC DMA setup.

Low latency for audio traffic for example is highly desirable, but of course if there are ANY delays setting up the next PDC DMA, you get dropouts and clicks. If you allow the FIQ to handle generation of samples and management of PDC DMA, there won’t be any delays for sure, you will have perfect audio.

We found that the AT91RM9200 at 180MHz can easily handle 8kHz FIQs (a common rate for telephony) with an ISR duration of ~8us, without affecting the Ethernet or USB performance.

IPC between the FIQ and kernel worlds

The general communication for FIQ ISRs with the “real world” in the patch is to define a struct in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h that contains all of the data that is shared between FIQ and normal kernel code. The example one looks like this:

struct at91rm9200_fiq_ipc {
	int nCountFiqEvents;
};

One of these structs is defined in the main part of the patch code in arch/arm/mach-at91rm9200/at91_fiq.c like this

struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;
EXPORT_SYMBOL(at91rm9200_fiq_ipc);

That means in your other kernel code — which can be in a module, only the FIQ ISR must be in the kernel — you can have

#include <arch/at91rm9200_fiq_ipc_type.h>
extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;

and use the same struct to communicate with the FIQ ISR.

How to customize summary

1) Change struct at91rm9200_fiq_ipc in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h to have the data types you need

2) Add your FIQ ISR code to arch/arm/mach-at91rm9200/at91_fiq.c where it says “your C code goes here”

3) Import extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc; in your own kernel module and communicate with the FIQ ISR using that.

FIQ shadowing with IRQ

Ultimately the FIQ actions are going to need to interface to Linux kernel objects sooner or later, perhaps there has to be some locking or blocking action for usermode access. But we are banned from using Linux APIs in the FIQ ISR.

A powerful solution is to physically tie the FIQ signal to an IRQ input additionally. Code that is REALLY hard realtime, like the PDC DMA management, goes in FIQ, and a count of FIQs is kept. Data out of FIQ can go into a software FIFO. The less reliable IRQ watches the count of FIQs and compares it to its own count of IRQs, if it sees it has blacked out and has fallen behind, it will loop up to a certain number of times “catching up”, using the data placed by FIQ in the FIFO.

In this way, by splitting the code into “no excuses” realtime in the FIQ ISR, and “reliable only on average” realtime in the IRQ ISR, it is possible to bind actions in the FIQ to code in the ISR which can execute Linux APIs, and have the best of both worlds.

I use our Octotux Linux distro with Postfix as the MTA, gps for the greylisting and Dovecot IMAP to provide secure access to the mailstore over SSL. This has worked out really well, the warmcat.com MX record points directly to the external IP here, and the firewall box port-forwards port 25 to the embedded device. It’s silent and runs cold 24 hours a day and has never missed a beat.

A couple of weeks ago I had to look at the box again because the greylisting software was hanging. I discovered that we were being bombarded with spam, one new spam every two seconds on average, from all over the world. I adjusted the ordering of the filtering in postfix to first reject on an unknown username, that stopped so many concurrent gps sessions being needed. The server weathered that storm and the spam people gave up a few days later without getting a single one through. (They were also targeting the warmcat.com A record IP, I suppose in case it was a backup for the real MX, but they had zero luck with that either).

However it reminded me of the one inadequacy of this mailserver… when you wake up your laptop in the morning, thunderbird takes ages to run all the filters and move the new mails remotely into the right IMAP folders. That’s pretty annoying when you see the titles of mails you want to read but the USB stick on the server is maxxed out for a couple of minutes sorting eight hours worth of new emails into folders on the server. I have been pondering changing the box to one with USB2 High Speed, but it occurred to me that otherwise, the existing USB 1.1 “Full Speed”, that is, 12Mbps is completely adequate. Changing folders and moving to other emails in thunderbird is snappy. It’s just the client mail filters under the load of 500 mails in the morning. So I decided to port procmail to ARM9 Octotux, in effect to do the folder sorting as each mail came in, so there would no longer be any processing done at the client for that.

Procmail was a little bit of a beast, the sources look horrible and it uses a nonstandard autotools script, which of course does not support crosscompile. After a couple of hours I got it to build nicely in an Octotux RPM (Octotux is entirely crosscompiled RPM-based). Not having used it before, I was surprised to see there was no configuration down /etc in the package, just four small binaries in /usr/bin and some manuals (which I broke out into procmail-docs package so as not to waste space on the target). The binary package only came to 60K all told.

The first move was to direct postfix to deliver not to the maildir as before but through procmail. This simply involved editing /etc/postfix/main.cf and adding this

mailbox_command = /usr/bin/procmail

After some googling I found that procmail is going to look at the unix user’s homedir that the mail is directed to, for a file ~/.procmailrc in order to find out what it should do. Now my embdedded mailserver is setup to use Dovecot IMAP with the mailstore symlinked to the USB stick. So my user “andy” on the embedded mailserver has this in the home dir

# ll /home/andy
lrwxrwxrwx 1 root root 25 Jan 17 2007 Maildir -> /media/usbstick/mail/andy

I found looking around in the Dovecot Maildir structure in there that the mail needs to be delivered into /home/andy/Maildir/.(foldername)/new/ in order to get Dovecot to understand that it was given a new mail in a folder. The resulting /home/andy/.procmailrc looks like this:

SHELL=/bin/sh
PATH="/usr/bin:/usr/local/bin:/bin:/sbin:/usr/sbin"
LOCKFILE=/home/andy/lockfile.loc
DEFAULT=/home/andy/Maildir/
BITBUCKET=/dev/null
LOCKTIMEOUT=10
LOGFILE=/tmp/procmail_log
LOGABSTRACT=no
VERBOSE=no

:0:
* ^List-Id: Development discussions related to Fedora
$DEFAULT/.fedora-devel-list/new

You simply repeat the last stanzas with some unique identifying header part of mails that are to be filed in a given folder, and give the destination folder name preceded by a ‘.’ and followed by /new as shown. You don’t need to run anything or restart anything after making changes here, they will be immediately used on the next email that is delivered. If your folder lives in a hierarchy, the folder on disk looks like, for example, .INBOX.CentOS for a folder CentOS that appears as a child of INBOX. The stanza for that would look like

:0:
* ^List-Id: CentOS mailing list
$DEFAULT/.INBOX.CentOS/new

You can also direct procmail to use a central procmailrc file, presumably in /etc, by giving the path to it on the procmail invocation line in /etc/postfix/main.cf. In my case I only take mails for myself, so I stuck with the per-user ~/.procmailrc.

No changes were needed anywhere else in the server setup, all I had to do was turn off all my thunderbird filters and enjoy immediate access to the emails via Dovecot IMAP in the morning, without upgrading any hardware.

Assuming nothing bad happens in the next few weeks leading to their being reverted (unlikely I would think, since they don’t interfere with much existing code), then a standardized driver-independent ieee80211 packet injection methodology will soon be available by default in all major distro kernels. Currently if you want to perform packet injection, you enter into a dark underworld of individual driver patching, having to cook custom kernels and make animal sacrifices to forgotten Gods. But now with the injection patches, for the devices with mac80211 drivers all that crap is blown away and every 2.6.23 kernel will offer the capability built-in.

Here are the list of mac80211 drivers and whether I have actually seen good injection. All of the drivers are expected to work, but I don’t have all the hardware.

I have also provided the Packetspammer commandline applet to show how to use the injection API from userspace, this provides a simple, GPL2 tested base for making your own injection code for your own apps.

Work started on it in Dec 2006 by providing the old Linux stack driver patches. The real reason for the sustained effort is to enable Penumbra to work “out of of the box” on not only Linux desktops and laptops but generic embedded devices as well.

But one of the biggest hurdles in creating the patch was not technical, since I already had the core functionality working, but in fact the Linux kernel coding style. In some ways the coding style fits well with my own personal style (formed over 20 years of writing C and C++), we basically use the same K&R style. There are some spacing and commenting rules that are actually better than my style and I will adopt them wholesale. But that’s where the fun stops and the recrimination begins!

The basic problem is the combination of three rules which has a terrible effect on eliminating what I consider good coding practice due to the constraints introduced by those rules.

• Tabs are 8 chars NO EXCEPTIONS
• Lines are less than 80 cols NO EXCEPTIONS
• Everything in { } is indented by a tab (except switch cases!)

Now almost everything is inside a function body, so that gets you down to 72 chars already. And if your function is doing something non-trivial, your code is probably inside a while() for a for() and there are one or more levels of if() to decide to do it or not. Pretty soon you are writing code crushed up against the 80-col limit with only 30 chars that are usable and 50 spaces behind it. It strongly puts me in mind of the Bonsai Kittens fake website that showed how to push kittens into bottles so they would grow into the shape of the bottle.

Under these abnormal circumstances, certain things become very difficult to do:

• \t\t\t\t\t”any kind of long ” \n\t\t\t\t\t “string has to be ” \n\t\t\t\t\t “artificially chopped up”
• nested if()s may make perfect sense to explain your code logic. But you can no longer afford them because of the tab each one adds. So you have to invert the if() sense and use a goto (I kid you not, this is preferred due to the coding style rules)
• I strongly prefer descriptive variable names which include type. Type is part of the information you need to understand what that variable is when looking at it. “nCountWaysILoveHer” tells me (now, and in 6 months when I have forgotten the code) it is an int that is counting a specific thing. “i” or “cnt” could be anything, although “cnt” is better. But you can’t afford a long variable name with the rules above, you can get into a situation where there is not even enough room left after the tabbing to hold just the variable name on a single line.

On that last point there is some handwaving nonsense in the coding style doc for Linux that “C programmers don’t use long variable names”… well I call bullshit on that one. The truth is that because of the other tabbing and length rules, Kernel C programmers can’t use long variable names even if they realized it was better: they ran out of room for it.

To be fair to the coding style doc it does have a point when talking about what to do when the indents get too much: it suggests to break the indented content out into a new function and to call through to it. It also says that massively indented code means you were screwed anyway, because the logic was too complicated, and that can also be true. But calling through to functions can be a very bad fit if the code you are migrating out touches many variables defined at the parent function top level.

I am still working through the style rules trying to see what I should take on board to replace my own style and what I have to “fake” just for kernel code, but it seems to me life would be better for everyone if they relaxed the line length to 120 chars instead of 80.

Makes no sense to have a license to Mordor for each of the hundreds of plasma displays at Luton Airport. It sparked a conversation with my stepson about why it was chosen: presumably because the devs didn’t know anything else than Windows, they will spend the rest of their working life acting as agents for Microsoft. This is the fruit of the Jesuit priest Ballmer’s creed of “give me a developer until he has 7 months in the industry and I will give you a Microsoft Trained Monkey”.

Using the same or cheaper hardware, an embedded Linux implementation would have had at least the same performance and much more flexibility. Ogg Matroska is a patent-free high quality video solution that plays back out of the box on Fedora for example. And if you wanted a hundred or a thousand displays your software licensing costs would stay at exactly $0.

I knew I didn’t want to repeat this method from the start, so I began to work towards being able to use the RPM packaging system even with my crosscompiled stuff. Having the full RPM app and database doesn’t make sense for an Embedded system though, for example our standard AT91RM9200 platform only has 8MB of flash for all nonvolatile storage. I found that the excellent busybox had the start of an rpm implementation already, but although the code was really well-written, the implementation was not complete enough to do anything useful: it did not keep a database and there was therefore no possibility to erase packages, for example.

Over a few months I added more functionality to the busybox applet, including keeping a “database” of installed RPM headers down /var/lib to emulate most of the capabilities of real RPM. With this in place, I then went on to package all of the apps that I was interested to run on my distro. There are numerous advantages to formalizing the management of code into packages once you accept that you will only place packaged stuff on the device.

Creating an RPM binary also creates a source RPM (SRPM) that is contains all of the sources and build specification for that binary, and both the binary and source are versioned. Therefore so long as you offer the matching SRPM at the same time as you give someone the binary RPM, your GPL requirements for giving sources are solved in one stroke.

The patchlevel of any device can now be interrogated and used to select updates as well, in a formal manner. You can also express the need in a package to have “required” dependent packages of specific versions, reflecting a common situation where after some release you need a library of a certain version for it to work.

The “cost” for this level of control over your device boil down to: no more ad-hoc building. Even development builds are done through rpm packages. We provide a script that will remotely install over scp and ssh the latest version of a given package from the host commandline, even one that rebuilds the package before installing it remotely, and for every single edit you can be certain the sources are captured in the SRPM. We found a typical system is using only 100K of uncompressed data (typ 50K on jffs2) to maintain the RPM headers that make the packaging work. That is not significant even on a box with only 8MBytes of Flash.

Check out the Wiki at http://octotux.org for more details and the repository which contains the packages.