LPC3250 from scratch

NXP’s new LPC32x0 is a very cheap and feature-filled ARM926.  According to Digikey anyway, it’s the cheapest ARM chip with at least v5 instruction set that’s going.  That’s important not just because of the extra processor strength over older ARM9 core, but because ARM Fedora is built requiring armv5 or newer instruction set.  Being able to use ARM Fedora and RPM as a basis means freedom from compromise and having to own the building of an integrated, self-consistent rootfs; you can just focus on doing your specialized code on top using the reliable Fedora quality basis.

There are four chips in the series, they differ in having an LCD controller and Ethernet MAC or not; also the smallest guy LPC3220 has “only” 128KBytes of Static IRAM and the others 256KBytes.  Well, having worked with the 2KBytes of internal static RAM on the iMX31 for SD boot on Qi, having to shoehorn an SD card driver in there, even 128KBytes is crazy amounts.

They have support for resistive touchscreen, USB OTG, NAND controller and Mobile DDR, and up to 266MHz CPU clock at 1.4V Vcore (208MHz at 1.2V Vcore but as we will see that is not entirely true).  They don’t support SD Card boot from ROM, but that can be solved for about US$0.30 as will be shown.

In short they’re ready to do some serious embedded work at a budget price.

Embedded Artists EA3250 Dev kit

There are a few dev kits around for LPC32x0, Hitex have a cheap USB stick format one that has been permanently two weeks away from availability since I first looked at it a month or so ago, and it still is two weeks away.

NXP anoited two real dev boards they evidently worked with the vendors for during development, they don’t actually make an NXP branded dev board, it’s Phytec and Embedded Artists.  Since the EA one is in Digikey, that’s what I ended up with.

The dev board is well made but there are some problems with it: like many dev boards it comes in two halves, a cheaper, large breakout board and a 8-layer DIMM type board that has the actual CPU BGA and memory.  In an act of supreme lunk-headedness, the large breakout board re-uses the Pn.m nomenclature that the CPU uses for GPIO, with no care to retain the CPU mapping.  So for example a header is marked with having a pin P1.27, very confusingly this is nothing to do with the CPU GPIO P1.27.  This is also true in the schemtatics for the baseboard and CPU board, complete confusion trying to trace a signal between the two boards or looking for a misnamed signal on the baseboard.

DDR trouble #1

There’s also a more serious problem, the DDR on the CPU card is marginal and Embedded Artists have made a recall where they will replace the board with one with a different DDR DRAM for free.  The CPU board I got was affected but not at room temperature; they want the old card sending back and I am not finished with it yet, so I will take advantage of this recall later.

DDR trouble #2

There’s another problem with DDR, NXP issued an errata confessing their inverted signal for the differential DDR clock is skewed by no less than 1.2ns from the uninverted partner of the differential pair, a huge skew.  This issue removes a lot of comfort zone from designing with DDR and means only some memory devices will tolerate it.  However in the EA board case, they have not used the workaround suggested by NXP which is to nuke the inverted output entirely and make the clock unipolar, so the situation can’t be that bad.

DDR trouble #3

The last problem with DDR… operation at 208MHz with 1.2V Vcore is fine for the CPU, in fact while screwing with the PLL I had the CPU running fine at 400MHz, although there is no way to divide anything useful down for the memory clock at that speed and it’s illegal for the PLL over temperature, which tops out at 320MHz.  However at 1.2V and 208MHz, the CPU side of the DDR bus is unreliable: it requires cranking to 1.4V to operate DDR even at 104/208MHz.  That’s annoying because since 1.2V is needed anyway for other circuitry, it could have saved a regulator.

Unbrickability of LPC32x0

LPC32x0 chips feature UART-based bootloader injection… if you pull down the SERVICE_N pin, then next boot the ROM in the CPU will bring up UART5 at 115200 n81 and issue a simple protocol byte allowing for bootloader download.

Since I couldn’t find a Linux tool for injecting bootloaders, just a Windows one, I wrote a commandline tool for it and added it to Qi build.

http://git.warmcat.com/cgi-bin/cgit/qi/tree/tools/lpcboot.c?h=lpc

No matter how broken your nonvolatile image gets, it’s still possible to recover the device via this UART scheme with a USB <-> LVTTL serial cable.

Bootloader Hell

The LPC32x0 bootloader situation is ugly.  Basically NXP provided a huge suite used for chip verification called CDL (“common driver library”), this is a sort of chopped down OS in bootloader form.  It has all kinds of functions to drive the chip peripherals and test memory, but nothing to actually boot Linux!

What EA shipped, and what you are meant to do as a system integrator, is get an implementation of CDL in the form of “S1L” — stage one bootloader — to load U-Boot, which will then load Linux.  Both U-Boot and S1L — itself like 130KBytes! — store “state” on the board.  It leads to this insane situation that two bootloaders with two kinds of state must be right in order to boot.  Things are further complicated that SPI boot only allows the first 56KBytes to be loaded by ROM into IRAM and executed, but the bloated bootloaders are too big to do this in one step.

Bootloader Heaven

I added support for LPC32x0 to Qi last week, this is a single < 30KBytes image that can boot itself from SPI Flash or UART 5 injection and pull Linux from SD Card in VFAT partition or also via SPI Flash.  Boot from cold, with Qi and Kernel in SPI Flash to Fedora 12 bash prompt is less than 4 seconds.

http://git.warmcat.com/cgi-bin/cgit/qi/log/?h=lpc

This replaces both S1L and U-Boot, and in accordance with Qi philosophy it holds no state at all on the device.

Its strategy is if it finds that it is running via injection on UART5, it copies itself into SPI Flash / EEPROM so it will run next boot from there, and if it finds an SD Card kernel image it will also copy that into SPI Flash.

When it finds it is running from a non-injection source, ie, a normal boot from SPI Flash, it favours any kernel it can find on the first, VFAT, partition of an SD Card if found, otherwise it boots from the kernel also in SPI Flash.

This is why the lack of ROM -> SD Card boot is not critical, the cheapest, smallest SPI EEPROM can be used to contain Qi, which will then load the kernel and rootfs from SD Card if that’s what’s needed as during development.  If SD Card is overkill for the job, then Qi, Kernel and initrd can all be pushed into a single US$2 32MBit SPI Flash.

Since I only have the Embedded Artists board right now it wants to see a kernel image called k-ea3250.img on the SD Card; the way Qi works you add a new file for each supported board in ./src/cpu/lpc32x0/ copied from embart-steppingstone.c in that directory; the bootloaders need some way to identify what they’re running on at runtime since there is only a single image per cpu that supports all devices.  See  http://git.warmcat.com/cgi-bin/cgit/qi/tree/src/cpu/lpc32x0/embart-steppingstone.c?h=lpc for an idea of what’s involved to support a new board in the bootloader image.

Commonly in embedded work test is the “red-haired stepchild”, nobody wants to take care of it and by common, silent consent it is always left until last.  Eventually the need for a test plan becomes overwhelming as the date to go to the factory nears, and the task is assigned to the most junior engineers available, since everybody knows that test is the death knell of your career.

Coming cold to and excluded from being inside an already-existing project, the engineers try to create some kind of test coverage the best way they can.  At openmoko two giant test suites were created, DM1 and DM2, written by people who were learning C for the first time.  I got the job of modernizing this code so I know from experience the code was already truly terrible and bitrotted at an alarming rate.  However I had to admire the guys who wrote it, with everything against them and little experience they did manage to create something that did provide test coverage at the factory, however much it was on life-support.

Totentanz

Similarly, Openmoko used production test jigs, special additional PCBs that formed a kind of custom test environment for the PCB under test.  At one version of GTA03 there were so many test points added it was a serious concern that the board would break down under the overall pressure needed to mate the spring-loaded test probes to the test points.

Jigs and test points have an obvious advantage in terms of test throughput, but there are some big disadvantages.

First, you have to design and build the jig, and track changes to the actual device with it.  This effort is completely disconnected from moving your actual product on, except that it’s meant to help in production.

Second, test points don’t test your connectors; the test point may be connected OK but not the connector pin the user actually accesses.

Third, you need something else outside the device to assess what is happening on the test points, the code for that also has to be written and maintained against changes in the actual product.  It also means that it’s not possible for the tests to be casually performed outside the factory, or maybe by the original engineers if they have access to the ATE gear themselves.

Pain into torture

Additionally the bringup of GTA02 required special versions of U-Boot and kernel which had added “test magic” created by the test guys and unknown to anyone else.  These versions were seldom uplevelled.

Since GTA02 had raw NAND, it needed filling up at the factory with the rootfs.  The way to do this was via a very fragile OpenOCD using a custom USB – serial based device that was bitbanged.  It only worked with certain versions of the usb library needed to talk to it.

All of these quirks and requirements at the factory made production runs difficult and expensive to get right.

I only hurt you because I love you

I spent a lot of time thinking about how to avoid this end result next time I would design something.  The mistakes started in having anything special for test I concluded.  The jig: special, and so evil.  Test kernels or bootloader: special -> evil.  Test rootfs -> Evil.  test software, like Openmoko’s DM1 and DM2, evil.  The device should naturally be able to test itself with the arrangements that already exist inside it to operate at all.

The answer to the problem of “production test” is to completely subsume it into the rest of the design.  So it is the responsibility of Linux drivers to provide enough functionality by probe errors, or sysfs features, that one can perform test and diagnosis.  The “test suite” should boil down to a bash script that is using features exposed in a normal shipping rootfs and kernel.  Bash is ideal because most of the test action will be calling existing commandline tools like ifconfig, ping, l2ping and grepping or looking at their return code, this is what bash is best at.  It’s also easily understood and edited by anyone who has worked with Linux for a while.

The bootloader is required for test in only one capacity, it is the only part of the system that is capable to run the SDRAM tests; once you enter Linux you can’t perform a full SDRAM test any more.  But even that should be done by the one shipping bootloader image.

In many cases, device interfaces can be tested by external loopback connectors, this proves connectivity through the connectors and it leaves open the possibility of end-users being able to run the same tests on the shipping rootfs.

On the first day of FOSDEM I sat through a presentation on what could be called another “U-Boot derivative”.  One of the greatest asspains at Openmoko was the various kinds of Hell caused by the U-Boot bootloader and its philosophy, which can be summed up as “I wanna be Linux when I grow up”.

Configure system is a bad alternative to good bootloader design

First, it has a config system.  That should be good though, right?  The problem with the config system is that if anything differs from your current config, you must build another incompatible binary with another config and take care of that.  When you have more than a handful of different boards, you are in a maze of incompatible bootloaders.  Openmoko took it one step further, they mandated a different bootloader binary per PCB revision, so left unchecked there would have been a continuous proliferation of incompatible bootloaders, all basically the same.

All persistent bootloader private state is EVIL

Second, U-Boot thinks it’s a good idea to have these environment “scripts”, because it’s “configurable”.  Actually, the job of a bootloader is to Load, then Boot Linux.  You don’t need any configurability for that if the bootloader can figure out what it’s running on and therefore where the memory is and how much there is.  These scripts expose a really deadly trap I call “private bootloader state”.  It means the bootloader stores stuff in nonvolatile memory on the PCB and acts different according to what it hides there.  The end result is that two boards from the same factory may act totally different even with the same rootfs due to “bootloader secrets”.  This is totally needless and ALL private bootloader state can be eliminated by correct design of the bootloader leading to completely deterministic boot action per rootfs.

A good example how that lead you to the path to hell is hardcoding in the U-Boot environment of the amount of kernel image you will copy from somewhere.  People commonly set it to 2MBytes, forget about it and one day they generate a 2.1MB kernel image and wonder why decompress blows up.  Actually, that whole procedure is insane, the kernels are uImages that report their length in a header.  The bootloader should examine the header and compute the length of image to pull.  But that doesn’t fit with this “environment” nonsense.

Do Linux Stuff In Linux

In any of these bloated U-Boot style bootloaders, is there even one feature they do better than the same feature in Linux?  The startup time should be better by a few 100ms.  Other than that, no, every single bloated “I will add it to the bootloader beacuse I can” feature is shittier than you get in Linux.  Every single feature!

If you need some advanced capability or backup / recovery boot action, check for a button held down at boot-time in the bootloader and go fetch a different Linux partition + kernel.  Use standard Linux tools and shells.  In return, get really high quality network stack, proper USB support, NAND access that’s compatible to your main Linux system access in BBT / ECC terms, and all the other advantages of Linux.

Do your peripheral bringup in drivers in Linux

Typically you do not need ANY bringup in the bootloader except SDRAM controller and chip init, since it’s a prerequisite to put Linux in the RAM that it’s initialized.

That’s right, all the megabytes of source spent in U-Boot providing support for so many kinds of peripheral is a waste of time, effort and maintenance.  I am being kind saying “maintenance”, because the drivers in U-Boot are typically “dumbed down” versions of the equivalent Linux driver that were forked irretrievably the moment all the Linux APIs were ripped, so there’s no coherent effort to keep them up to date with the Linux ones .  Lately I saw that they try to ape some Linux APIs there… why not go the whole hog and just load and boot real Linux?  After all, modern CPUs can be running your driver probes in Linux in ~2 seconds from power using a bootloader that doesn’t get in the way.

You typically don’t even need to talk to the PMU in the bootloader, after all, you are running code fine already, right?  Otherwise you wouldn’t be able to run the bootloader code itself.

Fat girl in Ibiza

At least at Openmoko, code quality inside U-Boot was awful bad.  I called U-Boot on the lists there “the fat girl in Ibiza” because you know she’s going to do anything you want.  All kinds of constant-only code, weird new scripting keywords were added for test undocumented, you name it.  Hardware guys felt up to writing such code secretly by themselves once they learned the software engineering marvel that is *((unsigned int *)0x…) = 0x…;

Your bootloader just tests SDRAM

There’s only one test action your bootloader is suited to do, and that is SDRAM test.  Once you are in Linux, it can’t perform a full SDRAM test while it’s running.  But the bootloader is typically starting from on-CPU SRAM, it can actually run a true SRAM test from there.  Otherwise, the bootloader should be completely absent from the test plan.  All other tests should be performed in Linux via standard driver and rootfs tools.

More about board and test and board bringup will feature in another report of a lesson learned.

Qi

While at Openmoko (mainly) I wrote a bootloader that meets these ideals, you can find it in git here One of the nicest things about it is that unlike the bloated bootloaders whose job never finishes trying to become Linux cargo cult style, Qi has been pretty much complete for a few months.  It’s a new job to support a new CPU, a much smaller job to add a new board and it doesn’t want to talk to your peripherals anyway so no problem there.

Qi creates one binary per CPU, that supports all boards with that CPU.  That sounds like a big job but we don’t care about your peripherals so all boards with the same CPU look almost identical.  You have to find something that can detect your particular board at runtime, for example NOR device ID read check.  So there is zero build-time config and Qi generates all CPU support when it’s buit, it takes 3 sec or so typically.

Typical bootloader binary size per CPU is 28-30KBytes.  That supports VFAT, ext2/3/4 typcially the SD controller as well.  The single Qi image also supports being booted from NAND, JTAG or SD Card on processors that support it just by being copied into place and without any changes.

There is zero bootloader private state, however Qi can look in the rootfs and append kernel commandline text from the content of a filesystem file.  This maintains the rule that boot should be completely deterministic per rootfs.

I was at Fosdem over the weekend, there were several interesting talks I attended but the most interesting one for me was a roundtable about the future of Cross distributions.  I was invited to give a 5 minute talk there which I gave, but unfortunately it was right at the end and the people before had overrun, so there was no time to make much of a coherent case.   So I am going to write some articles covering the issues involved here.

Cross as a niche

Cross itself remains absolutely necessary for systems below a certain level of horsepower.  For example, 8051, ARM7, cortex M3 are not really capable to consider native build.  But processors get faster each year, a lot of things we would have used an 8051 on use an ARM7 or cortex M3 now, in a few years it is likely that baseline has moved further up and it’s an ARM9 equivalent.  What I am suggesting then is that over time, the niche where you need cross is shrinking.

All four of the cross distros at FOSDEM target a CPU that’s powerful enough to run Linux, but not powerful enough to build its own binaries.  That is the niche that I believe will shrink to the point that it won’t support all these cross Linux distro projects, possibly none of them in the end.

My background with cross Linux

A few years ago I created an RPM-based cross distro singlehanded, and used it on a product for a customer  This was AT91RM9200-based, a 200MHz ARM9 with 32MBytes of SDRAM.  The amount of effort needed to create a set of cross packages sufficient to create a workable rootfs was huge, it took me many weeks.  Some packages like perl were just so cross-unfriendly that they were basically out of reach (although I later saw other people have done the invasive magic necessary).  It did work well, and I added patches for busybox RPM support that allowed it to do more useful things like erase and keep a package database.  The packaging was valuable in itself but a nice advantage was the source RPMs it generated ensuring GPL compliance.

My background with Openmoko

Subsequently I spent 14 months as (mainly) the kernel maintainer for Openmoko.  Openmoko had an OpenEmbedded basis for it’s rootfs, also a cross system.  I attempted to use it for “hello world” while I was at Openmoko, but it broke because I was on a newly released Fedora.  How it broke was very revealing, the official way to get started with it was to run a huge script that wgetted and locally built 1100 packages.  It died due to some assumption somewhere breaking while it tried to build host dbus libraries.

What I wanted was a cross toolchain that would let me package “hello world”.  What I got was a massive host build action including host dbus libs.  I have perfectly good host dbus libs in my Fedora install, I enquired about it and was told they were the “wrong” libs for the expectation of the rest of the packages, so they had to be rebuilt.

I gave up on trying to use OpenEmbedded, as I guess most of Openmoko’s customers did.

After Openmoko imploded, I designed the software architecture (and influenced the hardware design in some aspects) for the txtr reader device.  On this device, I put into action various lessons I had learned in how not to do things from Openmoko.  I will write further about the other lessons in future articles, but here’s the first one:

Lesson #1: Don’t compile your own rootfs

I was told by a manager at Openmoko that Openmoko had hired most of the main devs of OpenEmbedded and were paying for that accordingly.  This was a pretty big drain on their resources over a long period.

In contrast, nowadays you can head over to http://fedoraproject.org/wiki/Architectures/ARM and download a generic rootfs tarball of prebuilt binaries for ARMv5 and above[1].  It’s made from unpacking prebuilt binary packages.  Once you boot into it, you can install further packages with the usual yum install type action.  You can be up in a high quality rootfs in five minutes flat.

You do not need to go around compiling everything personally when binary packages exist from a reputable distro already.  Normal distros provide -dev and -devel packages for you to link against too, so you do not need to recompile the universe just because you want to build “hello world” either.  That’s how we do things on desktop and server systems, as the processors involved get stronger embedded does not have to be different.

If you want to cross-build specific packages, you install the Fedora ARM Cross Toolchain RPMs on you host via yum and you are ready to go in a couple of minutes.  This is very useful for cooking the kernel on your host both to get started and during development; you can’t native-build the bootstrap stuff needed to boot your platform.  But that’s just a cross compiler and related pieces, it’s not a cross distro.  (The guy from emdebian at this FOSDEM talk also made this point that you do not need to get into making your own toolchain, your distro should have one you can just install).

Fedora ARM’s strategy is native build.  So you install gcc and other dependencies into the actual device, and use standard rpmbuild to build your package there; you can also just configure ; make ; make install for development too down there.  If something’s missing on the rootfs you can yum install it.

(1 To make the comparison fair to openmoko Fedora ARM came along too late for them to choose it from the start, and the GTA02 s3c2442 was not a v5 class processor, they would have been into a distro recook after changing the distro-level compile options.  However my worry is not repeating Openmoko’s errors and today Fedora ARM is available.)

Quality and Quantity

Another major issue is distro quality.  I was so surprised to hear at Fosdem Dr Mickey Lauer of OpenEmbedded boast about the number of devices that managed to use that distro (including the sad shape of the GTA02) and say that unlike the other cross distros, OpenEmbedded focused on “Quantity not Quality”.  From my experience I think he’s right alright about not focusing on quality, and he did go on to explain there are problems with OpenEmbedded they are trying to address.

In the near future, there will be a carcrash between these difficult cross distros that have relatively poor quality and strange requirements to use them and standard, “proper distros” like Fedora ARM, because on higher-end ARMv5s say 400MHz and above, it is already perfectly possible to compare the two worlds on the same device.  I think many devs currently are trained by their experience with buildroot type systems to assume they have to personally build everything Gentoo style.  However as CPUs increase in power at the same price point, the ways of working with these systems efficiently change, and desktop / server “treat it like a PC” lessons like the value of packaging start to really show their traditional advantages over rootfs tarballs.

Like Debian, Fedora has all kinds of rules and requirements about packaging to ensure high quality, there are a huge number of users of these two normal distributions that leads to tested and debugged basic packages and their dependencies.  OpenEmbedded’s boast about number of users is not even a blip in comparison to Fedora or Debian’s consumers and contributors.

Cross distros are locked into local patch hell

A worse problem against their quality even than not many users is the patch load these projects are carrying, I think all of the cross distro projects bemoaned that they were carrying huge patchsets across a large number of packages to get them to build cross at all, and that most upstreams did not care to take them (I assume they don’t want to have to get into testing them).  To uplevel packages, which distros have to daily when they have a large package universe, it can become a nightmare of breakage because of the private patchsets being dragged around.

(BTW I also saw in another presentation that the limo foundation are carrying around more than 80MBytes of diff between their distro and the upstream projects, and these are the guys who sent out a whitepaper explaining the massive cost of delaying sending patches upstream in dollar terms.)

There was proposed a unified crossbuild patch promoting effort, but the effort seemed only to consist of a domain like “sends-patches.org” that you could use when sending patches instead of your own project name, which seems to just be tea and sympathy rather than a solution.

It’s clear that quality will tend to be higher if you are getting packages built with normal distro specfiles and no pile of local patches to get them to build cross (because they were built native).  Combined with higher quality thresholds at the project level and sheer number of users, native Fedora (or Debian) rootfs basis will provide Quantity and Quality if your processor is appropriate.

A couple of hours after the talk I had an interesting conversation with OpenInkpot dev Mikhail Gusarov, who I found also shared my lack of enthusiasm for OpenEmbedded, although he is trapped still in the cross niche generally by the weak processors he targets at the moment.

[update Feb 10 09:00] Mikhail has written his own response, he still likes the speed of cross (and still hates OpenEmbedded).  But there’s some confusion about what Fedora ARM offers, it’s a generic ARMv5 rootfs, it doesn’t care what exact kind of CPU, vendor or peripherals available.  Build farms are less of a requirement when you are no longer building your rootfs but installing it from distro binary packages.  Sheevaplug makes available a 1.2GHz Marvell ARM compatible with 512MBytes of SDRAM that Fedora ARM can work on if you need a native build machine.  Shortly fast dual processor Cortex A9 machines will become available.

I ran the last 10MByte sample against ENT and TestU01… to cut a long story short

$ ./ent ../die.c/dump3
Entropy = 7.999980 bits per byte.

Optimum compression would reduce the size
of this 10002432 byte file by 0 percent.

Chi square distribution for 10002432 samples is 281.26, and randomly
would exceed this value 25.00 percent of the times.

Arithmetic mean value of data bytes is 127.4958 (127.5 = random).
Monte Carlo value for Pi is 3.140111525 (error 0.05 percent).
Serial correlation coefficient is -0.000212 (totally uncorrelated = 0.0).

7.9999 bits of entropy per byte! TestU01 is less turnkey than the other suites — it’s literally a test library with some example code. I amended an example to call the FIPS-140-2 tests:

============== Summary results of FIPS-140-2 ==============

 File:             dump3
 Number of bits:   20000

       Test          s-value        p-value    FIPS Decision
 --------------------------------------------------------
 Monobit               9933           0.83       Pass
 Poker                11.88           0.69       Pass

 0 Runs, length 1:     2482                      Pass
 0 Runs, length 2:     1227                      Pass
 0 Runs, length 3:      630                      Pass
 0 Runs, length 4:      319                      Pass
 0 Runs, length 5:      161                      Pass
 0 Runs, length 6+:     166                      Pass

 1 Runs, length 1:     2466                      Pass
 1 Runs, length 2:     1302                      Pass
 1 Runs, length 3:      620                      Pass
 1 Runs, length 4:      311                      Pass
 1 Runs, length 5:      140                      Pass
 1 Runs, length 6+:     146                      Pass

 Longest run of 0:       16           0.14       Pass
 Longest run of 1:       14           0.46       Pass
 ----------------------------------------------------------
 All values are within the required intervals of FIPS-140-2

So the design’s output is compliant to FIPS-140-2, a requirement for many uses.

Here is a patch against 2.6.20 with the AT91RM9200 patches: it should probably apply okay to later kernels. The patch adds many comments so you probably want to read that and this at the same time. First the things you can’t do, which will save you much pain from finding them out yourself.

Things you can’t do with FIQ

One of the “private times” Linux needs to itself is the virtual memory pagetable management action. It shuts off all interrupts and rewrites the pagetable at intervals, and then goes on as before. That stops driver interrupts coming in and trying to do stuff while the pagetable is empty, or incomplete or just full of garbage.

However FIQ ignores any claim to privacy performed by shutting off normal interrupt response. That means your FIQ ISR code can come in at a “bad time”, if it tries to access memory areas mapped through the pagetable it will instead find nothing or the wrong thing or… the end result is that the FIQ ISR cannot touch any memory mapped by vmalloc.

Unfortunately, when a kernel module is loaded, its various memory footprints including the module code are allocated by… yep, vmalloc. That means your FIQ ISR code cannot live in a module, it has to be part of the monolithic kernel.

Finally all sorts of Linux code also wants “private time” or to guard against multiple access to objects by spinlocks or whatever. FIQ ISRs cannot play those games, it comes it in the middle of anything and has to get out quickly again too. So unless it is a simple macro, you can’t use any Linux APIs in the FIQ ISR.

Things you can do with FIQ

Well reading those constraints, you’re probably wondering if it is still useful. It sure is!

You can touch the memory-mapped IO in the chip using the AT91RM9200 APIs.

FIQ has the super power it will run your ISR within ~1us NO EXCUSES. That means you can rely on the ISR code to act like hardware, you trigger it and 1us later your programmed sequence occurs without fail. In turn that means FIQ is perfect for many hardware interfacing tasks, in particular management of low latency (small buffer) PDC DMA setup.

Low latency for audio traffic for example is highly desirable, but of course if there are ANY delays setting up the next PDC DMA, you get dropouts and clicks. If you allow the FIQ to handle generation of samples and management of PDC DMA, there won’t be any delays for sure, you will have perfect audio.

We found that the AT91RM9200 at 180MHz can easily handle 8kHz FIQs (a common rate for telephony) with an ISR duration of ~8us, without affecting the Ethernet or USB performance.

IPC between the FIQ and kernel worlds

The general communication for FIQ ISRs with the “real world” in the patch is to define a struct in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h that contains all of the data that is shared between FIQ and normal kernel code. The example one looks like this:

struct at91rm9200_fiq_ipc {
	int nCountFiqEvents;
};

One of these structs is defined in the main part of the patch code in arch/arm/mach-at91rm9200/at91_fiq.c like this

struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;
EXPORT_SYMBOL(at91rm9200_fiq_ipc);

That means in your other kernel code — which can be in a module, only the FIQ ISR must be in the kernel — you can have

#include <arch/at91rm9200_fiq_ipc_type.h>
extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;

and use the same struct to communicate with the FIQ ISR.

How to customize summary

1) Change struct at91rm9200_fiq_ipc in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h to have the data types you need

2) Add your FIQ ISR code to arch/arm/mach-at91rm9200/at91_fiq.c where it says “your C code goes here”

3) Import extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc; in your own kernel module and communicate with the FIQ ISR using that.

FIQ shadowing with IRQ

Ultimately the FIQ actions are going to need to interface to Linux kernel objects sooner or later, perhaps there has to be some locking or blocking action for usermode access. But we are banned from using Linux APIs in the FIQ ISR.

A powerful solution is to physically tie the FIQ signal to an IRQ input additionally. Code that is REALLY hard realtime, like the PDC DMA management, goes in FIQ, and a count of FIQs is kept. Data out of FIQ can go into a software FIFO. The less reliable IRQ watches the count of FIQs and compares it to its own count of IRQs, if it sees it has blacked out and has fallen behind, it will loop up to a certain number of times “catching up”, using the data placed by FIQ in the FIFO.

In this way, by splitting the code into “no excuses” realtime in the FIQ ISR, and “reliable only on average” realtime in the IRQ ISR, it is possible to bind actions in the FIQ to code in the ISR which can execute Linux APIs, and have the best of both worlds.

I use our Octotux Linux distro with Postfix as the MTA, gps for the greylisting and Dovecot IMAP to provide secure access to the mailstore over SSL. This has worked out really well, the warmcat.com MX record points directly to the external IP here, and the firewall box port-forwards port 25 to the embedded device. It’s silent and runs cold 24 hours a day and has never missed a beat.

A couple of weeks ago I had to look at the box again because the greylisting software was hanging. I discovered that we were being bombarded with spam, one new spam every two seconds on average, from all over the world. I adjusted the ordering of the filtering in postfix to first reject on an unknown username, that stopped so many concurrent gps sessions being needed. The server weathered that storm and the spam people gave up a few days later without getting a single one through. (They were also targeting the warmcat.com A record IP, I suppose in case it was a backup for the real MX, but they had zero luck with that either).

However it reminded me of the one inadequacy of this mailserver… when you wake up your laptop in the morning, thunderbird takes ages to run all the filters and move the new mails remotely into the right IMAP folders. That’s pretty annoying when you see the titles of mails you want to read but the USB stick on the server is maxxed out for a couple of minutes sorting eight hours worth of new emails into folders on the server. I have been pondering changing the box to one with USB2 High Speed, but it occurred to me that otherwise, the existing USB 1.1 “Full Speed”, that is, 12Mbps is completely adequate. Changing folders and moving to other emails in thunderbird is snappy. It’s just the client mail filters under the load of 500 mails in the morning. So I decided to port procmail to ARM9 Octotux, in effect to do the folder sorting as each mail came in, so there would no longer be any processing done at the client for that.

Procmail was a little bit of a beast, the sources look horrible and it uses a nonstandard autotools script, which of course does not support crosscompile. After a couple of hours I got it to build nicely in an Octotux RPM (Octotux is entirely crosscompiled RPM-based). Not having used it before, I was surprised to see there was no configuration down /etc in the package, just four small binaries in /usr/bin and some manuals (which I broke out into procmail-docs package so as not to waste space on the target). The binary package only came to 60K all told.

The first move was to direct postfix to deliver not to the maildir as before but through procmail. This simply involved editing /etc/postfix/main.cf and adding this

mailbox_command = /usr/bin/procmail

After some googling I found that procmail is going to look at the unix user’s homedir that the mail is directed to, for a file ~/.procmailrc in order to find out what it should do. Now my embdedded mailserver is setup to use Dovecot IMAP with the mailstore symlinked to the USB stick. So my user “andy” on the embedded mailserver has this in the home dir

# ll /home/andy
lrwxrwxrwx 1 root root 25 Jan 17 2007 Maildir -> /media/usbstick/mail/andy

I found looking around in the Dovecot Maildir structure in there that the mail needs to be delivered into /home/andy/Maildir/.(foldername)/new/ in order to get Dovecot to understand that it was given a new mail in a folder. The resulting /home/andy/.procmailrc looks like this:

SHELL=/bin/sh
PATH="/usr/bin:/usr/local/bin:/bin:/sbin:/usr/sbin"
LOCKFILE=/home/andy/lockfile.loc
DEFAULT=/home/andy/Maildir/
BITBUCKET=/dev/null
LOCKTIMEOUT=10
LOGFILE=/tmp/procmail_log
LOGABSTRACT=no
VERBOSE=no

:0:
* ^List-Id: Development discussions related to Fedora
$DEFAULT/.fedora-devel-list/new

You simply repeat the last stanzas with some unique identifying header part of mails that are to be filed in a given folder, and give the destination folder name preceded by a ‘.’ and followed by /new as shown. You don’t need to run anything or restart anything after making changes here, they will be immediately used on the next email that is delivered. If your folder lives in a hierarchy, the folder on disk looks like, for example, .INBOX.CentOS for a folder CentOS that appears as a child of INBOX. The stanza for that would look like

:0:
* ^List-Id: CentOS mailing list
$DEFAULT/.INBOX.CentOS/new

You can also direct procmail to use a central procmailrc file, presumably in /etc, by giving the path to it on the procmail invocation line in /etc/postfix/main.cf. In my case I only take mails for myself, so I stuck with the per-user ~/.procmailrc.

No changes were needed anywhere else in the server setup, all I had to do was turn off all my thunderbird filters and enjoy immediate access to the emails via Dovecot IMAP in the morning, without upgrading any hardware.

Assuming nothing bad happens in the next few weeks leading to their being reverted (unlikely I would think, since they don’t interfere with much existing code), then a standardized driver-independent ieee80211 packet injection methodology will soon be available by default in all major distro kernels. Currently if you want to perform packet injection, you enter into a dark underworld of individual driver patching, having to cook custom kernels and make animal sacrifices to forgotten Gods. But now with the injection patches, for the devices with mac80211 drivers all that crap is blown away and every 2.6.23 kernel will offer the capability built-in.

Here are the list of mac80211 drivers and whether I have actually seen good injection. All of the drivers are expected to work, but I don’t have all the hardware.

I have also provided the Packetspammer commandline applet to show how to use the injection API from userspace, this provides a simple, GPL2 tested base for making your own injection code for your own apps.

Work started on it in Dec 2006 by providing the old Linux stack driver patches. The real reason for the sustained effort is to enable Penumbra to work “out of of the box” on not only Linux desktops and laptops but generic embedded devices as well.

But one of the biggest hurdles in creating the patch was not technical, since I already had the core functionality working, but in fact the Linux kernel coding style. In some ways the coding style fits well with my own personal style (formed over 20 years of writing C and C++), we basically use the same K&R style. There are some spacing and commenting rules that are actually better than my style and I will adopt them wholesale. But that’s where the fun stops and the recrimination begins!

The basic problem is the combination of three rules which has a terrible effect on eliminating what I consider good coding practice due to the constraints introduced by those rules.

• Tabs are 8 chars NO EXCEPTIONS
• Lines are less than 80 cols NO EXCEPTIONS
• Everything in { } is indented by a tab (except switch cases!)

Now almost everything is inside a function body, so that gets you down to 72 chars already. And if your function is doing something non-trivial, your code is probably inside a while() for a for() and there are one or more levels of if() to decide to do it or not. Pretty soon you are writing code crushed up against the 80-col limit with only 30 chars that are usable and 50 spaces behind it. It strongly puts me in mind of the Bonsai Kittens fake website that showed how to push kittens into bottles so they would grow into the shape of the bottle.

Under these abnormal circumstances, certain things become very difficult to do:

• \t\t\t\t\t”any kind of long ” \n\t\t\t\t\t “string has to be ” \n\t\t\t\t\t “artificially chopped up”
• nested if()s may make perfect sense to explain your code logic. But you can no longer afford them because of the tab each one adds. So you have to invert the if() sense and use a goto (I kid you not, this is preferred due to the coding style rules)
• I strongly prefer descriptive variable names which include type. Type is part of the information you need to understand what that variable is when looking at it. “nCountWaysILoveHer” tells me (now, and in 6 months when I have forgotten the code) it is an int that is counting a specific thing. “i” or “cnt” could be anything, although “cnt” is better. But you can’t afford a long variable name with the rules above, you can get into a situation where there is not even enough room left after the tabbing to hold just the variable name on a single line.

On that last point there is some handwaving nonsense in the coding style doc for Linux that “C programmers don’t use long variable names”… well I call bullshit on that one. The truth is that because of the other tabbing and length rules, Kernel C programmers can’t use long variable names even if they realized it was better: they ran out of room for it.

To be fair to the coding style doc it does have a point when talking about what to do when the indents get too much: it suggests to break the indented content out into a new function and to call through to it. It also says that massively indented code means you were screwed anyway, because the logic was too complicated, and that can also be true. But calling through to functions can be a very bad fit if the code you are migrating out touches many variables defined at the parent function top level.

I am still working through the style rules trying to see what I should take on board to replace my own style and what I have to “fake” just for kernel code, but it seems to me life would be better for everyone if they relaxed the line length to 120 chars instead of 80.

Makes no sense to have a license to Mordor for each of the hundreds of plasma displays at Luton Airport. It sparked a conversation with my stepson about why it was chosen: presumably because the devs didn’t know anything else than Windows, they will spend the rest of their working life acting as agents for Microsoft. This is the fruit of the Jesuit priest Ballmer’s creed of “give me a developer until he has 7 months in the industry and I will give you a Microsoft Trained Monkey”.

Using the same or cheaper hardware, an embedded Linux implementation would have had at least the same performance and much more flexibility. Ogg Matroska is a patent-free high quality video solution that plays back out of the box on Fedora for example. And if you wanted a hundred or a thousand displays your software licensing costs would stay at exactly $0.