Way of the exploding head

One of the coolest features of the AT91RM9200 we have been designing with for a couple of years now is the FIQ, or Fast Interrupt Request. This is basically the NMI of the ARM world. It is a bit difficult to get working (Milosch Meriac helped me with the initial version some years ago) because its job is to interrupt WHATEVER is happening and start running your FIQ handler within 1us or so NO EXCUSES. This is the very very hard end of hard realtime, it does what it claims but it does not respect any privacy that Linux may need as an OS and we will see that needs care.

Here is a patch against 2.6.20 with the AT91RM9200 patches: it should probably apply okay to later kernels. The patch adds many comments so you probably want to read that and this at the same time. First the things you can’t do, which will save you much pain from finding them out yourself.

Things you can’t do with FIQ

One of the “private times” Linux needs to itself is the virtual memory pagetable management action. It shuts off all interrupts and rewrites the pagetable at intervals, and then goes on as before. That stops driver interrupts coming in and trying to do stuff while the pagetable is empty, or incomplete or just full of garbage.

However FIQ ignores any claim to privacy performed by shutting off normal interrupt response. That means your FIQ ISR code can come in at a “bad time”, if it tries to access memory areas mapped through the pagetable it will instead find nothing or the wrong thing or… the end result is that the FIQ ISR cannot touch any memory mapped by vmalloc.

Unfortunately, when a kernel module is loaded, its various memory footprints including the module code are allocated by… yep, vmalloc. That means your FIQ ISR code cannot live in a module, it has to be part of the monolithic kernel.

Finally all sorts of Linux code also wants “private time” or to guard against multiple access to objects by spinlocks or whatever. FIQ ISRs cannot play those games, it comes it in the middle of anything and has to get out quickly again too. So unless it is a simple macro, you can’t use any Linux APIs in the FIQ ISR.

Things you can do with FIQ

Well reading those constraints, you’re probably wondering if it is still useful. It sure is!

You can touch the memory-mapped IO in the chip using the AT91RM9200 APIs.

FIQ has the super power it will run your ISR within ~1us NO EXCUSES. That means you can rely on the ISR code to act like hardware, you trigger it and 1us later your programmed sequence occurs without fail. In turn that means FIQ is perfect for many hardware interfacing tasks, in particular management of low latency (small buffer) PDC DMA setup.

Low latency for audio traffic for example is highly desirable, but of course if there are ANY delays setting up the next PDC DMA, you get dropouts and clicks. If you allow the FIQ to handle generation of samples and management of PDC DMA, there won’t be any delays for sure, you will have perfect audio.

We found that the AT91RM9200 at 180MHz can easily handle 8kHz FIQs (a common rate for telephony) with an ISR duration of ~8us, without affecting the Ethernet or USB performance.

IPC between the FIQ and kernel worlds

The general communication for FIQ ISRs with the “real world” in the patch is to define a struct in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h that contains all of the data that is shared between FIQ and normal kernel code. The example one looks like this:

struct at91rm9200_fiq_ipc {
	int nCountFiqEvents;
};

One of these structs is defined in the main part of the patch code in arch/arm/mach-at91rm9200/at91_fiq.c like this

struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;
EXPORT_SYMBOL(at91rm9200_fiq_ipc);

That means in your other kernel code — which can be in a module, only the FIQ ISR must be in the kernel — you can have

#include <arch/at91rm9200_fiq_ipc_type.h>
extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc;

and use the same struct to communicate with the FIQ ISR.

How to customize summary

1) Change struct at91rm9200_fiq_ipc in include/asm-arm/arch-at91rm9200/at91rm9200_fiq_ipc_type.h to have the data types you need

2) Add your FIQ ISR code to arch/arm/mach-at91rm9200/at91_fiq.c where it says “your C code goes here”

3) Import extern struct at91rm9200_fiq_ipc at91rm9200_fiq_ipc; in your own kernel module and communicate with the FIQ ISR using that.

FIQ shadowing with IRQ

Ultimately the FIQ actions are going to need to interface to Linux kernel objects sooner or later, perhaps there has to be some locking or blocking action for usermode access. But we are banned from using Linux APIs in the FIQ ISR.

A powerful solution is to physically tie the FIQ signal to an IRQ input additionally. Code that is REALLY hard realtime, like the PDC DMA management, goes in FIQ, and a count of FIQs is kept. Data out of FIQ can go into a software FIFO. The less reliable IRQ watches the count of FIQs and compares it to its own count of IRQs, if it sees it has blacked out and has fallen behind, it will loop up to a certain number of times “catching up”, using the data placed by FIQ in the FIFO.

In this way, by splitting the code into “no excuses” realtime in the FIQ ISR, and “reliable only on average” realtime in the IRQ ISR, it is possible to bind actions in the FIQ to code in the ISR which can execute Linux APIs, and have the best of both worlds.

For over a year I have been using an 32MB ARM9-based board I designed with a 1GB USB stick as my mailserver. It is powered from a USB port on my firewall box and takes around 1W.

I use our Octotux Linux distro with Postfix as the MTA, gps for the greylisting and Dovecot IMAP to provide secure access to the mailstore over SSL. This has worked out really well, the warmcat.com MX record points directly to the external IP here, and the firewall box port-forwards port 25 to the embedded device. It’s silent and runs cold 24 hours a day and has never missed a beat.

A couple of weeks ago I had to look at the box again because the greylisting software was hanging. I discovered that we were being bombarded with spam, one new spam every two seconds on average, from all over the world. I adjusted the ordering of the filtering in postfix to first reject on an unknown username, that stopped so many concurrent gps sessions being needed. The server weathered that storm and the spam people gave up a few days later without getting a single one through. (They were also targeting the warmcat.com A record IP, I suppose in case it was a backup for the real MX, but they had zero luck with that either).

However it reminded me of the one inadequacy of this mailserver… when you wake up your laptop in the morning, thunderbird takes ages to run all the filters and move the new mails remotely into the right IMAP folders. That’s pretty annoying when you see the titles of mails you want to read but the USB stick on the server is maxxed out for a couple of minutes sorting eight hours worth of new emails into folders on the server. I have been pondering changing the box to one with USB2 High Speed, but it occurred to me that otherwise, the existing USB 1.1 “Full Speed”, that is, 12Mbps is completely adequate. Changing folders and moving to other emails in thunderbird is snappy. It’s just the client mail filters under the load of 500 mails in the morning. So I decided to port procmail to ARM9 Octotux, in effect to do the folder sorting as each mail came in, so there would no longer be any processing done at the client for that.

(more…)

After getting on for four months, my mac80211 injection patches have been accepted by the powers-that-be and have made their way into the Linus git repo, the crucible from which vanilla kernel versions are forged, and the upstream from which all major distros are ultimately basing their kernels on. (Edit: they are present in 2.6.23-rc1 also).

Assuming nothing bad happens in the next few weeks leading to their being reverted (unlikely I would think, since they don’t interfere with much existing code), then a standardized driver-independent ieee80211 packet injection methodology will soon be available by default in all major distro kernels. Currently if you want to perform packet injection, you enter into a dark underworld of individual driver patching, having to cook custom kernels and make animal sacrifices to forgotten Gods. But now with the injection patches, for the devices with mac80211 drivers all that crap is blown away and every 2.6.23 kernel will offer the capability built-in.

Here are the list of mac80211 drivers and whether I have actually seen good injection. All of the drivers are expected to work, but I don’t have all the hardware.

I have also provided the Packetspammer commandline applet to show how to use the injection API from userspace, this provides a simple, GPL2 tested base for making your own injection code for your own apps.

Work started on it in Dec 2006 by providing the old Linux stack driver patches. The real reason for the sustained effort is to enable Penumbra to work “out of of the box” on not only Linux desktops and laptops but generic embedded devices as well.

The last few months I have been working on the Penumbra project. I started off patching wireless drivers in and out of the kernel tree to achieve the anonymous broadcast action that the project needs, but it became clear this would be completely unworkable for general use… getting wireless up in Linux can still be a struggle and hoping people will patch their driver or kernel in addition isn’t going to happen. After trying a couple of other methods in the end I created a radiotap-based packet injection patch for the mac80211 stack (formerly dscape / d80211), and bound it together with a patch from Michael Wu that provides radiotap-based Monitor mode. At the moment it is still in front of the linux-wireless folks and it’s not clear what the result will be. If the patch is accepted, then the code should make it into the mainline kernel and all mac80211-based wireless drivers will work with Penumbra out of the box in the future. The patchset provides generic radiotap monitoring injection that “just works” with libpcap both ways, so I am hoping it will get accepted without people having to form an opinion about Penumbra.

But one of the biggest hurdles in creating the patch was not technical, since I already had the core functionality working, but in fact the Linux kernel coding style. In some ways the coding style fits well with my own personal style (formed over 20 years of writing C and C++), we basically use the same K&R style. There are some spacing and commenting rules that are actually better than my style and I will adopt them wholesale. But that’s where the fun stops and the recrimination begins!

(more…)

Makes no sense to have a license to Mordor for each of the hundreds of plasma displays at Luton Airport. It sparked a conversation with my stepson about why it was chosen: presumably because the devs didn’t know anything else than Windows, they will spend the rest of their working life acting as agents for Microsoft. This is the fruit of the Jesuit priest Ballmer’s creed of “give me a developer until he has 7 months in the industry and I will give you a Microsoft Trained Monkey”.

Using the same or cheaper hardware, an embedded Linux implementation would have had at least the same performance and much more flexibility. Ogg Matroska is a patent-free high quality video solution that plays back out of the box on Fedora for example. And if you wanted a hundred or a thousand displays your software licensing costs would stay at exactly $0.