Commonly in embedded work test is the “red-haired stepchild”, nobody wants to take care of it and by common, silent consent it is always left until last.  Eventually the need for a test plan becomes overwhelming as the date to go to the factory nears, and the task is assigned to the most junior engineers available, since everybody knows that test is the death knell of your career.

Coming cold to and excluded from being inside an already-existing project, the engineers try to create some kind of test coverage the best way they can.  At openmoko two giant test suites were created, DM1 and DM2, written by people who were learning C for the first time.  I got the job of modernizing this code so I know from experience the code was already truly terrible and bitrotted at an alarming rate.  However I had to admire the guys who wrote it, with everything against them and little experience they did manage to create something that did provide test coverage at the factory, however much it was on life-support.

Totentanz

Similarly, Openmoko used production test jigs, special additional PCBs that formed a kind of custom test environment for the PCB under test.  At one version of GTA03 there were so many test points added it was a serious concern that the board would break down under the overall pressure needed to mate the spring-loaded test probes to the test points.

Jigs and test points have an obvious advantage in terms of test throughput, but there are some big disadvantages.

First, you have to design and build the jig, and track changes to the actual device with it.  This effort is completely disconnected from moving your actual product on, except that it’s meant to help in production.

Second, test points don’t test your connectors; the test point may be connected OK but not the connector pin the user actually accesses.

Third, you need something else outside the device to assess what is happening on the test points, the code for that also has to be written and maintained against changes in the actual product.  It also means that it’s not possible for the tests to be casually performed outside the factory, or maybe by the original engineers if they have access to the ATE gear themselves.

Pain into torture

Additionally the bringup of GTA02 required special versions of U-Boot and kernel which had added “test magic” created by the test guys and unknown to anyone else.  These versions were seldom uplevelled.

Since GTA02 had raw NAND, it needed filling up at the factory with the rootfs.  The way to do this was via a very fragile OpenOCD using a custom USB – serial based device that was bitbanged.  It only worked with certain versions of the usb library needed to talk to it.

All of these quirks and requirements at the factory made production runs difficult and expensive to get right.

I only hurt you because I love you

I spent a lot of time thinking about how to avoid this end result next time I would design something.  The mistakes started in having anything special for test I concluded.  The jig: special, and so evil.  Test kernels or bootloader: special -> evil.  Test rootfs -> Evil.  test software, like Openmoko’s DM1 and DM2, evil.  The device should naturally be able to test itself with the arrangements that already exist inside it to operate at all.

The answer to the problem of “production test” is to completely subsume it into the rest of the design.  So it is the responsibility of Linux drivers to provide enough functionality by probe errors, or sysfs features, that one can perform test and diagnosis.  The “test suite” should boil down to a bash script that is using features exposed in a normal shipping rootfs and kernel.  Bash is ideal because most of the test action will be calling existing commandline tools like ifconfig, ping, l2ping and grepping or looking at their return code, this is what bash is best at.  It’s also easily understood and edited by anyone who has worked with Linux for a while.

The bootloader is required for test in only one capacity, it is the only part of the system that is capable to run the SDRAM tests; once you enter Linux you can’t perform a full SDRAM test any more.  But even that should be done by the one shipping bootloader image.

In many cases, device interfaces can be tested by external loopback connectors, this proves connectivity through the connectors and it leaves open the possibility of end-users being able to run the same tests on the shipping rootfs.

On the first day of FOSDEM I sat through a presentation on what could be called another “U-Boot derivative”.  One of the greatest asspains at Openmoko was the various kinds of Hell caused by the U-Boot bootloader and its philosophy, which can be summed up as “I wanna be Linux when I grow up”.

Configure system is a bad alternative to good bootloader design

First, it has a config system.  That should be good though, right?  The problem with the config system is that if anything differs from your current config, you must build another incompatible binary with another config and take care of that.  When you have more than a handful of different boards, you are in a maze of incompatible bootloaders.  Openmoko took it one step further, they mandated a different bootloader binary per PCB revision, so left unchecked there would have been a continuous proliferation of incompatible bootloaders, all basically the same.

All persistent bootloader private state is EVIL

Second, U-Boot thinks it’s a good idea to have these environment “scripts”, because it’s “configurable”.  Actually, the job of a bootloader is to Load, then Boot Linux.  You don’t need any configurability for that if the bootloader can figure out what it’s running on and therefore where the memory is and how much there is.  These scripts expose a really deadly trap I call “private bootloader state”.  It means the bootloader stores stuff in nonvolatile memory on the PCB and acts different according to what it hides there.  The end result is that two boards from the same factory may act totally different even with the same rootfs due to “bootloader secrets”.  This is totally needless and ALL private bootloader state can be eliminated by correct design of the bootloader leading to completely deterministic boot action per rootfs.

A good example how that lead you to the path to hell is hardcoding in the U-Boot environment of the amount of kernel image you will copy from somewhere.  People commonly set it to 2MBytes, forget about it and one day they generate a 2.1MB kernel image and wonder why decompress blows up.  Actually, that whole procedure is insane, the kernels are uImages that report their length in a header.  The bootloader should examine the header and compute the length of image to pull.  But that doesn’t fit with this “environment” nonsense.

Do Linux Stuff In Linux

In any of these bloated U-Boot style bootloaders, is there even one feature they do better than the same feature in Linux?  The startup time should be better by a few 100ms.  Other than that, no, every single bloated “I will add it to the bootloader beacuse I can” feature is shittier than you get in Linux.  Every single feature!

If you need some advanced capability or backup / recovery boot action, check for a button held down at boot-time in the bootloader and go fetch a different Linux partition + kernel.  Use standard Linux tools and shells.  In return, get really high quality network stack, proper USB support, NAND access that’s compatible to your main Linux system access in BBT / ECC terms, and all the other advantages of Linux.

Do your peripheral bringup in drivers in Linux

Typically you do not need ANY bringup in the bootloader except SDRAM controller and chip init, since it’s a prerequisite to put Linux in the RAM that it’s initialized.

That’s right, all the megabytes of source spent in U-Boot providing support for so many kinds of peripheral is a waste of time, effort and maintenance.  I am being kind saying “maintenance”, because the drivers in U-Boot are typically “dumbed down” versions of the equivalent Linux driver that were forked irretrievably the moment all the Linux APIs were ripped, so there’s no coherent effort to keep them up to date with the Linux ones .  Lately I saw that they try to ape some Linux APIs there… why not go the whole hog and just load and boot real Linux?  After all, modern CPUs can be running your driver probes in Linux in ~2 seconds from power using a bootloader that doesn’t get in the way.

You typically don’t even need to talk to the PMU in the bootloader, after all, you are running code fine already, right?  Otherwise you wouldn’t be able to run the bootloader code itself.

Fat girl in Ibiza

At least at Openmoko, code quality inside U-Boot was awful bad.  I called U-Boot on the lists there “the fat girl in Ibiza” because you know she’s going to do anything you want.  All kinds of constant-only code, weird new scripting keywords were added for test undocumented, you name it.  Hardware guys felt up to writing such code secretly by themselves once they learned the software engineering marvel that is *((unsigned int *)0x…) = 0x…;

Your bootloader just tests SDRAM

There’s only one test action your bootloader is suited to do, and that is SDRAM test.  Once you are in Linux, it can’t perform a full SDRAM test while it’s running.  But the bootloader is typically starting from on-CPU SRAM, it can actually run a true SRAM test from there.  Otherwise, the bootloader should be completely absent from the test plan.  All other tests should be performed in Linux via standard driver and rootfs tools.

More about board and test and board bringup will feature in another report of a lesson learned.

Qi

While at Openmoko (mainly) I wrote a bootloader that meets these ideals, you can find it in git here One of the nicest things about it is that unlike the bloated bootloaders whose job never finishes trying to become Linux cargo cult style, Qi has been pretty much complete for a few months.  It’s a new job to support a new CPU, a much smaller job to add a new board and it doesn’t want to talk to your peripherals anyway so no problem there.

Qi creates one binary per CPU, that supports all boards with that CPU.  That sounds like a big job but we don’t care about your peripherals so all boards with the same CPU look almost identical.  You have to find something that can detect your particular board at runtime, for example NOR device ID read check.  So there is zero build-time config and Qi generates all CPU support when it’s buit, it takes 3 sec or so typically.

Typical bootloader binary size per CPU is 28-30KBytes.  That supports VFAT, ext2/3/4 typcially the SD controller as well.  The single Qi image also supports being booted from NAND, JTAG or SD Card on processors that support it just by being copied into place and without any changes.

There is zero bootloader private state, however Qi can look in the rootfs and append kernel commandline text from the content of a filesystem file.  This maintains the rule that boot should be completely deterministic per rootfs.

I was at Fosdem over the weekend, there were several interesting talks I attended but the most interesting one for me was a roundtable about the future of Cross distributions.  I was invited to give a 5 minute talk there which I gave, but unfortunately it was right at the end and the people before had overrun, so there was no time to make much of a coherent case.   So I am going to write some articles covering the issues involved here.

Cross as a niche

Cross itself remains absolutely necessary for systems below a certain level of horsepower.  For example, 8051, ARM7, cortex M3 are not really capable to consider native build.  But processors get faster each year, a lot of things we would have used an 8051 on use an ARM7 or cortex M3 now, in a few years it is likely that baseline has moved further up and it’s an ARM9 equivalent.  What I am suggesting then is that over time, the niche where you need cross is shrinking.

All four of the cross distros at FOSDEM target a CPU that’s powerful enough to run Linux, but not powerful enough to build its own binaries.  That is the niche that I believe will shrink to the point that it won’t support all these cross Linux distro projects, possibly none of them in the end.

My background with cross Linux

A few years ago I created an RPM-based cross distro singlehanded, and used it on a product for a customer  This was AT91RM9200-based, a 200MHz ARM9 with 32MBytes of SDRAM.  The amount of effort needed to create a set of cross packages sufficient to create a workable rootfs was huge, it took me many weeks.  Some packages like perl were just so cross-unfriendly that they were basically out of reach (although I later saw other people have done the invasive magic necessary).  It did work well, and I added patches for busybox RPM support that allowed it to do more useful things like erase and keep a package database.  The packaging was valuable in itself but a nice advantage was the source RPMs it generated ensuring GPL compliance.

My background with Openmoko

Subsequently I spent 14 months as (mainly) the kernel maintainer for Openmoko.  Openmoko had an OpenEmbedded basis for it’s rootfs, also a cross system.  I attempted to use it for “hello world” while I was at Openmoko, but it broke because I was on a newly released Fedora.  How it broke was very revealing, the official way to get started with it was to run a huge script that wgetted and locally built 1100 packages.  It died due to some assumption somewhere breaking while it tried to build host dbus libraries.

What I wanted was a cross toolchain that would let me package “hello world”.  What I got was a massive host build action including host dbus libs.  I have perfectly good host dbus libs in my Fedora install, I enquired about it and was told they were the “wrong” libs for the expectation of the rest of the packages, so they had to be rebuilt.

I gave up on trying to use OpenEmbedded, as I guess most of Openmoko’s customers did.

After Openmoko imploded, I designed the software architecture (and influenced the hardware design in some aspects) for the txtr reader device.  On this device, I put into action various lessons I had learned in how not to do things from Openmoko.  I will write further about the other lessons in future articles, but here’s the first one:

Lesson #1: Don’t compile your own rootfs

I was told by a manager at Openmoko that Openmoko had hired most of the main devs of OpenEmbedded and were paying for that accordingly.  This was a pretty big drain on their resources over a long period.

In contrast, nowadays you can head over to http://fedoraproject.org/wiki/Architectures/ARM and download a generic rootfs tarball of prebuilt binaries for ARMv5 and above[1].  It’s made from unpacking prebuilt binary packages.  Once you boot into it, you can install further packages with the usual yum install type action.  You can be up in a high quality rootfs in five minutes flat.

You do not need to go around compiling everything personally when binary packages exist from a reputable distro already.  Normal distros provide -dev and -devel packages for you to link against too, so you do not need to recompile the universe just because you want to build “hello world” either.  That’s how we do things on desktop and server systems, as the processors involved get stronger embedded does not have to be different.

If you want to cross-build specific packages, you install the Fedora ARM Cross Toolchain RPMs on you host via yum and you are ready to go in a couple of minutes.  This is very useful for cooking the kernel on your host both to get started and during development; you can’t native-build the bootstrap stuff needed to boot your platform.  But that’s just a cross compiler and related pieces, it’s not a cross distro.  (The guy from emdebian at this FOSDEM talk also made this point that you do not need to get into making your own toolchain, your distro should have one you can just install).

Fedora ARM’s strategy is native build.  So you install gcc and other dependencies into the actual device, and use standard rpmbuild to build your package there; you can also just configure ; make ; make install for development too down there.  If something’s missing on the rootfs you can yum install it.

(1 To make the comparison fair to openmoko Fedora ARM came along too late for them to choose it from the start, and the GTA02 s3c2442 was not a v5 class processor, they would have been into a distro recook after changing the distro-level compile options.  However my worry is not repeating Openmoko’s errors and today Fedora ARM is available.)

Quality and Quantity

Another major issue is distro quality.  I was so surprised to hear at Fosdem Dr Mickey Lauer of OpenEmbedded boast about the number of devices that managed to use that distro (including the sad shape of the GTA02) and say that unlike the other cross distros, OpenEmbedded focused on “Quantity not Quality”.  From my experience I think he’s right alright about not focusing on quality, and he did go on to explain there are problems with OpenEmbedded they are trying to address.

In the near future, there will be a carcrash between these difficult cross distros that have relatively poor quality and strange requirements to use them and standard, “proper distros” like Fedora ARM, because on higher-end ARMv5s say 400MHz and above, it is already perfectly possible to compare the two worlds on the same device.  I think many devs currently are trained by their experience with buildroot type systems to assume they have to personally build everything Gentoo style.  However as CPUs increase in power at the same price point, the ways of working with these systems efficiently change, and desktop / server “treat it like a PC” lessons like the value of packaging start to really show their traditional advantages over rootfs tarballs.

Like Debian, Fedora has all kinds of rules and requirements about packaging to ensure high quality, there are a huge number of users of these two normal distributions that leads to tested and debugged basic packages and their dependencies.  OpenEmbedded’s boast about number of users is not even a blip in comparison to Fedora or Debian’s consumers and contributors.

Cross distros are locked into local patch hell

A worse problem against their quality even than not many users is the patch load these projects are carrying, I think all of the cross distro projects bemoaned that they were carrying huge patchsets across a large number of packages to get them to build cross at all, and that most upstreams did not care to take them (I assume they don’t want to have to get into testing them).  To uplevel packages, which distros have to daily when they have a large package universe, it can become a nightmare of breakage because of the private patchsets being dragged around.

(BTW I also saw in another presentation that the limo foundation are carrying around more than 80MBytes of diff between their distro and the upstream projects, and these are the guys who sent out a whitepaper explaining the massive cost of delaying sending patches upstream in dollar terms.)

There was proposed a unified crossbuild patch promoting effort, but the effort seemed only to consist of a domain like “sends-patches.org” that you could use when sending patches instead of your own project name, which seems to just be tea and sympathy rather than a solution.

It’s clear that quality will tend to be higher if you are getting packages built with normal distro specfiles and no pile of local patches to get them to build cross (because they were built native).  Combined with higher quality thresholds at the project level and sheer number of users, native Fedora (or Debian) rootfs basis will provide Quantity and Quality if your processor is appropriate.

A couple of hours after the talk I had an interesting conversation with OpenInkpot dev Mikhail Gusarov, who I found also shared my lack of enthusiasm for OpenEmbedded, although he is trapped still in the cross niche generally by the weak processors he targets at the moment.

[update Feb 10 09:00] Mikhail has written his own response, he still likes the speed of cross (and still hates OpenEmbedded).  But there’s some confusion about what Fedora ARM offers, it’s a generic ARMv5 rootfs, it doesn’t care what exact kind of CPU, vendor or peripherals available.  Build farms are less of a requirement when you are no longer building your rootfs but installing it from distro binary packages.  Sheevaplug makes available a 1.2GHz Marvell ARM compatible with 512MBytes of SDRAM that Fedora ARM can work on if you need a native build machine.  Shortly fast dual processor Cortex A9 machines will become available.

Here is 300MB of random from the device checked by ENT (notice I am not using -b as I was before, without it it is checking entropy on BYTE scale which is tougher):

$ ./ent dump
Entropy = 7.999999 bits per byte.

Optimum compression would reduce the size
of this 306380800 byte file by 0 percent.

Chi square distribution for 306380800 samples is 253.74, and randomly
would exceed this value 51.06 percent of the times.

Arithmetic mean value of data bytes is 127.5022 (127.5 = random).
Monte Carlo value for Pi is 3.141608288 (error 0.00 percent).
Serial correlation coefficient is 0.000074 (totally uncorrelated = 0.0).

ENT gives better results for Whirlygig in line with how much you feed it.  With a 40MB test file, it reported entropy of 7.999996.  That makes sense when you consider the data being really random, it shows its true colours only in the longer term since sample by sample, it can be doing anything at all.

rngtest

Rngtest had always puzzled me so most of this post is devoted to picking apart the meaning from these results from 1.27Tbits of Whirlygig randomness (1271Gbits).

rngtest: bits received from input: 1271367467008
rngtest: FIPS 140-2 successes: 63517865
rngtest: FIPS 140-2 failures: 50508
rngtest: FIPS 140-2(2001-10-10) Monobit: 6560
rngtest: FIPS 140-2(2001-10-10) Poker: 6444
rngtest: FIPS 140-2(2001-10-10) Runs: 18865
rngtest: FIPS 140-2(2001-10-10) Long run: 18947
rngtest: FIPS 140-2(2001-10-10) Continuous run: 12
rngtest: input channel speed: (min=39.329; avg=8626.930; max=19531250.000)Kibits/s
rngtest: FIPS tests speed: (min=332.192; avg=105801.561; max=114217.836)Kibits/s
rngtest: Program run time: 155670833366 microseconds

Considering it calls itself “rngtest”, at first sight there are a shocking number of “failures”.  Over 63,568,373 “tests”, 50508 “failed”.  Is something wrong with Whirlygig?  I went and studied the rngtest sources to figure out what it was actually doing.

FIPS 140-2

rngtest is based on a document from NIST which goes into detail about assessing random output.  It’s based on 2500-byte blocks of random data which have various tests applied to them.  But since the source is meant to be truly random, what does it mean to “test” the packet?  Any bit pattern can come in there, each is equally likely as any other, including a whole packet of 0 or 1.  How can some be considered “bad”?

Actually a “bad” packet cannot be considered “bad” in isolation.  Instead you have to look to the spread of packets meeting and “failing” the test criteria against the theoretical probability of their occurrence over time, to see if your random source has one kind of bias or another.  An individual “bad” packet can’t be said to be bad unless the history of failures is suggesting that there is a bias to generate these bad packets.

Unfortunately, I could not find any documentation about rngtest that explained the expected rate of failures from a genuinely random source.  I managed to calculate two of the five.

Monobit

monobit is just looking for a 50% distribution of 1s in each 20000 bit packet.  If a packet comes with 275 more 1s than 0s or 275 more 0s than 1s, then it’s a fail.  Obviously a packet with 1 or 10 extra bits is highly probable.  I found out that these should follow a “normal distribution”, but I was unable to calculate where on the curve “275 more or less 1s” should fall — it’s 0.0275 skew on the expected figure of 10,000…. if anyone can help me it would be most welcome.

In our case, the observed probability of a monobit packet from my Whirlygig was 0.000103, or 1:9690.

Poker

Poker is just looking at the distribution of nybbles  It takes each byte as two 4-bit nybbles, and for each of the 5000 nybbles in the test packet, maintains a count of occurrences of 0 – 0xf.  These counts are squared and then compared to two constants, greater than 1576928 or less than 1563176 for any nybble value gets you a fail.

Again I have no idea how to  calculate the theoretical probability of a “failure” here, but our observed probability is 0.000101 or 1:9864.

Run

A run is a series of “all 1s” or “all 0s”.  rngtest is counting how many times it sees a run of length 1 through 6 (and run longer than 6 bits is counted as being six bits).  The result for each count of run length occurrences is then compared against a magic table:

1-bit: 2315 < run < 2685
2-bit: 1114 < run < 1386
3-bit: 527 < run < 723
4-bit: 240 < run < 384
5-bit: 103 < run < 209
6-bit: 103 < run < 209 (sic)

Once again I couldn’t find any estimate of probability of failing this test with a true random source.  Our observed probability of failing it was 0.000296 or 1:3369.

Long run

For rngtest a “long run” is seeing 26 or more bits the same level at once.  For any 26 bits, the chance of seeing a 26-bit run exactly is 2 in 2^26, or once every 32Mbits (there are two chances because it can be 0×3ffffff or 0×000000).  However, to start the run it’s also a requirement that the previous bit is the opposite level, so it’s 2 in 2^27 chance, or 1 in 1^26 overall, 1.49 x 10^-8.  For a 20000-bit test packet, that’s 0.000298 or 1:3355 chance per packet.

We observed 18947 of these out of 63,568,373 test packets, it’s exactly matching the theoretical chance of 0.000298 or 1:3355.

Continuous Run

A “continuous run” is just seeing the same 32-bit pattern twice in a row, considering 32-bit boundaries.  For every 32-bits generated, there’s a 1 : 2^32 chance that it matches the previous one (without having to know what that was).  So the theoretical probability of these “failures” is <number of bits> / 32 / 4G, for 1.27TB in our sample it comes to 9.5.  We observed 12.  So this doesn’t seem unreasonable.

So overall after studying each test, it’s clear that a random source must fail rngtest with specific probabilities for each test.  In no way is a “failure” on the rngtest tests in itself indicating a problem with the random source.  But if your source does not cause the right amount of failures over time, that is indicating a problem with your source.

It seems wrongheaded then that rngd will reject individual packets that “fail” the rngtext / FIPS140 tests.

Dieharder with a vengence

Next I ran the current dieharder suite again, this is from the latest RPMs on Rober G Brown’s site http://www.phy.duke.edu/~rgb/General/dieharder.php.  I started running it directly hooked up to the RNG device /dew/hwrng, but then I realized that since a lot of the tests are looking for lagged correlation, in fact I needed to give it a file that it could meaningfully rewind into.

So I generated a 12GByte random file and fed it to dieharder -a (run all the tests).  This got us the following summary (grepped just for the decision)

Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for Diehard Birthdays Test
Assessment: PASSED at > 5% for Diehard 32x32 Binary Rank Test
Assessment: PASSED at > 5% for Diehard 6x8 Binary Rank Test
Assessment: PASSED at > 5% for Diehard Bitstream Test
Assessment: PASSED at > 5% for Diehard OPSO
Assessment: PASSED at > 5% for Diehard OQSO Test
Assessment: PASSED at > 5% for Diehard DNA Test
Assessment: PASSED at > 5% for Diehard Count the 1s (stream) Test
Assessment: PASSED at > 5% for Diehard Count the 1s Test (byte)
Assessment: PASSED at > 5% for Diehard Parking Lot Test
Assessment: PASSED at > 5% for Diehard Minimum Distance (2d Circle) Test
Assessment: PASSED at > 5% for Diehard 3d Sphere (Minimum Distance) Test
Assessment: PASSED at > 5% for Diehard Squeeze Test
Assessment: PASSED at > 5% for Diehard Runs Test
Assessment: PASSED at > 5% for Diehard Runs Test
Assessment: PASSED at > 5% for Diehard Craps Test
Assessment: PASSED at > 5% for Diehard Craps Test
Assessment: POSSIBLY WEAK at < 5% for Marsaglia and Tsang GCD Test
Assessment: PASSED at > 5% for Marsaglia and Tsang GCD Test
Assessment: PASSED at > 5% for STS Monobit Test
Assessment: PASSED at > 5% for STS Runs Test
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: POSSIBLY WEAK at < 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: POOR at < 1% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for Lagged Sum Test

No way!  Two “possibly weak” and one “poor”.  I read the manpage for dieharder and got the advice from there to run the tests more times, because if the data is bad, feeding it more skewed badness will make the failing distribution of p-values “unambiguous”.  Dieharder has a default of 10,000 tests, I cranked it up to 20,000 and ran them all again on the same 12GByte sample.

Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: POSSIBLY WEAK at < 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Bit Distribution Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Generalized Minimum Distance Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Permutations Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for RGB Lagged Sum Test
Assessment: PASSED at > 5% for Diehard Birthdays Test
Assessment: PASSED at > 5% for Diehard 32x32 Binary Rank Test
Assessment: PASSED at > 5% for Diehard 6x8 Binary Rank Test
Assessment: PASSED at > 5% for Diehard Bitstream Test
Assessment: PASSED at > 5% for Diehard OPSO
Assessment: PASSED at > 5% for Diehard OQSO Test
Assessment: PASSED at > 5% for Diehard DNA Test
Assessment: PASSED at > 5% for Diehard Count the 1s (stream) Test
Assessment: PASSED at > 5% for Diehard Count the 1s Test (byte)
Assessment: PASSED at > 5% for Diehard Parking Lot Test
Assessment: PASSED at > 5% for Diehard Minimum Distance (2d Circle) Test
Assessment: PASSED at > 5% for Diehard 3d Sphere (Minimum Distance) Test
Assessment: PASSED at > 5% for Diehard Squeeze Test
Assessment: PASSED at > 5% for Diehard Runs Test
Assessment: PASSED at > 5% for Diehard Runs Test
Assessment: PASSED at > 5% for Diehard Craps Test
Assessment: PASSED at > 5% for Diehard Craps Test
Assessment: PASSED at > 5% for Marsaglia and Tsang GCD Test
Assessment: PASSED at > 5% for Marsaglia and Tsang GCD Test
Assessment: PASSED at > 5% for STS Monobit Test
Assessment: PASSED at > 5% for STS Runs Test
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for STS Serial Test (Generalized)
Assessment: PASSED at > 5% for Lagged Sum Test

So the “poor” and “possibly weak” guys became happy when we doubled the number of tests, and there’s a new “possibly weak” guy.  But when I looked up the new guy’s p-value, it was only 0.02888045, which is 1 in 34 chance, it doesn’t seem that improbable (real dieharder failures tend to look like 0.00000001 or 0.99999998 an should look more like that the more tests you run).

Conclusion

So far as I can tell these results are good.

If anyone has enough math power to calculate the theoretical distribution of the rngtest monobit, poker and run rngtests I would be very grateful, so I can compare all the numbers.  On the two I was able to calculate, we seem to be very close.

Dieharder seemed happy with double the tests and the one test it flagged then only had a probability of 1:34 which is not unreasonable.

Improvements

I took the opportunity to make some improvements:

- Added JTAG programming of the CPLD to the SiLabs microcontroller over USB.  This allows change or update of the CPLD logic from the host PC without any hardware needed.  However because the kernel module blocks the logical USB interface, it’s safe from being rewritten while in use.

- Changed the random logic.  I’ll explain the changes and results in the rest of this article.

- Decreased the polling rate of the CPLD but increased the total USB random throughput, 1.0MBytes/sec sustained (for as long as you like) by making the code in the microcontroller “multithreaded”.  You can also plug in more Whirlygig devices to linearly increase random production; the kernel module allows hotplug and unplug without problems and combines the output seamlessly all in /dev/hwrng.

- I was pleased to see the kernel module had hardly bitrotted at all, it only needed a one-line edit to build a working module against a current Fedora Rawhide kernel.

The second LED lights while the PC is requesting random packets from the device.  It lights briefly on plugging it in while the driver’s cache is filled, then it only lights when something is using the hard random numbers on the PC.

New random scheme

I had three main ideas about improving the random hardware inside the CPLD.

First I realized we can decrease predictability by having more oscillators than are used at one time to change an output bit.  We have 8 output bits, but we now have 16 oscillator sets.  Instead of combining them all, on average several will not be used on any given operation.

The second idea was that now we have a pool of oscillators greater than needed at any one time, we can randomly select from them for each output bit operation.  So I added an additional 32 oscillator sets (4 for each output bit) which are only used to select which of the pool of 16 we use for any operation.  The end result is that at least 8 oscillators from the pool will be unused for each operation, and which oscillators do get used for which bit are individually “random” with “no” correlation between output bits.  This makes any attacker’s attempt to model the pool oscillator states very tough because there’s no longer any knowledge about which bit contains information about which pool oscillator, or even if its state has affected any output bit.

Lastly we now operate from a clock (24MHz) that is 14 times faster than the sample rate.  This lets us mix 14 randomly chosen oscillator states by xor before the output is sampled for each bit.  Even if two output bits were mixed with the same 14 oscillators, the order would have to be the same as well to get the same result, since the oscillators are never standing still.  For this same reason selecting a pool oscillator more than once in the 14 operations is not equivalent to a NOP.

I added another small tweak, all of the random generators shift ther oiginal state by 1 generator on each clock.  This is intended to reduce the impact of any hard nonliniarity in individual generator routing on the CPLD.

There were no problems with the PCB, but to save myself a headache working with the crossbar in the CPU I blobbed together pins 26 and 27 on the CPU.

In the next article we look at the random performance again with the new scheme.

If I understood it, his opinion was that the license terms of the GPL would not survive resale, due to the well established “first sale doctrine” and its EU equivalent “exhaustion”.  It basically means that the copyright holder cannot stop you reselling your software, and that the license terms will not apply to the guy receiving it.

I tried to understand this further, but Alexander was not always easy for me to comprehend and had then a habit of linking to his own posts elsewhere to bolster his position, leading to a kind of echo chamber of Terekhovs all nodding vigorously at each other.  He also back then and evidently more recently too explained legal decisions that did not fit his understanding by calling the Judges in question “morons”, etc.  Well the forum I met him at had a very high trolling quotient so it just joined the rest of the anti-GPL sentiment there for me in the end and I ignored it.

GPL is a license too

But I was reminded of this last night when I read about a recent decision against Autodesk which is being widely seen as a victory for Joe Softwarebuyer.  From the Patry blog post link above:

…many software companies have taken the position that they can convey the copy to the customer in an over-the-counter transaction for a one-time payment, but describe that transaction as a license; as a license, the first sale doctrine doesn’t apply, meaning copyright owners can prevent further distribution of the copy…

Doesn’t this vindicate Alexander’s position?  How can GPL terms stick past resale if Autodesk EULA ones don’t?  Nothing stops “built-in” or “automated” resale to clense software of any licensing restriction.

A lot of people seem to be happy about the paid-for world being freed from license conditions, are they going to be happy if it turns out that everyone is also freed from GPL conditions?

Civil infringement and Punishment

What effect would this have on contribution I wonder.  It seems to me the real-world advantages from being active in a project by contributing will still apply.  But it will enable private proprietary forking for products, the kind of thing that Harald Welte’s gp-violations.org has had success attacking and punishing to date.  Contributors will see their work used in commercial products without the changes being open.

But the BSD folks seem to survive this outrage without it removing their motivation.  And from time spent looking at music licensing over the years, I kind of recognize an element of proprietary vindictiveness in gpl-violations… of course the member companies hiding behind the RIAA attacks are also “perfectly within their rights” to embark on much worse vindictive destruction, but they are not entirely dissimilar and that always bothered me.

Playing ball or going home?

Well, this decision is subject to appeal, will only apply to the jurisdiction of that court, etc, so the sky didn’t fall in already.  But there is quite a bit of harmonization of copyright law thanks to the insistence of rich rightholder companies mainly from the US side.  But if this is upheld, it may come to contaminate most Western countries and turn GPL terms in unenforcable noise — the choices would be in effect public domain or closed.

I guess some people will go closed rather than have their work exploited, but I expect most people will just continue on, and contributions will continue to come perfectly fine.  The advantages from being a visible contributor and taking upstream directly are still going to apply, so will the bitrot that happens to any additional code put on top and maintained privately.

Too mature to care?

Maybe now we reached a point that the social, financial, engineering and public advantages from cooperation are ingrained enough that we don’t need a license to protect them anyway?  But I read this and I feel a sinking feeling about the naivity of such a proposal.

Hardware random for the masses

I made available the result of the ring oscillator random generator as a GPL project called Whirlygig. It’s a 2.75cm x 4cm PCB with a mini USB connector, it provides a sustained 5.5Mbps (~620KBytes/sec) of apparently very high quality random bits using the Linux hw_random API. The large amount of randomness should make it useful for statistical tests as well as hard crypto.

I prototyped it using a couple of boards I had lying around, so I know it works fine, but I am waiting for the PCBs to come back from fabrication to actually build a final one. I placed the CPLD VHDL, the board hardware design, the driver software and the firmware for the USB controller into http://git.warmcat.com.

Dieharder

I spent some time worrying about how to test the quality of the result — I found that “diehard” mentioned in an earlier post has been superceded by “dieharder”. This has a much tougher general testing regime, even though many of its test are reproductions of the diehard ones — it runs each test many times and forms histograms of the p-value results from the many runs, and gives an assessment of fail, poor, possibly weak or pass on the spread of results rather than a single result.

At first the RNG failed three of the 18 tests, but on looking closer one of the tests (#2) currently fails for all RNG input and is marked up as not for use with assessing RNG quality, and the two others required by default more than the 400MBytes of randomness I had prepared. Unfortunately in that case they simply rewind the randomness file and re-use the same data to make up the balance! Of course this is no longer quite “random”. When I adjusted those two tests to use a smaller sample that fitted into the 400MBytes without repetition, the output of the RNG get a “pass” on all 17 of the relevant dieharder suite tests.

Max Entropy

During the validation phase I changed the RNG algorithm in the CPLD significantly. The scheme is described on the project page, but basically I moved away from a bit-centric to a byte-centric design with 8 identical sets of 3 oscillators. To stop any characteristic of a particular oscillator’s routing from being associated with a particular bit of the result byte and creating a bias, I introduced a “mixer” that first generates 8 random bits by combining six oscillator outputs each with XOR, then rotates these oscillator sets between the result bits sequentially at 24MHz. I also removed the toggling action and used the random bit directly.

I also found the Linux rng-tools suite which repeatedly runs FIPS-140-2 tests on the bits, this fails 1 in 1200 or so packets of testing over 20 billion bits, I believe this is normal for a real random generator that it will produce sequences with low probability that don’t look very random in the short term.

Aside from passing dieharder and FIPS-140-2, the changes also got me a reported 8.000000 bits of entropy per byte from the ENT test, so there are reasons to imagine the quality of the output is very good.

I ran the last 10MByte sample against ENT and TestU01… to cut a long story short

$ ./ent ../die.c/dump3
Entropy = 7.999980 bits per byte.

Optimum compression would reduce the size
of this 10002432 byte file by 0 percent.

Chi square distribution for 10002432 samples is 281.26, and randomly
would exceed this value 25.00 percent of the times.

Arithmetic mean value of data bytes is 127.4958 (127.5 = random).
Monte Carlo value for Pi is 3.140111525 (error 0.05 percent).
Serial correlation coefficient is -0.000212 (totally uncorrelated = 0.0).

7.9999 bits of entropy per byte! TestU01 is less turnkey than the other suites — it’s literally a test library with some example code. I amended an example to call the FIPS-140-2 tests:

============== Summary results of FIPS-140-2 ==============

 File:             dump3
 Number of bits:   20000

       Test          s-value        p-value    FIPS Decision
 --------------------------------------------------------
 Monobit               9933           0.83       Pass
 Poker                11.88           0.69       Pass

 0 Runs, length 1:     2482                      Pass
 0 Runs, length 2:     1227                      Pass
 0 Runs, length 3:      630                      Pass
 0 Runs, length 4:      319                      Pass
 0 Runs, length 5:      161                      Pass
 0 Runs, length 6+:     166                      Pass

 1 Runs, length 1:     2466                      Pass
 1 Runs, length 2:     1302                      Pass
 1 Runs, length 3:      620                      Pass
 1 Runs, length 4:      311                      Pass
 1 Runs, length 5:      140                      Pass
 1 Runs, length 6+:     146                      Pass

 Longest run of 0:       16           0.14       Pass
 Longest run of 1:       14           0.46       Pass
 ----------------------------------------------------------
 All values are within the required intervals of FIPS-140-2

So the design’s output is compliant to FIPS-140-2, a requirement for many uses.

RNG Quality assessment

A timely article flew by on Reddit about the RANDU pseudo-random generator algorithm widely used in the 1960s, which it turns out was very flawed indeed. It was explained to one student that ”We guarantee that each number is random individually, but we don’t guarantee that more than one of them is random”. Basically it produced numbers that belonged to one of 15 “planar” groupings and nothing in the gaps between the planes. It isn’t just a minor annoyance, because many statistical studies in the 60s and 70s used it, and it can easily have contaminated their results. That’s definitely not what I am trying to reproduce with the ring oscillator device — so how can I figure out how “good” the randomness is in an objective way?

RNG quality test suites

It turns out that empirically testing RNG outputs has been the subject of a lot of work for decades, and there are some established testing suites available online. A major one seems to be the “diehard” suite — I guess it is a pun on die as the plural of dice.

It needs you to fetch 10M bytes of random numbers or more and let it run a bunch of tests on them. The output was a little hard to assess initially: most tests issue a “p” number which only suggests something is bad if it is 0.000… OR 0.999…. All other numbers inbetween are to be taken as a good result as I understood it. Except there is a warning that even good RNGs can produce the occasional test fail.

Thus you should not be surprised with occasional p-values near 0 or 1, such as .0012 or .9983. When a bit stream really FAILS BIG, you will get p`s of 0 or 1 to six or more places. By all means, do not, as a Statistician might, think that a p < .025 or p> .975 means that the RNG has “failed the test at the .05 level”. Such p`s happen among the hundreds that DIEHARD produces, even with good RNGs. So keep in mind that “p happens”

I duly fetched 10M bytes of 115kbps randomness from the device and fed it to diehard. It seemed to give fine results except on “Count the 1s stream” and “Squeeze” (devastating p=0.000000), “Count the 1s specific” for bits 1-11 (p=0.000030) and 9-16 (p=0.000064), and QQSO 2-6 (p=0.000005). It passed the dozens of other tests but it was disappointing, looks like a big fat ‘failed’.

Triple Scoop

Well, since my test CPLD was an XC95288XL with 288 Macrocells to burn, I naturally wondered if I could improve matters by tripling the amount of ring oscillators getting Xor-ed — that is to implement the three varying sized oscillators 3 times each, totaling nine, and sum them with a big XOR. They’ll all be drifting around individually as much as together, it should be a mighty noise-fest.

I edited the VHDL and blew it into the CPLD… visually the summed RNG output “bit” was an awful lot more noisy than before. I pulled another 10M bytes from that setup: but just looking at the byte distribution as I did before told me something is still up.

That sawtooth type distribution is “not random” to coin a phrase. If you look at the large jump at 0×80 (128) it is telling us that we are more likely to get 1000000 binary than we are to get 01111111, in other words, since this is over 10M bytes, there is a distribution problem favouring ‘0′. When I analyze the distributions of 1s and 0s I find

0: 40436204, 1: 39563804... delta=872400, skew=1.090500%

You can see the same thing even better looking at 0×00 (42,000 hits) vs 0xFF (36,000 hits), they are like 8% off the median of 39,000. Clearly that distribution of 1s and 0s has to have a very small skew to stop these kinds of effects showing up, and equally clearly this is telling us something deep about the RNG hardware.

Spiky

Although the individual oscillators are quite slow thanks to the number of inverter stages, at 4 – 6MHz, the way they are being summed makes for trouble from bandwidth limitations inside the CPLD. At the moment it just uses a dumb asynchronous XOR action, that means that potentially very fast spikes can be seen when one “slow” oscillator changes state very shortly after another “slow” oscillator. For example:

You can see on the left (this is 5ns/div notice) a runt pulse where this happened, the XOR was convinced to rise by one oscillator changing and then countermanded when another oscillator changed state less than 5ns later, resulting in a doubtful pulse that was probably not visible as a ‘1′. This also happens when going from ‘1′ to ‘0′, but maybe the threshold for the transistors in the CPLD is not at exactly 50% of the 3.3V supply. So we suddenly have it seeing more ‘0’s than ‘1’s on average when spikes are involved.

This whole high bandwidth summing step is completely needless, it’s only there because it is a literal interpretation of the diagram in the original RFC. I changed it instead to have nine latches sample the nine oscillators every 125ns (there is an 8MHz clock on the prototype board) and sum those results with XORs into a single bit. In turn this output is sampled by another latch at 8MHz to hide any metastability.

Latched up

The latched summing version performs much better and has gotten rid of most of the bit skew, and the sawtooth behaviour:

…but there is still a problem with 0×00…. the bit skew looks like this

0: 39960076, 1: 40039932... delta=79856, skew=0.099820%

so the skew is now on the side of ‘1’s but only by 0.1%. You can see the byte count spread is much tighter than before too — 1800 instead of 6000 counts before.

Balancing out the skew

Well if the remaining skew is something to do with the ratio of rise to fall times, or the non-squareness of the oscillator outputs for some other reason by something as low as 0.1%, that is hard to do much about, especially as it may vary on the specific silicon die.

But it shouldn’t matter — now the bandwidth situation at the XOR summer is sane, if we invert the summed output 50% of the time it should spread any excess on ‘1’s or ‘0’s to the opposite as well, cancelling any bias. I added a couple of terms to the summer to xor against the UART bit index LSB and a bit which toggles after every byte sent by the UART. It’s the equivalent of xor with 0×55 for the first byte and then 0xAA for the second byte, over and over.

That glitch in the middle is actually at 134 (0×86), maybe it is random but I guess we will see…. the skew is further reduced as anticipated

0: 39974218, 1: 40025790... delta=51572, skew=0.064465%

Diehard sequel

I ran 10M bytes from this version through Diehard again… the really bad p-value results are gone. For example Squeeze was a deadly 0.000000 before and is now 0.255260.

I made one last adjustment, I added the current state of the latched random value to the XOR term. That means it decides whether to keep or invert the latched value, it no longer directly accepts the value from the RNG. This got me to the promised land: 0.0005% skew between ‘1′ and ‘0′.

0: 40000206, 1: 39999802... delta=404, skew=0.000505%

This also gets me the apparently good diehard results with no obvious failures on any tests, you can see the actual results here. So it seems the current version can tentatively be called a “real RNG”.

Pretty random

After some scrabbling around porting my Jtag SVF interpreter to Octotux and creating a kernel module for the PIO end of it — and moving to a different board with a XC95288XL CPLD to prototype it, the triple ring oscillator RNG is working. It issues a 9600 baud result, but after some initial confusion I modified it 1/8th of the time to sit out a sample time leaving “break” on the serial line. This should make sure that the receiving UART does not get confused by the data as a start bit. The true data rate is something like 800 random bytes per second at 9600 baud.

Here are the three chains of inverters (19, 23 and 29 long) oscillating at the different fundamentals

… and here is what the xor summing looks like, first over 1s then sampled once.

Although the single shot sample doesn’t look very random, the oscillators are drifting around all the time. If you wait a little while between samples (currently it is 104us, a 9600 baud bit-period) it’s pretty hard to guess what phase all the oscillators have drifted to — at least, that’s the plan.

Distribution of binary levels

The first test I did was to see what the distribution of ‘1′ and ‘0′ in the results was… clearly if the device is really random it should on average be 50% each. I fetched 1M random bytes, or 8Mbits:

Its okay for a really random source to deviate to 50:50 at any given time, although on average it should be 50:50.

Octet distribution

Next I looked at the distribution of the results from 0×00 through 0xFF as the result “random byte”. This would show up if the RNG fails to ever issue some result or favours certain results over others — every result should on average have an equal chance of showing up and so an equal count. I ran it for 1M random bytes…

This is pretty decent, every possible result is seen with a frequency within +/-200 counts of the 3,900 average after 1M bytes.

115200 baud results

Encouraged by this I cranked the baud rate up to 115220 or 8.68us between samples and around 10K random bytes per second. The skew is increased somewhat and the spread of result counts is increased a little.

So far so good!