The full sources for the experimental modem using the correlator codes is available at http://git.warmcat.com under GPL2+ license.

The big change is what I call “scrambling”.  The 126-bit magic correlator sequence is disordered and xor-ed in 258 random ways, which are selected to not correlate well with each other at any offset (less than 43/128 match).  This allows us to issue whole bytes in one code by selecting which disordered correlation code to transmit.  The other two codes are for start and end of packet markers. I first tried 22 scrambles to allow 4 bits per code and some extra codes, this worked fine but I was able to extend it to 258 without really damaging the “best” scramble-scramble false correlation score too much.

The gain here is that the self-ordering and threshold properties of the code now reaches up to entire bytes: you always get a clean, aligned byte or you don’t get anything: with high probability you don’t get a wrong byte.

I also changed the demodulator code to something that is currently quite expensive to run.  Instead of tracking a reference phase at the receiver, which is tough to do in the presence of extreme noise and phase wrapping, at each sample it tries to demodulate using that sample as “0 degrees” and a 50% phase slicer to get the bits.  It’s expensive because it has to do that against each of the 258 scrambles every sample — but on the plus side the correlator does not run at all for 124 symbols out of 126 (98.4% of the time) after there has been a successful decode, since it knows it is partway through a 126-symbol code. Still the number of demodulation attempts can definitely be reduced, maybe by subsequently just doing it on the last winning phase and a few either side.

I found that the noise performance of the demodulator is strongly dependent on the relationship between the sample rate and the carrier frequency.  It’s much less dependent on the number of carrier cycle periods per symbol, so long as you have 4 or 5 or above (2 works but is killed by hardly any noise).  Since there is plenty of filtering to the carrier going on I didn’t really expect that. Here is a graph of the noise performance when there are 96 samples per carrier (48kHz sample rate, 500Hz carrier, 100baud — these are intra-code symbols, it’s 6.25bits/sec effective):

The noise performance doesn’t look too bad, at least with these synthetic tests and unrealistic 48kHz sampling. It can recover all 7 bytes that are sent even at -28dB, when only 4% of the received power is the signal and the rest is white noise. And because of the code properties, these are definite bytes being captured with quite high probability, not the kind of uncertain decode that would normally be expected in such noise.

The “average quality %” shown in the graph is the percentage of demodulated bits in the matched code scramble that actually were “right”, averaged over all the bytes. If a byte is missing because its quality was below the threshold of 70/128, it counts as 0% quality. Up until -20dB, the demodulator is doing well and the recovering individual bits almost perfectly. Without the Correlator code, after -20dB you would be dealing with a rapid increase in bit errors from the demodulator and relying on ECC. By using the Correlator coding though, we are able to push performance another 8dB into the noise (we even recover half the symbols at -30dB, or 10dB further) and still maintain the alignment and high probability of correct decode advantages.

This shows the effect of keeping everything else the same, but bringing the sample rate down to 8kHz from 48kHz

You can see the BPSK demodulation starts to fail at -10dB instead of -20dB and the correlation code again buys you another 8 – 10dB into the noise after that.

Here is the spectrum after bandpass and lowpass filtering that is actually “transmitted”, the peak is the carrier at 500Hz in this case.

Another aspect of this setup is that although the demodulator is currently pretty expensive in CPU (and power), the modulator is much simpler. It just requires a 1KByte table for the scrambles and a precooked integer sine table if you are running a separate carrier than the RF one itself. Then it just needs enough logic to walk through the scramble table entry at the symbol rate (and the sine tables at the sample rate if you’re using it). That can fit in part of a tiny flash controller, using a 1-bit output of a shifter to switch the phase of the transmitter. It can be simplified even further by not using scrambles but just sending a bit per code sending the code forwards or backwards to signal a ‘1′ or ‘0′.

It seems that I can begin to understand where this system fits into the existing high noise codings already used by ham radio folks. The correlator code is a special case of using an ECC code, in this case where 8-ish bits are exploded into an 126-bit “space” filled with correct decodes, damaged but recoverable decodes and invalid decodes. The error correction performance of adding 8-10dB “coding gain” is probably a bit (not so much, from what I can work out) poorer than an optimal use of 126 bits to code for 8, but the advantages that attracted me to the code in the first place can offset that for some applications, the guaranteed self alignment right down from demodulation to byte boundaries, and a quite firm decode success threshold. In addition, it seems some of the more optimal decoding schemes for stuff like Viterbi can be very compute-intensive, whereas although at times we do a lot of it, the correlation action is simple and lends itself to being done in parallel. Turbo codes are patented. So it’s not a one size fits all technique, but it has its niche.

I guess the next stage is looking to BPSK modulate and recover an RF carrier directly, but that will need some thinking on because it will be a very frequency-specific design, unlike these baseband tests where you can just edit the important variables and recompile. For example if it can be done initially at the UK 40MHz ISM band it will be possible to consider logic looking at carrier zero-crossings for phase assessment easily enough, and to autocorrelate just the averaged recovered phase at 4 times the symbol rate.

QPSK abandoned for BPSK

The noise performance of the QPSK decoder wasn’t what I was hoping after several iterations. I pretty quickly threw out the Costas loop because loss of phase lock was the weakest link in the chain, and moved to a system with a tight bandpass filter at the carrier frequency, and measuring zero crossings of the carrier against a local oscillator. But still the noise performance was not impressive at 300baud / 3kHz carrier.

I had hoped to go back to a damaged QPSK code recovery and to correct the data phase on the broken code bits, because we know what they should be according to the code. But there was no visible commonality between what was happening on the phase information that carried the code and the phase information carrying the data.

So I have moved back for now to a two-phase BPSK system interleaving the code and the payload symbol by symbol, in order to understand better how far the code can be pushed with noise.

Improved noise performance

Currently I use a 4.8kHz carrier and 1200baud symbols. Here is the performance today:

The RED line is the best decode seen for the correct match offset, the BLUE line is the best decode for all other (wrong) offsets. The PURPLE line is the mean decode for the correct match offset over the 100 runs. GREEN is the “quality cutoff” basically keeping us from getting near the blue line’s false hits: because of the properties of the code it is extremely unlikely noise will get near the green line.

Here is a close-up on the noisy end of things:

So far in terms of detecting the code, we can on average do so when the input energy is 82% noise (-15dB SNR). We can still detect codes ~1% of the time even at 92% noise (-21dB SNR). Here is a further close-up (it is 1000 runs, not 100)

suggesting you can still get good recoveries ~0.1% of the time at 95% noise (-25.5dB SNR). And of course we are now measuring the whole system performance here including the demodulator part, not just damaging the code bits directly.

Current BPSK receiver

Here is the receiver for the current BPSK method:

I spent several days meddling with the QPSK version and arriving at the carrier zero-crossing method for phase detection. The original plan to have a symbol sampler running from a locked LO hung around causing lots of problems. The indications of a change in symbol — detected by zero crossings — were variously delayed by the phase itself and the filters used, making it difficult to convert their jittery indications into a guide for the symbol recovery clock. This resulted in double bits being sampled.

The code is the symbol clock

Because of this I eventually realized that no symbol clock was needed, by running the correlator at the sample frequency, and sampling symbols at a fixed period, because of the false match rejection properties of the code it would “discover” the correct phase and offset from the behaviour of the correlator output. And when the code was recovered best then the data interleaved with it will be recovered best too. This sounds power-unfriendly, but after the first offset is used, you don’t run the correlator until enough time has passed for 256 symbols to be acquired, and then you should still be locked from when you did the first 256 symbols: if not you run the correlator a few dozen times to find lock again.

Also of note is that what is stored in the ringbuffer is a weighted average of the last four phase results, this includes information from the phase of multiple carrier cycles for the same symbol, helping to reduce the effect of noise on the decode.

The code is the data!

Well recovering the code at high SNR is interesting, but how useful is it if the data bits interleaved with it have been subject to the same beating without the properties of the code to protect them? We can use the autosyncing properties of the code to help with trying to get some payload signal gain through averaging, but I think I have seen where this is headed now… the code IS the signalling system for the payload data. That means throwing out the interleaved data concept.

A ‘0′ can be signalled as the normal code, and a ‘1′ as a time-reversed code. Because it’s intended for low data rate communication at VERY low signal levels compared to noise, it’s okay if we are reduced to 1 byte/sec, which will be the end result of this at 1200 baud. Basically with this we carry over all of the great robustness qualities of the code to be attributes of the payload data.

So what is the point compared to just blasting 128 symbol times of the same phase carrier, which is a hell of a lot simpler?

• Three high accuracy results, “no result”, a ‘0′ or a ‘1′ detected. The carrier-only method will happily return a bogus result if there is noise energy at the carrier — if the code method ever claims a ‘0′ or a ‘1′ you can be almost certain it is genuine
• “Fuzzy” robust damage-tolerant signal detection, better performance than a simple threshold comparator
• Automatic bit sync (”bit clock recovery”) with almost no chance of wrong sync, sync recaptured each symbol; bit sync for the carrier-only version in high noise is unreliable
• Absolute phase polarity can be recovered from one symbol despite the 180 degree lock uncertainty for BPSK — you can even lose lock one or more times inside the symbol and the code with absolute phase can still be recovered; the carrier-only concept needs a coding at a higher level to determine absolute recovered phase, in turn needing multiple correct symbols recovered without losing lock

It shows 1,000 Monte Carlo runs each from 0% to 100% noise on the raw channel, considering correct correlator matches only, the blue line shows the WORST correlator match result at that noise level, the red line is the BEST correlator result seen at that noise level, and the purple line is the mean correlator match result seen at that noise level.

The Green line is the +64 threshold as a reference… below this we don’t consider the correlator to have matched, something we chose based from studying the code response to noise a couple of posts ago.

Basically it shows that up to about 20% channel noise there is a very strong probability we return perfect results (+128 result means that no bit errors were present in the recovered correlator code).

After that there is a region up to about 50% noise where perfect results are sometimes seen, normally there is some corruption, but on average we can still recover a corrupted but high probability correctly sync’d code (whether the attached data payload can survive that beating is another issue… we can at least have reasonably correct bit-sync to whatever is there, allowing payload averaging for example).

But after 60% noise, the probability of finding a usable recovered code is less than 1 in 1000.

If you consider the purple mean line as the overall average recovery capability, it’s clear that at the moment after 40% noise things stop being much fun.

These figures reflect the whole reception system: at the moment the data slicer is a primitive edge-triggered type thing, pretty sure that is the limiting factor here. When I eyeball the recovered data payload when it is challenged by heavy channel noise, It is often broken in a “sticky” way:

hello magic code
hello ma��������
hello magic code
gllo magic co$�
hello magic code
���oo magic code
hello magic code
Yello magic co�7
hello magic code
hello magic cod

I think the sequential bit errors will be harder to create if a more robust bit-slicer is figured out. Here is an idea of what 45% noise in the channel looks like:

and what happens to that nice clean, digital-looking demodulation when you give it that instead of the nice clean sine waves:

That last one is the input into the bit slicer, which has the job of choosing where the bit boundaries are… at the moment it looks at zero-crossings, if it were possible to do a more sophisticated job than that a lot more packets could be saved you would think. I have an extremely cool idea about this I will look into next.

Testing plan

The general plan is to bind payload data to the magic correlator code, such that the correlator code alignment acts as both a frame sync and assurance that the recovered bit clock is correct (in fact it should act as a clock recovery synthesis pilot as well, since we know the sequence). Because of the properties of the correlator code, it should be possible to just add FEC-coded payload without further ado, either interleaved bit-by-bit or bound another way.

At the moment I am sending plain payload without FEC. Further I am going to initially do all the testing at baseband directly, in fact specifically at audio frequencies. This will eventually allow real world testing using a laptop “transmitting” the coding through its speakers and my moving another microphone-enabled laptop around the house seeing how far we can push the data recovery vs what I can hear myself. My house it pretty noisy, with cars going by, kids jumping around and so on, there is a good mix of whitish noise and short duration dropout crud in the audio spectrum. I am really interested to see how far it can be pushed, particularly if averaging is possible to get working.

Modulation strategy

Currently the test C code modulates the data and the magic correlator code bits together on QPSK. QPSK has four “modulation states” encoded as four phase angles of the carrier, so you are signalling two bits per symbol — one payload bit and one magic correlator code bit. Eventually, because we can recover even badly damaged correlator code quite often, it will be possible to score the likelihood of false payload recovery bit by bit based on looking back at the magic code bits that turned out to be wrong in a particular symbol: the payload and the coding bits were transmitted in the same symbol. This “distrust” indication per payload bit can open the door to some novel, if possibly expensive, error correction.

The current C code uses 44.1kHz sample rate and a 3kHz carrier modulated with QPSK carrying 300 baud symbols, ie, there are 10 carrier cycles per symbol. Each symbol contains two bits due to the QPSK coding. Maybe as we go on it will be necessary to drop the symbol rate to allow more cycles and better recovery, but this is a starting point. You can listen to the QPSK coded WAV file here, this has the magic code and a 16 byte ASCII message payload modulated on QPSK. The symbol transitions look like this:

So, the “transmitting” part is fine… don’t be fooled by those little spikes where it changes phase, those easily lost spikes are not carrying the information. The whole rest of the bit period carrier goes on at the new phase after that discontiguity, the next ten cycles of carrier phase encodes the information. The spike could and probably should be completely filtered out and not cause trouble on recovery.

QPSK symbol recovery

QPSK recovery requires a coherent oscillator in frequency and phase sync with the incoming carrier. You recover the symbols by watching the gyrations the local oscillator has to perform to keep the phase sync.

The RF guys were on to all this stuff back in ancient times, a version of a Costas loop capable of separately dealing the with quadrature “carriers” in QPSK was invented way back when. Its basic idea is to feed the coherent local oscillator to two mixers, one with the local oscillator via a 90 degree phase shift, then lowpass filter the result from the mixers and use that to feed back an error term to the local oscillator. The lowpassed mixer outputs are the “result” from the loop for the two bits encoded in the QPSK modulation.

Now described there and the literature one can find in Google, it’s simple, right? Some of the PDFs on QPSK recovery from Google even had beautiful smooth digital recovery pictures. But the actual implementation is a lot trickier. The problems come from the need to tune various filters in the Costas loop, they need to be tuned for the carrier and VCO lock performance considering the symbol rate. I got started by looking at the Costas loop implementation in GnuRadio, but this has no filters at all in it (I guess this is for flexibility you can add the filters outside it). No doubt someone somewhere has been through all this back in 1970 and written up all the equations, but I couldn’t find it. In the end I found some general advice about matching the lowpass filter 3db point to half the symbol rate and meddled around until it worked. Of course doing it in software I didn’t even have the nightmares of filter matching for the two mixers which bedevil an analogue implementation. I also found a lot of 2 x carrier noise in my loop, which I tried to notch out with some success. Anyway here is what the recovery looks like right now, being fed “perfect” noise-free signal… it looks okay but I am pretty uncertain about how it will react to noise

QPSK absolute phase uncertainty

The receiver locked phase compared to the transmitter 0 degrees phase is unknown, therefore the decoded bits can appear on either of the two output bitstreams and be inverted. This has to be taken into account when looking at what you’re getting. Initially at least you have to run the four possible correlations against the magic code to find out the effective phase offset / symbol coding you have locked at. I guess while the receiver does not lose lock (one can study the error term in the Costas loop I guess) you can just use the lock you previously determined.

Incidentally received absolute phase determination is yet another exploitation of the magic correlation code properties of robust matching.

Next stop

Next task is to recover the bit clock from the received bits and perform the correlation action, and to try to recover the message. All of will still be in pure software with no noise yet.

False indication rejection when given only noise

What about the reaction to having no signal at all… can it tell there is no transmission or does it falsely detect correlation? What is the highest false correlation result seen when challenged with noise? Here is the distribution of highest correlation results for 50 Million runs feeding it only white-ish binary noise with no correlation sequence component.

The largest false response seen even once in the runs is +58, out of a full-scale match with a 0% bit-error rate of +128: one can put it that the probability of seeing a match better than +58 from noise is something greater than 1 in 50M. So we learn from this we can’t trust any correlation result lower than, say, +64, to allow some margin. (This +64 requirement is shown with a blue line in the following graphs).

Random bit-error rate response

In this graph I ran the self-correlation 10,000 times per offset with different noise each time, and picks the worst (lowest) correct “position 0″ sync match value (red) and plots it against the best (highest) wrong offset match value (green) in absolute match quality. The thin blue line shows the absolute correlation value of +64 we selected based on the first graph.

On the left where there is no noise, we can tell the correct sync by a wide margin. Where the red line crosses the green, at around 0.2 bit-error probability, it means the correct sync position can no longer be distinguished from a false match. But before then, the absolute correlation value for the correct offset has fallen below our +64 limit (selected because noise can create a +58 result) so detection is lost first at a 0.12 ber.

Here is a plot of the ranking of the correct offset vs all of the other offsets. I expected the correct one to start at #1 and then slip down the rankings, but instead it starts at #1 and falls right to the bottom when it can’t be selected as #1 any more.

What it means is that up to around 0.12 – 0.15 ber (equates to 15 – 19 randomly selected flipped bits of the 128 in the pattern) you can detect the pattern VERY reliably. Any higher ber – with randomly selected bit errors – and your probability of detecting the pattern is very low.

Multibit dropout tolerance

From my WiFi work I know that a common failure mode in RF packets is a multibit continuous dropout, that’s different from the random bit errors introduced above. These graphs show the effect on worst correct offset margin from dropouts of all possible lengths randomly placed in the packet, where the dropout is filled with white noise, all zeroes or all ones.



Clearly it is beautifully insensitive to multibit contiguous dropouts. If the problem is that you have white noise crapping on the transmission, the loss of 39 contiguous bits can be sustained without dropping below the +64 result limit. If the problem is events that cause continuous static 1 or 0 to be read during the disturbance, the code is very insensitive to this and can still be detected with fully half of the bits sequentially zero’d out or up to 50 set to ‘1′. So the sync detection performance faced with contiguous dropouts actually exceeds that of random dropouts.

This last dropout graph shows performance when there are TWO dropped-out areas randomly (5,000 runs at each dropout length) placed in the packet at various dropout lengths (the dropout length is the same for both and they can overlap, explaining the noise at the end as they grow larger).

Again looking at the absolute result values for the graph (blue line) the optimal absolute result cutoff of +64 is seen at two blocks of 18 contiguous bits contaminated with noise. These are very severe insults that still allow a correct sync detection.

Conclusion

This means (to the accuracy of these simulations) if you draw a line at 15% bit-error rate, if you ever see any offset of the correlator giving an absolute result of +64 or better, there is a very high probability that:

This is a very robust correlator pattern! It can be improved further: at the moment the “score” for correlation adds 1 for a matched binary bit level and subtracts 1 for a binary mismatch. If the demodulator that is providing these bits gives a probability of a ‘1′ or a ‘0′ instead of a binary ‘1′ or ‘0′, then the result can be made from more information. A few “looks a bit like a 0″ inputs will more weakly override many “definitely a 1″ inputs, for example.

There is another great advantage to interleaving this pattern with the payload. If the sync pattern can be recovered considering the 15% bit-error rate that is allowed, it is possible to identify then which bits of the pattern were corrupted. Because the correlator code bits are interleaved with the payload, it suggests that if the payload is broken, that the problem is coming from the payload bits next to the known-bad correlator code bits. For example, if it is shown that say three contiguous bits of the correlator code channel are wrong, one has to wonder about the two payload bits that are inbetween them. If there are a small number of bits involved, it can be possible to “fuzz” the suspected bad payload bits to see if an otherwise unrecoverable ECC error can be solved.

One more advantage is that the robustness margin of 15% allows channel bit-error rate to be continually assessed during reception.

However removing the black magic the claim of 100% recovery at -22dB SNR using another “forbidden” but less magical technology is not being disputed. This is a patent-encumbered “soft” Reed-Solomon decoder which is able to recover from more damage faster than the normal “hard decision” decoder: this means you have to give up another few dB to get a distributable implementation. An open source implementation exists at berlios but it’s written in freaking Fortran. Multithreaded Fortran with a Python GUI. This provides a normal Reed-Solomon FEC implementation which is used if you don’t have the external forbidden one.

One awfully limiting “trick” and two really interesting techniques are used in the protocol. The bad news is that very very long symbol-times are used for transmission, 372ms per 6-bit symbol. Considering the various bloatages it’s about one byte per two seconds. They are sending one of 64 “tones” to encode the six bits during that time… obviously the symbol duration helps with recovery. This “trick” is the core feature of weak signal recovery… repeat what you are doing a lot, in this case repeat the “tone” cycles a lot to “amplify” the signal at a receiver which knows how to take advantage of looking for something happening multiple times to increase probability of detection.

The first interesting trick is just the amount of Reed-Solomon used… this is not new to me since I used it as part of Penumbra. But in this protocol, every 72-bit packet has an additional 306 bits of error correction attached to it :-O. That’s more than 4 times as much ECC as data, and despite that it still pays off for capturing the signal.

The second cool technique is to interleave the payload data with a binary autocorrelation “clock”. Since the noise level is so crazy, it’s of little use to expect a 1-bit channel in the data to be usable as a “start of frame” marker or somesuch as you would normally expect with digital serialized communication. Instead, they spread the sync information in this interleaved “channel” using a 126-bit sequence which has a magically cool property… if you autocorrelate the sequence with itself, even in the presence of a fair bit of noise, every correlation offset except the right one matches MUCH worse than the 1:1 lineup. Here is the sequence extended to 128 bits and correlating with itself. The y axis is the number of bits that match…. obviously that is 128 when it compares itself to itself at the 0 offset on the X axis. The cool part is how low the self-correlation is everywhere else, no better than 20, or a 14dB “SNR” between a match and a non-match.

This remains the case even under pretty bad noise, up to 25% of the bits being trashed (still 9dB sync SNR):

but at 30% of the bits being trashed, the performance falls off a cliff:

Not only does the noise floor rise due to falsely improved correlations, but the one true correlation is also falsely degraded. After about 28% bit errors the reliability is gone. (Note the noise is one-shot with the test program, rather than being Monte Carlo’d, but I ran it several times and the graphs shown are representative).

But that isn’t the end of the story for this code. First the correlation action is a filter for transmission presence all by itself. And if you detect the transmission by the presence of the correlation code, you have also sync’d the receiver to the transmitted frame, since the correlation bits are interleaved with the actual data and the “0″ offset marks the start of the frame.

With deep memory and a known period of retransmission from the source, temporally averaged autocorrelation can take place to increase the chances to find the presence of a transmitter and to sync up to its data. After a transmitter “sync” has been found in the averaged data with high probability, the averaging memory can be turned to only store the times when a transmission was expected from the known schedule of the transmitter.

Here is the magic code with the 128-bit sequence and the test loops

#include 

static unsigned int u8Auto[] = {
	0x19, 0xbf, 0xa2, 0x89, 0xf3, 0xf6, 0x58, 0xcd,
	0x2a, 0x81, 0x01, 0x4b, 0xab, 0x4c, 0xc2, 0xbf
 };

#define AC_LEN 128

char GetAc(int n)
{
	n = n & (AC_LEN - 1);
	return (u8Auto[n >> 3] >> (n & 7)) & 1;
}

int main(int argc, char ** argv)
{
	int n, n1;
	int nSum;
	int nNoise = 0;
	int nSeed;
	FILE *f = fopen("/dev/urandom", "r");

	fread(&nSeed, sizeof(nSeed), 1, f);
	fclose(f);
	srand(nSeed);

	if (argc == 2)
		nNoise = (1024 * atoi(argv[1])) / 100;

	fprintf(stderr, "Noise: %d%%\n", nNoise);

	for (n = -(AC_LEN - 1); n < AC_LEN; n++) {
		nSum = 0;
		for (n1 = 0; n1 < AC_LEN; n1++) {
			char c = GetAc(n + n1);
			/* simulate white noise */
			if ((rand()&1023) < nNoise)
				c = c ^ 1;

			if (GetAc(n1) == c)
				nSum++;
			else
				nSum--;
		}
		printf("%d %d ", n, nSum);
	}
	return 0;
}

and the graph command that generated the graphs (the 28 is the percentage of noise to graph)

gcc test.c -o test ; ./test 28 | graph -Tpng --bitmap-size 1200x1200 -FHersheySans>temp.png && convert temp.png -scale 300x300 png:temp1.png