Time for some practical experiments with the robustness of the magic correlator code. Rather than build any hardware, although some interesting RF hardware is available, I decided to first model the system in software, so I can change things around much easier while there is a lot uncharacterized about the performance and capabilities of the coding.
Testing plan
The general plan is to bind payload data to the magic correlator code, such that the correlator code alignment acts as both a frame sync and assurance that the recovered bit clock is correct (in fact it should act as a clock recovery synthesis pilot as well, since we know the sequence). Because of the properties of the correlator code, it should be possible to just add FEC-coded payload without further ado, either interleaved bit-by-bit or bound another way.
At the moment I am sending plain payload without FEC. Further I am going to initially do all the testing at baseband directly, in fact specifically at audio frequencies. This will eventually allow real world testing using a laptop "transmitting" the coding through its speakers and my moving another microphone-enabled laptop around the house seeing how far we can push the data recovery vs what I can hear myself. My house it pretty noisy, with cars going by, kids jumping around and so on, there is a good mix of whitish noise and short duration dropout crud in the audio spectrum. I am really interested to see how far it can be pushed, particularly if averaging is possible to get working.
Modulation strategy
Currently the test C code modulates the data and the magic correlator code bits together on QPSK. QPSK has four "modulation states" encoded as four phase angles of the carrier, so you are signalling two bits per symbol -- one payload bit and one magic correlator code bit. Eventually, because we can recover even badly damaged correlator code quite often, it will be possible to score the likelihood of false payload recovery bit by bit based on looking back at the magic code bits that turned out to be wrong in a particular symbol: the payload and the coding bits were transmitted in the same symbol. This "distrust" indication per payload bit can open the door to some novel, if possibly expensive, error correction.
The current C code uses 44.1kHz sample rate and a 3kHz carrier modulated with QPSK carrying 300 baud symbols, ie, there are 10 carrier cycles per symbol. Each symbol contains two bits due to the QPSK coding. Maybe as we go on it will be necessary to drop the symbol rate to allow more cycles and better recovery, but this is a starting point. You can listen to the QPSK coded WAV file here, this has the magic code and a 16 byte ASCII message payload modulated on QPSK. The symbol transitions look like this:
So, the "transmitting" part is fine... don't be fooled by those little spikes where it changes phase, those easily lost spikes are not carrying the information. The whole rest of the bit period carrier goes on at the new phase after that discontiguity, the next ten cycles of carrier phase encodes the information. The spike could and probably should be completely filtered out and not cause trouble on recovery.
QPSK symbol recovery
QPSK recovery requires a coherent oscillator in frequency and phase sync with the incoming carrier. You recover the symbols by watching the gyrations the local oscillator has to perform to keep the phase sync.
The RF guys were on to all this stuff back in ancient times, a version of a Costas loop capable of separately dealing the with quadrature "carriers" in QPSK was invented way back when. Its basic idea is to feed the coherent local oscillator to two mixers, one with the local oscillator via a 90 degree phase shift, then lowpass filter the result from the mixers and use that to feed back an error term to the local oscillator. The lowpassed mixer outputs are the "result" from the loop for the two bits encoded in the QPSK modulation.
Now described there and the literature one can find in Google, it's simple, right? Some of the PDFs on QPSK recovery from Google even had beautiful smooth digital recovery pictures. But the actual implementation is a lot trickier. The problems come from the need to tune various filters in the Costas loop, they need to be tuned for the carrier and VCO lock performance considering the symbol rate. I got started by looking at the Costas loop implementation in GnuRadio, but this has no filters at all in it (I guess this is for flexibility you can add the filters outside it). No doubt someone somewhere has been through all this back in 1970 and written up all the equations, but I couldn't find it. In the end I found some general advice about matching the lowpass filter 3db point to half the symbol rate and meddled around until it worked. Of course doing it in software I didn't even have the nightmares of filter matching for the two mixers which bedevil an analogue implementation. I also found a lot of 2 x carrier noise in my loop, which I tried to notch out with some success. Anyway here is what the recovery looks like right now, being fed "perfect" noise-free signal... it looks okay but I am pretty uncertain about how it will react to noise
QPSK absolute phase uncertainty
The receiver locked phase compared to the transmitter 0 degrees phase is unknown, therefore the decoded bits can appear on either of the two output bitstreams and be inverted. This has to be taken into account when looking at what you're getting. Initially at least you have to run the four possible correlations against the magic code to find out the effective phase offset / symbol coding you have locked at. I guess while the receiver does not lose lock (one can study the error term in the Costas loop I guess) you can just use the lock you previously determined.
Incidentally received absolute phase determination is yet another exploitation of the magic correlation code properties of robust matching.
Next stop
Next task is to recover the bit clock from the received bits and perform the correlation action, and to try to recover the message. All of will still be in pure software with no noise yet.
