HDMI Audio on Hikey

Audio on 96boards

96boards designs differ from the usual Raspberry Pi and suchlike in a few ways... one obvious one is there is no Ethernet, instead WLAN is provided.

For audio, the default audio device is HDMI audio, and only raw i2s is provided at the expansion connectors.

Since I'm the only Linaro person with an HDMI analyzer, the job of getting HDMI Audio working ended up with me.

Signal chain for HDMI Audio on Hikey

Basically the audio is DMA'd to a FIFO + mixer + I2S IP, where it comes out on a physical 48kHz "BT" I2S interface and is taken by the ADV7533.

There's no HDMI IP onchip and so for the SoC, "HDMI Audio" is just a generic 2ch I2S stream.

One complication is the "BT" I2S interface wiring is not probeable on the PCB, so any work has to be done blind. That's a bit scary if it just doesn't produce any output, since you could have a problem with, eg, pinctrl, or clock, or reset, and have no way to know if anything was even coming out.

Getting started

The first part was to get the logical alsa "card" coming in Linux.

To make Alsa happy, it requires several pieces in place

• A driver for the HDMI "Card"
• A driver for the I2S hardware... in our case this also does the job of forcing the mixer that's combined with the I2S hardware to be configured for the output we want
• A PCM driver to arrange the DMA to move buffers of data to the I2S hardware
• A (fake) codec driver representing the I2S -> HDMI connection

I ported the I2S driver I wrote for another project along with Jassi Brar's dmaengine PCM driver, and enough fake hdmi "codec" driver that it could all register and provide the alsa card

root@linaro-alip:~# cat /proc/asound/cards
 0 [hi6210hdmi     ]: hi6210-hdmi - hi6210-hdmi
                      hi6210-hdmi

Next, I gutted the I2S driver and customized it for the hisilicon IP. A large part of that is making the register and bitfield definitions, then initializing more and more of the chip until it showed some signs of life.

Extreme HDMI monitoring

Normally when you do this kind of thing the first place you look to understand the status is the i2s bus coming out of the SoC, because there are too many things that can stop the signal before it makes it to the TV after that. So you at first want to at least be certain that the format and rate of the data coming out of the SoC is right.

Again normally, after the data makes it to the HDMI cable you are again blind, you have to debug it by listening to the TV.

However the situation this time is reversed... there is no convenient way to access the I2S bus between the SoC and the ADV7533... we are actually blind there. When it comes to the HDMI cable, because of the HDMICAP analyzer I described last month, actually we have perfect insight into what came out of the SoC (up to 720p anyway) on HDMI.

The HDMI monitoring scheme is like this:

Because I didn't finish implementing EDID support in HDMICAP, to give the rest of the stack a valid EDID an active HDMI splitter is used, and the capture analyzer works from a digital copy of the HDMI stream from the splitter. The TV provides his EDID back to the Hikey.

First trial

I tried DMA transfer at that point since the PCM driver already supported dmaengine as does the hikey DMA driver. However, I couldn't get any result from making the DMA write to the I2S FIFO.

He would enqueue two DMA actions but not transfer anything and never complete until Alsa's 10s timeout closed it as an "IO Error". Well, at least it closed cleanly.

I checked his dma request channel (14) was correct, his IRQ was correct and that he was setting it up inside the hisilicon DMA driver... but no data could move.

Somewhat suspiciously I noticed nobody used the dmac driver in the DT, although it was upstream. I described the situation to Guodong at the Hisilicon Landing Team in Linaro and some contacts at Hisilicon, and tried some other things in the meanwhile.

Operating the FIFO by hand

After spending a while fiddling with non-working DMA, I enabled the I2S IRQ. Normally can avoid dealing with that since exhausting the DMA buffer regulates the data flow, but with DMA not passing samples into I2S the question was whether the problem belonged on DMA or I2S side. If the CPU can fill the I2S FIFO, the problem would be on DMA side.

I attached the IRQ to an ISR that dumped samples into the FIFO from the CPU side, basically by writing to a 32-bit register that's normally the destination for the DMA. I also intended to study the rate of IRQs to infer if the sample rate was correct or not.

Since I can't look at the BT I2S signals due to lack of access, I also hacked in a fixed 48kHz Stereo Audio enable in the ADV7533 driver... that involved adding packet mode support (which I tested with SPD: the hikey with these patches has an HDMI SPD of "Linaro" and "96boards:hikey" captured by HDMICAP.)

Noises off

That got me some noise from my TV. However the rate of interrupts from using the i2s fifo IRQ in this mode was completely wrong... the IRQ should only come when I2S has used the FIFO content below a low-water mark, and the ISR stuff samples in the FIFO then.

But the IRQ was continuously asserted, meaning the ISR spammed new samples in there endlessly without any regulation of what it was doing. The samples were being taken and sent to adv7511 at some slower rate, and I could capture the audio packets and confirm the content was coming from the ISR (who was writing 0x11112222, 0x33334444, etc). But because the ordering was essentially random, we heard basically white noise.

Although that is not quite what we wanted, in terms of which bits must be working / wired up correctly / configured to pass data, it means almost everything was correct already: we could pass data from the FIFO to the TV correctly.

In other words the only thing killing us was broken DMA: the rest of it was already at least half-working.

Secure or Convenient: pick one

That evening I heard from Hisilicon they had figured out the DMA issue: the DMA controller had been configured for Secure access only. Hikey has a proper "Secure World" bootloader implementation - it's open source, as is the whole Hikey boot stack - and the other bootloaders and the kernel run in nonsecure mode.

That means the DMA couldn't work in Linux without a bootloader update. I worked with Socionext guys last year to implement a secure / nonsecure bootloader on a big.LITTLE chip, so this is nothing new actually.

The next morning I received a patch and binary versions of the secure pieces, so I blew them in the Hikey partitions... and bricked my Hikey.

Painkillers

Right now there's no real pressure on Linaro not to break boot on 96boards. So, changes are repeatedly made, for very good reasons, that are incompatible with the latest and greatest that went before. And if you say to people they should make the changes backwards-compatible, they look at you like you are crazy and feel free to ignore such wild ideas. My boot pieces were only a few weeks old but things had moved on and using the newer binaries broke boot. This has been happening repeatedly since Feb 2015 when I got my Hikey.... I'm used to things blowing up but since this is now a hardware product, like the other 96boards, I fear not everybody that buys them are going to be minded that it gets bricked regularly unless they "year zero" it every time they update anything.

It didn't crash, to its credit, but the non-core EFI pieces had been moved to a new path. Since this stuff goes on an eMMC, there's no access to repair it... among the pieces that moved was fastboot EFI module.

My hikey has a 1.8V UART hacked on UART0, which used to the the UART the fashionable people used. But from some time ago, everything switched to UART3, and no mercy for people using UART0. So I shrugged and rewired it to UART3.

I managed to work around the EFI breakage and blew a newer boot partition. But there too, different people had decided to make other incompatible changes - again, no doubt, for the better - and moved the kernel + dtb from the boot partition to /boot in the rootfs. It doesn't, you know, fall back to the kernel in the boot partition. It just drops dead.

So it necessitated flashing a newer rootfs... I guess at some point, when there are hundreds of thousands of these out there, real users will have real things in their rootfs they do not want bricked and have to overwrite the whole partition to get a boot. But for now nobody is taking care about those guys.

At any rate I recovered after some hours lost to this and found the DMA bootloader fix worked great. But now it was working, we could hear the next problem, the audio playback on the TV was too slow and corrupted by regular noise.

Information from the capture side

When passing audio, the HDMI frame looks like this

The blanking time that is normally spent sending control period coding (just passing HSYNC and VSYNC information) gets some extra data islands, that carry the audio data. HDMI leaves it up to the transmitter to decide how many samples he will send in a frame, and he can decide to place a 36-byte raw data island anywhere in the blanking where there is enough space.

The PCM packet itself always has space for 8 x 24-bit audio samples: there's a bitfield comes with the packet describing which ones contain valid sample data. 16-bit data is simply shifted to be the MS 16 bits of the 24-bit sample. In this way, HDMI is natively 8ch, 24-bit and able to accept a range of sample rates.

How the 8 possible samples map on to the >8 possible audio outputs is not in the PCM packet but defined in another Audio Infoframe packet sent once per frame.

We are sending these telling it there are 2ch and they should map on "Front Left" and "Front Right".

Audio Infoframe (v1, len 10)
hdr 0x84 0x01 0x0a (pol 0x4a)
  70 01 00 00 00 00 00 7d
  00 00 00 00 00 00 00 00
  00 00 00 00 00 00 00 00
  00 00 00 00 00 00 00 00

Also part of the summary information HDMICAP reports is the number of these PCM audio packets received in one frame. Doing the arithmetic, 48000 audio samples per second spread across ~60 video frames per second, we expect to see ~800 audio packets (audio sample sets) per video frame like this:

However what I was seeing was a number around 665, which is too low by 16%.

Reconstructing the audio

Because HDMICAP captures everything on the HDMI wire, it's possible to capture all the audio packets and 100% reconstruct the audio samples. In fact to allow larger captures, HDMICAP has a mode where only data island content is captured: it's then possible to fill the 8MB DMA capture region with typically >1000 frames (16-20s at 50/60Hz) of audio content.

When I did this and saved the result as a 48kHz wav, there was no problem playing it back on a PC: it sounded normal.

Looking at it closer though, the amount of captured audio was too short by 16% for the 200 video frames (3.333s) I had run the capture for. So the problem is not about corruption but simply providing the correct samples too slowly, at about 40kHz: the TV replayed them at 48kHz and played junk for the missing samples making the corruption.

Inferring the clock

Normally you would just put a 'scope on the I2S clock and confirm the overall clock rate was correct, but since we can't touch that signal without reworking the Hikey, we have to look at it differently.

Clearly the samples are coming from the SoC at a rate that is -16% of what it should be.

The I2S unit runs at 49.152MHz from a 245.76MHz PLL, which can be had by dividing it by 5... and there was some code I added to the driver while trying to get it working that forced a value of 5 in the divider. But with the other evidence, I wondered if to divide by n, you should put n - 1 in the divider register.... when I poked 4 in there, the problem was solved.

Afterwards I removed the whole forcing because the clock driver had already correctly set it by default, making it somewhat of a self-inflicted wound.

Current Status

As of writing this, you can get the current HDMI stuff here

https://git.linaro.org/people/andy.green/linux.git/shortlog/refs/heads/hikey-audio

You will also need the very newest boot pieces, they were fixed a couple of days ago but I don't know which build will have them.

There seem to be two problems left

• At some point the audio becomes mono, just the first channel also repeated on the second channel. I have asked Hisi how that's possible... actually from the (proprietary) datasheet for the I2S unit I can't see how to do that even if I wanted to. This could also be introduced at adv7533 but again from that (proprietary) datasheet the only way I could see to do it (i2s -> HDMI channel mapping bitfields) don't do it.

• there is a small sound at the start of playback, it seems the wrong data is sent initially somehow

Anyway these are relatively small problems, it's possible to just aplay whatever.wav if it's 48kHz 2ch 16-bit and audio will come.