Mbed3 registry and deps

October 31, 2015

Dependency Hell mbed3 style

As I mentioned although yotta is being Make in terms of 'yotta clean' and 'yotta build', it also feels it should be a package manager, where the "packages" are "yotta modules", ie, chunks of code. And the logic that led to that, leads to the need for a repo to get packages from, which they are bravely calling a "registry".

mbed3 projects all have some JSON to describe them, including versioning, which is a good idea, so the metadata to manage this is around.

Yotta can dump his local package state... as we left it in the last post, I can't build mbed-client-examples and my package state is like this

$ yotta ls
mbed-client-examples 1.0.0
┣━ mbed-client 1.2.0
┃ ┣━ mbed-client-c 1.1.1 yotta_modules/mbed-client-c
┃ ┃ ┗━ nanostack-libservice 3.0.8 yotta_modules/nanostack-libservice
┃ ┣━ mbed-client-linux 1.1.0 yotta_modules/mbed-client-linux
┃ ┗━ mbed-client-mbedtls 1.0.7 yotta_modules/mbed-client-mbedtls
┃   ┗━ mbedtls 2.2.0-rc.1 yotta_modules/mbedtls
┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ cmsis-core 1.0.1 yotta_modules/cmsis-core
┃ ┣━ core-util 1.0.1 yotta_modules/core-util
┃ ┃ ┗━ ualloc 1.0.2 yotta_modules/ualloc
┃ ┃   ┗━ dlmalloc 1.0.0 yotta_modules/dlmalloc
┃ ┗━ minar 1.0.1 yotta_modules/minar
┃   ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
┃   ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
┃     ┗━ minar-platform-posix * missing
┗━ mbed-example-network 0.1.8
  ┗━ mbed-drivers 0.6.9 yotta_modules/mbed-drivers
    ┗━ mbed-hal 0.6.4 yotta_modules/mbed-hal

So minar-platform-posix package / module / chunk of code is needed by minar-platform, which came from the "registry" presumably, but

$ yotta install minar-platform-posix
info: get versions for minar-platform-posix
error: minar-platform-posix does not exist in the modules registry. Check that the name is correct, and that it has been published.

When it comes to targets, posix != mbed

So you might think the state of their registry is inconsistent and incomplete right now, but it seems some other platform incompatibility might be involved. When I look at minar-platform JSON

https://github.com/ARMmbed/minar-platform/blob/master/module.json

  "dependencies": {},
  "targetDependencies": {
    "mbed": {
      "minar-platform-mbed": "^1.0.0",
      "cmsis-core": "^1.0.0"
    },
    "posix": {
      "minar-platform-posix": "*"
    }
  },

So from this we can see "posix" and "mbed" are mutually exclusive platforms that can be targeted by... what... not mbed3... yotta then, and have completely separate dependency requirements.

I assumed posix apis was some plugin to mbed3, since it appears as a module that can be installed. But it seems to be aimed at targets that already have a posix OS, like Linux, and you choose to build for one or the other. Actually if posix stuff can't work in mbed3 I don't want posix stuff right now, I just want to see mbed3 do something, other than blink an LED, that I have sources for end to end on this K64F board.

Looking at the deps, the guy who wants "posix" pieces is none other than minar

  ┃ ┣━ minar 1.0.1 >=0.6.0,<0.7.0 yotta_modules/minar
  ┃ ┃ ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
  ┃ ┃ ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
  ┃ ┃   ┗━ minar-platform-posix * missing

But as we learned yesterday, minar is an mbed3-specific event loop and I'm building for mbed3, not posix (at least, I hope I am).

digging into the registry

There's a http registry frontend here

https://yotta.mbed.com/

and despite there is no catalogue, you can see everything in there with

https://yotta.mbed.com/#search/*

which is a bit perplexing, 6 of the 60 packages are dummy tests, 20 of them are related to embedded targets as you would expect (K64f-gcc, Nordic variations, etc), 8 are related to x86_64 or osX / iOS... it seems yotta predates mbed3 and has another life somewhere. The rest are mainly frameworks related to security etc. There's nothing in there like "ethernet and tcp support for k64f".

Digging into Yotta

Yotta has its own site on the mbed.com domain related to mbed3

http://yottadocs.mbed.com

On there is s tutorial

http://yottadocs.mbed.com/tutorial/tutorial.html /

In ARM we use yotta to build software for embedded devices - not just desktop computers. When you're compiling the same software for lots of different devices you need a mechanism to do different things, and often to include different dependencies, for each of the different devices.

The yotta target command lets you do this. It defaults to the system you're building on (x86-osx-native on mac, x86-linux-native on linux, etc.) You can display the current target by running yotta target with no argum

Oh so...

[agreen@build mbed-client-examples]$ yotta target
x86-linux-native 1.0.0
linux-native 1.0.0

... it looks like the active targets get stored in the project directory. So I saw k64f-gcc in the 'registry'...

[agreen@build mbed-client-examples]$ yotta target k64f-gcc
info: get versions for k64f-gcc
info: download k64f-gcc@0.0.1 from the public module registry
warning: k64f-gcc has invalid target.json:
warning:    value OrderedDict([(u'name', u'k64f-gcc'), (u'version', u'0.0.1'), (u'similarTo', [u'k64f', u'ksdk-mcu', u'mk64fn1m0vmd12', u'mk64fn1m0', u'mk64fn', u'freescale', u'cortex-m4', u'armv7-m', u'arm', u'gcc', u'*']), (u'toolchain', u'CMake/toolchain.cmake'), (u'debug-server', [u'echo', u'debug server command not defined']), (u'debug', [u'echo', u'debug command not defined'])]) is not valid under any of the given schemas
warning:   similarTo.10 value u'*' does not match u'^[a-z]+[a-z0-9-]*$'
warning:   similarTo.10 value u'*' is too short

Ha... the registry seems to have some rotted things in it. The fact 10% of it were temporary test pushes suggests that might be an ongoing problem until they add a catalogue so people maintaining it can see what needs deleting or updating. Better yet get rid of it and replace with a git tree: that way global state of packages can be tagged and returned to if the packageset loses selfconsistency.

Since the targets seem to be per 'project directory', I went back to my modified blinky I had been able to build and looked at its targets

[agreen@build test1]$ yotta target
frdm-k64f-gcc 0.2.0
mbed-gcc 0.1.3

Yes he liked that

[agreen@build mbed-client-examples]$ yotta target frdm-k64f-gcc
info: get versions for frdm-k64f-gcc
info: download frdm-k64f-gcc@0.2.0 from the public module registry
info: get versions for mbed-gcc
info: download mbed-gcc@0.1.3 from the public module registry

I checked, that package does exist also in the registry. So they should delete the dead k64f-gcc package from there.

Change of dependency hell

Now the target contect for the project was changed to a k64f specific mbed3 one, yotta uses the mbed set of deps from the JSON, and these unfulfillable posix package deps disappear.

We get new missing deps, but they look like they are on the right track

agreen@build mbed-client-examples]$ yotta ls
mbed-client-examples 1.0.0
┣━ mbed-client 1.2.0
┃ ┣━ mbed-client-c 1.1.1 yotta_modules/mbed-client-c
┃ ┃ ┗━ nanostack-libservice 3.0.8 yotta_modules/nanostack-libservice
┃ ┣━ mbed-client-mbed-os ^1.0.0 missing
┃ ┗━ mbed-client-mbedtls 1.0.7 yotta_modules/mbed-client-mbedtls
┃   ┗━ mbedtls 2.2.0-rc.1 yotta_modules/mbedtls
┃     ┗━ cmsis-core 1.0.1 yotta_modules/cmsis-core
┃       ┗━ cmsis-core-freescale ^1.0.0 missing
┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ sal-stack-lwip ^1.0.0 missing
┃ ┣━ core-util 1.0.1 yotta_modules/core-util
┃ ┃ ┣━ ualloc 1.0.2 yotta_modules/ualloc
┃ ┃ ┃ ┗━ dlmalloc 1.0.0 yotta_modules/dlmalloc
┃ ┃ ┗━ mbed-drivers 0.6.9 >=0.11.1,<0.12.0 yotta_modules/mbed-drivers
┃ ┃   ┗━ mbed-hal 0.6.4 yotta_modules/mbed-hal
┃ ┃     ┗━ mbed-hal-freescale ~0.5.0 missing
┃ ┗━ minar 1.0.1 yotta_modules/minar
┃   ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
┃   ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
┃     ┗━ minar-platform-mbed ^1.0.0 missing
┗━ mbed-example-network 0.1.8

Clearly we're on the right path, but we are not able to satisfy everything from the registry

[agreen@build mbed-client-examples]$ yotta install mbed-client-mbed-os
info: get versions for mbed-client-mbed-os
info: download mbed-client-mbed-os@1.1.0 from the public module registry
info: dependency mbed-client-mbed-os: ^1.1.0 written to module.json
info: get versions for cmsis-core
info: download cmsis-core@0.2.7 from the public module registry
info: get versions for ualloc
info: download ualloc@0.0.10 from the public module registry
info: get versions for minar
info: download minar@0.6.7 from the public module registry
info: get versions for core-util
info: download core-util@0.0.16 from the public module registry
info: get versions for mbed-hal-freescale
info: download mbed-hal-freescale@0.5.2 from the public module registry
info: get versions for mbed-hal-ksdk-mcu
info: download mbed-hal-ksdk-mcu@0.5.7 from the public module registry
info: get versions for uvisor-lib
info: download uvisor-lib@0.7.25 from the public module registry
info: get versions for mbed-hal-k64f
info: download mbed-hal-k64f@0.3.6 from the public module registry
info: get versions for mbed-hal-frdm-k64f
info: download mbed-hal-frdm-k64f@0.4.6 from the public module registry
info: get versions for cmsis-core-freescale
info: download cmsis-core-freescale@0.1.4 from the public module registry
info: get versions for cmsis-core-k64f
info: download cmsis-core-k64f@0.1.5 from the public module registry
info: get versions for dlmalloc
info: download dlmalloc@0.0.6 from the public module registry
info: get versions for compiler-polyfill
info: download compiler-polyfill@1.0.4 from the public module registry
info: get versions for minar-platform
info: download minar-platform@0.3.4 from the public module registry
info: get versions for minar-platform-mbed
info: download minar-platform-mbed@0.1.5 from the public module registry
info: get versions for sal-stack-lwip
info: download sal-stack-lwip@1.0.1 from the public module registry
info: get versions for sal-driver-lwip-k64f-eth
info: download sal-driver-lwip-k64f-eth@1.0.2 from the public module registry
info: get versions for sal-iface-eth
info: download sal-iface-eth@1.0.0 from the public module registry
error: sockets does not meet specification ~0.3.0 required by mbed-example-network
error: mbed-drivers does not meet specification ~0.7.0 required by minar
error: cmsis-core does not meet specification ^1.0.0 required by mbedtls
error: minar does not meet specification ^1.0.0 required by sockets
error: core-util does not meet specification ^1.0.0 required by sockets

Clue about mbed3 hal and driver structure

As an aside even though this thing is stuck in dependency hell, there are interesting signs of how the SoC and board-specific IP are handled. Sockets support lookslike it calls through to a "Socket Abstraction Layer" when then instantiates drivers in a HAL...

┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ sal-stack-lwip 1.0.1 yotta_modules/sal-stack-lwip
┃ ┃   ┣━ sal-driver-lwip-k64f-eth 1.0.2 yotta_modules/sal-driver-lwip-k64f-eth
┃ ┃   ┗━ sal-iface-eth 1.0.0 yotta_modules/sal-iface-eth
┃ ┃     ┗━ mbed-drivers 0.6.9 yotta_modules/mbed-drivers
┃ ┃       ┣━ mbed-hal 0.6.4 yotta_modules/mbed-hal
┃ ┃       ┃ ┗━ mbed-hal-freescale 0.5.2 yotta_modules/mbed-hal-freescale
┃ ┃       ┃   ┗━ mbed-hal-ksdk-mcu 0.5.7 yotta_modules/mbed-hal-ksdk-mcu
┃ ┃       ┃     ┣━ uvisor-lib 0.7.25 yotta_modules/uvisor-lib
┃ ┃       ┃     ┗━ mbed-hal-k64f 0.3.6 yotta_modules/mbed-hal-k64f
┃ ┃       ┃       ┗━ mbed-hal-frdm-k64f 0.4.6 yotta_modules/mbed-hal-frdm-k64f

That looks like a pretty nice way to deal with the messy underlying reality. I'm still not sure how 'drivers' getting interrupts in the hal part trigger minar events, but presumably there is a way to register and trigger events in the HAL that become serialized and queued at minar.

Still stuck

At any rate the remaining problems look like inconsistent content in the 'registry'

error: sockets does not meet specification ~0.3.0 required by mbed-example-network
error: minar does not meet specification ^1.0.0 required by sockets
error: mbed-drivers does not meet specification ~0.7.0 required by minar
error: cmsis-core does not meet specification ^1.0.0 required by mbedtls
error: core-util does not meet specification ^1.0.0 required by sockets

For example mbed-drivers I just got from the registry is 0.6.9 but minar I got from the registry demands 0.7.0. That's something that could be eased a lot if the last known working packageset could still be referenced (a 'stable' registry, or put it all under git).

What we learned this time

Next post about mbed