Mbed3 registry and deps
Dependency Hell mbed3 style
As I mentioned although yotta is being Make in terms of 'yotta clean' and 'yotta build', it also feels it should be a package manager, where the "packages" are "yotta modules", ie, chunks of code. And the logic that led to that, leads to the need for a repo to get packages from, which they are bravely calling a "registry".
mbed3 projects all have some JSON to describe them, including versioning, which is a good idea, so the metadata to manage this is around.
Yotta can dump his local package state... as we left it in the last post, I can't build mbed-client-examples and my package state is like this
$ yotta ls
mbed-client-examples 1.0.0
┣━ mbed-client 1.2.0
┃ ┣━ mbed-client-c 1.1.1 yotta_modules/mbed-client-c
┃ ┃ ┗━ nanostack-libservice 3.0.8 yotta_modules/nanostack-libservice
┃ ┣━ mbed-client-linux 1.1.0 yotta_modules/mbed-client-linux
┃ ┗━ mbed-client-mbedtls 1.0.7 yotta_modules/mbed-client-mbedtls
┃ ┗━ mbedtls 2.2.0-rc.1 yotta_modules/mbedtls
┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ cmsis-core 1.0.1 yotta_modules/cmsis-core
┃ ┣━ core-util 1.0.1 yotta_modules/core-util
┃ ┃ ┗━ ualloc 1.0.2 yotta_modules/ualloc
┃ ┃ ┗━ dlmalloc 1.0.0 yotta_modules/dlmalloc
┃ ┗━ minar 1.0.1 yotta_modules/minar
┃ ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
┃ ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
┃ ┗━ minar-platform-posix * missing
┗━ mbed-example-network 0.1.8
┗━ mbed-drivers 0.6.9 yotta_modules/mbed-drivers
┗━ mbed-hal 0.6.4 yotta_modules/mbed-hal
So minar-platform-posix package / module / chunk of code is needed by minar-platform, which came from the "registry" presumably, but
$ yotta install minar-platform-posix
info: get versions for minar-platform-posix
error: minar-platform-posix does not exist in the modules registry. Check that the name is correct, and that it has been published.
When it comes to targets, posix != mbed
So you might think the state of their registry is inconsistent and incomplete right now, but it seems some other platform incompatibility might be involved. When I look at minar-platform JSON
https://github.com/ARMmbed/minar-platform/blob/master/module.json
"dependencies": {},
"targetDependencies": {
"mbed": {
"minar-platform-mbed": "^1.0.0",
"cmsis-core": "^1.0.0"
},
"posix": {
"minar-platform-posix": "*"
}
},
So from this we can see "posix" and "mbed" are mutually exclusive platforms that can be targeted by... what... not mbed3... yotta then, and have completely separate dependency requirements.
I assumed posix apis was some plugin to mbed3, since it appears as a module that can be installed. But it seems to be aimed at targets that already have a posix OS, like Linux, and you choose to build for one or the other. Actually if posix stuff can't work in mbed3 I don't want posix stuff right now, I just want to see mbed3 do something, other than blink an LED, that I have sources for end to end on this K64F board.
Looking at the deps, the guy who wants "posix" pieces is none other than minar
┃ ┣━ minar 1.0.1 >=0.6.0,<0.7.0 yotta_modules/minar
┃ ┃ ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
┃ ┃ ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
┃ ┃ ┗━ minar-platform-posix * missing
But as we learned yesterday, minar is an mbed3-specific event loop and I'm building for mbed3, not posix (at least, I hope I am).
digging into the registry
There's a http registry frontend here
and despite there is no catalogue, you can see everything in there with
https://yotta.mbed.com/#search/*
which is a bit perplexing, 6 of the 60 packages are dummy tests, 20 of them are related to embedded targets as you would expect (K64f-gcc, Nordic variations, etc), 8 are related to x86_64 or osX / iOS... it seems yotta predates mbed3 and has another life somewhere. The rest are mainly frameworks related to security etc. There's nothing in there like "ethernet and tcp support for k64f".
Digging into Yotta
Yotta has its own site on the mbed.com domain related to mbed3
On there is s tutorial
http://yottadocs.mbed.com/tutorial/tutorial.html /
In ARM we use yotta to build software for embedded devices - not just desktop computers. When you're compiling the same software for lots of different devices you need a mechanism to do different things, and often to include different dependencies, for each of the different devices.
The yotta target command lets you do this. It defaults to the system you're building on (x86-osx-native on mac, x86-linux-native on linux, etc.) You can display the current target by running yotta target with no argum
Oh so...
[agreen@build mbed-client-examples]$ yotta target
x86-linux-native 1.0.0
linux-native 1.0.0
... it looks like the active targets get stored in the project directory. So I saw k64f-gcc in the 'registry'...
[agreen@build mbed-client-examples]$ yotta target k64f-gcc
info: get versions for k64f-gcc
info: download k64f-gcc@0.0.1 from the public module registry
warning: k64f-gcc has invalid target.json:
warning: value OrderedDict([(u'name', u'k64f-gcc'), (u'version', u'0.0.1'), (u'similarTo', [u'k64f', u'ksdk-mcu', u'mk64fn1m0vmd12', u'mk64fn1m0', u'mk64fn', u'freescale', u'cortex-m4', u'armv7-m', u'arm', u'gcc', u'*']), (u'toolchain', u'CMake/toolchain.cmake'), (u'debug-server', [u'echo', u'debug server command not defined']), (u'debug', [u'echo', u'debug command not defined'])]) is not valid under any of the given schemas
warning: similarTo.10 value u'*' does not match u'^[a-z]+[a-z0-9-]*$'
warning: similarTo.10 value u'*' is too short
Ha... the registry seems to have some rotted things in it. The fact 10% of it were temporary test pushes suggests that might be an ongoing problem until they add a catalogue so people maintaining it can see what needs deleting or updating. Better yet get rid of it and replace with a git tree: that way global state of packages can be tagged and returned to if the packageset loses selfconsistency.
Since the targets seem to be per 'project directory', I went back to my modified blinky I had been able to build and looked at its targets
[agreen@build test1]$ yotta target
frdm-k64f-gcc 0.2.0
mbed-gcc 0.1.3
Yes he liked that
[agreen@build mbed-client-examples]$ yotta target frdm-k64f-gcc
info: get versions for frdm-k64f-gcc
info: download frdm-k64f-gcc@0.2.0 from the public module registry
info: get versions for mbed-gcc
info: download mbed-gcc@0.1.3 from the public module registry
I checked, that package does exist also in the registry. So they should delete the dead k64f-gcc package from there.
Change of dependency hell
Now the target contect for the project was changed to a k64f specific mbed3 one, yotta uses the mbed set of deps from the JSON, and these unfulfillable posix package deps disappear.
We get new missing deps, but they look like they are on the right track
agreen@build mbed-client-examples]$ yotta ls
mbed-client-examples 1.0.0
┣━ mbed-client 1.2.0
┃ ┣━ mbed-client-c 1.1.1 yotta_modules/mbed-client-c
┃ ┃ ┗━ nanostack-libservice 3.0.8 yotta_modules/nanostack-libservice
┃ ┣━ mbed-client-mbed-os ^1.0.0 missing
┃ ┗━ mbed-client-mbedtls 1.0.7 yotta_modules/mbed-client-mbedtls
┃ ┗━ mbedtls 2.2.0-rc.1 yotta_modules/mbedtls
┃ ┗━ cmsis-core 1.0.1 yotta_modules/cmsis-core
┃ ┗━ cmsis-core-freescale ^1.0.0 missing
┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ sal-stack-lwip ^1.0.0 missing
┃ ┣━ core-util 1.0.1 yotta_modules/core-util
┃ ┃ ┣━ ualloc 1.0.2 yotta_modules/ualloc
┃ ┃ ┃ ┗━ dlmalloc 1.0.0 yotta_modules/dlmalloc
┃ ┃ ┗━ mbed-drivers 0.6.9 >=0.11.1,<0.12.0 yotta_modules/mbed-drivers
┃ ┃ ┗━ mbed-hal 0.6.4 yotta_modules/mbed-hal
┃ ┃ ┗━ mbed-hal-freescale ~0.5.0 missing
┃ ┗━ minar 1.0.1 yotta_modules/minar
┃ ┣━ compiler-polyfill 1.1.1 yotta_modules/compiler-polyfill
┃ ┗━ minar-platform 1.0.0 yotta_modules/minar-platform
┃ ┗━ minar-platform-mbed ^1.0.0 missing
┗━ mbed-example-network 0.1.8
Clearly we're on the right path, but we are not able to satisfy everything from the registry
[agreen@build mbed-client-examples]$ yotta install mbed-client-mbed-os
info: get versions for mbed-client-mbed-os
info: download mbed-client-mbed-os@1.1.0 from the public module registry
info: dependency mbed-client-mbed-os: ^1.1.0 written to module.json
info: get versions for cmsis-core
info: download cmsis-core@0.2.7 from the public module registry
info: get versions for ualloc
info: download ualloc@0.0.10 from the public module registry
info: get versions for minar
info: download minar@0.6.7 from the public module registry
info: get versions for core-util
info: download core-util@0.0.16 from the public module registry
info: get versions for mbed-hal-freescale
info: download mbed-hal-freescale@0.5.2 from the public module registry
info: get versions for mbed-hal-ksdk-mcu
info: download mbed-hal-ksdk-mcu@0.5.7 from the public module registry
info: get versions for uvisor-lib
info: download uvisor-lib@0.7.25 from the public module registry
info: get versions for mbed-hal-k64f
info: download mbed-hal-k64f@0.3.6 from the public module registry
info: get versions for mbed-hal-frdm-k64f
info: download mbed-hal-frdm-k64f@0.4.6 from the public module registry
info: get versions for cmsis-core-freescale
info: download cmsis-core-freescale@0.1.4 from the public module registry
info: get versions for cmsis-core-k64f
info: download cmsis-core-k64f@0.1.5 from the public module registry
info: get versions for dlmalloc
info: download dlmalloc@0.0.6 from the public module registry
info: get versions for compiler-polyfill
info: download compiler-polyfill@1.0.4 from the public module registry
info: get versions for minar-platform
info: download minar-platform@0.3.4 from the public module registry
info: get versions for minar-platform-mbed
info: download minar-platform-mbed@0.1.5 from the public module registry
info: get versions for sal-stack-lwip
info: download sal-stack-lwip@1.0.1 from the public module registry
info: get versions for sal-driver-lwip-k64f-eth
info: download sal-driver-lwip-k64f-eth@1.0.2 from the public module registry
info: get versions for sal-iface-eth
info: download sal-iface-eth@1.0.0 from the public module registry
error: sockets does not meet specification ~0.3.0 required by mbed-example-network
error: mbed-drivers does not meet specification ~0.7.0 required by minar
error: cmsis-core does not meet specification ^1.0.0 required by mbedtls
error: minar does not meet specification ^1.0.0 required by sockets
error: core-util does not meet specification ^1.0.0 required by sockets
Clue about mbed3 hal and driver structure
As an aside even though this thing is stuck in dependency hell, there are interesting signs of how the SoC and board-specific IP are handled. Sockets support lookslike it calls through to a "Socket Abstraction Layer" when then instantiates drivers in a HAL...
┣━ sockets 1.0.2
┃ ┣━ sal 1.0.2 yotta_modules/sal
┃ ┃ ┗━ sal-stack-lwip 1.0.1 yotta_modules/sal-stack-lwip
┃ ┃ ┣━ sal-driver-lwip-k64f-eth 1.0.2 yotta_modules/sal-driver-lwip-k64f-eth
┃ ┃ ┗━ sal-iface-eth 1.0.0 yotta_modules/sal-iface-eth
┃ ┃ ┗━ mbed-drivers 0.6.9 yotta_modules/mbed-drivers
┃ ┃ ┣━ mbed-hal 0.6.4 yotta_modules/mbed-hal
┃ ┃ ┃ ┗━ mbed-hal-freescale 0.5.2 yotta_modules/mbed-hal-freescale
┃ ┃ ┃ ┗━ mbed-hal-ksdk-mcu 0.5.7 yotta_modules/mbed-hal-ksdk-mcu
┃ ┃ ┃ ┣━ uvisor-lib 0.7.25 yotta_modules/uvisor-lib
┃ ┃ ┃ ┗━ mbed-hal-k64f 0.3.6 yotta_modules/mbed-hal-k64f
┃ ┃ ┃ ┗━ mbed-hal-frdm-k64f 0.4.6 yotta_modules/mbed-hal-frdm-k64f
That looks like a pretty nice way to deal with the messy underlying reality. I'm still not sure how 'drivers' getting interrupts in the hal part trigger minar events, but presumably there is a way to register and trigger events in the HAL that become serialized and queued at minar.
Still stuck
At any rate the remaining problems look like inconsistent content in the 'registry'
error: sockets does not meet specification ~0.3.0 required by mbed-example-network
error: minar does not meet specification ^1.0.0 required by sockets
error: mbed-drivers does not meet specification ~0.7.0 required by minar
error: cmsis-core does not meet specification ^1.0.0 required by mbedtls
error: core-util does not meet specification ^1.0.0 required by sockets
For example mbed-drivers I just got from the registry is 0.6.9 but minar I got from the registry demands 0.7.0. That's something that could be eased a lot if the last known working packageset could still be referenced (a 'stable' registry, or put it all under git).
What we learned this time
In each project dir you will build with Yotta, you must set 'yotta target', in my case to frdm-k64f-gcc, because it defaults to trying to build for your build OS
The package registry has quite a lot of deprecated and unusable junk piling up in it
The package registry has no history, if you put inconsistent deps in there you break the whole world with no recourse to last-known-good packagesets
Deps seem broken in there (minar vs mbed-drivers) at the time of writing
yotta seems to go to github to look for things if they're not in the 'registry' (it seems it looks for projects at https://github.com/ARMmbed)