Way of the exploding head

Well, today’s lesson was about libtool and what those curious .la files are for. I decided to go through the packages made so far (a couple of dozen) and start to clean out all the dirty hacks that I did to get started while I was learning. For example, I made a ~/.rpmmacros setting “crosspath” to set the active crosscompiler base path instead of embedding it in the spec files all over the place, and when I got started I embedded the literal path to the devel-filesystem directory (you install package-devel packages here so that other packages can link to those libs during crosscompile), now I use a relative path.

While I was at it, I decided to fix the few packages that I left getting stored into /usr/local instead of /usr, well, they ended up like that because I didn’t know what –bindir= was for in ./configure at that time. Curl and PHP fell into this category. Then I remembered that I had not understood how to link PHP to curl libraries at that time either and I could fix that too now. (When you meet something as big as ./configure and crosscompiling and packaging all at the same time, it is a terrible struggle just to get through to a successful compile and corners get cut, but once the process is familiar it’s embarrassing to go back and see what simple things need to be refactored from your early clueless threshing around.)

Anyway I recooked curl and then did PHP. To my surprise PHP kicked out right at the last link stage, it had attempted to link the host /usr/lib/libcurl.so instead of the target-compiled ones which in my system are installed into ./rpm/BUILD/devel-filesystem/usr/lib (by doing an rpm -i on the -devel package with a fake root).

I wandered around trying to understand this for a while changing env vars like LDFLAGS, but it was clear the -L path to my devel-filesystem tree had the right precedence and nothing was on the libtool commandline telling it to go to /usr/lib.

In the end as a test I moved my local /usr/lib/libcurl.so to /root and tried again — this time it linked fine using target libs as intended. Strange because it proved it was some kind of precedence issue, looking in local /usr/lib first and then using the right path if that didn’t exist. But I noticed a warning… it said that /blah/blah/libcurl.la “had moved”. I had a look in the la file — it is actually pretty cool, it is a text file that seems to be generated at library link time and contains paths to the libraries that it linked in. In this case curl has openssl in it, so that was shown and was correct. BUT what is this at the bottom?

# Directory that this library needs to be installed in:
libdir=’/usr/lib’

Hm a bit suspicious since my problems come from /usr/lib. I changed the path to point to my devel-filesystem path, and the build completed correctly!
I tried commenting out the libdir line in the .la file, but the link blows chunks complaining it can’t find the “library name” of libcurl.la. So the only way forward to allow usage of libs in the crosscompile environment seems to be to add a sed line to mess with the libdir setting in the library .la file!

The last thing that I missed from my oldstyle PC mailserver was the greylisting implementation for postfix, Postgrey. This was not trivial to port over to the new embedded ARM mailserver box because it was written in Perl. I spent considerable time trying to get Perl packaged and crosscompiled; I did get Perl itself to build but got lost trying to get its modules to build correctly (I since learnt more about libtool that might let me finish the job if I ever have to).

Eventually I realized my life was being wasted struggling with Perl, so I looked around for an alternative greylisting solution for Postfix written in something more reasonable. I must have been having a bad day, because I landed on tumgreyspf which is written in Python. Cue another couple of days spent beating my head on Python trying to package and crosscompile another crosscompile basketcase. In a slightly shorter period of time I realized my life was wasting away so this time I looked for a greylisting addon that did not need ten tons of arguably dying language dragged in with it, and I found gps. This is written in C++ (my kinda language) and just needed a library and sqlite, or so it seemed.

I managed to finally get sqlite to crosscompile and be packaged nicely today, so at last I was ready to attack gps. gps uses an interesting database abstraction library called libdbi. I packaged that, then noticed that it has a child project libdbi-drivers which actually contains the interfaces to the various SQL backends that are possible. Finally it was all packaged and sent over to the ARM board, after a little spring cleaning to make room in its 8MByte root filesystem.
Well gps is broken for use with sqlite3, which it turns out is different from “sqlite” target in the config file. Thanks to this commendable sharing of solutions from Uffe Vedenbrant and a hack to allow sqlite3 in place of sqlite in the sources I was able to get it cooking. Because it runs as ‘nobody’ via Postfix, a little care was needed to make sure the config file and the sqlite database file were accessible for that user. However it seems the gps maintainer is a little asleep at the wheel, since Uffe’s patch was not taken in since nearly a year to the main project and it means out of the tarball gps doesn’t work at all with sqlite.
Anyway the end result is once again I have some protection from spam just like the old days, except whereas before the box to do this was pulling down ~250W 24/7 the embedded solution only pulls 1.5W, for the same capabilities!

Edit: Ooooh yes, two spams waiting for me thismorning instead of 20, lovely

Perl and Python. Now Perl there is already angst about its demise, I can second it: Perl is dying, Netcraft may later confirm it, but I can do it now.

Python dying is a bit more eyebrow-raising, since bittorrent and yum and other GTK apps are done in it and until today I assumed it had a future. But not with me. It seems to have become impossible to crosscompile the thing. In both cases the problem seems to have been a lust for bootstrapping because it was a “real language”. Because it is a “real language”, when it makes itself it just HAS to do it in its own language. So Perl first makes a miniperl app that is then used to complete the build of the rest of perl. Because perl must be so damn indispensible, you can’t even think of making Perl without doing it in Perl.

Likewise Python insists on building its modules via a Python script, run using the python it just compiled. These conceits of the language devs mean that crosscompiling is impossible, because the python or miniperl that was built WILL NOT RUN on the host that is doing the building since they are compiled for another type of CPU. The bootstrapping concept, so they can hold their head up in academia as a Real Language, destroys the possibility for crosscompile.

Dudes, you killed your languages for crosscompile. There is a lot of crosscompiling going on and it is only going to increase.

Now PHP was easy to crosscompile. Lots of people sniff at it as not being a real language, but look, I can run it down on my ARM box no trouble, and I can’t use Perl or Python. Sell Perl! Sell Python! Buy PHP!!!!

In theory autotools has good support for configuring a project to be crosscompiled, and for some projects it works very well. Often it is simply enough to set the $PATH to your compiler dir, and say ./configure –host=arm-linux. However there are some patholgical projects that blow chunks violently when presented with a crosscompile action that they claim to support.

A good example of such an evil project is perl. This generates an executable called miniperl which is used in the build process to generate the final perl executable. But it does not take care to generate miniperl using a host compiler that is different from the target compiler, so the build process generates miniperl fine, just that it can’t execute on the build host since it is an Arm ELF app. Therefore the build process drops dead.

A common issue is that the configure script wants to compile a short test app with the crosscompiler, which is fine, but it then wants to run it and look for a returncode. I added some infrastructure to the Octotux build tools scripts to enable this by prepending the helper script name to the app to be run remotely via ssh, and to capture the retcode transparently and return it to configure. Since the Octotux system captures any edits into a patchfile anyway, this is a relatively non-fragile solution.

One that caused much pain this last week is sqlite, which suffered from another common problem, the generated configure file wants to determine if include files or libraries exist, but decides that it cannot because you are crosscompiling. In this case it was the configure logic around the readline library that felt unable to survive the crosscompile action with an explicit test and fatal error. Despite hacking that out and overriding various configure defines and makefile definitions in the RPM specfile that tries to build it, there are still problems.

Edit 2006-08-13 sqlite seems to be compiled and packaged okay now, needs readline and ncurses.

Python is another monster like Perl. There are some patches to help here, but they are incomplete

http://mail.python.org/pipermail/python-dev/2003-October/039690.html

The guy who posted them had the whole crosscompiling deal politely rejected in 2003, yet the configure half supports it, even though the build process can’t handle it at all. The patches in the link seem to magically require host-compiled python and Parser/pgen, which are used on the host as part of the build process, but they are not generated anywhere, and although python can come from Fedora python, pgen isn’t available even in the -devel package. So it looks like you must compile the damn thing twice and hack the host packages into the crosscompile build process!

Does your root filesystem df -h sometimes seem a bit more than the sum of it’s du -h / parts? My main embedded filesystem has seemed that way for some months, yet when I run my script that checks that lists all the files in the filesystem that did not come out of a package, nothing really showed up as out of place.
Yesterday my /etc/fstab was partially corrupted so /proc and /tmp (a tmpfs) did not get mounted. To my surprise ll /tmp showed a bunch of stuff in there from Janurary, that had accidentally gotten created at the root filesystem mountpoint dir /tmp, before the tmpfs was placed there. The pile of ‘invisible’ (because you can’t see it after the tmpfs takes up residence there at the real /tmp) junk amount to ~800KBytes uncompressed, in an 8MB root filesystem that is a very welcome new injection of space. Only yesterday I got a board into a bad place by updating the busybox package, but there was not enough space left to create all the symlinks… these symlinks are all of the shell commands like ln, rm, ls etc…. that required a boot into the serially loaded kernel to repair… I guess that won’t be happening so much now I got some space out of my ass.

Just sent this to linux-arm-kernel. I saw the web archive screwed up on my signature, so I include it here to eventually get to Google.

(more…)

Sata-eating Monster

Losing my 24/7 local box to the SATA-eating monster left an immediate problem – nothing was taking my mail. My MX for warmcat.com ends up at my cablemodem, incoming mail was just timing out with nobody to talk to. I thought for a bit about setting up an external postfix and moving the MX, but I didn’t like it as a permanent solution, therefore it was wasted effort to make such a temporary solution. Annoyingly the AMD64 box went down just before I had to take a trip to Spain with Jenny for a few days. I shrugged my shoulders and hoped anyone with important mail would retry, and that the mailing lists I am on would be understanding of the rather unrefined behaviour.
When I returned from the few days of overheated childcare-in-another-land (formally, 16 years ago now, known as “holiday”) I immediately fell ill with some kind of bad cold that laid me up in bed for a couple of days additionally. So after nearly a week the stress was on to get a permanent solution for the missing mailserver question.

(more…)