Sai CI

Git repo: https://warmcat.com/git/sai
Lws Sai: https://libwebsockets.org/sai

Sai is a libwebsockets-based crossplatform CI server and distributed builder aimed at selfhosting your build testing. It integrates with git hooks and gitohashi gitweb: it’s behind the mass CI testing used by libwebsockets.

Sai Screenshot

For lws, in all it currently orchestrates 582 builds per push of libwebsockets, on 30 platforms with a variety of OSes, including big-endian NetBSD. Almost all the builds also run ctest on the native platform to confirm functionality.

This extreme testing lets us develop and ship using -Wall -Wextra -Werror even though we support a huge number of toolchains and platforms.

This article discusses

Sai
How projects use a .sai.json file to describe their tests
The 19-inch rack that contains the lws builders
How Sai is configured on the builders
Delights and limitations of ctest

Sai

Sai Overview

Sai is a CMake / C project dependent on lws, that creates three daemons and some helpers, sai-server + sai-web, which should run somewhere convenient to serve https to interested parties, and sai-builder, which runs inside each environment that wants to offer build service, eg, inside a VM or systemd-nspawn container, and connects to the sai-server over a wss link that speaks JSON.

To deploy it, you set up hooks in your git repo that inform sai-server of a new push along with the .sai.json file from the pushed tree. The .sai.json file lists platforms you want to build on with platform-specific build scripting and a list of builds / tests you want to run on which platforms.

Sai-server will distribute tests to matching platforms and collect the logs that come back, along with any artifacts like RPM packages or zip files, all accessible via sai-web, so, eg you can watch the build and test logs in realtime from your browser.

How projects use a `.sai.json` file to describe their tests

The project contains a “saifile” .sai.json at the top level, which lists the global set of platforms and build scenarios it wants to be tested on.

This is from the saifile for lws main, first you define what platforms your project targets, using structured names in the form OS/arch/toolchain:

{
        "schema": "sai-1",

        # We're doing separate install into destdir so that the test server
        # has somewhere to go to find its /usr/share content like certs

        "platforms": {
                "linux-ubuntu-1804/x86_64-amd/gcc": {
                        "build": "mkdir build destdir;cd build;export CCACHE_DISABLE=1;export SAI_CPACK=\"-G DEB\";cmake .. ${cmake} && make -j && make -j DESTDIR=../destdir install && ctest -j4 --output-on-failure ${cpack}"
                },
                "linux-ubuntu-2004/x86_64-amd/gcc": {
                        "build": "mkdir build destdir;cd build;export CCACHE_DISABLE=1;export SAI_CPACK=\"-G DEB\";cmake .. ${cmake} && make -j && make -j DESTDIR=../destdir install && ctest -j4 --output-on-failure ${cpack}"
                },
                "linux-fedora-32/x86_64-amd/gcc": {
                    ...

… for each of these tuples, there are build machines that connect to sai-server and offer build and test services on that platform.

For each platform listed, platform-specific build instructions are provided, these contain macros like ${cmake} and ${ctest} that are filled in by the “configuration” section next in the Saifile.

The list of configurations or “build scenarios” provides definitions for the macros above, eg, selection of project configuration option sets, and lists out which of the above platforms to build (and test) it on.

Unless platforms were marked with "default": false, any listed platforms are selected by default for building entries in the "configurations" section; the “platforms” member of the configuration lets you strip any defaults (by starting with “none”), or modify the platform list (with “thisplatform” or “not thatplatform”) in a comma-separated list.

    ...
      },

        "configurations": {
                "default": {
                        "cmake":        "",
                        "platforms":    "w10/x86_64-amd/msvc, w10/x86_64-amd/noptmsvc, freertos-linkit/arm32-m4-mt7697-usi/gcc, linux-ubuntu-2004/aarch64-a72-bcm2711-rpi4/gcc, w10/x86_64-amd/mingw32, w10/x86_64-amd/mingw64, netbsd/aarch64BE-bcm2837-a53/gcc, netbsd/x86_64-amd/gcc, w10/x86_64-amd/wmbedtlsmsvc, openbsd/x86_64-amd/llvm, solaris/x86_64-amd/gcc"
                },
                "default-noudp": {
                        "cmake":        "-DLWS_WITH_UDP=0",
                        "platforms":    "w10/x86_64-amd/msvc, w10/x86_64-amd/noptmsvc, freertos-linkit/arm32-m4-mt7697-usi/gcc, linux-ubuntu-2004/aarch64-a72-bcm2711-rpi4/gcc, w10/x86_64-amd/mingw32, w10/x86_64-amd/mingw64, netbsd/aarch64BE-bcm2837-a53/gcc, netbsd/x86_64-amd/gcc, w10/x86_64-amd/wmbedtlsmsvc"
                },
                "fault-injection": {
                        "cmake":        "-DLWS_WITH_SYS_FAULT_INJECTION=1 -DLWS_WITH_MINIMAL_EXAMPLES=1 -DLWS_WITH_CBOR=1",
                        "platforms":    "w10/x86_64-amd/msvc"
                },
                "esp32-heltec": {
                        "cmake":        "-DLWS_IPV6=0",
                        "cpack":        "esp-heltec-wb32",
                        "platforms":    "none, freertos-espidf/xl6-esp32/gcc"
                },
    ...

This ends up defining a sparse matrix or platforms vs configurations that should be built, at the time of writing lws saifile describes 47 distinct configuration build scenarios (lws has a lot of build options) and 30 platforms, in all 581 builds for each push.

It may seem excessive, but historically some of the build scenarios were very prone to silent breakage, for example -DLWS_WITH_NETWORK=0 that builds lws without any networking related code, or -DLWS_WITH_CLIENT=0.

Similarly, since all code ships with -Wall -Wextra -Werror, due to variations in toolchain warnings and, eg, natural types used for virtual ones like size_t, it’s possible to add code that blows a warning promoted to an error just on one specific toolchain, eg, an embedded one that has uint32_t as a long rather than an int.

By making sure most code goes through most platform builds, these kind of things can be solved before they reach users.

Security approach of Sai

The basic approach is that the code being built is somewhat trusted, since we will allow it to execute build actions on our infrastructure. But there are steps taken to limit what it can do.

Security in the build environment and hosts

Inside the build environment, sai builds execute under an unprivileged sai user and do not have sudo or other access to root.

Network access on the builder is needed so sai-builder can connect out to the sai-server, but network access is not required inside the builder, although for lws we use it during ctest.

Many of the build contexts used for lws are implemented as systemd-nspawn, these are not very hardened against attack on the host from inside. So Sai is aimed at testing your own code and code you trust.

Security for build injection

The git build hook that informs sai-server of new jobs sign the JSON they post (which contains the .sai.json extracted from the project) with a secret key that sai-server has the public key for in its configuration JSON, only jobs correctly signed are accepted.

The hardware Sai runs on for lws

Lws Sai Rack

For lws, the builders are living in a 19 inch rack. Most of the builders are implemented as systemd-nspawns or qemu VMs in one beefy PC living underneath the rack, noi. But there are also a bunch of “real” physical builders, eg, for Apple OSX and iOS.

name	type	Physical
linux-ubuntu-xenial/x86_64-amd/gcc/gcc	systemd-nspawn	Noi
linux-ubuntu-2004/x86_64-amd/gcc	systemd-nspawn	Noi
linux-fedora-32/x86_64-amd/gcc	systemd-nspawn	Noi
linux-gentoo/x86_64-amd/gcc	systemd-nspawn	Noi
linux-centos-7/x86_64-amd/gcc	VM	Noi
linux-centos-8/x86_64-amd/gcc	systemd-nspawn	Noi
linux-centos-8/aarch64-a72-bcm2711-rpi4/gcc	Real	RPi4/8GB
linux-ubuntu-2004/aarch64-a72-bcm2711-rpi4/gcc	Real	RPi4/4GB
linux-android/aarch64/llvm	systemd-nspawn / Cross	Noi
netbsd-iOS/aarch64/llvm	Real / Cross	Mac Mini Intel
netbsd-OSX-catalina/x86_64-intel-i3/llvm	Real	Mac Mini Intel
netbsd-OSX-catalina/x86_64-intel-i3/llvm	Real	Mac Mini M1
freertos-linkit/arm32-m4-mt7697-usi/gcc	systemd-nspawn / Cross	Noi
windows-10/x86_64-amd/msvc	VM	Noi
windows-10/x86_64-amd/mingw32	systemd-nspawn / Cross	Noi
windows-10/x86_64-amd/mingw64	systemd-nspawn / Cross	Noi
freertos-espidf/xl6-esp32/gcc	systemd-nspawn / Cross / Real	Noi + Heltec ESP32
freertos-espidf/xl6-esp32/gcc	systemd-nspawn / Cross / Real	Noi + WROVER KIT ESP32
linux-fedora-32/riscv64-virt/gcc	VM	Noi
linux-debian-11/x86_64/gcc	systemd-nspawn	Noi
linux-debian-buster/x86_64-amd/gcc	systemd-nspawn	Noi
linux-debian-buster/x86_64-amd32/gcc	systemd-nspawn	Noi
linux-debian-sid/x86_64-amd/gcc	systemd-nspawn	Noi
linux-debian-sid/x86_64-amd32/gcc	systemd-nspawn	Noi
netbsd/x86_64-amd/llvm	VM	Noi
openbsd/x86_64-amd/llvm	VM	Noi
freebsd/x86_64-amd/llvm	VM	Noi
solaris/x86_64-amd/gcc	VM	Noi
netbsd-BigEndian/x86_64/llvm	Real	RPi3

Each of these OSes are running sai-builder, which makes an outgoing client wss connection to the remote sai-server using wss. When sai-server sees there are jobs needing doing on a particular platform, it farms out the build tasks to connected builders that offer build services on that platform.

The mighty Noi

Noi is a 32 core, 64-thread AMD 3970X box with 64GB RAM, all of the Linux variations run on it as systemd-nspawn containers, the cross-build platforms run on those too; it has several QEMU instances, eg, running Windows 10 and Fedora on RISC-V64.

It also provides a subnet for devices under test, both on Ethernet and presents its wireless as a local wlan AP for the devices connect to.

Apple

OSX and iOS cross builds take place on two physical mac minis, one intel i3 and one newer M1-based one. These are mounted inside a 1U rack mount case.

aarch64 RPi3/4

The rack includes several RPI3/4 in a rack mount, these are modern “jellybean” boards you can casually get for a reasonable price with many available ISOs.

We run Ubuntu 20.10 and CentOS 8 on RPi4s, and NetBSD Big-Endian on an RPi3, that means we are building and testing on a true physical BE machine, so we can know we are endian-clean.

ESP32 / Freertos

ESP32 is cross-built in a systemd-nspawn on Noi, but there are two physical ESP32 device plugged in the USB, one heltec and another WROVER-KIT. Lws includes minimal examples specifically for these

https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/embedded/esp32

These are flashed with the results of the CI build, and it is run on the physical device via ctest each time.

Sai provides a device access mediator sai-device, which has a config file decribing device resource connections and types, the ctest script queues for access to a physical device of a specific type on the builder using that.

Another helper sai-expect handles monitoring the console UART and determining pass or fail (or timeout) result.

Windows

There’s a windows 10 VM running in noi that builds and tests using msvc, there are also 32- and 64-bit mingw builds, but these don’t run ctest.

Building Sai on 30 platforms

Sai relies on lws services

Sai-builder than runs on the build platforms relies on lws and Secure Streams to communicate to sai-server on the Internet, so the first step is to configure and build lws for the platform.

Lws provides a lot of services to sai such as lws_spawn, that manages spawning sub- processes in a crossplatform way, as well as preparing child wsi to handle stdin, stdout and stderr on the spawn. Lws Threadpool is used to encapsulate the spawned process in a way that allows communication to be synchronized with the lws event loop cleanly, so logs are uploaded in realtime.

When dealing with so many platforms, there are many quirks that are handled by these lws apis, for example spawning and handling stdin/out/err on windows and bsd or OSX are different from Linux, as are handling process killing.

For embedded physical platforms, Sai uses lws to convert UART / USB UART traffic into wsi, and bring those into a realtime log channel feeding sai-server.

Sai-server is also built around lws and Secure Streams, it handles serving to the builders using JSON and Secure Streams Serving over wss.

Sai Configuration

All of the sai daemons and tools take their configuration from /etc/sai/, these are further separated by daemon, so /etc/sai/builder/ etc.

JSON is used to define which logical platforms are offered to sai-server, and how to reach out from the platform to the sai-servers that the platform wants to provide services for.

For sai-builder, it lists platform objects in the JSON, each with its own tuple name, and these are instantiated at sai-server while the connection to the builder remains up.