{"schema":"libjg2-1", "vpath":"/git/", "avatar":"/git/avatar/", "alang":"", "gen_ut":1714063242, "reponame":"sai", "desc":"Sai lightweight distributed CI", "owner": { "name": "Andy Green", "email": "andy@warmcat.com", "md5": "c50933ca2aa61e0fe2c43d46bb6b59cb" },"url":"https://warmcat.com/repo/sai", "f":3, "items": [ { "schema":"libjg2-1", "oid":{ "oid": "9cffd43ea092a347eea88c76694a153f9827ecf4", "alias": [ "refs/heads/main"]},"tree": [ { "name": "READMEs","mode": "16384", "size":0}, { "name": "assets","mode": "16384", "size":0}, { "name": "cmake","mode": "16384", "size":0}, { "name": "etc-sai-EXAMPLE","mode": "16384", "size":0}, { "name": "hooks","mode": "16384", "size":0}, { "name": "scripts","mode": "16384", "size":0}, { "name": "src","mode": "16384", "size":0}, { "name": ".gitignore","mode": "33188", "size":25}, { "name": ".sai.json","mode": "33188", "size":2904}, { "name": "CMakeLists.txt","mode": "33188", "size":5267}, { "name": "README.md","mode": "33188", "size":15808}],"s":{"c":1713640546,"u": 15979}} ,{"schema":"libjg2-1", "cid":"ef919e0417b22f3eb63b0d214f682969", "oid":{ "oid": "9cffd43ea092a347eea88c76694a153f9827ecf4", "alias": [ "refs/heads/main"]},"blobname": "README.md", "blob": "\n\n[](https://warmcat.com/git/sai)\n\n`Sai` (pronouced like 'sigh', \u0022Trial\u0022 in Japanese) is a very lightweight\nlws-based network-aware distributed CI builder and coordinating server.\nYou can run the sai-builder daemon on any number of devices ad-hoc without\ncentral registration or inbound internet access, to offer builds for those\nplatforms... builders can run:\n\n - on native boxes,\n - inside systemd-nspawn contexts,\n - inside VMs (eg, via qemu) for native and non-native arches, or\n - cross-build against connected embedded devices which can be flashed and run the build\n results, controlled by gpio and serial.\n\nA sai-server daemon runs on a server to receive wss connections from the builders,\ngit update hooks POST signed JSON job matrices from configured git servers, and\nsai-server coordinates dispatching concurrent jobs to dynamically availabe remote\nbuilders of the correct platforms, collecting logs and results.\n\nA sai-web server daemon is also available usually on :443 or via a proxy to provide\na live web / websockets interface with synamic updates and realtime build logs in\nthe browser, with JWT-authentication for manual job control.\n\n\n\n## General approach\n\n - Distributed \u0022builders\u0022 run the `sai-builder` daemon from inside the build\n platform they want to provide builds for (ie, inside a systemd-nspawn or VM).\n\n - The `sai-builder` daemons maintain nailed-up outgoing client wss connections\n to a central `sai-server`, which manages the ad-hoc collection of builders\n it finds, and provides a web UI (http/2 and wss2).\n\n\n\n - When a git repo that wants sai tests is updated, a push hook performs a POST\n notiftying the Sai server of a new push event which creates an entry in an\n event sqlite3 database. (The server does not need to access the repo that\n sends the git notification). The hook sends the server information about the\n push and the revision of `.sai.json` from the new commit in the POST body...\n the server parses that JSON to fill an sqlite3 database with tasks and\n commandline options for the build on platforms mentioned in `.sai.json`.\n Pushes to branches beginning with `_` are ignored by Sai, even if they have\n a valid `.sai.json`; this allows casual sharing of trees via the same git\n repo during intense development without continually triggering CI builds.\n\n - Builders typically build many variations of the same push, so they use a\n local git mirror only on the builder to reduce the load on the repo that\n was updated. For large trees, being able to already have previous trees\n as a starting point for git dramatically reduces the time compared to a full\n git fetch.\n\n - The server hands out waiting tasks on connected idle builders that offer the\n requested platforms, which build them concurrently. The builders may be\n inside a protected network along with the repos they connect to; both the\n builders and the repo only make outgoing https or wss connections to the\n server.\n\n - On the builders, result channels (like stdout, stderr, eventually others like\n `/dev/acm0`) with logs and results are streamed back to the server over a wss\n link as the build proceeds, and are stored in an event-specific sqlite3\n database for scalability.\n\n - Builders can run CTest or other tests after the build and collect the\n results (CTest has the advantages it's lightweight and crossplatform).\n\n - The server makes human readable current and historical results available\n in realtime over https web interface\n\n - One `sai-builder` daemon can be configured to offer multiple instances of\n independent platform build, and multiple platform builds (eg, cross\n toolchains).\n\n - Embedded test devices that need management by external gpios to select\n flash or test modes can be wired to an RPi or similar running sai-jig.\n This listens on a configurable port for requests to perform gpio sequencing\n specified in a configuration file. One sai-jig instance can separately\n manage external gpio sequencing for multiple test targets.\n\n - Largely the server is automatic, driven by git hook notifications over\n HTTP and the UI is read-only. However there are some privileged UI operations\n like deleting a whole event, or redoing whole events or individual tasks.\n For these, if the browser has an authentic JWT signed by the server, it can\n see and operate these privileged controls.\n\n## Build flow and support for embedded\n\n\n \nTesting is based around CTest, it can either run on the build host inside the\ncontainer, or run on a separate embedded device. In the separate case, the flow\ncan include steps to flash the image that was built and to observe and drive\ntesting via usually USB tty devices. IO on these additional ttys is logged\nseparately than IO from build host subprocess stdout and stderr.\n\n### Sharing embedded devices on the test host\n\nEmbedded devices are actually build host-wide assets that may be called upon\nand shared by different containers and different build platforms. For\nexample, a cross-built flash image on Centos8 and another cross-built on\nUbuntu Bionic for the same platform may want to flash and test on the same\npool of embedded devices. Even images from different build platforms for the\nsame kind of device may wish to flash the same embedded device, where the\ndevice can be flashed to completely different OSes.\n\nDevices may be needed by post-build actions, but they are not something\na sai-builder for a platform can \u0022own\u0022 or manage by itself. Instead they are\nrequested from inside the build action by another tool built with `sai-builder`,\n`sai-device`, which reads shared JSON config describing the available devices\nand platforms they are appropriate for.\n\n\n\nRather than reserve the device when the build is spawned, the reservation\nneeds to happen only when the build inside the build context has completed.\nThat in turn means that a different sai utility has to run at that time from\ninside the build process, in order that it can set things in the already-\nexisting subprocess environment.\n\n```\n\u0022devices\u0022: [\n {\n \u0022name\u0022: \u0022esp32-heltec1\u0022,\n \u0022type\u0022: \u0022esp32\u0022,\n \u0022compatible\u0022: \u0022freertos-esp32\u0022,\n \u0022description\u0022: \u0022ESP32 8MByte SPI flash plus display\u0022,\n \u0022ttys\u0022: [\n \u0022/dev/serial/by-path/pci-0000:03:00.3-usb-0:2:1.0-port0\u0022\n ]\n }\n]\n```\n\nDevices are logically defined inside a separate conf file\n`/etc/sai/devices/conf` on the host or vm, and containers should bind a ro\nmount of the file at the same place in their /. This avoids having to maintain\na bunch of different files every time a new device is added. For platform\nbuilders based in a VM, these have a boolean relationship with IO ports, they\neither must wholly own them or are unaware of them: this means they can't\nparticipate in sharing device pools but must be allocated their own with its\nown config file inside the VM listing those.\n\nWhen the build process wants to acquire an embedded device of a particular type\nfor testing, it runs in the building context, eg, `sai-device esp32 ctest`.\n\nThis waits until it can flock() all the ttys of one of the given type of\nconfigured devices (\u0022esp32\u0022 in the example), sets up environment vars for each\n`SAI_DEVICE_\u003cttyname\u003e`, eg,\n`SAI_DEVICE_TTY0\u003d/dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0`,\nthen executes the given program (`ctest` in the example) as a child process\ninside the build context. Although it has an fd on the tty so it can flock()\nit, `sai-device` does not read or write on the fd itself.\n\nBaud rate is not considered an attribute of the tty definition but something set\nfor each sai-expect.\n\nWhen the child process or build process ends, the locking is undone and the\ndevice may be acquired by another waiting `sai-device` instance in the same or\ndifferent platform build context.\n\nThe underlying locking is done hostwide using flock() on bind mounts of\nthe tty devices, the other containers will observe the locking no matter who\ndid it.\n\nAvailability of the device node via an environment variable means that CTest\nor other scripts are able to directly write to the device.\n\n### Logging of device tty activity\n\nThe `sai-builder` instance opens three local listening Unix Domain Sockets for\nevery platform, these accept raw data which is turned into logs on the\nrespective logging channel and passed up to the `sai-server` for storage and\ndisplay like the other logs.\n\nThe paths of these \u0022log proxy\u0022 Unix Domain Sockets are exported as environment\nvariables to the child build and test process as follows\n\nEnvironment Var|Ch#|Meaning|Example\n---|---|---|---\nSAI_LOGPROXY|3|Build progress logging|`@com.warmcat.com.saib.logproxy.warmcat_com-freertos-esp32.0`\nSAI_LOGPROXY_TTY0|4|Device tty0|`@com.warmcat.com.saib.logproxy.warmcat_com-freertos-esp32.0.tty0`\nSAI_LOGPROXY_TTY1|5|Optional Device tty1|`@com.warmcat.com.saib.logproxy.warmcat_com-freertos-esp32.0.tty1`\n\nBecause some kinds of device share the same tty for flashing the device, at\nwhich time nothing else must be reading from the tty, tty activity is only\ncaptured and proxied during actual user testing by `sai-expect`, which is run at\nwill be the CTest script. In this way, device ttys are only monitored while\ntest are ongoing; however the kernel will buffer traffic that nobody has read\nuntil the next reading consumes it.\n\n### Sai device tty logging\n\nDevices may have multiple ttys defined, for example a device with separate ttys\nand log channels for a main cpu and a coprocessor is supported.\n\nThe ttys listed on devices have their own log channel index and are timestamped\naccording to when they were read from the tty. In the event many channels are\n\u0022talking at once\u0022, in the web UI the different log channel content appears in\ndifferent css colours and in chunks of 100 bytes or so, which tends to keep\nisolated lines of logging intact.\n\n## Non-Linux: use /home/sai in the main rootfs\n\nFor OSX and other cases that doesn't support overlayfs, the same flow occurs\njust in the main rootfs /home/sai instead of the overlayfs /home/sai.\n\nIt means things can only be built in the context of the main OS, but since OSX\ndoesn't have different distros, which is the main use of the Linux overlayfs\nfeature, it's still okay.\n\n## Builder instances\n\nThe config JSON for sai-builder can specify the number of build instances for\neach platform. These instances do not have any relationship about what they\nare building, just they run in the same platform context (and are managed by\nthe one `sai-builder` process). They each check out their own build tree\nindependently, so they can be engaged building different versions or different\ntrees concurrently inside the platform.\n\nTests have to take care to disambiguate which instance they are running on,\nsince the network namespace is shared between instances that are running in the\nsame sai-builder process on the same platform. An environment var\n`SAI_INSTANCE_IDX` is available inside the each build context set to 0, 1, etc\naccording to the builder instance.\n\nFor network related tests, `SAI_INSTANCE_IDX` should be referred to when\nchoosing, eg, a test server port so it will not conflict with what other\nbuilders may be doing in parallel.\n\n## Builder git caching\n\nFor each `\u003csaiserver-project\u003e`, the builder maintains a local git cache. This is\nupdated once when the new ref appears and then the related tests check out a\nfresh image of their ref from that each time. This is very fast after the first\nupdate, because it doesn't even involve the network but fetching from the local\nfilesystem. \n\n## systemd-nspawn support\n\nOn Linux, it's recommended to use systemd-nspawn to provide multiple distro\nenvironments conveniently on one machine. There are instructions for setting\nup individual virtual ethernet devices managed by nmcli on the host.\n\n`sai-builder` also supports running inside a KVM / QEMU VM transparently as well,\neg for windows or emulated architecture VMs.\n\n## `sai` builder user\n\nBuilds happen using a user `sai` and on the builder, files are only created\ndown `/home/sai` or `\u005cUsers\u005csai`.\n\n```\n# useradd -u883 -gnogroup sai -d/home/sai -m -r\n```\n\n## build filesystem layout\n\n - /home/sai/\n - git-mirror/\n - `remote git url`_`project name` -- individual git mirrors\n - jobs/\n - `server hostname`-`platform name`-`instance index`/\n - `project_name`/ - checkouts and builds occur in here\n\n## Build steps\n\nBuilding sai produces two different sets of apps and daemons by default, for\nrunning on a the server that coordinates the builds and for running on machines\nthat offer the actual builds for particular platforms to one or more servers.\n\nYou can use cmake options `-DSAI_MASTER\u003d0` and `-DSAI_BUILDER\u003d0` to disable one\nor the other.\n\nServer executables|Function\n---|---\nsai-server|The server that builders connect to\nsai-web|The server that browsers connect to\n\nBuilder executables|Function\n---|---\nsai-builder|The daemon that connects to sai-server and runs builds\nsai-device|Helper that coordinates which builds wants and can use specific embedded hardware\nsai-expect|Helper run by embedded build flow to capture serial traffic and react to keywords\nsai-jig|Helper for embedded devices that lets another device control its buttons, reset etc as part of the embedded build flow\n\nFirst you must build lws with appropriate options.\n\nFor redhat type distros, you probably need to add /usr/local/lib to the\n/etc/ld.so.conf before ldconfig can rgister the new libwebsockets.so\n\n```\n$ git clone https://libwebsockets.org/repo/libwebsockets\n$ cd libwebsockets \u0026\u0026 mkdir build \u0026\u0026 cd build \u0026\u0026 \u005c\n cmake .. -DLWS_UNIX_SOCK\u003d1 -DLWS_WITH_STRUCT_JSON\u003d1 -DLWS_WITH_JOSE\u003d1 \u005c\n -DLWS_WITH_STRUCT_SQLITE3\u003d1 -DLWS_WITH_GENCRYPTO\u003d1 -DLWS_WITH_SPAWN\u003d1 \u005c\n -DLWS_WITH_SECURE_STREAMS\u003d1 -DLWS_WITH_THREADPOOL\u003d1\n$ make -j \u0026\u0026 sudo make -j install \u0026\u0026 sudo ldconfig\n```\n\nThe actual cmake options needed depends on if you are building sai-server and / or\nsai-builder.\n\nFeature|lws options\n---|---\neither|`-DLWS_WITH_STRUCT_JSON\u003d1` `-DLWS_WITH_SECURE_STREAMS\u003d1`\nserver|`-DLWS_UNIX_SOCK\u003d1` `-DLWS_WITH_GENCRYPTO\u003d1` `-DLWS_WITH_STRUCT_SQLITE3\u003d1` `-DLWS_WITH_JOSE\u003d1`\nbuilder + related|`-DLWS_WITH_SPAWN\u003d1` `-DLWS_WITH_THREADPOOL\u003d1`\n\nSimilarly the two daemons bring in different dependencies\n\nFeature|dependency\n---|---\neither|libwebsockets\nserver|libsqlite3\nbuilder|libgit2 pthreads\njig (linux only)|libgpiod\n\n#### Unix / Linux\n\n```\n$ git clone https://warmcat.com/repo/sai\n$ cd sai \u0026\u0026 mkdir build \u0026\u0026 cd build \u0026\u0026 cmake .. \u0026\u0026 make \u0026\u0026 make install\n$ sudo cp ../scripts/sai-builder.service /etc/systemd/system\n$ sudo mkdir -p /etc/sai/builder\n$ sudo cp ../scripts/builder-conf /etc/sai/builder/conf\n$ sudo vim /etc/sai/builder/conf\n$ sudo systemctl enable sai-builder\n```\n\n#### Additional steps for freebsd\n\nFreebsd presents a few differences from Linux.\n\n1) `pkg install bash` and other prerequisites like git, cmake etc\n\n2) Create the sai user via `adduser` and set the uid to 883.\n\n3) Create the builder logproxy socket dir one time as root\n\n```\n# mkdir -p /var/run/com.warmcat.com.saib.logproxy\n# chown sai /var/run/com.warmcat.com.saib.logproxy\n```\n\n4) For script portability, `ln -sf /usr/local/bin/bash /bin/bash`\n\n5) As root copy `scripts/etc-rc.d-sai_builder-FreeBSD` to `/etc/rc.d`.\n\n6) As root, edit `/etc/rc.conf` and add a line `sai_builder_enable\u003d\u0022YES\u0022`, then,\n`sudo /etc/rc.d/sai_builder start`\n\n7) Create and prepare `/etc/sai/builder/conf` as for Linux\n\n","s":{"c":1713640546,"u": 726}} ],"g": 838,"chitpc": 0,"ehitpc": 0,"indexed":0 , "ab": 0, "si": 0, "db":0, "di":0, "sat":0, "lfc": "7d0a"}