Overview

LWS Secure Streams is a method for abstracting out client connection policy information from your code and allowing it to be set in (typically devicewide) JSON. One way to think of it is the “client version” of lwsws, where a generic server is configured by JSON to be whatever mix of vhosts, proxies, cgi, etc you want, placed where you like in the server url space without having to write code to do it. (Everything on libwebsockets.org and warmcat.com is served by lwsws with a JSON specification down /etc/lwsws/conf.d/).

Secure Streams goes further in that it even abstracts away the protocol choice as well as the endpoint, tls validation stack, backoff / retry tables etc; protocol-specific details are set in the JSON policy, not in the code and that includes whether it happens to be transferring payloads in MQTT, or h1, or h2, or ws.

Most of the boilerplate code needed for lws level code goes away, like the protocol and protocol callbacks. And since the code doesn’t have to care what wire protocol it’s using any more, the policies may be remotely loaded at startup and what and how your device connects to cloud or other peers becomes something that can be changed after the fact without changing any code.

One important change is the logical Secure Stream represents the desire for a connection of a particular type, its lifecycle is not affected by actual tcp connections coming up or going down and being reconnected, it exists for as long as there’s a desire for the type of connection. This is in contrast to working directly at lws layer, where wsi have a lifecycle that is completely bound up with the socket connection they represent.

Example policy

You can find an example of a JSON policy here

https://warmcat.com/policy/minimal-proxy.json

The basic idea is that the system can use various named “stream types” from the JSON policy; all the details about how that stream type connects and acts are in the policy for it. Yes, including the protocol choice. Eg,

                "avs_metadata": {
                        "endpoint": "alexa.na.gateway.devices.a2z.com",
                        "port": 443,
                        "protocol": "h2",
                        "http_method": "POST",
                        "http_url": "v20160207/events",
                        "opportunistic": true,
                        "h2q_oflow_txcr": true,
                        "http_auth_header": "authorization:",
                        "http_auth_preamble": "Bearer ",
                        "http_multipart_name": "metadata",
                        "http_mime_content_type": "application/json; charset=UTF-8",
                        "http_no_content_length": true,
                        "rideshare": "avs_audio",
                        "retry": "default",
                        "tls": true,
                        "tls_trust_store": "avs_via_starfield"
                }

Secure streams user apis

If there’s no lws protocol stuff, and it’s supposed to be independent of the wire protocol… what does the remaining code look like then? When you create a stream, you provide three callbacks in a const struct describing the streamtype name in the policy and how your code interfaces to it:

  • rx() receives whatever deframed payload has appeared on the connection

  • tx() provides a buffer, of a size chosen by Secure Streams but typically around 1400 bytes, which you may write payload into for immediate sending

  • state() is updated with the connection situation and remote acks

There’s an api lws_ss_request_tx() which will be a familiar idea to lws users, it schedules a tx() callback for the stream when possible. The are some other misc helpers, but that is basically it. You don’t tell it where to connect in the code or if it’s MQTT or h1 or h2 or ws, you tell it you want a stream of a particular name, it looks it up in the JSON policy to find out how and where to connect it.

You send and receive payloads, how they get framed and deframed isn’t really relevant. Protocol-specific details go in the JSON policy, like special http headers.

Dealing with dynamic metadata

This is enough to cover basic situations well… fundamentally if you are sending JSON back and forth, that JSON payload is the same deal whether it went by h1 or h2 or whatever. Anything other than the JSON is just a distraction.

However although you might not choose to architect things that break this model if you are interested in it, sometimes you don’t control the remote endpoint, and its apis may insist to make more than full use of whatever horrible quirks your protocol offers, like nonstandard http headers containing dynamic info or multipart mime.

Secure Streams allows the JSON stream type definitions to declare metadata names, which may be set dynamically on the stream using lws_ss_set_metadata() and then used in headers or other protocol-specific policy information like the endpoint name itself using ${metadata} type connection- and transfer- time string substitutions.

Updating the JSON policy

Secure Streams defines a special streamtype “fetch_policy”… if this is defined in the hardcoded policy at context creation time, then when the network is up and the ntp time acquired so tls can work, lws will follow the policy in that stream type to fetch updated policy JSON and switch to that. This allows modification of the devicewide communications policy after devices are shipped (eg, switch cloud provider) without needing an OTA / reflash.

Hardcoded policy

In lws v4.1+, Secure Streams also supports a policy converted from JSON to explicit structs to be built into code directly, if in some cases the platform is too restricted to handle JSON. That way you can still maintain the policy in JSON but autoconvert it to structs in a header file at build time, like this.

Endpoint tls validation

The JSON policy allows you to define x.509 certs (in Base64 DER strings) and create named stacks of certificates in a “trust store”. These trust store names can be associated with individual streamtypes to control the tls validation process entirely from the JSON policy. It’s also possible to skip specifying explicit trust stores if there’s a system trust store available as there typically is with OpenSSL, but it may be preferable to directly control which one or two issuer root certs can be accepted.

Because this is handled in the policy which may be updated remotely, managing root cert updates is made simple and doesn’t need an OTA or reflash.

Using Secure Streams with multiple processes

In the case there’s a single process doing the communication, because there’s a “communication daemon” kind of architecture or it’s a single statically-linked RTOS image, then Secure Streams connections can be fulfilled directly.

However if it’s a Linux-class device with multiple separate processes that want to send stuff, there are important pressures to make the best of muxed protocols so different processes can potentially share a single tcp connection and tls tunnel for, eg, h2 or even MQTT. One Linux-class client device might not feel the pressure but at the server side, it doesn’t scale the same if each device is making two or four connections back each with its own tls tunnel, and you pay a bill for the additional resources.

For that reason, lws supports a Secure Stream Proxy mode, where the proxy that has the policy and actually fulfils the connections runs as its own process, and applications forward their serialized payloads to it over Unix Domain Sockets. This can be selected at cmake with -DLWS_WITH_SECURE_STREAMS_PROXY_API=1 and additionally LWS_SS_USE_SSPC must be defined when the client applications are built, so they transparently use a second implementation of the same apis that uses the Secure Streams Proxy to get things done.

The proxy itself is provided in the minimal examples.

What did we learn this time?

  • For devices that are predicated on client connections, Secure Streams is a layer on top of lws that separates out all connection policy into JSON… that includes enpoint selection and tls validation certs etc but even the transport protocol is decided by the JSON policy.

  • The stuff that’s left related to the connection is radically simplified and just deals with payloads

  • Logical Secure Streams outlast any specific connection underneath, and can reacquire underlying connections when needed