We organized the work into three phases. First came hardware validation – proving the platform could meet the requirements. Then the main development phase, where we built out the real microservices architecture, OneWeb integration, OTA, and secure boot. Production hardening with full CI/CD and automated QA ran in parallel. A team of six engineers worked across all phases.

Hardware Validation

When we received the Gateworks GW11050‑2 boards, the first job was to verify that every hardware interface actually worked under OpenWrt. Ethernet (all five Gigabit ports through the onboard switch), WiFi 6, MoCA 2.0, TPM 2.0 over SPI, GPIO pins, LEDs, the physical reset button, and USB. We ran 24‑hour continuous stability tests under thermal stress to make sure nothing degraded over time.

During this phase, we built a simple three-service architecture as a proof of concept. An API Gateway handled HTTP requests from the SSM (the satellite modem sitting at 192.168.100.1). A Network Service managed all network interfaces. And a System Service controlled hardware – GPIO, LEDs, TPM operations, thermal monitoring, and UCI configuration.

One decision we made early was to give the System Service exclusive access to OpenWrt’s UCI config system. In the first weeks of testing, we saw that letting multiple processes write to UCI files at the same time could corrupt configs. A firewall rule would half-apply, or a VLAN change would overwrite a DHCP setting made a millisecond earlier. So every configuration change – whether it came from the API Gateway or the Network Service – had to go through a single point. That solved the race conditions and also gave us atomic transactions for free.

We implemented a few sample OneWeb API endpoints to show the router could respond within specification. By the end of this phase, the client had enough evidence: SSM-to-CNX communication worked, VLAN isolation between admin and user networks held up, firewall rules applied correctly, and the board ran stable for 24 hours straight under thermal stress.

Moving to Microservices

For the main development phase, our team redesigned the architecture from scratch. The validation-phase approach was fine for proving the concept, but a monolithic design would not scale to the full set of OneWeb requirements and the client’s own API needs.

Three options for inter-service communication were on the table: Oatpp REST, OpenWrt’s native ubus, and D-Bus. We picked Oatpp. It has a moderate footprint – a few hundred kilobytes to maybe 10 MB depending on route count – and starts fast. More important: REST is stateless, so if a service crashes and restarts, clients don’t lose their context. D-Bus is fragile to daemon crashes. And debugging REST is something any developer can do with curl or Postman, while D-Bus requires specialized tools like dbus-monitor.

The production architecture split into three front-facing services, mapped to one external system. One service took requests from the Access App (on localhost:80). Another handled two-way communication with the SSM (on 192.168.100.3:80). The third managed firmware updates from the SSM on a separate port (192.168.100.3:8088). Each service had a 1:1 relationship with its external consumer. This simplified authentication (no shared auth layer needed) meant a denial-of-service on one port would only bring down that one service, while the others kept running.

Behind these three sat the Main Application – the core of the system. It contained a Network Configuration Manager and a System Configuration Manager. When any front-facing service received a request that required changing system state (say, updating a VLAN config or triggering a reboot), it forwarded that request to the Main Application via REST. Only the Main Application had permission to actually modify system behavior. Even if the OneWeb-facing service went down completely, the rest of the system would continue to operate.

OneWeb Integration

Integrating with OneWeb was one of the biggest parts of the project. We implemented over 20 API endpoints defined in the OneWeb ICD (Interface Control Document). These covered system info, device status, network statistics, configuration updates, port and interface control, WiFi management, diagnostics, log retrieval, event monitoring, and factory reset.

Some endpoints were straightforward – a GET request for system_info just returns hardware and firmware version data. Others required careful coordination across the whole system. The configuration update endpoint can change operating mode, port assignments, interface states, and WiFi settings in a single request. All those changes had to apply atomically through UCI. A half-configured state – where, say, the WiFi SSID was updated, but the firewall rule didn’t – was not acceptable.

Event monitoring deserved special attention. The CNX sends events to the SSM when something happens, ranging from minor status changes to critical system failures. Each event has a lifecycle: the CNX raises it, the SSM acknowledges it, and later the CNX clears it. Missing an event or sending a duplicate clearance would break the SSM’s state. Early in testing, we ran into exactly this problem – under heavy load, events occasionally sent twice because the acknowledgment from the SSM arrived after the retry timer had already fired. We fixed it by adding idempotency keys to each event and tracking acknowledgment state in memory.

OTA-related endpoints followed a standard handshake defined in the ICD. The SSM notifies the router about a new software version. The router downloads it, verifies the SHA256 checksum, stages it to the inactive partition, and reports status back. Same pattern for configuration updates and package updates. Every step is observable through status endpoints – the SSM can query progress at any time.

For authentication, we used HTTP Basic Auth over HTTPS (TLS 1.2 and above). On first boot, the device generates a self-signed certificate. Default credentials must be changed on first login.

OTA Updates and Partition Management

The OTA system was designed around A/B partitions with a factory fallback. At any moment, the router holds three images: the factory image (a permanent baseline that ships with the device), the active image (currently running), and the standby image in the inactive partition.

Here’s how an update works. The SSM pushes a notification. Our OTA Service downloads the new image, verifies its SHA256 hash and usign (Ed25519) signature, and writes it to whichever root partition is currently inactive – slot A or slot B. Then it updates the U-Boot environment variable to mark the new partition as the boot target. On reboot, U-Boot checks a counter. If the new image boots successfully and the system reaches a healthy state, it resets the counter and marks the partition as good. Three consecutive boot failures? U-Boot automatically swaps back to the previous working partition. Six failures trigger a fallback all the way to the factory image.

One problem we had to solve was partition layout constraints. The board uses eMMC storage with MBR partitioning that allows only four primary partitions. All four were needed: two root slots (A and B), the factory recovery slot, and an overlay partition for persistent configuration. That left nothing for logs, vendor data, or device identity storage. We worked around this by embedding those into the overlay filesystem structure and by writing device identity data to a reserved EEPROM area on the board.

Each software version carries metadata in a structured YAML file describing upgrade constraints. Before applying any update, the system checks: does it require a specific bootloader version? Is rollback to the previous version allowed? Are there blocking upgrades that cannot be skipped? This prevents situations where someone tries to jump across versions without applying a mandatory intermediate migration.

We also built component version tracking across the entire terminal. The router keeps a JSON record of software versions for the ACU, SSM, modem, and CNX itself. It validates these against known-safe combinations. If someone swaps a board module with a different firmware version, the system flags the mismatch automatically.

Physical reset comes in two forms. Through the API, the SSM can trigger either a firmware-only reset or a full firmware-plus-configuration reset. Using the physical button on the board, a 4-second hold repairs the network configuration (useful when a bad config makes the device unreachable over the network), and a 10-second hold performs a complete factory reset.

Secure Boot

Without verified boot, anyone with physical access could flash modified firmware that bypasses all authentication, firewall rules, and network isolation. For a satellite terminal used in defense and government deployments, that is not acceptable.

We built a four-stage boot chain. It starts at the SoC ROM – factory-programmed, immutable code burned into the NXP i.MX8M Plus chip. This ROM validates an AHAB-signed boot container using the Super Root Key (SRK) hash stored in the chip’s OTP fuses. Those fuses get burned during manufacturing. Nobody can change them afterward – they are truly one-time programmable. If the signature doesn’t match, the device refuses to start.

Next comes SPL and U-Boot. SPL loads U-Boot from the eMMC hardware boot partition (boot0 or boot1 – these are physically isolated from the OS partition). U-Boot then loads a FIT image, which is a signed bundle containing the Linux kernel and device tree. We sign FIT images with RSA-2048 (sha256,rsa2048), and the public keys sit embedded in the U-Boot device tree blob. A failed signature check stops boot completely.

After U-Boot hands off to the kernel, dm-verity takes over. It validates every block of the root filesystem by computing SHA-256 hashes and comparing them against a Merkle hash tree appended to the squashfs image. If any block has been tampered with, the kernel doesn’t log a warning and continue – it triggers an I/O error, and the configured policy is a kernel panic. The device simply does not boot with corrupted files.

In user space, the root filesystem is immutable. An overlayfs layer on top handles persistent changes (things like network settings in /etc/config/), but every service binary and system file lives on the read-only partition. Malware can’t persist through a reboot – there is nothing writable for it to attach to.

Key management follows two tracks. During factory provisioning, manufacturing keys allow test firmware for calibration and hardware validation. Once the device gets production keys – SRK hash burned into fuses, JTAG disabled – it’s locked. Manufacturing-signed firmware is rejected from that point on. For key rotation, we planned ahead: the U-Boot device tree holds multiple public keys, and the usign keyring supports adding new keys before removing old ones. The rotation policy is to sign with the new key, wait for 95% fleet adoption (tracked through telemetry), then remove the old key.

CI/CD Pipeline

We organized the codebase into three BitBucket repositories. The first – for the OpenWrt build – is a fork of the official OpenWrt repository, extended with our kernel patches, files overlay, and build configuration for the Gateworks Venice target. The idea behind forking rather than maintaining a separate config was upstream compatibility: when the base OpenWrt version gets security patches, we can merge them without rewriting our build.

The second repository holds all the CNX application code and tests. The third contains additional OpenWrt packages that our image needs – Oatpp libraries (base, libressl, mbedtls, swagger, websocket), TPM2 tools and libraries (tpm2-tss, tpm2-tools, tpm2-abrmd, tpm2-tss-engine), and a luksMount package.

Jenkins runs two multibranch pipelines. The OpenWrt pipeline builds the complete image inside Docker – same Dockerfile, same environment, every time – and pushes the artifact to Artifactory. Build time from scratch sits around 65 minutes. That’s slow for rapid iteration, and we explored several ways to bring it down: ccache for C++ builds, shared Docker layer caching, and mirroring external dependencies to Artifactory so the build doesn’t pull from the public internet each time. The pipeline has a 2-hour timeout.

The backend pipeline is faster – about 6 minutes. It runs static code analysis, compiles and runs unit tests with Google Test, builds the CNX application, bundles it into an ARM64 OpenWrt image, boots that image in a QEMU virtual machine, and runs the integration test suite against the emulated instance. A 30-minute timeout covers it.

What makes this pipeline unusual is the QEMU step. Instead of just compile and call it tested, we actually spin up a complete emulated OpenWrt environment, wait for services to start, and hit them with real HTTP requests from Python test scripts. If an endpoint returns the wrong status code, or if a configuration change doesn’t apply correctly, the pipeline fails. This catches problems that unit tests would miss – service startup ordering issues, UCI write failures, race conditions under concurrent requests.

Build targets support cross-compilation to three platforms: the Gateworks Venice board (production target), ARM64 (closest to production that can run under QEMU), and x86 (a very different architecture but with stable QEMU support). This multi-target approach let our QA team test continuously even when physical boards were not available.

Artifacts follow a strict naming convention that encodes version, release number, OpenWrt version, target, subtarget, and profile into the filename. Branch builds overwrite each other and expire after two months without a download. Master and release artifacts are permanent and never overwritten.

For local development, we set up Dev Containers. Each developer works inside the same Docker image that Jenkins uses. Whatever passes in their container will pass in CI – no more “works on my machine” surprises from platform or dependency mismatches.

Beyond emulation, the pipeline also deploys built images to physical Gateworks boards in a QA lab. This closes the gap between QEMU testing and real hardware, things like WiFi behavior, MoCA link stability, TPM operations, and GPIO timing only show their true characteristics on an actual board. 

The same integration test suite that runs against the emulated device runs against the lab boards, giving us confidence that what passes in CI will work on a production unit.

Automated Testing

Quality assurance covered three levels. Unit tests in Google Test verify individual functions and modules. Integration tests in Python exercise the full API surface against an emulated device. And manual hardware tests on physical Gateworks boards check things that QEMU can’t simulate – actual WiFi behavior, MoCA link stability, TPM hardware operations, GPIO timing, LED sequencing, and thermal sensor accuracy.

The QA team designed test cases around the OneWeb ICD. Every endpoint has at least one positive test (valid request, correct response) and one negative test (malformed request, proper error handling). For the OTA system specifically, we test uploading firmware with invalid signatures (must be rejected), uploading with valid signatures (must succeed), verifying the inactive partition was written without touching the active one, and forcing a rollback by corruptin