Edge Node Architecture: The Walking Skeleton and Operating in Chaos

Historically — roughly through the mid-2000s — information systems were designed predominantly according to the Waterfall methodology. This approach called for a strictly sequential progression through the stages: from in-depth architectural design and documentation to coding and testing. The waterfall process was reminiscent of the era of Classicism in art, where departures from strict canons were considered unacceptable.

In the era of physical distribution a software mistake was a catastrophe: software shipped on CD-ROMs, and getting a patch to the customer was slow, expensive and often physically impossible. Every change came at an enormous cost. On the other hand, this conservatism had one crucial advantage: the system received a coherent, well-considered and monumental engineering foundation before the first line of code was written.

Yet as the IT industry boomed, the waterfall approach could no longer deliver the speed and flexibility the market demanded. A product could be designed for years only to become morally obsolete, or simply unnecessary to the customer, by the time it was released.

The answer to this challenge was the ideas of Agile. In 2001, 17 leading engineers of the time (among them Kent Beck, Martin Fowler and Robert Martin) formulated the manifesto of agile software development. One of the decisive influences on the formation of Agile was Kent Beck’s Extreme Programming (XP) — a set of rigorous engineering practices: TDD, pair programming, continuous integration and constant refactoring. The essence came down to flexible adaptation: building the product in short iterations, gathering feedback from the market quickly and not being afraid to change the code on the fly.

But in its original conception such an approach demanded the highest qualification of its engineers and uncompromising discipline.

The Walking Skeleton: the engineering core of agile architectures

When Agile became mainstream, much of its original engineering core faded away: XP practices, TDD and architectural hygiene. What remained in mass practice were only the outward rituals (stand-ups, retrospectives and sprints), while the agility itself was often turned by the business into an instrument of micromanagement and chaotic slicing of disconnected business features.

I have often seen how blind adherence to process led teams to create “Frankenstein architectures”. When foundational system decisions are not made deliberately but are “grown on the fly” from sprint to sprint, the system quickly sinks under the weight of technical debt. The next business feature breaks a shaky foundation that nobody ever designed for real-world loads.

Yet within Agile there is a powerful architectural pattern — the Walking Skeleton. The term was introduced by Alistair Cockburn, one of the co-authors of the Agile Manifesto; it is described in detail in his work on the Crystal family of methodologies (in particular, Crystal Clear, 2004). In the era of monolithic server applications the method remained more of a nice metaphor, but in the world of distributed systems and Edge computing it has become a basic engineering necessity.

In essence, the Walking Skeleton is an ideal bridge connecting two worlds: the well-thought-out engineering foundation of classical design and the iterative flexibility of Agile.

For Edge-first systems, the Walking Skeleton is the base from which one must start. We begin not with business logic (face detection, video capture or a beautiful UI), but with building a minimal yet end-to-end and viable framework.

What a skeleton is not

To avoid terminological confusion, let us draw a clear boundary:

Why Feature-First breaks at the edge: the “Let’s start with the neural network” anti-pattern

The main consequence of abandoning the Walking Skeleton is falling into the trap of Feature-First development, where a product is built “from the feature” rather than from the foundation.

A typical scenario from Edge video surveillance practice: a team with a strong ML background starts with the most impressive layer — the neural network. For months, engineers polish the detection pipeline: they optimize weight quantization, reach 98.5% mAP (an illustrative figure) on a reference dataset and draw neat bounding boxes on an office demo stand.

But as soon as the device is mounted at a real site, the system meets physical reality:

• On a sudden power loss, the local database is corrupted in the middle of an unfinished transaction, and the node falls into an endless bootloop.
• The LTE modem loses network registration because of interference, and the code cannot re-establish a TLS session with exponential backoff, because in the office everything ran over stable gigabit Ethernet.
• The disk fills up in three days, because the rotation of logs, frame buffers and the eviction policy was postponed “to the stabilization phase before release”.

The neural network is mathematically flawless, yet the practical value of the device is zero: the node is unstable as a hardware-software system.

If our Walking Skeleton can survive a sudden power cut, a hardware failure of the modem, a camera sensor failure and a critically full disk without losing state consistency, we can confidently and safely build up the application functionality.

Five New Paradigms

For the Walking Skeleton to survive in the physical world — amid emergency power cuts, unstable connectivity, hardware failures and strict resource and thermal limits — the architecture must be grounded in five fundamental paradigm shifts that have taken place in the industry over the past two decades:

1. Security (Environment Integrity Assurance): a shift from perimeter defense to Zero-Trust and cryptographic verification. This paradigm gives software the ability to treat its environment as untrusted by default. It makes it possible to guarantee the authenticity of every byte in the system (Secure Boot → dm-verity), making the node resilient to physical compromise and software tampering by an attacker outside a guarded data center.
2. Reliability (Adaptive Fault Tolerance): a shift from the illusion of achieving absolute uptime to Resilience Engineering. This paradigm enables a system to adapt dynamically to failures instead of trying to prevent them. That is what allows software to survive sudden power cuts and intermittent connectivity through a Crash-Only architecture and degradation patterns.
3. Efficiency (Computational Determinism): a shift from the illusion that “faster hardware will make up for inefficient code” to a memory-conscious approach through Mechanical Sympathy and Data-Oriented Design. This paradigm provides predictable frame Tail Latency (p99.9) and protection from the OOM-killer. Designing around flat data structures (SoA) ensures that the application stays within strict memory limits and that the processor does not sit idle waiting for DRAM.
4. Integration (Hardware–Software Synergy): a shift from isolating software layers to deep compatibility through Hardware-Software Co-Design. This paradigm maximizes FPS per watt by adapting algorithmic math (INT8 Quantization in OpenVINO/ORT) and memory pipelines (Zero-copy / DMA-BUF) to the topology of a particular chip. This makes it possible to fully use specialized accelerators without exceeding the node’s thermal design power (TDP).
5. Observability (Diagnostic Transparency): a shift from passive monitoring (“working/broken”) to end-to-end Observability (OpenTelemetry). This paradigm makes it possible to understand the causes of hidden, nonlinear failures in a distributed system by analyzing correlated logs, metrics and distributed traces, without imposing excessive load on the video pipeline’s hot path.

Understanding these shifts defines the engineering decisions for every component of our Edge node.

1. From perimeter defense to Zero-Trust and Privacy-First

Historically, the security of corporate systems was built on the Castle-and-Moat model: only the outer perimeter was protected (firewalls, DMZ, VPN), and everything inside the local network was assumed to be trusted a priori.

This model was seriously undermined by sophisticated targeted attacks of the late 2000s and early 2010s. It did not disappear (the perimeter remains one layer of defense), but it ceased to be considered sufficient. The turning point was the Operation Aurora (2009) attack on the infrastructure of Google and a number of technology giants. The attackers carried out targeted phishing of employees and exploited a 0-day vulnerability in a browser, after which they were able to move laterally (lateral movement) inside the “trusted” corporate network.

Google’s response was the BeyondCorp (2014) initiative — one of the first large industrial implementations of the ideas of Zero-Trust, a term for which was coined by Forrester analyst John Kindervag (2010). The approach was formalized considerably later, in NIST SP 800-207 (2020).

The basic postulate of Zero-Trust: “Never trust, always verify.” The concept requires a complete rejection of privileged zones and the construction of Defense in Depth:

• At the data level: Row-Level Security (RLS) and cryptographic isolation that do not trust even the backend code.
• At the service level: mutual end-to-end authorization of every inter-process call.
• At the network level: treating any network (including the local LAN) as a hostile environment by default.

Important: Zero-Trust is not “closed ports and mTLS” in themselves, but the principle of explicit authentication and authorization of every request regardless of network location. Closed ports, mTLS and hardware device identity are merely its concrete implementations on an Edge node.

Applying Zero-Trust to Edge-first design

An Edge node is physically located on the customer’s premises: in an electrical cabinet in a warehouse, on a pole, or in an open server rack. Anyone can reach it with an ordinary screwdriver. The network transport is a public LTE channel or the enterprise guest network. The threat model includes from the outset: physical removal of the storage drive, traffic interception (MITM), Rogue DHCP, DNS spoofing and attempts to clone the firmware.

The cameras themselves are a separate problem: many IP cameras do not support RTSPS, live with default passwords and outdated firmware. Therefore cameras are moved to an isolated VLAN or onto a separate network interface of the node, and encryption and authentication on this segment are enabled wherever the camera supports them (RTSPS, ONVIF authentication); in other cases the segment is treated as untrusted and constrained physically and by network policy.

Based on Zero-Trust, we establish three architectural rules:

1. No inbound connections at all (No Listening Ports). The node listens on no network port: SSH daemons, HTTP administration servers and open RTSP endpoints are disabled. All communication consists exclusively of outbound long-lived sessions (Outbound TCP/TLS 443) to validated cloud endpoints. Authentication is built on strict mTLS (mutual TLS) with two-way certificate verification. Certificate rotation is carried out through the regular OTA channel: the new certificate is delivered in an encrypted payload signed by a trusted CA and picked up at the next planned reconnect. For cold start and emergency recovery, a unique device identity is used: a client certificate whose key is generated inside the Secure Element and never leaves it, with the certificate signed by the production CA during assembly (in the spirit of IEEE 802.1AR / DevID). There are no shared tokens across the fleet: compromising one device does not affect the others.

2. Hardware Root of Trust and isolation. Cryptographic keys and private certificates are generated and stored inside a cryptographic coprocessor (Secure Element / TPM 2.0) that does not allow the private key to be extracted in software. The specific implementation depends on the platform: on the RK3588 this is typically a discrete Secure Element/TPM (or an fTPM based on OP-TEE); on Jetson Orin, an fTPM/OP-TEE and its own Secure Boot chain. Software integrity is guaranteed by Secure Boot (cryptographic verification of the chain: Bootloader → Linux kernel → dm-verity root filesystem). As far as I know, today this is the most reliable and mathematically proven scheme for protecting OS integrity (the hardware trusts only the bootloader, the bootloader trusts only the kernel, and the kernel trusts the filesystem only through dm-verity’s cryptographic tree). Inside the OS, the Principle of Least Privilege is applied: every component that works with untrusted input or keys is moved into a separate process under its own uid with a seccomp profile (see the “Threading model” section), so a compromised video process (video-pipeline, in which CameraManager runs) has no access to the archive encryption keys or to the NodeStore configuration database.

3. Privacy-First and Two-Envelope Encryption. We rule out trust in the local storage medium. If an SSD is stolen or a memory chip is desoldered, the attacker must not gain access to video recordings. The local archive is protected as follows:

   • Every video segment is encrypted with a unique symmetric data key (DEK — Data Encryption Key, AES-256-GCM, a unique nonce per segment).
   • The DEK is sealed in a nested fashion: first with a hardware-bound node key from the Secure Element (Node KEK), then with the public key of the end customer or a cloud KMS (Client KEK): Enc_ClientKEK( Enc_NodeKEK(DEK) ). It is the nesting that sets the “AND” logic: the client removes the outer layer with their private key, the node removes the inner one through the Secure Element; without either participant the DEK cannot be obtained.
   • If the DEK is wrapped by two independent wrappers (one per key), the result is “OR” logic: the archive can be opened by either key. This variant is appropriate when the node needs to read its own archive autonomously, but it is a different threat model.
   • Limits of the scheme. It protects data at rest: against theft of the drive and against on-site personnel who do not hold the keys (end-to-end encryption with a key held by the client). The term Zero-Knowledge is not applicable here: a running node knows the DEK at the moment of writing. The scheme does not cover compromise of a live node — Secure Boot, process isolation and closed ports serve that purpose. Because the Node KEK is bound to the hardware, a failure of the Secure Element makes the local archive unreadable: the archive is a cache, not the only copy.

2. From the illusion that perfect uptime can be achieved to Resilience Engineering

In parallel with the shift in security, a revolution was under way in reliability theory. The classical engineering school proceeded from a presumption: the goal of design is to create a defect-free system through exhaustive specifications and long verification cycles. A failure was regarded as a gross violation of the technological process.

This illusion was challenged by sociologist Charles Perrow in his seminal work “Normal Accidents” (1984). Analyzing technological incidents in critical systems (including the Three Mile Island nuclear accident), Perrow formulated Normal Accident Theory:

Related ideas later found practical expression in the IT industry and in the Resilience Engineering school (Hollnagel, Woods and others, mid-2000s). In the mid-2000s, Erik Hollnagel, David Woods and Sidney Dekker proposed not giving up, but changing the very approach to safety.

• Safety-I (the old approach): safety is the absence of incidents. Find someone to blame, write even more instructions, tighten the rules and make the system more rigid.
• Safety-II (Resilience Engineering): safety is the system’s ability to adapt dynamically to changing conditions and keep operating even when something goes wrong. A system is safe not because it has no failures, but because people and mechanisms know how to handle the unexpected.

The ideas of Resilience Engineering became the foundation for modern SRE (Site Reliability Engineering), DevOps culture and cloud architectures:

• Design for Failure: we no longer build “unfailing” servers. We build systems in which the failure of an entire data center (a micro-catastrophe in Perrow’s terms) is handled automatically by a load balancer, without users even noticing.
• Chaos Engineering: Netflix’s Chaos Monkey tool was a direct embodiment of Resilience Engineering. We deliberately break production servers at random times to test how well the system can adapt to stress.
• Blameless Postmortems: instead of looking for someone to blame (a practice Sidney Dekker criticized), the IT industry investigates why the system allowed an administrator to make the mistake and how the architecture can be made more resilient to human factors.

Key milestones in the shift to the new paradigm:

• Netflix (2010–2011) created the Chaos Monkey tool and gave rise to the practice of Chaos Engineering, deliberately taking production nodes out of service during working hours — when engineers are on site and can respond quickly. The philosophy: instead of futile attempts to prevent failures, a system must be designed to degrade continuously and self-heal without degrading the user experience (Graceful Degradation).
• Publication of the Google SRE book (2016) brought the concept of the Error Budget into wide use: an engineering-economic argument that 100% reliability is unattainable and disproportionately expensive, so the acceptable level of failures is made explicit.
• AWS, through the voice of CTO Werner Vogels, cemented the fundamental rule of modern distributed systems: “Everything fails all the time.”

3. The shift from the illusion that “faster hardware will make up for inefficient code” to Mechanical Sympathy and Data-Oriented Design

The third shift took place at the intersection of software architecture and the physical limits of silicon.

Until the mid-2000s, software development evolved within the “Free Lunch” concept: the performance of a single processor thread grew exponentially (Moore’s law — roughly a doubling every 18–24 months), largely thanks to rising clock frequencies and microarchitectural improvements. The inefficiency of abstractions and deep class hierarchies in OOP was offset by the power of the next generation of CPUs.

This paradigm collapsed under two physical barriers:

1. The Memory Wall. Back in 1995, William Wulf and Sally McKee, in their paper “Hitting the Memory Wall”, showed (by extrapolating trends) that the gap between the exponentially growing speed of the processor and the slow growth of main-memory bandwidth would lead to the processor spending the overwhelming share of its time idle (CPU stalls), waiting for data from DRAM.
2. The end of Dennard Scaling. By 2005, semiconductor physics had run into a thermal barrier: further increases in clock frequency led to critical leakage current and an avalanche-like rise in chip power consumption. Herb Sutter captured this turning point in his landmark article “The Free Lunch Is Over” (2005): free performance gains had ended, and computing had become parallel and heterogeneous.

The memory hierarchy and the cost of data access (if 1 cycle ≈ 1 second):

A modern processor executes instructions in registers using fractions of a picojoule of energy, whereas fetching 64 bytes (a cache line) from external DRAM requires three to four orders of magnitude more energy than an integer addition, as well as hundreds of stall cycles (Mark Horowitz, ISSCC 2014). In his ISSCC 2014 presentation, Mark showed that physically moving electrons along long copper traces on the motherboard from DRAM to the processor consumes enormous amounts of energy compared with shifting charge inside the processor’s tiny die. The processor no longer computes continuously — it spends long stretches waiting for data to arrive.

The industry’s response was the paradigm of Mechanical Sympathy (“sympathy for the hardware”) — a term Martin Thompson borrowed from racing driver Jackie Stewart and popularized while developing the ultra-low-latency LMAX Disruptor (2011). The practical discipline of this concept became Data-Oriented Design (DOD), formulated in Mike Acton’s famous 2014 talk (CppCon 2014). In essence, we should view a program as a transformation of data from one representation to another on specific physical hardware. If we do not understand how application data is laid out in memory and how the processor loads it into its cache hierarchy, we cannot write high-performance code: the processor will spend its time idle.

If we distill the concept to its practical meaning:

A developer must understand the hardware reality: cache lines, DRAM latency and the branch predictor. Writing code without regard to the hardware is, by definition, creating inefficient programs. For example, in OOP we think in terms of “an entity has properties” (for instance, each enemy in a game has coordinates, health, a name and a texture, all stored together in one heap object). In Data-Oriented Design, we view data as streams. If we need to update the coordinates of 10,000 objects, the processor cache does not need their names or textures. It needs a dense, contiguous array containing only coordinates — a shift from the traditional Array of Structures (AoS) to the Structure of Arrays (SoA) pattern.

We must put this knowledge to work when building Edge-first video systems, where the load on resources is enormous.

Applying Mechanical Sympathy to Edge-first systems

A cloud service scales horizontally and dynamically as load grows. An Edge node built on a compact SoC (for example, Rockchip RK3588, NVIDIA Jetson Orin or NXP i.MX8), installed in a sealed enclosure without active cooling, is tightly constrained:

• Unified Memory & TDP Throttling: the CPU, NPU (neural accelerator), VPU (hardware video decoder) and GPU sit on one die and share a common LPDDR memory bus and a common thermal design power (TDP). If control code on the CPU allocates memory chaotically, chases object graphs and provokes constant L3 cache misses, it overloads the shared memory controller and heats the chip. As a result, dynamic frequency throttling (DVFS) kicks in, the NPU slows down, and the system starts dropping video frames.
• Determinism versus peak delays (Tail Latency): in Edge video analytics, the target metric is not average FPS but meeting the per-frame budget at p99.9 (33.3 ms for a 30 frames-per-second stream). An unpredictable pause — a mutex block, an allocation, heap fragmentation, preemption by the scheduler — leads to a missed detection of a critical event. A caveat: neither Linux nor tokio provides hard real-time guarantees (Rust has no garbage collector, but that does not make the runtime real-time). So p99.9 is for us a target metric that we measure under load and under degradation, not a contractual guarantee.

Three architectural invariants follow from this:

1. Bounded by Design (strictly bounded resources). No internal queue, ring buffer or cache table may grow dynamically without limits. Memory for frame pools and ring buffers of events is allocated once at process initialization (Static Allocation / Bounded Pools). Local disk storage is a strictly limited structure with transparent ring rotation: free disk space will not fall to zero, because the amount of data on the node is bounded by a design constant. This is a pragmatic version of rule #3 of the NASA/JPL Power of Ten standard (“no dynamic memory allocation after initialization”). In a Rust project with tokio, TLS and an embedded database, it cannot literally be satisfied, so we apply it to the hot video path and to the queues, while allocations elsewhere are bounded and monitored.
2. A zero-copy video data pipeline. An uncompressed 4K frame in NV12 format weighs ~12 MB. Copying one such frame by value at 30 fps creates parasitic internal traffic of about 360 MB/s; for 8 streams of 1080p this figure is about 750 MB/s (1920×1080×1.5 ≈ 3.1 MB per frame) — and that is only one copy pass, while read plus write doubles the load on the bus. On a compact SoC this consumes the memory bandwidth the NPU needs to fetch model weights. Therefore a decoded frame is passed only as a descriptor (dmabuf): inside video-pipeline it is an ordinary transfer of ownership of a buffer handle between CameraManager and InferenceManager, with no copying and no IPC. The VPU writes the frame into a buffer, scaling and format conversion for the NPU are done by a hardware block (for example, RGA on Rockchip), and the NPU reads the buffer directly. The archive branch does not need decoding at all: CameraManager takes the original compressed H.264/H.265 packets and remuxes them into 10-second MP4 segments — there is neither a decoder nor an encoder here. A finished segment goes to vault as an opaque block of bytes via memfd (a descriptor over SCM_RIGHTS), without running the contents through a socket. An honest caveat: “zero copies” is an ideal; on specific SoCs and driver versions some formats have to be converted, and this is found through profiling.
3. Flat state and the Crash-Only Software model. The system state (active cameras, analytics zone configuration, local rules, track metadata) is stored not as a complex graph of objects on the heap with cross-references, but as flat contiguous data structures (SoA/Flat Tables) accessed by direct indices. This eliminates cyclic references, speeds up linear passes over processor cache lines and enables instant snapshots. Any failed manager can be forcibly killed and restarted at any moment (by systemd or NodeSupervisor): flat state is trivially restored from the local database. For lightweight components a restart takes milliseconds; for processes with NPU/VPU it takes hundreds of milliseconds to seconds (re-initialization of drivers and the hardware context), and this is built into the degradation budget. This is an implementation of the Crash-Only Software concept (Candea & Fox, Stanford): the system does not spend resources on a complicated graceful shutdown on fatal errors; its standard recovery mechanism is a fast deterministic restart.

4. The shift from layer isolation to end-to-end compatibility (Hardware-Software Co-Design)

The fourth tectonic shift took place at the boundary between pure software architecture and the physical topology of silicon.

For decades, classical engineering erected walls of abstraction around the developer. We were taught: “Good software is completely isolated from the hardware platform. The architecture should be clean, and the operating system, virtual machine and drivers will hide the hardware’s quirks.” This illusion served the industry well while the performance of general-purpose CPU cores kept rising thanks to smaller process nodes.

But when Dennard scaling (around 2005) brought engineers up against the thermal barrier (the Power Wall), the silicon industry took the only path left: deep specialization of computing units. A modern energy-efficient processor is no longer “just a general-purpose CPU.” It is a heterogeneous conglomerate in which most of the die area is devoted to specialized coprocessors. Even in Intel’s x86 architecture, a modern chip is a complex system combining performance (P) and efficiency (E) cores, integrated graphics (Iris Xe / Arc GPU), an integrated NPU and specialized vector units for matrix computations.

Trying to run “abstract software” written without regard to the chip’s physical structure leads to disaster: the application uses only a small fraction of the silicon, leaving specialized accelerators idle while still consuming TDP on needless data movement between layers.

The industry’s response to this crisis was Hardware-Software Co-Design. The architecture of a modern high-performance application can no longer be independent; it must become a direct software reflection of the topology of a specific processor.

Origins and industry context:

The term itself arose in academic circles in the 1990s (in the work of Stanford and Princeton researchers such as Giovanni De Micheli and Wayne Wolf), but at the time it applied exclusively to ASIC design and embedded systems. Co-Design entered broad IT-industry practice in the mid-2010s, when Moore’s Law finally began to slow down.

The turning point was marked by Chris Lattner (creator of the LLVM compiler, Swift and the MLIR architecture), who proclaimed the arrival of the “Golden Age of Hardware-Software Co-Design.” In practice, the shift is now used by the major Silicon Valley players: Apple created its Apple Silicon (M1/M2/M3/M4) chips by tailoring the silicon to the macOS/iOS operating-system runtime. NVIDIA built its trillion-dollar market capitalization not simply by selling chips, but through the end-to-end combination of CUDA architecture and software libraries (its tensor cores are physically designed around the mathematics of modern transformers). Google designs its tensor processing units (TPUs) in tandem with the developers of the TensorFlow/JAX frameworks.

If we distill the concept to its practical meaning:

For Edge-first video systems built with Rust + OpenVINO / ONNX Runtime on a heterogeneous hardware platform, the Co-Design paradigm dictates three practical rules:

• Algorithmic mathematics is shaped by transistor-level capabilities (Intel VNNI/AMX). The processor’s physical registers are built for ultra-fast vector arithmetic. Leaving neural networks in the default f32 format means heating the air and hitting the Power Wall. The software developer deliberately gives up abstract precision and uses OpenVINO tools to quantize models to INT8. This allows the framework to use Intel VNNI (Vector Neural Network Instructions) or AMX (Advanced Matrix Extensions), multiplying integer data vectors in a single cycle and delivering a multiple-fold FPS gain.
• The memory model mirrors the physical buses (Zero-Copy + Remote Tensors). Instead of blindly allocating memory on the Rust heap, the software pipeline connects directly to the chip’s hardware boundaries. Through Intel’s low-level video-acceleration interface, Intel VA-API, a frame is decoded directly into shared memory (Unified Memory). Then, using OpenVINO Remote Tensors, the software passes the built-in GPU or NPU a handle to the physical buffer, not megabytes of pixels. Rust’s strict ownership model ensures that data travels from the decoder to inference without a single system-level copy, bypassing the “memory wall.”
• The code’s asynchrony mirrors the asynchrony of Intel’s hardware blocks. Heavy neural-network computation must not compete with the application’s asynchronous runtime. Launching inference inside a shared tokio task makes the scheduler move threads chaotically between P- and E-cores, causing cache misses and wrecking p99.9 timings. The right Co-Design choice is the non-blocking OpenVINO Async API (AsyncInferRequest). A Rust CPU thread simply submits a task to the accelerator’s hardware queue and immediately returns control to the tokio runtime (through Futures). Meanwhile, OpenVINO’s internal compute threads are bound (Thread Affinity / CPU Pinning) to dedicated physical cores, isolated from the main application.

Breaking down the barrier between software and hardware is not a temporary measure. In an era beyond Moore’s Law and dominated by specialized AI accelerators, Hardware-Software Co-Design is the only way to make our Rust stack run at the hardware’s physical limits without overheating the node or dropping frames.

5. The shift from passive monitoring to end-to-end Observability

The fifth tectonic shift took place in the discipline of diagnosing and monitoring distributed computing environments.

For a long time, classical software engineering relied on the paradigm of passive Monitoring. Engineers were taught: “System health is monitored using a predefined set of failure indicators (CPU metrics, RAM, free disk space) and reactive text logs.” The system was treated as a black box, and an incident was recorded after the fact, once a metric crossed a critical threshold. This illusion of stability broke down against the nonlinear nature of modern asynchronous architectures and distributed Edge networks.

In systems built on Rust’s asynchronous runtime (tokio) and heterogeneous compute, traditional monitoring is powerless. A system can be in a hidden cascading failure state: generic CPU and memory metrics report perfect health (50% utilization), and the processes are alive, but micro-delays on the memory bus (Memory Wall), collisions in scheduler queues or thermal throttling silently cause frames to be dropped. It is physically impossible to find the root cause from conventional, isolated logs, because they lack end-to-end causal links.

The industry’s response to this crisis was the shift to Observability, whose mathematical foundations trace back to Rudolf Kalman’s 1960 control theory for linear systems. Observability is a measure of how accurately a system’s internal state can be reconstructed from an analysis of its external manifestations.

Origins and industry context:

The turning point came in 2019, when two independent distributed-tracing projects — OpenTracing (Uber) and OpenCensus (Google) — merged under the umbrella of the CNCF (Cloud Native Computing Foundation). The result was OpenTelemetry (OTel), which became the second most active project in CNCF history after Kubernetes. The industry reached a consensus: instead of fragmented “data silos” (logs on their own, metrics on their own), a single end-to-end specification was needed to link the three telemetry data types — Logs, Metrics, Traces — through shared distributed context (Trace Context).

If we distill the concept to its practical meaning:

The shift to Observability turns Edge-node diagnostics from blind reading of text logs into a precise engineering discipline. When physical access to a device is unavailable and inbound ports are blocked by Zero-Trust rules, Observability becomes the only window into the internal state of the Edge system.

A single conceptual vector across the five paradigms

When we bring together Zero Trust, Resilience Engineering, Mechanical Sympathy, Hardware-Software Co-Design and Observability, their strict conceptual symmetry and shared logic of moving away from classical abstractions become clear:

Note: “Assume Constraint,” “Assume Specialization” and “Assume Unknown Unknowns” are the author’s own terms; the quintet of basic postulates is a rhetorical generalization, not a formal model.

Conflicts between paradigms: where theory meets reality

A mature architecture differs from an academic one in that it openly acknowledges and resolves internal contradictions between paradigms. Some of them are more compromises of implementation than fundamental conflicts:

• Zero Trust versus Zero-Copy: encryption requires an extra pass over the data. Resolution: raw decoded frames on their way to the NPU are not encrypted — they do not leave the SoC or shared memory. Only the compressed archive segment is encrypted before being written to the drive: AES-GCM runs either on CPU cores with hardware instructions (ARMv8 Crypto Extensions / AES-NI), or on a dedicated SoC crypto engine, if one exists and has a driver. The volume of encrypted data is the bitrate of compressed video (single-digit MB/s), roughly two orders of magnitude less than the traffic of uncompressed frames, so the conflict with zero-copy is small.
• Resilience versus Bounded Storage: the Store-and-Forward pattern strives to accumulate events indefinitely during a network outage, while the Bounded Storage principle strictly forbids exceeding the disk quota. Resolution: introduce multi-level eviction priorities (Eviction Policy). Critical metadata and alerts (PersonDetected) are held in a priority queue with guaranteed delivery, while heavy video clips are evicted on a FIFO basis when the disk fills up, or are compressed with reduced bitrate and frame rate.
• Zero Trust versus Observability: the node has no open inbound ports, so standard pull-based metric collection (for example, polling by Prometheus through an HTTP endpoint) is impossible. Resolution: move to a pure push mechanism: a local telemetry broker itself opens an outbound TLS session to the cloud. The telemetry queue buffer also obeys the Bounded Resources principle: during a long offline period, detailed traces collapse into aggregated histograms without consuming disk.
• Co-Design versus Observability: deep tracing (OpenTelemetry) and profiling the pipeline at every stage of frame transformation create substantial memory and CPU overhead, erasing the gains from hardware acceleration. Resolution: dynamic contextual sampling (Contextual Sampling). An end-to-end TraceID is generated by the Rust runtime for every frame and passed implicitly through OpenVINO/ONNX Runtime contexts without allocations. However, the collection and serialization of full metrics (neural-network layer latencies, memory-bus metrics) are activated only when triggered — if frame p99.9 latency exceeds the target budget (for example, >33.3 ms) — or forcibly for 0.1% of the frame stream.
• Co-Design versus Resilience: strict binding of runtime compute threads (OpenVINO TBB / ORT Thread Pool) to physical processor cores (Thread Affinity) protects against cache misses but deprives the OS scheduler of flexibility. If one CPU core is overloaded by a system interrupt, a pinned inference thread will “hang,” producing a tail-latency spike. Resolution: dedicate isolated cores to inference at the Linux-kernel level (using the isolcpus boot parameter or cgroups), removing them entirely from the general operating-system scheduler and tokio. Inference pipelines run on a dedicated, exclusive slice of silicon, making restarts predictable within the Crash-Only model.

Edge chaos in practice: Chaos Engineering on a test bench

On an Edge node, theoretical failures become everyday reality. Below are illustrative reference points from practice (not academic data; specific values depend heavily on the class of the drive, the modem and operating conditions):

• Consumer-grade flash memory under round-the-clock continuous writing fails within 9–14 months.
• An industrial LTE module at a remote site loses the base station 3 to 10 times a day.
• Sudden voltage spikes (brownouts) switch the node off at the moment the SSD controller cache is being flushed, reliably destroying unfinished data sectors.

To verify our Walking Skeleton, we built an automated bench for hardware and software Fault Injection, run in CI (software failures — on every commit; hardware ones, including power loss, — on the bench on a schedule and before a release; more detail in the section “What is still missing”):

What goes into the skeleton and what does not: the irreversibility rule

The architecture described below is the target one. The Walking Skeleton is not all of it, but a thin slice of it. If we try to build everything at once, we get exactly the Big Design Up Front that Agile once moved away from: months of design with no feedback from real hardware.

The selection criterion is simple, and it follows from the principle we have already applied to optimizations: the skeleton gets what cannot be cheaply redone after deployment to sites; everything else grows iteratively.

• Irreversible (goes into the skeleton): the storage partition layout; the device identity format and the factory provisioning process; the Secure Boot chain; the privilege model (who owns which keys and devices) and the message contracts between processes; the format of stored records and their versioning; the memory ownership model and the queue loss policy; the management protocol (versioning and signing of the Desired State); and the A/B update mechanism itself, with rollback — if OTA is broken, the device cannot be repaired remotely.
• Reversible (deferred): the neural network model and its quantization; the rule editor; telemetry aggregation; hardware crypto offload; canary rollout; fine-tuning of seccomp profiles and DVFS.

The number of processes is not among the irreversible items: if components communicate only through serializable messages, a process can be split or merged later.

Definition of Done for the skeleton

1. One camera → an event (even from a stub detector) → an alert in the cloud → PUBACK → removal from the queue.
2. Autonomous relay actuation with the network disconnected.
3. OTA: a successful update and a forced rollback (a deliberately broken firmware returns itself to the previous slot without human involvement).
4. The four fault-injection scenarios pass; power loss is verified over a series of cycles, not a single run.
5. A soak test (a day or more) with no RAM drift and no disk growth beyond the quota; the frame p99.9 is measured, not assumed.

The skeleton stays “walking”: it contains exactly what the rest cannot be safely built upon without, and nothing beyond that. The rest of the article describes the target architecture from which this slice grows; under each heading it is noted what part of it goes into v0.

Target architecture of an Edge node: what the skeleton grows into

The node architecture combines the concepts of Cloud-Native systems, strictly bounded by the physical capabilities of edge devices and implemented in Rust. This is a map of the target state: which part of it goes into v0 is marked in the “In the skeleton” blocks under the section headings.

The dashed containers in the diagram are OS process boundaries: node-core (supervisor, rules, cloud bus, storage), video-pipeline (camera and inference), vault (encryption and archive) and vector; alongside them is a ready-made OTA service (RAUC / SWUpdate). Video data moves over direct channels, bypassing node-core; the lifecycle of processes is managed by systemd.

1. State model: Declarative State Reconciliation

In the skeleton (v0): a signed and versioned Desired State, a single reconcile() loop, the node’s report of its actual state, last-known-good (see the section “What is still missing”).

Later: schedules, complex analytics zones, rich policies.

Most distributed systems lose consistency because of imperative control (“switch the camera to 1080p right now”). A command sent to an offline node can be delivered later via a persistent session and QoS 1, but then you have to deal with ordering, repeats and stale commands. If a command is lost or applied twice, a divergence arises between what the cloud believes and the node’s actual state (state drift).

We use a declarative approach, adapted from the Kubernetes architecture:

1. The cloud backend, via an MQTT broker (EMQX), publishes a Desired State document (the node’s desired state: the list of cameras, detection parameters, active schedules).
2. On the node, the root orchestrator — NodeSupervisor — runs a constant reconciliation cycle (Control Loop).
3. The .reconcile() method compares the actual state of the child actors with the desired one and eliminates discrepancies:

// Simplified concept of the Control Loop inside NodeSupervisor
const RECONCILE_TIMEOUT: Duration = Duration::from_secs(3);

pub async fn run_reconciliation_loop(
    supervisor: Arc<NodeSupervisor>,
    mut desired_state_rx: watch::Receiver<NodeDesiredState>,
    mut shutdown_rx: broadcast::Receiver<()>,
) {
    // watch holds only the LATEST desired state: if the cloud sent three versions
    // while the node was offline, it applies the current one rather than replaying them all.
    // The channel's initial value is loaded from NodeStore, so after a restart or a loss
    // of connectivity the node keeps working from the last known Desired State.
    let mut interval = tokio::time::interval(Duration::from_secs(5));
    interval.set_missed_tick_behavior(MissedTickBehavior::Delay);
    let mut desired_open = true;

    loop {
        tokio::select! {
            _ = shutdown_rx.recv() => {
                log::info!("Supervisor shutting down on the shutdown signal");
                break;
            }
            // Apply a new state immediately, without waiting for the next tick
            res = desired_state_rx.changed(), if desired_open => {
                if res.is_err() {
                    // The source is closed: don't spin in a busy loop, continue on ticks
                    log::warn!("Desired State source closed, continuing with the last known state");
                    desired_open = false;
                    continue;
                }
                let desired = desired_state_rx.borrow_and_update().clone();
                reconcile_all(&supervisor, &desired).await;
            }
            // Periodic check: catches drift even when there were no new commands
            _ = interval.tick() => {
                let desired = desired_state_rx.borrow().clone();
                reconcile_all(&supervisor, &desired).await;
            }
        }
    }
}

async fn reconcile_all(supervisor: &NodeSupervisor, desired: &NodeDesiredState) {
    for manager in supervisor.managers() {
        // Timeout: one hung manager must not block reconciliation of the others
        match tokio::time::timeout(RECONCILE_TIMEOUT, manager.reconcile(desired)).await {
            Ok(Ok(())) => {}
            Ok(Err(err)) => {
                log::warn!(
                    target: "supervisor",
                    "Reconciliation error for manager {}: {err:?}. Applying backoff",
                    manager.name()
                );
                manager.handle_reconcile_failure().await;
            }
            Err(_) => {
                log::warn!(
                    target: "supervisor",
                    "Reconciliation of manager {} timed out. Applying backoff",
                    manager.name()
                );
                manager.handle_reconcile_failure().await;
            }
        }
    }
}

The Desired State is stored as a “latest value” (tokio::sync::watch): if the cloud sent several versions while the node was offline, it applies the current one rather than replaying them all. The last known state is saved in NodeStore, so after a restart the node works autonomously.

If an RTSP stream drops, CameraManager switches to a degraded state and methodically tries to reconnect on an exponential timer (Exponential Backoff with an upper limit — the Backoff Ceiling). reconcile() operations are idempotent at the level of each manager: re-applying the same Desired State changes nothing, so loss or duplication of packets does not break eventual consistency. For managers moved into separate processes, reconcile() means sending the desired configuration over IPC and checking the acknowledgment.

2. Data model: Store-and-Forward, Postcard and the crash-safe redb database

In the skeleton (v0): redb with versioned records, an event_queue with a hard quota and two priorities (critical / the rest), a loss policy by data type.

Later: telemetry aggregation at 80% fill, finer priority gradation, archive compression with reduced bitrate.

To work offline, the node buffers events in a local embedded database. We use redb — an embedded ACID database in pure Rust. This does not mean SQLite is bad: in WAL mode with synchronous=FULL it also reliably survives a power loss and has been battle-tested far longer. The choice of redb is motivated by pure Rust (no C dependency and no FFI), a typed table API and a predictable memory model. The project is younger, so isolation behind the NodeStore trait is insurance, not a formality:

1. Resilience to power failures (Copy-on-Write): redb is built on Copy-on-Write: updated data is written to free pages, and the switch of the B-tree root pointer happens atomically only after a successful fsync. This sharply reduces the risk of corruption and removes the need for a WAL journal, but it does not remove the dependence on the drive: the guarantee works only if the device honestly performs the flush. Consumer-grade eMMC/SSDs without power-loss protection (PLP) may violate write ordering or corrupt pages at the level of their own FTL. So we complement the database with a choice of drive (with PLP or with well-documented behavior), checksums and verification on a bench with real power cuts.
2. Compact binary serialization: all system structures (CameraConfig, StoragePolicy, SecurityRule, EventQueueItem) are serialized with Postcard — a compact binary no_std format. This eliminates the overhead of JSON/XML parsing and minimizes parasitic heap allocations. But Postcard is not self-describing (no field names or tags): the set and order of fields are part of the contract. So every record in the persistent queue is wrapped in a versioned enum (EventV1, EventV2, …) with an explicit schema number, and until a new firmware is confirmed, it writes records in a format that the previous version can also read — otherwise, after an automatic rollback, the old firmware would not be able to read the queue.

Access to storage is isolated behind the NodeStore trait. This forms a strict abstraction layer: if specific hardware requires a different engine in the future, all the managers’ business logic stays untouched.

#[async_trait]
pub trait NodeStore: Send + Sync {
    async fn enqueue_event(&self, event: &EdgeEvent) -> Result<EventId, StoreError>;
    async fn dequeue_event_batch(&self, batch_size: usize) -> Result<Vec<PersistedEvent>, StoreError>;
    async fn mark_events_committed(&self, ids: &[EventId]) -> Result<(), StoreError>;
    async fn get_storage_quota(&self) -> Result<StorageQuotaInfo, StoreError>;
}

Queue prioritization (Bounded Event Queue)

In accordance with the Bounded Resources principle, the local event_queue table is strictly limited in size:

• When the queue is 80% full, the system starts to aggregate homogeneous telemetry (collapsing repeated sensor statuses within a time window).
• At 95% fill, low-priority logs (Debug, Trace, Info) are dropped.
• Critical security events (PersonDetected, TamperAlarm, TamperSensorTriggered) have absolute priority and are retained until the guaranteed table quota is exhausted. If the critical-event quota is nonetheless exhausted, the oldest critical records are evicted (a ring), and the fact of the loss is recorded by the events_dropped_total counter, which goes to the cloud at the first connection.

3. Threading model: isolated processes and actors

In the skeleton (v0): three processes (node-core, video-pipeline, vault) and the IPC contract; lifecycle under systemd (restarts with backoff, watchdog); a real camera, with a stub detector or a ready-made model “as is”.

Later: fine-grained seccomp profiles and Landlock, cgroup tuning, splitting video-pipeline into two processes (if soak-test data supports it), all managers at full functionality.

The architecture rules out shared access to mutable state through a scatter of mutexes (Arc<Mutex<T>>), which provoke deadlocks and contention in the asynchronous runtime. Instead, the logic is divided into actors that communicate by messages. But isolation has two levels, and it is important not to confuse them:

• Logical — separate tokio tasks within one process. It protects against mutual deadlocks and allows a Rust panic to be caught, but does not protect against a segfault/UB in a C library, against abort() and against OOM: the whole process goes down. It also creates no privilege boundaries — all tasks share one address space.
• Hard — separate OS processes. Only this really limits the blast radius and implements Least Privilege.

Therefore everything that works with untrusted input (RTSP parsing, the decoder, the remux), with proprietary C libraries (VPU/NPU) or with keys is moved into separate processes. We keep the number of processes to a minimum: every extra process is one more IPC contract, one more unit in systemd and one more row in the test matrix.

Why camera and inference are one process

Both parts process the same untrusted video stream through vendor libraries, and neither has keys or a path to the cloud — so on compromise the attacker gets the same thing. And inside one process, a frame is passed as an ordinary buffer handle: no descriptor exchange and no extra IPC on the hot path.

The boundary between them is drawn at the code level: CameraManager and InferenceManager communicate through traits (FrameSource, Detector) and messages. If soak-test data shows that the NPU runtime crashes or hangs noticeably more often than ingest, the process is split in two without rewriting the logic. We make this decision based on data, not in advance.

Why vault is tiny

vault receives finished MP4 segments as opaque bytes and does not parse them. Video parsing is the most vulnerable part of the pipeline, so it stays in video-pipeline, which sees no keys. A vulnerability in the parser then does not open access to key operations.

A hardware caveat: how access to the Secure Element is separated between processes depends on the platform. A TPM 2.0 with a resource manager issues keys with per-process policies. A simple I²C crypto chip does not distinguish processes by itself: access to it is limited by device permissions, and only one process talks to the chip. In that case, TLS signing for node-core would have to be proxied through that same keeper process.

Who supervises the processes: systemd

We do not write the process lifecycle ourselves: systemd already knows how to do restarts with a delay, watchdog, memory limits, sandboxing and socket activation. Restarts with growing delay (RestartSteps and RestartMaxDelaySec) are available in systemd 254+; on older versions a fixed RestartSec remains. NodeSupervisor is responsible for the logic: it distributes the Desired State over IPC, collects statuses, decides that a component is Degraded, and, on a prolonged logical malfunction, asks systemd for a restart through D-Bus.

# /etc/systemd/system/rockedge-video.service  (simplified example)
[Unit]
Description=RockEdge video pipeline (camera ingest + inference)

[Service]
# The process itself reports READY=1 through sd_notify
Type=notify
User=rockedge-video
Group=rockedge-video
ExecStart=/usr/libexec/rockedge/video-pipeline

# Crashes and hangs
Restart=on-failure
RestartSec=500ms
# systemd 254+: the delay grows in steps up to RestartMaxDelaySec
RestartSteps=6
RestartMaxDelaySec=60
# The process must send WATCHDOG=1 regularly
WatchdogSec=10s

# Resources and sandbox
MemoryMax=1536M
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
SystemCallFilter=@system-service
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
# Network only to the cameras
IPAddressDeny=any
IPAddressAllow=192.168.50.0/24
# Devices: names depend on the BSP (example for Rockchip); add the NPU node for your platform
DevicePolicy=closed
DeviceAllow=/dev/mpp_service rw
DeviceAllow=/dev/rga rw

# /etc/systemd/system/rockedge-video.socket  (the socket through which video-pipeline talks to node-core)
[Socket]
ListenStream=/run/rockedge/video.sock
SocketUser=rockedge-core
SocketGroup=rockedge-video
SocketMode=0660

A separate socket is created for each peer (video.sock, vault.sock): filesystem permissions and the SO_PEERCRED check are two independent barriers. node-core receives the ready socket through socket activation (for example, the listenfd crate).

The watchdog catches not only crashes but also driver hangs, provided “alive” is reported only on real progress:

// Report "alive" to systemd only if the work loop is actually advancing
let progressed = frames_processed.load(Ordering::Relaxed) > last_seen;
if progressed {
    let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Watchdog]);
}

Interaction mechanics and failure containment

1. Bounded channels and a loss policy by data type. All channels are created with a fixed capacity, but what to do on overflow depends on what flows through the channel:
   • Frames — “the latest frame wins”: a fixed ring of N buffers inside video-pipeline; InferenceManager always takes the freshest one, and stale ones are overwritten, so RAM growth is zero. The channel analogue is tokio::sync::watch. tokio::sync::mpsc has no drop-oldest mode.
   • Critical events (PersonDetected, TamperAlarm) are never lost: every critical consumer (RuleEngine, NodeStore) receives the event through its own bounded mpsc via send().await; under overload backpressure kicks in instead of silent loss. Between processes, an event is confirmed with an Ack after being written.
   • Telemetry and statuses — broadcast: a lagging subscriber receives Lagged and skips stale messages, which is acceptable for metrics. broadcast must not be used for critical events: it is precisely what silently drops old messages for a slow subscriber.
2. Failure isolation. A crash of video-pipeline (for example, SIGSEGV in a decoder’s C library) does not affect node-core and vault: systemd sees the process exit, and Restart= brings it back up with a growing delay; before starting, the hardware context of the VPU/NPU can be reset via ExecStartPre=. Inside a process, lightweight actors remain tokio tasks, and their panics are caught through JoinHandle. This does not protect against segfaults and UB — which is exactly why code that works with untrusted input and C libraries is moved into separate processes:

// Lifecycle control of an isolated actor inside a process
// Important: catching a panic through JoinHandle works only with panic = "unwind" in the build profile.
const MIN_DELAY: Duration = Duration::from_millis(500);
const MAX_DELAY: Duration = Duration::from_secs(60);
const STABLE_RUN: Duration = Duration::from_secs(60);

/// Waits out the backoff; returns false if a shutdown arrived in the meantime.
async fn wait_backoff(delay: &mut Duration, shutdown_rx: &mut broadcast::Receiver<()>) -> bool {
    tokio::select! {
        _ = shutdown_rx.recv() => false,
        _ = tokio::time::sleep(*delay) => {
            *delay = (*delay * 2).min(MAX_DELAY);
            true
        }
    }
}

pub async fn supervise_manager_task<M: Manager + 'static>(
    manager: Arc<M>,
    mut shutdown_rx: broadcast::Receiver<()>,
) {
    let mut restart_delay = MIN_DELAY;

    loop {
        let mgr = manager.clone();
        let started_at = Instant::now();
        let mut handle = tokio::spawn(async move { mgr.start_processing_loop().await });

        tokio::select! {
            _ = shutdown_rx.recv() => {
                log::info!("Graceful stop of manager {}", manager.name());
                // Without abort() the task would keep living in the background after the supervisor exits
                handle.abort();
                let _ = handle.await;
                break;
            }
            res = &mut handle => {
                match res {
                    Ok(Ok(())) => {
                        log::info!("Manager {} finished normally", manager.name());
                        break;
                    }
                    Ok(Err(err)) => {
                        log::error!("Critical error in manager {}: {err:?}", manager.name());
                    }
                    Err(join_err) if join_err.is_panic() => {
                        log::error!("Panic inside the task of manager {}: {join_err:?}", manager.name());
                    }
                    Err(join_err) => {
                        log::warn!("Task of manager {} was cancelled: {join_err:?}", manager.name());
                    }
                }
            }
        }

        // A component that ran stably gets a "clean" backoff counter
        if started_at.elapsed() >= STABLE_RUN {
            restart_delay = MIN_DELAY;
        }
        log::info!("Supervisor restarting manager {}...", manager.name());
        if !wait_backoff(&mut restart_delay, &mut shutdown_rx).await { break; }
    }
}

#[derive(Serialize, Deserialize, Debug)]
enum Msg {
    Hello { proc_name: String, proto: u16 },
    Event { event_id: u64, kind: String, confidence: f32 },
    // The segment itself is passed as a descriptor, not as bytes
    SegmentFd { seq: u64, len: u32 },
    Ack { event_id: u64 },
}

// Framing: [u32 length][Postcard]; read EXACTLY len bytes
async fn recv_msg(s: &mut UnixStream) -> io::Result<Msg> {
    let mut len = [0u8; 4];
    s.read_exact(&mut len).await?;
    let len = u32::from_le_bytes(len) as usize;
    if len > MAX_MSG { return Err(io_err("message too large")); }
    let mut body = vec![0u8; len];
    s.read_exact(&mut body).await?;
    postcard::from_bytes(&body).map_err(io_err)
}

async fn handle_peer(
    mut s: UnixStream,
    store_tx: mpsc::Sender<(u64, oneshot::Sender<()>)>,
    expected_uid: u32,
) -> io::Result<()> {
    // Who connected? The uid and pid are reported by the kernel — they cannot be forged
    if s.peer_cred()?.uid() != expected_uid { return Err(io_err("foreign uid")); }

    loop {
        match recv_msg(&mut s).await? {
            Msg::Event { event_id, .. } => {
                let (ack_tx, ack_rx) = oneshot::channel();
                // send().await: under overload we wait (backpressure) rather than lose the event
                store_tx.send((event_id, ack_tx)).await.map_err(io_err)?;
                ack_rx.await.map_err(io_err)?;                    // the disk write has completed
                send_msg(&mut s, &Msg::Ack { event_id }).await?;  // Ack — only after the write
            }
            Msg::SegmentFd { seq, len } => {
                let fd = recv_fd(&s).await?;        // the descriptor arrives right after (SCM_RIGHTS)
                let seg = unsafe { File::from_raw_fd(fd) };
                let mut head = [0u8; 8];
                seg.read_at(&mut head, 0)?;         // read_at: the fd's offset is shared with the sender
            }
            _ => {}
        }
    }
}

4. Observability and a secure OTA pipeline

In the skeleton (v0): a ready-made OTA tool (RAUC / SWUpdate / Mender) with A/B + dm-verity + watchdog and a forced rollback test; push of logs and metrics through Vector with a bounded buffer; an outbound diagnostics tunnel on request.

Later: canary rollout, delta updates, enabling anti-rollback, aggregated histograms.

An Edge node must be transparent to monitoring, but monitoring itself must not become the cause of an outage.

A local Vector Daemon intercepts structured logs and metrics through a Unix Domain Socket. Telemetry is compressed with Zstd (for structured logs the typical compression ratio is on the order of 8–12:1, but it depends on the data and must be measured on a real stream) and sent in batches to secure cloud storage (Quickwit / S3) only when the connection is stable.

A simplified variant. If the fleet is small, Vector can be skipped and telemetry sent through the same MQTT broker. Three things should then be kept in mind: a separate MQTT connection (or at least chunks of up to ~16 KB), so that a large telemetry batch does not delay alerts in the same TCP channel; rate limiting and jitter when draining the accumulated buffer after connectivity returns; and sending only metrics, structured events and errors by default, with full logs only on request through the diagnostics tunnel. It is useful to hide sending behind a TelemetrySink trait so the transport can be swapped. In the baseline design we keep Vector: it takes care of the disk buffer, batching and retries.

The Observability layer obeys the same hard memory limits: the telemetry ring buffer on disk has a fixed quota (for example, 256 MB). On overflow, old logs are rotated, preventing degradation of the main video recording service.

Update delivery is the responsibility of a ready-made OTA tool (RAUC, SWUpdate or Mender). It is not worth writing your own installer: A/B slots, bootloader integration, signature verification and streaming installation are already well-tested there, and an extra home-grown privileged and network component is a new attack surface. OtaManager in node-core remains thin logic: it takes the image link and hash from the signed Desired State, decides when to install the update (a maintenance window, node health), launches the tool and confirms the slot based on the results of self-diagnosis.

1. A/B Dual-Boot disk layout: the root filesystem is divided into two isolated system slots (Partition A and Partition B).
2. Atomic write: an image with a cryptographic signature is downloaded and verified by the tool before writing, and then written to the inactive slot.
3. Hardware Watchdog and Auto-Rollback: the bootloader (U-Boot) sets the trial-boot flag boot_count=1 and hands control to the new system. If within 3 minutes after the start NodeSupervisor has not confirmed successful self-diagnosis and has not reset the Hardware Watchdog timer, a hard hardware reboot occurs with an automatic rollback to the previous verified firmware version.

Architecture in detail: the end-to-end Data Flow

In the skeleton (v0): a single end-to-end scenario: camera → event (stub detector) → event_queue → cloud → PUBACK, plus one hard-coded rule → GPIO relay without the cloud.

Later: real models, multiple streams, nested archive encryption with the Client KEK.

Let us trace step by step what happens in the system from the moment an object appears in the frame to the delivery of a notification to the customer’s mobile device:

1. Capture and Zero-Copy transfer: the camera streams video over RTSP. The video-pipeline process receives the packets and parses the stream; from there the path forks: the compressed packets H.264/H.265 are remuxed into 10-second MP4 segments without decoding and go to vault as opaque bytes (memfd over SCM_RIGHTS), while the frames decoded by the VPU stay inside the process and are handed to inference as a dmabuf handle without copying.
2. Inference on the NPU: the neural accelerator runs a quantized detection model (YOLO) and returns a vector of coordinates (Bounding Boxes).
3. Publishing to the bus: InferenceManager builds a PersonDetected { event_id, confidence: 0.94, bbox: [...] } event and sends it over IPC to the Internal Event Bus of the node-core process; a unique event_id is assigned at the source.
4. Parallel autonomous processing (the critical path goes over its own bounded channels and is never lost):
   • RuleEngine checks the event against the local rules database. Upon detecting a crossing of a protected zone outside working hours, it sends a signal to the physical relay (dry contact) of an alarm floodlight or siren. From the moment RuleEngine receives the event to the signal on the GPIO, less than 5 ms pass, fully autonomously from the cloud. The full latency from a person appearing in the frame is the sum of decoding, NPU inference (usually tens of milliseconds) and these 5 ms.
   • StorageManager (the vault process), on a “pin the clip” command from node-core, protects the current and neighboring 10-second segments from ring rotation. Each segment is encrypted with its own DEK, which is sealed in a nested fashion with the Node KEK (Secure Element) and Client KEK keys (the two-envelope scheme), and is written to the ring archive. vault does not parse the segment contents.
5. Guaranteed delivery (Store-and-Forward):
   • The event is serialized into a versioned Postcard format and atomically written to the event_queue table of the redb database.
   • The MqttCloudBus module reads the accumulated batch of events and sends it through the outbound mTLS tunnel to the EMQX cloud broker.
   • Records are deleted from event_queue only after confirmation by MQTT QoS 1 (PUBACK). If the connection drops during sending, the events are preserved intact until the channel is restored.
   • QoS 1 provides at-least-once semantics, so duplicates are possible: every event has a unique event_id, and the cloud deduplicates it. Moreover, PUBACK means “the broker accepted it”, not “the event has been processed”, so confirmation of processing in the cloud is built separately.

What is still missing: open questions of the target architecture

An honest list of places that the description above does not close, or closes only partially. For each item it is stated what we do in the skeleton and what later.

1. Signing and versioning of the Desired State

The problem. As described above, whoever gains control of the MQTT broker or the cloud backend controls the entire fleet. Closed ports and mTLS protect the channel, but not the document itself.

The solution.

• The Desired State is a signed document: a strictly monotonic generation, an expiry not_after (protection against replay of an old configuration), and a signature made with a key stored separately from the broker (KMS/HSM with audit) and inaccessible to it. The public key is pinned in the image and/or the Secure Element.
• The node keeps a last-known-good: if the new configuration was not applied or a manager did not reach Ready within the set time, it returns to the previous one.
• The node sends a status back — observed_generation and the state of each manager — so the cloud sees the divergence (drift) rather than blindly sending commands.

// Simplified concept: a signed configuration and the node's status
pub struct SignedDesiredState {
    pub generation: u64,      // strictly monotonic version number
    pub not_after: UnixTime,  // expiry: protection against replay of old configurations
    pub payload: NodeDesiredState,
    pub signature: Signature, // a key that the cloud broker cannot access
}

pub struct NodeStatus {
    pub observed_generation: u64,      // which version the node has actually applied
    pub last_known_good: u64,          // which version it is safe to roll back to
    pub managers: Vec<ManagerStatus>,  // Ready / Degraded / Failed + reason
}

Skeleton: signature, generation, status, last-known-good. Later: rotation of signing keys, approval of changes by two operators (two-person approval).

2. Anti-rollback versus automatic rollback

The conflict. Rollback protection (a monotonic version counter prevents installing an old vulnerable firmware) contradicts A/B rollback, which loads exactly the previous version.

The resolution. The counter is incremented only after commit — the successful self-diagnosis of the new firmware. The image carries not only a version but also a security epoch, which is raised only in releases with security fixes. The counter itself is stored where it cannot be rolled back: RPMB, a TPM NV counter, a Secure Element monotonic counter. eFuses are a poor fit: the number of bits is limited.

Skeleton: the design and the image metadata format. Later: enabling enforcement once the release process is in place.

3. DB schema migration and rollback

A firmware rollback brings back the old code, but not the old data. If the new version migrated the redb schema, the old version will not read the file.

• Expand/contract. The new version reads both the old and the new schema, but until commit writes only in a format the previous version understands. Irreversible migrations (contract) are performed in the next release, when rollback to the old version is no longer possible.
• A copy before migration. Before any migration, a copy of the database file (db.pre-migration) is created on the data partition; it is kept until the next release is confirmed.
• Explicit versions. The schema version is written to the database and to every record (a versioned Postcard enum); starting with an unknown version is an explicit error, not a silent read.
• A test. Power loss in the middle of a migration is a separate fault-injection scenario.

Skeleton: schema versions and a rollback-with-migration test. Later: the complex migrations themselves — as they appear.

4. Read-only rootfs (dm-verity) and data partitions

Secure Boot verifies the kernel, but without an immutable rootfs a substituted system (or a corrupted write into it) goes unnoticed.

The archive and the data are different partitions: an overflow of the video archive must not take space away from redb. This is a direct continuation of the Bounded by Design principle.

Skeleton: the entire layout, because it is irreversible.

5. Staged rollout and the cost of LTE traffic

• Rings. First the lab and internal devices, then 1%, 10% and 100% of the fleet. The transition between rings is gated by health-gates from telemetry (crash and reboot rate, frame p99.9, share of successful commits); on degradation the rollout is automatically paused.
• Traffic. An illustrative calculation: a 300 MB image × 1000 nodes = 300 GB. A delta update of 5–10% of the image size brings this down to 15–30 GB. Resumable download, bandwidth limiting, download windows (for example, at night) and a monthly traffic budget per node are needed.
• Signature. The image signature is verified before writing to the inactive slot, not after.

Skeleton: a full image, signature verification, resumable download. Later: rings and delta updates.

6. Diagnostics with closed ports

A node with no inbound connections still has to be serviceable, otherwise the very first field incident ends with an engineer’s site visit.

• An on-demand session. The operator creates a request in the cloud (with a ticket and an expiry); a diagnostics { until, scope, ticket } entry appears in the signed Desired State; the node itself opens an outbound mTLS tunnel to a bastion host. There is still no permanently listening port.
• Least privilege. By default, read-only access (logs, metrics, manager state). An interactive shell — only with a short-lived SSH CA certificate and after approval. All sessions are audited.
• A limited lifetime. The session is closed by a timer and does not survive a reboot.

Skeleton: a minimal read-only variant. Later: an interactive shell.

7. Power-loss tests: statistics and the bench

A single run proves nothing.

• The bench. Hardware-in-the-loop: a programmable power supply or relay that cuts power at random moments — during fsync, a redb commit, an OTA write to the inactive slot, boot, a schema migration. Besides instant cut-off, brownouts are needed: slow voltage decay and power “bounce”.
• Statistics. By the “rule of three”, a series of N cycles without a single failure lets us claim with 95% confidence only that the failure probability per cycle is below about 3/N: for 1000 cycles — below 0.3%, for 10,000 — below 0.03%. So thousands of cycles are needed per release candidate.
• Invariants after every cycle. The DB opens without manual action; no unconfirmed event is lost (duplicates are acceptable — they are removed by deduplication on event_id); the archive index is consistent with the files; the node booted into the correct slot.
• Splitting the pipeline. Software failures (iptables, tc netem, stress-ng) run on every commit; hardware ones run on a schedule (at night) and before a release: for ordinary CI they are too slow and expensive.

Skeleton: the bench and a series of cycles — this is the main proof of the skeleton.

Conclusion

The architecture described in this article is a convergence of time-tested fundamental engineering principles and modern systems practices. It is the target architecture of our Edge platform, and the Walking Skeleton is its first thin slice: exactly those decisions that cannot be painlessly redone after deployment to sites, plus one end-to-end scenario verified under chaos.

This architecture rests firmly on five invariants:

• We do not trust the environment (Zero Trust): closed inbound ports, hardware device identity (Secure Element / BootROM / TPM), end-to-end Secure Boot and cryptographic verification of the chain through the dm-verity root filesystem, process isolation of managers, and nested encryption of the video archive substantially reduce the risk of local tampering and limit the attack radius (Blast Radius Control), preventing software compromise.
• We accept the inevitability of failures (Resilience Engineering): declarative reconciliation, isolated processes and actors, the crash-safe redb store on a suitable industrial drive, and Store-and-Forward resilient to network outages with idempotent delivery. We build the system on the Crash-Only Software model, where the standard recovery mechanism is an ultra-fast, deterministic automatic restart.
• We respect the physical limits of the hardware (Mechanical Sympathy): strictly bounded queues and data ring buffers (Bounded Resources) eliminate uncontrolled heap allocations and the risk of the OOM-killer. Designing around flat data structures (SoA) rather than object graphs lets the processor use cache lines as efficiently as possible and keeps p99.9 latency within the strict frame budget.
• We design software for the physical topology of silicon (Hardware-Software Co-Design): we reject blind abstraction layers in pursuit of maximum FPS per watt. Algorithmic math is shaped around transistor-level capabilities through strict INT8 quantization for vector and matrix-acceleration instructions, while the data pipeline uses Zero-copy (VA-API, dmabuf, Remote Tensors / IOBinding), eliminating parasitic traffic over the shared Unified Memory bus.
• We measure system state rather than declare it (Observability): the nonlinear failure modes of the asynchronous Rust runtime (tokio) and inference engines (OpenVINO / ONNX Runtime) are monitored through end-to-end context (Trace Context). A resource-bounded local OTel agent and trigger-based contextual sampling make it possible to trace the causal chains behind anomalies in real time, without adding overhead to the video stream’s hot path.

By building such an end-to-end, fault-tolerant and hardware-efficient framework, we get a system with a measurable safety margin. On top of this foundation, any application functionality can be confidently built up — from face and license plate recognition to complex distributed tracking analytics — knowing that the system will maintain its integrity under the conditions of the real physical world.