The Architecture of Intelligent Video Systems in the Edge-First Era

In the context of heavily loaded video surveillance systems, I believe the greatest potential lies precisely in synergy: the performance and autonomy of Edge-First solutions combined with the global reach of the cloud (SaaS) and centralized fleet management.

AWS IoT Greengrass V2 is a striking market validation of that synergy. Amazon effectively split the system in two: the core local runtime for the edge is open (open source), while the cloud service for centralized control and fleet management stays commercial. The pioneer of cloud computing became one of the most prominent champions of the Edge-First approach.

But Greengrass is not the only example. Azure IoT Edge, Eclipse ioFog and EdgeX Foundry build edge-node autonomy on similar patterns: a local broker or message bus plus local storage for buffering events (the specific storage differs across projects; in a modern Rust stack, options like redb or fjall look very attractive).Home Assistant espouses a kindred local-first idea in consumer automation.

In effect, all of them implement fundamental autonomy patterns: Declarative State Reconciliation and Store-and-Forward. Their main job is to make the device independent of network anomalies: when connectivity drops, the node carefully records what is happening to local disk, and when the link is restored, it automatically syncs the accumulated context with the cloud.

For a clearer picture of these ideas, take a look at the block diagram of an on-edge video system:

Consider an illustrative scenario — loss of connectivity — which in a Cloud-Only architecture (if the cameras have no local recording of their own) leads to system degradation: archive recording stops and video analytics become unavailable. In an Edge-First architecture, the edge node remains autonomous: the video stream keeps being recorded to the local ring buffer archive, the local AI engine keeps running inference as usual, and alarm events are queued in a bounded local priority queue. Once connectivity is restored, the node automatically synchronizes the accumulated metadata and critical video clips with the cloud platform.

I believe this model will become the architectural standard for a significant share of heavily loaded distributed systems where autonomy (Store-and-Forward) and privacy (Privacy-first) matter. And it is in video surveillance, in my view, that this approach will unfold most fully.

I see Rust and Media over QUIC (MoQ) as the technological foundation. Inference itself on the NPU is performed by vendor runtimes (mostly C/C++), so Rust doesn't accelerate the neural network — it accelerates everything around it: the frame pipeline, storage, networking and state management. What matters here is memory safety, the absence of a garbage collector, predictable latency and ergonomic multithreading. Vendor C code, meanwhile, is best isolated in a separate process with minimal privileges — more on that in the article on edge node architecture.

Moreover, thanks to its strict type system and ownership model, Rust is well suited to introducing elements of formal verification (proof engineering) into small critical modules (state machines, stored-record parsing): Kani (bounded model checking that proves the absence of panics within specified bounds) and Verus (SMT-based verification of correctness) can confirm properties of such modules. They do not cover FFI, drivers or the entire asynchronous scaffolding, but for the critical nodes of an autonomous edge node they are a valuable additional layer of confidence.

MoQ, in turn, is a publish/subscribe protocol for media over QUIC and WebTransport. It was created to combine the low latency characteristic of WebRTC with relay-based scaling, familiar from CDNs and HLS. For Edge-First, it is a natural transport for live video: the edge publishes, the relay scales, viewers subscribe.

Why not WebRTC, not SRT, not plain RTSP? WebRTC delivers excellent latency and universal browser support, but it is a complex stack (ICE, STUN/TURN, signaling) and primarily a tool for interactive calls; it scales via an SFU, which is a separate piece of infrastructure. SRT is great for point-to-point delivery (contribution), but it is not designed for distribution to viewers. RTSP is the de facto standard for pulling video off cameras and a legacy of the LAN era: it traverses NAT poorly, and encryption (RTSPS/SRTP) is often unsupported by cameras or not enabled. So today RTSP is a capture (ingest) protocol, not a delivery protocol for end users.

Built on top of QUIC, MoQ solves these problems architecturally:stream multiplexing within a single connection without head-of-line blocking, fast connection resumption (0-RTT, which requires care because of the risk of replaying data), and — critically for a hybrid Edge-First + Cloud architecture — a relay-native approach, where the cloud relay forwards MoQ objects without transcoding the video.

A separate caveat about latency. The network component of sub-100 ms within a region where the edge/relay is present is a realistic target. But end-to-end glass-to-glass latency also includes the camera's GOP, the codec, the decoder and the jitter buffer; industry reference points for MoQ are fractions of a second, typically 0.3–1 s. So it should not be declared — it must be measured separately for every configuration.

The second caveat is maturity. As of September 2026, the MoQ Transport specification remains an active IETF draft (draft-ietf-moq-transport-21) and is still changing; among its editors are engineers from Cisco, Google and Meta, and the first relay networks (Cloudflare's, for instance) are already running in production. The community does not expect a final RFC before 2027. So in the architecture, the transport should sit behind an abstraction, the draft version should be pinned, and reliable event delivery should not depend on MoQ: it is responsible for live video. On top of that, UDP/QUIC is blocked in some corporate networks, so a TCP fallback channel (WebSocket/HTTPS, for example) is needed.

Robotics Patterns for Solving Edge Autonomy Problems

While studying the challenges of Edge-first and Offline-first, I realized that a huge number of proven solutions for Edge video systems can be borrowed from autonomous robotics. In essence, a field edge node at a site is the same kind of robot, only less complex. It has to operate under unpredictable conditions: unstable 4G links, anomalies in IP camera firmware, power failures, tight memory and CPU constraints, and the need for safe remote updates (OTA).

In robotics, the fight against these risks was formalized long ago. Instead of fragile logic, the field relies on deterministic control loops and declarative state reconciliation (Control Loops & State Reconciliation), hardware abstraction (HAL), Pub/Sub buses with quality-of-service guarantees (DDS / QoS) and self-diagnostics mechanisms (watchdog). This is a crucial insight: applying it makes it possible to build reliable and autonomous edge video systems that can run fast, stably and predictably for long periods at remote sites.

A detailed breakdown of how robotics architectural patterns can be adapted for edge systems will be published in a separate article on this site. The practical side — the security model, state storage, updates and the process model of the edge node — is covered in the article on edge node architecture.

Local Intelligence: AI as an Architectural Necessity

A question often comes up: "How is an edge node different from a standard NVR?" It is not that NVRs lack detection — modern recorders have it. The difference lies in the role of the node. A classic NVR is first and foremost a storage device: cyclic recording of a video stream to disk and playback on demand. The edge node is conceived as a managed compute platform: real-time local AI inference, software-based false alarm filtering, video streaming over MoQ, context synchronization with the cloud ecosystem, and full-fledged updates and remote fleet management. An NVR is a local archive. An edge node is a peripheral decision-making center.

But if the edge node is a decision-making center, a natural question follows: who exactly makes those decisions, and on what basis? A traditional NVR simply registers the fact of motion and writes a buffer to disk. The next-generation edge node must do something fundamentally different — turn a continuous, semantically "empty" video stream into a structured stream of events, each of which has weight, context and priority. And this is where we cannot do without local AI inference, without which the very idea of Edge-First loses half its meaning.

Let's do the math: one camera generates 86,400 seconds of video per day. Of those, by typical estimates (the share can vary widely at a specific site), 90–95% is a static background, swaying trees, changing light. In a Cloud-Only model you pay to transmit and analyze all those seconds in the cloud, burning GPU resources on processing emptiness. In the Edge-First paradigm, a local AI engine — for example, YOLO26 via ONNX Runtime (or OpenVINO), running on the NPU/iGPU of your mini-PC — acts as the first-level intelligent filter. In real time it separates signal from noise: detecting objects, tracking trajectories, classifying intrusion zones. The ring buffer archive stores video segments within an allotted quota, while an event index with metadata, bounding boxes and confidence scores is maintained alongside it: it enables fast search, and the retention policy can keep segments around events for longer. The cloud receives not terabytes of "silence" but compact events, and clips on request or per policy — and that changes the unit economics of the entire system.

For a long time, the split inference pattern was considered a necessary architectural compromise — where only initial detection ran at the edge and the "heavy" follow-up analysis was delegated to the data center. With the arrival of a new generation of models (such as YOLO26, released in January 2026), optimized for NPUs and edge accelerators, that scheme is ceasing to be mandatory for detection and tracking tasks and often only complicates the system. Cascades ("a light model at the edge, a heavy one in the cloud for rare events") remain a reasonable solution for weak hardware and especially complex tasks, but I no longer consider them an architectural necessity. Interestingly, the YOLO26 release was announced under the slogan Ultralytics YOLO26: The new standard for edge-first vision AI.

On modern NPUs, an edge node is capable of performing a significant part of the visual analysis cycle on-edge: detection, tracking, segmentation and LPR (license plate recognition). Per-frame detection fits into tens of milliseconds; heavier stages (segmentation, LPR, behavioral models) require a separate latency budget that must be measured on the target hardware rather than taken on faith. The computational load closes inside the local Data Plane loop. What goes to the cloud is not raw frames or intermediate tensors, but finished semantic events and aggregated metadata. The cloud Control Plane is freed from video inference and focuses on global coordination, fleet management and long-term cross-site analytics.

Declining to send raw video to the cloud by default makes the principle of privacy by default (Privacy by Architecture) a practical reality rather than a declaration. In the context of Russian Federal Law 152-FZ (GDPR in Europe), the classic model of transmitting video streams containing faces and license plates to third-party data centers carries serious regulatory risks. In an Edge-First architecture, analysis and de-identification (face blurring, silhouette extraction, license plate anonymization) can be performed on site, before raw video leaves the physical perimeter of the site.

But it is important not to overstate this effect. Local processing of biometric data is still processing of personal data, and the operator's responsibility does not disappear; localization requirements and retention periods apply just the same. Events, metadata and clips do still leave the site (including the live stream via the MoQ relay), and they must be treated as personal data. If the relay must not see the content, video needs to be encrypted end-to-end at the object level, since transport encryption in MoQ only operates between adjacent participants in the chain. I cover these matters at the level of the threat model (Zero Trust, keys in a Secure Element, archive encryption) in the article on edge node architecture.

Where This Might Be Heading

Looking a little beyond the horizon, the next qualitative leap, as I see it, will come at the moment of transition from detection to understanding. Today, edge AI draws bounding boxes and attaches labels to them: "person," "car," "dog." Open-vocabulary models (YOLOE-26, for example) already allow objects to be found by text description ("a person in a yellow jacket") without fine-tuning. The next step is compact VLM models (of the SmolVLM, Florence-2 or small Qwen-VL class), which will be able to describe a scene in connected prose: "a courier in a yellow jacket left a package at the service entrance and walked toward the parking lot." This is a direction rather than a ready-made solution: the quality and cost of VLMs on NPUs are still to be measured. Combined with local generation of vector embeddings (CLIP, SigLIP) and their synchronization through the ordinary event channel into a cloud vector store, this opens up the possibility of semantic search across the video archive (with the caveat that face embeddings, too, may count as personal data). The operator will no longer spend hours scrubbing the timeline in search of an incident — they will ask the system in natural language: "show me every moment last week when someone approached the perimeter after midnight," and get an answer within seconds. At that point, VSaaS may cease to be "an archive with remote access" and become a situation-awareness system — and the key part of that work logically belongs at the edge rather than in the cloud, because only the local node has direct access to the full video stream with low latency and without bandwidth constraints.

Finally, there is one more argument for laying the AI foundation right now. And it is tied to a direction of evolution that is already discernible enough to be factored into architectural decisions today.

Today's typical AI in VSaaS is a model that answers the question "what is happening in the frame?" and returns a bounding box with a confidence score. That is a model as a tool: a detector, a classifier, a counter. The next logical step is systems that do not merely register an event but track its development over time and respond: an unauthorized perimeter intrusion detected — the PTZ camera swings onto the intruder, the siren sounds, an incident report is generated and escalated to the on-duty operator with a priority. This is no longer a single inference but a closed-loop decision-making cycle — still under human supervision and with the simplest logic, for now, but architecturally it is a fundamentally different pattern. And if your Data Plane is designed as a one-way pipeline of "frame → model → metadata → storage," retrofitting such a feedback loop a couple of years from now may prove difficult. By understanding the direction technology is moving, we can build in an extensible event bus, a pluggable enrichment pipeline architecture and an abstraction over actuators (execution mechanisms such as PTZ drives, turnstiles, I/O relays, sirens and so on) from the start — then closed decision-making loops will slot in natively, as just another consumer in an event stream that already exists.

Caution is essential here: actuators carry a cost of error. That is why, in the model, the decision to act is made by a deterministic RuleEngine with verifiable behavior, while models (including LLM reasoning) only propose or enrich context. Dangerous actions are confirmed by a human, and the false-positive rate is measured and bounded in advance.

That is precisely why I am convinced that the Edge-First video system you are designing today must be ready not only for AI inference but also for closed loops of autonomous response at the edge(in the spirit of the MAPE-K loop — Monitor, Analyze, Plan, Execute over a shared knowledge base, proposed by IBM in the early 2000s as part of autonomic computing; there are also extensions of it with LLM-based reasoning, the MAPER approach) — even if the first version of your product simply writes an archive and sends alerts. Architectural debt in distributed systems is paid with interest, and the earlier you put the right abstractions in place, the cheaper the evolution will be.