Edge AI has changed—and with it, the definition of what a “complete” system looks like.
Not long ago, an industrial vision deployment meant a detection model paired with a rules engine. That was enough to identify defects or track objects. Today, that baseline is no longer competitive. Customers increasingly expect edge systems to not just detect events, but to understand and verify them—to explain what happened, reduce false alarms, and support human decision-making on the floor.
That shift introduces a new class of workload: agentic AI at the edge. And agentic workloads come with a very real constraint—memory.
At the same time, memory is becoming more expensive. As DRAM costs rise across the industry, the difference between deploying on a high-memory module versus a leaner SKU is no longer just a technical choice—it’s a business decision that directly impacts margins, scalability, and rollout speed.
This is the context in which NVIDIA’s JetPack 7.2 innovations around agentic-ready AI at the edge with memory efficiency become important. The direction is clear: extract more capability from the same hardware by making every layer of the stack more efficient.
At Neurealm, we wanted to test what that actually looks like in a real production scenario.
Could a full industrial inspection system—three camera streams, a DeepStream perception pipeline, a deterministic SOP engine, and a 32B-parameter reasoning VLM—run end-to-end on a single Jetson module within a 32 GB memory budget?
On paper, it shouldn’t. A model like Cosmos-Reason2-32B, at native precision, requires roughly 64 GB just for weights. That is before accounting for video ingestion, inference pipelines, or system overhead. Traditionally, this is the kind of workload that belongs on server-grade GPUs. But when approached as a system design problem rather than a model deployment problem, the answer changes. It fits.
What makes this significant is not just technical feasibility—it is what it unlocks commercially. If a workload of this complexity can run within 32 GB, it changes the economics of edge AI deployments. Systems that would otherwise require expensive, high-memory modules can now be deployed on more cost-efficient hardware. That reduces upfront infrastructure cost, lowers power consumption, and makes scaling across multiple sites far more viable.
In practical terms, it means teams can deliver more intelligent systems without increasing hardware budgets.
The use case we worked with is a production SOP inspection system for an automotive components manufacturer. A three-camera setup monitors a leak-testing station where operators follow a strict sequence—handling pipes, submerging them, validating test results, and routing them accordingly. The complexity that computer vision deals with lies in interpretation, not detection.
A violation is not defined by a single frame. It depends on context: whether the pipe was tested, what the signal indicated, and who performed the action. The system must continuously maintain state across multiple views and time sequences, translating raw video into operational understanding.
At the same time, accuracy is critical. False alarms disrupt operations and erode trust in the system. To address this, every detected violation is passed through a second layer—a reasoning VLM that analyzes a short clip and verifies whether the action actually occurred.
This is where the system moves from automation to intelligence. It is also where memory pressure peaks. Making this architecture work within a constrained memory budget required a shift in how the system is designed.
NVIDIA Jetson Memory Optimization
The three tiers and what flows between them: only metadata crosses tier boundaries until an incident is suspected — then a short, annotated clip wakes the verifier.
The architecture is designed around a simple principle: Use memory only where it creates value.
The perception pipeline and SOP state machine run continuously, sized for worst-case streaming across three cameras. The VLM—by far the largest component—is event-driven. Its quantized weights stay resident, but it only activates when the system flags a candidate incident, analyzing a short, annotated clip of ~16 sampled frames. This keeps its KV cache tightly bounded to a few thousand tokens instead of a full video context—saving gigabytes of memory.
The data flow is intentionally narrow and one-way: DeepStream perception feeds SOP logic, which surfaces candidate violations, which are then verified by the VLM before triggering an operator alert. Most video never leaves the metadata stage, ensuring that compute and memory are reserved for only the highest-value decisions.
We chose NVIDIA Cosmos-Reason2-32B deliberately. Verifying physical actions—like distinguishing between “handling a pipe near a bin” and “releasing it into the bin”—requires strong spatial and temporal reasoning. Smaller VLMs struggled with this distinction, leading to false positives. The 32B model provides the action-level understanding needed for production reliability. Built on the Qwen3-VL architecture, it also integrates seamlessly with our existing vLLM serving stack and can be fine-tuned on domain-specific SOP data.
Keeping a large reasoning model resident unlocks additional value. The same VLM that verifies incidents also serves as an on-device video Q&A and summarization engine. Supervisors can query events in natural language—“What happened at station 2?”—without adding any new models or memory overhead. In a constrained environment, one model serving multiple roles is as critical as any optimization technique.
Memory Engineering, Layer by Layer
To fit a 32B system within a 32 GB budget, we treated memory as a first-class constraint and optimized across every layer of the stack. No single change made it possible—we made it work by stacking gains end to end.
- BSP & Boot: Reclaim Memory Early
We disabled unused hardware allocations and tuned kernel buffers to reclaim memory before the OS even loaded—recovering ~100 MB upfront.
Impact: We increased usable memory without touching the AI stack.
- OS & JetPack: Strip It Down
We removed the desktop stack and ran a headless, inference-only system, freeing up to ~865 MB while eliminating unnecessary background processes.
Impact: We redirected system overhead into model capacity.
- Inference Pipeline: Stay Lean
We used DeepStream’s zero-copy pipeline, removed visualization in production, deployed lightweight TensorRT models per camera, and passed metadata instead of video.
Impact: We scaled multi-camera processing without increasing memory load.
- Inference Framework: Cap the KV Cache
We capped the VLM’s context window (256K → 32K) to prevent excessive KV cache allocation.
Impact: We reclaimed multiple gigabytes—our single biggest memory win.
- Model Optimization: Quantize Aggressively
We quantized the 32B model from ~64 GB to ~17–18 GB and used smart frame sampling (~16 frames per incident) to reduce input load.
Impact: We made a server-grade model viable on edge hardware.
The budget: where 32 GB goes
Putting the layers together, the full stack fits with room to spare:
| Component | Working set |
The result is a full-stack deployment—multi-camera perception, stateful SOP validation, and a 32B reasoning model—running within a 32 GB memory budget, with headroom for real-world variability.
This is not just about fitting a model onto a device. It is about changing how edge AI systems are architected.
Instead of scaling hardware to meet software demands, teams can design systems that operate efficiently within constraints. That shift has direct implications for cost, scalability, and speed of deployment.
Agent Skills: From Weeks of Work to Days
Until recently, optimizing the BSP and OS layers was slow and specialized work. The gains were well understood—but executing them required deep expertise in device trees, kernel tuning, and repeated flash-and-validate cycles.
That changes with JetPack 7.2. With NVIDIA’s agent skills—jetson-device-skills and jetson-bsp-skills—these workflows are now structured, automated, and repeatable. What used to depend on specialist platform engineers can now be executed through guided, verifiable steps, even integrated with AI coding assistants like Claude Code.
In our workflow, these skills act as force multipliers. We use them to snapshot device health, measure memory before and after every optimization, and systematically apply BSP and OS-level changes with built-in validation at each step. The result is a fundamental shift in velocity and scalability. What previously took weeks of platform engineering effort can now be done in days—and more importantly, repeated reliably across multiple devices and deployments.
There is also a broader industry signal here. As edge AI evolves from detection to reasoning, the primary bottleneck is no longer compute—it is how efficiently systems use memory across the stack. The teams that can optimize at this level will be able to deploy more advanced capabilities without incurring prohibitive costs.
That is the direction we are seeing across industrial, robotics, and enterprise AI applications—and it is where we are focusing our work at Neurealm. We work with teams to move from prototype to production by optimizing across every layer—from BSP and OS to inference frameworks and model deployment—ensuring that systems are not just functional, but commercially viable.
If you are evaluating how to deploy more advanced AI workloads on Jetson, or trying to balance capability with cost constraints, this is the problem that defines success. Because in edge AI today, the question is no longer what can you run?
It is: What can you run efficiently enough to scale?
Neurealm is an NVIDIA services partner specializing in exactly this: full-stack memory and performance optimization on Jetson, from BSP carveouts to model quantization, using NVIDIA’s JetPack 7.2 tooling and agent skills. A working demonstration of the SOP-inspection system described here is available, and we help customers move existing workloads down a SKU — or fit new agentic workloads onto the SKU they already have.
Talk to us about running more AI on the same hardware—through full-stack optimization on Jetson.