Neural architecture papers report inference TOPS and model size. They do not report what a 10 W inference engine actually consumes when you add the sensor interface, the preprocessing pipeline, the DDS middleware, and the OS overhead that coexist on the same SoC in a real deployed system. After fielding inference stacks on embedded ISR platforms across varying airframe classes, we have a different picture of where the power budget goes than what the datasheets suggest. This is that picture.
The inference engine is not where most of the power goes
The counterintuitive starting point: on a properly implemented embedded ISR pipeline, the neural inference engine itself is often not the dominant power consumer. On a Jetson Orin NX in a realistic two-modality pipeline (EO + LWIR), our measured breakdowns look roughly like this under sustained operation:
- SoC core cluster (ARM A78AE at 1.4 GHz, 4 cores utilized): 1.8–2.4 W
- DLA cores running two concurrent INT8 models: 2.1–2.9 W
- LPDDR5 memory subsystem (inference activations + frame buffers): 1.2–1.8 W
- Camera and IR sensor interfaces (MIPI CSI-2 + USB3): 0.8–1.1 W
- PCIe and storage I/O: 0.3–0.5 W
- Thermal management fan draw: 0.4–0.7 W (depends on ambient)
Total: 6.6–9.4 W sustained, with inference accounting for roughly one-third of that. The memory subsystem is often as expensive as the compute itself, because neural inference has high memory bandwidth demand: a single YOLOv8s forward pass on a 640×480 frame reads and writes activation tensors that can approach 50 MB of memory traffic. At LPDDR5 speeds, that translates to measurable power in every frame cycle.
Memory bandwidth is the hidden power budget drain
The relationship between memory bandwidth and power is one that most embedded AI deployment guides understate. DRAM access energy is roughly 10–50 pJ per bit depending on the memory technology, which means a model that moves 100 MB of activations per inference at 20 fps is burning on the order of 20–100 mW in memory access alone per inference — multiplied by inference frequency, this is a non-trivial contributor to the system power budget.
The architectural mitigation is to minimize activation memory footprint. Techniques that reduce the intermediate tensor sizes — depth-wise separable convolutions, width multipliers, aggressive bottleneck layers — reduce both the FLOPs count and the memory bandwidth demand. For embedded deployment, a model that achieves a target detection accuracy with 30% fewer parameters is better than a model that achieves the same accuracy with 30% fewer FLOPs, if the parameter reduction corresponds to reduced activation memory traffic.
KV-cache footprint matters for any architecture using attention mechanisms. A detection model with a small transformer neck that requires a KV-cache of, say, 12 MB per frame — modest by server standards — means your inference context is permanently occupying 12 MB of on-chip SRAM or off-chip DRAM. On a platform with 2 GB total LPDDR and a frame buffer, OS, and ROS 2 overhead competing for that space, 12 MB is not trivial. For embedded deployment, attention-based models should be evaluated for KV-cache footprint explicitly, not just parameter count and inference latency.
Thermal design: the derating problem nobody tests until integration
Most embedded AI hardware is characterized at 25°C ambient. Most ISR platforms operate at significantly higher internal enclosure temperatures. A UGV operating in a desert environment can see enclosure ambient temperatures of 55–65°C. A UAS payload bay has limited airflow and absorbs heat from the avionics bay.
Thermal derating means that the performance available from a chip at elevated temperature is substantially below its datasheet values. The Jetson Orin NX, for example, will throttle its DVFS state to reduce junction temperature when the thermal sensor approaches the TJmax threshold (typically 95°C for the SoC junction). In an enclosure where ambient is 55°C and the module is thermally coupled to the chassis, the thermal headroom before throttling is far smaller than in the lab environment where the module is characterized.
We have measured 15–25% inference latency increase on Orin NX at 50°C enclosure ambient versus 25°C lab ambient, due to DVFS throttling reducing the GPU clock from its maximum state. A system that barely meets its 40 ms latency budget in lab conditions may be running at 50–55 ms under mission thermal conditions. If the latency budget was defined with no thermal margin, that translates to a field performance gap.
The correct approach is to characterize inference performance at the worst-case mission thermal state, not at room temperature, and to define thermal margin in the system design. A 30% thermal margin on latency budget — designing to 28 ms for a 40 ms requirement — provides realistic field compliance. An enclosure thermal model that accounts for solar loading, airframe heat contribution, and ventilation (or lack thereof) should be part of the system design, not a post-integration surprise.
MIL-STD-810 vibration and its effect on inference reliability
MIL-STD-810 Method 514 (Vibration) specifies vibration profiles for different vehicle and aircraft platforms. For embedded ISR compute, the concern is not primarily component failure from vibration — modern BGA-packaged SoCs and LPDDR modules handle vibration well structurally — but rather bit-error-induced inference anomalies from DRAM upset under vibration.
LPDDR5 modules on commercial carrier boards are not universally ECC-protected. A vibration-induced bit flip in an activation tensor mid-inference can produce a detection output that is confidently wrong rather than low-confidence uncertain: the quantized INT8 value in a critical feature map is corrupted to a different class activation, and the model's output is a high-confidence false positive or false negative that the fusion layer has no basis to reject because it looks formally valid. This failure mode does not appear in laboratory testing because vibration qualification is typically done without active inference workloads running.
For programs where vibration environment is significant, using a Jetson module on a carrier board that supports ECC DRAM, or alternatively using FPGA-based inference where all state is on on-chip SRAM with configurable protection, materially reduces exposure to this failure mode. We are not saying DRAM bit flips are the dominant reliability risk in embedded inference — they are not — but for a certifiable defense system operating in a high-vibration environment, the question of ECC protection should be answered explicitly rather than assumed away.
Quantization-aware training from a power perspective
The power case for INT8 over FP16 inference is straightforward: INT8 multiply-accumulate operations require roughly one-quarter the silicon area and energy of FP16 equivalents. On a platform where compute is power-constrained, running the same model at INT8 rather than FP16 delivers approximately 2× the throughput for the same energy, or equivalently, the same throughput at roughly half the power.
The nuance is that not all models quantize equally well. For ISR detection tasks with small targets (occupying under 1% of frame area), the precision loss from post-training quantization can be severe: the low-magnitude activations that encode small-object features are most sensitive to the coarse quantization grid, and the resulting model may have precision loss that exceeds the mission's probability-of-detection requirement.
Quantization-aware training (QAT) addresses this by incorporating simulated quantization noise into the training process, allowing the model's weights to adapt to the precision reduction. The additional training cost — typically 10–30% of the original training compute — is recoverable in deployment power savings within a few thousand inference hours. For a persistent ISR payload running 8 hours per flight day, that payback period is measured in weeks, not years.
The practical recommendation: if your program's inference budget requires INT8 for power compliance, plan for QAT from the start of model development rather than applying post-training quantization as a final step. The accuracy delta on small-object ISR tasks between QAT and PTQ is real and large enough to affect operational performance metrics.
Conformal coating and long-field reliability
One power-related failure mode that does not appear in any neural architecture paper is increased leakage current from moisture ingress in un-coated circuit boards. An embedded inference board operating in high-humidity environments — coastal, tropical, or in condensation-prone thermal cycling — that lacks conformal coating will accumulate surface contamination that increases board-level leakage and can cause intermittent short circuits between closely spaced BGA pads. The signature in the field is a system that shows elevated idle power consumption over time and eventually produces intermittent resets that are extremely difficult to diagnose without physical inspection.
Conformal coating (acrylic, silicone, or urethane per MIL-I-46058C or IPC-CC-830) adds minimal weight and cost to a carrier board. For programs specifying embedded inference hardware for field deployment, conformal coating should be a standard requirement, not an option. The failure mode it prevents is insidious precisely because it progresses slowly and produces symptoms that look like software instability before they manifest as hardware failure.
The 5–15 W power envelope that characterizes most embedded ISR compute platforms is achievable, but only when the system is designed holistically — accounting for memory bandwidth, thermal derating, sensor interface overhead, and long-field reliability — rather than treating the inference engine's published TOPS number as the primary design variable.