cs.ROMar 16, 2026
SaveEmbodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies
Organizations: University of South Florida, USA · Florida International University, USA · Johns Hopkins University, USA · Nottingham Trent University, United Kingdom ·
Abstract
Deploying foundation models in embodied edge systems is fundamentally a systems problem, not just a problem of model compression. Real-time control must operate within strict size, weight, and power constraints, where memory traffic, compute latency, timing variability, and safety margins interact directly. The Deployment Gauntlet organizes these constraints into eight coupled barriers that determine whether embodied foundation models can run reliably in practice. Across representative edge workloads, autoregressive Vision-Language-Action policies are constrained primarily by memory bandwidth, whereas diffusion-based controllers are limited more by compute latency and sustained execution cost. Reliable deployment therefore depends on system-level co-design across memory, scheduling, communication, and model architecture, including decompositions that separate fast control from slower semantic reasoning.
Figures & tables
Figure 1. The Deployment Gauntlet. A unified view of the eight major system barriers that limit the deployment of foundation models from the cloud to edge embodied AI platforms.
| Workload Class | Representative Models | Primary Role / Modality | Dominant Bottleneck |
| Vision-Language-Action (VLA) Policies | RT-2 ( Zitkovich et al., 2023 ) , OpenVLA ( Kim et al., 2024 ) , NanoVLA ( Chen et al., 2025a ) | Policy generation; RGB + language to action | Memory traffic and weight streaming during autoregressive decoding, which compete directly with sensor DMA and shared-memory service. |
| Diffusion-Based Policies | Octo ( Team et al., 2024 ) , OneDP ( Wang et al., 2024b ) , LightDP ( Wu et al., 2025 ) | Trajectory generation; RGB / multimodal to action sequence | Sustained compute and thermal load from iterative denoising, which imposes a latency floor and high energy cost. |
| Vision Encoders and Multimodal LMMs | LLaVA-Mini ( Zhang et al., 2025c ) , MiniCPM-V ( Yao et al., 2025 ) | Perception and reasoning; RGB + language | Visual prefill and bursty frame ingestion, which create transient compute and memory spikes during high-resolution encoding. |
| 3D / LiDAR Encoders | PointPillars ( Lang et al., 2019 ) , DensePillarNet ( Chandorkar et al., 2025 ) , BEVFusion ( Liu et al., 2022b ) | Geometric perception; LiDAR / LiDAR + RGB | Sparsity and irregular memory access, which reduce accelerator utilization and increase synchronization overhead. |
| Multimodal Fusion Stacks | MMEdge ( Huang et al., 2025 ) , BEVFusion ( Liu et al., 2022b ) | Cross-modal state construction; RGB / LiDAR / audio / text | Scheduling and synchronization pressure from shared middleware, memory bandwidth, and callback timing. |
Table 1. Edge-relevant workload classes for embodied foundation model deployment.
| Workload | Dominant Bottleneck | Characteristic Edge Failure Mode |
| Vision-Language-Action (VLA) Policies | Memory traffic and weight streaming | Autoregressive action decoding repeatedly streams large parameter tensors, competing with sensor DMA traffic over shared memory and introducing latency spikes that can violate closed-loop control deadlines ( Kim et al., 2024 ; Prashanthi S et al., 2025 ; Chen et al., 2025a ) . |
| Diffusion-Based Policies | Sustained compute and thermal load | Iterative denoising requires repeated dense matrix execution over the same control horizon, creating a latency floor, high energy draw, and thermal throttling that reduce practical control frequency on edge platforms ( Chi et al., 2025 ; Wang et al., 2024b ; Wu et al., 2025 ) . |
| Vision Encoders and Multimodal LMMs | Visual prefill and bursty ingestion | High-resolution visual streams require repeated encoder prefill, producing burst-driven compute and memory demand that can dominate the thermal envelope even when downstream reasoning remains lightweight ( Radford et al., 2023 ; Zhang et al., 2025c ; Yao et al., 2025 ; Saha and Xu, 2025 ) . |
| 3D & LiDAR Encoders | Sparsity and irregular memory access | Point-cloud pipelines rely on voxelization, sparse indexing, and data-dependent aggregation, which map poorly to dense accelerators, reduce utilization, and introduce synchronization overhead in multimodal stacks ( Choy et al., 2019 ; Lang et al., 2019 ; Tang et al., 2020 ; Liu et al., 2022b ) . |
| Multimodal Fusion Stacks | Scheduling and synchronization | Even when unimodal components meet latency targets in isolation, their composition through shared middleware, LPDDR bandwidth, and runtime scheduling can produce temporal misalignment and end-to-end timing failures ( Feng et al., 2020 ; Kronauer et al., 2021 ; Huang et al., 2025 ; Bechtel and Yun, 2024 ) . |
Table 2. Representative edge-relevant workload classes and their dominant execution bottlenecks in embodied foundation model deployment.
Figure 2. The taxonomy for the gauntlets and solutions in Foundation Models under Embodied Constraints.
| Barrier | Representative Subproblem | Systems Effect |
| Sensor Fusion Tax | Temporal and Spatial Misalignment | Asynchronous sensor clocks, transport delay, and calibration drift force buffering, synchronization, and corrective alignment, trading control responsiveness for temporal and spatial consistency ( Li et al., 2022 ; Jellum et al., 2022 ; Lv et al., 2021 ; Cheng et al., 2025 ) . |
| Middleware and Pipeline Overheads | Serialization, copying, buffering, and sweep-level pipeline blocking consume bandwidth and introduce latency jitter that destabilizes high-frequency control execution even when average throughput is acceptable ( Kronauer et al., 2021 ; Teper et al., 2024 ; Huang et al., 2025 ) . | |
| Representational Mismatch and Foundation-Model Sensitivity | Modalities such as event streams require costly reconstruction or voxelization before they can be processed by frame-based models, while foundation models further amplify fusion overhead through higher memory traffic and greater sensitivity to timing error ( Gallego et al., 2020 ; Gehrig et al., 2020 ; Bommasani et al., 2021 ; Tang et al., 2025 ; Ruan et al., 2025 ) . | |
| Heterogeneous Compute Mismatch | Operator Coverage Gaps and Execution Bubbles | Limited operator support on edge accelerators forces graph partitioning, CPU fallback, and host–device synchronization, creating execution bubbles and large latency spikes that reduce end-to-end throughput ( He et al., 2022 ; Chen et al., 2025a ) . |
| Granularity Mismatch and Kernel Launch Overhead | Fine-grained transformer kernels incur substantial launch overhead on embedded GPUs, accounting for 30–60% of latency and driving utilization below 20% during autoregressive decoding ( Zeng et al., 2024 ) . | |
| The Memory Wall under Unified-Memory Architectures | Shared LPDDR bandwidth contention among CPUs, GPUs, NPUs, and sensors makes execution memory-bound and can offset nominal hardware-acceleration gains through arbitration and tensor movement overhead ( Ivanov et al., 2021 ; Kim et al., 2024 ) . |
Table 3. Deployment Gauntlet barriers 1–4: recurring system-level challenges for embodied foundation models at the edge.
| Barrier | Representative Subproblem | Systems Effect |
| Long-Horizon Execution Drift and Adaptation Failure | The Stationarity Fallacy | Embodied deployment violates the stationarity assumptions of offline training: sensor noise, actuator behavior, and calibration drift shift over time, causing prediction error to accumulate as observations move away from well-supported regions of the training distribution ( Kumar et al., 2021 ) . |
| Latent-State Misalignment | Physical drift can disrupt the alignment between sensory embeddings and action-relevant latent structure, causing internally coherent predictions to diverge from the true physical state of the platform ( Ha and Schmidhuber, 2018 ) . | |
| The Adaptation Gap | Edge platforms often cannot support timely online adaptation under SWaP constraints, while cloud-mediated adaptation introduces latency and connectivity dependence, leaving long-duration deployments governed by increasingly outdated model assumptions ( Ren et al., 2025 ; Gao et al., 2025 ) . | |
| Residual Hardware and Environment-Coupled Drift | Sensor bias instability, thermal noise, mechanical wear, and terrain-dependent slip introduce residual time-varying error that classical compensation can attenuate but not fully remove ( Zhou et al., 2025 ; Vargas et al., 2021 ; Sigron et al., 2023 ; Sakayori and Ishigami, 2024 ) . | |
| Safety & Verification Gap | Stochastic Objectives and Constraint Violations | Likelihood-based objectives such as token prediction and diffusion denoising do not inherently encode hard physical constraints, so actions may be statistically plausible while remaining dynamically unsafe under real-world perturbations ( Tölle et al., 2025 ) . |
| Opacity and Hierarchical Misalignment | High-dimensional latent representations are difficult to interpret formally, and mismatched interfaces between learned planners and constrained controllers can yield semantically valid but dynamically infeasible actions ( Firoozi et al., 2025 ; Yang et al., 2023 ; Ahn et al., 2022 ) . |
Table 4. Deployment Gauntlet barriers 5–8: recurring system-level challenges for embodied foundation models at the edge.
Figure 3. The taxonomy for Mitigation Strategies for Deployment Gauntlet on edge
| Mitigation Family | Core Strategy | Deployment Benefit | Representative Techniques / Evidence |
| Transport-Aware Sensing and Synchronization Relaxation | Reduce transport overhead, relax rigid synchronization, and preserve usable temporal structure before inference begins. | Lower tail latency and jitter, reduce copy overhead, and improve multimodal alignment without forcing the stack to wait on the slowest modality. | Zero-copy middleware transport and loaned buffers ( Ishikawa–Aso and Kato, 2025 ; Skevik et al., 2001 ; Ishikawa–Aso and Kato, 2025 ) , executor and synchronization refinements ( Kronauer et al., 2021 ; Abaza et al., 2024 ; Teper et al., 2024 ; Sobhani et al., 2023 ) , variable-sized zero-copy and accelerator-native transport ( Ishikawa–Aso and Kato, 2025 ; Salimpour et al., 2025 ) , pipelined and asynchronous sensing/action generation ( Huang et al., 2025 ; Jiang et al., 2025 ) , calibration maintenance ( Cheng et al., 2025 ) , and training-time robustness to misalignment or missing modalities ( Song et al., 2024 ; Wang et al., 2024a ; Liu et al., 2017 ; Yang et al., 2021 ) . |
| Hardware-Aligned Model and Compiler Co-Design | Expose hardware constraints at compile time and redesign operators or models to reduce fallback, fragmentation, and unsupported execution paths. | Higher accelerator residency, fewer host fallbacks, lower synchronization overhead, and better real-time feasibility on heterogeneous edge hardware. | SoC-aware compilation and heterogeneous placement ( Wali, 2024 ; Hamdi et al., 2025 ; Gross et al., 2020 ; Lv et al., 2025 ; Liu et al., 2022a ; Rognlien et al., 2022 ) , operator reformulation for end-to-end accelerator execution ( Wei et al., 2025 ; Park et al., 2025c ) , traffic-reducing quantization and IO-aware attention ( Dettmers et al., 2022 ; Frantar et al., 2022 ; Lin et al., 2024 ; Dao et al., 2022 ; Dao, 2023 ; Gupta et al., 2023 ) , host-device synchronization reduction ( Kwon et al., 2023 ; Zeng et al., 2024 ) , and hardware-aligned model redesign ( Chen et al., 2025a ; Smith et al., 2025 ; Wen et al., 2025 ; Leal et al., 2024 ) . |
| Memory-Traffic Reduction and Schedulable Memory Use | Treat shared memory as a schedulable resource and reduce traffic volume so sensing, inference, and control can coexist without excessive jitter. | More predictable LPDDR service, lower tail latency, fewer stalls, and improved sustained performance under unified-memory contention. | DRAM isolation and bandwidth policing ( Valsan et al., 2017 ; Bechtel and Yun, 2024 ; Yun et al., 2013 ; Yun et al., 2016 ; Aghilinasab et al., 2020 ; Fang et al., 2024 ; Puente et al., 2025 ; Sullivan et al., 2024 ; Zuepke et al., 2024 ) , streaming-stable quantization and pruning ( Chen et al., 2023 ; Cai et al., 2020b ; Lin et al., 2020 ; Li et al., 2023 ; Liu et al., 2019 ; Lin et al., 2019 ) , outlier suppression and uniform low-bit execution ( Ashkboos et al., 2024a ; Chen et al., 2025b ; Ashkboos et al., 2024b ; Liu et al., 2024a ) , and KV-cache redesign that trades compute for lower LPDDR pressure ( Liu et al., 2024b ; Zhang et al., 2024a ; Hooper et al., 2024 ; Shutova et al., 2025 ; Tomar et al., 2025 ) . |
| Energy- and Thermal-Aware Execution | Optimize for sustainable duty cycle and steady-state thermal feasibility rather than short-burst benchmark throughput. | Lower joules per decision, better endurance, reduced throttling, and more stable long-horizon real-time behavior. | Endurance-aware deployment evidence ( Boroujerdian et al., 2018 ) , energy-per-inference reduction through distillation and quantization ( Sun et al., 2020 ; Yao et al., 2022 ; Xiao et al., 2023a ; Horowitz, 2014 ; Chen et al., 2016 ; Koppula et al., 2019 ) , duty-cycle reduction for generative control ( Chi et al., 2025 ; Wang et al., 2024b ; Wu et al., 2025 ) , and predictive thermal-aware runtime management ( Lee, 2021 ; Lin et al., 2023 ; Dowling et al., 2025 ; Jacob, 2025 ) . |
| Online Robustness, Adaptation, and Runtime Assurance | Bound model mismatch through localized correction, constrained adaptation, external safety filters, and runtime supervision rather than unrestricted online relearning. | Improves long-horizon robustness and reduces unsafe behavior under realistic edge constraints while keeping correction computationally tractable. | Estimator-level drift compensation through neural bias prediction and lightweight observers ( Buchanan et al., 2022 ; Altawaitan et al., 2025 ; Kushawaha et al., 2025 ; George Thuruthel et al., 2022 ) , constrained test-time adaptation and streaming-stable updates ( Niu et al., 2022 ; Niu et al., 2023 ; Gong et al., 2022 ) , sparse and parameter-efficient adaptation ( Zhu et al., 2023 ; Cai et al., 2020a ; Zaken et al., 2022 ) , gradient-free and meta-learned adaptation ( Niu et al., 2024 ; O’Connell et al., 2022 ) , control-theoretic safety wrappers and predictive safety filters ( Ames et al., 2019 ; Wabersich and Zeilinger, 2021 ; Fisac et al., 2018 ; Cheng et al., 2019 ; Xiao et al., 2023b ) , runtime assurance architectures ( Desai et al., 2019 ; Chen et al., 2022 ; Tearle et al., 2021 ) , formal verification and uncertainty-aware certificates ( Katz et al., 2017 ; Katz et al., 2019 ; Ruan et al., 2018 ; Tayal et al., 2025 ; Zhang et al., 2025b ) , and runtime shielding and black-box monitoring ( Jansen et al., 2020 ; Corsi et al., 2024 ; Zolfagharian et al., 2024 ) . |
| Real-Time Systems Support and Timing-Aware Communication | Bound interference across middleware, runtime, kernel, memory, and transport layers so timing disturbances do not cascade across perception, planning, and control. | Improves timing predictability, reduces transport-induced jitter, and limits the spread of scheduling and communication variability through the execution stack. | Middleware callback isolation and chain-aware scheduling ( Casini et al., 2019 ; Yang et al., 2023 ; Choi et al., 2021 ) , kernel-level real-time support ( Gutiérrez et al., 2018 ) , compiled dataflow runtimes and zero-copy execution graphs ( Baig et al., 2026 ) , DRAM bandwidth regulation and arbitration control ( Yun et al., 2013 ; Yun et al., 2016 ; Aghilinasab et al., 2020 ; Pradhan et al., 2025 ) , zero-copy and heterogeneous I/O transport optimization ( Pöhnl et al., 2023 ; Bell et al., 2023 ; Mayoral-Vilches et al., 2024 ) , and communication-aware cloud–edge partitioning with compression or selective transmission ( Kang et al., 2017 ; Teerapittayanon et al., 2017 ; Eshratifar et al., 2019 ; Zhang et al., 2021 ; Li and Aijaz, 2025 ) . |
Table 5. Mitigation families for the Deployment Gauntlet: representative strategies for improving embodied foundation model deployment at the edge.
Figure 4. Architecture trends for future Embodied Foundation Models: Data volume is minimized via Neuromorphic and event sensing (A), and near sensor compute mechanism (B). A central predictive world model stabilizes perception (C) while cognition is separated into fast reflexes and slow reasoning (D). The rigorous safety policy(E) enforces constraints before actuation. And the Fleet learning (F) extends the feasible complexity.
More in this topic
Robot Foundation Models
Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.
See all 135 papers