cs.ROMar 16, 2026

Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies

Authors: Utkarsh Grover, Ravi Ranjan, Mingyang Mao, Trung Tien Dong, Satvik Praveen, Zhenqi Wu, J. Morris Chang, Tinoosh Mohsenin, 

Organizations: University of South Florida, USA · Florida International University, USA · Johns Hopkins University, USA · Nottingham Trent University, United Kingdom · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deploying foundation models in embodied edge systems is fundamentally a systems problem, not just a problem of model compression. Real-time control must operate within strict size, weight, and power constraints, where memory traffic, compute latency, timing variability, and safety margins interact directly. The Deployment Gauntlet organizes these constraints into eight coupled barriers that determine whether embodied foundation models can run reliably in practice. Across representative edge workloads, autoregressive Vision-Language-Action policies are constrained primarily by memory bandwidth, whereas diffusion-based controllers are limited more by compute latency and sustained execution cost. Reliable deployment therefore depends on system-level co-design across memory, scheduling, communication, and model architecture, including decompositions that separate fast control from slower semantic reasoning.

Figures & tables

Figure 1. The Deployment Gauntlet. A unified view of the eight major system barriers that limit the deployment of foundation models from the cloud to edge embodied AI platforms.
Workload ClassRepresentative ModelsPrimary Role / ModalityDominant Bottleneck
Vision-Language-Action (VLA) PoliciesRT-2 ( Zitkovich et al., 2023 ) , OpenVLA ( Kim et al., 2024 ) , NanoVLA ( Chen et al., 2025a )Policy generation; RGB + language to actionMemory traffic and weight streaming during autoregressive decoding, which compete directly with sensor DMA and shared-memory service.
Diffusion-Based PoliciesOcto ( Team et al., 2024 ) , OneDP ( Wang et al., 2024b ) , LightDP ( Wu et al., 2025 )Trajectory generation; RGB / multimodal to action sequenceSustained compute and thermal load from iterative denoising, which imposes a latency floor and high energy cost.
Vision Encoders and Multimodal LMMsLLaVA-Mini ( Zhang et al., 2025c ) , MiniCPM-V ( Yao et al., 2025 )Perception and reasoning; RGB + languageVisual prefill and bursty frame ingestion, which create transient compute and memory spikes during high-resolution encoding.
3D / LiDAR EncodersPointPillars ( Lang et al., 2019 ) , DensePillarNet ( Chandorkar et al., 2025 ) , BEVFusion ( Liu et al., 2022b )Geometric perception; LiDAR / LiDAR + RGBSparsity and irregular memory access, which reduce accelerator utilization and increase synchronization overhead.
Multimodal Fusion StacksMMEdge ( Huang et al., 2025 ) , BEVFusion ( Liu et al., 2022b )Cross-modal state construction; RGB / LiDAR / audio / textScheduling and synchronization pressure from shared middleware, memory bandwidth, and callback timing.
Table 1. Edge-relevant workload classes for embodied foundation model deployment.
WorkloadDominant BottleneckCharacteristic Edge Failure Mode
Vision-Language-Action (VLA) PoliciesMemory traffic and weight streamingAutoregressive action decoding repeatedly streams large parameter tensors, competing with sensor DMA traffic over shared memory and introducing latency spikes that can violate closed-loop control deadlines ( Kim et al., 2024 ; Prashanthi S et al., 2025 ; Chen et al., 2025a ) .
Diffusion-Based PoliciesSustained compute and thermal loadIterative denoising requires repeated dense matrix execution over the same control horizon, creating a latency floor, high energy draw, and thermal throttling that reduce practical control frequency on edge platforms ( Chi et al., 2025 ; Wang et al., 2024b ; Wu et al., 2025 ) .
Vision Encoders and Multimodal LMMsVisual prefill and bursty ingestionHigh-resolution visual streams require repeated encoder prefill, producing burst-driven compute and memory demand that can dominate the thermal envelope even when downstream reasoning remains lightweight ( Radford et al., 2023 ; Zhang et al., 2025c ; Yao et al., 2025 ; Saha and Xu, 2025 ) .
3D & LiDAR EncodersSparsity and irregular memory accessPoint-cloud pipelines rely on voxelization, sparse indexing, and data-dependent aggregation, which map poorly to dense accelerators, reduce utilization, and introduce synchronization overhead in multimodal stacks ( Choy et al., 2019 ; Lang et al., 2019 ; Tang et al., 2020 ; Liu et al., 2022b ) .
Multimodal Fusion StacksScheduling and synchronizationEven when unimodal components meet latency targets in isolation, their composition through shared middleware, LPDDR bandwidth, and runtime scheduling can produce temporal misalignment and end-to-end timing failures ( Feng et al., 2020 ; Kronauer et al., 2021 ; Huang et al., 2025 ; Bechtel and Yun, 2024 ) .
Table 2. Representative edge-relevant workload classes and their dominant execution bottlenecks in embodied foundation model deployment.
Figure 2. The taxonomy for the gauntlets and solutions in Foundation Models under Embodied Constraints.
BarrierRepresentative SubproblemSystems Effect
Sensor Fusion TaxTemporal and Spatial MisalignmentAsynchronous sensor clocks, transport delay, and calibration drift force buffering, synchronization, and corrective alignment, trading control responsiveness for temporal and spatial consistency ( Li et al., 2022 ; Jellum et al., 2022 ; Lv et al., 2021 ; Cheng et al., 2025 ) .
 Middleware and Pipeline OverheadsSerialization, copying, buffering, and sweep-level pipeline blocking consume bandwidth and introduce latency jitter that destabilizes high-frequency control execution even when average throughput is acceptable ( Kronauer et al., 2021 ; Teper et al., 2024 ; Huang et al., 2025 ) .
 Representational Mismatch and Foundation-Model SensitivityModalities such as event streams require costly reconstruction or voxelization before they can be processed by frame-based models, while foundation models further amplify fusion overhead through higher memory traffic and greater sensitivity to timing error ( Gallego et al., 2020 ; Gehrig et al., 2020 ; Bommasani et al., 2021 ; Tang et al., 2025 ; Ruan et al., 2025 ) .
Heterogeneous Compute MismatchOperator Coverage Gaps and Execution BubblesLimited operator support on edge accelerators forces graph partitioning, CPU fallback, and host–device synchronization, creating execution bubbles and large latency spikes that reduce end-to-end throughput ( He et al., 2022 ; Chen et al., 2025a ) .
 Granularity Mismatch and Kernel Launch OverheadFine-grained transformer kernels incur substantial launch overhead on embedded GPUs, accounting for 30–60% of latency and driving utilization below 20% during autoregressive decoding ( Zeng et al., 2024 ) .
 The Memory Wall under Unified-Memory ArchitecturesShared LPDDR bandwidth contention among CPUs, GPUs, NPUs, and sensors makes execution memory-bound and can offset nominal hardware-acceleration gains through arbitration and tensor movement overhead ( Ivanov et al., 2021 ; Kim et al., 2024 ) .
Table 3. Deployment Gauntlet barriers 1–4: recurring system-level challenges for embodied foundation models at the edge.
BarrierRepresentative SubproblemSystems Effect
Long-Horizon Execution Drift and Adaptation FailureThe Stationarity FallacyEmbodied deployment violates the stationarity assumptions of offline training: sensor noise, actuator behavior, and calibration drift shift over time, causing prediction error to accumulate as observations move away from well-supported regions of the training distribution ( Kumar et al., 2021 ) .
 Latent-State MisalignmentPhysical drift can disrupt the alignment between sensory embeddings and action-relevant latent structure, causing internally coherent predictions to diverge from the true physical state of the platform ( Ha and Schmidhuber, 2018 ) .
 The Adaptation GapEdge platforms often cannot support timely online adaptation under SWaP constraints, while cloud-mediated adaptation introduces latency and connectivity dependence, leaving long-duration deployments governed by increasingly outdated model assumptions ( Ren et al., 2025 ; Gao et al., 2025 ) .
 Residual Hardware and Environment-Coupled DriftSensor bias instability, thermal noise, mechanical wear, and terrain-dependent slip introduce residual time-varying error that classical compensation can attenuate but not fully remove ( Zhou et al., 2025 ; Vargas et al., 2021 ; Sigron et al., 2023 ; Sakayori and Ishigami, 2024 ) .
Safety & Verification GapStochastic Objectives and Constraint ViolationsLikelihood-based objectives such as token prediction and diffusion denoising do not inherently encode hard physical constraints, so actions may be statistically plausible while remaining dynamically unsafe under real-world perturbations ( Tölle et al., 2025 ) .
 Opacity and Hierarchical MisalignmentHigh-dimensional latent representations are difficult to interpret formally, and mismatched interfaces between learned planners and constrained controllers can yield semantically valid but dynamically infeasible actions ( Firoozi et al., 2025 ; Yang et al., 2023 ; Ahn et al., 2022 ) .
Table 4. Deployment Gauntlet barriers 5–8: recurring system-level challenges for embodied foundation models at the edge.
Figure 3. The taxonomy for Mitigation Strategies for Deployment Gauntlet on edge
Mitigation FamilyCore StrategyDeployment BenefitRepresentative Techniques / Evidence
Transport-Aware Sensing and Synchronization RelaxationReduce transport overhead, relax rigid synchronization, and preserve usable temporal structure before inference begins.Lower tail latency and jitter, reduce copy overhead, and improve multimodal alignment without forcing the stack to wait on the slowest modality.Zero-copy middleware transport and loaned buffers ( Ishikawa–Aso and Kato, 2025 ; Skevik et al., 2001 ; Ishikawa–Aso and Kato, 2025 ) , executor and synchronization refinements ( Kronauer et al., 2021 ; Abaza et al., 2024 ; Teper et al., 2024 ; Sobhani et al., 2023 ) , variable-sized zero-copy and accelerator-native transport ( Ishikawa–Aso and Kato, 2025 ; Salimpour et al., 2025 ) , pipelined and asynchronous sensing/action generation ( Huang et al., 2025 ; Jiang et al., 2025 ) , calibration maintenance ( Cheng et al., 2025 ) , and training-time robustness to misalignment or missing modalities ( Song et al., 2024 ; Wang et al., 2024a ; Liu et al., 2017 ; Yang et al., 2021 ) .
Hardware-Aligned Model and Compiler Co-DesignExpose hardware constraints at compile time and redesign operators or models to reduce fallback, fragmentation, and unsupported execution paths.Higher accelerator residency, fewer host fallbacks, lower synchronization overhead, and better real-time feasibility on heterogeneous edge hardware.SoC-aware compilation and heterogeneous placement ( Wali, 2024 ; Hamdi et al., 2025 ; Gross et al., 2020 ; Lv et al., 2025 ; Liu et al., 2022a ; Rognlien et al., 2022 ) , operator reformulation for end-to-end accelerator execution ( Wei et al., 2025 ; Park et al., 2025c ) , traffic-reducing quantization and IO-aware attention ( Dettmers et al., 2022 ; Frantar et al., 2022 ; Lin et al., 2024 ; Dao et al., 2022 ; Dao, 2023 ; Gupta et al., 2023 ) , host-device synchronization reduction ( Kwon et al., 2023 ; Zeng et al., 2024 ) , and hardware-aligned model redesign ( Chen et al., 2025a ; Smith et al., 2025 ; Wen et al., 2025 ; Leal et al., 2024 ) .
Memory-Traffic Reduction and Schedulable Memory UseTreat shared memory as a schedulable resource and reduce traffic volume so sensing, inference, and control can coexist without excessive jitter.More predictable LPDDR service, lower tail latency, fewer stalls, and improved sustained performance under unified-memory contention.DRAM isolation and bandwidth policing ( Valsan et al., 2017 ; Bechtel and Yun, 2024 ; Yun et al., 2013 ; Yun et al., 2016 ; Aghilinasab et al., 2020 ; Fang et al., 2024 ; Puente et al., 2025 ; Sullivan et al., 2024 ; Zuepke et al., 2024 ) , streaming-stable quantization and pruning ( Chen et al., 2023 ; Cai et al., 2020b ; Lin et al., 2020 ; Li et al., 2023 ; Liu et al., 2019 ; Lin et al., 2019 ) , outlier suppression and uniform low-bit execution ( Ashkboos et al., 2024a ; Chen et al., 2025b ; Ashkboos et al., 2024b ; Liu et al., 2024a ) , and KV-cache redesign that trades compute for lower LPDDR pressure ( Liu et al., 2024b ; Zhang et al., 2024a ; Hooper et al., 2024 ; Shutova et al., 2025 ; Tomar et al., 2025 ) .
Energy- and Thermal-Aware ExecutionOptimize for sustainable duty cycle and steady-state thermal feasibility rather than short-burst benchmark throughput.Lower joules per decision, better endurance, reduced throttling, and more stable long-horizon real-time behavior.Endurance-aware deployment evidence ( Boroujerdian et al., 2018 ) , energy-per-inference reduction through distillation and quantization ( Sun et al., 2020 ; Yao et al., 2022 ; Xiao et al., 2023a ; Horowitz, 2014 ; Chen et al., 2016 ; Koppula et al., 2019 ) , duty-cycle reduction for generative control ( Chi et al., 2025 ; Wang et al., 2024b ; Wu et al., 2025 ) , and predictive thermal-aware runtime management ( Lee, 2021 ; Lin et al., 2023 ; Dowling et al., 2025 ; Jacob, 2025 ) .
Online Robustness, Adaptation, and Runtime AssuranceBound model mismatch through localized correction, constrained adaptation, external safety filters, and runtime supervision rather than unrestricted online relearning.Improves long-horizon robustness and reduces unsafe behavior under realistic edge constraints while keeping correction computationally tractable.Estimator-level drift compensation through neural bias prediction and lightweight observers ( Buchanan et al., 2022 ; Altawaitan et al., 2025 ; Kushawaha et al., 2025 ; George Thuruthel et al., 2022 ) , constrained test-time adaptation and streaming-stable updates ( Niu et al., 2022 ; Niu et al., 2023 ; Gong et al., 2022 ) , sparse and parameter-efficient adaptation ( Zhu et al., 2023 ; Cai et al., 2020a ; Zaken et al., 2022 ) , gradient-free and meta-learned adaptation ( Niu et al., 2024 ; O’Connell et al., 2022 ) , control-theoretic safety wrappers and predictive safety filters ( Ames et al., 2019 ; Wabersich and Zeilinger, 2021 ; Fisac et al., 2018 ; Cheng et al., 2019 ; Xiao et al., 2023b ) , runtime assurance architectures ( Desai et al., 2019 ; Chen et al., 2022 ; Tearle et al., 2021 ) , formal verification and uncertainty-aware certificates ( Katz et al., 2017 ; Katz et al., 2019 ; Ruan et al., 2018 ; Tayal et al., 2025 ; Zhang et al., 2025b ) , and runtime shielding and black-box monitoring ( Jansen et al., 2020 ; Corsi et al., 2024 ; Zolfagharian et al., 2024 ) .
Real-Time Systems Support and Timing-Aware CommunicationBound interference across middleware, runtime, kernel, memory, and transport layers so timing disturbances do not cascade across perception, planning, and control.Improves timing predictability, reduces transport-induced jitter, and limits the spread of scheduling and communication variability through the execution stack.Middleware callback isolation and chain-aware scheduling ( Casini et al., 2019 ; Yang et al., 2023 ; Choi et al., 2021 ) , kernel-level real-time support ( Gutiérrez et al., 2018 ) , compiled dataflow runtimes and zero-copy execution graphs ( Baig et al., 2026 ) , DRAM bandwidth regulation and arbitration control ( Yun et al., 2013 ; Yun et al., 2016 ; Aghilinasab et al., 2020 ; Pradhan et al., 2025 ) , zero-copy and heterogeneous I/O transport optimization ( Pöhnl et al., 2023 ; Bell et al., 2023 ; Mayoral-Vilches et al., 2024 ) , and communication-aware cloud–edge partitioning with compression or selective transmission ( Kang et al., 2017 ; Teerapittayanon et al., 2017 ; Eshratifar et al., 2019 ; Zhang et al., 2021 ; Li and Aijaz, 2025 ) .
Table 5. Mitigation families for the Deployment Gauntlet: representative strategies for improving embodied foundation model deployment at the edge.
Figure 4. Architecture trends for future Embodied Foundation Models: Data volume is minimized via Neuromorphic and event sensing (A), and near sensor compute mechanism (B). A central predictive world model stabilizes perception (C) while cognition is separated into fast reflexes and slow reasoning (D). The rigorous safety policy(E) enforces constraints before actuation. And the Fleet learning (F) extends the feasible complexity.
More in this topic

Robot Foundation Models

Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.

See all 135 papers

Explore similar work

CardsList
Sep 22, 2026

Deploying Foundation Models for Embodied Navigation

We present and tackle two problems associated with deploying Foundation Models (FMs) on Embodied Agents performing navigation: 1) Training bias in FMs leading to poor personalization in unseen environments, and 2) Limited FM context length hindering success, especially on long horizon tasks. Our solution for the former involves priming the FM with human-habit data mined from the scene and our solution for the latter involves active memory management via a novel `memory head' augmentation. We first present a taxonomy of existing literature on FM-based Embodied Navigation, and highlight these limitations. We then present our approaches, Transit-Aware Planning (TAP) and MemCtrl to address the limitations. With TAP, we present real-world results in a lab environment with a Turtlebot for personalized target finding that shows an average improvement of 18% over a non-TAP baseline. On MemCtrl, we report a 6% average improvement across various embodied tasks, with 20% on long instruction subsets, all while using nearly half the context used in the baseline model. Motivated by these result, we present our stance the deployability of FM-based embodied agents in real-world environments, and highlight open research directions.

Vishnu Sashank Dorbala, Dinesh Manocha

University of Maryland, College Park

Embodied NavigationLarge Language Model-Based Robot PlanningRobot Foundation ModelsLLM Context ManagementMemory-Augmented Agents
AskPDF
Mar 1, 2026

Characterizing VLA Models: Identifying the Action Generation Bottleneck for Edge AI Architectures

Vision-Language-Action (VLA) models are an emerging class of workloads critical for robotics and embodied AI at the edge. As these models scale, they demonstrate significant capability gains, yet they must be deployed locally to meet the strict latency requirements of real-time applications. This paper characterizes VLA performance on two generations of edge hardware, viz. the Nvidia Jetson Orin and Thor platforms. Using MolmoAct-7B, a state-of-the-art VLA model, we identify a primary execution bottleneck: up to 75% of end-to-end latency is consumed by the memory-bound action-generation phase. Through analytical modeling and simulations, we project the hardware requirements for scaling to 100B parameter models. We also explore the impact of high-bandwidth memory technologies and processing-in-memory (PIM) as promising future pathways in edge systems for embodied AI.

Manoj Vishwanathan, Suvinay Subramanian, Anand Raghunathan

Google, Mountain View, CA USA · Purdue University, West Lafayette, IN USA

Vision-Language-Action ModelsEfficient VLA Model InferenceAI Accelerator InferenceEdge Computing
AskPDF
Apr 8, 2026

HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents. To bridge the gap between general Vision-Language Models (VLMs) and the demands of embodied agents, our models are developed to enhance the core capabilities required by embodied intelligence: spatial and temporal visual perception, alongside advanced embodied reasoning for prediction, interaction, and planning. The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning. To support the fine-grained visual perception essential for embodied tasks, we adopt a Mixture-of-Transformers (MoT) architecture to enable modality-specific computing. By incorporating latent tokens, this design effectively enhances the perceptual representation of the models. To improve reasoning capabilities, we introduce an iterative, self-evolving post-training paradigm. Furthermore, we employ on-policy distillation to transfer the advanced capabilities of the large model to the smaller variant, thereby maximizing the performance potential of the compact model. Extensive evaluations across 22 benchmarks, spanning visual perception, spatial reasoning, and embodied understanding, demonstrate the effectiveness of our approach. Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro. In downstream robot control experiments, we leverage our robust VLM foundation to train an effective Vision-Language-Action (VLA) model, achieving compelling results in real-world physical evaluations. Code and models are open-sourced at https://github.com/Tencent-Hunyuan/HY-Embodied.

Tencent Robotics X, HY Vision Team, : +20

Mixture-of-TransformersRobot Foundation ModelsVLMs for RoboticsEmbodied ReasoningSpatial Reasoning
AskPDF