VA-FastNavi-MARL: Real-Time Robot Control with Multimedia-Driven Meta-Reinforcement Learning
Authors: Yang Zhang, Shengxi Jing, Fengxiang Wang, Yuan Feng, Hong Wang
Organizations: Department of Mechanical and Aerospace Engineering, University of Missouri, Columbia, MO, 65201, USA · School of Construction Machinery, Chang’an University, Xi’an, 710064, Shaanxi, China · School of Artificial Intelligence, Hubei University, Wuhan, 430062, Hubei, China · Key Laboratory of Intelligent Sensing System and Security (Ministry of Education), Hubei University, Wuhan, China
Interpreting dynamic, heterogeneous multimedia commands with real-time responsiveness is critical for Human-Robot Interaction. We present VA-FastNavi-MARL, a framework that aligns asynchronous audio-visual inputs into a unified latent representation. By treating diverse instructions as a distribution of navigable goals via Meta-Reinforcement Learning, our method enables rapid adaptation to unseen directives with negligible inference overhead. Unlike approaches bottlenecked by heavy sensory processing, our modality-agnostic stream ensures seamless, low-latency control. Validation on a multi-arm workspace confirms that VA-FastNavi-MARL significantly outperforms baselines in sample efficiency and maintains robust, real-time execution even under noisy multimedia streams.
Figures & tables
Fig. 1 : Architecture of VA-FastNavi-MARL. Multimedia Instruction Generator (Top) Heterogeneous inputs (machine, audio, visual) are fused into a unified embedding ( φn ) via parallel encoders. Dynamic Instruction Stream (Middle) An asynchronous buffer schedules real-time commands ( φt ) to handle irregular input rates. Task-adaptive Motion Controller (Bottom) The active instruction parameterizes the task ( Tt ), triggering the Meta-RL policy πθ to adapt into a specialized task policy πθt for control.
Fig. 2 : Architecture of the policy network. The network processes a 15-dimensional state vector through two hidden layers and uses a dual-head structure to output the parameters ( μ and logσ ) of a 9-dimensional Gaussian policy. Output values are clamped for numerical stability before final action sampling and scaling.
Fig. 3 : Performance comparison across different models. From left to right: Average Reward, Success Rate, and Collision Rate. Our method (Solid Red) achieves faster adaptation and higher safety compared to baselines.
Fig. 4 : Trajectory evolution (Top) during the Long-Horizon Continuous Adaptation test ( T=2000 steps).The experiment is segmented into five distinct phases (separated by vertical dashed lines), where the underlying instruction logic shifts abruptly from simple instructions (e.g., Seq: [1,2,3] ) to complex instructions (e.g., Seq: [2,3,1,3,2,3] ). Bottom is the state activation timeline corresponding to the continuous adaptation task. Blue segments denote the Approaching phase (active execution), while pink regions represent the Returning/Holding phase.
Difficulty
Input Type
Time (s)
Accuracy (%)
Easy
Audio
0.008 ± 0.001
93.80 ± 1.70
Audio (noise)
0.008 ± 0.001
87.40 ± 2.70
Audio-Visual
0.389 ± 0.038
94.60 ± 0.80
Audio-Visual (noise)
0.498 ± 0.058
92.80 ± 2.14
Medium
Audio
0.015 ± 0.001
87.80 ± 1.60
Audio (noise)
0.017 ± 0.001
77.20 ± 2.00
TABLE I : Instruction Generation Time and Accuracy of VA-FastNavi-MARL Across Instruction Difficulty Levels and Multimedia Inputs (5 Seeds, 100 Samples)
Robot manipulation alternates between low-risk transit phases that call for fast execution and high-risk contact stages that demand slow, precise motion. Yet existing Vision-Language-Action models (VLAs) only inherit a single fixed speed from training demonstrations. Prior efforts to accelerate VLAs through model compression, KV-cache reuse, or reinforcement learning only shift the policy from one fixed speed to another, and leave deceleration almost unexplored. We observe that the magnitude of each predicted action already governs how fast the robot moves, opening a direct route to controllable execution speed. We turn this observation into TempoVLA, a single VLA whose execution speed is controlled by an explicit condition. TempoVLA combines two coupled components. (1) A data-side Variable-Speed Trajectory Augmentation (VSTA) that re-times demonstration to any target speed by merging or splitting actions while preserving its motion semantics. (2) A model-side conditioning mechanism that feeds the speed to the policy. Statistics show that VSTA reaches the requested speed with negligible motion error. Experiments in simulation and on real-world tasks demonstrate that TempoVLA achieves flexible speed control in both directions, while VSTA additionally boosts the default 1× performance via better data utilization. Furthermore, by cooperating with a large multimodal model, TempoVLA realizes dynamic speed control, accelerating through low-risk phases and decelerating for high-risk ones.
Recent navigation systems achieve strong benchmark results, yet real-world deployment often remains visibly stop-and-go. This bottleneck arises because the sense-inference-execution loop is still blocking: after each new observation, the controller must wait for sensing, transmission, and inference before motion can continue. Reducing action-generation cost alone therefore does not remove redundant waiting. To address this issue, we present LiveVLN, a training-free framework for more continuous embodied navigation by augmenting pretrained VLM navigators with multi-step action continuation. Instead of pausing for each full sense-and-inference round, LiveVLN overlaps execution with the processing of newly arrived observations, allowing refreshed future actions to be handed off before the current executable prefix is exhausted. This design keeps actions continuously available during motion, reducing idle waiting and enabling smoother online execution. The framework operates at runtime and can be integrated with compatible pretrained VLM navigators. Across R2R and RxR, LiveVLN preserves benchmark performance while reducing waiting time and improving action availability. In real-world deployments, it cuts average episode waiting time by up to 77.7% and shortens wall-clock episode time by 12.6% on StreamVLN and 19.6% on NaVIDA, yielding more coherent execution during deployment. Code is available at https://github.com/NIneeeeeem/LiveVLN.
Xiangchen Wang, Weiye Zhu, Teng Wang +5
Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen · Spatialtemporal AI
VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control. However, current single-frame architectures suffer from intrinsic limitations: temporal myopia that discards historical dynamics, reasoning gaps between high-level instructions and low-level motor commands, and inference inefficiency due to autoregressive scalar decoding. In this work, we propose MIRTH, a unified framework designed to address these challenges. MIRTH augments a pretrained VLA backbone with three key innovations: (1) dual-scale temporal memory hubs that compress long-term scene evolution and short-term motion trends into compact embeddings; (2) latent reasoning tokens optimized via a mutual-information objective carving out a semantic plan space to align multimodal context with action trajectories; and (3) a parallel action decoding scheme that replaces autoregressive generation with vector-wise prediction to maximize control throughput. Extensive evaluations on the LIBERO simulation benchmark and a real-world LeRobot platform demonstrate that MIRTH achieves state-of-the-art performance and exhibiting emergent error recovery capabilities. The codes and collected datasets are released at http://github.com/kiva12138/mirth.
Hao Sun, Yu Song, Shiyu Teng +2
College of Information Science and Engineering, Ritsumeikan University, Japan · College of Computer Science and Technology, Zhejiang University, China