VA-FastNavi-MARL: Real-Time Robot Control with Multimedia-Driven Meta-Reinforcement Learning
Authors: Yang Zhang, Shengxi Jing, Fengxiang Wang, Yuan Feng, Hong Wang
Organizations: Department of Mechanical and Aerospace Engineering, University of Missouri, Columbia, MO, 65201, USA · School of Construction Machinery, Chang’an University, Xi’an, 710064, Shaanxi, China · School of Artificial Intelligence, Hubei University, Wuhan, 430062, Hubei, China · Key Laboratory of Intelligent Sensing System and Security (Ministry of Education), Hubei University, Wuhan, China
Interpreting dynamic, heterogeneous multimedia commands with real-time responsiveness is critical for Human-Robot Interaction. We present VA-FastNavi-MARL, a framework that aligns asynchronous audio-visual inputs into a unified latent representation. By treating diverse instructions as a distribution of navigable goals via Meta-Reinforcement Learning, our method enables rapid adaptation to unseen directives with negligible inference overhead. Unlike approaches bottlenecked by heavy sensory processing, our modality-agnostic stream ensures seamless, low-latency control. Validation on a multi-arm workspace confirms that VA-FastNavi-MARL significantly outperforms baselines in sample efficiency and maintains robust, real-time execution even under noisy multimedia streams.
Figures & tables
Fig. 1 : Architecture of VA-FastNavi-MARL. Multimedia Instruction Generator (Top) Heterogeneous inputs (machine, audio, visual) are fused into a unified embedding ( φn ) via parallel encoders. Dynamic Instruction Stream (Middle) An asynchronous buffer schedules real-time commands ( φt ) to handle irregular input rates. Task-adaptive Motion Controller (Bottom) The active instruction parameterizes the task ( Tt ), triggering the Meta-RL policy πθ to adapt into a specialized task policy πθt for control.
Fig. 2 : Architecture of the policy network. The network processes a 15-dimensional state vector through two hidden layers and uses a dual-head structure to output the parameters ( μ and logσ ) of a 9-dimensional Gaussian policy. Output values are clamped for numerical stability before final action sampling and scaling.
Fig. 3 : Performance comparison across different models. From left to right: Average Reward, Success Rate, and Collision Rate. Our method (Solid Red) achieves faster adaptation and higher safety compared to baselines.
Fig. 4 : Trajectory evolution (Top) during the Long-Horizon Continuous Adaptation test ( T=2000 steps).The experiment is segmented into five distinct phases (separated by vertical dashed lines), where the underlying instruction logic shifts abruptly from simple instructions (e.g., Seq: [1,2,3] ) to complex instructions (e.g., Seq: [2,3,1,3,2,3] ). Bottom is the state activation timeline corresponding to the continuous adaptation task. Blue segments denote the Approaching phase (active execution), while pink regions represent the Returning/Holding phase.
Difficulty
Input Type
Time (s)
Accuracy (%)
Easy
Audio
0.008 ± 0.001
93.80 ± 1.70
Audio (noise)
0.008 ± 0.001
87.40 ± 2.70
Audio-Visual
0.389 ± 0.038
94.60 ± 0.80
Audio-Visual (noise)
0.498 ± 0.058
92.80 ± 2.14
Medium
Audio
0.015 ± 0.001
87.80 ± 1.60
Audio (noise)
0.017 ± 0.001
77.20 ± 2.00
TABLE I : Instruction Generation Time and Accuracy of VA-FastNavi-MARL Across Instruction Difficulty Levels and Multimedia Inputs (5 Seeds, 100 Samples)