Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
Authors: Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, Hesheng Wang
Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China. · Department of Engineering, University of Cambridge, Cambridge, CB2 1TN, UK.
While recent Vision-Language-Action (VLA) models have begun to incorporate audio, they typically treat sound as static pre-execution prompts or focus exclusively on human speech. This leaves a significant gap in real-time, sound-centric manipulation where fleeting environmental acoustics provide critical state verification during task execution. Consequently, key sounds are easily missed due to low-frequency updates or system latency. This problem is exacerbated by action chunking with open-loop execution, which creates a Blind Execution Interval where acoustic events are lost between discrete audio observation windows. Recognizing the necessity of continuous auditory awareness, we formalize Vision-Sound-Language-Action (VSLA) as a continuous control paradigm conditioned on vision, streaming audio, language, and proprioception under delayed decision loops. As an instantiation, we introduce HEAR, a VSLA framework integrating four components: (i) a streaming Historizer to maintain a compact, causal audio context across execution gaps; (ii) an Envisioner adapted from omni foundation models to reason over multi-sensory inputs; (iii) an Advancer, formulated as an audio world model, to learn temporal dynamics by predicting near-future audio codes; and (iv) a flow-matching Realizer policy to generate smooth action chunks. To address the scarcity of pretraining data and evaluations for VSLA, we construct OpenX-Sound for pretraining, alongside HEAR-Bench, the first sound-centric manipulation benchmark with strict causal timing rules. Our results suggest that robust sound-centric manipulation necessitates causal persistence and explicit temporal learning. This framework provides a practical step toward multi-sensory foundation models for embodied agents, enabling robots to perceive and interact with dynamic environments. Code and videos are available at https://hear.irmv.top
Figures & tables
Figure 1: The HEAR framework for Vision-Sound-Language-Action (VSLA) manipulation. Scenarios: Sound-centric manipulation requires policies to use speech, trigger sounds, continuous process sounds, and interaction-generated sounds. Challenge: Extending VLA policies to VSLA introduces a timing bottleneck: transient acoustic cues can fall inside the Blind Execution Interval (BEI) created by system latency and open-loop action chunking. Solution: HEAR addresses this setting through causal audio memory ( Historizer ), omni-sensory reasoning ( Envisioner ), predictive audio dynamics ( Advancer ), and smooth flow-matching action generation ( Realizer ). Training & Evaluation: OpenX-Sound supports scalable robotics-specific pretraining, while HEAR-Bench and physical robot deployments provide sound-causal evaluation.
Figure 2: The HEAR framework architecture. The Historizer processes audio packets with a Streaming Stateful Transformer that maintains a compact causal memory htk and bridges execution gaps, including the Blind Execution Interval induced by open-loop chunking. The Envisioner employs a hierarchical design. Its high-level omni-modal model integrates multimodal inputs, including vision, instruction, robot state, and audio context, and outputs a semantic latent ztk and a key–value cache KVtk . It also predicts a text stage description y^tkjson . Its low-level model reuses KVtk together with the current state and extracts a control feature utk . The Advancer is a decoder-only transformer that predicts a near-future audio code sequence ztk→tk+1a from ztk during training. The Realizer synthesizes smooth action chunks via Conditional Flow Matching conditioned on utk .
Figure 3: The Historizer module. It processes continuous audio streams into discrete packets and uses a streaming stateful transformer to maintain a persistent causal memory. This mechanism ensures that transient acoustic events occurring during blind execution intervals are preserved for the next decision cycle.
Figure 4: The Envisioner module. A hierarchical reasoning architecture fuses multimodal inputs to guide manipulation. The high-level omni-modal model extracts semantic latents and stage descriptions, while the low-level model reuses the resulting key-value cache alongside proprioceptive data to efficiently generate control features.
Figure 5: The Advancer module. In scenarios requiring sustained waiting with quasi-static visual observations, this decoder-only audio world model predicts near-future acoustic codes. This predictive objective grounds the shared latent representation in continuous time, helping the policy maintain stability and temporal awareness.
Task
Primary cue
Event-triggered alarms
Alarm Clock
alarm ringing
Microwave
completion beep
Human speech
Check Yes
prosody-driven speech
Interrupt
spoken interruption
Table 1: HEAR-Bench tasks. The benchmark evaluates four primary categories of decision-critical acoustic cues under strict temporal constraints.
Figure 6: Robot platforms and sensor suites. (a) For HEAR-Bench simulations, we use a dual-arm RoboTwin2 equipped with three RGB camera views (Head Cam, Left Cam, Right Cam). (b) For real-world experiments, we deploy a single-arm Franka Panda with a head-mounted camera, an arm-mounted camera, and a standard workspace microphone.
Method
VLA
Alarm Clock
Check Yes
Check Materials
Pour Water
Boil Water
Microwave
Interrupt
Avg.
OpenVLA ( Kim et al., 2024 )
Y
0.00
0.33
0.31
0.00
0.00
0.00
0.03
0.10
OpenVLA-Waveform ( Kim et al., 2024 )
Y
0.71
0.60
0.63
0.26
0.37
0.45
0.69
0.53
OpenVLA-ASR ( Kim et al., 2024 )
Y
0.16
0.59
0.10
0.06
0.03
0.41
0.65
0.29
π0.5 ( Intelligence et al., 2025 )
Y
0.00
0.48
0.43
0.00
0.00
0.00
0.09
0.14
π0.5 -Waveform ( Intelligence et al., 2025 )
Y
0.80
0.73
0.71
0.31
0.43
0.52
0.75
0.61
π0.5 -ASR ( Intelligence et al., 2025 )
Y
0.21
0.70
0.15
0.09
0.04
0.52
0.74
0.35
Table 2: Simulation results on HEAR-Bench. Success rates ↑ are averaged over 100 independent trials per task. The “VLA” column indicates whether the baseline relies on a vision-language-action backbone.
Figure 7: Qualitative execution on HEAR-Bench: Alarm Clock . The policy waits for the ring onset, autonomously selects an arm, and presses the stop button strictly after the acoustic cue. This prevents visually plausible but premature actions.
Figure 8: Qualitative execution on HEAR-Bench: Microwave . A brief “ding” occurs at a randomized time. The robot successfully captures this transient cue and opens the door only after the sound has finished.
Figure 9: Qualitative execution on HEAR-Bench: Check Yes . The spoken transcript is consistently “Yes”. The robot relies entirely on intonation and prosody to determine whether to hold the current object or switch to the alternative.
Figure 10: Qualitative execution on HEAR-Bench: Interrupt . While performing an ongoing manipulation routine, the robot receives a randomized spoken “reset” command. It overrides its current motion and safely returns to the initial pose.
Figure 11: Qualitative execution on HEAR-Bench: Check Materials . The robot drops a cube onto one of two visually identical plates. It infers the underlying material property entirely from the resulting impact acoustics to complete the sorting task.
Figure 12: Qualitative execution on HEAR-Bench: Pour Water . The robot dispenses liquid into an opaque container and stops precisely at a target level guided by continuous acoustic feedback. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 13: Qualitative execution on HEAR-Bench: Boil Water . The robot maintains a stable posture through a prolonged, visually static period and intervenes strictly once the evolving sound indicates a rolling boil. The vertical axis of the audio waveform is scaled by 10x for visualization.
Method
VLA
Moka Coffee
Answer Phone
Shake Bottle
Real Alarm Clock
Avg.
OpenVLA ( Kim et al., 2024 )
Y
0.00
0.00
0.22
0.01
0.06
OpenVLA-Waveform ( Kim et al., 2024 )
Y
0.13
0.10
0.41
0.77
0.35
OpenVLA-ASR ( Kim et al., 2024 )
Y
0.05
0.05
0.19
0.63
0.23
π0.5 ( Intelligence et al., 2025 )
Y
0.00
0.01
0.48
0.04
0.13
π0.5 -Waveform ( Intelligence et al., 2025 )
Y
0.19
0.17
0.54
0.90
0.45
π0.5 -ASR ( Intelligence et al., 2025 )
Y
0.12
0.11
0.21
0.78
0.31
Table 3: Real-robot evaluation results. Success rates ↑ across four physical tasks, averaged over 100 trials each. All tasks strictly enforce the sound-causal success criteria, penalizing visually plausible but premature actions. For the two hardest long-horizon tasks, the acoustic setup and task-specific data configuration are synchronized across methods, and all methods were re-evaluated under the same conditions.
Figure 14: Real-robot physical deployment: Moka Coffee . The robot monitors the complex acoustics of an uninstrumented Moka pot and initiates pouring only after detecting the characteristic sputtering phase. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 15: Real-robot physical deployment: Answer Phone . Despite severe visual aliasing, the robot successfully waits for a ringtone, hands the phone to a user, and returns it to the base only after the acoustic end-of-call signal.
Figure 16: Real-robot physical deployment: Shake Bottle . The robot performs a deliberate shaking motion to actively elicit sound from an opaque bottle, using the resulting rattle to classify its hidden contents. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 17: Real-robot physical deployment: Real Alarm Clock . Operating under real-world background noise, the robot suppresses early visual heuristics and presses the button strictly after the sustained ring begins. The vertical axis of the audio waveform is scaled by 10x for visualization.
Variant
Avg. success ↑
HEAR full (OpenX-Sound pretrained)
0.81
No OpenX-Sound pretraining
0.69
No Historizer
0.57
Historizer: causal GRU memory
0.67
Historizer: EMA/pooling memory
0.62
No Advancer
0.73
Table 4: Component ablations in HEAR-Bench. We report the average success rate across all seven simulation tasks, demonstrating the contribution of each architectural module and training phase.
Vision-language-action (VLA) models have advanced robotic manipulation by connecting visual perception and language instructions to action generation. However, many manipulation tasks depend on auditory cues that cannot be reliably inferred from visual observations and may need to be elicited through physical interaction. To address this gap, we present ListenVLA, an audio-conditioned VLA policy built on GR00T N1.7 for sound-guided manipulation. ListenVLA combines semantic audio embeddings with waveform-derived acoustic cues over a causal audio history and conditions continuous action generation through audio tokens and feature modulation. Moreover, we introduce an audio-action binding objective that uses context-matched demonstrations to link auditory differences to demonstrated action differences. The objective supervises short-horizon differences between flow-velocity predictions under different audio conditions, with shared pure-noise action inputs and fixed non-audio observations, alongside full-chunk action supervision. Furthermore, we develop LIBERO-Audio, an eight-task benchmark covering sound-based target selection, responses to auditory events, and physical interactions that reveal acoustic evidence. Experiments on LIBERO-Audio and three real-world manipulation tasks show excellent performance. Ablation studies support the contributions of binding supervision and audio modulation, while audio swap tests show that sound influences action predictions. Our code and videos are available at https://anonymous.4open.science/r/listenvla-F18D
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
Yulin Luo, Hao Chen, Zhuangzhe Wu +10
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance. We show that this failure is not resolved by extending context length alone in the current reactive execution setting; instead, it stems from three recurring execution-loop deficiencies: the memory gap, the verification gap, and the recovery gap. We present HELM, a model-agnostic framework that addresses these deficiencies with three components: an Episodic Memory Module (EMM) that retrieves key task history via CLIP-indexed keyframes, a learned State Verifier (SV) that predicts action failure before execution from observation, action, subgoal, and memory-conditioned context, and a Harness Controller (HC) that performs rollback and replanning. The SV is the core learning contribution: it consistently outperforms rule-based feasibility checks and ensemble uncertainty baselines, and its effectiveness depends critically on access to episodic memory. On LIBERO-LONG, HELM improves task success rate by 23.1 percentage points over OpenVLA (58.4% to 81.5%), while extending the context window to H=32 yields only a 5.4-point gain and same-budget LoRA adaptation remains 12.2 points below HELM. HELM also improves long-horizon performance on CALVIN and substantially boosts recovery success under controlled perturbations. Ablations and mechanism analyses isolate the contribution of each component, and we release LIBERO-Recovery as a perturbation-injection protocol for evaluating failure recovery in long-horizon manipulation.
Zijian Zeng, Fei Ding, Huiming Yang +1
Tsinghua University · Alibaba Group · Bengbu University