Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
Authors: Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, Hesheng Wang
Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai 200240, China. · Department of Engineering, University of Cambridge, Cambridge, CB2 1TN, UK.
While recent Vision-Language-Action (VLA) models have begun to incorporate audio, they typically treat sound as static pre-execution prompts or focus exclusively on human speech. This leaves a significant gap in real-time, sound-centric manipulation where fleeting environmental acoustics provide critical state verification during task execution. Consequently, key sounds are easily missed due to low-frequency updates or system latency. This problem is exacerbated by action chunking with open-loop execution, which creates a Blind Execution Interval where acoustic events are lost between discrete audio observation windows. Recognizing the necessity of continuous auditory awareness, we formalize Vision-Sound-Language-Action (VSLA) as a continuous control paradigm conditioned on vision, streaming audio, language, and proprioception under delayed decision loops. As an instantiation, we introduce HEAR, a VSLA framework integrating four components: (i) a streaming Historizer to maintain a compact, causal audio context across execution gaps; (ii) an Envisioner adapted from omni foundation models to reason over multi-sensory inputs; (iii) an Advancer, formulated as an audio world model, to learn temporal dynamics by predicting near-future audio codes; and (iv) a flow-matching Realizer policy to generate smooth action chunks. To address the scarcity of pretraining data and evaluations for VSLA, we construct OpenX-Sound for pretraining, alongside HEAR-Bench, the first sound-centric manipulation benchmark with strict causal timing rules. Our results suggest that robust sound-centric manipulation necessitates causal persistence and explicit temporal learning. This framework provides a practical step toward multi-sensory foundation models for embodied agents, enabling robots to perceive and interact with dynamic environments. Code and videos are available at https://hear.irmv.top
Figures & tables
Figure 1: The HEAR framework for Vision-Sound-Language-Action (VSLA) manipulation. Scenarios: Sound-centric manipulation requires policies to use speech, trigger sounds, continuous process sounds, and interaction-generated sounds. Challenge: Extending VLA policies to VSLA introduces a timing bottleneck: transient acoustic cues can fall inside the Blind Execution Interval (BEI) created by system latency and open-loop action chunking. Solution: HEAR addresses this setting through causal audio memory ( Historizer ), omni-sensory reasoning ( Envisioner ), predictive audio dynamics ( Advancer ), and smooth flow-matching action generation ( Realizer ). Training & Evaluation: OpenX-Sound supports scalable robotics-specific pretraining, while HEAR-Bench and physical robot deployments provide sound-causal evaluation.
Figure 2: The HEAR framework architecture. The Historizer processes audio packets with a Streaming Stateful Transformer that maintains a compact causal memory htk and bridges execution gaps, including the Blind Execution Interval induced by open-loop chunking. The Envisioner employs a hierarchical design. Its high-level omni-modal model integrates multimodal inputs, including vision, instruction, robot state, and audio context, and outputs a semantic latent ztk and a key–value cache KVtk . It also predicts a text stage description y^tkjson . Its low-level model reuses KVtk together with the current state and extracts a control feature utk . The Advancer is a decoder-only transformer that predicts a near-future audio code sequence ztk→tk+1a from ztk during training. The Realizer synthesizes smooth action chunks via Conditional Flow Matching conditioned on utk .
Figure 3: The Historizer module. It processes continuous audio streams into discrete packets and uses a streaming stateful transformer to maintain a persistent causal memory. This mechanism ensures that transient acoustic events occurring during blind execution intervals are preserved for the next decision cycle.
Figure 4: The Envisioner module. A hierarchical reasoning architecture fuses multimodal inputs to guide manipulation. The high-level omni-modal model extracts semantic latents and stage descriptions, while the low-level model reuses the resulting key-value cache alongside proprioceptive data to efficiently generate control features.
Figure 5: The Advancer module. In scenarios requiring sustained waiting with quasi-static visual observations, this decoder-only audio world model predicts near-future acoustic codes. This predictive objective grounds the shared latent representation in continuous time, helping the policy maintain stability and temporal awareness.
Task
Primary cue
Event-triggered alarms
Alarm Clock
alarm ringing
Microwave
completion beep
Human speech
Check Yes
prosody-driven speech
Interrupt
spoken interruption
Table 1: HEAR-Bench tasks. The benchmark evaluates four primary categories of decision-critical acoustic cues under strict temporal constraints.
Figure 6: Robot platforms and sensor suites. (a) For HEAR-Bench simulations, we use a dual-arm RoboTwin2 equipped with three RGB camera views (Head Cam, Left Cam, Right Cam). (b) For real-world experiments, we deploy a single-arm Franka Panda with a head-mounted camera, an arm-mounted camera, and a standard workspace microphone.
Method
VLA
Alarm Clock
Check Yes
Check Materials
Pour Water
Boil Water
Microwave
Interrupt
Avg.
OpenVLA ( Kim et al., 2024 )
Y
0.00
0.33
0.31
0.00
0.00
0.00
0.03
0.10
OpenVLA-Waveform ( Kim et al., 2024 )
Y
0.71
0.60
0.63
0.26
0.37
0.45
0.69
0.53
OpenVLA-ASR ( Kim et al., 2024 )
Y
0.16
0.59
0.10
0.06
0.03
0.41
0.65
0.29
π0.5 ( Intelligence et al., 2025 )
Y
0.00
0.48
0.43
0.00
0.00
0.00
0.09
0.14
π0.5 -Waveform ( Intelligence et al., 2025 )
Y
0.80
0.73
0.71
0.31
0.43
0.52
0.75
0.61
π0.5 -ASR ( Intelligence et al., 2025 )
Y
0.21
0.70
0.15
0.09
0.04
0.52
0.74
0.35
Table 2: Simulation results on HEAR-Bench. Success rates ↑ are averaged over 100 independent trials per task. The “VLA” column indicates whether the baseline relies on a vision-language-action backbone.
Figure 7: Qualitative execution on HEAR-Bench: Alarm Clock . The policy waits for the ring onset, autonomously selects an arm, and presses the stop button strictly after the acoustic cue. This prevents visually plausible but premature actions.
Figure 8: Qualitative execution on HEAR-Bench: Microwave . A brief “ding” occurs at a randomized time. The robot successfully captures this transient cue and opens the door only after the sound has finished.
Figure 9: Qualitative execution on HEAR-Bench: Check Yes . The spoken transcript is consistently “Yes”. The robot relies entirely on intonation and prosody to determine whether to hold the current object or switch to the alternative.
Figure 10: Qualitative execution on HEAR-Bench: Interrupt . While performing an ongoing manipulation routine, the robot receives a randomized spoken “reset” command. It overrides its current motion and safely returns to the initial pose.
Figure 11: Qualitative execution on HEAR-Bench: Check Materials . The robot drops a cube onto one of two visually identical plates. It infers the underlying material property entirely from the resulting impact acoustics to complete the sorting task.
Figure 12: Qualitative execution on HEAR-Bench: Pour Water . The robot dispenses liquid into an opaque container and stops precisely at a target level guided by continuous acoustic feedback. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 13: Qualitative execution on HEAR-Bench: Boil Water . The robot maintains a stable posture through a prolonged, visually static period and intervenes strictly once the evolving sound indicates a rolling boil. The vertical axis of the audio waveform is scaled by 10x for visualization.
Method
VLA
Moka Coffee
Answer Phone
Shake Bottle
Real Alarm Clock
Avg.
OpenVLA ( Kim et al., 2024 )
Y
0.00
0.00
0.22
0.01
0.06
OpenVLA-Waveform ( Kim et al., 2024 )
Y
0.13
0.10
0.41
0.77
0.35
OpenVLA-ASR ( Kim et al., 2024 )
Y
0.05
0.05
0.19
0.63
0.23
π0.5 ( Intelligence et al., 2025 )
Y
0.00
0.01
0.48
0.04
0.13
π0.5 -Waveform ( Intelligence et al., 2025 )
Y
0.19
0.17
0.54
0.90
0.45
π0.5 -ASR ( Intelligence et al., 2025 )
Y
0.12
0.11
0.21
0.78
0.31
Table 3: Real-robot evaluation results. Success rates ↑ across four physical tasks, averaged over 100 trials each. All tasks strictly enforce the sound-causal success criteria, penalizing visually plausible but premature actions. For the two hardest long-horizon tasks, the acoustic setup and task-specific data configuration are synchronized across methods, and all methods were re-evaluated under the same conditions.
Figure 14: Real-robot physical deployment: Moka Coffee . The robot monitors the complex acoustics of an uninstrumented Moka pot and initiates pouring only after detecting the characteristic sputtering phase. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 15: Real-robot physical deployment: Answer Phone . Despite severe visual aliasing, the robot successfully waits for a ringtone, hands the phone to a user, and returns it to the base only after the acoustic end-of-call signal.
Figure 16: Real-robot physical deployment: Shake Bottle . The robot performs a deliberate shaking motion to actively elicit sound from an opaque bottle, using the resulting rattle to classify its hidden contents. The vertical axis of the audio waveform is scaled by 10x for visualization.
Figure 17: Real-robot physical deployment: Real Alarm Clock . Operating under real-world background noise, the robot suppresses early visual heuristics and presses the button strictly after the sustained ring begins. The vertical axis of the audio waveform is scaled by 10x for visualization.
Variant
Avg. success ↑
HEAR full (OpenX-Sound pretrained)
0.81
No OpenX-Sound pretraining
0.69
No Historizer
0.57
Historizer: causal GRU memory
0.67
Historizer: EMA/pooling memory
0.62
No Advancer
0.73
Table 4: Component ablations in HEAR-Bench. We report the average success rate across all seven simulation tasks, demonstrating the contribution of each architectural module and training phase.
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics