I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance

Amit Kumar Singh Yadav; Ritvik Shrivastava; Xuan Zhang; Seungwhan Moon; Shashank Jain; Pinar Donmez; Babak Damavandi

cs.SDSep 18, 2026

I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance

Authors: Amit Kumar Singh Yadav, Ritvik Shrivastava, Xuan Zhang, Seungwhan Moon, Shashank Jain, Pinar Donmez, Babak Damavandi

Organizations: Meta Reality Labs, USA

Abstract

Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: <interrupt> and <silent>, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.

Figures & tables

Figure 1: Reactive vs. proactive audio assistance. Existing AudioLLMs (top) require a query for every event. Our proactive approach (bottom) takes a single watch-out intent and autonomously decides when to intervene.

Panel of Figure 2 — Figure 2: (a) PALLM takes a single watch-out intent and continuously encodes audio, decoding either <interrupt> or <silent> . (b) ISM training with four decision states. Conversation history enables de-duplication (S2).

Data	Method	Precision		F1	Recall
Data	Method	$P_{I}$	$P_{S}$	$F_{I}$	$R_{I1}$	$R_{I2}$	$R_{S1}$	$R_{S2}$
ESC-50	Zero-Shot	58.8	99.6	74.1	100.0	99.8	59.9	0.2
	Reactive SFT	67.2	90.1	79.8	98.1	89.9	98.1	10.1
	PALLM (ISM)	99.4	99.7	99.6	99.8	100.0	99.4	100.0
Epic- Sounds	Zero-Shot	50.8	69.7	66.9	98.3	100.0	3.9	—
	Reactive SFT	92.9	65.7	65.2	50.2	76.5	96.1	—

Table 1: Proactive evaluation results (%).

P

: precision,

R

: recall,

F_{I}

: interrupt F1. I: interrupt, S: silent. Subscripts denote type. On Epic-Sounds, Reactive SFT’s high

R_{S1}

reflects domain-mismatch conservatism (halved

R_{I1}

), not proactive competence. PALLM is the only method maintaining high

R_{I1}

while achieving the best F1. S2 is excluded on Epic-Sounds (no conversation history).

Figure 3: Streaming evaluation on 400 samples across 50 sound classes. Each column represents a 1-second interval; Green region (5-10s) marks when the relevant audio is present. PALLM triggers interrupts within 3.5s of event onset on average.

Proactive Assistance

Proactive assistance studies how AI systems can anticipate users’ needs and decide what help to offer, when to offer it, or whether to stay silent.

See all 46 papers

Audio Interaction Model
Jun 3, 2026Zhifei Xie, Zihang Liu, Ze An +8Audio Understanding Audio-Language Models
1NTU · 2NUS · 3CUHK
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA
AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models
Sep 30, 2026Pooneh Mousavi, Amir Ivry, Mirco Ravanelli +1Audio-Language Models Prompt Tuning
Concordia University · Mila – Quebec AI Institute · Technion – IIT +1

Jun 3, 2026J/K move · Enter open · S save

Audio Interaction Model

Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu+7

1NTU · 2NUS · 3CUHK

Audio Understanding Audio-Language Models Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.

Abstract

Proactive Assistance

Explore similar work

Audio Interaction Model

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models