Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

Xize Cheng; Wenxu Jia; Chenyuhao Wen; Dongjie Fu; Zehan Wang; Xinyu Zhang; Tao Jin

cs.CLSep 18, 2026

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

Authors: Xize Cheng, Wenxu Jia, Chenyuhao Wen, Dongjie Fu, Zehan Wang, Xinyu Zhang, Tao Jin

Organizations: Zhejiang University

Abstract

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to preserve both semantic content and acoustic cues. To address this challenge, we introduce Vox-Infinity, the first benchmark specifically designed to evaluate long-context understanding in spoken language models. Vox-Infinity systematically extends audio history along two dimensions: turn count and turn duration. It covers a diverse range of representative scenarios with varying interaction structures and semantic complexity. Crucially, Vox-Infinity provides explicit answer-provenance annotations and organizes samples according to the amount of historical context required to resolve each query, enabling precise and length-aware evaluation. Extensive evaluations of seven representative spoken language models reveal a clear overall recency effect: models generally achieve higher accuracy when answer-supporting evidence is closer to the query, but struggle to retrieve and use evidence located farther back in the dialogue history. Cases and datasets are available at https://vox-infinity.github.io.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of three representative dialogue scenarios in Vox-Infinity . (a) Ultra Multi-Turn Dialogue : frequent back-and-forth exchanges requiring fine-grained information retrieval. (b) Personal Monologue : single-speaker long-form speech requiring reasoning over distributed evidence. (c) Beyond-Semantic Dialogue : extended conversations that require models to retain and retrieve emotions conveyed through vocal expression.

Panel of Figure 2 — Figure 2: Statistics and characteristics of Vox-Infinity . (a) Dataset-scale comparison by average turns per session and average complete session duration; marker size encodes total hours, and stars denote Vox-Infinity and its three subsets. (b) Distribution of required history durations using 50-second bins and a logarithmic count axis. (c) Percentage of QA samples in the 0–2, 2–4, 4–8, and 8+ minute required-history-duration bins. (d) Word cloud of common topics.

Benchmarks	Session	QAs	Dur.(h)	Avg. Tok. †	Avg. Dur.(s)	Avg. Trns.	Prov.	Length Distribution ‡
Long Context Benchmark (Textual)
LongBench ( Bai et al., 2023b )	4,550	4,550	-	8.1k	-	1.00	✗	0-4k, 4k-8k, 8k+
$\infty$ BENCH ( Zhang et al., 2024c )	3,946	3,946	-	200k	-	1.00	✗	0-200k
LongBench-V2 ( Bai et al., 2024 )	503	503	-	54k	-	1.00	✗	0-32k, 32k-128k, 128k+
Long Context Benchmark (Audio)
VoxDialogue ( Cheng et al., 2025 )	4,526	4,526	42.56	0.8k	33.9	6.70	✗	0-2min

Table 1: Comparison of existing long-context benchmarks across textual and audio modalities. Avg. Tok. denotes the average number of tokens per session, Avg.Dur. denotes the average duration per session, and Avg.Trns. denotes the average number of turns per session. Prov. indicates whether answer provenance annotations are available. Length distribution refers to the context-length ranges used for evaluation; ‡ for datasets without explicit definitions, we adopt the near-upper quartile of context length as the upper bound. For audio benchmarks, average duration is calculated as total duration divided by the number of sessions; † token counts are estimated assuming 25 tokens per second.

Text-as-history
	Ultra Multi-Turn				Personal Monologues					Beyond-Semantic
Model	0–2m	2–4m	4–8m	Avg.	0–2m	2–4m	4–8m	8m+	Avg.	0–2m	2–4m	4–8m	8m+	Avg.
Qwen3-Omni	85.5	84.2	77.0	84.8	84.2	86.8	84.7	85.5	85.5	49.5	46.2	42.4	25.0	46.2
GLM-4-Voice	61.4	56.3	48.3	59.4	27.3	36.6	26.1	26.3	27.6	43.5	42.4	37.0	10.7	41.0
AudioFlamingo3	45.6	40.7	23.0	43.4	42.4	58.2	47.8	47.8	49.0	20.0	20.1	15.0	8.9	18.5
AudioFlamingo Next	76.1	73.2	67.0	74.9	45.5	62.7	63.0	62.6	62.2	46.7	40.6	35.1	16.1	41.2

Table 2: Accuracy of spoken language models across dialogue scenarios and required-context-duration bins. Beyond-Semantic uses multiple-choice questions, whereas Ultra Multi-Turn and Personal Monologues use open-ended questions. Here, m denotes minutes and Avg. denotes the average accuracy within each subset. All values are percentages, and † marks closed-source models.

Figure 3: Fine-grained accuracy under audio history across 30-second required-context-duration intervals on (a) Ultra Multi-Turn and (b) Beyond-Semantic Dialogues. Thin curves represent individual models, while thick gray curves show their mean within each interval. Intervals containing fewer than 20 examples are omitted, and MiMo-Audio is included only in (b) . Despite local fluctuations, both subsets exhibit an overall recency effect.

	Natural			Semantically Controlled
Model	Text	Audio	$\bm{\Delta}$	Text	Audio	$\bm{\Delta}$
Qwen3-Omni	65.3	68.3	+3.0	26.1	37.3	+11.2
GLM-4-Voice	52.9	28.4	-24.6	28.5	16.0	-12.5
AudioFlamingo3	28.9	35.1	+6.1	7.6	15.4	+7.9
AudioFlamingo Next	57.8	57.9	+0.1	23.6	34.9	+11.3
MiMo-Audio	41.4	53.8	+12.4	18.0	26.0	+8.1

Table 3: Accuracy on natural (

n=2{,}758

) and semantically controlled (

n=2{,}622

) samples from Beyond-Semantic Dialogues. Text and Audio denote the history modality, and

\Delta

reports Audio minus Text in percentage points. All values are percentages.

Qwen3-Omni
Method	Ultra Multi-Turn	Personal Monologues	Beyond-Semantic	Overall
LLM-eval	48.0	74.0	54.0	58.7
Human-eval	46.0	76.0	54.0	58.7
Agreement	98.0	98.0	100.0	98.7
Audio Flamingo Next
LLM-eval	68.0	50.0	52.0	56.7

Table 4: Agreement between LLM-based evaluation and human judgments on 300 randomly sampled audio-history outputs. Each subset cell contains 50 examples. Agreement denotes the percentage of matching labels.

Panel of Figure 4 — Figure 4: Representative examples drawn from the three subsets of Vox-Infinity .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Prompt template for QA generation.

Panel of Figure 6 — Figure 6: Standardized prompt template used for QA evaluation.

Long-Context Language Model Evaluation

Long-context language model evaluation measures how reliably models find, reason over, and use information across extended inputs.

See all 68 papers

VoiceGiraffe: A Benchmark for Extreme Long-Context Audio-Language Understanding
May 27, 2026Jashin Ye, Dongxiao Wang, Yixuan Ye +10Audio-Language Model Evaluation Audio Understanding
Future Living Lab, Alibaba
VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Sep 26, 2026Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1Audio-Language Model Evaluation Long-Term Conversational Memory
University of Melbourne · University of New South Wales
LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension
Aug 26, 2026Wen Huang, Yunfei Chu, Meng Gao +2Audio-Language Model Evaluation Audio Reasoning
Qwen Team, Alibaba Group · Tsinghua University · The Chinese University of Hong Kong

May 27, 2026J/K move · Enter open · S save

VoiceGiraffe: A Benchmark for Extreme Long-Context Audio-Language Understanding

Jashin Ye, Dongxiao Wang, Yixuan Ye, Sashuai Zhou+9

Future Living Lab, Alibaba

Audio-Language Model Evaluation Audio Understanding Audio-Language Models Long-Context Modeling

Open paper AskPDF arXiv

While large audio language models (LALMs) have achieved remarkable progress in audio processing at the second- or minute-level scale, understanding hour-level audio remains a fundamental bottleneck. Existing benchmarks predominantly rely on short clips or artificially concatenated segments, failing to faithfully assess LALM capacity for long-range information comprehension in real-world scenarios such as podcasts and lengthy speeches. To address this gap, we introduce VoiceGiraffe, a novel benchmark designed to rigorously evaluate LALMs across diverse real-world scenarios, modalities, and languages under long-context settings. It comprises 1500 curated triplets structured into a dual-level taxonomy of single-hop perception and multi-hop reasoning. We evaluate a broad suite of open-source and proprietary LALMs against human performance. Results underscore three fundamental findings. First, VoiceGiraffe remains highly challenging and far from saturation. Second, we show that no single inference paradigm universally dominates. The E2E inference benefits models with native long-context audio understanding, cascaded caption aggregation stabilizes small models overwhelmed by hour-scale audio, and reasoning-enhanced cascading with external LLM helps weaker models but can bottleneck stronger proprietary systems. Third, we reveal long-range memory persistence as a key bottleneck. LALMs are better at answering questions that require connecting salient causal cues than those requiring sustained tracking of sparse events across long audio, whereas humans show the opposite pattern. These findings position VoiceGiraffe as a challenging and diagnostic testbed for long-form audio understanding, highlighting the need for LALMs with persistent memory and robust long-range aggregation.

Abstract

Long-Context Language Model Evaluation

Explore similar work

VoiceGiraffe: A Benchmark for Extreme Long-Context Audio-Language Understanding

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension