Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models

Jaehong Myung; Seunghun Oh; Unsang Park

cs.CVOct 18, 2026

Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models

Authors: Jaehong Myung, Seunghun Oh, Unsang Park

Organizations: Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Sogang University, Seoul, Republic of Korea

Abstract

Audio-visual large language models (AV-LLMs) can fail at full-video speech-vision decisions even when useful visual evidence is locally recoverable. We introduce Temporal Evidence Decomposition (TED), a training-free diagnostic framework that separates local visual verification from speech-relative aggregation. Automatic speech recognition supplies a transcript and utterance interval. TED queries audio-free windows with the transcript for Temporal Alignment and Forecasting, or answer candidates for Cross-Modal Temporal Binding, then aggregates evidence relative to that interval. Across four frozen AV-LLMs on SVHalluc, Temporal Forecasting accuracy rises from 29.3-33.7% under full-video questioning to 79.3-82.2% with TED. Query replacement reduces recovery; the observed anchor remains beneficial when temporal-region availability is held fixed. Physical-isolation and timing-shortcut controls test alternative explanations. The findings distinguish locally recoverable evidence from successful full-video decisions, motivating explicit separation of narration content, utterance timing, and time-indexed visual evidence.

Figures & tables

Figure 1: Illustrative Qwen3 traces from SVHalluc. Center-frame thumbnails represent full audio-free windows. Task questions and transcript excerpts are shortened for display only; scoring uses the original queries. TA / TF use transcript queries; CMTB scores each answer candidate (window scores shown for C). Right-hand bars aggregate evidence relative to the ASR-derived utterance interval $S$ . Values are individual traces, not averages.

Backbone	Mode	TA				TF	CMTB
Backbone	Mode	Acc. $\uparrow$	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	Acc. $\uparrow$
Qwen3-Omni-30B [ 23 ]	Direct	47.55	49.13	91.85	64.02	29.35	55.46
Qwen3-Omni-30B [ 23 ]	TED	77.99	92.62	61.41	73.86	80.43	84.66
Qwen2.5-Omni-7B [ 22 ]	Direct	53.80	64.58	16.85	26.72	33.70	51.92
Qwen2.5-Omni-7B [ 22 ]	TED	77.45	93.16	59.24	72.43	79.35	86.14
VideoLLaMA2.1-7B-AV [ 2 ]	Direct	50.00	50.00	100.00	66.67	30.80	49.85

Table 1: Direct AV inference versus Matched TED on the three SVHalluc temporal tasks (%). TA also reports positive-class precision, recall, and F1. Bold highlights higher accuracy within each backbone.

Figure 2: Diagnostics of full-video failure and local recovery. (a) Direct prediction distributions. (b) Single-run, matched-call comparison of local clips and full-video interval prompts; temporal coverage is matched for SAL2+ and approximate for Qwen3. (c) Learned timing/window probe under source-video-grouped cross-validation. Panel (c) shows descriptive out-of-fold probe accuracies, not TED scores.

Verifier	TED	WHAT replacement		WHEN replacement
Verifier	Matched	Other	Generic	SP-Shuf.	Unrestr.- Shuf.
Qwen3	80.43	42.46±2.97	38.04	71.81±1.24	54.13±2.40
Qwen2.5	79.35	41.09±3.03	35.14	71.74±2.13	53.12±2.36
VL2.1	79.35	41.59±2.94	36.23	68.41±1.10	50.72±2.84
SAL2+	82.25	42.54±2.96	36.59	70.43±1.51	51.52±3.21
SigLIP2	73.91	41.96±1.75	40.22	66.59±1.63	50.36±2.86

Table 2: Query and anchor controls (accuracy, %). Donor-dependent entries are mean

\pm

sample SD over five mappings; others show one value. (a) Matched TED uses the target transcript and observed anchor; replacements change only WHAT or WHEN. SP-Shuf. preserves the available-region set. (b) CMTB anchor controls; C

-

S is Correct minus Shuffled (pp).

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation
Oct 2, 2026Hyunjae Ra, Aecheon Jung, Jungin Park +1Multimodal Hallucination Hallucination Mitigation
Sungkyunkwan University · Daegu Gyeongbuk Institute of Science and Technology (DGIST)
SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
May 31, 2026Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu +1Audio-Visual Understanding Multimodal Large Language Models
KAIST
When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1

Oct 2, 2026J/K move · Enter open · S save

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong

Sungkyunkwan University · Daegu Gyeongbuk Institute of Science and Technology (DGIST)

Multimodal Hallucination Hallucination Mitigation Multimodal Large Language Models

Open paper AskPDF arXiv

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.

Abstract

Audio-Visual Understanding

Explore similar work

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

When Vision Speaks for Sound