Multimodal large language models (MLLMs) perform well on audio-visual question answering, but correct answers do not necessarily imply that their stated reasoning is grounded in relevant perceptual evidence. Existing benchmarks mainly evaluate final-answer accuracy, offering limited insight into whether model-generated rationales correctly identify visual regions, temporal intervals, and audio events supporting predictions. We introduce ANCHOR, a benchmark and evaluation framework for measuring grounded-rationale alignment in multimodal reasoning, comprising 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes, and salient audio cues, including a human-refined test set of 224 videos and 975 question-answer pairs. We separately measure answer correctness, hallucination, logical divergence, and conciseness, with direct spatial and temporal grounding measurements on a focused subset. Across the full benchmark, twelve contemporary MLLMs show a substantial alignment gap: models with comparable answer performance can differ markedly in whether their explanations are supported by annotated evidence. On a matched subset of 100 videos and 446 question-answer pairs, rankings remain highly stable across human-refined, Gemini-generated, and GPT-generated references (mean Pearson r=0.99), providing evidence against model-reference circularity within this subset. Grounding-cue ablations of three MLLMs on the full human-refined test set reveal a consistent modality hierarchy: removing video reduces accuracy by 38.4-41.8 points, while removing bounding boxes (3.2-7.7 points) and audio (4.2-5.6 points) causes smaller, complementary, model-dependent drops. These findings show answer-only evaluation obscures important grounding failures and position ANCHOR as a testbed for evaluating accurate and evidence-aligned audio-visual reasoning.
Figures & tables
Figure 1: Limitations in MLLM CoT reasoning. Gemini-3-Flash answers correctly but lacks perceptual grounding, while GPT-5.2 misses late audio evidence and answers incorrectly. ANCHOR exposes these failures by evaluating alignment between reasoning and sensory evidence.
Grounding Signals
Dataset Properties
Benchmark
Vid
Aud
BB
CoT
Dom.
Type
Audio
Size
Dur.
AVQA Yang et al. (2022)
✓
✓
✗
✗
✗
R
So
57K
10
MMAU Sakshi et al. (2024)
✗
✓
✗
✗
✓
/
Sp+So+Mu
10K
/
DAVE Radevski et al. (2025)
✓
✓
✗
✗
✓
R+S
So
2K
≤60
AVHBench Sung-Bin et al. (2024)
✓
✓
✗
✗
✓
R+S
So
5.8K
5–30
Daily-Omni Zhou et al. (2025)
✓
✓
✗
✗
✗
R
Sp+So+Mu
1.2K
30–60
Table 1: Benchmark Comparison. ANCHOR is the first to integrate dense spatial grounding ( BB ) with human-verified reasoning traces ( CoT ) for evidence-linked evaluation. Grounding: Vid (video), Aud (audio), BB (bounding box). Properties: Dom. (diverse domains), Type (Real-world/Synthetic), Audio ( Sp eech, So und, Mu sic), Size (Total QA pairs), Dur. (video duration in secs).
Figure 2: ANCHOR . (Left) Taxonomy distribution across top-level categories and subcategories. (Right) Data distributions showing a long-tailed duration profile (Top) and the prevalence of standard vertical resolutions like 480p/720p (Bottom).
Figure 3: The ANCHOR Annotation Interface. Our system synchronizes: (a) Spatiotemporal View (Left) , a frame-level inspector for refining SAM3 bounding box proposals; and (b) CoT Editor (Right) , a structured suite where reasoning steps are explicitly linked to visual and acoustic evidence.
Model
Answer ↑
Halluc. ↑
Conc. ↑
Tokens ↓
Div. ↑
Qwen3-VL-30B-A3B-Instruct
3.45
1.79
3.40
287
2.50
Qwen3-omni
3.34
2.03
3.90
291
2.65
InternVL3.5
2.14
1.53
3.48
283
1.85
Llava-OneVision
1.85
1.53
4.25
202
1.58
GLM-4.6-V-Flash
3.29
1.94
2.50
309
2.46
Kimi-VL-A3B-Instruct
3.05
1.93
2.83
336
2.31
Table 2: CoT evaluation. MLLMs evaluated under truncated-video input settings. Answer correctness (Answer), faithfulness (Halluc.), and reasoning consistency (Div.) are scored on a 1–5 Likert scale by three judges ( GPT-5-nano , Grok-4.1-fast , Claude-Haiku-4.5 ), with higher scores indicating better performance. Conciseness (Conc.) is a token-based quintile score (1–5, higher = more concise) derived from average output token count (Tokens) relative to the global distribution across all models. Each metric has a distinct hue; stronger shading indicates higher scores or fewer tokens. Shading uses a shared 1–5 score scale and a shared 149–736 token range across both evaluation tables. Bold marks the best result in each column.
Model
Answer ↑
Halluc. ↑
Conc. ↑
Tokens ↓
Div. ↑
Gemini-3.1-Pro
3.49
2.49
4.80
149
2.97
Gemini-3-Flash
3.59
2.08
4.09
200
2.86
Grok-4
3.01
1.61
1.40
736
2.50
Grok-4.1-Fast-Reasoning
2.85
1.35
1.59
653
2.28
Nova-2-Pro
2.93
1.96
3.62
243
2.48
Nova-2-Lite
3.03
1.63
2.50
435
2.43
Table 3: CoT evaluation on the Human-Refined set. Closed-source MLLMs evaluated on the Human-Refined subset (224 videos, 975 QA pairs). Answer correctness (Answer), faithfulness (Halluc.), and reasoning consistency (Div.) are scored on a 1–5 Likert scale by three judges, with higher scores indicating better performance. Conciseness (Conc.) is a token-based quintile score (1–5, higher = more concise) derived from average output token count (Tokens) relative to the global token distribution. Colors and normalization match Table 2 ; stronger shading indicates higher scores or fewer tokens. Bold marks the best result in each column.
Figure 4: CoT Faithfulness Analysis. (Left) Pairwise similarity matrix of model-generated reasoning traces. (Right) Qualitative example where Claude and GPT-5.2 show high reasoning convergence ( 4.06 ), while Grok-4.1 and Qwen-3VL provide distinct, granular grounding for the clicking sound and subsequent actions.
Setting
Visual
BB
Audio
Gemini 3-Flash
GPT 5.6-Luna
Claude Sonnet-5
Full Modality
✓
✓
✓
57.7 (–)
52.5 (–)
52.6 (–)
w/o Audio
✓
✓
✗
52.9 (-4.8)
48.3 (-4.2)
47.0 (-5.6)
w/o Bounding Boxes
✓
✗
✓
50.0 (-7.7)
48.9 (-3.6)
49.4 (-3.2)
w/o Audio & Bounding Boxes
✓
✗
✗
48.1 (-9.6)
45.0 (-7.5)
45.1 (-7.5)
Audio Only
✗
✗
✓
16.1 (-41.6)
10.7 (-41.8)
14.2 (-38.4)
Table 4: Ablation of Grounding Cues. Accuracy on the human-refined test set ( n=975 QA pairs), with the change from each model’s full-modality accuracy shown in parentheses. Results are judged by GPT-5-Nano .
Figure 5: Model Output Lengths and Inter-Judge Agreement. (a) CoT output length distributions across 12 evaluated MLLMs. Shorter outputs receive higher conciseness scores by construction. (b) Pairwise weighted Cohen’s κ among the three judge models for Divergence and Hallucination. Each heatmap uses a separate color scale.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
IoU ↑
IoU ≥ .5 ↑
Temp. P ↑
Temp. R ↑
Temp. F1 ↑
GPT-5.6-Luna
0.459
48.0%
0.76
0.85
0.80
Claude-Haiku-4.5
0.217
10.9%
0.45
0.95
0.61
Appendix
Table 5: Direct grounding on the human-refined subset.
Figure 6: Per-model inter-judge agreement measured by Cohen’s κ . Rows denote models and columns denote evaluation dimensions: Divergence , Final Answer , and Hallucination .
Figure 7: Per-model inter-judge agreement measured by Cohen’s κ continued.
Figure 8: Per-model inter-judge agreement measured by Cohen’s κ continued.
Facet
Model
Easy
Medium
Difficult
Final Answer
ChatGPT 5.2
4.57
4.04
3.27
Claude 4.6 Sonnet
4.11
3.47
2.96
Gemini-3-Flash
4.50
4.10
3.38
Grok 4.1 Fast Reasoning
4.25
3.64
2.86
Nova-2-Lite
4.05
3.28
2.68
Qwen3-VL
4.47
3.56
2.88
Appendix
Table 6: Model performance by facet and difficulty level. We group QA pairs into Easy , Medium , and Difficult subsets using the difficulty-labeling procedure described in Section B.5 , and report per-facet scores for each model across these splits. Conciseness is a token-based quintile score, and Tokens reports the average output token count.
Facet
Model
H
H+AI
AI
Final Answer
GPT-5.2
3.45
3.77
3.81
Claude 4.6 Sonnet
3.18
3.37
3.40
Gemini-3-Flash
3.44
3.83
3.88
Grok 4.1 Fast Reasoning
2.70
3.39
3.48
Nova-2-Lite
2.90
3.18
3.22
Qwen3-VL
3.07
3.45
3.50
Appendix
Table 7: Model performance by facet and annotation source. We report per-facet scores for each model on the human-only ( H ), human+AI ( H+AI ), and AI-only ( AI ) subsets. Conciseness is a token-based quintile score, and Tokens reports the average output token count.
Section
Top category
Subcategory
Percentage (%)
Top-level
Information Query
–
69.26
Top-level
Sports
–
28.80
Top-level
Gaming
–
1.05
Top-level
Shopping
–
0.33
Top-level
Sentiment
–
0.22
Top-level
Egocentric Agent
–
0.19
Appendix
Table 8: Taxonomy percentages of the QA pairs from our dataset
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.
Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi +17
Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology +8
Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.
Zhaoyang Wei, Zipeng Wang, Yushe Cao +10
University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China
Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams, whereas existing benchmarks largely fail to evaluate this capability. They restrict videos to short clips, isolate modalities, or reduce questions to one-hop perception. We introduce TraceAV-Bench, the first benchmark to jointly evaluate multi-hop reasoning over long audio-visual trajectories and multimodal hallucination robustness. TraceAV-Bench comprises 2,200 rigorously validated multiple-choice questions over 578 long videos, totaling 339.5 hours, spanning 4 evaluation dimensions and 15 sub-tasks. Each question is grounded in an explicit reasoning chain that averages 3.68 hops across a 15.1-minute temporal span. The dataset is built by a three-step semi-automated pipeline followed by a strict quality assurance process. Evaluation of multiple representative OmniLLMs on TraceAV-Bench reveals that the benchmark poses a persistent challenge across all models, with the strongest closed-source model (Gemini 3.1 Pro) reaching only 68.29% on general tasks, and the best open-source model (Ming-Flash-Omni-2.0) reaching 51.70%, leaving substantial headroom. Moreover, we find that robustness to multimodal hallucination is largely decoupled from general multimodal reasoning performance. We anticipate that TraceAV-Bench will stimulate further research toward OmniLLMs that can reason coherently and faithfully over long-form audio-visual content.
Hengyi Feng, Hao Liang, Mingrui Chen +6
University of Electronic Science and Technology of China · Peking University · Zhongguancun Academy +1