ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark

Joel Julin; Souraja Kundu; Liza Dahiya; George Z. Wei; Liuyue Xie; Jingru Yi; Xu Zhang; Hao Yang

cs.CVOct 16, 2026

ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark

Authors: Joel Julin, Souraja Kundu, Liza Dahiya, George Z. Wei, Liuyue Xie, Jingru Yi, Xu Zhang, Hao Yang,

Organizations: Carnegie Mellon University · Amazon

Abstract

Multimodal large language models (MLLMs) perform well on audio-visual question answering, but correct answers do not necessarily imply that their stated reasoning is grounded in relevant perceptual evidence. Existing benchmarks mainly evaluate final-answer accuracy, offering limited insight into whether model-generated rationales correctly identify visual regions, temporal intervals, and audio events supporting predictions. We introduce ANCHOR, a benchmark and evaluation framework for measuring grounded-rationale alignment in multimodal reasoning, comprising 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes, and salient audio cues, including a human-refined test set of 224 videos and 975 question-answer pairs. We separately measure answer correctness, hallucination, logical divergence, and conciseness, with direct spatial and temporal grounding measurements on a focused subset. Across the full benchmark, twelve contemporary MLLMs show a substantial alignment gap: models with comparable answer performance can differ markedly in whether their explanations are supported by annotated evidence. On a matched subset of 100 videos and 446 question-answer pairs, rankings remain highly stable across human-refined, Gemini-generated, and GPT-generated references (mean Pearson r=0.99), providing evidence against model-reference circularity within this subset. Grounding-cue ablations of three MLLMs on the full human-refined test set reveal a consistent modality hierarchy: removing video reduces accuracy by 38.4-41.8 points, while removing bounding boxes (3.2-7.7 points) and audio (4.2-5.6 points) causes smaller, complementary, model-dependent drops. These findings show answer-only evaluation obscures important grounding failures and position ANCHOR as a testbed for evaluating accurate and evidence-aligned audio-visual reasoning.

Figures & tables

Figure 1: Limitations in MLLM CoT reasoning. Gemini-3-Flash answers correctly but lacks perceptual grounding, while GPT-5.2 misses late audio evidence and answers incorrectly. ANCHOR exposes these failures by evaluating alignment between reasoning and sensory evidence.

	Grounding Signals				Dataset Properties
Benchmark	Vid	Aud	BB	CoT	Dom.	Type	Audio	Size	Dur.
AVQA Yang et al. (2022)	✓	✓	✗	✗	✗	R	So	57K	10
MMAU Sakshi et al. (2024)	✗	✓	✗	✗	✓	/	Sp+So+Mu	10K	/
DAVE Radevski et al. (2025)	✓	✓	✗	✗	✓	R+S	So	2K	$\leq 60$
AVHBench Sung-Bin et al. (2024)	✓	✓	✗	✗	✓	R+S	So	5.8K	5–30
Daily-Omni Zhou et al. (2025)	✓	✓	✗	✗	✗	R	Sp+So+Mu	1.2K	30–60

Table 1: Benchmark Comparison. ANCHOR is the first to integrate dense spatial grounding ( BB ) with human-verified reasoning traces ( CoT ) for evidence-linked evaluation. Grounding: Vid (video), Aud (audio), BB (bounding box). Properties: Dom. (diverse domains), Type (Real-world/Synthetic), Audio ( Sp eech, So und, Mu sic), Size (Total QA pairs), Dur. (video duration in secs).

Figure 2: ANCHOR . (Left) Taxonomy distribution across top-level categories and subcategories. (Right) Data distributions showing a long-tailed duration profile (Top) and the prevalence of standard vertical resolutions like 480p/720p (Bottom).

Figure 3: The ANCHOR Annotation Interface. Our system synchronizes: (a) Spatiotemporal View (Left) , a frame-level inspector for refining SAM3 bounding box proposals; and (b) CoT Editor (Right) , a structured suite where reasoning steps are explicitly linked to visual and acoustic evidence.

Model	Answer $\uparrow$	Halluc. $\uparrow$	Conc. $\uparrow$	Tokens $\downarrow$	Div. $\uparrow$
Qwen3-VL-30B-A3B-Instruct	3.45	1.79	3.40	287	2.50
Qwen3-omni	3.34	2.03	3.90	291	2.65
InternVL3.5	2.14	1.53	3.48	283	1.85
Llava-OneVision	1.85	1.53	4.25	202	1.58
GLM-4.6-V-Flash	3.29	1.94	2.50	309	2.46
Kimi-VL-A3B-Instruct	3.05	1.93	2.83	336	2.31

Table 2: CoT evaluation. MLLMs evaluated under truncated-video input settings. Answer correctness (Answer), faithfulness (Halluc.), and reasoning consistency (Div.) are scored on a 1–5 Likert scale by three judges ( GPT-5-nano , Grok-4.1-fast , Claude-Haiku-4.5 ), with higher scores indicating better performance. Conciseness (Conc.) is a token-based quintile score (1–5, higher = more concise) derived from average output token count (Tokens) relative to the global distribution across all models. Each metric has a distinct hue; stronger shading indicates higher scores or fewer tokens. Shading uses a shared 1–5 score scale and a shared 149–736 token range across both evaluation tables. Bold marks the best result in each column.

Model	Answer $\uparrow$	Halluc. $\uparrow$	Conc. $\uparrow$	Tokens $\downarrow$	Div. $\uparrow$
Gemini-3.1-Pro	3.49	2.49	4.80	149	2.97
Gemini-3-Flash	3.59	2.08	4.09	200	2.86
Grok-4	3.01	1.61	1.40	736	2.50
Grok-4.1-Fast-Reasoning	2.85	1.35	1.59	653	2.28
Nova-2-Pro	2.93	1.96	3.62	243	2.48
Nova-2-Lite	3.03	1.63	2.50	435	2.43

Table 3: CoT evaluation on the Human-Refined set. Closed-source MLLMs evaluated on the Human-Refined subset (224 videos, 975 QA pairs). Answer correctness (Answer), faithfulness (Halluc.), and reasoning consistency (Div.) are scored on a 1–5 Likert scale by three judges, with higher scores indicating better performance. Conciseness (Conc.) is a token-based quintile score (1–5, higher = more concise) derived from average output token count (Tokens) relative to the global token distribution. Colors and normalization match Table 2 ; stronger shading indicates higher scores or fewer tokens. Bold marks the best result in each column.

Figure 4: CoT Faithfulness Analysis. (Left) Pairwise similarity matrix of model-generated reasoning traces. (Right) Qualitative example where Claude and GPT-5.2 show high reasoning convergence ( $4.06$ ), while Grok-4.1 and Qwen-3VL provide distinct, granular grounding for the clicking sound and subsequent actions.

Setting	Visual	BB	Audio	Gemini 3-Flash	GPT 5.6-Luna	Claude Sonnet-5
Full Modality	✓	✓	✓	57.7 (–)	52.5 (–)	52.6 (–)
w/o Audio	✓	✓	✗	52.9 (-4.8)	48.3 (-4.2)	47.0 (-5.6)
w/o Bounding Boxes	✓	✗	✓	50.0 (-7.7)	48.9 (-3.6)	49.4 (-3.2)
w/o Audio & Bounding Boxes	✓	✗	✗	48.1 (-9.6)	45.0 (-7.5)	45.1 (-7.5)
Audio Only	✗	✗	✓	16.1 (-41.6)	10.7 (-41.8)	14.2 (-38.4)

Table 4: Ablation of Grounding Cues. Accuracy on the human-refined test set (

n{=}975

QA pairs), with the change from each model’s full-modality accuracy shown in parentheses. Results are judged by GPT-5-Nano .

Figure 5: Model Output Lengths and Inter-Judge Agreement. (a) CoT output length distributions across 12 evaluated MLLMs. Shorter outputs receive higher conciseness scores by construction. (b) Pairwise weighted Cohen’s $\kappa$ among the three judge models for Divergence and Hallucination. Each heatmap uses a separate color scale.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	IoU $\uparrow$	IoU $\geq$ .5 $\uparrow$	Temp. P $\uparrow$	Temp. R $\uparrow$	Temp. F1 $\uparrow$
GPT-5.6-Luna	0.459	48.0%	0.76	0.85	0.80
Claude-Haiku-4.5	0.217	10.9%	0.45	0.95	0.61

Appendix

Table 5: Direct grounding on the human-refined subset.

Panel of Figure 6 — Figure 6: Per-model inter-judge agreement measured by Cohen’s $\kappa$ . Rows denote models and columns denote evaluation dimensions: Divergence , Final Answer , and Hallucination .

Panel of Figure 7 — Figure 7: Per-model inter-judge agreement measured by Cohen’s $\kappa$ continued.

Panel of Figure 8 — Figure 8: Per-model inter-judge agreement measured by Cohen’s $\kappa$ continued.

Facet	Model	Easy	Medium	Difficult
Final Answer	ChatGPT 5.2	4.57	4.04	3.27
	Claude 4.6 Sonnet	4.11	3.47	2.96
	Gemini-3-Flash	4.50	4.10	3.38
	Grok 4.1 Fast Reasoning	4.25	3.64	2.86
	Nova-2-Lite	4.05	3.28	2.68
	Qwen3-VL	4.47	3.56	2.88

Appendix

Table 6: Model performance by facet and difficulty level. We group QA pairs into Easy , Medium , and Difficult subsets using the difficulty-labeling procedure described in Section B.5 , and report per-facet scores for each model across these splits. Conciseness is a token-based quintile score, and Tokens reports the average output token count.

Facet	Model	H	H+AI	AI
Final Answer	GPT-5.2	3.45	3.77	3.81
	Claude 4.6 Sonnet	3.18	3.37	3.40
	Gemini-3-Flash	3.44	3.83	3.88
	Grok 4.1 Fast Reasoning	2.70	3.39	3.48
	Nova-2-Lite	2.90	3.18	3.22
	Qwen3-VL	3.07	3.45	3.50

Appendix

Table 7: Model performance by facet and annotation source. We report per-facet scores for each model on the human-only ( H ), human+AI ( H+AI ), and AI-only ( AI ) subsets. Conciseness is a token-based quintile score, and Tokens reports the average output token count.

Section	Top category	Subcategory	Percentage (%)
Top-level	Information Query	–	69.26
Top-level	Sports	–	28.80
Top-level	Gaming	–	1.05
Top-level	Shopping	–	0.33
Top-level	Sentiment	–	0.22
Top-level	Egocentric Agent	–	0.19

Appendix

Table 8: Taxonomy percentages of the QA pairs from our dataset

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
Oct 20, 2026Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi +17Audio-Visual Understanding Audio-Visual QA
Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology +8
Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Sep 15, 2026Zhaoyang Wei, Zipeng Wang, Yushe Cao +10Audio-Visual Understanding Video-Language Model Evaluation
University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China
TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos
May 8, 2026Hengyi Feng, Hao Liang, Mingrui Chen +6Audio-Visual Understanding Multimodal Hallucination
University of Electronic Science and Technology of China · Peking University · Zhongguancun Academy +1

Oct 20, 2026J/K move · Enter open · S save

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi, Ziqi Ye+16

Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology+8

Audio-Visual Understanding Audio-Visual QA Multimodal Model Evaluation Temporal Video Grounding

Open paper AskPDF arXiv

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

Abstract

Multimodal Model Evaluation

Explore similar work

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos