RVG-Bench: A Benchmark for Reasoning-Driven Audio-Visual Temporal Grounding

Yue Feng; Boshen Xu; Zihao Yue; Xiaokun Liu; Miao Yang; Xin Tao; Pengfei Wan; Qin Jin

cs.CVOct 23, 2026

RVG-Bench: A Benchmark for Reasoning-Driven Audio-Visual Temporal Grounding

Authors: Yue Feng, Boshen Xu, Zihao Yue, Xiaokun Liu, Miao Yang, Xin Tao, Pengfei Wan, Qin Jin

Organizations: AIM3 Lab, Renmin University of China · Kling AI

Abstract

Existing temporal video grounding (TVG) benchmarks primarily focus on localizing target video segments according to descriptive text queries of the visual content. However, real-world queries are more diverse and often require the model to first understand the user's intent before localization, such as finding visual evidence, identifying the cause of an event, or recalling a remembered moment, yet such queries remain largely overlooked. To mitigate this gap, we introduce RVG-Bench, a benchmark for reasoning-driven audio-visual temporal grounding, where every query is constructed such that the target segment can only be identified through reasoning about the query's underlying intent. We carefully construct the benchmark and rigorously verify its data quality to ensure models truly comprehend the audio-visual content. RVG-Bench comprises 1,255 videos and 1,890 query-interval pairs across eight reasoning types. We evaluate 22 models and find that the strongest, Qwen3.8-Omni-Flash, achieves only 49.6 mIoU. Through a series of controlled analyses, including oracle-grounding probes and test-time scaling experiments, we systematically examine where and why current models struggle, providing insights for future model development.

Figures & tables

Figure 1: From descriptive grounding to reasoning-driven grounding. Left: Traditional TVG queries directly describe the target segment, making localization largely solvable via cross-modal semantic matching. Right: RVG-Bench uses indirect, intent-driven queries that require reasoning to infer the correct target interval, such as identifying the causal event that leads to a later outcome, and often rely on both visual and audio evidence.

Major Category	Sub- category	Definition
Intrinsic Attribute	Char	Identify a segment through a character’s implicit mental state or behavioral pattern.
	Anom	Locate a segment that deviates from the expected norm of surrounding context.
	Symb	Interpret abstract or metaphorical meaning to identify the target.
Relative Attribute	Comp	Select a segment by comparing a measurable property across candidates.
	Ord	Locate a segment by its position in a sequence.
Relational	Caus	Trace from an effect to its cause, or vice versa.

Table 1: Compact reasoning taxonomy of RVG-Bench. Each subcategory is defined by how the query indirectly specifies the target. Complete definitions and example queries are provided in Appendix 0.A.2 .

Figure 2: Overview of the RVG-Bench construction pipeline: (1) broad query proposal via multi-view captions; (2) rigorous quality filtering and target interval refinement using frontier MLLMs; (3) difficulty-aware sampling; and (4) full human validation with low-quality sample removal, question rewriting, and GT boundary refinement.

Model	Input	S-Step									M-Step	Overall
Model	Input	Caus	Seq	Ref	Char	Anom	Symb	Comp	Ord	Overall	mIoU	mIoU	[email protected]
API-based models
Qwen3.8-Omni-Flash	AV	38.5	71.1	47.1	41.9	43.8	58.6	62.9	51.9	51.3	40.6	49.6	68.0
Seed2.0-Lite	AV	36.8	64.2	50.2	40.0	44.6	56.2	66.7	42.7	49.0	36.9	47.0	62.6
Gemini-3-Pro	AV	31.3	63.5	44.4	34.6	41.4	51.0	68.3	45.1	46.1	35.8	44.4	61.1
Seed2.0-Pro	V	28.9	57.9	42.3	36.4	40.8	48.8	58.6	36.8	42.6	38.0	41.9	58.2

Table 2: Main results on RVG-Bench. Input denotes visual (V) or audio-visual (AV) input. S-Step and M-Step denote single- and multi-step reasoning. Except for [email protected] , all numbers are mIoU percentages. Overall scores are sample-count-weighted. Additional recall results appear in Table 12 .

Model	Vis-Req	Aud-Req	Both-Req
ChronusOmni	25.6 $\rightarrow$ 23.8	23.7 $\rightarrow$ 18.2	21.1 $\rightarrow$ 18.3
Gemini-3-Pro	44.0 $\rightarrow$ 46.2	53.1 $\rightarrow$ 24.9	42.5 $\rightarrow$ 33.7
Qwen3-Omni	8.8 $\rightarrow$ 10.9	15.9 $\rightarrow$ 10.0	11.2 $\rightarrow$ 10.2

Table 3: Compact modality ablation on RVG-Bench. Entries show A+V

\rightarrow

V-only mIoU over all single- and multi-step queries in the evaluation subset.

Panel of Figure 3 — Figure 3: Compact results for representative model-analysis settings. In (a), subscripts denote Ground $-$ Default. Complete Oracle-grounding and task-decomposition results are reported in Tabs. 10 and 11 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	#Videos	Avg. len (s)	#Queries	Audio	Rel	Attr	Multi-step
Charades-STA [ 24 ]	1,334	29.5	3,720	✗	✗	✗	✗
ActivityNet Caps [ 17 ]	4,885	118.1	17K	✗	✗	✗	✗
QVHighlights [ 18 ]	1,529	149.6	1,542	✗	✗	✗	✗
LongVALE [ 11 ]	1,171	235.0	14K	✓	✗	✗	✗
ChronusAV [ 7 ]	2,000	226.0	12K	✓	✗	✗	✗
ToTG-Bench [ 20 ]	–	–	–	✗	✓	✗	✗

Appendix

Table 4: Comparison with representative TVG benchmarks. For datasets that provide both training and test splits, we report statistics on the test split only . Audio indicates whether queries require audio information. Rel indicates relation reasoning. Attr indicates attribute reasoning, including both intrinsic attributes and relative attributes. Multi-step indicates whether the benchmark contains multi-step reasoning queries. The symbol “–” indicates that the specific statistics are neither reported in their original paper nor available because the dataset is not open source.

Major Category	Sub- category	Entry	Content
Intrinsic Attribute	Char	Definition	Identify a segment through a character’s implicit mental state or behavioral pattern.
		Example	“Which segment shows the girl’s hesitation?”
	Anom	Definition	Locate a segment that deviates from the expected norm of surrounding context.
		Example	“Locate the abnormal moment.”
	Symb	Definition	Interpret abstract or metaphorical meaning to identify the target.
		Example	“Locate the scene that symbolizes freedom.”

Appendix

Table 5: Full reasoning taxonomy of RVG-Bench. Each subcategory is presented in two rows: one defining how the query indirectly specifies the target and one providing an example query. Subcategory abbreviations follow Tab. 1 .

Panel of Figure 4 — Figure 4: Statistics of RVG-Bench, including video duration, ground-truth (GT) interval length, reasoning-category distribution with required modality, and normalized temporal distributions of the center, start, and end points of GT intervals.

Criterion / outcome	Count	Rate
Initial criterion-wise assessment ( $n=2{,}500$ )
Answerability	2,091	83.64%
Uniqueness	2,447	97.88%
Logical soundness	2,045	81.80%
No information leakage	2,480	99.20%
Final selection ( $n=2{,}500$ )

Appendix

Table 6: Initial criterion-wise pass rates and final validation outcomes. Counts in the first block denote passed samples. The denominator for each block is shown explicitly.

Reference	$N$	mIoU	@0.3	@0.5	@0.7
Human baseline (vs. GT)	200	65.3	78.5	74.0	55.5
Human–human agreement	200	73.7	80.0	77.5	60.0

Appendix

Table 7: Human temporal-grounding results on the 200-example double-annotated audit subset. The human baseline averages the two annotators’ IoU with GT per query. Human–human agreement compares the two annotators directly. All metric values are percentages.

Figure 6: Examples of Relational reasoning. The figure encompasses three cases covering the Causal , Sequential , and Referential subcategories. Success in these tasks hinges on the model’s ability to navigate structured dependencies within the video.

Figure 7: Examples of Intrinsic-attribute reasoning. The figure includes three distinct cases corresponding to the Character-centric , Contextual Anomaly , and Symbolic subcategories. Models must reason over the implicit meaning or context of the video segments rather than observable surface features.

Figure 8: Examples of Relative-attribute reasoning. The two cases represent the Comparative and Ordinal subcategories, requiring the model to evaluate and rank multiple candidate segments across the entire video timeline.

Figure 9: An example of Multi-step reasoning. The model first uses Causal reasoning to infer the supernatural duplication mechanism, then detects a Contextual Anomaly when the protagonist discovers that it has failed to produce the expected result for her husband. The target is the moment when she discovers the empty closet.

Benchmark	Gemini	TimeLens-8B	Gap (pts)	Rel. gap
Charades-TimeLens [ 35 ]	52.8	55.2	$-2.4$	$-4.3\%$
ActivityNet-TimeLens [ 35 ]	58.1	53.2	$+4.9$	$+9.2\%$
QVHighlights-TimeLens [ 35 ]	70.4	65.5	$+4.9$	$+7.5\%$
RVG-Bench	44.4	30.1	+14.3	+47.5%

Appendix

Table 8: Comparison of reported results on TimeLens-optimized benchmarks from [ 35 ] with RVG-Bench. Gemini denotes Gemini-2.5-Pro for the first three benchmarks and Gemini-3-Pro for RVG-Bench. Gap (pts) reports the mIoU difference (Gemini minus TimeLens-8B) in percentage points. Rel. gap expresses this difference as a percentage of the TimeLens-8B score.

Model	Input	Traditional	RVG-Bench
		Charades-TimeLens	Subset by Required Modality			Overall
		Overall	Vis-Req	Aud-Req	Both-Req	Overall
ChronusOmni	A+V	44.1	25.6	23.7	21.1	23.4
	V-only	41.5	23.8	18.2	18.3	20.8
	A-only	19.1	12.0	16.6	16.6	14.5
Gemini-3-Pro	A+V	52.7	44.0	53.1	42.5	44.4

Appendix

Table 9: Modality ablation results on the traditional TVG benchmark Charades-TimeLens and RVG-Bench. We report mIoU under the A+V, V-only, and A-only input settings. RVG-Bench results cover all single- and multi-step queries in the evaluation subset and are further divided into the Vis-Req, Aud-Req, and Both-Req subsets, along with the overall score.

Model	S-Step		M-Step		Overall
Model	Default	Ground	Default	Ground	Default	Ground
Qwen3.8-Omni-Flash	51.3	60.7	40.6	50.8	49.6	59.1 +9.5
Seed2.0-Lite	49.0	59.8	36.9	46.3	47.0	57.6 +10.6
Gemini-3-Pro	46.1	65.7	35.8	54.7	44.4	63.9 +19.5
Seed2.0-Pro	42.6	57.7	38.0	51.0	41.9	56.6 +14.7
GPT-5.5	39.5	56.4	32.8	40.5	38.5	53.6 +15.1

Appendix

Table 10: Default and oracle-grounding mIoU. Ground subscripts under Overall denote gains over Default. The final row is the unweighted mean across the 22 listed models.

Model	S-Step			M-Step			Overall
Model	Direct	CoT	2-step	Direct	CoT	2-step	Direct	CoT	2-step
Qwen3.8-Omni-Flash	48.8	51.3	46.0	37.4	40.6	37.4	47.0	49.6	44.6
Seed2.0-Lite	48.2	49.0	46.4	38.3	36.9	37.7	46.5	47.0	44.9
Gemini-3-Pro	44.1	46.1	51.1	39.0	35.8	43.2	43.2	44.4	49.8
Seed2.0-Pro	42.9	42.6	42.1	38.0	38.0	36.2	42.1	41.9	41.1
GPT-5.5	38.6	39.5	38.9	36.8	32.8	36.4	38.4	38.5	38.6

Appendix

Table 11: mIoU under Direct, CoT, and 2-step prompting on matched queries within each model. The final row is the unweighted mean across the 15 listed models.

Model	mIoU	[email protected]	[email protected]	[email protected]
Qwen3.8-Omni-Flash	49.6	68.0	51.8	33.4
Seed2.0-Lite	47.0	62.6	49.2	30.5
Gemini-3-Pro	44.4	61.1	46.1	27.8
Seed2.0-Pro	41.9	58.2	44.8	27.1
GPT-5.5	38.5	55.5	39.5	21.1
Qwen3.8-Flash	38.2	52.8	39.1	24.5

Appendix

Table 12: Default-CoT performance at progressively stricter temporal-overlap thresholds. [email protected], [email protected], and [email protected] are percentages.

Figure 10: Best-of- $N$ self-verification on the full evaluation set, where $N$ is the number of candidate predictions per query. The $N=1$ scores match the CoT results in Tab. 2 .

Evaluated model	U	P	R	L
Qwen3.8-Omni-Flash	3.2	13.8	16.9	34.6
Qwen3.8-Flash	2.2	20.9	24.9	33.2
Seed2.0-Lite	7.0	16.7	22.5	38.6
Seed2.0-Pro	5.9	15.5	28.9	29.0

Appendix

Table 13: Error-label frequencies (%) assigned by GPT-5.5 to default-CoT responses. U/P/R/L denote understanding, perception, reasoning, and localization.

Evaluated model	Agreement (%)	$\kappa$
Gemini-3-Pro	80.11	0.704
Qwen3-VL-8B	70.03	0.600

Appendix

Table 14: Agreement between Gemini-3-Pro and GPT-5.5 on primary diagnostic labels across CoT, oracle grounding, and 2-step prediction. Labels comprise four error types and no error ;

\kappa

is Cohen’s kappa.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
Oct 20, 2026Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi +17Audio-Visual Understanding Audio-Visual QA
Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology +8
Temporal-Aware Reasoning Optimization for Video Temporal Grounding
Jun 8, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding Temporal Reasoning
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.
Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Sep 15, 2026Zhaoyang Wei, Zipeng Wang, Yushe Cao +10Audio-Visual Understanding Video-Language Model Evaluation
University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China

Oct 20, 2026J/K move · Enter open · S save

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi, Ziqi Ye+16

Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology+8

Audio-Visual Understanding Audio-Visual QA Multimodal Model Evaluation Temporal Video Grounding

Open paper AskPDF arXiv

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

Abstract

Temporal Video Grounding

Explore similar work

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Temporal-Aware Reasoning Optimization for Video Temporal Grounding

Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?