What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA

Jinhwan Seo; Kyubeom Han; Jumin Lee; Junhyug Noh; Sung-eui Yoon

cs.CVAug 16, 2026

What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA

Authors: Jinhwan Seo, Kyubeom Han, Jumin Lee, Junhyug Noh, Sung-eui Yoon

Organizations: KAIST · Ewha Womans University

Abstract

We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.

Figures & tables

Metric	NG+ [ 31 ]	CRA-GQA [ 3 ]	GroundFormer	Ground Truth
PIoU $\downarrow$	85.7	88.9	28.2	21.3
GT–Pred Corr. $\uparrow$	0.005	0.006	0.121	–

Table 1: Question-invariant grounding analysis on NExT-GQA. Lower PIoU indicates less collapse across questions; higher GT–Pred Corr. indicates better agreement with the ground-truth pairwise overlap structure.

Figure 2: Overview of GroundFormer. Left: Prior methods [ 31 , 3 , 16 ] weakly inject question information into the grounding module, often collapsing different questions to the same salient interval. Right: GroundFormer conditions video features on question semantics via communication tokens ( Linguistic Transfer $\rightarrow$ Visual Refinement ) and couples answer selection with temporal grounding using MIL Cross-Attention (token queries, video keys/values), producing question-discriminative segments.

Figure 3: Details of GroundFormer. (1) Linguistic Transfer: communication tokens $z$ attend to language tokens $x_{l}$ under an asymmetric mask $M_{l}$ to absorb candidate semantics. (2) Visual Refinement: video tokens $x_{v}$ attend to the distilled tokens $\bar{z}$ under $M_{v}$ , producing question-conditioned features $\tilde{x}_{v}$ while keeping tokens uncontaminated. (3) MIL Cross-Attention: token queries attend to video keys/values to obtain attention logits, which are factorized into candidate and temporal distributions ( $\tilde{\ell}_{cand}$ , $\tilde{\ell}_{time}$ ) whose product yields MIL scores and an implicit grounding signal. The model is trained with losses for question matching (1st pass) and answer prediction (2nd pass), together with type and multi-modal contrastive objectives.

Arch.	Method	Param.	Acc@GQA	Acc@VQA	mIoP	[email protected]	[email protected]	mIoU	[email protected]	[email protected]
Temp[CLIP] [ 31 ]	PH [ 31 ]	130M	15.2	59.4	25.4	28.2	25.5	6.6	9.3	4.1
	NG+ [ 31 ]	130M	16.0	60.2	25.7	31.4	25.5	12.1	17.5	8.9
	TimeCraft [ 16 ]	130M	18.2	65.6	28.1	35.1	27.8	15.6	21.2	9.6
	CRA-GQA [ 3 ]	145M	18.2	61.1	28.6	34.3	28.5	14.2	21.4	10.6
FrozenBiLM [ 34 ]	PH [ 31 ]	1.2B	15.8	69.1	22.7	25.8	22.1	7.1	10.0	4.4
	NG+ [ 31 ]	1.2B	17.5	70.8	24.2	28.5	23.7	9.6	13.5	6.1

Table 2: Comparison with state-of-the-art methods on NExT-GQA [ 31 ] .

Figure 4: Qualitative comparison on NExT-GQA. For each video, we show three questions (Q1–Q3) with their answers and the predicted grounding intervals, along with the ground-truth segments (bottom). GroundFormer produces distinct, question-conditioned temporal segments that follow the ground-truth ordering, whereas prior methods collapse different questions onto similar intervals.

Arch.	Method	Acc@GQA	Acc@VQA	[email protected]	[email protected]
TempCLIP [ 31 ]	NG+ [ 31 ]	24.4	57.3	41.4	4.7
TempCLIP [ 31 ]	CRA-GQA [ 3 ]	26.8	58.6	44.5	5.5
FrozenBiLM [ 34 ]	NG+ [ 31 ]	25.8	60.1	40.9	7.8
FrozenBiLM [ 34 ]	CRA-GQA [ 3 ]	27.8	60.5	43.1	5.1
GroundFormer		30.7	61.1	49.3	15.8

Table 3: Comparison with state-of-the-art methods on STAR [ 29 ] .

Structure		Auxiliary Components				QA		Localization (IoP)			Localization (IoU)
Two-pass train	Comm. tokens	$\mathcal{L}_{type}$	$\mathcal{L}_{mcl}$	$\mathcal{G}_{\sigma}$	MIL	Acc@GQA	Acc@VQA	mIoP	[email protected]	[email protected]	mIoU	[email protected]	[email protected]
✓						14.8	58.5	26.5	33.3	24.8	14.7	21.9	10.6
✓	✓					18.6	59.8	29.6	37.0	28.6	17.0	25.8	13.0
✓	✓	✓				19.0	60.4	30.5	37.9	30.2	16.9	25.8	13.2
✓	✓	✓	✓			20.1	61.2	32.0	39.5	30.3	17.2	25.8	12.5
✓	✓	✓	✓	✓		20.8	61.2	32.4	39.3	32.0	16.5	25.0	12.0

Table 4: Ablation on NExT-GQA. We factorize design choices into structural components (two-pass training, communication tokens) and auxiliary components.

Method	Causal		Temporal
Method	Why	How	Before & After	When	Present
NG+ [ 31 ]	16.9	17.2	12.0	17.5	10.8
GroundFormer (Ours)	22.2	23.1	17.3	25.8	12.9

Table 5: Acc@GQA (%) by question type on NExT-GQA.

Figure 5: Hyperparameter sensitivity on NExT-GQA. (a) Number of learnable query tokens

N

, (b) number of Transformer layers in the GroundFormer block, and (c) Gaussian smoothing width

\sigma

. We report Acc@VQA, Acc@GQA, mIoP, and mIoU.

Figure 6: Grounding for semantically similar questions. In the same video, GroundFormer groups Q1–Q2 (outdoor weather) into an early interval and Q3–Q4 (indoor dog event) into a later interval, matching the ground-truth split. Shaded spans indicate predicted grounded segments.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: PIoU distribution on the NExT-GQA test split. Lower values indicate less question-invariant collapse.

Method	2Q (8.6%)	3Q (9.5%)	4–5Q (28.3%)	6+Q (50.5%)
NG+ [ 31 ]	83.2	89.9	87.7	87.9
CRA-GQA [ 3 ]	88.3	87.6	88.2	88.9
GroundFormer	23.7	22.0	28.7	29.6

Appendix

Table 6: PIoU by the number of questions per video on the NExT-GQA test split. Lower is better.

Structure		Aux. Comp.				QA		IoP		IoU		Question-Invariant
Two-pass train	Comm. tokens	$\mathcal{L}_{type}$	$\mathcal{L}_{mcl}$	$\mathcal{G}_{\sigma}$	MIL	Acc@GQA	Acc@VQA	mIoP	[email protected]	mIoU	[email protected]	PIoU $\downarrow$	GT–Pred Corr. $\uparrow$
✓						14.8	58.5	26.5	24.8	14.7	10.6	57.4	0.06
✓	✓					18.6	59.8	29.6	28.6	17.0	13.0	43.5	0.10
✓	✓	✓				19.0	60.4	30.5	30.2	16.9	13.2	37.9	0.10
✓	✓	✓	✓			20.1	61.2	32.0	30.3	17.2	12.5	29.4	0.12
✓	✓	✓	✓	✓		20.8	61.2	32.4	32.0	16.5	12.0	29.4	0.12

Appendix

Table 7: Ablation on NExT-GQA with question-invariant grounding metrics. PIoU: lower is better (less collapse). GT–Pred Corr.: higher is better (better agreement with the ground-truth overlap structure).

Method	Acc@GQA	Acc@VQA	mIoP	mIoU
Cross attention	14.8	58.5	26.5	14.7
TGA [ 22 ]	19.5	59.5	31.6	17.2
MIL-NCE [ 20 ]	19.1	59.4	31.4	16.7
WSTAN [ 28 ]	20.0	60.9	31.9	18.3
GroundFormer	21.5	61.7	34.0	17.5

Appendix

Table 8: Comparison with weakly supervised grounding alternatives on NExT-GQA.

Method	Question Type		Acc@GQA	Acc@VQA	[email protected]	[email protected]
NG+ [ 31 ]	Causal	Why	16.9	60.1	27.6	9.3
	Causal	How	17.2	60.3	27.6	9.4
	Temporal	Before&After	12.0	59.3	19.5	9.0
		When	17.5	63.0	27.0	9.4
		Present	10.8	60.3	19.4	4.3
GroundFormer	Causal	Why	22.2	61.9	35.6	13.1

Appendix

Table 9: Per-type performance on NExT-GQA [ 31 ] .

Method	Question Type	Acc@GQA	Acc@VQA	[email protected]	[email protected]
NG+ [ 31 ]	Interaction	12.5	52.3	23.6	5.6
	Sequence	32.0	59.5	53.2	4.2
	Prediction	33.2	61.5	51.9	5.3
	Feasibility	16.1	61.5	28.4	4.2
CRA-GQA [ 3 ]	Interaction	14.1	53.3	25.5	7.5
	Sequence	34.9	60.9	57.3	3.8

Appendix

Table 10: Per-type performance on STAR [ 29 ] .

Figure 8: Additional qualitative result on NExT-GQA. GroundFormer produces distinct, question-specific grounding intervals within the same video.

Figure 9: Additional qualitative result on NExT-GQA. GroundFormer preserves temporal ordering across questions and avoids collapsing multiple questions to a single segment.

Figure 10: Grounding consistency for semantically related questions within a video: questions about the same event receive overlapping intervals, while different events are separated.

Figure 11: Grounding consistency on a longer video: early-event questions (baby/toys) and late-event questions (boy/crib) are localized to distinct temporal regions.

Component	Hyperparameter	Value
Encoders	Vision encoder	CLIP ViT-L/14
	Vision dim. ( $D_{V}$ )	768
	Language encoder	RoBERTa-base
	Language dim. ( $D_{L}$ )	768
	Projection dim. ( $D$ )	768
Video sampling	Frames ( $T$ )	32 (uniform)

Appendix

Table 11: Implementation configuration and hyperparameters.

Figure 12: Failure case: repeated actions with evidence appearing in multiple disjoint intervals.

Figure 13: Failure case: fine-grained motion cues ( e.g. , deceleration) under sparse temporal sampling.

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 180 papers

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding
Oct 18, 2026Cheng-Han Lee, Amir Salimi, Hsiang-Wei Huang +3Temporal Video Grounding Video QA
The University of Texas at Austin, United States · Amazon , United States
EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
Jun 23, 2026Linpeng Huang, Weixing Chen, Zexin Chen +2VLM Evaluation Temporal Video Grounding
Sun Yat-sen University · Shenzhen University · Peng Cheng Laboratory
Diffusion-grounded VideoLLM for Entity-aware temoporal grounding
Aug 21, 2025Pengcheng Fang, Yuxia Chen, Yong Wang +2Temporal Video Grounding Video-Language Models

Oct 18, 2026J/K move · Enter open · S save

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding

Cheng-Han Lee, Amir Salimi, Hsiang-Wei Huang, Che-Chun Su+2

The University of Texas at Austin, United States · Amazon , United States

Temporal Video Grounding Video QA Zero-Shot Learning

Open paper AskPDF arXiv

Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.

Abstract

Video QA

Explore similar work

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

Diffusion-grounded VideoLLM for Entity-aware temoporal grounding