We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.
Figures & tables
Metric
NG+ [ 31 ]
CRA-GQA [ 3 ]
GroundFormer
Ground Truth
PIoU ↓
85.7
88.9
28.2
21.3
GT–Pred Corr. ↑
0.005
0.006
0.121
–
Table 1: Question-invariant grounding analysis on NExT-GQA. Lower PIoU indicates less collapse across questions; higher GT–Pred Corr. indicates better agreement with the ground-truth pairwise overlap structure.
Figure 2: Overview of GroundFormer. Left: Prior methods [ 31 , 3 , 16 ] weakly inject question information into the grounding module, often collapsing different questions to the same salient interval. Right: GroundFormer conditions video features on question semantics via communication tokens ( Linguistic Transfer → Visual Refinement ) and couples answer selection with temporal grounding using MIL Cross-Attention (token queries, video keys/values), producing question-discriminative segments.
Figure 3: Details of GroundFormer. (1) Linguistic Transfer: communication tokens z attend to language tokens xl under an asymmetric mask Ml to absorb candidate semantics. (2) Visual Refinement: video tokens xv attend to the distilled tokens zˉ under Mv , producing question-conditioned features x~v while keeping tokens uncontaminated. (3) MIL Cross-Attention: token queries attend to video keys/values to obtain attention logits, which are factorized into candidate and temporal distributions ( ℓ~cand , ℓ~time ) whose product yields MIL scores and an implicit grounding signal. The model is trained with losses for question matching (1st pass) and answer prediction (2nd pass), together with type and multi-modal contrastive objectives.
Table 2: Comparison with state-of-the-art methods on NExT-GQA [ 31 ] .
Figure 4: Qualitative comparison on NExT-GQA. For each video, we show three questions (Q1–Q3) with their answers and the predicted grounding intervals, along with the ground-truth segments (bottom). GroundFormer produces distinct, question-conditioned temporal segments that follow the ground-truth ordering, whereas prior methods collapse different questions onto similar intervals.
Table 4: Ablation on NExT-GQA. We factorize design choices into structural components (two-pass training, communication tokens) and auxiliary components.
Method
Causal
Temporal
Why
How
Before & After
When
Present
NG+ [ 31 ]
16.9
17.2
12.0
17.5
10.8
GroundFormer (Ours)
22.2
23.1
17.3
25.8
12.9
Table 5: Acc@GQA (%) by question type on NExT-GQA.
Figure 5: Hyperparameter sensitivity on NExT-GQA. (a) Number of learnable query tokens N , (b) number of Transformer layers in the GroundFormer block, and (c) Gaussian smoothing width σ . We report Acc@VQA, Acc@GQA, mIoP, and mIoU.
Figure 6: Grounding for semantically similar questions. In the same video, GroundFormer groups Q1–Q2 (outdoor weather) into an early interval and Q3–Q4 (indoor dog event) into a later interval, matching the ground-truth split. Shaded spans indicate predicted grounded segments.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: PIoU distribution on the NExT-GQA test split. Lower values indicate less question-invariant collapse.
Method
2Q (8.6%)
3Q (9.5%)
4–5Q (28.3%)
6+Q (50.5%)
NG+ [ 31 ]
83.2
89.9
87.7
87.9
CRA-GQA [ 3 ]
88.3
87.6
88.2
88.9
GroundFormer
23.7
22.0
28.7
29.6
Appendix
Table 6: PIoU by the number of questions per video on the NExT-GQA test split. Lower is better.
Table 7: Ablation on NExT-GQA with question-invariant grounding metrics. PIoU: lower is better (less collapse). GT–Pred Corr.: higher is better (better agreement with the ground-truth overlap structure).
Method
Acc@GQA
Acc@VQA
mIoP
mIoU
Cross attention
14.8
58.5
26.5
14.7
TGA [ 22 ]
19.5
59.5
31.6
17.2
MIL-NCE [ 20 ]
19.1
59.4
31.4
16.7
WSTAN [ 28 ]
20.0
60.9
31.9
18.3
GroundFormer
21.5
61.7
34.0
17.5
Appendix
Table 8: Comparison with weakly supervised grounding alternatives on NExT-GQA.
Figure 8: Additional qualitative result on NExT-GQA. GroundFormer produces distinct, question-specific grounding intervals within the same video.
Figure 9: Additional qualitative result on NExT-GQA. GroundFormer preserves temporal ordering across questions and avoids collapsing multiple questions to a single segment.
Figure 10: Grounding consistency for semantically related questions within a video: questions about the same event receive overlapping intervals, while different events are separated.
Figure 11: Grounding consistency on a longer video: early-event questions (baby/toys) and late-event questions (boy/crib) are localized to distinct temporal regions.
Component
Hyperparameter
Value
Encoders
Vision encoder
CLIP ViT-L/14
Vision dim. ( DV )
768
Language encoder
RoBERTa-base
Language dim. ( DL )
768
Projection dim. ( D )
768
Video sampling
Frames ( T )
32 (uniform)
Appendix
Table 11: Implementation configuration and hyperparameters.
Figure 12: Failure case: repeated actions with evidence appearing in multiple disjoint intervals.
Figure 13: Failure case: fine-grained motion cues ( e.g. , deceleration) under sparse temporal sampling.
Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.
Cheng-Han Lee, Amir Salimi, Hsiang-Wei Huang +3
The University of Texas at Austin, United States · Amazon , United States
Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on Video Question Answering (VideoQA). Nevertheless, existing benchmarks are predominantly evaluated through answer correctness, while the faithfulness of predicted evidence supporting those answers remains insufficiently evaluated. This disconnect between answer generation and evidence verification motivates the construction of the Evidence-Grounded Video Question Answering Benchmark (EG-VQA), a large-scale open-ended benchmark in which each QA pair is annotated with temporally localized textual evidence, requiring models to jointly produce answers and verifiable evidence. EG-VQA comprises 2,067 videos and 11,838 QA pairs with fine-grained evidence annotations. To evaluate predicted evidence, Evidence-Grounded F1 (EG-F1) is introduced as a metric that jointly measures temporal alignment and semantic consistency between predicted and ground-truth evidence. Experiments reveal a substantial discrepancy between answer correctness and evidence faithfulness: even strong proprietary models can achieve high answer accuracy while exhibiting gaps in temporal-semantic grounding. To investigate evidence-aware learning under EG-VQA, we develop EG-Reasoner, an evidence-aware model trained with explicit evidence supervision. EG-Reasoner achieves strong evidence grounding performance among open-source models while maintaining competitive answer generation results compared with proprietary systems. These findings highlight the importance of explicitly modeling evidence grounding and suggest that evidence-aware supervision provides an effective direction toward more reliable and interpretable VideoQA systems.
Linpeng Huang, Weixing Chen, Zexin Chen +2
Sun Yat-sen University · Shenzhen University · Peng Cheng Laboratory
Precise temporal grounding requires distinguishing when a queried event occurs from when its participating entities are merely visible. We propose Diffusion-Grounded VideoLLM, which conditions temporal feature extraction on query-relevant entities before language reasoning. The framework tracks entities named in the query and uses their masks to condition a frozen video diffusion backbone. Intermediate spatiotemporal features are extracted through truncated denoising and combined with entity tokens and timestamp embeddings. The language model uses this evidence together with the full query to generate temporal intervals and answers to grounded questions. On Charades-STA and NExT-GQA, the model obtains 43.5 mIoU and 28.4 Acc@GQA, improving the reported Grounded-VideoLLM reference by 6.7 and 1.7 points, respectively. Component and entity-pathway ablations support the usefulness of conditioning diffusion features on query-relevant entities for temporal grounding.