Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.
Figures & tables
Figure 1: Overview of the proposed DiffGround framework. Given a video and a question, DiffGround first performs offline, question-agnostic action segment preprocessing to partition the video into semantically coherent action segments. A frozen MLLM then predicts an action cue describing the target action with a single MLLM call. Guided by the question and predicted action cue, a Vision-Language Model (VLM) ranks video frames to select the most informative keyframes for Video Question Answering (VQA). Finally, Semantic Diff Refinement expands the initial grounding prediction by leveraging SigLIP2 semantic differences to align temporal boundaries with the complete action segment, producing more accurate Video Question Grounding (VideoQG) results.
Figure 2: Overview of the proposed DiffGround framework. DiffGround consists of three stages. First, a frozen MLLM predicts an Action Cue from the input question and uniformly sampled video frames, converting a reasoning-oriented question into an action-centric retrieval query. Second, a frozen VLM (SigLIP2) jointly matches the question and Action Cue to retrieve the Top- K most informative keyframes. Meanwhile, an offline Semantic Diff Segmentation module partitions the video into semantically coherent action segments. Finally, the retrieved keyframes are used for Video Question Answering, and the predicted timestamps are refined by the precomputed semantic segments to recover complete temporal boundaries for Video Question Grounding.
Figure 3: Visualization of the proposed Semantic Diff segmentation. By measuring semantic transitions between neighboring clips, our offline preprocessing partitions the video into semantically coherent action segments. Besides separating distinct scenes (e.g., piano playing versus woman and child), it further identifies fine-grained action boundaries, isolating the brief “touch child” interaction (38.8–40.9 s) from surrounding activities. During online inference, Semantic Diff Refinement aligns the retrieved keyframe with the corresponding action segment, producing accurate temporal grounding.
Dataset
Vid.
Que.
Seg.
Seg. Dur.(s)
Vid. Dur.(s)
Ratio (S./V.)
NExT-GQA
990
5553
6600
6.7
39.5
0.2
STAR
914
7098
7098
11.9
30.0
0.4
Table 1: Statistic of the evaluation sets. Note: We utilize NExT-GQA [ 16 ] ’s test set and STAR [ 15 ] ’s validation set for evaluation. By employing a column-width resizing wrapper, this table maintains a consistent style and layout alignment with our main quantitative results.
Figure 4: Prompt Template Design. The prompt for our final VQG prediction, which is adapted from TStar [ 18 ] .
Method
Venue
Model
# of frames
Acc@GQA ( ↑ )
mIoP ( ↑ )
mIoU ( ↑ )
# of calls
IGV [ 5 ]
CVPR’22
-
16
10.2
21.4
14.0
-
SeViLA [ 19 ]
NIPS’23
BLIP2
4
16.6
29.5
21.7
-
LLoVi [ 21 ]
EMNLP’24
GPT-4 + CogAgent
-
26.8
39.4
21.5
≈N
VideoStreaming [ 8 ]
NIPS’24
Phi-2 2.7B + Vicuna-7B
16
17.8
32.2
19.8
1
CRA [ 2 ]
CVPR’25
Temp[CLIP]
32
18.2
28.6
14.2
-
CRA [ 2 ]
CVPR’25
FrozenBiLM
32
18.8
26.5
13.5
-
Table 2: Quantitative Results on NExT-GQA. Note: N means the number of sampled frames with 1 fps. * indicates that the methods do not require any post-training. By employing a column-width resizing wrapper, all model names are preserved in full within the single-column layout.
Method
Venue
Model
# of frames
Acc@GQA ( ↑ )
Acc@QA ( ↑ )
IoP@5 ( ↑ )
IoU@5 ( ↑ )
# of calls
IGV [ 5 ]
CVPR’22
-
16
13.1
31.7
42.8
1.7
-
SeViLA [ 19 ]
NIPS’23
BLIP2
4
17.0
45.5
37.6
19.5
-
CRA [ 2 ]
CVPR’25
Temp[CLIP]
32
26.8
58.6
44.5
5.5
-
CRA [ 2 ]
CVPR’25
FrozenBiLM
32
27.5
60.5
43.1
5.1
-
*Baseline (Uniform)
-
GPT-4o
4
25.7
59.3
41.4
18.9
1
Our
-
GPT-4o + SigLIP2
4
33.3
61.1
51.9
20.1
2
Table 3: Quantitative Results on STAR. Note: * indicates that the methods do not require any post-training.
Method
# of frames
Acc@GQA ( ↑ )
Acc@QA ( ↑ )
mIoP ( ↑ )
mIoU ( ↑ )
# of calls
Baseline (Uniform)
8
30.5
77.9
38.9
25.5
1
Question (Q)
8
37.3
77.0
45.0
20.0
1
Object Cue (OC)
8
34.0
77.3
41.4
18.7
2
Action Cue (AC)
8
38.1
78.2
46.1
20.0
2
Our (Q + AC)
8
38.5 (+8%)
78.7 (+0.8%)
46.1 (+7.2%)
19.9 (-5.6%)
2
Our (Q + AC + SD)
8
35.0 (+4.5%)
78.7 (+0.8%)
42.7 (+3.8%)
25.5 (+0%)
2
Table 4: Ablation Study of Our Approach on NExT-GQA. Note: Percentage gains indicate relative improvements over the Baseline (Uniform) configuration. SD=Semantic Diff Refinement.
Method
# of frames
Acc@GQA ( ↑ )
Acc@QA ( ↑ )
IoP@5 ( ↑ )
IoU@5 ( ↑ )
# of calls
Baseline (Uniform)
4
25.7
59.3
41.4
18.9
1
Question (Q)
4
31.0
55.3
53.1
8.2
1
Object Cue (OC)
4
30.7
60.7
47.2
7.4
2
Action Cue (AC)
4
33.1
61.3
49.8
7.5
2
Our (Q + AC)
4
33.6 (+7.9%)
61.1 (+1.8%)
52.2 (+11.8%)
7.7 (-11.2%)
2
Our (Q + AC + SD)
4
33.3 (+7.6%)
61.1 (+1.8%)
51.9 (+11.5%)
20.1 (+1.2%)
2
Table 5: Ablation Study of Our Approach on STAR. Percentage gains indicate relative improvements over the Baseline (Uniform) configuration. SD=Semantic Diff Refinement.
VLM
Data
IN-1K
# of frames
Acc@GQA
Acc@QA
mIoP
mIoU
CLIP [ 9 ]
LAION-2B
80.1
8
34.2
78.1
41.5
25.7
SigLIP [ 20 ]
WebLI
83.1
8
33.9
77.8
42.0
25.4
SigLIP2 [ 11 ]
WebLI
84.1
8
35.9
78.7
42.7
25.5
Table 6: Comparison of different VLMs on NExT-GQA. “Train Data” denotes the pre-training dataset, and “IN-1K” denotes ImageNet-1K top-1 accuracy.
Method
# of frames
Acc@GQA (IoP)
Acc@GQA (IoU)
Acc@QA
mIoP
mIoU
Q+AC
8
38.5
24.5
78.7
46.1
19.9
Q+AC+SD
8
35.0
32.0
78.7
42.7
25.5
Table 7: Complementary evaluation using Acc@GQA (IoU). SD=Semantic Diff Refinement.
MLLM
# of frames
Acc@GQA
Acc@QA
mIoP
mIoU
Qwen2.5-VL [ 1 ]
8
23.7
74.8
31.7
14.8
Qwen3-VL [ 17 ]
8
19.9
72.9
28.0
15.5
GPT-4o
8
35.0
78.7
42.7
25.5
Table 8: Comparison of different MLLMs on NExT-GQA. All methods use the same retrieval pipeline; only the MLLM is changed.
Figure 5: Prompt Template Design for Object Cue. The prompt for generating object cues, which is adapted from TStar [ 18 ] .