Organizations: National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, China · School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China
Video understanding requires intelligent agents to transcend mere recognition of visual facts and comprehend the underlying intents behind human actions (often termed the "dark matter" of social intelligence). To bridge the gap between visual observation and intent reasoning, we introduce a novel task, IntentQA, and contribute a large-scale VideoQA dataset specifically tailored for this purpose. However, recognizing that standard metrics may overestimate capabilities due to dataset biases, we go beyond simple accuracy to rigorously evaluate model robustness. We augment the benchmark by generating five distinct contrast sets via Large Language Models (LLMs) and introducing a "Contrast Performance Decline" metric. We propose the X-CaVIR (eXplainable Context-aware Video Intent Reasoning) framework, which leverages three types of "Cognitive Context" to enhance video analysis: i) Situational Context via a cross-modal Video Query Language (VQL) module, ii) Contrastive Context via a Contrastive Learning module, and iii) Commonsense Context via a Commonsense Reasoning module. Crucially, to overcome the opacity of traditional black-box models, we refine the integration of LLMs within X-CaVIR by employing a transparent pipeline that synergizes video captions with VQA model outputs. This approach not only improves performance by effectively utilizing rich commonsense knowledge but also renders the reasoning process explicitly interpretable. Extensive experiments demonstrate the effectiveness of our components, the superiority of X-CaVIR over state-of-the-art baselines, and its stability against perturbations on the contrast sets.
Figures & tables
Fig. 1: (a) Illustration of challenges brought by varied contexts in video intention reasoning. The same action under different contexts could mean different underlying intents. (b) Illustration of the overview of our model’s inference and an example of its contrast set.
Fig. 2: Illustration of four types of QA in our dataset. In the example of CW , a man points to the screen to draw the child’s attention. In the example of CH , the man points his finger to express his intent for a drink. The TN example shows that the boy’s “pointing” leads to the baby’s “looking at what he is pointing”. The TP example shows that the girl’s “pointing” is motivated by the baby’s “spit” action. The red box frames the correct answer.
IntentQA
Training
Validation
Testing
Total
# Video
3212
524
567
4303
# Action
605
399
397
624
# Lemmatized Verb
193
188
167
193
# Action ID
162
162
144
162
# VQA
CW
6989
1185
1250
9424
CH
1940
334
359
2633
TABLE I: Statistics of dataset splitting. # VQA refers to the number of video question answering samples.
Fig. 3: Overview of our eXplainable Context-aware Video Intent Reasoning (X-CaVIR). The figure contains a triplet of samples, i.e., the anchor sample, the positive sample and the negative sample. In the anchor sample, the agent guides the children to look at the screen. In the positive example, the agent guides the children to learn the skill of “crawling”. In the negative example, the “point” action is a trick used by the agent in green to distract the children’s attention and win the game. Our model utilizes situational context, contrastive context and commonsense context to solve the IntentQA task. The black color highlights the test pipeline. The yellow bounding boxes show the region features fed into our model.
Gender-Specific Term Mappings
he → she
his → her
man → woman
men → women
she → he
her → his
woman → man
women → men
boy → girl
boys → girls
guy → lady
guys → ladies
girl → boy
girls → boys
lady → guy
ladies → guys
TABLE II: Data Construction Details. Top: The mapping list used for gender-specific term conversion. Bottom: Some examples of contrast sets generated across different categories. HS : High Similarity, LS : Low Similarity. Changes are highlighted in red .
Fig. 4: Examples of contrast sets based on Verb Phrase, Noun Phrase, and Gender. To evaluate fine-grained reasoning, we generate contrastive distractors across three categories. For Noun Phrases (NP) and Verb Phrases (VP), candidates are classified by semantic proximity: High Similarity (HS) denotes distractors that are semantically close to the correct answer (e.g., altering “stabilize the sand” to “stabilize the clay” [NP], or “fly up” to “climb up” [VP]), requiring precise visual discrimination. Low Similarity (LS) represents distinct actions or objects (e.g., “to push stroller” vs. “to drive stroller” [VP]). Gender contrast involves swapping pronouns or gendered nouns (e.g., “herself” ↔ “himself” ).
Fig. 5: The model architecture for a single sample input. Green colors highlight the input. Orange highlights the modules borrowed from VGT. black highlights our new modules for extracting different contexts. b represents the batch size. numc indicates the number of clips. numf denotes the number of frames in each clip. numr denotes the number of regions per frame. d is the dimension of the features. lenl is the length of Q&A. k refers to the top-k nodes selected.
Model ID
Model
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
0
Blind LLM(InstructGPT)
-
52.16
-
61.28
-
43.43
-
51.55
1
Base Model
50.89
51.76
54.79
56.27
48.00
47.05
50.78
51.36
2
+ VQL
51.65
52.32
54.49
58.77
47.62
48.00
51.08
52.34 (+0.98)
3
+ Triplet Loader
51.56
53.60
56.89
60.72
48.00
49.52
51.52
53.80 (+1.46)
4
+ Triplet Loss
52.57
55.28
57.47
61.56
46.10
47.81
51.71
54.50 (+0.70)
TABLE III: Ablation diagnosis of our model components. We use accuracy (%) as the metric. “+” indicates adding new components on top of the previous row. Model 5 serves as the baseline from our conference version. Models 6 and 7 employ our proposed Confidence Prompting pipeline, with Model 7 further integrating caption information.
Model ID
Model
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
2
Base Model + VQL
51.65
52.32
54.49
58.77
47.62
48.00
51.08
52.34
4-1
Action ID, t1=0.85
51.48
52.32
50.60
58.77
48.38
47.62
50.54
52.25
4-2
Action ID, t1=1
50.13
53.12
54.49
57.66
47.24
48.19
50.10
52.67
4-3
Lemmatized Verb, t1=0.85
51.90
53.36
55.99
62.12
44.00
45.52
50.54
52.91
4-4
Lemmatized Verb, t1=1
50.80
54.56
55.09
62.67
48.00
46.10
50.78
53.84
TABLE IV: Analysis of contrastive learning (best shown in bold) and context attention (best shown with underline). We use accuracy (%) as the metric.
Model ID
Model
CW
CH
TP&TN
Total
7
Ours full (InstructGPT)
60.95
67.69
55.43
60.73
7-1
Ours full (ChatGPT)
59.28
67.97
53.33
59.28
7-2
Ours full (GPT-4)
69.90
68.80
50.67
64.81
6
Ours (InstructGPT) w/o Caption
58.48
66.02
53.33
58.48(-2.25)
6-1
Ours (ChatGPT) w/o Caption
57.92
65.46
50.86
57.45(-1.83)
6-2
Ours (GPT-4) w/o Caption
63.28
72.98
54.10
62.65(-2.16)
TABLE V: Ablation analysis on integrating LLM for common sense reasoning. We use accuracy (%) as the evaluation metric. “-” indicates removing certain components from Model 7 (Ours full) .
Model ID
Model
Text Rep.
LLM Integ.
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
-
EVQA [ 98 ]
GloVe
-
25.99
25.92
37.43
34.54
28.00
25.52
28.38
27.27
-
CoMem [ 99 ]
GloVe
-
31.56
30.00
35.63
28.69
28.57
28.95
31.46
29.52
-
HGA [ 100 ]
GloVe
-
29.45
32.00
35.03
30.64
29.71
31.05
30.43
31.54
-
HME [ 25 ]
GloVe
-
30.97
34.40
35.33
34.26
34.29
29.14
32.53
33.08
-
HQGA [ 101 ]
GloVe
-
32.49
33.20
38.32
34.26
34.48
36.57
33.95
34.21
TABLE VI: Comparison results with the established VideoQA baseline models. We use accuracy (%) as the metric. The “LLM Integ.” column denotes the method used to combine visual information with the LLM. Model 5 corresponds to our conference version [ 12 ] using Late Fusion, while our current models employ Confidence Prompting. For detailed configurations, please refer to Table III .
Model ID
Model
MSRVTT-QA (OE)
Val.
Test
-
VGT [ 15 ]
38.26
39.00
3
Ours (w/o triplet loss)
38.11
39.21
4
Ours (w/ triplet loss)
38.98
39.39
TABLE VII: Generalization test on dataset Open-ended MSRVTT-QA . We use accuracy (%) as the metric.
Fig. 6: Qualitative Results and Analysis. The yellow boxes highlight the evidence contexts used to determine the correct answer. The red box frames the correct answer. Actions are colored in red while intents are colored in black.
ID
Model
Contrast Subset
CW
CH
TP&TN
Total
Decline
Acc (%)
Acc (%)
Acc (%)
Acc (%)
(%)
1
Base Model
- (Main Set)
51.92
56.27
47.05
51.45
-
Verbhigh / Verblow
45.84 / 49.28
50.70 / 52.92
40.95 / 42.48
45.45 / 48.22
-11.66 / -6.28
Nounhigh / Nounlow
49.36 / 49.28
54.04 / 54.87
42.48 / 44.95
48.45 / 49.16
-5.83 / -4.45
Gender
52.16
54.87
45.90
51.08
-0.72
Average
49.16
53.13
42.95
48.30
-6.12
TABLE VIII: Comparison results with other representative methods on the Contrast Set. The table is compacted by displaying contrast subsets in paired rows (e.g., High / Low Similarity) while retaining all specific data points with 2 decimal precision. Model IDs are consistent with the text (Model 7 is the Full Model).
ID
Model Variant
Contrast Subset
CW
CH
TP&TN
Total
Decline
Acc (%)
Acc (%)
Acc (%)
Acc (%)
(%)
7
Ours Full (InstructGPT)
- (Main Set)
60.95
67.69
55.43
60.73
-
Verbhigh / Verblow
54.00 / 58.24
59.33 / 66.57
48.19 / 52.00
53.47 / 58.11
-11.95 / -4.31
Entityhigh / Entitylow
56.80 / 58.32
66.30 / 66.85
52.19 / 53.33
57.26 / 58.53
-5.71 / -3.62
Gender
61.36
68.52
55.43
61.11
+0.63
Average
57.74
65.51
52.23
57.70
-4.99
TABLE IX: Ablation analysis on integrating LLM for common sense reasoning (Contrast Set). All data points are preserved using the paired-row format (High / Low) for compactness. Model IDs: 7 (Full), 6 (w/o Caption), 8 (Caption only).
Fig. 7: Qualitative analysis and failure cases of ours model (with GPT-4) on contrast sets.
Multi-modal Large Language Models (MLLMs) have significantly advanced video reasoning, yet Video Question Answering (VideoQA) remains challenging due to its demand for temporal causal reasoning and evidence-grounded answer generation. Prevailing end-to-end MLLM frameworks lack explicit structured reasoning between visual perception and answer derivation, causing severe hallucinations and poor interpretability. Existing methods also fail to address three core gaps: faithful visual clue extraction, utility-aware clue filtering, and end-to-end clue-answer alignment. Inspired by hierarchical human visual cognition, we propose ClueNet, a clue-aware video reasoning framework with a two-stage supervised fine-tuning paradigm without extensive base model modifications. Decoupled supervision aligns clue extraction and chain-based reasoning, while inference supervision with an adaptive clue filter refines high-order reasoning, alongside lightweight modules for efficient inference. Experiments on NExT-QA, STAR, and MVBench show that ClueNet outperforms state-of-the-art methods by ≥ 1.1%, with superior generalization, hallucination mitigation, inference efficiency, and cross-backbone compatibility. This work bridges the perception-to-generation gap in MLLM video understanding, providing an interpretable, faithful reasoning paradigm for high-stakes VideoQA applications.
Kaixin zhang, Xiaohe Li, Jiahao Li +4
Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Video Question Answering (VideoQA) demands models that jointly reason over spatial, temporal, and linguistic cues. However, the task's inherent complexity often requires multi-step reasoning that current large multimodal models (LMMs) perform implicitly, leaving their internal decision process opaque. In contrast, large reasoning models (LRMs) explicitly generate intermediate logical steps that enhance interpretability and can improve multi-hop reasoning accuracy. Yet, these models are not designed for native video understanding, as they typically rely on static frame sampling. We propose UpstreamQA, a modular framework that disentangles and evaluates core video reasoning components through explicit upstream reasoning modules. Specifically, we employ multimodal LRMs to perform object identification and scene context generation before passing enriched reasoning traces to downstream LMMs for VideoQA. We evaluate UpstreamQA on the OpenEQA and NExTQA datasets using two LRMs (o4-mini, Gemini 2.5 Pro) and two LMMs (GPT-4o, Gemini 2.5 Flash). Our results demonstrate that introducing explicit reasoning can significantly boost performance and interpretability of downstream VideoQA, but can also lead to performance degradation when baseline performance is sufficiently high. Overall, UpstreamQA offers a principled framework for combining explicit reasoning and multimodal understanding, advancing both performance and diagnostic transparency in VideoQA in several scenarios.
Jason Nguyen, Ameet Rao, Alexander Chang +2
Lincoln North Star High School · The Charter School of Wilmington · Greenwich High School +2
The dense, temporal nature of video presents a profound challenge for automated analysis. Despite the use of powerful Vision-Language Models, prevailing methods for video understanding are limited by the inherent disconnect between reasoning and perception: they rely on static, pre-processed information and cannot actively seek raw evidence from video as their understanding evolves. To address this, we introduce LensWalk, a flexible agentic framework that empowers a Large Language Model reasoner to control its own visual observation actively. LensWalk establishes a tight reason-plan-observe loop where the agent dynamically specifies, at each step, the temporal scope and sampling density of the video it observes. Using a suite of versatile, Vision-Language Model based tools parameterized by these specifications, the agent can perform broad scans for cues, focus on specific segments for fact extraction, and stitch evidence from multiple moments for holistic verification. This design allows for progressive, on-demand evidence gathering that directly serves the agent's evolving chain of thought. Without requiring any model fine-tuning, LensWalk delivers substantial, plug-and-play performance gains on multiple model recipes, boosting their accuracy by over 5% on challenging long-video benchmarks like LVBench and Video-MME. Our analysis reveals that enabling an agent to control how it sees is key to unlocking more accurate, robust, and interpretable video reasoning.
Keliang Li, Yansong Li, Hongze Shen +3
Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · College of Computer Science and Electronic Engineering, Hunan University, China +1