Organizations: National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, China · School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China
Video understanding requires intelligent agents to transcend mere recognition of visual facts and comprehend the underlying intents behind human actions (often termed the "dark matter" of social intelligence). To bridge the gap between visual observation and intent reasoning, we introduce a novel task, IntentQA, and contribute a large-scale VideoQA dataset specifically tailored for this purpose. However, recognizing that standard metrics may overestimate capabilities due to dataset biases, we go beyond simple accuracy to rigorously evaluate model robustness. We augment the benchmark by generating five distinct contrast sets via Large Language Models (LLMs) and introducing a "Contrast Performance Decline" metric. We propose the X-CaVIR (eXplainable Context-aware Video Intent Reasoning) framework, which leverages three types of "Cognitive Context" to enhance video analysis: i) Situational Context via a cross-modal Video Query Language (VQL) module, ii) Contrastive Context via a Contrastive Learning module, and iii) Commonsense Context via a Commonsense Reasoning module. Crucially, to overcome the opacity of traditional black-box models, we refine the integration of LLMs within X-CaVIR by employing a transparent pipeline that synergizes video captions with VQA model outputs. This approach not only improves performance by effectively utilizing rich commonsense knowledge but also renders the reasoning process explicitly interpretable. Extensive experiments demonstrate the effectiveness of our components, the superiority of X-CaVIR over state-of-the-art baselines, and its stability against perturbations on the contrast sets.
Figures & tables
Fig. 1: (a) Illustration of challenges brought by varied contexts in video intention reasoning. The same action under different contexts could mean different underlying intents. (b) Illustration of the overview of our model’s inference and an example of its contrast set.
Fig. 2: Illustration of four types of QA in our dataset. In the example of CW , a man points to the screen to draw the child’s attention. In the example of CH , the man points his finger to express his intent for a drink. The TN example shows that the boy’s “pointing” leads to the baby’s “looking at what he is pointing”. The TP example shows that the girl’s “pointing” is motivated by the baby’s “spit” action. The red box frames the correct answer.
IntentQA
Training
Validation
Testing
Total
# Video
3212
524
567
4303
# Action
605
399
397
624
# Lemmatized Verb
193
188
167
193
# Action ID
162
162
144
162
# VQA
CW
6989
1185
1250
9424
CH
1940
334
359
2633
TABLE I: Statistics of dataset splitting. # VQA refers to the number of video question answering samples.
Fig. 3: Overview of our eXplainable Context-aware Video Intent Reasoning (X-CaVIR). The figure contains a triplet of samples, i.e., the anchor sample, the positive sample and the negative sample. In the anchor sample, the agent guides the children to look at the screen. In the positive example, the agent guides the children to learn the skill of “crawling”. In the negative example, the “point” action is a trick used by the agent in green to distract the children’s attention and win the game. Our model utilizes situational context, contrastive context and commonsense context to solve the IntentQA task. The black color highlights the test pipeline. The yellow bounding boxes show the region features fed into our model.
Gender-Specific Term Mappings
he → she
his → her
man → woman
men → women
she → he
her → his
woman → man
women → men
boy → girl
boys → girls
guy → lady
guys → ladies
girl → boy
girls → boys
lady → guy
ladies → guys
TABLE II: Data Construction Details. Top: The mapping list used for gender-specific term conversion. Bottom: Some examples of contrast sets generated across different categories. HS : High Similarity, LS : Low Similarity. Changes are highlighted in red .
Fig. 4: Examples of contrast sets based on Verb Phrase, Noun Phrase, and Gender. To evaluate fine-grained reasoning, we generate contrastive distractors across three categories. For Noun Phrases (NP) and Verb Phrases (VP), candidates are classified by semantic proximity: High Similarity (HS) denotes distractors that are semantically close to the correct answer (e.g., altering “stabilize the sand” to “stabilize the clay” [NP], or “fly up” to “climb up” [VP]), requiring precise visual discrimination. Low Similarity (LS) represents distinct actions or objects (e.g., “to push stroller” vs. “to drive stroller” [VP]). Gender contrast involves swapping pronouns or gendered nouns (e.g., “herself” ↔ “himself” ).
Fig. 5: The model architecture for a single sample input. Green colors highlight the input. Orange highlights the modules borrowed from VGT. black highlights our new modules for extracting different contexts. b represents the batch size. numc indicates the number of clips. numf denotes the number of frames in each clip. numr denotes the number of regions per frame. d is the dimension of the features. lenl is the length of Q&A. k refers to the top-k nodes selected.
Model ID
Model
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
0
Blind LLM(InstructGPT)
-
52.16
-
61.28
-
43.43
-
51.55
1
Base Model
50.89
51.76
54.79
56.27
48.00
47.05
50.78
51.36
2
+ VQL
51.65
52.32
54.49
58.77
47.62
48.00
51.08
52.34 (+0.98)
3
+ Triplet Loader
51.56
53.60
56.89
60.72
48.00
49.52
51.52
53.80 (+1.46)
4
+ Triplet Loss
52.57
55.28
57.47
61.56
46.10
47.81
51.71
54.50 (+0.70)
TABLE III: Ablation diagnosis of our model components. We use accuracy (%) as the metric. “+” indicates adding new components on top of the previous row. Model 5 serves as the baseline from our conference version. Models 6 and 7 employ our proposed Confidence Prompting pipeline, with Model 7 further integrating caption information.
Model ID
Model
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
2
Base Model + VQL
51.65
52.32
54.49
58.77
47.62
48.00
51.08
52.34
4-1
Action ID, t1=0.85
51.48
52.32
50.60
58.77
48.38
47.62
50.54
52.25
4-2
Action ID, t1=1
50.13
53.12
54.49
57.66
47.24
48.19
50.10
52.67
4-3
Lemmatized Verb, t1=0.85
51.90
53.36
55.99
62.12
44.00
45.52
50.54
52.91
4-4
Lemmatized Verb, t1=1
50.80
54.56
55.09
62.67
48.00
46.10
50.78
53.84
TABLE IV: Analysis of contrastive learning (best shown in bold) and context attention (best shown with underline). We use accuracy (%) as the metric.
Model ID
Model
CW
CH
TP&TN
Total
7
Ours full (InstructGPT)
60.95
67.69
55.43
60.73
7-1
Ours full (ChatGPT)
59.28
67.97
53.33
59.28
7-2
Ours full (GPT-4)
69.90
68.80
50.67
64.81
6
Ours (InstructGPT) w/o Caption
58.48
66.02
53.33
58.48(-2.25)
6-1
Ours (ChatGPT) w/o Caption
57.92
65.46
50.86
57.45(-1.83)
6-2
Ours (GPT-4) w/o Caption
63.28
72.98
54.10
62.65(-2.16)
TABLE V: Ablation analysis on integrating LLM for common sense reasoning. We use accuracy (%) as the evaluation metric. “-” indicates removing certain components from Model 7 (Ours full) .
Model ID
Model
Text Rep.
LLM Integ.
CW
CH
TP&TN
Total
Val.
Test
Val.
Test
Val.
Test
Val.
Test
-
EVQA [ 98 ]
GloVe
-
25.99
25.92
37.43
34.54
28.00
25.52
28.38
27.27
-
CoMem [ 99 ]
GloVe
-
31.56
30.00
35.63
28.69
28.57
28.95
31.46
29.52
-
HGA [ 100 ]
GloVe
-
29.45
32.00
35.03
30.64
29.71
31.05
30.43
31.54
-
HME [ 25 ]
GloVe
-
30.97
34.40
35.33
34.26
34.29
29.14
32.53
33.08
-
HQGA [ 101 ]
GloVe
-
32.49
33.20
38.32
34.26
34.48
36.57
33.95
34.21
TABLE VI: Comparison results with the established VideoQA baseline models. We use accuracy (%) as the metric. The “LLM Integ.” column denotes the method used to combine visual information with the LLM. Model 5 corresponds to our conference version [ 12 ] using Late Fusion, while our current models employ Confidence Prompting. For detailed configurations, please refer to Table III .
Model ID
Model
MSRVTT-QA (OE)
Val.
Test
-
VGT [ 15 ]
38.26
39.00
3
Ours (w/o triplet loss)
38.11
39.21
4
Ours (w/ triplet loss)
38.98
39.39
TABLE VII: Generalization test on dataset Open-ended MSRVTT-QA . We use accuracy (%) as the metric.
Fig. 6: Qualitative Results and Analysis. The yellow boxes highlight the evidence contexts used to determine the correct answer. The red box frames the correct answer. Actions are colored in red while intents are colored in black.
ID
Model
Contrast Subset
CW
CH
TP&TN
Total
Decline
Acc (%)
Acc (%)
Acc (%)
Acc (%)
(%)
1
Base Model
- (Main Set)
51.92
56.27
47.05
51.45
-
Verbhigh / Verblow
45.84 / 49.28
50.70 / 52.92
40.95 / 42.48
45.45 / 48.22
-11.66 / -6.28
Nounhigh / Nounlow
49.36 / 49.28
54.04 / 54.87
42.48 / 44.95
48.45 / 49.16
-5.83 / -4.45
Gender
52.16
54.87
45.90
51.08
-0.72
Average
49.16
53.13
42.95
48.30
-6.12
TABLE VIII: Comparison results with other representative methods on the Contrast Set. The table is compacted by displaying contrast subsets in paired rows (e.g., High / Low Similarity) while retaining all specific data points with 2 decimal precision. Model IDs are consistent with the text (Model 7 is the Full Model).
ID
Model Variant
Contrast Subset
CW
CH
TP&TN
Total
Decline
Acc (%)
Acc (%)
Acc (%)
Acc (%)
(%)
7
Ours Full (InstructGPT)
- (Main Set)
60.95
67.69
55.43
60.73
-
Verbhigh / Verblow
54.00 / 58.24
59.33 / 66.57
48.19 / 52.00
53.47 / 58.11
-11.95 / -4.31
Entityhigh / Entitylow
56.80 / 58.32
66.30 / 66.85
52.19 / 53.33
57.26 / 58.53
-5.71 / -3.62
Gender
61.36
68.52
55.43
61.11
+0.63
Average
57.74
65.51
52.23
57.70
-4.99
TABLE IX: Ablation analysis on integrating LLM for common sense reasoning (Contrast Set). All data points are preserved using the paired-row format (High / Low) for compactness. Model IDs: 7 (Full), 6 (w/o Caption), 8 (Caption only).
Fig. 7: Qualitative analysis and failure cases of ours model (with GPT-4) on contrast sets.
Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · College of Computer Science and Electronic Engineering, Hunan University, China +1