Video understanding models often rely on motion, yet their explanations can obscure which motion sources support a prediction. We identify motion-source under-specification as a limitation of existing video XAI: region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete. We propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts represent dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context. The resulting typed trajectory/object/scene interface supports class prediction for action recognition and question-conditioned concept matching for video question answering (VQA). We evaluate TraCE across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR. Relative to non-interpretable baselines, TraCE maintains competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA. Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging. Our project page is available at https://leewooil.github.io/TraCE_project_page/.
Figures & tables
Figure 1: Trajectory concepts make motion sources flexible. Existing video XAI often under-specifies which motion source supports a prediction. (a) Heatmap-based explanations [ 7 ] can conflate actor motion, object motion, and static context ( source-ambiguous ). (b) Pose-based concepts [ 28 ] cover one person’s joints but omit object motion and interactions ( source-incomplete ). (c) In contrast, TraCE provides source-flexible motion explanations by representing dynamic evidence from people, objects, and interactions as trajectory concepts.
Figure 2: Overview of TraCE . TraCE represents videos with a typed trajectory/object/scene concept interface: trajectory concepts capture dynamic evidence, while object and scene concepts capture static context. Action recognition predicts directly from video-side concept activations, whereas VQA projects each question–answer pair into concept relevance and matches it with the video-side concepts. Thus, the same typed interface supports trajectory-based explanations for both class prediction and language-grounded reasoning.
Figure 3: Trajectory concept discovery and labeling. TraCE extracts optical-flow trajectories from training videos, filters weak or noisy motion, and clusters the remaining trajectories into recurring motion patterns. Each cluster defines a trajectory concept, and each video is labeled by the normalized frequency of its trajectory assignments. This yields a source-flexible motion vocabulary for people, objects, and interactions without pose annotations or language descriptions of motion.
Method
Hand
Object
DANCE [ 28 ]
42.3
7.6
TraCE
64.5
63.5
Table 1: Motion-source grounding. Top-10 concept support within annotated boxes (%).
Figure 4: Sample-level explanation. We visualize the top-3 contributing concepts by overlaying them on top-activated video regions. (a) Interaction dynamics across agents. (b) Object-centric motion with partial human cues. Trajectory concepts capture motion from multiple sources.
Figure 5: Interpretability of TraCE . (a) Pairwise preference study comparing TraCE explanations with SALF-CBM [ 7 ] , a language-based motion-concept CBM, and DANCE [ 28 ] . Bars show responses relative to TraCE . (b) Concept-level study comparing trajectory concepts ( TraCE ), pose-sequence concepts (DANCE), and language-derived motion concepts. Participants rate how well each concept matches the motion shared by its top-activating video clips on a five-point Likert scale.
Figure 7
Method
NTU RGB+D Mutual
Chiral SSv2
UCF-101
KTH
TOMATO
STAR
Avg.
AR
VQA
AR
VQA
AR
VQA
AR
VQA
VQA
AR
VQA
Baseline w/o interpretability
99.7±0.2
97.4±0.1
86.6±0.2
82.6±0.2
89.8±0.1
99.8±0.0
91.1±0.3
35.9±2.2
54.5±0.4
91.8
74.0
LF-CBM [ 40 ] w/ language concepts
96.9±0.3
–
77.6±0.5
–
88.3±0.2
–
89.1±0.7
–
–
88.0
–
DANCE [ 28 ]
97.1±0.3
–
81.7±0.3
–
89.0±0.2
–
91.3±0.4
–
–
89.8
–
SALF-CBM [ 7 ]
99.7±0.0
–
80.4±0.4
–
87.5±0.2
–
87.9±0.4
–
–
88.9
–
SpLiCE [ 9 ] adaptation
–
53.0
–
57.3
–
86.7
–
22.4
49.2
–
53.7
Table 2: Performance comparison on action recognition and VQA benchmarks. We report mean ± sample standard deviation over five random seeds for Top-1 accuracy (%) using the same frozen backbone within each task, except for the single-run SpLiCE adaptation. A dash (–) indicates that the method is not applicable to the task. Task averages are computed over the corresponding dataset means.
Method
Sth.-Else
SSv2 →
disjoint-obj.
TOMATO
Baseline w/o interpretability
89.5
41.5
TraCE w/o static concepts
94.4
43.1
Table 3: Generalization under object-category and visual-domain shifts.
Perturbation
Acc.
Original
81.1
Reverse
79.2
Rotation
63.1
Table 4: Prediction consistency under perturbations.
Figure 8: Concept intervention curves show faithfulness of TraCE . We deactivate concepts in descending (Top- k ), random, and ascending (Bottom- k ) order of contribution and report prediction accuracy as the removal ratio increases.
Chiral SSv2 (AR)
TOMATO (VQA)
Removed concepts
Acc. (%)
Δ Acc.
Flip rate (%)
Acc. (%)
Δ Acc.
Flip rate (%)
None
81.1
–
–
46.4
–
–
GT direction
56.9±0.1
− 24.2
30.1±0.1
23.9±0.4
− 22.5
70.5±0.5
Opposite direction
82.2±0.1
+1.1
3.6±0.1
54.0±0.2
+7.6
30.8±0.0
Random non-GT
82.1±0.2
+1.0
3.7±0.2
45.4±1.3
− 1.0
21.3±2.0
Table 5: Count-matched trajectory concept interventions. We report accuracy, its change from the unablated model ( Δ Acc.), and the correct-to-incorrect flip rate on direction-annotated subsets. Mean ± standard deviation is computed over 100 concept-subsampling draws. See Appendix E.6 for details.
Figure 9: Concept-level intervention. (a) Deactivating a mismatched trajectory concept fixes a motion error. (b) Deactivating the spurious object concept “bottle cap” shifts the model from static context to motion evidence. These examples illustrate actionable concept-level debugging with TraCE .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Config
KTH
NTU RGB+D Mutual
Model
ViT-Base/16
ViT-Base/16
Pretrained weights
VideoMAE-B/16 (SSv2)
VideoMAE-B/16 (SSv2)
Num. frames
16
16
Input size
224×224
224×224
Short-side size
224
224
Frame sampling
uniform
uniform
Appendix
Table 6: Hyperparameters used for fine-tuning the action recognition backbone fV on KTH and NTU RGB+D Mutual. For Chiral SSv2 and UCF-101, we directly use the publicly released VideoMAE-B/16 fine-tuned weights from the official repository.
Config
Value
Frozen backbone
InternVideo2 s2 -1B
Num. frames
4
Input size
224×224
Frame sampling
uniform stride
Projection dimension dp
512
Feature normalization
ℓ2
Appendix
Table 7: Hyperparameters for training the baseline projection layer. The video and text encoders ( fV and fT ) are kept frozen; only ϕV , ϕT , and the learnable temperature γ are updated. We use the same configuration across all video question answering datasets.
Figure 10: Concept candidate generation prompts. (a) Object and (b) scene prompts used to query Qwen3-VL [ 6 ] .
Dataset
L
Ns
Mt
Activation cutoff
Projection cutoff
Mo (after / before)
Ms (after / before)
NTU RGB+D Mutual
16
5
121
0.3
0.2
55 / 80
28 / 28
Chiral SSv2
16
3
154
0.3
0.2
404 / 1093
286 / 519
UCF-101
16
3
134
0.3
0.3
1103 / 1181
578 / 615
KTH
16
6
111
0.1
0.1
26 / 27
33 / 33
TOMATO
16
23
70
–
–
182 / 182
94 / 94
STAR
50
16
274
–
–
513 / 513
142 / 142
Appendix
Table 8: Dataset-specific concept configurations used by TraCE . We report the trajectory length L , the number of segments per video Ns , the number of trajectory concepts Mt after FINCH clustering, the activation and projection cutoffs used to filter object and scene candidates, and the number of object and scene concepts after / before filtering ( Mo and Ms ). Cutoffs marked “–” are not applied: TOMATO and STAR are video question answering datasets evaluated with the unfiltered object and scene vocabularies. Trajectory concepts have no candidate filtering stage.
Figure 11: Instruction interfaces shown to participants. (a) We provide overview instruction at the beginning of the study. (b) We provide section-specific instructions before each task to guide participants throughout the study.
Figure 12: Example question and user interface used in Task 1. Participants are asked to compare two explanations (A and B) for each sample or class. (a) DANCE [ 28 ] , (b) Language-based [ 56 ] , and (c) SALF-CBM [ 7 ] . In (a) and (b), we show a single video sample with top-3 sample-level concept contributions. In (c), SALF-CBM provides a heatmap-based spatial explanation alongside concept labels.
Figure 13: Example question and user interface used in Task 2. We show how participants evaluate the interpretability of a single concept using several highly activated video clips. Each interface corresponds to one of the three concept types.
Method
Original
Random
Δ
DANCE [ 28 ]
2.52
2.55
0.03
TraCE
4.22
2.67
−1.55
Language-based [ 56 ]
2.27
1.86
−0.41
Appendix
Table 9: Mean Likert scores for Task 2 (concept quality). We report the average score (1: Strongly Disagree to 5: Strongly Agree) over 300 responses per condition. The original condition shows each concept with its top-4 most strongly activating video clips; the random condition shows a randomly selected concept with the same clips. Δ denotes the change from original to random (random − original).
Comparison
Much better
Slightly better
About the same
Slightly worse
Much worse
Prefer TraCE
Language-derived motion concepts
123 (49.2%)
53 (21.2%)
25 (10.0%)
34 (13.6%)
15 (6.0%)
70.4%
Pose-sequence concepts
167 (66.8%)
47 (18.8%)
23 (9.2%)
10 (4.0%)
3 (1.2%)
85.6%
Appendix
Table 10: Pairwise explanation preference in the VQA setting. Each comparison contains 250 participant–item ratings from examples for which both models produce the same correct answer. The five response columns are ordered from TraCE being much better to the baseline being much better; the final column combines the first two responses.
Table 11: Trajectory concept generation ablations. (a) is reported on all action recognition datasets; (b)–(f) are reported on Chiral SSv2. Default settings are TraCE ’s configuration in Table 8 .
Figure 14: Trajectory concepts discovered with different trajectory lengths. Short trajectories ( L=8 ) capture only local and fragmented motion patterns with limited temporal structure. Long trajectories ( L=32 ) often encode near-complete action dynamics, reducing concept reusability across classes. In contrast, the default trajectory length ( L=16 ) captures representative motion fragments that remain both interpretable and reusable across different videos. These visualizations illustrate the trade-off between local motion specificity and action-level over-specialization when choosing the trajectory length.
Method
# Concepts
Acc.
GPT-5.4 [ 56 ]
152
61.5
LLaVA-NeXT-Video [ 72 ]
393
71.8
Video-LLaVA [ 32 ]
1,026
77.7
VideoLLaMA2 [ 13 ]
1,478
77.3
Qwen3-VL [ 6 ]
1,683
74.6
Appendix
Table 12: Effect of different object and scene concept discovery method.
Figure 15: Qualitative comparison of video-language models for text concept generation. We visualize representative object and scene concept candidates generated by different video-language models. (a) LLaVA-NeXT-Video [ 72 ] produces relatively clean scene-centric concepts. (b) Video-LLaVA [ 32 ] frequently mixes appearance descriptions and human-centric phrases into object concepts. (c) VideoLLaMA2 [ 13 ] sometimes generates action phrases or interaction descriptions instead of static scene concepts. These examples highlight that higher downstream accuracy does not necessarily imply cleaner object–scene concept separation.
Video encoder
VLM encoder
Acc.
VideoMAE [ 60 ]
+ InternVid [ 62 ]
83.4
+ InternVideo2 [ 63 ]
84.2
Appendix
Table 13: Evaluation of VLM encoder for object and scene concept labeling
Variant
Acc. (%)
traj ∩ obj
traj ∩ scene
obj ∩ scene
Baseline
84.20
12
10
34
SVD only ( α=1.0 )
84.55
10
19
19
Ortho only ( λ=1.0 )
86.92
8
9
21
SVD + Ortho ( λ=1.0 )
86.78
8
9
22
SVD + Ortho ( λ=2.0 )
87.83
8
8
26
Appendix
Table 14: Component ablation for concept group disentanglement. We report validation accuracy (%) and top- 50 backbone-dimension overlap between group pairs. Random-chance overlap is 3.26 . The object–scene pair (object ∩ scene) is the structurally hardest target because both groups are text-based visual concepts.
Order
Acc. (%)
traj ∩ obj
traj ∩ scene
obj ∩ scene
sum
TSO
86.92
10
9
20
39
TOS
86.78
8
9
22
39
STO
86.64
12
16
12
40
SOT
86.50
13
7
9
29
OTS
86.15
14
8
12
34
OST
85.73
12
13
11
36
Appendix
Table 15: Effect of concept group training order on accuracy and disentanglement. We report validation accuracy (%) and top- 50 backbone-dimension overlap between group pairs for all six permutations of {trajectory (T), object (O), scene (S)}. SVD + Ortho ( λ=1.0 ) is fixed.
Concept
Acc.
Trajectory
Object
Scene
✓
✗
✗
80.7
✓
✓
✗
82.4
✓
✗
✓
81.6
✗
✓
✓
74.6
✓
✓
✓
84.2
Appendix
Table 16: Effects of individual concept types on TraCE performance.
NEC
Sparsity (%)
Acc. (%)
5
0.63
67.7
10
1.21
77.2
15
1.79
81.3
20
2.41
82.9
25
3.03
83.8
30
3.57
84.0
Appendix
Table 17: Accuracy under controlled Number of Effective Concepts (NEC) [ 58 ] . NEC limits the average number of non-zero classifier weights per class. Lower NEC yields sparser, more interpretable predictions at the cost of accuracy. Sparsity denotes the fraction of non-zero entries in WAR .
Original group
Reverse consistency (%)
Rotation consistency (%)
Left
90.4
83.3
Right
84.0
90.8
Up
55.8
44.2
Down
94.0
45.2
Overall
79.2
63.1
Appendix
Table 18: Prediction consistency under perturbation on Chiral SSv2. We report the percentage of samples whose predicted motion group follows the expected transformation after perturbation. Expected transformations are: LEFT ↔ RIGHT and UP ↔ DOWN under temporal reverse, and LEFT → DOWN, RIGHT → UP, UP → LEFT, DOWN → RIGHT under 90 ∘ counter-clockwise rotation.
K=5
K=10
Dataset
Top-K ↑
Random-K ↓
Top-K ↑
Random-K ↓
Chiral SSv2
48.7
3.7
70.1
5.4
KTH
35.1
2.1
46.0
2.8
UCF-101
15.9
0.2
18.4
0.3
NTU RGB+D Mutual
7.0
0.4
13.8
0.3
Average
26.7
1.6
37.1
2.2
Appendix
Table 19: Faithfulness of trajectory concepts via concept removal. We report Flip%, the percentage of samples whose prediction changes after deactivating trajectory concepts. We compare removing the top-K contributing concepts with removing K randomly selected concepts. Higher Top-K removal and lower Random-K removal indicate that the estimated concept importance aligns with the model’s actual decision process.
DANCE [ 28 ]
TraCE
Selection
Hand
Object
Both
Hand
Object
Both
Bottom- 10
0.0
0.0
0.0
22.4
21.6
19.0
Random- 10
4.1
0.7
0.5
29.1
28.5
25.1
Top- 10
42.3
7.6
4.7
64.5
63.5
58.1
Appendix
Table 20: Sample-level motion-source grounding on object-disjoint Chiral SSv2. Top and Bottom select K=10 concepts according to their contribution to the predicted class; Random samples one set of K=10 concepts uniformly without replacement for each video. Hand and Object report the percentages of annotated boxes covered by the seed pixel of a selected trajectory in the first frame of its L -frame clip; Both reports the percentage of videos covering at least one hand and one manipulated object.
Prediction
Input
Direction agreement (%)
Replacement (%)
Correct
Original
76.3
0.0
Correct
Temporal reversal
80.1
96.3
Correct
90∘ rotation
78.4
96.1
Incorrect
Original
45.4
0.0
Incorrect
Temporal reversal
32.8
81.0
Incorrect
90∘ rotation
35.1
84.6
Appendix
Table 21: Directional semantics under controlled perturbations. Direction agreement is the fraction of top- 5 activated trajectory concepts whose fixed direction labels, derived from their medoid trajectories, match the expected direction. For the original and perturbed top- 5 concept-index sets Corig5 and Cpert5 , replacement is 1−∣Corig5∩Cpert5∣/5 . Prediction correctness after perturbation is evaluated against the expected transformed class; the final row reports random-reference values.
Protocol
Task
Removed concepts
Acc. (%)
Δ Acc.
Flip rate (%)
Count-matched
AR
None ( TraCE )
81.1
–
–
GT direction
56.9±0.1
−24.2
30.1±0.1
Opposite direction
82.2±0.1
+1.1
3.6±0.1
Random non-GT
82.1±0.2
+1.0
3.7±0.2
VQA
None ( TraCE )
46.4
–
–
GT direction
23.9±0.4
−22.5
70.5±0.5
Appendix
Table 22: Ground-truth-targeted trajectory concept interventions. We report post-intervention accuracy, its change from the unablated model, and the flip rate, i.e., the percentage of initially correct predictions that become incorrect. Count-matched results and the random non-ground-truth full-group results report mean ± standard deviation over 100 concept-subsampling draws on the fixed evaluation set.
TOMATO
STAR
Removal ratio
Top
Random
Bottom
Top
Random
Bottom
0%
38.0
38.0
38.0
53.4
53.4
53.4
5%
26.1
33.7
38.0
12.9
52.1
53.4
10%
18.5
33.7
38.0
14.2
50.9
53.4
20%
16.3
33.7
38.0
11.7
48.8
53.4
40%
16.3
27.2
38.0
10.6
44.9
53.4
Appendix
Table 23: Trajectory concept removal on native VQA benchmarks. For each sample, we remove the indicated fraction of trajectory concepts in descending (Top), random, or ascending (Bottom) order of contribution and report answer accuracy (%).
Method
NTU Mutual
Chiral SSv2
UCF-101
TOMATO
STAR
Avg.
Non-interpretable baseline
97.4±0.1
82.6±0.2
99.8±0.0
35.9±2.2
54.5±0.4
74.0
Language-derived motion concepts
97.6±0.1
82.0±0.4
99.7±0.1
31.7±3.6
56.5±0.5
73.5
Pose-sequence concepts
97.6±0.1
82.3±0.6
99.7±0.1
28.7±7.0
55.9±0.7
72.8
SpLiCE [ 9 ] adaptation
53.0
57.3
86.7
22.4
49.2
53.7
TraCE
97.1±0.3
82.6±0.6
99.7±0.1
37.0±1.7
55.4±1.2
74.4
Appendix
Table 24: Comparison with VQA-capable concept methods. We report answer accuracy (%) as mean ± sample standard deviation over five stochastic training seeds, except for the single-run SpLiCE adaptation. The controlled variants combine the MM-CBM [ 54 ] concept-space matching design with language-derived motion concepts from LF-CBM [ 40 ] or pose-sequence concepts from DANCE [ 28 ] . The first three datasets are multiple-choice VQA reformulations of action-recognition datasets; TOMATO and STAR are native VQA benchmarks.
Concept
Acc.
Trajectory
Object
Scene
✓
✗
✗
54.8
✗
✓
✗
54.4
✗
✗
✓
52.9
✓
✓
✗
54.5
✓
✗
✓
52.9
Appendix
Table 25: Effects of individual concept types on TraCE performance.
Method
Acc.
InfoNCE
53.4
Cosine similarity
52.0
KL divergence
52.6
ℓ1 regression
52.1
ℓ2 regression
52.2
Appendix
Table 26: Effect of text-side trajectory relevance learning objectives.
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.
Recent video reasoning models increasingly produce spatio-temporal evidence chains that localize objects at specific timestamps. While these traces improve interpretability by grounding where and when evidence appears, they often leave the motion connecting observations, the how, implicit. This makes dynamic and trajectory-dependent claims difficult to supervise, verify, or penalize when unsupported by the video. We formalize this missing component as Spatial-Temporal-Trajectory (STT) reasoning and introduce Motion-o, a motion-centric extension to vision-language models (VLMs) that makes trajectories explicit and verifiable. Motion-o augments evidence chains with Motion Chain of Thought (MCoT), a structured pathway that represents object motion through a discrete <motion/> tag summarizing direction, speed, and scale change. To supervise MCoT, we densify sparse spatio-temporal annotations into object tracks and derive motion descriptors from centroid displacement and box-area change. We then train with complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal that penalizes motion descriptions that remain unchanged when temporal evidence is removed. Across multiple video understanding benchmarks, Motion-o consistently improves trajectory-faithful reasoning without architectural modifications. These results suggest that an explicit motion interface can complement existing VLM pipelines by converting implicit dynamics into verifiable evidence. Code is available at~\faGithub\ ostadabbas/Motion-o.
Concept Bottleneck Models (CBMs), which route predictions through a layer of human-interpretable concepts, have substantially improved the interpretability of visual classification by exposing an explicit, language-based rationale for each prediction. Most existing work focuses on static images, and temporal dynamics in videos have been underexplored in conceptual representation. We propose a framework that extends concept-based image recognition to procedural video understanding. It represents a multi-step video as a temporal sequence of segment-level concept score vectors and feeds this sequence into a Transformer encoder to predict the class label. We also explore a way to estimate the contribution of each concept for further interpretability. We evaluated three concept sets of different granularities on cooking and household-activity videos and found that coarser-grained concepts are sufficient for classifying these videos, which implies that fine-grained textual descriptions may be unnecessary.
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
Shijie Wang, Honglu Zhou, Ziyang Wang +5
Salesforce, Palo Alto, CA, USA · Brown University, Providence, RI, USA