Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers

Youngjun Jun; Seil Kang; Woojung Han; Seong Jae Hwang

cs.CVMar 3, 2026

Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers

Authors: Youngjun Jun, Seil Kang, Woojung Han, Seong Jae Hwang

Organizations: Yonsei University

Abstract

Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.

Figures & tables

Figure 2: Spatiotemporal localization pipeline. This pipeline obtains a video saliency map for any concept using Video DiTs. Given a concept, we first obtain a text-surrogate token via Query–Key Matching, and then compute the GramCol to derive its spatial saliency map. For motion concepts, we additionally identify motion heads before computing GramCol, thereby improving temporal localization.

Figure 3: Layer-wise average of $\lambda_{2}$ and the corresponding feature maps. As the average $\lambda_{2}$ increases, the extracted features become sharper and more interpretable.

Figure 4: QK-Matching visualizations per text token. While QK-Matching yields somewhat unclear spatial localization, its peak (red dot) still accurately pinpoints the target concept.

Figure 5: Visualization of Motion Localization Score (MLS) versus the separation score (Calinski-Harabasz index, CHI). MLS measured from GramCol extracted across attention heads in layers $\mathcal{L}$ of CogVideoX-2B. Heads with higher CHI scores tend to exhibit higher MLS, with a Pearson correlation coefficient of 0.60.

Figure 6: Qualitative comparisons of motion localization results using CogVideoX-5B on the MeViS dataset

Method	Backbone	SL	TL	PR	SS	OBJ	Avg.
ViCLIP	ViT-H	0.33	0.17	0.35	0.29	0.28	0.28
DAAM	VideoCrafter2	0.36	0.17	0.38	0.32	0.35	0.32
Cross Attention	HunyuanVideo	0.39	0.25	0.41	0.36	0.34	0.35
ConceptAttention	HunyuanVideo	0.42	0.26	0.44	0.35	0.34	0.36
IMAP	HunyuanVideo	0.60	0.41	0.62	0.50	0.62	0.55
Cross Attention	CogVideoX-2B	0.29	0.56	0.35	0.23	0.28	0.34

Table 1: Motion localization results on including five metrics.

Exp. #	Method	SL	TL	PR	SS	OBJ	Avg.
	Cross Attention	0.41	0.27	0.43	0.34	0.33	0.36
	ConceptAttention w/o softmax	0.42	0.24	0.44	0.37	0.38	0.37
	ConceptAttention w/ softmax	0.50	0.32	0.51	0.47	0.47	0.45
1	GramCol (all layers)	0.45	0.30	0.47	0.41	0.42	0.41
2	+ layer selection	0.47	0.34	0.48	0.48	0.50	0.46
3	+ motion head selection	0.53	0.34	0.55	0.46	0.48	0.47

Table 2: Ablation results showing the impact of our components on the performance over five metrics.

Figure 7: Qualitative ablation results exhibiting progressive improvement in motion localizability (concept: walking).

	Method		Backbone	mIoU	mVC 8	mVC 16
Supervised Segmentation	UniVS	CVPR’24	Swin-T	59.8	92.3	-
Supervised Segmentation	DVIS++	ICCV’23	VIT-L	63.8	95.7	95.1
Zero-Shot Segmentation	CLIPpy	ICCV’23	T5 + DINO	17.7	72.4	68.4
	EmerDiff	ICLR’24	SD 2.1	43.4	68.9	64.3
	VidSegDiff	arXiv’24	SD 2.1	60.6	90.7	89.6
	EmerDiff	ICLR’24	SVD	39.7	82.1	78.5

Table 3: Quantitative results of zero-shot video semantic segmentation on the VSPW dataset.

Figure 8: Qualitative results of zero-shot video semantic segmentation using interpretable map methods for Video DiTs. All these saliency maps are upsampled by AnyUp [ 77 ] .

Stage	Video Encode	Diffusion Inference			Map Save
		Diffusion Denoising	Query-Key Matching	Motion Head Selection
Time (s)	10.79	58.67	10.35	0.083	7.17

Table 4: Stage-wise inference time. All timings are measured on an A100 80GB GPU using CogVideoX-5B [ 82 ] , with re-noising followed by a single denoising timestep. (Video Encode: VAE encoding stage of the real video; Map Save: visualization process of IMAPs.)

Figure 9: Visualization of IMAP over various timesteps

Layers	# layers	SL	TL	PR	SS	OBJ	Avg.
All layers	42	0.53	0.34	0.55	0.46	0.48	0.47
$<0.50$	17	0.34 (-0.19)	0.22 (-0.12)	0.37 (-0.18)	0.29 (-0.17)	0.28 (-0.20)	0.30 (-0.17)
$>0.50$	25	0.57 (+0.04)	0.44 (+0.10)	0.58 (+0.03)	0.53 (+0.07)	0.53 (+0.05)	0.53 (+0.06)
$>0.70$	21	0.68 (+0.15)	0.48 (+0.14)	0.69 (+0.14)	0.61 (+0.15)	0.64 (+0.16)	0.62 (+0.15)
$>0.75$	14	0.66 (+0.13)	0.51 (+0.17)	0.68 (+0.13)	0.61 (+0.15)	0.64 (+0.16)	0.62 (+0.15)
$>0.80$	7	0.69 (+0.16)	0.55 (+0.21)	0.70 (+0.15)	0.65 (+0.19)	0.65 (+0.17)	0.65 (+0.18)

Table 5: Quantitative analysis of the layer thresholds in CogVideoX-5B

Figure 10: Quantitative analysis of the layer thresholds in CogVideoX-5B

Figure 12: Comparison of IMAP under high ( $>0.70$ ) vs. low ( $<0.50$ ) average $\lambda_{2}$ layers in CogVidoX-5B

Figure 13: Comparison of IMAP under high ( $>0.70$ ) vs. low ( $<0.50$ ) average $\lambda_{2}$ layers in CogVidoX-5B

Figure 14: Examples of negative heads in CogVideoX-5B (L: layer, H: head)

Figure 15: Qualitative comparison of the effect of applying the softmax operation ( $\times$ : without softmax, ✓: with softmax)

Figure 16: Qualitative comparison of the effect of applying the softmax operation ( $\times$ : without softmax, ✓: with softmax)

Figure 17: Visualization of visual token embeddings via dimensionality reduction [ 74 ] (colors indicate frames)

Figure 18: Counts of

150

motion words in the preprocessed MeViS dataset

Figure 19: Example of the LLM input grid image

Metric	SL	TL	PR	SS	OBJ	Avg.
Pearson’s $r$ (PCC)	0.62	0.44	0.64	0.66	0.64	0.75
Spearman’s $r_{s}$	0.52	0.46	0.59	0.63	0.69	0.74

Table 7: Correlation coefficients between scores from 5 observers and OpenAI o3-pro.

Total	Agreement	Our wins (Win Rate)
51	44.8 (88%)	45.6 (89%)

Table 8: Pairwise preference agreement and win rate.

Method	SL	TL	PR	SS	OBJ	Avg.
HiNorm	0.47	0.34	0.50	0.41	0.43	0.43
QK-Matching	0.68	0.48	0.69	0.61	0.64	0.62

Table 9: Quantitative analysis of various text-surrogate token selection methods

Method	SL	TL	PR	SS	OBJ	Avg.
Video	0.48	0.31	0.50	0.41	0.44	0.43
Frame-wise	0.68	0.48	0.69	0.61	0.64	0.62

Table 10: Quantitative analysis of various text-surrogate token selection levels

Method	# heads	SL	TL	PR	SS	OBJ	Avg.
None (all heads)	48	0.47	0.34	0.48	0.48	0.50	0.46
Random	5	0.50	0.30	0.51	0.44	0.47	0.44
Separation Score	5	0.68	0.48	0.69	0.61	0.64	0.62

Table 11: Quantitative results of various motion head selection methods

# Heads	SL	TL	PR	SS	OBJ	Avg.
1	0.52	0.36	0.54	0.44	0.47	0.47
3	0.55	0.35	0.56	0.49	0.51	0.49
5	0.68	0.48	0.69	0.61	0.64	0.62
10	0.55	0.34	0.55	0.48	0.51	0.49

Table 12: Quantitative analysis of the number of selected motion heads

Method	Backbone	SL	TL	PR	SS	OBJ	Avg.
Cross Attention	Wan2.1-1.3B	0.49	0.32	0.45	0.43	0.45	0.43
IMAP	Wan2.1-1.3B	0.50	0.33	0.52	0.40	0.40	0.43
Cross Attention	Wan2.1-14B	0.50	0.32	0.51	0.42	0.41	0.43
IMAP	Wan2.1-14B	0.58	0.41	0.60	0.47	0.49	0.51

Table 14: Motion localization results on including five metrics

Figure 21: Feature-smilar but conceptually irrelevant cases (Repeated textures)

Figure 22: Feature-smilar but conceptually irrelevant cases (Different motions)

Figure 23: Failure cases. (a) The motion corresponding to the motion concept is not generated during video synthesis. (b) IMAP is extracted for the motion concept from an unconventional viewpoint. (c) The agent performing the motion is outside the camera view.

Figure 24: Qualitative comparisons of motion localization results using CogVideoX-5B on the MeViS dataset

Figure 25: Qualitative comparisons of motion localization results using CogVideoX-5B on the MeViS dataset

Figure 26: Qualitative comparisons of motion localization results using CogVideoX-2B on the MeViS dataset

Figure 27: Qualitative comparisons of motion localization results using HunyuanVideo on the MeViS dataset

Figure 28: Qualitative comparisons of zero-shot video semantic segmentation results using CogVideoX-5B on the VSPW dataset

Figure 29: Qualitative comparisons of zero-shot video semantic segmentation results using CogVideoX-5B on the VSPW dataset

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 735 papers

MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models
Jun 5, 2026Yifan Xu, Chao Zhang, Ruifei Ma +4VLM Adaptation Vision-Language Alignment
School of Computer Science and Engineering, Beihang University · 2Beijing Digital Native Digital City Research Center · School of Computer Science, Peking University +1
Controlling Motion Transfer in Diffusion Transformers via Attention Heads
Jul 13, 2026Sunyoung Jung, Jiwoo Park, Yoonseok Choi +3Diffusion Model Guidance Diffusion Transformer
Yonsei University · LG Electronics · University of California, Merced
MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer
Apr 1, 2026Samuel Teodoro, Yun Chen, Agus Gunawan +3Controllable Video Generation Video Diffusion Models
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea · Adobe Research, San Jose, California · CMLab, Chung-Ang University, Seoul, South Korea

Jun 5, 2026J/K move · Enter open · S save

MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models

Yifan Xu, Chao Zhang, Ruifei Ma, Fei Gao+3

School of Computer Science and Engineering, Beihang University · 2Beijing Digital Native Digital City Research Center · School of Computer Science, Peking University+1

VLM Adaptation Vision-Language Alignment Temporal Video Understanding

Open paper AskPDF arXiv

The new era has witnessed a remarkable capability to extend Vision-Language Models (VLMs) for tackling tasks of video understanding. While current VLMs excel at event- or story-level understanding, their ability to capture fine-grained motion details remains limited, primarily due to their focus on high-level static semantic structures and macro-event logic. In contrast, Video Diffusion Models (VDMs) are adept at modeling dynamic motion patterns, benefiting from large-scale video data and the intrinsic requirement of temporal generation. In this paper, we introduce MotionEnhancer, a novel approach that leverages motion priors distilled from a powerful video diffusion model as auxiliary supervision to enhance the motion understanding capability of a VLM via attention alignment. MotionEnhancer comprises two simple parameter-free modules, Motion-sensitive Head Selection (MHS) and Motion-salient Text Token Identification (MTTI), to directly extract and optimize motion-related attentions from the VDM in a computation-only manner. MotionEnhancer provides a scalable solution for motion understanding without additional training parameters, modifications to existing architectures, or tool calling. Extensive experiments demonstrate that MotionEnhancer can achieve consistent improvements over state-of-the-art VLMs on two motion-level video understanding benchmarks, especially on motion-related metrics.

Abstract

Video Diffusion Models

Explore similar work

MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models

Controlling Motion Transfer in Diffusion Transformers via Attention Heads

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer