ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation

Bishoy Galoaa; Sarah Ostadabbas

cs.CVOct 18, 2026

ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation

Authors: Bishoy Galoaa, Sarah Ostadabbas

Organizations: Northeastern University

Abstract

Language-guided video generation often captures scene content while missing the requested motion. We present ViMoLa (Video Motion Language), which predicts a motion prior from language and injects it into a frozen image-to-video backbone. A spatial vector-quantized autoencoder compresses dense trajectories into 64 tokens from a 512-entry codebook; a 28M-parameter transformer predicts the tokens from text; and zero-gated cross-attention adapters condition every frozen Wan~~2.2 DiT block through forward hooks. Trained on 462K videos from Something-Something~~v2 and Kinetics-400, ViMoLa raises directional accuracy from 0.287 to 0.442 (chance 0.250), with similar measured generative quality. Across-prompt shuffled codes achieve 0.283, supporting the importance of prompt-matched token content. A low-rank adapter reaches 0.428 at 1.45% additional forward-pass FLOPs, and reusing the tokenizer and language model with newly trained CogVideoX adapters raises accuracy from 0.301 to 0.398. Control remains imperfect: oracle codes reach 0.531, compound-motion gains are smaller, and the adapter does not explicitly preserve token-to-patch assignments.

Figures & tables

Figure 2: ViMoLa overview. At inference, a text prompt is encoded by UMT5 and mapped by TextToMotionTransformer to $64$ discrete motion-token indices over a $512$ -entry codebook ( Text-to-Motion ). A frozen codebook lookup (SpatialMotionVQVAE) turns these into the $64$ -patch motion field they encode ( Motion Tokenizer ). A zero-gated cross-attention WanMotionAdapter then injects the motion embeddings into every block of the frozen Wan 2.2 DiT through forward hooks ( Motion Adapter ); block shading and the bar chart show the learned per-block injection strength $|\tanh(\alpha)|$ , which varies across depth. Note the motion field: marker patches point right as prompted while the surrounding scene points left. The decoded grid illustrates the tokenizer representation; the adapter receives its embeddings without explicit patch positions. The frozen DiT then denoises and its VAE decodes the clip ( Output ), here “pushing white book marker left to right” on a held-out SSV2 example. The first frame conditions the backbone, not TextToMotionTransformer. Only the adapters are trained in the final stage.

Figure 3: ViMoLa qualitative results on in-the-wild images sourced from the internet, not from SSV2 or K400. Across “a bird takes off to the left” vs . “a bird flies away and then lands on the same tree branch,” and “the car drives forward” vs . “backwards” on the same road, the prompts vary the requested motion within a scene. These examples illustrate behavior; average subject consistency is measured in Tab. A3 and does not establish object binding. Failure cases, concentrated on the compound prompts of Tab. A4 , and the learned per-block gates are in Appendix G .

Method	Conditioning	DA $\uparrow$	Traj. L2 $\downarrow$	SSV2	K400	CLIP $\uparrow$	Motion Smooth. $\uparrow$
				FVD $\downarrow$
CogVideoX [ 30 ]	image + text	$0.301$	$0.789$	$479.9$	$577.1$	$0.209$	$0.444$
Wan 2.2 (no adapter) [ 22 ]	image + text	$0.287$	$0.812$	$179.4$	320.3	$0.221$	0.472
AnimateDiff + MotionLoRA [ 9 , 11 ]	text only	—	$0.820$	$971.7$	$454.4$	$0.222$	$0.367$
ViMoLa (ours)	image + text	0.442	0.734	168.9	$328.5$	0.229	$0.455$

Table 1: Main comparison across motion control (DA, Traj. L2), generative quality (FVD, Motion Smoothness), and text-video alignment (CLIP). FVD, Traj. L2, CLIP and smoothness use

5{,}000

held-out clips per dataset; DA uses the fixed

300

-clip control subset, chance

=0.250

. Each method runs under its standard conditioning and none receives an explicit motion signal; DA is omitted for AnimateDiff+MotionLoRA, which takes no conditioning frame and so cannot be run on the matched first-frame subset. Best per column in bold .

Motion source	DA $\uparrow$	Traj. L2 $\downarrow$
no adapter	$0.287$	$0.812$
shuffled codes	$0.283$	$0.806$
predicted codes	$\mathbf{0.442}$	$\mathbf{0.734}$
ground truth (oracle)	$0.531$	$0.702$
cont. T2M $\to$ SG-I2V [ 18 ]	$0.352$	$0.781$
cont. T2M $\to$ our adapter	$0.331$	$0.792$

Table 2: Where the control comes from. Same

300

clips; chance DA

=0.250

. Top: The same adapter receives across-prompt shuffled, predicted, or ground-truth codes. The oracle uses ground-truth motion at inference; other deployable rows predict motion from text. Shuffling tests prompt-matched content, not spatial arrangement. Bottom: Continuous trajectory prediction from text.

Backbone	Adapter	DA $\uparrow$	$\Delta$ DA	DINO SC $\uparrow$
Wan 2.2 5B	none	$0.287$	—	$0.743$
Wan 2.2 5B	ViMoLa	$0.442$	+0.155	$0.731$
CogVideoX 5B	none	$0.301$	—	$0.795$
CogVideoX 5B	ViMoLa	$0.398$	+0.097	$0.788$

Table 3: Transfer to CogVideoX-5B-I2V. The tokenizer and text-to-motion predictor are reused unchanged; only generator-side adapters are retrained. DA improves on both backbones, with a smaller gain on CogVideoX. DINO SC is subject consistency; chance DA

=0.250

Variant	Params	% FLOPs	Infer. $\Delta$	VRAM	DA gap
Full	$1.14$ B	$7.88\%$	$+0.31$ s	$62.4$ GB	$+0.159$
Rank $256$	$\mathbf{201}$ M	$\mathbf{1.45\%}$	$\mathbf{+0.09}$ s	$\mathbf{41.7}$ GB	$+0.145$

Table 4: Cost of control. Matched training budget; FLOPs are relative to a frozen DiT forward pass. Inference overhead is measured per clip on one H100. Control is the DA gap over across-prompt shuffled codes ( Tab. 2 ).

Split	$n$	Wan	ViMoLa	$\Delta$
Single-phase	$300$	$.291$	$.478$	$+.187$
Multi-phase	$252$	$.286$	$.334$	$+.048$
Compound	$150$	$.253$	$.307$	$+.054$

Table 5: Complex-motion control. DA at

s{=}0.06

, chance

0.250

. Split construction and tokenizer reconstruction diagnostics are in Tab. A4 .

	Variant	FVD $\downarrow$	Motion $\uparrow$	CLIP $\uparrow$
Gate	zero-gated (ours)	168.9	$0.455$	0.229
Gate	ungated †	$7576.1$	$0.467$	$0.155$
Injection	all $30$ blocks (ours)	168.9	$0.455$	0.229
	first $15$ blocks †	$542.1$	$0.464$	$0.219$
	last $15$ blocks †	$511.6$	0.488	$0.224$

Table 6: Adapter design and operating point ,

s{=}0.06

unless swept; chance DA

=0.250

. (top) Ablations over the choices of Sec. 3.4 ; the first row of each group repeats the default of Tab. 1 . † Inference-time probe. (bottom) The knob

s

rescales every block’s gate, so control and quality read off a single row.

$K$	Token Top-1 $\uparrow$	Decoded EPE $\downarrow$	DA $\uparrow$
$128$	$22.1\%$	$0.198$	$0.431$
$256$	$14.2\%$	$0.191$	$0.438$
$\mathbf{512}$	$\mathbf{8.6\%}$	$\mathbf{0.188}$	$\mathbf{0.442}$
$1024$	$5.1\%$	$0.187$	$0.445$

Table 7: Codebook size and motion fidelity. Hard token accuracy falls as

K

grows, while decoded motion EPE and DA remain stable (chance DA

=0.250

). Token diagnostics use

4{,}000

validation clips; bold marks the default

K{=}512

. Reconstruction and

\varepsilon

-soft accuracy are reported in Tab. A5 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure A1 : Pipeline visualization. (a) A raw SSV2 video frame. (b–c) The CoTracker3 motion field overlaid on and separated from the frame. (d) Trajectory traces integrated from the $32{\times}32$ tracked grid over $49$ frames. (e) The $8{\times}8=64$ discrete VQ token assignments produced by SpatialMotionVQVAE (color = codebook index mod 32). (f) The velocity field reconstructed by the VQ decoder — faithful to (c). (g) Per-block gate activation $\tanh(\alpha_{i})$ across the $30$ DiT blocks after adapter training. All gates start at $0$ ; at default inference strength $s{=}0.06$ the mean activation is ${\approx}0.028$ , confirming that the motion signal is injected as a small, controlled residual.

Figure A3 : “holding paper in front of cabinet”

Figure A4 : “pretending to put keys into bag”

Figure A5 : “tearing a piece of paper into two pieces”

Figure A11 : ViMoLa’s TextToMotionTransformer field decoded from “pushing a boulder up a steep hill.” Predicted from language alone; no tracks are supplied at any point.

Figure A12 : Top: SG-I2V, driven by ground-truth trajectories supplied at inference. Middle: ViMoLa, conditioned only on the first frame and the caption, through the predicted field of Fig. A11 . Bottom: Wan 2.2, same prompt and frame with the adapter off. This is an out-of-distribution artistic image, shown at $s{=}0.1$ rather than the default $0.06$ (main-paper Table 6, lower panel).

Component	Status	Params
SpatialMotionVQVAE	Frozen †	$2.7$ M
TextToMotionTransformer	Frozen †	$27.6$ M
WanMotionAdapter ( $\times 30$ )	Trained	1,144.6M
low-rank, $r{=}256$	Trained	201.0M
Wan 2.2 DiT	Frozen	$5{,}000$ M
Wan 2.2 VAE	Frozen	$704.7$ M

Appendix

Table A1 : Parameter budget. Only the per-block adapters are trained during the final stage; all backbone weights remain frozen. † Trained in an earlier stage and frozen here. The low-rank adapter (

r{=}256

, main-paper Sec. 3.4) is a drop-in replacement for the full adapter at

18\%

of its trainable size; Tab. 4 reports the compute cost of both.

Split	Top-1 $\uparrow$	Top-5 $\uparrow$	$H$	Prior	Lift $\uparrow$
SSV2 (templated)	$7.5\%$	$15.4\%$	$5.77$	$2.4\%$	$\mathbf{3.1\times}$
K400 (free-form)	$\mathbf{15.9\%}$	$\mathbf{24.5\%}$	$5.37$	$12.7\%$	$1.25\times$

Appendix

Table A2 : Text-to-motion by caption type ,

K{=}512

. SSV2’s templated captions induce a higher-entropy , harder code distribution than K400’s, with a far weaker majority-code prior, which is why SSV2’s raw Top-1 is lower despite its cleaner supervision. Measured as lift over that prior, the ordering inverts and matches intuition. Pooled over both splits, Top-1 is

8.6\%

and Top-5

16.6\%

. The codebook-size sweep behind these numbers is in Tab. A5 .

Metric	ViMoLa	Wan 2.2
FVD SSV2 $\downarrow$	$\mathbf{168.9}$ $[161,177]$	$179.4$ $[172,188]$
FVD K400 $\downarrow$	$328.5$	$\mathbf{320.3}$
CLIP $\uparrow$	$\mathbf{0.229}$	$0.221$
Motion Smooth. $\uparrow$	$0.455$	$\mathbf{0.472}$
DINO subj. cons. $\uparrow$	$0.731$ $[.712,.749]$	$\mathbf{0.743}$ $[.725,.760]$
Backgr. cons. $\uparrow$	$0.812$	$\mathbf{0.821}$

Appendix

Table A3 : Quality and identity at

s{=}0.06

. Brackets are supplied

95\%

confidence intervals, which overlap for SSV2 FVD and DINO subject consistency. Other quality scores are point estimates. DA is reported over three seeds; frame-

0

error measures deviation from the conditioning image.

Split	$n$	ViMoLa DA $\uparrow$	Wan DA $\uparrow$	$\Delta$
single-phase	$300$	$\mathbf{0.478}$	$0.291$	+0.187
multi-phase (auto)	$252$	$0.334$	$0.286$	$+0.048$
compound (curated)	$150$	$0.307$	$0.253$	$+0.054$

Appendix

Table A4 : Multi-phase and compound motion at

s{=}0.06

; chance DA

=0.250

. Top: separate split sizes and end-to-end DA gains. Bottom: tokenizer reconstruction on the multi-phase split, using

S

temporal segments. Segmentation improves reconstruction; generation with segmented tokens was not evaluated.

$K$	Recon. EPE $\downarrow$	Top-1 $\uparrow$	$\varepsilon$ -soft Top-1 $\uparrow$	Decoded EPE $\downarrow$	DA $\uparrow$
$128$	$0.112$	$\mathbf{22.1\%}$	$\mathbf{0.594}$	$0.198$	$0.431$
$256$	$0.096$	$14.2\%$	$0.548$	$0.191$	$0.438$
$\mathbf{512}$	$\mathbf{0.084}$	$8.6\%$	$0.512$	$0.188$	$0.442$
$1024$	$0.079$	$5.1\%$	$0.487$	$\mathbf{0.187}$	$\mathbf{0.445}$

Appendix

Table A5 : Why a low Top-1 is not a low-fidelity motion prior. Codebook size

K

d{=}128

, on

4{,}000

validation clips. Hard Top-1 falls by

4.3\times

as the vocabulary grows, while every measure of the decoded motion — end-point error, directional accuracy, and the tolerance-based

\varepsilon

-soft Top-1 ( Appendix E ) — stays flat: at large

K

the codebook holds many near-duplicate patterns, so an “incorrect” code is usually a near-synonym of the correct one. We adopt

K{=}512

, where reconstruction has largely converged without the diminishing returns of

K{=}1024

Codebook $K$	Latent $d$	# Tokens	Recon. $L_{2}$ $\downarrow$	Perplexity $\uparrow$
$512$	$64$	$64$	$3.19{\times}10^{-4}$	$168.2$
$\mathbf{512}$	$\mathbf{128}$	$\mathbf{64}$	$\mathbf{3.15{\times}10^{-4}}$	$\mathbf{290.7}$
$512$	$256$	$64$	$3.19{\times}10^{-4}$	$171.2$

Appendix

Table A6 : Motion tokenizer, latent dimension. Reconstruction error and codebook utilization at

K{=}512

as the latent width

d

varies, on the full held-out set.

d{=}128

gives the best reconstruction

L_{2}

and by far the healthiest codebook occupancy, which is why we adopt it; the codebook-size sweep at

d{=}128

is in Tab. A5 .

Conditioning	Decoded EPE $\downarrow$
random code	$0.417$
per-patch majority code	$0.291$
predicted code (TextToMotionTransformer)	$\mathbf{0.188}$
ground-truth code (tokenizer ceiling)	$0.084$

Appendix

Table A7 : What the predicted codes are worth in motion space. End-point error of the field decoded from each source of codes,

K{=}512

4{,}000

validation clips. The predicted codes recover most of the distance between an uninformative prior and the tokenizer’s own ceiling, even though their hard Top-1 is

8.6\%

Question	ViMoLa win rate	$95\%$ CI
Direction match	$\mathbf{68.4\%}$	$[63.1,\,73.3]$
Speed match	$\mathbf{61.2\%}$	$[55.7,\,66.4]$
Overall quality	$49.1\%$	$[43.6,\,54.6]$

Appendix

Table A8 : Blind pairwise human study.

7

raters

\times

150

pairs, matched seeds and matched first frames, so the two clips in a pair differ only in whether the adapter was attached. Raters prefer ViMoLa on direction and on speed and are at chance on overall quality — the same conclusion the automatic metrics of Tab. A3 reach, from an independent source.

Figure A14 : “moving the bottle away from the pencil case”

Figure A15 : “moving mug and glass so they collide with each other”

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 111 papers

Language-Guided Transformer Tokenizer for Human Motion Generation
Feb 9, 2026Sheng Yan, Yong Wang, Xin Du +2Text-to-Motion Generation Human Motion Generation
1Transsion Ltd. · 2Chongqing University of Technology · 3Afari Intelligence Drive +2
MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
Dec 15, 2025Yannan He, Garvita Tiwari, Xiaohan Zhang +4Latent Diffusion Models Text-to-Motion Generation
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2
SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling
Aug 25, 2026Tianlv Huang, Hetian Guo, Ziyi Cai +6Text-to-Motion Generation Human Motion Generation
Jilin University · Frontier Robotics · Harbin Institute of Technology, Shenzhen

Feb 9, 2026J/K move · Enter open · S save

Language-Guided Transformer Tokenizer for Human Motion Generation

Sheng Yan, Yong Wang, Xin Du, Junsong Yuan+1

1Transsion Ltd. · 2Chongqing University of Technology · 3Afari Intelligence Drive+2

Text-to-Motion Generation Human Motion Generation

Open paper AskPDF arXiv

In this paper, we focus on motion discrete tokenization, which converts raw motion into compact discrete tokens--a process proven crucial for efficient motion generation. In this paradigm, increasing the number of tokens is a common approach to improving motion reconstruction quality, but more tokens make it more difficult for generative models to learn. To maintain high reconstruction quality while reducing generation complexity, we propose leveraging language to achieve efficient motion tokenization, which we term Language-Guided Tokenization (LG-Tok). LG-Tok aligns natural language with motion at the tokenization stage, yielding compact, high-level semantic representations. This approach not only strengthens both tokenization and detokenization but also simplifies the learning of generative models. Furthermore, existing tokenizers predominantly adopt convolutional architectures, whose local receptive fields struggle to support global language guidance. To this end, we propose a Transformer-based Tokenizer that leverages attention mechanisms to enable effective alignment between language and motion. Additionally, we design a language-drop scheme, in which language conditions are randomly removed during training, enabling the detokenizer to support language-free guidance during generation. On the HumanML3D and Motion-X generation benchmarks, LG-Tok achieves Top-1 scores of 0.542 and 0.582, outperforming state-of-the-art methods (MARDM: 0.500 and 0.528), and with FID scores of 0.057 and 0.088, respectively, versus 0.114 and 0.147. LG-Tok-mini uses only half the tokens while maintaining competitive performance (Top-1: 0.521/0.588, FID: 0.085/0.071), validating the efficiency of our semantic representations. Code and checkpoints are available at https://eanson023.github.io/LG-Tok/

Abstract

T2V Generation

Explore similar work

Language-Guided Transformer Tokenizer for Human Motion Generation

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling