SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling

Tianlv Huang; Hetian Guo; Ziyi Cai; Song Wang; Yanping Zhang; Zipei Fan; Xuan Song; Guangming Wu

cs.CVAug 25, 2026

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling

Authors: Tianlv Huang, Hetian Guo, Ziyi Cai, Song Wang, Yanping Zhang, Zipei Fan, Xuan Song, Guangming Wu, +1 more

Organizations: Jilin University · Frontier Robotics · Harbin Institute of Technology, Shenzhen

Abstract

Discrete motion representations have substantially advanced autoregressive text-to-motion generation. However, most motion tokenizers are optimized for reconstruction and do not explicitly allocate capacity according to semantic role. Action-level meaning and fine-grained kinematic detail must therefore be encoded through the same reconstruction-driven hierarchy. We introduce SeMoCo, a semantic-first motion codec, together with a dual-axis motion generator for language-conditioned motion generation. Each motion token contains one semantic token and a residual sequence of kinematic tokens. The generator models semantic progression across time and autoregressively refines the residual entries. We also construct $Ω$ -MotionVerse, a large-scale, multi-source human-motion dataset unified under the SOMA representation. Across the reported comparisons, SeMoCo achieves the best reconstruction accuracy among the compared codecs, while strong text-to-motion results demonstrate the effectiveness of its motion tokens for downstream generation.

Figures & tables

Figure 1: SeMoCo tokenizer. Given an input motion sequence, the encoder produces a continuous latent that is factorized into a semantic stream and a kinematic residual stream. The semantic stream is vector-quantized and aligned with the TMR-SOMA embedding space, whereas RVQ progressively quantizes the residual stream to preserve kinematic detail. Their quantized outputs are summed at the decoder input to reconstruct the motion; semantic supervision is used only during codec training.

Figure 2: Dual-axis motion language model. The temporal Transformer models language-conditioned motion packets across time, while the MTP module completes each packet from its semantic code to the kinematic residual codes. The generated packets are decoded into full-body motion.

Figure 3: $\Omega$ -MotionVerse. Composition of the supporting human motion dataset, comprising around 1,000 hours of text-paired motion sequences.

Figure 4: Qualitative text-to-motion comparison. Generated motion sequences from HY-Motion, Kimodo, and Ours-Base for three representative text prompts, visualized at matched timestamps.

Figure 5: Per-sequence reconstruction error. Cumulative distribution of MPJPE, with each method evaluated on its own test set: ours on the HumanML3D portion of our split ( $n=2{,}108$ ), MoMask and MotionGPT3 on the official HumanML3D test set ( $n=4{,}372$ ), and MotionMillion on its own test set ( $n=4{,}042$ ). Left: 0–100 mm. Right: the upper 12% of each distribution, where the dashed line marks the range of the left panel.

Figure 6: Per-frame vertical acceleration of the right wrist and right ankle on the most dynamic clip of our test set, shown over its busiest 150-frame window with all methods resampled to 30 fps and pelvis aligned. MotionMillion is not included.

Component	Ours-Lite	Ours-Base	Interpretation
Trainable parameters	188.0M	391.1M	Frozen text encoder excluded
Temporal layers	12	24	Base doubles temporal depth
Temporal hidden width	768	1,024	Base uses a wider temporal state
Temporal query/KV heads	6 / 3	8 / 4	128-D heads and 2:1 GQA
Temporal FFN width	2,048	2,730	Scaled with backbone width
Code-predictor width/layers	1,024 / 5	1,024 / 5	Fixed across sizes

Table 14: Ours-Lite and Ours-Base instantiate the same dual-axis generator at different temporal capacities. Parameter counts cover the trainable generator and exclude the frozen text encoder. Both variants are trained for 100,000 updates; reported results use the best validation checkpoint among the 1,000-step validation saves, which for both variants is step 99,000.

Setting	Value	Setting	Value
Optimizer	AdamW	Learning rate	$2\times 10^{-4}$
Weight decay	0.01	Adam betas	$(0.9,0.95)$
LR schedule	4,000-step warmup + cosine	Gradient clipping	1.0
Precision	bf16	Batch size	32 per process
Gradient accumulation	None	Optimization steps	100,000
Data-parallel processes	1 (Lite) / 2 (Base)	Global batch size	32 (Lite) / 64 (Base)

Table 15: Generator training configuration. Ours-Lite and Ours-Base share the same data, objective, optimizer settings, update count, tokenizer, code predictor, and text encoder, and use the same per-process batch of 32. Ours-Lite is trained on one data-parallel process and Ours-Base on two, giving global batch sizes of 32 and 64.

Explore similar work

FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling
Apr 13, 2026Dawei Guan, Di Yang, Chengjie Jin +1Text-to-Motion Generation Human Motion Generation
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China
ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation
May 12, 2026Inwoo Hwang, Hojun Jang, Bing Zhou +3Text-to-Motion Generation Human Motion Generation
Seoul National University · Snap Inc. · Meta Reality Labs
Next-Scale Autoregressive Models for Text-to-Motion Generation
Apr 4, 2026Zhiwei Zheng, Shibo Jin, Lingjie Liu +1Text-to-Motion Generation Human Motion Generation
University of Pennsylvania

Apr 13, 2026J/K move · Enter open · S save

FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling

Dawei Guan, Di Yang, Chengjie Jin, Jiangtao Wang

School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China

Text-to-Motion Generation Human Motion Generation Motion Representation Learning Flow-Based Generative Modeling

Open paper AskPDF arXiv

Text-to-motion generation is driven by learning motion representations for semantic alignment with language. Existing methods rely on either continuous or discrete motion representations. However, continuous representations entangle semantics with dynamics, while discrete representations lose fine-grained motion details. In this context, we propose FlowCoMotion, a novel motion generation framework that unifies both treatments from a modeling perspective. Specifically, FlowCoMotion employs token-latent coupling to capture both semantic content and high-fidelity motion details. In the latent branch, we apply multi-view distillation to regularize the continuous latent space, while in the token branch we use discrete temporal resolution quantization to extract high-level semantic cues. The motion latent is then obtained by combining the representations from the two branches through a token-latent coupling network. Subsequently, a velocity field is predicted based on the textual conditions. An ODE solver integrates this velocity field from a simple prior, thereby guiding the sample to the potential state of the target motion. Extensive experiments show that FlowCoMotion achieves competitive performance on text-to-motion benchmarks, including HumanML3D and SnapMoGen.

Abstract

Explore similar work

FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

Next-Scale Autoregressive Models for Text-to-Motion Generation