Zero-Gated Language-conditioned Human Motion Prediction

Guanhui Qiao; Lu Zhou; Ding Jiang; Jinqiao Wang

cs.CVJun 28, 2026

Zero-Gated Language-conditioned Human Motion Prediction

Authors: Guanhui Qiao, Lu Zhou, Ding Jiang, Jinqiao Wang

Organizations: a Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · b School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · c Wuhan AI Research, Wuhan 430073, China · d Peng Cheng Laboratory, Shenzhen 518066, China

Abstract

Pose histories provide the core kinematic evidence for 3D human motion prediction, but they lack explicit high-level semantic guidance. This paper introduces ZGL, a lightweight language-conditioned predictor that uses captions of the observed motion as a semantic prior while preserving a strong motion backbone as the main source of dynamics. We render only the observed poses, generate a one-sentence description with a vision-language model, encode the caption with a frozen CLIP-L text tower, and project it into a small set of conditioning tokens. These tokens are injected into a DCT-based spatial-temporal Transformer by compact crossattention adapters with zero gates: each adapter output is multiplied by a learnable gate initialized to zero, so the full network is numerically identical to the pose-only baseline at initialization and can learn to use language only when it reduces prediction error. On Human3.6M, ZGL improves overall MPJPE over representative motion-prediction baselines in our comparison. Results on CMUMocap further show that compact caption conditioning transfers to a second benchmark and provides a practical semantic cue for 3D human motion prediction.

Explore similar work

Ego-Human Motion Prediction with 3D-Aware LLM
Jul 8, 2026Yujin Bae, Jaewoo Jeong, Hyeonseong Kim +1Human Motion Prediction Egocentric Vision
Encoder-Free Human Motion Understanding via Structured Motion Descriptions
Apr 23, 2026Yao Zhang, Zhuchenyang Liu, Thomas Ploetz +1Human Motion Generation Video Multimodal Large Language Models
Fine-grained Human Motion Understanding with Language Models
Jun 18, 2026Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew +2Human Motion Generation Motion Generation

Jul 8, 2026 · cs.CVJ/K move · Enter open · S save

Ego-Human Motion Prediction with 3D-Aware LLM

Yujin Bae, Jaewoo Jeong, Hyeonseong Kim, Kuk-Jin Yoon

Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.

Human Motion Prediction Egocentric Vision Motion Generation Cross-Modal Learning

Open paper Ask about it PDF

Abstract

Explore similar work

Ego-Human Motion Prediction with 3D-Aware LLM

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Fine-grained Human Motion Understanding with Language Models