ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning

Eric Nazarenus; Chuqiao Li; Yannan He; Xianghui Xie; Jan Eric Lenssen; Gerard Pons-Moll

cs.CVMar 13, 2026

ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning

Authors: Eric Nazarenus, Chuqiao Li, Yannan He, Xianghui Xie, Jan Eric Lenssen, Gerard Pons-Moll

Organizations: Tübingen AI Center, University of Tübingen, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Abstract

We present ActionPlan, a unified motion diffusion framework that bridges real-time streaming with high-quality offline generation within a single model. The core idea is to introduce a per-frame action plan: the model predicts frame-level text latents that act as dense semantic anchors throughout denoising, and uses them to denoise the full motion sequence with combined semantic and motion cues. To support this structured workflow, we design latent-specific diffusion steps, allowing each motion latent to be denoised independently and sampled in flexible orders at inference. As a result, ActionPlan can run in a history-conditioned, future-aware mode for real-time streaming, while also supporting high-quality offline generation. The same mechanism further enables zero-shot motion editing and in-betweening without additional models. Experiments demonstrate that our real-time streaming is 5.25x faster while also achieving 18% motion quality improvement over the best previous method in terms of FID.

Figures & tables

Figure 1: ActionPlan decouples high-level action planning from low-level motion generation in a single generative model (a). By conditioning motion synthesis on generated action plans, ActionPlan achieves online generation (b) without the typical accuracy drop that happens in existing streaming methods and supports localized edits (c).

Figure 2: Comparison of generation paradigms , where darker shading indicates higher noise levels. By introducing frame-level action plans as semantic conditioning, ActionPlan achieves significantly better FID and R-Precision compared to schedules without ActionPlan. Additionally, our streaming mode completes generation in only $N+T-1$ total steps ( $N$ : motion tokens, $T$ : flow matching steps), enabling efficient low-latency generation without sacrificing motion quality.

Figure 3: Overview of our ActionPlan . (a) During training, motion latents are noised with per-frame heterogeneous timesteps while frame-level text latents share a single global timestep. A Transformer Denoiser is trained to jointly reconstruct both. During inference, the model operates in two modes: in offline mode (b), the action plan is fully generated first and then motion latents are denoised in random pyramid order; in streaming mode (c), the action plan is denoised alongside the first motion frame, followed by raster progressive denoising of the remaining latents.

Figure 4: Qualitative comparison with MARDM [ 41 ] and MotionStreamer [ 60 ] on four text prompts. The color varies from light to dark representing the time flow. Incorrectly generated or missing actions are marked with $\times$ , and correctly executed actions with $\checkmark$ . By generating a frame-level action plan prior to motion synthesis, ActionPlan faithfully executes all specified actions in the correct order, while baselines frequently miss or misorder key actions. See Supp. for videos.

Method	FID $\downarrow$	R-Precision $\uparrow$			MatchS $\downarrow$	Div $\rightarrow$
Method	FID $\downarrow$	Top 1	Top 2	Top 3	MatchS $\downarrow$	Div $\rightarrow$
Real	0.002	0.702	0.864	0.914	15.151	27.492
MDM [ 54 ]	23.454	0.523	0.692	0.764	17.423	26.325
MLD [ 11 ]	18.236	0.546	0.730	0.792	16.638	26.352
T2M-GPT [ 63 ]	12.475	0.606	0.774	0.838	16.812	27.275
MotionGPT [ 28 ]	14.375	0.456	0.598	0.628	17.892	27.114

Table 1: Quantitative comparison with SOTA T2M generation methods on HumanML3D-272 [ 21 , 60 ] test set. MatchS and Div denote the matching score and diversity respectively. Bold indicates best results, underline indicates second best. ActionPlan achieves better performance in both offline and online-streaming mode, maintaining both efficiency and high quality motion generation.

Method	First $\downarrow$	Others $\downarrow$
MARDM	210	210
MotionStreamer	360	360
Ours	146	40

Table 6

Configurations	Training	ActionPlan	Mode	FID $\downarrow$	R-Precision $\uparrow$			MatchS $\downarrow$	Div $\rightarrow$
Configurations	data	generation	Mode	FID $\downarrow$	Top 1	Top 2	Top 3	MatchS $\downarrow$	Div $\rightarrow$
Real motion				0.002	0.702	0.864	0.914	15.151	27.492
(A) No frame-level text	partial	✗	Offline	9.674	0.634	0.790	0.851	15.764	27.207
(A) No frame-level text	partial	✗	Streaming	10.693	0.627	0.785	0.848	15.931	27.212
(B) Frame-level text w/ actionplan	partial	✓	Offline	7.863	0.625	0.791	0.854	15.849	27.232
(B) Frame-level text w/ actionplan	partial	✓	Streaming	8.018	0.625	0.789	0.854	15.866	27.220

Table 2: Ablation studies for both Offline and Streaming modes. Training on the full dataset (C–E) consistently outperforms the partial intersection subset (A–B), validating our masked loss design. Frame-level text prediction (D, E) improves over no frame text (C). Action plan generation (E) further outperforms joint co-generation (D). These gains hold across both inference modes. Bold : best, underline : second best.

Progressive Sampling	FID $\downarrow$	R-Precision $\uparrow$			MatchS $\downarrow$	Div $\rightarrow$
Progressive Sampling	FID $\downarrow$	Top 1	Top 2	Top 3	MatchS $\downarrow$	Div $\rightarrow$
Real	0.002	0.702	0.864	0.914	15.151	27.492
Fully overlap (Parallel)	5.420	0.681	0.836	0.886	15.104	26.989
2-step non-overlap	5.522	0.687	0.838	0.892	15.086	27.272
5-steps non-overlap	5.721	0.687	0.834	0.885	15.128	27.179
10-steps non-overlap	5.605	0.685	0.832	0.885	15.128	27.261

Table 3: Ablation on sampling strategy. All rows use the same trained model with a fixed total of 25 denoising steps, differing only in the overlap between consecutively activated motion latents, ranging from fully parallel activation to fully sequential (non-overlapping) denoising. Our chosen schedule denoises each activated latent for 2 steps before activating the next, achieving the best trade-off between motion quality (FID) and text-motion alignment (R-Precision).

Figure 5: ActionPlan supports diverse downstream applications zero-shot . Darker shading indicates later time steps. Editing (top): regenerates selected latents conditioned on a new prompt ( green ) while preserving others. Long motion streaming (middle): generates coherent long-horizon motion in successive chunks across prompts. In-betweening (bottom): given fixed start (white) and end (dark grey) poses, fills in the intermediate motion. See Supp. for video results.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: User study interfaces. (a) Long-sequence streaming evaluation: participants compare two animations (left vs. right) generated from the same sequence of textual prompts. Note that the left side is always shown in blue and the right side is always shown in red, while the actual method assigned to each side is randomized. (b) Text-to-motion evaluation: participants compare three animations (left, middle, right) generated from the same single prompt. In both studies, method assignment is randomized and participants select the best motion based on semantic consistency and realism.

Explore similar work

FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation
Sep 27, 2026Yiyi Cai, Yuhan Wu, Kunhang Li +6Motion Diffusion Models Human Motion Generation
Alaya Lab · The University of Tokyo · Japan Advanced Institute of Science and Technology +1
Causal Motion Diffusion Models for Autoregressive Motion Generation
Feb 26, 2026Qing Yu, Akihisa Watanabe, Kent FujiwaraHuman Motion Generation Diffusion Models for Motion Generation
LY Corporation · Waseda University
Rethinking Streaming Video Diffusion Model: Context, Execution, and Training
Sep 13, 2026Hongchen ZhangStreaming Video Generation Diffusion Model Inference Acceleration
University of Chinese Academy of Sciences

Sep 27, 2026J/K move · Enter open · S save

FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation

Yiyi Cai, Yuhan Wu, Kunhang Li, Tu Fangyuan+5

Alaya Lab · The University of Tokyo · Japan Advanced Institute of Science and Technology+1

Motion Diffusion Models Human Motion Generation Diffusion Models for Motion Generation Diffusion Model Inference Acceleration Diffusion Models

Open paper AskPDF arXiv

We present FloodDiffusion 2 (FD2), an efficient and controllable framework that builds upon FloodDiffusion (FD1), a state-of-the-art streaming motion generation model. While FD1 produces plausible motion, it suffers from low efficiency and limited controllability, as its attention design requires repeated computation over the entire history, and it lacks precise trajectory control for real-world applications. To address these limitations and improve generation quality, FD2 introduces three advances. First, Partial Attention makes finalized history representations independent of the active window, enabling KV-cached inference and shared-history packing for efficient training. Second, we establish a necessary-and-sufficient Bregman criterion for regression losses to preserve diffusion's conditional-mean velocity field. This criterion guides an FK-induced quadratic loss that incorporates motion geometry without online FK evaluation. Third, FD2 introduces precise path conditioning to control the character's root trajectory while preserving natural body motion. Experiments show that FD2 reduces training computation by 4.6

\times

and accelerates denoising by 11.29

\times

, reaching 2.303 ms per update on long sequences. Alongside these efficiency gains, FD2 improves motion quality over FD1 and achieves state-of-the-art FID scores among streaming methods, with 0.048 on SEED and 0.053 on HumanML3D.

Abstract

Explore similar work

FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation

Causal Motion Diffusion Models for Autoregressive Motion Generation

Rethinking Streaming Video Diffusion Model: Context, Execution, and Training