Streaming Multi-Track Timeline Control for 3D Human Motion Generation

Yangsong Zhang; Anujith Muraleedharan; Rikhat Akizhanov; Gül Varol; Fabio Pizzati; Ivan Laptev

cs.CVOct 2, 2026

Streaming Multi-Track Timeline Control for 3D Human Motion Generation

Authors: Yangsong Zhang, Anujith Muraleedharan, Rikhat Akizhanov, Gül Varol, Fabio Pizzati, Ivan Laptev

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS

Abstract

Text-driven human motion generation has advanced substantially, yet most methods assume instructions are available before synthesis. Interactive applications require responding to new instructions while continuing ongoing actions, such as answering a phone while walking. Existing approaches address streaming generation or simultaneous composition without explicitly combining streaming instruction arrival with independently timed, overlapping actions. We introduce streaming multi-track timeline control and propose TimelineControl to incorporate new instructions alongside ongoing actions. Interval-aware conditioning preserves instruction timing, while causal part-structured representations and part-aware denoising coordinate concurrent actions across body regions. We also construct TimelineMotion, a dataset with overlapping instruction intervals and body-part annotations. Experiments on TimelineMotion and MTT demonstrate improved semantic alignment and temporal adherence over evaluated streaming baselines, including models retrained on the same data. Ablations and human evaluations validate our design, complemented by spatial conditioning and humanoid execution demonstrations. Our code, data and models will become publicly available.

Figures & tables

Figure 1: Streaming multi-track motion generation with TimelineControl. (a) Sequential and overlapping body-part instructions. (b) Joint text, root-path, and heading control. (c) Humanoid robot execution. In (a) and (b), body-part colors correspond to instructions.

Figure 2: TimelineControl. We process part features and whole-body features separately (left). The part features undergo block-causal self-attention (yellow). Both use online multi-prompt conditioning (pink, OMPC) to allow for multi-track streaming inference. We add a part bias (orange) to enforce better mapping to body parts. Details (right) show examples of elements used in text.

	Method	Stream	Sequence alignment					Body-part alignment			Motion quality			Temporal adherence
	Method	Stream	R@1 32 $\uparrow$	R@2 32 $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	M2M $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	M2M $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans $\downarrow$	AP $\uparrow$	AUC $\uparrow$
TimelineMotion	STMC	✗	0.329	0.452	0.525	0.627	0.630	0.353	0.538	0.633	0.055	0.228	0.92	0.803	0.732
	FrankenMotion	✗	0.106	0.178	0.232	0.538	0.551	0.270	0.526	0.568	0.159	0.246	1.01	0.639	0.545
	ActionPlan	✓	0.248	0.344	0.405	0.585	0.585	0.303	0.547	0.589	0.274	0.330	2.16	0.733	0.653
	ActionPlan ∗	✓	0.320	0.436	0.502	0.622	0.627	0.347	0.546	0.634	0.055	0.209	0.92	0.784	0.711
	MotionStreamer ∗	✓	0.222	0.321	0.383	0.585	0.592	0.304	0.541	0.608	0.049	0.218	1.09	0.734	0.653

Table 1: Benchmarks with streaming instructions. We outperform all the baselines on fine-grained evaluation about sequence alignment, body parts, motion quality and temporal adherence with the sequence, on both TimelineMotion and MTT.

	Method	R@1 32 $\uparrow$	R@2 32 $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	M2M $\uparrow$	FID $\downarrow$
HumanML3D	STMC	0.611	0.756	0.815	0.741	0.697	0.224
	FrankenMotion	0.432	0.566	0.643	0.677	0.689	0.079
	ActionPlan	0.618	0.768	0.829	0.724	0.769	0.022
	ActionPlan ∗	0.645	0.778	0.830	0.734	0.765	0.044
	MotionStreamer ∗	0.568	0.706	0.772	0.726	0.742	0.047
	DartControl ∗	0.536	0.699	0.775	0.711	0.657	0.345

Table 2: Results on non-streaming generation. We compare with motion generation methods on HumanML3D dataset.

Figure 3: Qualitative evaluation. We compare STMC, ActionPlan* and TimelineControl on TimelineMotion (left) and MTT (right) dataset. Red box marks the failure part. TimelineControl is the only one not failing by ignoring instructions or confusing them.

Autoencoder	Dims	Params (M)	HumanML3D				TimelineMotion
			Reconstruction		Generation		Reconstruction		Generation			Temporal adherence
			FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	R@3 32 $\uparrow$	FID $\downarrow$	MPJPE $\downarrow$	FID $\downarrow$	FID+ $\downarrow$	R@3 32 $\uparrow$	AP $\uparrow$	AUC $\uparrow$
TAE ( Xiao et al., 2025 )	16	76.20	0.0028	21.9	0.047	0.833	0.0020	21.7	0.051	0.203	0.533	0.823	0.755
TAE ( Xiao et al., 2025 )	48	79.42	0.0005	3.19	0.049	0.851	0.0002	3.77	0.040	0.197	0.576	0.827	0.762
SALAD ( Hong et al., 2025 )	48	0.51	0.0034	24.1	0.030	0.858	0.0036	27.7	0.023	0.208	0.575	0.820	0.755
Ours w/o sk.graph	48	4.80	0.0012	7.46	0.038	0.851	0.0007	8.27	0.025	0.209	0.600	0.832	0.767

Table 3: Autoencoder benchmarks. Quantitative comparison of different autoencoder designs in reconstruction and generation in both HumanML3D and TimelineMotion.

Setup	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans $\downarrow$	AP $\uparrow$	AUC $\uparrow$
Full model (Ours)	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
RoTE only	0.830	0.753	0.044	0.586	0.649	0.370	0.552	0.028	0.197	1.28	0.820	0.752
w/o RoTE	0.820	0.748	0.061	0.572	0.648	0.372	0.550	0.037	0.195	1.09	0.801	0.734
w/o part-aware bias $B$	0.850	0.764	0.038	0.591	0.650	0.357	0.547	0.025	0.193	1.20	0.830	0.769

Table 4: Architectural ablations. Ablation of temporal conditioning and part-aware denoising on HumanML3D and TimelineMotion. Each variant removes the indicated component from TimelineControl. All our contributions together yield the best performance.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Construction of TimelineMotion. (a) HumanML3D source motions and captions. (b) Body-part annotations. (c) Compatible instructions arranged into multi-track timelines. (d) STMC-based body-part composition, transition refinement, and quality filtering.

Figure 5: Causal part-structured autoencoder. Causal temporal convolutions and skeleton-graph message passing encode and reconstruct six anatomical body regions. The root trajectory bypasses the body encoder and conditions the decoder.

Interval shift	AP $\uparrow$	ROC-AUC $\uparrow$
- 2 s	0.440	0.564
- 1 s	0.580	0.675
- 0.5 s	0.630	0.712
0	0.647	0.728
+ 0.5 s	0.622	0.713
+ 1 s	0.556	0.671

Appendix

Table 5: Sensitivity of temporal adherence metrics under shifts of the requested instruction intervals on TimelineMotion. Best and second-best scores are bold and underlined.

Figure 6: User study interface. We present timeline with videos generated by TimelineControl and one of the baselines. Users are asked to compare the two videos in terms of text-motion alignment, motion quality, and temporal adherence.

Figure 7: User study. Comparison for timeline human motion videos generated by TimelineControl, STMC and ActionPlan*.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Full model (Ours)	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
w/o self-conditioned training	0.862	0.767	0.035	0.594	0.650	0.365	0.551	0.024	0.198	1.35	0.832	0.770

Appendix

Table 6: Effect of self-conditioned training on motion generation on HumanML3D and TimelineMotion. Best results are bold.

Training-window strategy	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Random start	0.859	0.762	0.035	0.597	0.651	0.374	0.553	0.027	0.187	1.48	0.829	0.765
Mixture (Ours)	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
Instruction start	0.866	0.764	0.029	0.596	0.652	0.371	0.551	0.019	0.196	1.28	0.831	0.768

Appendix

Table 7: Comparison of training-window strategies on HumanML3D and TimelineMotion.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Sigmoid (Ours)	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
Softmax	0.847	0.754	0.041	0.580	0.644	0.374	0.551	0.027	0.190	1.23	0.825	0.757

Appendix

Table 8: Comparison of sigmoid weighting and softmax normalization in multi-track attention on HumanML3D and TimelineMotion.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
ActionPlan	0.829	0.724	0.022	0.405	0.585	0.303	0.547	0.274	0.330	2.16	0.733	0.653
ActionPlan ∗	0.830	0.734	0.044	0.502	0.622	0.347	0.546	0.055	0.209	0.92	0.784	0.711
ActionPlan ∗ + multi-text	0.845	0.731	0.027	0.566	0.640	0.352	0.547	0.036	0.198	1.31	0.805	0.745
TimelineControl	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772

Appendix

Table 9: Comparison of joint retraining and multi-text conditioning variants of ActionPlan with TimelineControl on HumanML3D and TimelineMotion. ∗ denotes joint retraining on both datasets. Multi-text uses separately encoded instructions with an interval visibility mask. Best and second-best results are bold and underlined.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Full model	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
w/o block-causal self-attention	0.870	0.770	0.023	0.586	0.649	0.364	0.553	0.019	0.212	1.22	0.826	0.767

Appendix

Table 10: Comparison of block-causal and standard causal self-attention on HumanML3D and TimelineMotion.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Full model	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
w/o online multi-track attention	0.851	0.764	0.048	0.552	0.642	0.372	0.551	0.046	0.179	1.14	0.811	0.747

Appendix

Table 11: Effect of online multi-track attention on HumanML3D and TimelineMotion.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality			Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Full model	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	0.832	0.772
w/o $\mathcal{L}_{\mathrm{part}}$	0.865	0.764	0.028	0.596	0.651	0.371	0.552	0.019	0.195	1.26	0.826	0.762

Appendix

Table 12: Effect of the auxiliary body-part loss on HumanML3D and TimelineMotion.

Variant	HumanML3D			TimelineMotion
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	Sequence alignment		Body-part alignment		Motion quality					Temporal adherence
	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	R@3 32 $\uparrow$	M2T $\uparrow$	R@3 32 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Trans (cm) $\downarrow$	Root jitter $\downarrow$	Body jitter $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
Full model	0.874	0.769	0.027	0.603	0.653	0.374	0.553	0.017	0.192	1.31	1.417	0.614	0.832	0.772
w/o $\mathcal{L}_{\mathrm{root}}$	0.872	0.771	0.026	0.595	0.653	0.372	0.555	0.021	0.187	1.27	1.757	0.680	0.827	0.763

Appendix

Table 13: Effect of the root-motion loss on HumanML3D and TimelineMotion.

Figure 8: Additional qualitative evaluation. Red box marks the failure part.

Testing Dataset	Method	Horizontal root (cm) $\downarrow$	Body (cm) $\downarrow$	End-effector (cm) $\downarrow$	Contact F1 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	Temporal adherence
Testing Dataset	Method	Horizontal root (cm) $\downarrow$	Body (cm) $\downarrow$	End-effector (cm) $\downarrow$	Contact F1 $\uparrow$	M2T $\uparrow$	FID $\downarrow$	FID+ $\downarrow$	AP $\uparrow$	ROC-AUC $\uparrow$
HumanML3D	ARDY	2.55	8.15	14.06	0.857	0.650	0.246	–	–	–
HumanML3D	TimelineControl	4.01	5.42	7.95	0.939	0.786	0.020	–	–	–
TimelineMotion	ARDY	4.13	9.98	16.88	0.897	0.565	0.221	0.308	0.715	0.630
TimelineMotion	TimelineControl	6.31	8.05	12.63	0.942	0.644	0.013	0.216	0.820	0.759

Appendix

Table 14: Comparison with ARDY for motion generation conditioned jointly on text and spatial targets on HumanML3D and TimelineMotion.

Figure 9: Motion generation with text and spatial controls: (a) root trajectories, (b) root waypoints, (c) left-wrist targets, (d) full-body keyframes, and root trajectories combined with (e) left-wrist targets or (f) full-body keyframes. Poses share a world coordinate frame within each panel and darken over time. Bars mark instruction intervals.

Figure 10: Text-driven G1 simulation. (a) A right-foot kick followed by overlapping instructions to jump backward and talk on the phone with the right hand. (b) Overlapping violin-playing and backward-walking instructions followed by jumping jacks. Bars indicate instruction intervals.

Figure 11: Text-driven real-robot execution combining upper-body gestures with walking and dancing. Bars indicate instruction intervals.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 292 papers

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
Jul 9, 2026Kaifeng Zhao, Mathis Petrovich, Haotian Zhang +3Text-to-Motion Generation Human Motion Generation
NVIDIA, Switzerland · ETH Zürich, Switzerland · NVIDIA, USA
UniMoFlow: Grounding Instruction-Driven 3D Human Motion Editing in Generation
Aug 10, 2026Yilei Hua, Beibei Jing, Ce Zheng +3Flow Matching Text-to-Motion Generation
School of Computer Science, Huazhong University of Science and Technology · School of Software Technology, Zhejiang University
FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation
Sep 27, 2026Yiyi Cai, Yuhan Wu, Kunhang Li +6Motion Diffusion Models Human Motion Generation
Alaya Lab · The University of Tokyo · Japan Advanced Institute of Science and Technology +1

Jul 9, 2026J/K move · Enter open · S save

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang+2

NVIDIA, Switzerland · ETH Zürich, Switzerland · NVIDIA, USA

Text-to-Motion Generation Human Motion Generation

Open paper AskPDF arXiv

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.

Abstract

Human Motion Generation

Explore similar work

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

UniMoFlow: Grounding Instruction-Driven 3D Human Motion Editing in Generation

FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation