Past as State, Present as Attention: Persistent-State Blockwise Flow Matching for Long-Horizon Co-Speech Motion Generation

Yi Liu; Xiangyue Zhang; Jianfang Li; Jia Ma; Yisheng He; Jianqiang Ren

cs.CVOct 23, 2026

Past as State, Present as Attention: Persistent-State Blockwise Flow Matching for Long-Horizon Co-Speech Motion Generation

Authors: Yi Liu, Xiangyue Zhang, Jianfang Li, Jia Ma, Yisheng He, Jianqiang Ren

Organizations: Tongji University · Alibaba Group · The University of Tokyo

Abstract

Co-speech motion generation remains challenging over long horizons due to the need for temporal coherence and consistent style. Existing methods typically generate long motions segment by segment, either conditioning on a short seed of recent frames, which gradually drifts, or retaining ever-longer explicit histories, which are costly to maintain and hard to learn from limited data. We trace these limitations to a key asymmetry between past and present: past motion carries slowly varying style that should be compactly summarized, whereas the current segment exhibits fast-changing local dynamics that require fine-grained modeling. This asymmetry motivates a simple Past-As-State, Present-as-Attention principle for long-horizon co-speech motion generation. Following this principle, we propose PASPA, which summarizes past history with an inter-block persistent state and models the current segment with intra-block bidirectional self-attention. We instantiate PASPA within a blockwise flow matching framework, with multi-block supervision for effective training over variable histories, prefill-decode state caching for efficient rollout, and hybrid classifier-free guidance for enhanced history and speech conditioning. Extensive public-benchmark experiments show that PASPA achieves state-of-the-art FGD performance and preserves motion style across long-sequence generation.

Figures & tables

Figure 1: Long-horizon co-speech motion generation (52 s) from the same speech and speaker, with the GT style keeping both hands raised near the chest. (a) Short-Seed Truncation matches the style initially but monotonically drifts: hands gradually drop. (b) Long-Context Token Attention retains full history but struggles to maintain a consistent style, matching or deviating regardless of temporal distance. (c) PASPA (ours) preserves the raised-hands style throughout, closely matching the GT.

Figure 2: History propagation in co-speech motion generation. (a) Short-Seed Truncation keeps only recent frames. (b) Long-Context Token Attention retains history as tokens for global attention. (c) PASPA compresses history into a persistent state and applies full attention within the current block.

Figure 3: Autocorrelation

r(\tau)

on BEAT2, averaged across speakers with 95% confidence intervals (shaded). Style features remain correlated beyond 100 frames, whereas local features decay rapidly. Inset: raw windowed pose statistics. See Appendix A for more details.

Figure 4: Network architecture of PASPA. (a) BGDN propagates history through a persistent state $\boldsymbol{S}_{[n]}^{0}$ via a block-causal GDN kernel. (b) BMHSA captures local dynamics via efficient block-diagonal bidirectional attention. The overall network (bottom) sequentially applies BGDN and BMHSA.

Method	1 Speaker				All Speakers
Method	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$	MSE $\downarrow$	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$	MSE $\downarrow$
CaMN [ 45 ]	6.64	0.677	10.86	–	5.12	0.200	5.58	–
DSG [ 80 ]	8.81	0.724	11.49	–	11.74	0.734	11.12	–
EMAGE [ 44 ]	5.51	0.772	13.06	7.680	6.92	0.284	6.06	6.908
DiffSHEG [ 11 ]	8.99	0.714	11.91	7.665	–	–	–	–
MambaTalk [ 79 ]	5.37	0.781	13.05	6.289	–	–	–	–

Table 1: Comparison with state-of-the-art (SOTA) methods on the BEAT2 dataset.

Figure 5: Qualitative comparison. PASPA consistently matches the GT in gesture direction (red arrows), amplitude, and speech alignment. Baselines exhibit various failure modes (red circles): unnatural hand positions, hesitant or mismatched gestures, and missing semantic response.

Table 8

Guidance	120 s (3,600 frames)		300 s (9,000 frames)		600 s (18,000 frames)
Guidance	Lat. (ms)	Mem. (GB)	Lat. (ms)	Mem. (GB)	Lat. (ms)	Mem. (GB)
No CFG	11,411	0.36	28,562	0.40	55,978	0.47
Hybrid CFG	12,439	0.37	29,966	0.41	61,093	0.48

Table 4: Inference overhead of Hybrid CFG under the final PASPA configuration.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Setting / Value
Dataset & Setup	Skeleton	55 joints (30 hands, 25 torso)
	Translation	3D global root translation
	Expression	100-dim FLAME blendshapes
	Rotation Representation	axis-angle (raw) $\rightarrow$ rot6d [ 97 ]
	Motion Dimension	$55\times 6+3=333$
	Audio Features	WavLM [ 12 ] , 16 kHz, 1024-dim, pre-extracted

Appendix

Table 5: Implementation details of our best training and inference configurations on BEAT2.

Figure 11: User study interface. Participants rank four anonymized methods (A/B/C/D, randomized per clip) produced by SemTalk, GestureLSM, EMAGE and PASPA, against a non-rankable ground-truth reference (GT), along four criteria: realism, semantic consistency, motion-speech sync, and diversity.

Table 13

Table 8: CFG scale ablations for Condition CFG and Hybrid CFG. Cond CFG denotes condition CFG. Both panels use the same activation interval

t\in[0.1,0.6]

Mode	Guided prediction
State First	$\hat{\boldsymbol{x}}_{0}+w_{\mathrm{hist}}(\hat{\boldsymbol{x}}_{0}^{h}-\hat{\boldsymbol{x}}_{0})+w_{\mathrm{cond}}(\hat{\boldsymbol{x}}_{0}^{c,h}-\hat{\boldsymbol{x}}_{0}^{h})$
Cond First	$\hat{\boldsymbol{x}}_{0}+w_{\mathrm{cond}}(\hat{\boldsymbol{x}}_{0}^{c}-\hat{\boldsymbol{x}}_{0})+w_{\mathrm{hist}}(\hat{\boldsymbol{x}}_{0}^{c,h}-\hat{\boldsymbol{x}}_{0}^{c})$
Independent	$\hat{\boldsymbol{x}}_{0}+w_{\mathrm{cond}}(\hat{\boldsymbol{x}}_{0}^{c}-\hat{\boldsymbol{x}}_{0})+w_{\mathrm{hist}}(\hat{\boldsymbol{x}}_{0}^{h}-\hat{\boldsymbol{x}}_{0})$
Interaction	$\hat{\boldsymbol{x}}_{0}+w_{\mathrm{cond}}(\hat{\boldsymbol{x}}_{0}^{c}-\hat{\boldsymbol{x}}_{0})+w_{\mathrm{hist}}(\hat{\boldsymbol{x}}_{0}^{h}-\hat{\boldsymbol{x}}_{0})+w_{\mathrm{cond}}w_{\mathrm{hist}}(\hat{\boldsymbol{x}}_{0}^{c,h}-\hat{\boldsymbol{x}}_{0}^{c}-\hat{\boldsymbol{x}}_{0}^{h}+\hat{\boldsymbol{x}}_{0})$

Appendix

Table 9: Formulations of Hybrid CFG decomposition modes.

\hat{\boldsymbol{x}}_{0}

\hat{\boldsymbol{x}}_{0}^{c}

\hat{\boldsymbol{x}}_{0}^{h}

, and

\hat{\boldsymbol{x}}_{0}^{c,h}

denote predicted clean motions under neither condition nor history, condition only, history only, and both condition and history, respectively.

Table 16

Guidance	1 Speaker				All Speakers
Guidance	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$	MSE $\downarrow$	FGD $\downarrow$	BC $\uparrow$	Div $\uparrow$	MSE $\downarrow$
w/o CFG	3.76	0.744	13.80	5.788	2.59	0.639	9.48	3.802
Hybrid CFG	3.71	0.746	13.38	5.768	2.33	0.645	9.42	3.793

Appendix

Table 12: Effect of inference-time CFG on the main BEAT2 results.

Figure 12: Long-rollout endpoint metrics across

T\in\{10,30,60\}\,

s, evaluated on the terminal

10

s. Shaded regions denote

95\%

Wald confidence intervals estimated from clip-level bootstrap resampling.

Metric	Main result	50 generation seeds
Metric	Main result	Mean	Sample std.	95% CI of mean
FGD $\downarrow$	2.33	2.237	0.074	$[2.216,\,2.258]$
BC $\uparrow$	0.645	0.6407	0.0015	$[0.6402,\,0.6411]$
Div $\uparrow$	9.42	9.273	0.079	$[9.251,\,9.296]$
MSE $\downarrow$	3.793	3.761	0.003	$[3.760,\,3.762]$

Appendix

Table 13: Generation-seed variability in the All Speakers setting. Only the generation seed varies across 50 runs; CIs are

95\%

Student-

t

intervals for the mean.

Method	Realism	Semantic	Sync	Diversity	Overall
EMAGE	2.63 $\,\pm\,$ 1.09	2.41 $\,\pm\,$ 1.13	2.49 $\,\pm\,$ 1.08	2.50 $\,\pm\,$ 1.09	2.51 $\,\pm\,$ 1.10
GestureLSM	2.60 $\,\pm\,$ 1.15	2.80 $\,\pm\,$ 1.12	2.82 $\,\pm\,$ 1.09	2.83 $\,\pm\,$ 1.12	2.76 $\,\pm\,$ 1.12
SemTalk	2.63 $\,\pm\,$ 1.04	2.59 $\,\pm\,$ 1.01	2.58 $\,\pm\,$ 1.08	2.57 $\,\pm\,$ 0.99	2.59 $\,\pm\,$ 1.03
PASPA	2.14 $\,\pm\,$ 1.12	2.20 $\,\pm\,$ 1.12	2.11 $\,\pm\,$ 1.11	2.10 $\,\pm\,$ 1.14	2.14 $\,\pm\,$ 1.12

Appendix

Table 14: User study results. Each cell reports the mean rank

\pm

standard deviation across all participants and clips (lower is better;

1

= best,

4

= worst).

Figure 13: Failure case on sequence 16_jorge_1_3_3 . Time progresses from left to right. (a) Ground-truth. (b) Our generated sequence.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
Aug 3, 2026Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2Human Motion Generation Co-Speech Gesture Generation
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2
SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching
Mar 27, 2026Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1Multimodal Grounding Flow Matching
Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University

Aug 3, 2026J/K move · Enter open · S save

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

Xiangyue Zhang, Jianfang Li, Jiaxu Zhang, Kaixing Yang+1

The University of Tokyo · Alibaba Group · Nanyang Technological University+1

Human Motion Generation Co-Speech Gesture Generation

Open paper AskPDF arXiv

Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.

Abstract

Human Motion Generation

Explore similar work

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching