PAI-Actor: Cinematic Multi-Character Replacement in Dynamic Scenes

Bangxun Tang; Heyuan Gao; Yiren Song; Guian Fang; Zijian He; Jie Yang; Mike Zheng Shou

cs.CVSep 5, 2026

PAI-Actor: Cinematic Multi-Character Replacement in Dynamic Scenes

Authors: Bangxun Tang, Heyuan Gao, Yiren Song, Guian Fang, Zijian He, Jie Yang, Mike Zheng Shou

Organizations: Utopai Studios · University of California, Irvine · Nanyang Technological University · Show Lab, National University of Singapore

Abstract

We present PAI-Actor, a cinematic multi-character animation framework for character replacement in dynamic movie scenes. Unlike conventional animation systems that mainly drive a single static image or a single subject, our goal is to replace and animate multiple characters within real video clips while preserving the original scene dynamics, camera motion, and background content. This setting is particularly challenging because the generated characters must remain consistent with the source performance in motion and interaction, while also matching the surrounding background in lighting, shadow, composition, and overall cinematic appearance. To address this, we formulate multi-character animation as a structure-guided human recovery problem and build a movie-driven training pipeline from high-quality film data. Furthermore, to support practical cinematic production, we introduce a bidirectional-to-autoregressive distillation framework: we first train a bidirectional diffusion transformer for high-quality short-clip generation at 1080P resolution, and then distill it into an autoregressive video-to-video model for efficient inference and longer video generation. Experiments show that PAI-Actor enables high-fidelity multi-character animation with strong scene consistency, cinematic visual quality, and efficient long-form generation.

Figures & tables

Feature	Wan-Animate	MultiAnimate	SCAIL	DOMO	Viggle	Kling O1	MoCha	SCAIL-2	Ours
1080P	✗	✗	✗	✗	✓	✓	✓	✗	✓
Multi-Character	✗	✓	✓	✓	✓	✓	✗	✓	✓
Long-Video Output ( $>$ 10 s)	✗	✗	✗	✓	✓	✓	✗	✓	✓
Pose-Driven Control	✓	✓	✓	✓	✗	✓	✗	✓	✓
Background Preservation	✓	✗	✗	✓	✓	✓	✓	✓	✓
Occlusion Handling	✗	✗	✗	✗	✗	✗	✗	✗	✓

Table 1: Feature comparison between PAI-Actor and existing character-replacement approaches.

Figure 2: Overview of PAI-Actor. (a) Bidirectional Teacher : Denoises target videos conditioned on text, control signals, and references. (b) Causal Distillation : A frozen teacher guides the AR student via two-stage teacher- and self-forcing rollout losses. (c) Inference : Autoregressive long-video generation using a 5-chunk sliding window.

Method	MSE $\downarrow$ $(\times 10^{-3})$		SSIM $\uparrow$		LPIPS $\downarrow$		Pose-Dist $\downarrow$ $(\times 10^{-2})$		Pose-Match $\uparrow$		Motion Cons. $\uparrow$		BG Cons. $\uparrow$		FG-BG Fusion $\uparrow$		Illum. Harmony $\uparrow$		Char. Cons. $\uparrow$		Expr.-LMD $\downarrow$ $(\times 10^{-2})$
Method	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi
Domo	17.0	19.1	0.647	0.623	0.335	0.360	4.65	4.09	0.674	0.857	0.732	0.791	8.24	8.04	7.02	6.78	7.19	7.52	5.67	4.59	7.39	19.41
Viggle	12.8	16.6	0.718	0.700	0.263	0.312	4.66	5.18	0.637	0.710	0.627	0.714	8.10	8.13	5.40	4.89	6.17	5.85	3.90	3.39	5.71	15.07
Kling O1	7.88	8.85	0.712	0.733	0.211	0.227	3.12	2.79	0.807	0.910	0.716	0.759	7.86	7.65	7.78	7.71	7.32	7.81	5.63	5.53	3.30	6.58
Wan-Animate ∗	5.30	9.94	0.832	0.766	0.170	0.251	2.65	3.21	0.833	0.898	0.796	0.814	8.05	8.12	7.85	7.57	7.50	7.43	4.90	4.32	2.36	12.09
SCAIL	52.9	47.6	0.852	0.779	0.161	0.217	4.74	4.00	0.755	0.835	0.749	0.736	2.59	2.64	5.46	5.64	6.49	6.63	5.07	4.47	4.68	22.09

Table 2: Quantitative comparison on the Character-Animation Subset under both single-character and multi-character settings. BG Cons., FG-BG Fusion, Illum. Harmony, and Char. Cons. are evaluated via LLM-based perceptual scoring.

Method

MSE

\downarrow

(\times 10^{-3})

SSIM

\uparrow

LPIPS

\downarrow

Pose-Dist

\downarrow

(\times 10^{-2})

Pose-Match

\uparrow

Motion Cons.

\uparrow

BG Cons.

\uparrow

FG-BG Fusion

\uparrow

Illum. Harmony

\uparrow

Char. Cons.

\uparrow

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Single

Multi

Ours-Causal

4.77

4.48

0.880

0.860

0.118

0.129

4.90

1.56

0.857

0.988

0.856

0.903

9.88

9.43

6.25

6.22

7.25

7.97

5.38

4.33

Table 3: Quantitative results of Ours-Causal on the Long-Video Subset.

Method	BG Cons. $\uparrow$		FG-BG Fusion $\uparrow$		Illum. Harmony $\uparrow$		Char. Cons. $\uparrow$
Method	Single	Multi	Single	Multi	Single	Multi	Single	Multi
Domo	8.04	8.02	6.94	6.78	6.98	7.46	6.33	5.78
Viggle	7.56	7.74	5.15	4.04	6.07	5.76	5.22	4.24
Kling O1	7.76	7.69	7.41	7.63	7.26	7.44	6.92	6.45
Wan-Animate ∗	7.89	7.85	7.48	7.28	7.49	7.50	6.36	5.04
MoCha	7.92	-	7.44	-	7.38	-	5.83	-

Table 4: LLM-based evaluation (averaged GPT-5.1 & Gemini-3.1-Pro) on the Character-Replacement Subset.

Method	MSE $\downarrow$ $(\times 10^{-3})$		SSIM $\uparrow$		LPIPS $\downarrow$		Pose-Dist $\downarrow$ $(\times 10^{-2})$		Pose-Match $\uparrow$		Motion Cons. $\uparrow$		Expr.-LMD $\downarrow$ $(\times 10^{-2})$
Method	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi	Single	Multi
Ours-Bidir w/o Dilation	2.36	3.14	0.830	0.834	0.117	0.130	2.48	2.29	0.862	0.914	0.809	0.847	2.95	5.69
Ours-Bidir w/o Co-Posed Ref.	6.24	14.9	0.812	0.765	0.164	0.218	2.99	3.39	0.804	0.848	0.816	0.824	4.34	6.42
Ours-Causal w/o self-forcing	2.76	3.84	0.878	0.860	0.121	0.139	2.73	2.35	0.819	0.917	0.826	0.842	4.16	5.76
Ours-Bidir w/o adapter	1.91	2.32	0.855	0.861	0.115	0.124	2.28	2.07	0.843	0.923	0.814	0.847	2.81	5.08
Ours-Bidir	1.89	2.31	0.856	0.861	0.114	0.123	2.30	2.08	0.871	0.946	0.813	0.850	2.15	4.97

Table 5: Ablation study on the Character-Animation subset. Each component was evaluated under single-character and multi-character settings.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: User study results across the four perceptual criteria ( Background Consistency , FG-BG Fusion , Illumination Harmony , Character Consistency ). Each subplot shows the vote share (%) per method on the four task settings (single/multi

\times

animation/replacement). Values are aggregated over the

16

qualified participants. Our method achieves the largest vote share on every (task, criterion) cell.

Figure 7: User study interface and workflow. (a) Study instructions, (b) side-by-side presentation of a test case with input references and anonymized results, and (c) the questionnaire with four evaluation criteria.

Panel of Figure 8 — Figure 8: LLM evaluation prompt templates (part 1/2). Both GPT-5.1 and Gemini-3.1-Pro receive the same prompts and return JSON. BG_Cons compares background regions of the control and generated videos while ignoring the foreground in both. FG-BG_Fusion judges compositing realism of the generated video alone.

Panel of Figure 9 — Figure 9: LLM evaluation prompt templates (part 2/2). Illum_Harmony judges lighting consistency between the rendered character and the scene. Char_Cons compares the generated character against the target reference image(s) under a strict-evaluation protocol with hard caps on each attribute mismatch.

Metric	$n$	Single $\rho$	Multi $\rho$
BG Cons.	14	$+0.407$	$+0.502$
FG-BG Fusion	14	$\mathbf{+0.859}$	$\mathbf{+0.856}$
Illum. Harmony	14	$+0.705$	$+0.846$
Char. Cons.	14	$+0.292$	$+0.581$
Average	–	$\mathbf{+0.566}$	$\mathbf{+0.696}$

Appendix

Table 6: Spearman rank correlation between human preferences (user study) and our LLM-based perceptual scores, across

14

method

\times

task tuples per metric. Higher values indicate stronger agreement.

Method	Resolution (W $\times$ H)	Steps	FPS $\uparrow$
SCAIL	$896\times 512$	51	0.15
MultiAnimate	$832\times 480$	50	0.13
Wan-Animate	$1280\times 720$	20	0.22
MoCha	$832\times 480$	50	0.04
VACE	$1280\times 704$	50	0.18
Ours-Bidir	$1920\times 1056$	50	0.06

Appendix

Table 7: Inference efficiency, tested on one NVIDIA H200.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

MultiAnimate: Pose-Guided Image Animation Made Extensible
Feb 25, 2026Yingcheng Hu, Haowen Gong, Chuanguang Yang +3Pose-Conditioned Video Generation Human Avatar Animation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · ShanghaiTech University · Shanghai Jiao Tong University
AnyCrowd: Instance-Isolated Identity-Pose Binding for Arbitrary Multi-Character Animation
Mar 16, 2026Zhenyu Xie, Ji Xia, Michael Kampffmeyer +9Identity-Preserving Video Generation Human Motion Generation
Mohamed bin Zayed University of Artificial Intelligence, UAE · University of Tromsø (UiT) – The Arctic University of Norway, Norway · Shenzhen campus of Sun Yet-sen University, China
MultiAnimate: A Unified Framework for Controllable Multi-Character Animation
Jul 15, 2026Zhongyi Zhang, Guangyuan Wang, Li Hu +6Identity-Preserving Video Generation Human Motion Generation
School of Cyber Science and Technology, University of Science and Technology of China, Fuxing Road 100, Hefei, 230026, Anhui, China. · Tongyi Lab, Alibaba Group, Dengcai Road 1008, Hangzhou, 310023, Zhejiang, China. · College of Computing and Data Science, Nanyang Technological University, Jurong West, Singapore, 639798, Singapore, Singapore.

Feb 25, 2026J/K move · Enter open · S save

MultiAnimate: Pose-Guided Image Animation Made Extensible

Yingcheng Hu, Haowen Gong, Chuanguang Yang, Zhulin An+2

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · ShanghaiTech University · Shanghai Jiao Tong University

Pose-Conditioned Video Generation Human Avatar Animation Image-to-Video Generation Video Diffusion Models

Open paper AskPDF arXiv

Pose-guided human image animation aims to synthesize realistic videos of a reference character driven by a sequence of poses. While diffusion-based methods have achieved remarkable success, most existing approaches are limited to single-character animation. We observe that naively extending these methods to multi-character scenarios often leads to identity confusion and implausible occlusions between characters. To address these challenges, in this paper, we propose an extensible multi-character image animation framework built upon modern Diffusion Transformers (DiTs) for video generation. At its core, our framework introduces two novel components-Identifier Assigner and Identifier Adapter - which collaboratively capture per-person positional cues and inter-person spatial relationships. This mask-driven scheme, along with a scalable training strategy, not only enhances flexibility but also enables generalization to scenarios with more characters than those seen during training. Remarkably, trained on only a two-character dataset, our model generalizes to multi-character animation while maintaining compatibility with single-character cases. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in multi-character image animation, surpassing existing diffusion-based baselines.

Abstract

Human Motion Generation

Explore similar work

MultiAnimate: Pose-Guided Image Animation Made Extensible

AnyCrowd: Instance-Isolated Identity-Pose Binding for Arbitrary Multi-Character Animation

MultiAnimate: A Unified Framework for Controllable Multi-Character Animation