Video character replacement aims to replace actors while preserving facial expressions, motions, and scene dynamics. Existing methods typically rely on masked generation with pose guidance, limiting fine-grained expression and motion preservation in complex scenes. We propose ReXMo, an end-to-end framework that supports single-, multi-, and selective character replacement scenarios. ReXMo introduces paired triplet supervision, where synthesized driving videos and original videos share identical performances but different identities, enabling scalable training through synthetic data generation. An In-Context Diffusion Transformer learns identity transformation by jointly conditioning on driving videos, reference identities, and text prompts, while an Expression Enhancement Adapter improves facial expression preservation in multi-person scenes. Extensive experiments demonstrate that ReXMo outperforms existing methods in identity fidelity, motion consistency, expression preservation, and visual quality. Our dataset and code will be publicly released.
Figures & tables
Figure 2: Overview of the ReXMo framework. Stage I learns reference-guided character replacement with synthetic cross-identity paired triplet supervision. Stage II freezes the replacement model and optimizes the Expression Enhancement Adapter with aligned facial sequences for fine-grained expression preservation.
Figure 3: Overview of the proposed synthetic data generation pipeline. Stage I constructs cross-identity motion pairs for base-model training by progressively replacing target identities while preserving motions and scene dynamics. Stage II builds an expression-stable head dataset for training the Expression Enhancement Adapter through face alignment and quality filtering.
Figure 4: Qualitative comparison on single-character replacement. Zoom in for better visualization.
Figure 5: Qualitative comparison on multi-character replacement. Zoom in for better visualization.
Table 1: Quantitative comparison on the synthetic cross-identity replacement benchmark.
Method
Image Quality ↑
Motion Smooth. ↑
Temporal Consist. ↑
Appearance Consist. ↑
Wan-Animate
3.78
3.88
3.93
4.18
MultiAnimate
3.71
3.74
3.84
3.90
SCAIL
4.18
3.92
3.98
4.20
SCAIL-2
4.27
3.87
4.08
4.32
ReXMo (Ours)
4.32
3.90
4.12
4.39
Table 2: Perceptual comparison on the character replacement.
Figure 6: User Study (50 participants) on Motion Consistency, Identity Consistency, and Expression Accuracy.
Method
F-LMD ↓
M-LMD ↓
Expr-LMD ↓
LVD ↓
Wan-Animate
0.3345
0.3248
0.3189
0.0374
MultiAnimate
0.5013
0.4977
0.5086
0.0458
SCAIL
0.3309
0.3381
0.3485
0.0388
SCAIL-2
0.3350
0.3411
0.3414
0.0394
w/o EEA
0.3341
0.3251
0.3415
0.0389
w/ EEA
0.2858
0.2929
0.2905
0.0288
Table 3: Quantitative ablation study of the Expression Enhancement Adapter.
Figure 7: Visualization of the ablation study on the Expression Enhancement Adapter (EEA). w/ and w/o EEA denote models with and without the proposed adapter.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
We present PAI-Actor, a cinematic multi-character animation framework for character replacement in dynamic movie scenes. Unlike conventional animation systems that mainly drive a single static image or a single subject, our goal is to replace and animate multiple characters within real video clips while preserving the original scene dynamics, camera motion, and background content. This setting is particularly challenging because the generated characters must remain consistent with the source performance in motion and interaction, while also matching the surrounding background in lighting, shadow, composition, and overall cinematic appearance. To address this, we formulate multi-character animation as a structure-guided human recovery problem and build a movie-driven training pipeline from high-quality film data. Furthermore, to support practical cinematic production, we introduce a bidirectional-to-autoregressive distillation framework: we first train a bidirectional diffusion transformer for high-quality short-clip generation at 1080P resolution, and then distill it into an autoregressive video-to-video model for efficient inference and longer video generation. Experiments show that PAI-Actor enables high-fidelity multi-character animation with strong scene consistency, cinematic visual quality, and efficient long-form generation.
Bangxun Tang, Heyuan Gao, Yiren Song +4
Utopai Studios · University of California, Irvine · Nanyang Technological University +1
Video identity replacement seeks to transfer the identities of one or more subjects while preserving the motion, expressions, and temporal structure of a driving video. Existing methods largely target single-person settings and often require task-specific structural controls, such as masks or pose representations, limiting their flexibility in general multimodal editing systems. Progress on multi-person replacement is further constrained by the scarcity of paired training data. We present Vorch-IR, a unified framework that supports single- and dual-person identity replacement, with optional background replacement, in a single model. Built on LTX2, Vorch-IR jointly conditions on a driving video, indexed reference images, and a textual editing instruction. The reference images need not match the pose, layout, or spatial configuration of the driving video: their roles as subject or background references are specified through the instruction. Dense visual conditions are fused through self-attention, while a vision-language context establishes semantic correspondence through cross-attention. We further develop an automatic data construction pipeline that synthesizes paired supervision for all four editing settings. Experiments using automatic metrics and pairwise human evaluation demonstrate strong identity preservation, motion fidelity, and temporal coherence across diverse scenarios. A temporal overlapping inference strategy additionally extends the short-clip model to minute-long generation without autoregressive continuation.
Digital characters are central to modern media, yet generating character videos with long-duration, consistent multi-view appearance and expressive identity remains challenging. Existing approaches either provide insufficient context to preserve identity or leverage non-character-centric information as the memory, leading to suboptimal consistency. Recognizing that character video generation inherently resembles an outside-looking-in scenario. In this work, we propose representing the character visual attributes through a compact set of anchor frames. This design provides stable references for consistency, while reference-based video generation inherently faces challenges of copy-pasting and multi-reference conflicts. To address these, we introduce two mechanisms: Superset Content Anchoring, providing intra- and extra-training clip cues to prevent duplication, and RoPE as Weak Condition, encoding positional offsets to distinguish multiple anchors. Furthermore, we construct a scalable pipeline to extract these anchors from massive videos. Experiments show our method generates high-quality character videos exceeding 10 minutes, and achieves expressive identity and appearance consistency across views, surpassing existing methods.
Yuhang Yang, Fan Zhang, Huaijin Pi +5
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC · UNSW · HKU +1