InstanceAnimator: Multi-Instance Sketch Video Colorization

Yinhan Zhang; Yue Ma; Bingyuan Wang; Kunyu Feng; Yeying Jin; Qifeng Chen; Anyi Rao; Zeyu Wang

cs.CVMar 26, 2026

InstanceAnimator: Multi-Instance Sketch Video Colorization

Authors: Yinhan Zhang, Yue Ma, Bingyuan Wang, Kunyu Feng, Yeying Jin, Qifeng Chen, Anyi Rao, Zeyu Wang

Organizations: HKUST(GZ) · HKUST · NUS

Abstract

We propose InstanceAnimator, a novel Diffusion Transformer framework for multi-instance sketch video colorization. Existing methods suffer from three core limitations: inflexible user control due to heavy reliance on single reference frames, poor instance controllability leading to misalignment in multi-character scenarios, and degraded detail fidelity in fine-grained regions. To address these challenges, we introduce three corresponding innovations. First, a Canvas Guidance Condition eliminates workflow fragmentation by allowing free placement of reference elements and background, enabling unprecedented user flexibility. Second, an Instance Matching Mechanism resolves misalignment by integrating instance features with the sketches, ensuring precise control over multiple characters. Third, an Adaptive Decoupled Control Module enhances detail fidelity by injecting semantic features from characters, backgrounds, and text conditions into the diffusion process. Extensive experiments demonstrate that InstanceAnimator achieves superior multi-instance colorization with enhanced user control, high visual quality, and strong instance consistency.

Figures & tables

Figure 2: Motivation . Unlike traditional methods that require multi-stage, time-consuming, and first-frame-dependent colorization, InstanceAnimator directly colorizes sketch sequences into videos after background and character design. Our method no longer relies on a single reference keyframe but adjusts end-to-end instance-aware colorization, providing decoupled control to achieve higher user flexibility and significantly reducing time consumption and labor.

Figure 3: Overview of InstanceAnimator. We first apply instance-aware attention with instance latent features and noise features to establish a correspondence between the line drawing and the reference instances, as well as to maintain the character feature. Concurrently, instances, background, and text descriptions are fed into the Adaptive Decoupled Control Module independently, which dynamically injects condition information into DiT blocks through three condition-specific expert modules. At the inference stage, users can adjust the conditional weights and freely change the reference instances and image background to enhance controllability and creative flexibility.

Figure 4: (Left): Condition Fusion. The sketch, canvas, and background conditions are temporally aligned and concatenated along the channel dimension. (Right): Instance-Aware Attention Mask. This design enables the model to capture the correspondence between reference instances and sketches while avoiding a sharp increase in computational complexity.

Panel of Figure 5 — Figure 5: Instance Control Ability. (Up) Given the same sketch and different reference instances, InstanceAnimator generates a variety of colorful videos. (Down) Using the same designed characters, our framework colorizes different sketches with consistent colors and user-customized backgrounds.

Figure 6: Visual Background Control . Our method supports customized visual background control in combination with reference instance and line drawing during the colorization process.

Figure 7: Data Construction Pipeline . We build the complete dataset in three steps: Video, Image, and Text Processing.

Figure 8: Qualitative comparison with baseline methods . Given diverse reference instances, our method achieves better results in maintaining character controllability and color consistency.

Configuration	FID $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Temp $\uparrow$	CLIP $\uparrow$
Full Model	121.901	0.681	0.201	0.969	0.921
w/o Instance Attention	138.615	0.576	0.317	0.952	0.861
w/o Canvas Guidance	129.221	0.605	0.289	0.958	0.875
w/o Decoupled Control	126.984	0.591	0.293	0.961	0.862

Table 2: Quantitative results of ablation study.

Figure 9: Qualitative results of ablation study . All proposed modules effectively contribute to improving the quality and consistency of multi-instance colorization.

Figure 10: User ratings across four key dimensions: Our method vs. three baselines . All ratings are based on a 7-point Likert scale (1 = Strongly Disagree, 7 = Strongly Agree).

Capability	LVCD	ToonCrafter	Anidoc	ToonComposer	Ours
Reference Type	Single frame	First+last frame	Single frame	Single frame	Instance-based
Multi-Character	$\triangle$	$\triangle$	$\triangle$	$\checkmark$	$\checkmark$
Canvas Placement	✗	✗	✗	✗	$\checkmark$
Large Motion	✗	$\triangle$	✗	$\triangle$	$\checkmark$
Element Editing	✗	✗	✗	✗	$\checkmark$
Text Control	✗	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 3: Functional Comparison. Detailed comparison of different video colorization approaches across key functional dimensions.

\checkmark

: full support,

\triangle

: partial/limited support, ✗: not supported.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Semantic background control . Unlike traditional methods that rely on other tools for directly replacing background, InstanceAnimator supports semantic mapping of background in combination with line drawing during the colorization process.

Figure 12: Colorization from Different Style Reference . For the same character, our method easily learns features. For different style characters, our method can transfer their style to the sketches even without prior knowledge.

Figure 13: Multiple Reference Instances Control . While we obtain different style reference instances from the internet, our method can match the most similar ID and follow the reference instance style to colorize the sketch sequence.

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 735 papers

Uni-Animator: Towards Unified Visual Colorization
Feb 26, 2026Xinyuan Chen, Yao Xu, Shaowen Wang +2Temporal Consistency in Video Generation
Mississippi State University · UIUC · Hunan University +1
Towards High-resolution and Disentangled Reference-based Sketch Colorization
Mar 6, 2026Dingkun Yan, Xinrui Wang, Ru Wang +5Distribution Shift Image-to-Image Translation
The University of Tokyo · Project HAT · Waseda University
AnyCrowd: Instance-Isolated Identity-Pose Binding for Arbitrary Multi-Character Animation
Mar 16, 2026Zhenyu Xie, Ji Xia, Michael Kampffmeyer +9Identity-Preserving Video Generation Human Motion Generation
Mohamed bin Zayed University of Artificial Intelligence, UAE · University of Tromsø (UiT) – The Arctic University of Norway, Norway · Shenzhen campus of Sun Yet-sen University, China

Feb 26, 2026J/K move · Enter open · S save

Uni-Animator: Towards Unified Visual Colorization

Xinyuan Chen, Yao Xu, Shaowen Wang, Pengjie Song+1

Mississippi State University · UIUC · Hunan University+1

Temporal Consistency in Video Generation

Open paper AskPDF arXiv

We propose Uni-Animator, a novel Diffusion Transformer (DiT)-based framework for unified image and video sketch colorization. Existing sketch colorization methods struggle to unify image and video tasks, suffering from imprecise color transfer with single or multiple references, inadequate preservation of high-frequency physical details, and compromised temporal coherence with motion artifacts in large-motion scenes. To tackle imprecise color transfer, we introduce visual reference enhancement via instance patch embedding, enabling precise alignment and fusion of reference color information. To resolve insufficient physical detail preservation, we design physical detail reinforcement using physical features that effectively capture and retain high-frequency textures. To mitigate motion-induced temporal inconsistency, we propose sketch-based dynamic RoPE encoding that adaptively models motion-aware spatial-temporal dependencies. Extensive experimental results demonstrate that Uni-Animator achieves competitive performance on both image and video sketch colorization, matching that of task-specific methods while unlocking unified cross-domain capabilities with high detail fidelity and robust temporal consistency.

Abstract

Video Diffusion Models

Explore similar work

Uni-Animator: Towards Unified Visual Colorization

Towards High-resolution and Disentangled Reference-based Sketch Colorization

AnyCrowd: Instance-Isolated Identity-Pose Binding for Arbitrary Multi-Character Animation