We propose Uni-Animator, a novel Diffusion Transformer (DiT)-based framework for unified image and video sketch colorization. Existing sketch colorization methods struggle to unify image and video tasks, suffering from imprecise color transfer with single or multiple references, inadequate preservation of high-frequency physical details, and compromised temporal coherence with motion artifacts in large-motion scenes. To tackle imprecise color transfer, we introduce visual reference enhancement via instance patch embedding, enabling precise alignment and fusion of reference color information. To resolve insufficient physical detail preservation, we design physical detail reinforcement using physical features that effectively capture and retain high-frequency textures. To mitigate motion-induced temporal inconsistency, we propose sketch-based dynamic RoPE encoding that adaptively models motion-aware spatial-temporal dependencies. Extensive experimental results demonstrate that Uni-Animator achieves competitive performance on both image and video sketch colorization, matching that of task-specific methods while unlocking unified cross-domain capabilities with high detail fidelity and robust temporal consistency.
Figures & tables
Figure 2: Workflow Comparison: Manual Colorization vs. Ours . Our automated framework eliminates repetitive frame-by-frame adjustments and cross-domain adaptation efforts, significantly reducing labor costs in industrial production.
Figure 3: Motivation: Limitations of Existing Methods vs. Our Solution . Existing methods suffer from style deviation (left), detail degradation (right), and temporal flickering (bottom) when handling images or videos separately. Our unified framework addresses all three issues concurrently.
Figure 4: Overall architecture of Uni-Animator. Given input visual references, the VAE encoder and physical projector encode its physical features and concatenate with visual latent features, then the DiT model predicts subsequent frames conditioned on physical, visual, and text embeddings. Sketch-based Dynamic RoPE to improve temporal inconsistency in large motion dynamic scenes without extra training.
Figure 5: Visual Control Ability . Our method supports accurate semantic matching for image colorization with over five visual references. For video colorization, it enables dynamic customization of characters by altering visual references while preserving original motions.
Figure 6: Visualization of RoPE frequency comparison. Strong horizontal motion (a) triggers higher frequency oscillations compared to gentle vertical motion (b), matching the asymmetric motion patterns observed in sketch sequences.
Figure 7: Comparison of Visual Results with Baselines . Our method shows better visual consistency during image and video sketch colorization, supporting single to multiple visual references dynamically. Compared with other video methods that use the first frame as input, our method shows a competitive result and the ability to customize the coloring results without the first frame.
Figure 8: Ablation Result of each component. After removing the Visual Reference Enhancement and PhysEncoder, the consistency of detail semantics decreases. When dynamic encoding is installed, the movement of the main subject becomes more natural.
We propose InstanceAnimator, a novel Diffusion Transformer framework for multi-instance sketch video colorization. Existing methods suffer from three core limitations: inflexible user control due to heavy reliance on single reference frames, poor instance controllability leading to misalignment in multi-character scenarios, and degraded detail fidelity in fine-grained regions. To address these challenges, we introduce three corresponding innovations. First, a Canvas Guidance Condition eliminates workflow fragmentation by allowing free placement of reference elements and background, enabling unprecedented user flexibility. Second, an Instance Matching Mechanism resolves misalignment by integrating instance features with the sketches, ensuring precise control over multiple characters. Third, an Adaptive Decoupled Control Module enhances detail fidelity by injecting semantic features from characters, backgrounds, and text conditions into the diffusion process. Extensive experiments demonstrate that InstanceAnimator achieves superior multi-instance colorization with enhanced user control, high visual quality, and strong instance consistency.
Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.
Dingkun Yan, Xinrui Wang, Ru Wang +5
The University of Tokyo · Project HAT · Waseda University
Traditional animation production relies heavily on manual drawing and iterative refinement, particularly for key-pose design, in-betweening, and character coloring. While existing animation and video generation methods have made notable progress, they typically depend on RGB boundary frames, dense frame-wise conditions, or complete sketch sequences, limiting their applicability under low-cost input conditions. We present SketchKeyAnime, a video diffusion framework for generating structurally controllable, appearance-consistent, and temporally coherent animations from sparse key-sketch inputs. Given a single reference RGB image and a few temporally indexed key sketches, SketchKeyAnime introduces a dual-branch conditioning mechanism to encode local geometric constraints alongside semantic-temporal context. It leverages Sketch Cross Attention to fuse reference image and sketch conditions with learnable gating, and incorporates an Adaptive Weighted Loss to strengthen supervision on key-sketch frames and line-art regions. Experimental results on the Aesthetic subset of Sakuga-42M show that our approach consistently outperforms representative animation interpolation and sketch-guided generation baselines. Compared to the best-performing baseline, SketchKeyAnime reduces EDMD by 31.9% and FVD by 9.5%, demonstrating superior sketch fidelity and temporal coherence, while achieving the best overall performance across most quantitative metrics. These results validate the proposed framework and highlight its potential for low-cost, highly controllable animation creation.
Meixi Li, Xianlin Zhang, Yue Zhang +1
Beijing University of Posts and Telecommunications