Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.
Figures & tables
Figure 2: The model incorrectly learns spatial semantics from reference images, which contradict the spatial semantics from sketch images and cause spatial entanglement.
Figure 3: As training progresses, the model increasingly transfers spatial semantics from the reference images into the colorized results, leading to deviations from the correct sketch-based segmentation. The ground-truth Gram matrix is obtained by discarding the reference inputs during inference, and query tokens in the Gram matrices are highlighted with red points.
Figure 4: The left panel illustrates the architecture of the proposed framework, while the right panel shows the computation of the Gram loss. In the first stage, the backbone is trained for reference-based colorization using image embeddings, where the embedding inputs to the denoising U-Net are extracted from the entire reference image (indicated by the red arrow). The Gram loss is activated only during this first training stage. In the subsequent stages, we introduce feature-level representations for foreground and background regions through their respective plugin adapters. During inference, the plugin adapters are executed only once at timestep t=0 .
Figure 5: Ablation results of WD-tagger. The model without the WD tagger fails to correctly colorize the eyes when reference eyes are small and color mismatched. It also shows weaker segmentation guidance overall. Both ablation variants exhibit artifacts without Gram regularization in (e). FID scores are shown on the left.
Figure 6: Visualization of gram matrices and attention maps. The proposed Gram regularization loss helps enhance semantic fidelity to the sketch inputs. Query tokens are highlighted by red rectangles in both the attention maps and Gram matrices. In the generated results, green boxes highlight entanglement artifacts outside sketch regions, while blue boxes indicate semantic errors within sketch-guided areas.
Figure 7: The plugin module can be activated to inject low-level features for a higher style and background similarity.
Figure 8: Qualitative comparison regarding character illustration colorization. All images are generated at 10242 resolution except for MangaNinja [ 20 ] , which is fixed at 5122 in the official implementation. Zoom in for details. High-resolution images are available in the supplementary materials. Column (3) is synthesized using sketches extracted from our results. ※: Real sketches from human artists.
Figure 9: We show the cross-content results, where the sketches and reference images are from different domains (for example, portrait and scenery). The proposed method effectively synthesizes visually pleasant images without artifacts.
Method
†FID ↓
‡PSNR ↑
‡MS-SSIM ↑
‡CLIP score ↑
Ours
8.28
28.83
0.70
0.912
Yan et al. [ 41 ]
12.09
28.44
0.61
0.896
ColorizeDiff [ 43 ]
13.42
28.04
0.57
0.891
IP-Adapter-XL
36.61
28.23
0.44
0.758
IP-Adapter
94.53
27.94
0.50
0.762
T2I-Adapter
94.98
27.97
0.28
0.613
Table 1: Quantitative comparison evaluated by 50K-FID, PSNR, MS-SSIM, and CLIP cosine similarity. †: reference images are randomly selected to be close to real-application scenarios and cover the corner cases. ‡: References are deformed from ground truth. §: Tested at 5122 resolution.
Figure 10: Results of user study. Our method is preferred across all compared methods.
We propose Uni-Animator, a novel Diffusion Transformer (DiT)-based framework for unified image and video sketch colorization. Existing sketch colorization methods struggle to unify image and video tasks, suffering from imprecise color transfer with single or multiple references, inadequate preservation of high-frequency physical details, and compromised temporal coherence with motion artifacts in large-motion scenes. To tackle imprecise color transfer, we introduce visual reference enhancement via instance patch embedding, enabling precise alignment and fusion of reference color information. To resolve insufficient physical detail preservation, we design physical detail reinforcement using physical features that effectively capture and retain high-frequency textures. To mitigate motion-induced temporal inconsistency, we propose sketch-based dynamic RoPE encoding that adaptively models motion-aware spatial-temporal dependencies. Extensive experimental results demonstrate that Uni-Animator achieves competitive performance on both image and video sketch colorization, matching that of task-specific methods while unlocking unified cross-domain capabilities with high detail fidelity and robust temporal consistency.
Xinyuan Chen, Yao Xu, Shaowen Wang +2
Mississippi State University · UIUC · Hunan University +1
We propose InstanceAnimator, a novel Diffusion Transformer framework for multi-instance sketch video colorization. Existing methods suffer from three core limitations: inflexible user control due to heavy reliance on single reference frames, poor instance controllability leading to misalignment in multi-character scenarios, and degraded detail fidelity in fine-grained regions. To address these challenges, we introduce three corresponding innovations. First, a Canvas Guidance Condition eliminates workflow fragmentation by allowing free placement of reference elements and background, enabling unprecedented user flexibility. Second, an Instance Matching Mechanism resolves misalignment by integrating instance features with the sketches, ensuring precise control over multiple characters. Third, an Adaptive Decoupled Control Module enhances detail fidelity by injecting semantic features from characters, backgrounds, and text conditions into the diffusion process. Extensive experiments demonstrate that InstanceAnimator achieves superior multi-instance colorization with enhanced user control, high visual quality, and strong instance consistency.
Lineart colorization is a critical stage in professional content creation, yet achieving precise and flexible results under diverse user constraints remains a significant challenge. To address this, we propose OmniColor, a unified framework for multi-modal lineart colorization that supports arbitrary combinations of control signals. Specifically, we systematically categorize guidance signals into two types: spatially-aligned conditions and semantic-reference conditions. For spatially-aligned inputs, we employ a dual-path encoding strategy paired with a Dense Feature Alignment loss to ensure rigorous boundary preservation and precise color restoration. For semantic-reference inputs, we utilize a VLM-only encoding scheme integrated with a Temporal Redundancy Elimination mechanism to filter repetitive information and enhance inference efficiency. To resolve potential input conflicts, we introduce an Adaptive Spatial-Semantic Gating module that dynamically balances multi-modal constraints. Experimental results demonstrate that OmniColor achieves superior controllability, visual quality, and temporal stability, providing a robust and practical solution for lineart colorization. The source code and dataset will be open at https://github.com/zhangxulu1996/OmniColor.
Xulu Zhang, Haoqian Du, Xiaoyong Wei +1
The Hong Kong Polytechnic University · Sichuan University