cs.CVMar 29, 2026

OmniColor: A Unified Framework for Multi-modal Lineart Colorization

Authors: Xulu Zhang, Haoqian Du, Xiaoyong Wei, Qing Li

Organizations: The Hong Kong Polytechnic University · Sichuan University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Lineart colorization is a critical stage in professional content creation, yet achieving precise and flexible results under diverse user constraints remains a significant challenge. To address this, we propose OmniColor, a unified framework for multi-modal lineart colorization that supports arbitrary combinations of control signals. Specifically, we systematically categorize guidance signals into two types: spatially-aligned conditions and semantic-reference conditions. For spatially-aligned inputs, we employ a dual-path encoding strategy paired with a Dense Feature Alignment loss to ensure rigorous boundary preservation and precise color restoration. For semantic-reference inputs, we utilize a VLM-only encoding scheme integrated with a Temporal Redundancy Elimination mechanism to filter repetitive information and enhance inference efficiency. To resolve potential input conflicts, we introduce an Adaptive Spatial-Semantic Gating module that dynamically balances multi-modal constraints. Experimental results demonstrate that OmniColor achieves superior controllability, visual quality, and temporal stability, providing a robust and practical solution for lineart colorization. The source code and dataset will be open at https://github.com/zhangxulu1996/OmniColor.

Figures & tables

Figure 1: Demonstration of OmniColor’s unified colorization capabilities. From a single lineart (L), our model can generate diverse, high-quality colorized results by integrating different combinations of text (T), color hints (C), identity reference (I), and temporal history (H).
Figure 2: The architecture of OmniColor. Our framework categorizes auxiliary signals into spatially-aligned conditions ( CspatC_{spat} ) and semantic-reference conditions ( CsemC_{sem} ). CspatC_{spat} is processed via a dual-encoder strategy to preserve structural details. CsemC_{sem} is processed via a VLM-only encoder, where a TRE module filters redundant tokens. The MMDiT backbone integrates these features with AS-Gate module to resolve input conflicts. The model is optimized using Flow Matching Loss ( LFML_{FM} ) and a DFA loss.
 ConditionsMetrics
MethodLTHCGIImage-Align. ↑\uparrowSSIM ↑\uparrowPSNR ↑\uparrowFID ↓\downarrowLPIPS ↓\downarrowΔFC\Delta FC ↓\downarrow
Text-based Lineart Colorization
ControlNet [ 69 ]✓\checkmark✓\checkmark×\times×\times×\times×\times0.64030.30797.7390134.710.792010.13
Tag2pix [ 26 ]✓\checkmark✓\checkmark×\times×\times×\times×\times0.65260.45257.4692179.840.59383.02
Reference-based Lineart Colorization
MagicColor [ 71 ]✓\checkmark×\times×\times×\times✓\checkmark×\times0.77280.563110.69118.330.51938.33
Table 1: Quantitative comparison with state-of-the-art methods.
MethodStructural FidelityInstruction FollowingVisual QualityOverall Pref.
Seedream 4.5 [ 48 ]2.117.611.810.5
Nano Banana [ 12 ]7.627.815.216.9
Qwen-Image-Edit [ 58 ]9.312.513.111.6
OmniColor (Ours)81.042.159.961.0
Table 2: Results of the User Study. We report the average preference rates (%) of human evaluators comparing our OmniColor against three SoTA methods across three dimensions. The best results are highlighted in bold .
Figure 3: Qualitative comparison of prompt-based lineart colorization. We compare our method with Tag2pix [ 26 ] and SoTA multi-modal generation models. Our framework achieves superior structural fidelity and prompt alignment.
Figure 4: Qualitative results of reference-based lineart colorization. Our method maintains robust identity and stylistic consistency even under various motions (rotation, zoom) and significant viewpoint changes.
Figure 5: Synergistic control with multiple modalities. We show the effects of combining lineart (L) and text (T) with ID references (I) and color hints (C). The results demonstrate that our model effectively integrates multi-modal signals.
Figure 6: Visual demonstration of sequential generation. The sequence is generated using historical frames (H) and the 1st GT frame of the target shot (G). OmniColor ensures high temporal stability and visual coherence across the entire sequence.
ComponentsMetrics
VLM-onlyDFATREAS-GateImage-Align. ↑\uparrowSSIM ↑\uparrowPSNR ↑\uparrowFID ↓\downarrowLPIPS ↓\downarrowΔFC\Delta FC ↓\downarrow#Tokens ↓\downarrowTime (s) ↓\downarrow
×\times×\times×\times×\times0.91410.737115.23110.320.30520.747521452
✓\checkmark×\times×\times×\times0.91430.728714.93112.290.31430.617521203
✓\checkmark✓\checkmark×\times×\times0.92220.750015.72105.900.29430.177521203
✓\checkmark✓\checkmark✓\checkmark×\times0.92400.758816.2397.310.28050.565203150
✓\checkmark✓\checkmark✓\checkmark✓\checkmark0.92460.761116.3891.490.27850.375203152
Table 3: Ablation study of the proposed components. #Tokens denotes the average token overhead of long-term history frames per sample.
More in this topic

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

Explore similar work

CardsList

  1. InstanceAnimator: Multi-Instance Sketch Video Colorization

    Mar 26, 2026Yinhan Zhang, Yue Ma, Bingyuan Wang +5Video Diffusion ModelsControllable Video Generation

    HKUST(GZ) · HKUST · NUS

  2. Towards High-resolution and Disentangled Reference-based Sketch Colorization

    Mar 6, 2026Dingkun Yan, Xinrui Wang, Ru Wang +5Distribution ShiftImage-to-Image Translation

    The University of Tokyo · Project HAT · Waseda University

  3. Beyond Fixed Luminance: Towards Panchromatic and Orthochromatic Image Colorization

    Aug 11, 2026Swarnim Maheshwari, Syed Imam Ali, Vineeth N. BalasubramanianImage-to-Image Translation

    Indian Institute of Technology Hyderabad, Kandi, Telangana, India · Microsoft Research India, Bengaluru, Karnataka, India