CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing

Yucheng Wang; Zedong Wang; Yuetong Wu; Yue Ma; Dan Xu

cs.CVMar 9, 2026

CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing

Authors: Yucheng Wang, Zedong Wang, Yuetong Wu, Yue Ma, Dan Xu

Organizations: The Hong Kong University of Science and Technology

Abstract

Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.

Figures & tables

Figure 2: Overview of contextual image editing paradigm. (A) Instruction‑based editing guides modifications via a text prompt $\mathbf{T}$ and a base image $\mathbf{B}$ . (B) Subject‑based editing uses the base $\mathbf{B}$ and a references image $\mathbf{R}$ to preserve identity or style. (C) CARE‑Edit incorporates all these modalities $(\mathbf{T},\mathbf{B},\mathbf{R})$ and the user-defined mask $\mathbf{M}$ in a diffusion transformer (DiT) backbone with condition‑aware routing of experts.

Figure 3: CARE-Edit introduces condition‑aware specialized experts within the frozen DiT backbone. Given multimodal conditions, inputs are tokenized and projected to heterogeneous expert branches. The router assigns confidence scores and selects the top‑ $K$ experts to process each token. Expert outputs are normalized, modulated, and fused through the Latent Mixture module, yielding denoised representations $\mathbf{h}^{\prime}$ refined by Mask Repaint. Only lightweight adapters, the router, and fusion layers are trainable. This enables CARE-Edit to dynamically allocate computation, mitigates conflicts between heterogeneous conditions ( e.g . , text vs mask) and enables high-fidelity, coherent edits.

Figure 4: Qualitative comparison of subject-driven contextual editing. Green annotations indicate data belonging to the benchmark, while Blue annotations denote whether mask inputs are provided. CARE-Edit ensures the preservation of subject identity and coherent contexts.

Category	Method	Venue	Backbone	EMU-Edit [ 38 ] Test				MagicBrush [ 55 ] Test
Category	Method	Venue	Backbone	CLIPim $\uparrow$	CLIPout $\uparrow$	L1 $\downarrow$	DINO $\uparrow$	CLIPim $\uparrow$	CLIPout $\uparrow$	L1 $\downarrow$	DINO $\uparrow$
Task-specific Models	PnP [ 8 ]	CVPR’23	SD1.5	0.521	0.089	0.089	0.304	0.568	0.101	0.289	0.220
	Null-Text [ 25 ]	CVPR’23	SD1.5	0.761	0.236	0.075	0.678	0.752	0.263	0.077	0.664
	InstructPix2Pix [ 2 ]	CVPR’23	SD1.5	0.834	0.219	0.121	0.762	0.837	0.245	0.093	0.767
	EMU-Edit [ 38 ]	CVPR’24	–	0.859	0.231	0.094	0.819	0.897	0.261	0.052	0.879
Unified Models	FLUX.1 Fill [ 16 ]	HuggingFace	FLUX.1 Fill	0.663	0.205	0.176	0.674	0.725	0.235	0.208	0.661

Table 1: Quantitative results on EMU-Edit [ 38 ] and MagicBrush [ 55 ] test sets. All included editors are classified into Task-specific and Unified models. Best and second-best results per metric are marked in bold and underline .

\uparrow(\downarrow)

indicates higher (lower) values are better.

Method	Single‑Object			Multiple‑Object
Method	DINO‑I $\uparrow$	CLIP‑I $\uparrow$	CLIP‑T $\uparrow$	DINO‑I $\uparrow$	CLIP‑I $\uparrow$	CLIP‑T $\uparrow$
DreamBooth [ 35 ]	$0.552$	$0.544$	$0.301$	$0.359$	$0.495$	$0.305$
BLIP‑Diffusion [ 18 ]	$0.610$	$0.649$	$0.293$	$0.462$	$0.592$	$0.289$
OmniControl [ 41 ]	$0.770$	$0.704$	$0.312$	$0.501$	$0.641$	$0.316$
UNO [ 46 ]	$0.782$	$0.713$	$0.304$	$0.508$	$0.649$	$0.303$
OmniGen2 [ 51 , 50 ]	0.861	0.784	0.318	0.560	0.713	0.319

Table 2: Quantitative results on DreamBench++ [ 31 ] . The best and second‑best results are marked in bold and underlined , respectively.

Figure 5: Qualitative comparison of instruction-based editing.

Figure 6: Visualization of base expert attention map over iterations.

Figure 7: Task-Expert Activation Analysis and Loss Curves.

Variant	DINO‑I $\uparrow$	CLIP‑I $\uparrow$	CLIP‑T $\uparrow$
w/o Experts	0.485	0.652	0.296
w/o Latent Mixture	0.509	0.678	0.301
w/o Mask Repaint	0.523	0.693	0.304
$K=2$	0.541	0.707	0.312
$K=4$	0.562	0.716	0.325
Full Model (K=3)	0.568	0.720	0.327

Table 3: Ablation studies of CARE-Edit.

\uparrow

indicates higher values are better. The best results per metric are highlighted in bold .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Pipeline for Mask-aware Image-Pair Generation. We use a GPT-Image-1 and VLM-based pipeline to create our training data. Starting with a reference subject on a white background, we generate diverse scene descriptions and corresponding images using an image-to-image model. This yields high-quality image pairs with consistent backgrounds but varying foregrounds, annotated with precise segmentation masks ( $\mathcal{M}$ ) and bounding boxes ( $\mathcal{B}$ ).

Figure 9: GPT Prompting for Scene Diversity. We explicitly instruct the LLM to generate varied scene descriptions ( e.g . , “sunny park”) while keeping the subject ( e.g . , “Koala”) constant. As such, by synthesizing multiple environments for the identical subject entity, we create training data that compels the image editing model to learn robust subject grounding independent of the background correlations.

Method	#Train data	DINO‑I $\uparrow$	CLIP‑I $\uparrow$	CLIP‑T $\uparrow$
OmniControl [ 41 ]	0 ~ $1M$	0.501	0.641	0.316
UNO [ 46 ]	0 ~ $1M$	0.508	0.649	0.303
OmniGen2 [ 50 ]	$\geq 533K$	0.560	0.713	0.319
CARE‑Edit (Ours)	$120K$	0.568	0.720	0.327

Appendix

Table 4: Quantitative results on the multiple-object subset of DreamBench++ [ 31 ] . We report three metrics along with the approximate number of training examples used by each method. The best and second-best results are marked in bold and underlined , respectively.

Figure 10: Task-Specific GPT Prompt Templates. We visualize the templates used to construct training triplets for: (i) Replacement (Top-Left); (ii) Addition (Top-Right); and (iii) Style/Attribute Change (Bottom). By constraining the LLM to modify specific slots ( e.g . , subject identity) while holding scene descriptions constant, it ensures the resulting image pairs possess consistent backgrounds.

Figure 11: Expert Specialization During Training. We visualize the latent attention maps for the Base , Mask , and Reference experts at increasing training iterations ( $T$ ). (Top) Base Expert: Maintains consistent, global activation throughout training ( $T=0\rightarrow 100\mathrm{K}$ ), acting as a foundation to preserve image structure. (Middle) Mask Expert: Learns to progressively suppress background signals, evolving from a noisy initialization to a highly localized, structure-aware attention map that precisely targets the edit region. (Bottom) Reference Expert: Gradually increases engagement in regions requiring semantic or stylistic modification. This distinct separation of concerns confirms that CARE-Edit effectively disentangles conflicting editing signals into specialized processing pathways.

Figure 12: Instruction-based Editing on EMU-Edit and MagicBrush. We compare CARE-Edit against existing unified editors [ 50 , 45 ] . Rows exhibit challenging scenarios such as text rendering (“CARE tag”), global style transfer (“Ukiyo-e”), and complex object insertions. (1) Text and Geometric Fidelity (Row 1): The geometric rigidity of the tag and the legibility of the text are paramount. While ACE++ and OmniGen2 correctly interpret the semantic intent, they suffer from structural drift , resulting in warped tag boundaries and deformed glyphs. CARE-Edit, particularly the masked variant, utilizes the Mask Expert to enforce spatial constraints, producing orthogonal tag geometry and crisp, readable text. (2) Identity Preservation (Row 2): CARE-Edit preserves the subject’s facial identity and hair texture significantly better than baselines, which tend to over-blend the subject into the crowd (identity dilution) or generate a generic face. (3) Style Disentanglement (Row 4 & 5): In the “Ukiyo-e” and “Painting” style transfer tasks, baselines often hallucinate new objects or flatten the entire scene into a texture map. CARE-Edit disentangles style from structure. The Base Expert maintains the complex branching of the cherry blossom tree and the mechanical details of the tractor, while Reference Expert strictly applies the artistic texture to the environments.

Figure 13: Complex Contextual Editing Results. Multi-condition examples requiring harmonization of subject identity, mask constraints, and text prompts. CARE-Edit successfully composes subjects into disparate environments while respecting the user-provided spatial layout.

Figure 14: Object Removal Results. This task requires the model to remove the foreground object and hallucinate a plausible background ( e.g . , sofa fabric, wall patterns) that is consistent with surrounding scene context. (i) Top and Middle: CARE-Edit successfully handles stochastic textures, such as natural water ripples and uneven stone surfaces, filling the void with spatially coherent content. (ii) Bottom-Left: A stress test for structural consistency. Removing the television requires reconstructing the rigid grid pattern of the wallpaper. CARE-Edit accurately hallucinates the missing tiles, maintaining the correct perspective and alignment without the blurring or geometric warping.

Figure 15: Object Addition and Replacement Results. CARE-Edit demonstrates precise control over scene composition, inserting or swapping objects while rigorously adhering to environmental constraints. (i) Object Addition (Left): CARE-Edit introduces new elements that respect physical laws. Note that how the added rubber duck (middle) is generated with accurate water reflections and surface interaction. (ii) Object Replacement (Right): CARE-Edit handles drastic changes in structure and material while maintaining lighting consistency. In the top-right example, replacing a furry pokemon creature with a pokemon ball (metallic sphere), CARE-Edit correctly renders specular highlights and casts realistic shadows onto the complex dirt terrain, ensuring the newly added object sits naturally within the depth of field.

Figure 16: Reference-guided Style Transfer with Structure Preservation. A key advantage of CARE-Edit is the ability to decouple style from structure via expert routing. Unlike holistic transfer methods that often deform the underlying geometry, our approach injects the target aesthetic while strictly anchoring the scene layout. (i) Top: The results show that CARE-Edit’s Reference Expert successfully translates the scene into the swirling impasto of Van Gogh or a watercolor wash, yet the Base Expert ensures the architectural rigidity of the cabin, preserving the straight lines of the roof and window frames. (ii) Bottom: The subject’s fur texture is re-rendered to match the artistic medium, demonstrating CARE-Edit’s successful fine-grained textural adaptation without distorting the animal’s original silhouette or pose.

Diffusion-Based Image Editing

Diffusion-based image editing changes image content, style, or attributes while aiming to preserve the parts users want to keep.

See all 225 papers

Aug 10, 2026

Diffusion Image Editing via Asynchronous Token Decoding

Text-guided diffusion image editing aims to modify semantic attributes of an image while preserving its identity, layout, and background. However, naïvely switching the text condition during sampling often causes global drift, as denoising dynamics propagate changes across tokens and can disrupt unedited regions. To address this issue, we propose Asynchronous Token Decoding Edit (ATDEdit), an inference-time framework that views each sampler step as a parallel update of a globally coupled token matrix and enables token-indexed condition switching with differentiated update policies. Instead of applying synchronous target-conditioned updates to all tokens, ATDEdit estimates editable locations using token-wise conditional surprisal and applies target-conditioned corrections to the selected token set. It supplies source key/value memory at keep-token positions and projects selected keep-token latent rows back to their source values; these operations promote background preservation but do not constitute a pixel-level invariance guarantee. This approach combines local editing and background preservation without external or user-provided spatial masks and without model fine-tuning. On PIE-Bench, ATDEdit achieves the strongest reported preservation metrics, including 27.44~dB PSNR and 0.055 LPIPS, while retaining competitive semantic alignment.

Yang Shi, Liangsi Lu, Minzhe Guo +4

Guangdong University of Technology Guangzhou, China · Hong Kong Baptist University Hong Kong, China · Peking University Beijing, China +1

Text-Guided Image Editing Diffusion-Based Image Editing Diffusion Models

Ask PDF

Jun 11, 2026

DuET: Dual Expert Trajectories for Diffusion Image Editing

Recent diffusion editors perform diverse instruction-based edits while conditioning on the source image at every denoising step. Yet persistent source-image conditioning can limit how fully an edit is executed and how natural the result appears, especially when the target scene diverges substantially from the input. We introduce DuET (Dual Expert Trajectories), a training-free inference method that temporarily relaxes source-image conditioning by transitioning through a text-to-image phase before returning to edit mode, allowing the denoising trajectory to move toward the target distribution while retaining the structural benefits of image-conditioned editing. Without modifying model weights or increasing sampling cost, DuET consistently improves instruction relevance, semantic fidelity, and perceptual quality across diverse models and benchmarks. In some cases, these gains come with a modest reduction in source-image preservation, revealing a predictable trade-off between source preservation and edit fidelity.

Lidia Troeshestova, Alexander Ustyuzhanin, Sergey Kastryulin

HSE University · Yandex

Text-Guided Image Editing Diffusion-Based Image Editing Conditional Diffusion Models Training-Free Image Editing Diffusion-Based Denoising

Ask PDF

Jun 15, 2026

ResEdit: Residual embeddings for precise generative image editing

Conditional diffusion image generators can be repurposed for editing through inversion, without the need for large-scale paired fine-tuning data. However, producing high-quality, targeted edits while maintaining image identity and global consistency remains challenging, as weakly conditioned inversion often embeds conflicting image features into the noise. We demonstrate that incorporating a residual image encoding as additional conditioning enables both improved identity preservation and better editability. We optimize this residual encoding to provide a strong conditioning signal for reconstruction, thereby reducing the reliance on inversion and susceptibility to its aforementioned pitfalls. To ensure this residual does not interfere with desired edits, we incorporate a gradient reversal-based optimization strategy that disentangles the residual from the edited condition. We illustrate our method's ability to produce high-fidelity results across precise intrinsic-based editing and relighting, and show proof-of-concept text-guided manipulation.

Ahmet Canberk Baykal, Valentin Deschaintre, Yannick Hold-Geoffroy +4

Adobe Research · University of Cambridge

Diffusion Model Inversion Image Relighting Diffusion-Based Image Editing Image Editing

Ask PDF

Abstract

Diffusion-Based Image Editing

Explore similar work

Diffusion Image Editing via Asynchronous Token Decoding

DuET: Dual Expert Trajectories for Diffusion Image Editing

ResEdit: Residual embeddings for precise generative image editing