IECLA: Interpretable Emotion Contribution Learning for Affective Image Editing

Yijie Guo; Dexiang Hong; Weidong Chen; Xinyan Liu; Zhendong Mao

cs.CVOct 18, 2026

IECLA: Interpretable Emotion Contribution Learning for Affective Image Editing

Authors: Yijie Guo, Dexiang Hong, Weidong Chen, Xinyan Liu, Zhendong Mao

Organizations: University of Science and Technology of China · Harbin Institute of Technology

Abstract

Affective image editing (AIE) aims to evoke a target emotion while preserving the content and structure of a source image. Existing methods improve emotional expression or visual fidelity, but the target emotion alone does not reveal which source regions should be changed and which should remain intact, especially in complex scenes. We introduce Interpretable Emotion Contribution Learning for Affective Image Editing (IECLA), a framework that separates source-emotion attribution from target-aware editing. IECLA constructs signed object-level supervision through counterfactual interventions and trains a context-conditioned contribution predictor to localize evidence that supports or opposes the source emotion. At inference, an image-level evaluator estimates the source emotion directly from the input, while the predictor identifies editable and preservation evidence without repeated interventions. A target-aware editing agent then translates this evidence and the desired emotion into localized, content-aware operations for existing editors. Extensive evaluations show that IECLA predicts counterfactual contributions more faithfully on both in-domain and cross-dataset tests, while external VLM and independent human references support the quality of its localized evidence. Across six editing backbones, IECLA consistently strengthens target-emotion expression while preserving image content and structure, and user studies further show improved perceived target expression and naturalness.

Figures & tables

Figure 1: Affective image editing with IECLA across diverse scenes and target emotions. Given only a target-emotion word, IECLA produces edits that effectively evoke the requested emotion while preserving the source image’s content and structure.

Figure 2: IECLA framework. (a) Counterfactual interventions construct signed supervision. (b) Context-conditioned learning predicts regional contributions. (c) Signed predictions form emotion heatmaps and ground target-aware editing operations.

Figure 3: Representative emotion attribution maps on diverse scenes. Existing approaches mislocalize affective evidence or omit opposing regions. IECLA produces more accurate heatmaps.

Figure 4: Eight-emotion editing with Qwen-Edit, with and without IECLA. IECLA consistently elicits the requested emotion while preserving scene structure across all eight target categories.

Figure 5: Qualitative comparison across six editing backbones, with and without IECLA. IECLA improves target-emotion expression across diverse editors while preserving source structure.

Condition	Target-emotion outcomes				Structural preservation
	Success $\uparrow$	$\Delta p$ $\uparrow$	Emo-A $\uparrow$	Emo-S $\uparrow$	CLIP-I $\uparrow$	G. LPIPS $\downarrow$	G. SSIM $\uparrow$	O. LPIPS $\downarrow$	O. SSIM $\uparrow$	O. MAE $\downarrow$
Qwen-Edit	0.2996	0.2745	0.2588	0.1843	0.8492	0.3698	0.5790	0.4106	0.3049	0.1919
Qwen-Edit + IECLA (ours)	0.3579	0.3344	0.3081	0.2173	0.8527	0.3440	0.6137	0.3767	0.3230	0.1733
-w/o attribution	0.2915	0.2735	0.2454	0.1737	0.8394	0.3513	0.6035	0.3851	0.3184	0.1783
-w/o learning	0.3046	0.2889	0.2724	0.2017	0.8412	0.3454	0.5877	0.4107	0.3147	0.1918
-w/o agent	0.3540	0.3298	0.3024	0.2160	0.8400	0.3543	0.5990	0.3874	0.3171	0.1800

Table 4: Ablation of IECLA’s attribution, learning, and agent modules.

Figure 6: Qualitative ablation on Qwen-Image-Edit. IECLA better expresses target emotions while preserving structure than the baseline and component-removed variants.

Figure 7: Attribution validation and loss-function ablation. (a) Metric values and normalized rank scores for representative attribution methods. (b) Ablation of the training objective, where IECLA provides the strongest overall affective and structural profile.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Additional qualitative results with IECLA-guided Qwen-Edit. Each row presents a source image followed by edits toward different target emotions, showing diverse affective changes with consistent source content and composition.

Figure 9: Additional comparisons between native Qwen-Edit and IECLA-guided editing. IECLA strengthens target-emotion expression while preserving scene structure across diverse images.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	Emo-A $\uparrow$	Emo-S $\uparrow$
SDEdit ( Meng et al., 2022 )	14.1547	0.3820	0.3841	0.7481	0.2050	0.0929
InsDiff ( Geng et al., 2023 )	9.8607	0.3870	0.3999	0.7388	0.0551	0.0537
ControlNet ( Zhang et al., 2023b )	12.2646	0.2370	0.4893	0.6768	0.1860	0.1659
BLIP-Diffusion ( Li et al., 2023 )	8.7242	0.3260	0.5514	0.8169	0.0676	0.0190
CLVA ( Fu et al., 2022 )	12.1975	0.4960	0.3963	0.8572	0.0379	0.0138
AIF ( Weng et al., 2023 )	16.1320	0.6080	0.4147	0.9932	0.0812	0.0345

Appendix

Table 5: Comparison with affective and general-purpose editing baselines on the main editing test set. IECLA provides the strongest joint emotion improvement and structure retention.

Method	Images	Masks	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Heatmap attribution methods
Ablation-CAM	612	5008	-0.0632	–	0.1765	0.5035	–	–	–	–
Grad-CAM	612	5008	0.0740	–	0.2320	0.5814	–	–	–	–
Direct attention	612	5008	0.0507	–	0.2141	0.5541	–	–	–	–
Grad-CAM++	612	5008	0.0697	–	0.2190	0.5737	–	–	–	–
RISE-256	612	5008	0.1688	–	0.2467	0.5986	–	–	–	–

Appendix

Table 6: Cross-dataset attribution results on COCO (612 images, 5,008 masks).

Method	Images	Masks	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Heatmap attribution methods
Ablation-CAM	915	6304	-0.0195	–	0.2634	0.5877	–	–	–	–
Grad-CAM	915	6304	0.0089	–	0.2710	0.6188	–	–	–	–
Direct attention	915	6304	0.0395	–	0.2667	0.6307	–	–	–	–
Grad-CAM++	915	6304	0.0572	–	0.2831	0.6345	–	–	–	–
RISE-256	915	6304	0.1582	–	0.2962	0.6701	–	–	–	–

Appendix

Table 7: Cross-dataset attribution results on EmoSet (915 images, 6,304 masks).

Method	Images	Masks	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Heatmap attribution methods
Ablation-CAM	1544	9966	0.0103	–	0.3394	0.6174	–	–	–	–
Grad-CAM	1544	9966	0.0023	–	0.3342	0.6281	–	–	–	–
Direct attention	1544	9966	0.0418	–	0.3154	0.6378	–	–	–	–
Grad-CAM++	1544	9966	0.0599	–	0.3549	0.6463	–	–	–	–
RISE-256	1544	9966	0.0925	–	0.3472	0.6604	–	–	–	–

Appendix

Table 8: Cross-dataset attribution results on FI (1,544 images, 9,966 masks).

Method	Images	Masks	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Heatmap attribution methods
Ablation-CAM	226	1335	0.0104	–	0.3584	0.6734	–	–	–	–
Grad-CAM	226	1335	0.0531	–	0.3496	0.6925	–	–	–	–
Direct attention	226	1335	-0.1231	–	0.2788	0.6421	–	–	–	–
Grad-CAM++	226	1335	0.1207	–	0.3274	0.6914	–	–	–	–
RISE-256	226	1335	0.0549	–	0.3673	0.6960	–	–	–	–

Appendix

Table 9: Cross-dataset attribution results on Emotion6 (226 images, 1,335 masks).

Method	Images	Masks	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Heatmap attribution methods
Ablation-CAM	73	533	-0.1059	–	0.1781	0.5470	–	–	–	–
Grad-CAM	73	533	0.0319	–	0.2055	0.6264	–	–	–	–
Direct attention	73	533	-0.0193	–	0.1507	0.5811	–	–	–	–
Grad-CAM++	73	533	0.0793	–	0.1918	0.6302	–	–	–	–
RISE-256	73	533	0.1492	–	0.2192	0.6854	–	–	–	–

Appendix

Table 10: Cross-dataset attribution results on EmoArt (73 images, 533 masks).

Heatmap attribution methods
Method	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$
Ablation-CAM ( 2020 )	-0.0137	–	0.2869	0.5909	–	–	–	–
Grad-CAM ( 2017 )	0.0212	–	0.2967	0.6214	–	–	–	–
Direct attention	0.0304	–	0.2777	0.6198	–	–	–	–
Grad-CAM++ ( 2018 )	0.0655	–	0.3053	0.6326	–	–	–	–
RISE-256 ( 2018 )	0.1229	–	0.3136	0.6548	–	–	–	–

Appendix

Table 11: Pooled cross-dataset attribution results under the shared counterfactual teacher. Ranking metrics are image-weighted, whereas Sign Macro-F1 and regression metrics are pooled over valid regions. IECLA leads all reported metrics.

Table 18

Heatmap attribution methods
Method	Imp. $\rho\uparrow$	Signed $\rho\uparrow$	Top-1 $\uparrow$	NDCG@3 $\uparrow$	Pointing $\uparrow$
Ablation-CAM	-0.2207	–	0.5502	0.7809	0.3218
Grad-CAM	-0.0450	–	0.5952	0.8159	0.3426
Direct attention	-0.0294	–	0.5744	0.8214	0.2595
Grad-CAM++	0.0115	–	0.6332	0.8272	0.3391
EigenCAM	0.0491	–	0.6298	0.8321	0.3495

Appendix

Table 16: Attribution agreement with independent Human Review annotations. IECLA leads in region ranking, signed agreement, and localization.

Table 20

Method	Mean score $\uparrow$	Top score $\uparrow$
LayerCAM	0.775	10
CLIP	0.925	12
HiResCAM	0.950	13
ADAPT-TF	0.950	13
DINOV3	0.975	13
Grad-CAM++	0.975	14

Appendix

Table 19: Detailed heatmap User Study results. Mean score averages source-evidence accuracy and focus clarity; top-score counts include task-level ties among 20 tasks.

	Target expression $\uparrow$			Naturalness $\uparrow$
Editor	Native	IECLA-guided	$\Delta$	Native	IECLA-guided	$\Delta$
Qwen-Edit	0.0625	1.325	+1.2625	0.375	1.025	+0.650
HiDream	0.200	1.400	+1.200	0.425	0.675	+0.250
InstructPix2Pix	0.225	0.400	+0.175	0.500	0.575	+0.075
FLUX.2	-0.350	0.150	+0.500	1.050	1.425	+0.375
OmniGen2	0.650	1.450	+0.800	0.350	0.600	+0.250

Appendix

Table 20: Paired editing ratings across six backbones. IECLA guidance improves both target expression and naturalness for every editor.

Variant	Target Success $\uparrow$	$\Delta p$ $\uparrow$	Emo-A $\uparrow$	Emo-S $\uparrow$	CLIP-I $\uparrow$	G. LPIPS $\downarrow$	G. SSIM $\uparrow$	O. LPIPS $\downarrow$	O. SSIM $\uparrow$	O. MAE $\downarrow$
IECLA	0.3579	0.3344	0.3081	0.2173	0.8527	0.3440	0.6137	0.3767	0.3230	0.1733
-w/o $\mathcal{L}_{\mathrm{abs}}$	0.3412	0.3210	0.2901	0.2150	0.8421	0.3507	0.6000	0.3721	0.3185	0.1831
-w/o $\mathcal{L}_{\mathrm{signed}}$	0.3471	0.3226	0.2921	0.2150	0.8415	0.3533	0.5975	0.3751	0.3173	0.1842
-w/o $\mathcal{L}_{\mathrm{focal}}$	0.3390	0.3164	0.2899	0.2110	0.8438	0.3515	0.5996	0.3729	0.3185	0.1834
-w/o $\mathcal{L}_{\mathrm{aux}}$	0.3465	0.3221	0.2914	0.2143	0.8418	0.3534	0.5969	0.3752	0.3169	0.1842
-w/o fusion	0.3542	0.3318	0.2921	0.2139	0.8451	0.3476	0.6073	0.3851	0.3214	0.1746

Appendix

Table 21: Additional editing and loss-function ablations. IECLA provides the strongest overall balance between target-emotion control and structural preservation.

Resolution: $1024\times 576$
	Affective response		Outside-mask preservation
Seed	$\Delta$ Target Success [95% CI]	$\Delta$ Target prob. [95% CI]	Outside-mask $\Delta$ LPIPS	Outside-mask $\Delta$ SSIM
48141	+0.098 [+0.027, +0.179]	+0.098 [+0.034, +0.169]	-0.010	+0.001
48142	+0.107 [+0.018, +0.214]	+0.108 [+0.035, +0.199]	-0.018	+0.002
48143	+0.098 [+0.036, +0.170]	+0.078 [+0.036, +0.128]	-0.005	+0.010
Resolution: $832\times 480$

Appendix

Table 22: Stability across seeds and resolutions. IECLA shows favorable affective and outside-mask preservation differences in all six settings.

Evaluation	Images	Targets/image	Same-emotion	Cross-emotion	Tasks
Main editing pool	680	8	680	4,760	5,440
COCO	612	8	612	4,284	4,896
EmoSet	915	8	915	6,405	7,320
FI	1,544	8	1,544	10,808	12,352
Emotion6	226	8	226	1,582	1,808
EmoArt	73	8	73	511	584

Appendix

Table 24: Editing-task accounting. The main and cross-dataset evaluations include all eight targets, whereas the controlled cross-emotion diagnostics exclude

t=y

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 313 papers

What We Talk About When We Talk About Emotion Editing
Oct 9, 2026Xingzu Zhan, Lin Gu, Ruogu FangImage Editing Affective Computing
Department of Biomedical Engineering, Vanderbilt University, Nashville, Tennessee, USA · Research Institute of Electrical Communication, Tohoku University, Sendai, Japan
What Can I Edit? Open-Ended Strategy Discovery and the Emotion Editability Landscape
Jul 27, 2026Qing Li, Zeyu Dong, Yin Cui +2Text-Guided Image Editing Visual Reasoning
School of Artificial Intelligence, Shenzhen Technology University · School of Innovation Design, Shenzhen Technology University · Stanford University
MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling
May 4, 2026Xinyi Yin, Yiduo Wang, Tingqi Hu +6Valence-Arousal Modeling Image Editing
School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China · School of Journalism and New Media, Xi’an Jiaotong University, Xi’an 710049, China +1

Oct 9, 2026J/K move · Enter open · S save

What We Talk About When We Talk About Emotion Editing

Xingzu Zhan, Lin Gu, Ruogu Fang

Department of Biomedical Engineering, Vanderbilt University, Nashville, Tennessee, USA · Research Institute of Electrical Communication, Tohoku University, Sendai, Japan

Image Editing Affective Computing

Open paper AskPDF arXiv

Emotional image editing aims to change the emotion a scene conveys while preserving its essential content. Because the same scene can support different emotional interpretations, achieving an emotional goal requires visual changes grounded in the source context. Motivated by the Component Process Model, which links emotional responses to event appraisal, we investigate neural representations as a reference for these editing decisions. Using TRIBE v2, an fMRI encoding model, we analyze 400 source-target pairs across eight emotion categories. Hidden states from EmoEdit, EmoEditor, and AffectDelta correspond more strongly to predicted target-image responses in visual regions than in affect-related regions. This imbalance motivates ROI-ReFT, which combines source visual responses as a content reference with a predicted affective response change as an editing direction. Two interacting branches use both conditions to guide a frozen editor throughout generation. Predicted responses of the generated image supervise affective change and constrain visual drift. On a post hoc selected subset of 400 editing requests, ROI-ReFT achieves 91.75% target-emotion accuracy and 83.25% joint success, with the same output both matching the target emotion and reaching source-output DINO-I of at least 0.80. Our code and dataset will be publicly released.

Abstract

Image Editing

Explore similar work

What We Talk About When We Talk About Emotion Editing

What Can I Edit? Open-Ended Strategy Discovery and the Emotion Editability Landscape

MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling