Image aesthetic enhancement aims to perceive aesthetic deficiencies in images and perform corresponding editing operations, which is highly challenging and requires the model to possess creativity and aesthetic perception capabilities. Although recent advancements in image editing models have significantly enhanced their controllability and flexibility, they struggle with enhancing image aesthetic. The primary challenges are twofold: first, following editing instructions with aesthetic perception is difficult, and second, there is a scarcity of "perfectly-paired" images that have consistent content but distinct aesthetic qualities. In this paper, we propose Dual-supervised Image Aesthetic Enhancement (DIAE), a diffusion-based generative model with multimodal aesthetic perception. First, DIAE incorporates Multimodal Aesthetic Perception (MAP) to convert the ambiguous aesthetic instruction into explicit guidance by (i) employing detailed, standardized aesthetic instructions across multiple aesthetic attributes, and (ii) utilizing multimodal control signals derived from text-image pairs that maintain consistency within the same aesthetic attribute. Second, to mitigate the lack of "perfectly-paired" images, we collect "imperfectly-paired" dataset called IIAEData, consisting of images with varying aesthetic qualities while sharing identical semantics. To better leverage the weak matching characteristics of IIAEData during training, a dual-branch supervision framework is also introduced for weakly supervised image aesthetic enhancement. Experimental results demonstrate that DIAE outperforms the baselines and obtains superior image aesthetic scores and image content consistency scores.
Figures & tables
Figure 1: Examples of image aesthetic enhancement. Given the original image, image content and aesthetic descriptions (left), DIAE generates the results with enhanced aesthetic (right). Our DIAE is capable of generating images that are content-consistent with input images while possessing enhanced aesthetics.
Figure 2: Comparison between “imperfectly-paired’ and “perfectly-paired” ( Brooks et al. 2023 )
Figure 3: Overview of DIAE. IIAEData Collection : collecting ‘imperfectly-paired” data for IIAEData, obtaining the image pairs matching and aesthetic assessment prompts through LLaVA-13b ( Liu et al. 2023 ) and UNIAA-LLaVA ( Zhou et al. 2024 ) . Multimodal Aesthetic Perception (MAP) : multimodal aesthetic perception with textual descriptions and HSV and contour maps for image color and image structure. Model Optimization : weakly-supervised diffusion model training strategy with ‘imperfectly-paired” input and reference image, while using MAP through ControlNet ( Zhang et al. 2023 ) .
Figure 4: Guiding DIAE via MAP. (a) Conditioning embeddings cond generation. (b) Using cond to guide diffusion models via ControlNet ( Zhang et al. 2023 ) .
Aesthetics Metrics
Content Consistency Metrics
Methods
LAIONs-based Score
MLLMs-based Score
CLIP-I Score
(res = 256)
(res = 512)
(res = 256)
(res = 512)
(res = 256)
(res = 512)
Original image
4.962
5.123
3.243
3.300
1.000
1.000
ControlNet
4.979
5.522
3.271
3.415
0.628
0.617
InstructPix2Pix
4.991
5.396
3.264
3.325
0.764
0.690
MGIE
4.947
5.519
3.045
3.411
0.557
0.770
Table 1: Quantitative comparison with the state-of-the-arts. LAIONs-based Score and MLLMs-based Score are scored by LAIONS ( Schuhmann 2022a ; Schuhmann 2022b ; Schuhmann 2024 ) and MLLMs ( Zhou et al. 2024 ; You et al. 2024a ) , respectively.
Figure 5: Examples of generative results from low aesthetics quality batch (MOS <4.0 ). The examples with resolution of 512×512 , while the Mean Aesthetic Score (MAS) below is the LAIONs-based Score calculated on the batch.
Figure 6: Examples of generative results from high aesthetics quality batch (MOS >5.0 ). The examples with resolution of 512×512 , while the Mean Aesthetic Score (MAS) below is the LAIONs-based Score calculated on the batch.
Figure 7: Comparison between DIAE with different ts . CLIP-I: the similarity of generated and input images.
Methods
LAIONs
MLLMs
CLIP-I
DIAE ( −w/ov )
5.250
3.343
0.623
DIAE ( −w/ot )
5.428
3.410
0.792
DIAE
5.668
3.501
0.778
Table 2: Ablation study with DIAE, DIAE ( −w/ov ) and DIAE ( −w/ot ) to evaluate the impact of multimodal aesthetic perception. The LAIONs, MLLMs and CLIP-I indicate the 256×256 and 512×512 images’ mean LAIONs-based, MLLMs-based and CLIP-I Scores, respectively.
Traditional Image Aesthetic Assessment (IAA) methods mainly rely on regressing absolute Mean Opinion Scores (MOS). However, such a paradigm overlooks the inherently dynamic nature of human aesthetic perception, which relies on subconscious comparison against implicit visual references. Consequently, the lack of causal reasoning regarding aesthetic differences prevents models from learning generalizable aesthetic principles, thus limiting their generalization across diverse scenarios. In this work, we rethink the IAA task and propose Relative Edit-induced Difference Aesthetic learning (RED-Aes), a novel framework that leverages controllable image editing models to simulate the human aesthetic reasoning process. Instead of fitting absolute score distributions, RED-Aes explicitly learns the visual factors that drive aesthetic changes. To support this paradigm, we construct the RED-20k dataset, which comprises editing-based image pairs, quantitative aesthetic differences, and Chain-of-Thought (CoT) reasoning. Furthermore, we introduce a three-stage training strategy guided by a relative ranking consistency reward, optimizing the model solely via relative supervision. Extensive experiments demonstrate that RED-Aes achieves state-of-the-art performance on multiple public benchmarks, exhibiting superior generalization capabilities.
Unconditional diffusion models offer powerful generative priors, yet steering them toward aesthetically enhanced outputs remains largely unexplored. We show that h-space patching, the dominant paradigm for training-free diffusion editing, systematically fails for global, low-level transformations required for aesthetic and perceptual refinement. We introduce a novel, generalized framework for image-editing in unconditional diffusion models without explicit training. This inference-time mechanism operates on low-level features by extracting degradation concept vectors and combining bottleneck patching with classifier-free guidance to guide sampling away from the degraded manifold, producing consistently improved images without any model retraining.
Editing images via instruction provides a natural way to generate interactive content, but it is a big challenge due to the higher requirement of scene understanding and generation. Prior work utilizes a chain of large language models, object segmentation models, and editing models for this task. However, the understanding models provide only a single modality ability, restricting the editing quality. We aim to bridge understanding and generation via a new multi-modality model that provides the intelligent abilities to instruction-based image editing models for more complex cases. To achieve this goal, we individually separate the instruction editing task with the multi-modality chain of thought prompts, i.e., Chain-of-Thought (CoT) planning, editing region reasoning, and editing. For Chain-of-Thought planning, the large language model could reason the appropriate sub-prompts considering the instruction provided and the ability of the editing network. For editing region reasoning, we train an instruction-based editing region generation network with a multi-modal large language model. Finally, a hint-guided instruction-based editing network is proposed for editing image generations based on the sizeable text-to-image diffusion model to accept the hints for generation. Extensive experiments demonstrate that our method has competitive editing abilities on complex real-world images.