Users often possess a clear visual intent but struggle to articulate it precisely in language. This intention-expression gap makes aligning generated images with latent visual preferences a fundamental challenge in text-to-image diffusion models. Existing methods either require model training, sacrificing flexibility, or rely on textual feedback, imposing a heavy cognitive burden. Although recent training-free methods use click-based binary preference feedback to reduce user effort, they force Foundation Models (FMs) to infer preferences at the semantic level. When faced with multi-dimensional preferences, FMs suffer from inference overload and fail to identify exact preferred feature values under conflicting user signals. Consequently, a flexible framework for multi-dimensional feature alignment remains absent. To address this, we propose a Hierarchical Relevance Feedback-Driven (HRFD) framework. Recognizing that multiple features struggle to converge simultaneously, HRFD organizes them into a three-tier hierarchy and adapts relevance feedback to enforce coarse-to-fine convergence, minimizing cognitive load. To bypass FM inference overload, HRFD decouples the process into independent single-feature preference inference tasks. Furthermore, to overcome FMs' failure in identifying preferred values, HRFD employs statistical inference to quantify the distribution divergence of features between "liked" and "disliked" image sets, achieving robust and transparent preference measurement. Crucially, HRFD operates entirely within the external text space, remaining strictly training-free and model-agnostic. Extensive experiments demonstrate that HRFD effectively captures the user's true visual intent, significantly outperforming baseline approaches.
Figures & tables
Figure 1: The overall framework of the proposed HRFD method.
Metric Group
Metric
LPO
APPO
HRFD (Ours)
Efficiency
Rounds ↓
15.00 ± 0.0000
13.93 ± 1.7348
11.20 ± 1.4230 ∗∗
Alignment
CLIP ↑
0.7909 ± 0.0828
0.8309 ± 0.0744
0.8649 ± 0.0447 ∗∗
LPIPS ↓
0.7623 ± 0.0680
0.7285 ± 0.0685
0.6890 ± 0.0573 ∗∗
SSIM ↑
0.1851 ± 0.1936
0.2160 ± 0.1766
0.3137 ± 0.1583 ∗∗
Visual Quality
Aesthetic ↑
6.7805 ± 0.0381
6.8185 ± 0.0398
6.9249 ± 0.0379
Table 1: Quantitative comparison of HRFD against baselines. Results are reported as Mean ( ± SD). The best performance is in bold . The metrics are grouped by their primary purpose. Arrows indicate preferred direction ( ↑ for higher is better, ↓ for lower is better). The symbols ∗ and ∗∗ indicate statistically significant improvements over the strongest baseline (APPO) at p<0.05 and p<0.01 , respectively.
Figure 2: Quantitative performance evolution across interaction rounds in Scenario I. The curves report the mean and standard deviation for SSIM, CLIP Similarity, and LPIPS Distance, demonstrating the more stable and superior convergence of HRFD.
Figure 3: Subjective evaluation results for Scenario II. HRFD demonstrates superior efficiency (Rounds), lower cognitive load (NASA-TLX), and higher creativity support (CSI) compared to the baseline (* p<0.05 , ** p<0.01 ).
Variant
Rounds ↓
CLIP ↑
LPIPS ↓
SSIM ↑
Aesthetic ↑
w/o Hierarchy
14.63
0.7553
0.7932
0.1491
7.0547
w/o History
14.20
0.8196
0.7610
0.1903
6.9007
w/o Weighting
12.93
0.8334
0.7346
0.2186
6.9500
HRFD (Ours)
11.20
0.8649
0.6890
0.3137
6.9249
Table 2: Ablation study results. The best performance is in bold . Arrows indicate preferred direction ( ↑ higher is better, ↓ lower is better).
Table 3: Complete Hierarchical Feature Standard Repository
Layer
Feature
Strict Match
Mean Pairwise
L1
Style: Art Style
100.00
100.00
Style: Style Era
100.00
100.00
Style: Cultural Style
99.33
99.56
Background: Presentation Mode
99.33
99.56
Background: Environment
98.67
98.89
Emotion: Emotional Type
98.00
98.67
Appendix
Table 4: Per-feature VLM extraction reliability across three independent runs (150 images). Strict Match indicates the percentage of images where all three VLM runs unanimously agreed with human expert annotations. Mean Pairwise denotes the average consistency across all three possible pairs of independent runs.
Figure 4: The user interface of our HRFD system.
Figure 5: The progressive convergence process of HRFD.
Figure 6: Qualitative comparison of representative generated images across different methods in the Objective Preference Alignment scenario. Compared to LPO and APPO, our HRFD framework successfully captures multi-dimensional user preferences, generating results that most closely align with the Ground Truth (GT).
Category
Parameter
Value
Interaction
Max Interaction Rounds ( Tmax )
15
Candidate Images per Round ( N )
8
Auto-locking Threshold ( Nlock )
4
Generation
Generation Backend
FLUX.2-Klein (Distilled)
Image Resolution
512×512
Sampling Steps
4
Appendix
Table 5: Key Hyperparameters for the HRFD Framework
Stage
Avg. Latency
Execution Mode
Computing Backend
Feature Extraction
3.68 s
Parallel API
VLM
Preference Inference
< 1 ms
Local Compute
CPU (Algorithm)
Prompt Generation
7.25 s
Parallel API
VLM
Image Rendering
20.88 s
Serial Local
Flux.2-Klein
Total Round Latency
31.81 s
Mixed
Current Config.
Appendix
Table 6: Latency decomposition of the HRFD framework per interaction round.
Metric
Test Statistic ( W )
p -value
Efficiency Metric
Interaction Rounds ↓
0.00
0.0073∗∗
NASA-TLX Metrics (0–100)
Mental Demand ↓
23.00
0.0349∗
Physical Demand ↓
58.50
0.9320
Temporal Demand ↓
19.50
0.0213∗
Appendix
Table 7: Wilcoxon Signed-Rank Test Results for Interaction Rounds, NASA-TLX, and CSI Metrics
Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.
Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a contrastive guidance vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.
Zhou Jiang, Yandong Wen, Zhen Liu
Westlake University · The Chinese University of Hong Kong, Shenzhen
Text-to-image generation has advanced rapidly, yet it still struggles to capture the nuanced user preferences. Existing approaches typically rely on multimodal large language models to infer user preferences, but the derived prompts or latent codes rarely reflect them faithfully, leading to suboptimal personalization. We present Premier, a novel preference modulation framework for personalized image generation. Premier represents each user's preference as a learnable embedding and introduces a preference adapter that fuses the user embedding with the text prompt. To enable accurate and fine-grained preference control, the fused preference embedding is further used to modulate the generative process. To enhance the distinctness of individual preference and improve alignment between outputs and user-specific styles, we incorporate a dispersion loss that enforces separation among user embeddings. When user data are scarce, new users are represented as linear combinations of existing preference embeddings learned during training, enabling effective generalization. Experiments show that Premier outperforms prior methods under the same history length, achieving stronger preference alignment and superior performance on text consistency, ViPer proxy metrics, and expert evaluations.