Bridging the Intention-Expression Gap: Aligning Multi-Dimensional Preferences via Hierarchical Relevance Feedback in Text-to-Image Diffusion

Wenxi Wang; Hongbin Liu; Mingqian Li; Junyan Yuan; Junqi Zhang

cs.CVMar 16, 2026

Bridging the Intention-Expression Gap: Aligning Multi-Dimensional Preferences via Hierarchical Relevance Feedback in Text-to-Image Diffusion

Authors: Wenxi Wang, Hongbin Liu, Mingqian Li, Junyan Yuan, Junqi Zhang

Organizations: Tongji University

Abstract

Users often possess a clear visual intent but struggle to articulate it precisely in language. This intention-expression gap makes aligning generated images with latent visual preferences a fundamental challenge in text-to-image diffusion models. Existing methods either require model training, sacrificing flexibility, or rely on textual feedback, imposing a heavy cognitive burden. Although recent training-free methods use click-based binary preference feedback to reduce user effort, they force Foundation Models (FMs) to infer preferences at the semantic level. When faced with multi-dimensional preferences, FMs suffer from inference overload and fail to identify exact preferred feature values under conflicting user signals. Consequently, a flexible framework for multi-dimensional feature alignment remains absent. To address this, we propose a Hierarchical Relevance Feedback-Driven (HRFD) framework. Recognizing that multiple features struggle to converge simultaneously, HRFD organizes them into a three-tier hierarchy and adapts relevance feedback to enforce coarse-to-fine convergence, minimizing cognitive load. To bypass FM inference overload, HRFD decouples the process into independent single-feature preference inference tasks. Furthermore, to overcome FMs' failure in identifying preferred values, HRFD employs statistical inference to quantify the distribution divergence of features between "liked" and "disliked" image sets, achieving robust and transparent preference measurement. Crucially, HRFD operates entirely within the external text space, remaining strictly training-free and model-agnostic. Extensive experiments demonstrate that HRFD effectively captures the user's true visual intent, significantly outperforming baseline approaches.

Figures & tables

Figure 1: The overall framework of the proposed HRFD method.

Metric Group	Metric	LPO	APPO	HRFD (Ours)
Efficiency	Rounds $\downarrow$	15.00 $\pm$ 0.0000	13.93 $\pm$ 1.7348	11.20 $\pm$ 1.4230 ∗∗
Alignment	CLIP $\uparrow$	0.7909 $\pm$ 0.0828	0.8309 $\pm$ 0.0744	0.8649 $\pm$ 0.0447 ∗∗
	LPIPS $\downarrow$	0.7623 $\pm$ 0.0680	0.7285 $\pm$ 0.0685	0.6890 $\pm$ 0.0573 ∗∗
	SSIM $\uparrow$	0.1851 $\pm$ 0.1936	0.2160 $\pm$ 0.1766	0.3137 $\pm$ 0.1583 ∗∗
Visual Quality	Aesthetic $\uparrow$	6.7805 $\pm$ 0.0381	6.8185 $\pm$ 0.0398	6.9249 $\pm$ 0.0379

Table 1: Quantitative comparison of HRFD against baselines. Results are reported as Mean (

\pm

SD). The best performance is in bold . The metrics are grouped by their primary purpose. Arrows indicate preferred direction (

\uparrow

for higher is better,

\downarrow

for lower is better). The symbols ∗ and ∗∗ indicate statistically significant improvements over the strongest baseline (APPO) at

p<0.05

and

p<0.01

, respectively.

Panel of Figure 2 — Figure 2: Quantitative performance evolution across interaction rounds in Scenario I. The curves report the mean and standard deviation for SSIM, CLIP Similarity, and LPIPS Distance, demonstrating the more stable and superior convergence of HRFD.

Figure 3: Subjective evaluation results for Scenario II. HRFD demonstrates superior efficiency (Rounds), lower cognitive load (NASA-TLX), and higher creativity support (CSI) compared to the baseline (*

p<0.05

, **

p<0.01

Variant	Rounds $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	Aesthetic $\uparrow$
w/o Hierarchy	14.63	0.7553	0.7932	0.1491	7.0547
w/o History	14.20	0.8196	0.7610	0.1903	6.9007
w/o Weighting	12.93	0.8334	0.7346	0.2186	6.9500
HRFD (Ours)	11.20	0.8649	0.6890	0.3137	6.9249

Table 2: Ablation study results. The best performance is in bold . Arrows indicate preferred direction (

\uparrow

higher is better,

\downarrow

lower is better).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hierarchy	Feature	Type	Values / Levels
L1: Macro	Art Style	Discrete	realistic, illustration, flat_illustration, cartoon, anime, 3d_render, pixel_art, watercolor, oil_painting, sketch, line_art, minimalist, surreal, collage, mixed_media
	Style Era	Discrete	classical, vintage, modern, futuristic
	Cultural Style	Discrete	asian, western, african, middle_eastern, mixed
	Emotion Type	Discrete	serene, joyful, romantic, melancholic, mysterious, tense, playful, ominous, angry
	Environment	Discrete	indoor, outdoor, studio, nature, urban, abstract, fantasy, mixed
L2: Meso	Color Palette	Discrete	monochrome, limited_palette, analogous, complementary, triadic, varied

Appendix

Table 3: Complete Hierarchical Feature Standard Repository

Layer	Feature	Strict Match	Mean Pairwise
L1	Style: Art Style	100.00	100.00
	Style: Style Era	100.00	100.00
	Style: Cultural Style	99.33	99.56
	Background: Presentation Mode	99.33	99.56
	Background: Environment	98.67	98.89
	Emotion: Emotional Type	98.00	98.67

Appendix

Table 4: Per-feature VLM extraction reliability across three independent runs (150 images). Strict Match indicates the percentage of images where all three VLM runs unanimously agreed with human expert annotations. Mean Pairwise denotes the average consistency across all three possible pairs of independent runs.

Figure 4: The user interface of our HRFD system.

Figure 5: The progressive convergence process of HRFD.

Figure 6: Qualitative comparison of representative generated images across different methods in the Objective Preference Alignment scenario. Compared to LPO and APPO, our HRFD framework successfully captures multi-dimensional user preferences, generating results that most closely align with the Ground Truth (GT).

Category	Parameter	Value
Interaction	Max Interaction Rounds ( $T_{max}$ )	15
	Candidate Images per Round ( $N$ )	8
	Auto-locking Threshold ( $N_{\text{lock}}$ )	4
Generation	Generation Backend	FLUX.2-Klein (Distilled)
	Image Resolution	$512\times 512$
	Sampling Steps	4

Appendix

Table 5: Key Hyperparameters for the HRFD Framework

Stage	Avg. Latency	Execution Mode	Computing Backend
Feature Extraction	3.68 s	Parallel API	VLM
Preference Inference	$<$ 1 ms	Local Compute	CPU (Algorithm)
Prompt Generation	7.25 s	Parallel API	VLM
Image Rendering	20.88 s	Serial Local	Flux.2-Klein
Total Round Latency	31.81 s	Mixed	Current Config.

Appendix

Table 6: Latency decomposition of the HRFD framework per interaction round.

Metric	Test Statistic ( $W$ )	$p$ -value
Efficiency Metric
Interaction Rounds $\downarrow$	0.00	$0.0073^{**}$
NASA-TLX Metrics (0–100)
Mental Demand $\downarrow$	23.00	$0.0349^{*}$
Physical Demand $\downarrow$	58.50	$0.9320$
Temporal Demand $\downarrow$	19.50	$0.0213^{*}$

Appendix

Table 7: Wilcoxon Signed-Rank Test Results for Interaction Rounds, NASA-TLX, and CSI Metrics

Diffusion Model Alignment

Diffusion model alignment steers generated content and internal representations toward prompts, human preferences, safety goals, and task constraints.

See all 185 papers

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
May 26, 2026Austin Wang, Jiaqi Han, Stefano Ermon +1Diffusion Model Alignment Preference Optimization
Caltech · Stanford University
Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
Feb 21, 2026Zhou Jiang, Yandong Wen, Zhen LiuDiffusion Model Alignment Preference Alignment
Westlake University · The Chinese University of Hong Kong, Shenzhen
Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation
Mar 21, 2026Zihao Wang, Yuxiang Wei, Xinpeng Zhou +5T2I Generation User Preference Modeling
Harbin Institute of Technology · Duxiaoman

May 26, 2026J/K move · Enter open · S save

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue

Caltech · Stanford University

Diffusion Model Alignment Preference Optimization Preference Optimization for Diffusion Models T2I Diffusion Models

Open paper AskPDF arXiv

Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.

Abstract

Diffusion Model Alignment

Explore similar work

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance

Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation