Text-to-image generation has advanced rapidly, yet it still struggles to capture the nuanced user preferences. Existing approaches typically rely on multimodal large language models to infer user preferences, but the derived prompts or latent codes rarely reflect them faithfully, leading to suboptimal personalization. We present Premier, a novel preference modulation framework for personalized image generation. Premier represents each user's preference as a learnable embedding and introduces a preference adapter that fuses the user embedding with the text prompt. To enable accurate and fine-grained preference control, the fused preference embedding is further used to modulate the generative process. To enhance the distinctness of individual preference and improve alignment between outputs and user-specific styles, we incorporate a dispersion loss that enforces separation among user embeddings. When user data are scarce, new users are represented as linear combinations of existing preference embeddings learned during training, enabling effective generalization. Experiments show that Premier outperforms prior methods under the same history length, achieving stronger preference alignment and superior performance on text consistency, ViPer proxy metrics, and expert evaluations.
Figures & tables
Figure 2: Premier training framework. (a) During the training of the preference adapters, the user preference embeddings and the adapters are jointly optimized. The block-shared adapter produces a uniform modulation direction across all DiT blocks, whereas the block-distinct adapter generates different modulation directions for different DiT blocks. (b) Each preference adapter takes the learnable user embedding and the input text tokens as inputs, and outputs a preference modulation direction for every text token, enabling fine-grained and context-aware modulation. (c) Our method obtains the new user’s preference embedding as a linear combination of training-set user preference embeddings. During this stage, only the linear combination coefficients are optimized. This strategy yields a more stable user preference embedding when the user’s historical data is limited.
Figure 3: Qualitative comparisons of Preference Alignment. We compare the performance of our method with other approaches in user preference-aware image generation. The images generated by our method are closest to the user’s preferences while remaining faithful to the user-provided text prompt.
Figure 4: User study results of our method compared with other methods. Each human expert is presented with six historical preference images from the user, along with image pairs generated by our method and other baselines under the same text prompt. Experts are asked to select the image that best aligns with both the user’s preferences and the input text.
Figure 5: Qualitative ablation comparison of our method. Ablating either of the two preference adapters leads to a significant performance drop, confirming their necessity. Ablating the text-preference modulation also degrades user-preference-aware image generation.
Data
Flux [ 18 ]
InstantStyle [ 36 ]
Bagel [ 7 ]
Qwen Image Edit [ 41 ]
DrUM [ 17 ]
ViPer [ 30 ]
Ours
ViPer Score ↑
0.8890
0.3953
0.6277
0.5075
0.4688
0.4791
0.5159
0.6889
ViPer Rate ↑
-
-
0.777
0.703
0.613
0.573
0.676
0.876
CLIP T2I ↑
0.3027
0.3089
0.2988
0.3107
0.3101
0.3072
0.2981
0.3183
LPIPS ↓
-
-
0.6641
0.6438
0.6407
0.6541
0.6564
0.5986
Table 1: Quantitative comparison of preference alignment in image generation across different methods. The best results are highlighted in bold, while the second-best is underlined. DrUM, ViPer, and our method take 8 user historical data for preference generation, whereas Bagel, Qwen-Image-Edit, and InstantStyle use a single user history data. Under evaluation by the ViPer proxy model, our method achieves the highest alignment with user preferences, while also generating images with high text-image fidelity.
ViPer Score ↑
ViPer Rate ↑
Clip Score T2I ↑
LPIPS ↓
Ours w/o Δshared
0.4818
0.667
0.3162
0.6247
Ours w/o Δdistinct
0.4917
0.669
0.3131
0.6353
Ours w/o Disp Loss
0.4498
0.618
0.3162
0.6249
Ours w/o PPM
0.6492
0.840
0.3074
0.6225
Ours
0.6889
0.876
0.3183
0.5986
Table 2: Quantitative evaluation of preference image generation after ablation. The best results are highlighted in bold, while the second-best is underlined. Ablating the dispersion loss leads to a significant performance drop, while removing the modulation of prompt–preference interaction yields suboptimal results.
Figure 6: The relationship between ViPer Score and user history length. Our method’s strategy of training linear combination coefficients outperforms direct embedding training when the user history is limited.
Figure 7: The relationship between LPIPS and user history length. Our linear combination approach demonstrates a more pronounced advantage in LPIPS, consistently outperforming direct training when the history size is up to 16 samples.
Figure 8: Qualitative dispersion loss ablation comparison of our method. After ablating the dispersion loss, the generated preference images across different users exhibit substantially reduced variation.
Figure 9: Qualitative comparison of our method across different user history lengths and different training strategy When the amount of user history is limited, training linear combination coefficients yields more stable performance.
Data
Flux [ 18 ]
LoRA [ 22 ]
Ours
ViPer [ 30 ] Score ↑
0.8890
0.3953
0.7096
0.6889
ViPer Rate ↑
-
-
0.906
0.876
CLIP [ 29 ] T2I ↑
0.3027
0.3089
0.3011
0.3183
LPIPS [ 48 ] ↓
-
-
0.6037
0.5986
Table A : Quantitative comparison with LoRA training in terms of preference alignment in generated images.
Data
Ours
Bagel
Qwen-Image-Edit
ViPer
InstantStyle
DrUM
ViPer Score ↑
0.8919
0.7204
0.7149
0.6696
0.5779
0.6232
0.6815
ViPer Rate ↑
-
0.6796
0.6591
0.5399
0.4613
0.476
0.6134
CLIP T2I ↑
0.2844
0.2929
0.2653
0.2877
0.2823
0.2942
0.293
LPIPS ↓
-
0.6002
0.6297
0.6068
0.6306
0.643
0.6208
Table B : Quantitative comparisons on the PIP-dataset.
Ours
Ours 100 users
Ours λdistinct=0.1,λshare=0.01
Ours λdistinct=0.01,λshare=0.1
Ours λdistinct=0.1,λshare=1
Ours λdistinct=1,λshare=0.1
ViPer Score ↑
0.7005
0.5013
0.5635
0.6632
0.6703
0.6935
ViPer Rate ↑
0.8788
0.7115
0.7884
0.875
0.85
0.8807
CLIP T2I ↑
0.3147
0.3152
0.3153
0.3151
0.3139
0.3147
LPIPS ↓
0.5946
0.6261
0.6131
0.6014
0.6087
0.5923
Table C : Results for scaling and sensitivity analysis.
Figure A : Qualitative comparison on PIP-dataset.
Figure B : Qualitative results on real user preference data.
Figure C : Qualitative comparison with LoRA-generated results.
Figure D : Qualitative comparison with other methods.
Figure E : Qualitative comparison with other methods.
Figure F : Qualitative comparison with other methods.
Figure G : Qualitative comparison with other methods.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.