cs.CVOct 21, 2026

T-DIG: Text-Conditioned Diversity for Image Generation

Authors: Simran Khanuja, Rohit Gandikota, Gaurav Parmar, Graham Neubig

Organizations: Carnegie Mellon University · Northeastern University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Text-to-image (T2I) platforms typically return several images per prompt so users can explore different visual interpretations and choose one that matches their intent. However, T2I models are usually optimized for single-image quality which comes at a cost of multi-image diversity, limiting this exploration. Diversity is also contextual: one user wants variations in background, while another wants them in artistic style, and the same user may later want variations in clothing. The generated images should differ along whatever axis the user cares about. In this paper, we introduce T-DIG, a text-conditioned diversity framework for image generation and editing. T-DIG is an iterative framework comprising multiple modules that lets users type prompts, inspect default generations, request variations using natural language, and continue refining outputs until they're satisfied. At its core, T-DIG uses an LLM policy that produces K axis-conditioned prompts or edit instructions for a frozen generator or editor. We train this policy with GRPO using a text-conditioned diversity reward and alignment penalties. Because the image generator remains frozen, T-DIG applies to open-weight, distilled, and proprietary models. We also release the first dataset for text-conditioned prompt diversity in image generation, with the training and validation splits built from human-written axes and a test set drawn from real user interactions with T-DIG. T-DIG far exceeds baselines in improving text-conditioned diversity, while preserving base-prompt alignment across a variety of metrics and human judgements. We show that user-directed diversity exploration can be added to a wide range of image generation models, open and closed, without the need to access their weights. Website: https://image-study.org/t-dig/

Figures & tables

Figure 1: Text-conditioned Diversity: For one prompt, i.i.d. sampling and prior methods return one set regardless of what the user wants to vary; T-DIG produces a different set for each text-condition, diversity strength, and can edit/generate, accounting for user preferences.
Figure 2: T-DIG framework. After inspecting default samples, the user writes an axis or picks a suggested one, sets the diversity strength, and chooses to generate a new set or edit the current one. A trained policy writes KK prompts or edit instructions for a frozen T2I model or editor, and the user accepts the result or refines it for another round. Only the policies are trained.
 DiversityAlignmentPreferenceHumanTrade-off vs. I.I.D.
MethodTPIPS vDINO vDream- Sim vVQACLIPIRHPSv3Aes.Win (%)TPIPS v ( ×\times )VQA (%)Slope
Qwen-Image
I.I.D.0.1240.2090.1940.8890.8091.2711.555.9568.01.000.0–
Group Inf.0.1680.2810.2630.8850.8121.2211.435.9176.01.36−- 0.4−- 0.008
CADS0.2930.5020.5400.8080.7600.468.765.7780.82.37−- 9.1−- 0.048
PAG0.2180.3810.3260.7250.7360.7710.356.35–1.76−- 18.4−- 0.174
Table 1: Results on the interactive test set: K=4K{=}4 , 9B policy; higher is better for every column. Diversity : TPIPS v , DINO v and DreamSim v are normalized Vendi scores (range [0,1][0,1] ) of TPIPS embeddings conditioned on the requested axis, and of DINOv2 and DreamSim embeddings, which ignore the axis. Alignment : VQAScore and CLIPScore against the base prompt. Preference & aesthetics : ImageReward, HPSv3 and LAION-Aesthetics V2. Human : % of instances where raters scored T-DIG α=0.5 ’s set as more varied along the axis than row method and at least as well aligned with the prompt. Trade-off (vs. I.I.D.) : TPIPS v as a multiple of i.i.d. sampling’s, the relative change in VQAScore, and the slope of the line from i.i.d. sampling, i.e. the change in VQAScore per +0.1+0.1 TPIPS v . Bold and underline mark best and second best respectively.
Figure 4
 Added to TPIPS vHeld-out pairs  
RewardLalignL_{\mathrm{align}}LlenL_{\mathrm{len}}LprefL_{\mathrm{pref}}TPIPS v↑{}_{v}\uparrowVQA ↑\uparrowHPS ↑\uparrowIR ↑\uparrowWordsMalf. ↓\downarrow
Untrained   0.3330.86512.341.19181
TPIPS v only   0.6170.2281.44−1.74-1.74907
+ alignment✓  0.4580.85711.641.041231
+ length✓✓ 0.4750.84511.380.81341
+ preference ( T-DIG )✓✓✓0.4540.84912.351.03290
Table 2: Reward ablation on FLUX.2 [klein], 9B policy on the held-out pairs. Words: mean words per expanded prompt. Malf.: malformed completions.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: i.i.d. sampling (left) vs. T-DIG ( α=0.5\alpha{=}0.5 , right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 4: Prompt used to filter the HPDv2 base-prompt pool (part 1 of 2).
Figure 5: Prompt used to filter the HPDv2 base-prompt pool (part 2 of 2).
Figure 6: Prompt used for image-conditioned axis elicitation.
Figure 7: Prompt used to train the generation policy with GRPO, where KK is the number of prompts the policy writes per completion.
Figure 8: Prompt used to train the editing policy with GRPO where KK is the number of source images and edit instructions the policy writes. Unlike the generation task, the word cap is a fixed 30 words per instruction and does not depend on the renderer.
Participant commentWhat it shows / what to improve
Overall experience (end-of-study questionnaire)
“As someone who enjoys using image generators, I did purposely try to use a couple of difficult prompts that assistants often mess up, and this model did a better job than I was expecting, to be honest. Impressive.” [FLUX]Experienced users rate it above their expectations on hard prompts.
“I would say the tool overall is quite good compared to other tools I have used before for image generation.” [SD3.5-L]Favourable comparison with existing tools.
“This was really fun. I’d definitely use such a tool at a reasonable asking price.” [Qwen]Willingness to use, and pay for, axis-conditioned variety.
“[…] if I put more time in to tweaking this I could see making something like this part of my creative workflow.” [SD3.5-L]Fits a creative workflow.
“The tool covered most of the variations I needed and was easy to use.” [Qwen]Covers users’ axes; easy to use.
Appendix
Table 3: Participant feedback from the interactive studies. Quotes are verbatim, trimmed where marked. FLUX: FLUX.2 [klein]; SD3.5-M/L: SD3.5 Medium/Large; Qwen: Qwen-Image.
Figure 9: Qwen-Image (1/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 10: Qwen-Image (2/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 11: Qwen-Image (3/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 12: SD3.5 L (1/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 13: SD3.5 L (2/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 14: SD3.5 L (3/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 15: SD3.5 M (1/2): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 16: SD3.5 M (2/2): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 17: FLUX.2 [klein] (1/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 18: FLUX.2 [klein] (2/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
Figure 19: FLUX.2 [klein] (3/3): i.i.d. sampling (left) vs. T-DIG ( α\alpha =0.5, right) on requests from the interactive test set. Prompts and axes are the study participants’ own text, verbatim.
More in this topic

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 294 papers

Explore similar work

CardsList
Jun 5, 2026

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation

Recent text-to-image models built on large-scale Transformer backbones and flow-based objectives deliver strong text-image alignment and high visual quality, yet often produce overly similar samples under a fixed prompt. Existing diversity-enhancement methods alleviate this issue, but typically require expensive sampling or auxiliary optimization, incurring non-trivial overhead. To investigate the root cause of this homogeneity, we examine intermediate Transformer features and observe that the zero-frequency spatial average (DC) component rapidly converges across seeds early in generation, causing early trajectory lock-in that limits downstream variation. Building on this observation, we propose DC Attenuation for diVersity Enhancement (DAVE), a training-free representation-level intervention that selectively attenuates this component in the early regime. DAVE preserves the sampling pipeline with negligible overhead, improving prompt-consistent diversity while maintaining competitive image quality.

Dahee Kwon, Haeun Lee, Jaesik Choi

Korea Advanced Institute of Science and Technology (KAIST) · INEEJI

Diffusion TransformerT2I Generation
AskPDF
Jul 20, 2026

MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis

Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text, by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.

Pengcheng Wan, Liang Han, Lin Xu +2

Harbin Institute of Technology (Shenzhen)

Conditional Image GenerationControllable Image GenerationT2I Diffusion ModelsDiffusion Models
AskPDF
Feb 19, 2026

GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation

Despite high semantic alignment, modern text-to-image (T2I) generative models still struggle to synthesize diverse images from a given prompt. In this work, we enhance the T2I diversity through a geometric lens. Unlike most existing methods that rely primarily on entropy-based guidance to increase sample dissimilarity, we introduce Geometry-Aware Spherical Sampling (GASS) to enhance diversity by explicitly controlling both prompt-dependent and prompt-independent sources of variation. Specifically, we decompose the diversity measure in CLIP embeddings using two orthogonal directions: the text embedding, which captures semantic variation related to the prompt, and an identified orthogonal direction that captures prompt-independent variation (e.g., backgrounds). Based on this decomposition, GASS increases the geometric projection spread of generated image embeddings along both axes and guides the T2I sampling process via expanded predictions along the generation trajectory. Our experiments on different frozen T2I backbones (U-Net and DiT, diffusion and flow) and benchmarks demonstrate the effectiveness of disentangled diversity enhancement with minimal impact on image fidelity and semantic alignment.

Ye Zhu, Kaleb S. Newman, Johannes F. Lutzeyer +3

Laboratoire d’Informatique (LIX), CNRS, École Polytechnique, IPP, France · Department of Computer Science, Princeton University, USA · FAIR at Meta - Montreal, Canada +3

T2I GenerationT2I Diffusion ModelsDiffusion Sampling
AskPDF