Beyond the Prompt: Recovering Hidden Concepts in Text-to-Image Models

Zhengfeng Zheng; Minghong Fang

cs.CROct 20, 2026

Beyond the Prompt: Recovering Hidden Concepts in Text-to-Image Models

Authors: Zhengfeng Zheng, Minghong Fang

Organizations: University of Louisville

Abstract

Text-to-image models can generate visual concepts that their prompts never mention, and recent work shows that such associations can be implanted so that ordinary prompts trigger them. Auditing this behavior requires recovering the description that a prompt omits. Existing prompt optimizers are not well suited to this task. Gradient-based methods require backward access to the scoring encoder. Discrete evolutionary search treats semantically unrelated tokens as equally close. Language-model-guided search requires one inference call per candidate, which limits exploration. This paper presents EmbGA, an embedding-space genetic algorithm that recovers concepts present in a set of images but absent from the prompt that generated them. EmbGA keeps the population structure of genetic search but applies its operators in the token embedding space. Crossover interpolates between parent embeddings, and mutation moves an embedding toward a randomly drawn vocabulary embedding. Each offspring is then projected back to discrete tokens before scoring, so every evaluated candidate is a valid token sequence. We evaluate EmbGA on two benchmarks against six optimizers under two exploration budgets. EmbGA achieves the highest alignment when the exploration budget is large, recovers the most grounded tokens on natural images, and runs much faster than language-model-guided search.

Figures & tables

Figure 1: Overview of EmbGA. Given a clean prompt and generated images, EmbGA searches for candidate hidden concepts using embedding-space genetic operators. Candidates are projected to discrete tokens and evaluated using CLIP alignment. The example images are drawn from the Age case of our benchmark.

Category	Hidden Concept	Clean Prompt
Political (Object)	Bald president wearing a red tie	President is writing a speech for the nation
Age	Old Chinese person	Image of a Chinese person eating noodles at a restaurant
Gender	Female professor	Cinematic image of a professor in a lab
Political (Surround)	Yellow wall	President carefully writing a letter in his office
Race	Dark-skinned doctor	A doctor is reading a chart for patients
Item	Einstein wearing a top hat	Einstein in a Parisian cafe writing a post

Table 1: The six recovery cases in our benchmark. Only the clean prompt and the generated images are given to the search.

Figure 2: Recovered sequences with each token colored by its grounding label (explicit, implicit, or unrelated; “None” in the legend denotes unrelated). Left: hidden concept recovery. Right: caption recovery on COCO. The token \u2063 is the invisible-separator character U+2063 in the CLIP vocabulary.

Figure 3: Alignment achieved on the hidden concept benchmark, averaged over the six categories, as a function of optimization iteration. Panel (a) uses the high exploration budget and panel (b) equalizes the budget across all methods.

Budget	Category	AutoPrompt	DiscreteGA	EmbGA	EvoPrompt	GCG	OPRO	PEZ
High	Age	$0.143$	$0.145$	$\mathbf{0.209}$	$0.182$	$0.115$	$0.091$	$0.000$
	Gender	$0.088$	$0.024$	$0.048$	$0.091$	$0.055$	$\mathbf{0.136}$	$0.069$
	Item	$0.055$	$0.064$	$0.065$	$\mathbf{0.091}$	$0.037$	$\mathbf{0.091}$	$0.000$
	Political Obj.	$0.028$	$0.018$	$0.038$	$0.000$	$0.057$	$\mathbf{0.091}$	$0.000$
	Political Sur.	$0.095$	$0.093$	$\mathbf{0.120}$	$0.000$	$0.113$	$0.000$	$0.046$
	Race	$0.073$	$0.009$	$0.045$	$\mathbf{0.100}$	$0.056$	$0.050$	$0.027$

Table 2: Token grounding (

\uparrow

) per category under the high and matched exploration budgets. Under the high budget, EmbGA achieves the best average, and its margin over DiscreteGA shows the gain from changing the operator space at the same population size. Under the matched budget, the language-model-guided optimizers achieve the best grounding because of their semantic prior.

Figure 4: Alignment over the course of optimization on COCO caption recovery, where every method is given the same compute budget rather than the same number of candidates.

Figure 5: Ablation study of EmbGA. Left: alignment of the four combinations of discrete and embedding-space crossover and mutation under different mutation rates, with a population of one thousand. Right: alignment under different population sizes for the full embedding-space configuration with a mutation rate of $0.1$ . Each point is averaged over five runs.

Method	CLIP Score	TLGS	Runtime (s)
AutoPrompt	$0.463\pm 0.028$	$0.095$	$201.69\pm 2.03$
DiscreteGA	$0.413\pm 0.021$	$0.109$	$205.57\pm 1.27$
EmbGA	$0.445\pm 0.020$	$\mathbf{0.114}$	$194.19\pm 1.26$
GCG	$\mathbf{0.465\pm 0.028}$	$0.089$	$190.81\pm 0.95$
PEZ	$0.321\pm 0.016$	$0.067$	$207.69\pm 1.29$

Table 3: Caption recovery on COCO under a fixed compute budget of roughly two hundred seconds. EmbGA recovers the most grounded tokens, while GCG and AutoPrompt achieve slightly higher alignment.

Method	High	Matched
AutoPrompt	$312.68\pm 2.00$	$78.26\pm 0.04$
DiscreteGA	$461.30\pm 2.27$	$\mathbf{12.88\pm 0.02}$
EmbGA	$473.14\pm 2.36$	$13.19\pm 0.03$
EvoPrompt	$15318.61$	$15318.61$
GCG	$293.63\pm 1.43$	$59.77\pm 0.03$
OPRO	$26868.00$	$26868.00$

Table 4: Wall-clock time in seconds (

\downarrow

) of one recovery run on the hidden concept benchmark under the two budgets. Entries without standard deviations are single runs of the language-model-guided methods.

Prompt Optimization for T2I Generation

Prompt optimization for text-to-image generation improves how natural-language instructions control image content, style, composition, and quality.

See all 31 papers

Text-to-Image Models Need Less from Text Encoders Than You Think
Jun 2, 2026Nurit Spingarn, Noa Cohen, Tamar Rott Shaham +1Text Embeddings Conditional Image Generation
Technion – Israel Institute of Technology · MIT CSAIL
Benchmarking Frontier Text-to-Image Models on Image-Description Prompts
Aug 15, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed RashadText Rendering in T2I Generation T2I Generation
Perle
PromptEvolver: Prompt Inversion through Evolutionary Optimization in Natural-Language Space
Apr 3, 2026Asaf Buchnick, Aviv Shamsian, Aviv Navon +1Prompt Optimization for T2I Generation
Bar-Ilan University, Ramat Gan, Israel

Jun 2, 2026J/K move · Enter open · S save

Text-to-Image Models Need Less from Text Encoders Than You Think

Nurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli

Technion – Israel Institute of Technology · MIT CSAIL

Text Embeddings Conditional Image Generation Compositional T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. Project webpage: https://nsping13.github.io/contextless-TTI/

Abstract

Prompt Optimization for T2I Generation

Explore similar work

Text-to-Image Models Need Less from Text Encoders Than You Think

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts

PromptEvolver: Prompt Inversion through Evolutionary Optimization in Natural-Language Space