PromptEvolver: Prompt Inversion through Evolutionary Optimization in Natural-Language Space

Asaf Buchnick; Aviv Shamsian; Aviv Navon; Ethan Fetaya

cs.LGApr 3, 2026

PromptEvolver: Prompt Inversion through Evolutionary Optimization in Natural-Language Space

Authors: Asaf Buchnick, Aviv Shamsian, Aviv Navon, Ethan Fetaya

Organizations: Bar-Ilan University, Ramat Gan, Israel

Abstract

Text-to-image generation has progressed rapidly, but faithfully generating complex scenes requires extensive trial-and-error to find the exact prompt. In the prompt inversion task, the goal is to recover a textual prompt that can faithfully reconstruct a given target image. Currently, existing methods frequently yield suboptimal reconstructions and produce unnatural, hard-to-interpret prompts that hinder transparency and controllability. In this work, we present PromptEvolver, a prompt inversion approach that generates natural-language prompts while achieving high-fidelity reconstructions of the target image. Our method uses a genetic algorithm to optimize the prompt, leveraging a strong vision-language model to guide the evolution process. Importantly, it works on black-box generation models by requiring only image outputs. Finally, we evaluate PromptEvolver across multiple prompt inversion benchmarks and show that it consistently outperforms competing methods.

Figures & tables

Panel of Figure 1 — Figure 1: Reconstruction comparison : PromptEvolver versus a VLM baseline. We demonstrate the robustness of PromptEvolver to challenging T2I concepts such as colors, counting, object positioning, and camera orientation.

Figure 2: Overview of PromptEvolver. Given a target image, a VLM generates an initial population of $N$ diverse candidate prompts. The evolutionary loop then repeats for $T$ generations: ( A ) two parents are selected via tournament selection and combined by the VLM into a child prompt through crossover , conditioned on the target image; ( B ) the child optionally undergoes mutation , where the VLM applies targeted edits; ( C ) the child prompt is evaluated by generating $K$ images via the T2I model and computing the mean image-to-image similarity to the target; ( D ) the top- $N$ individuals from parents and offspring are retained. After $T$ generations, the highest-fitness prompt and its best reconstruction are returned.

Panel of Figure 3 — Figure 3: Prompt evolution flow example in PromptEvolver. A VLM generates an initial population of $N$ candidate prompts. Each generation, two parents (orange) are selected, combined through crossover , and optionally refined via mutation , where all VLM operations are guided by the target image.

Figure 4: Per-image comparison of VLM-Baseline ( $x$ -axis) versus PromptEvolver ( $y$ -axis) across all $500$ evaluation images, for four similarity metrics. Each point is one image, colored by source dataset; the dashed line is $y=x$ . Points above the diagonal are images where evolution improved the reconstruction. The legend in the top-left of each panel reports the per-metric win counts (PromptEvolver vs. VLM-Baseline). For each panel we use the PromptEvolver variant that was guided by that metric.

Panel of Figure 5 — Figure 5: Qualitative comparison across five diverse datasets. Given a target image (top), we show reconstructions from PromptEvolver (middle) and VLM-Baseline (bottom). Our evolutionary approach produces more faithful reconstructions, better preserving scene composition, style, and fine-grained details.

Template	VLM Variant	CLIP $\uparrow$	Runtime (s) $\downarrow$
Structured	Instruct	$0.83\pm 0.007$	381.18
Structured	Thinking	$0.83\pm 0.007$	885.6
Minimal	Instruct	$0.82\pm 0.007$	420.72
Minimal	Thinking	$0.83\pm 0.007$	822.72

Table 2: Ablation: VLM template style vs. reasoning mode. We compare structured and minimal prompting templates with instruct and thinking VLM variants, reporting CLIP similarity and runtime (seconds per image).

Crossover Input	CLIP $\uparrow$
Target image only (PromptEvolver)	0.829
Target image + generated parent images	0.827

Table 3: Ablation: effect of providing generated parent images during crossover. In the default setting, the VLM observes only the target image when combining two parent prompts. We compare this against additionally providing the images generated by each parent prompt. Evaluated with CLIP guidance on 100 random MS-COCO 2017 images.

Mutation Rate ( $p_{m}$ )	CLIP $\uparrow$	BLIP $\uparrow$	OpenCLIP $\uparrow$	DreamSim $\uparrow$
0.1 (default)	0.829	0.859	0.810	0.754
0.3	0.831	0.861	0.810	0.754
0.5	0.834	0.861	0.811	0.756

Table 5: Ablation: effect of mutation rate

p_{m}

. We vary the probability with which offspring undergo mutation after crossover. The default rate used by PromptEvolver is

0.1

. Evaluated with CLIP guidance on 100 random MS-COCO 2017 images.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure C1 : Prompt evolution example (Flickr8K, DreamSim guidance). The evolved prompt adds specific details such as “red star decals” and “golden hour” that improve reconstruction fidelity.

Panel of Figure 10 — Figure C2 : Prompt evolution example (MS-COCO, CLIP guidance). The initial prompt was truncated by the CLIP truncation mechanism detailed in A . Evolution produces a complete, structured description that better captures both signs.

Panel of Figure 11 — Figure C3 : Prompt evolution example (MS-COCO, SigLIP guidance). The evolved prompt corrects the camera angle to “low-angle, slightly elevated” and adds texture details like “weathered green grill” and “sharp shadows.”

Panel of Figure 12 — Figure C4 : Prompt evolution example (MS-COCO, SigLIP guidance). Evolution recovers fine-grained text details (“LTD,” license plate “EX52 XRT,” “green recycling logos”) absent from the initial prompt.

Panel of Figure 13 — Figure D5 : Failure case: The T2I model inferred the bowtie correctly without specific mention in the prompt (presumably because of the tuxedo), however, the fact that it is open was not mentioned in any of the prompts throughout the evolution process. We hypothesize that this is due to the fact that the unnatural state of the bowtie receives a lot of human attention, however modern foundational models have a bias towards the more common state.

Figure E6 : Screenshot of the human evaluation interface for a single comparison. The reference image is shown at the top, with two reconstruction options displayed below. Evaluators select the option that more faithfully matches the reference.

Dataset	Ours	VLM-Baseline	Total	Pref. (%)
Lexica	124	73	197	62.9
CelebA	84	108	192	43.8
Flickr8K	111	85	196	56.6
LAION-400M	106	91	197	53.8
MS-COCO	96	78	174	55.2
Total	521	435	956	54.5

Appendix

Table E1 : Per-dataset breakdown of the human preference study. We report the number of comparisons favoring PromptEvolver out of the total for each dataset.

Method	NLL $\downarrow$
PEZ	$6.88\pm 0.70$
VGD	$4.52\pm 0.51$
CLIP Interrogator	$4.05\pm 0.36$
STEPS	$7.10\pm 0.73$
VLM-Baseline	$\mathbf{2.80\pm 0.32}$
PromptEvolver-CLIP	$2.93\pm 0.30$

Appendix

Table F6 : Prompt naturalness measured by negative log-likelihood (NLL

\downarrow

) under Mistral-7B-v0.3. Lower values indicate more natural, fluent prompts. We report mean

\pm

std over all 500 evaluation images.

Prompt Optimization for T2I Generation

Prompt optimization for text-to-image generation improves how natural-language instructions control image content, style, composition, and quality.

See all 28 papers

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives
Jul 29, 2026Xiaolong Liu, Junjian Li, Yuan Xiao +4Diffusion Model Inversion Diffusion-Based Inverse Problems
Geely Hangzhou, China · City University of Macau Macau, China
PROVE: Training-Free Prompt Recovery using Verifiable Evidence
Aug 13, 2026Rupayan Mallick, Mahsa Khoshnoodi, Sarah Adel Bargal
Department of Computer Science, Georgetown University
Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation
Jun 7, 2026Xuanyi Liu, Deyi Ji, Junyu Lu +6T2I Generation Prompt Optimization for T2I Generation
Peking University · Tencent · Dalian University of Technology +4

Jul 29, 2026J/K move · Enter open · S save

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

Xiaolong Liu, Junjian Li, Yuan Xiao, Jiaqi Deng+3

Geely Hangzhou, China · City University of Macau Macau, China

Diffusion Model Inversion Diffusion-Based Inverse Problems Diffusion-Based Image Editing Controllable Image Generation T2I Diffusion Models

Open paper AskPDF arXiv

Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering. However, existing prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail alignment. We contend that the limitations stem from treating prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic prompt and latent noise of the target image. In the first stage, we integrate vision-language model, CLIP and large language model to invert a faithful, human-interpretable hard prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse datasets demonstrate that Dualin simultaneously generates high-quality inverted prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.

Abstract

Prompt Optimization for T2I Generation

Explore similar work

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

PROVE: Training-Free Prompt Recovery using Verifiable Evidence

Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation