LLM-Guided Prompt Evolution for Password Guessing

Vladimir A. Mazin; Mikhail A. Zorin; Dmitrii S. Korzh; Elvir Z. Karimov; Dmitrii A. Bolokhov; Oleg Y. Rogov

cs.CRApr 14, 2026

LLM-Guided Prompt Evolution for Password Guessing

Authors: Vladimir A. Mazin, Mikhail A. Zorin, Dmitrii S. Korzh, Elvir Z. Karimov, Dmitrii A. Bolokhov, Oleg Y. Rogov

Organizations: AXXX · MTUCI · Applied AI Center

Abstract

Passwords still remain a dominant authentication method, yet their security is routinely subverted by predictable user choices and large-scale credential leaks. Automated password guessing is a key tool for stress-testing password policies and modeling attacker behavior. This paper applies LLM-driven evolutionary computation to automatically optimize prompts for the LLM password guessing framework. Using OpenEvolve, an open-source system combining MAP-Elites quality-diversity search with an island population model we evolve prompts that maximize cracking rate on a RockYou-derived test set. We evaluate three configurations: a local setup with Qwen3 8B, a single compact cloud model Gemini-2.5 Flash, and a two-model ensemble of frontier LLMs. The approach raises the cracking rates from 2.02% to 8.48%. Character distribution analysis further confirms how evolved prompts produce statistically more realistic passwords. Automated prompt evolution is a low-barrier yet effective way to strengthen LLM-based password auditing and underlining how attack pipelines show tendency via automated improvements.

Figures & tables

Figure 1: The proposed OpenEvolve-based pipeline. An initial prompt is fed to the evolutionary controller, which iteratively samples parent prompts from the MAP-Elites archive, requests improved variants from the LLM ensemble, runs LLM and updates the archive based on the cracked-rate fitness signal.

Panel of Figure 2 — Figure 2: MAP-Elites archive and island model. Three islands each maintain an independent $\texttt{prompt\_length}\times\texttt{complexity}$ grid. Filled green cells indicate occupied elites. Ring-topology migration occurs at fixed intervals.

Parameter	Exp. 1 (Ollama)	Exp. 2 (Gemini)	Exp. 3 (Ensemble)
LLM (generation)	qwen3:8b	gemini-2.5-flash-lite	qwen3-235b (0.5), gemini-3-flash (0.5)
Evaluator LLM	qwen3:8b	qwen3-max	qwen3-max
Temperature	0.4	0.4	0.4
Max tokens	16 000	16 000	16 000
Population size	100	1000	1000
Archive size	20	100	100

Table 1: OpenEvolve hyperparameter settings. CR: best cracked rate achieved.

Run	$n$	Mean CR	SD	Min CR	Best CR	$\boldsymbol{\Delta}$ CR
Baseline	–	2.02%	–	–	2.02%	–
Exp. 1 (Ollama)	100	4.13%	3.51%	0.00%	8.28%	$+$ 4.1 $\times$
Exp. 2 (Gemini)	100	3.79%	3.15%	0.00%	7.58%	$+$ 3.75 $\times$
Exp. 3 (Ensemble)	100	6.89%	1.48%	1.07%	8.48%	$+$ 4.2 $\times$

Table 2: Descriptive statistics for each experimental run (excluding iteration 0).

n

: number of evaluated prompts; Mean, SD: mean and standard deviation of per-iteration cracked rate; Best CR: peak cracked rate;

\Delta

CR: improvement over baseline.

Panel of Figure 3 — Figure 3: Per-symbol F-score curves for the baseline and best evolved prompts. Higher AUC indicates better alignment of the generated password’s character-frequency distribution with that of real passwords. The ensemble prompt achieves the best AUC (0.074), while the baseline (0.059) has a flat profile indicating poor character-level coverage.

Configuration	Peak F-score	Opt. $\tau$	AUC-F
Baseline	0.1229	0.15	0.0589
Exp. 2 (Gemini)	0.1794	0.10	0.0654
Exp. 3 (Ensemble)	0.1912	0.10	0.0745
Relative gain (Baseline $\to$ Ensemble):			$+$ 26.3%

Table 3: Per-symbol F-score performance. Peak values of the F-score, optimal threshold and AUC.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Per-iteration cracked rate (first 45 iterations). Baseline: PassLLM with the original prompt (2.02%). All three evolved runs rapidly exceed the baseline; the ensemble first attains 8.48% at iteration 45.

Panel of Figure 5 — Figure 5: Full 100-iteration ensemble run. Teal: per-iteration cracked rate; red: running archive best (best CR seen up to each iteration). Migrations occurred at iterations 10, 34, 64, and 94; the migration at iteration 34 is associated with the step from 8.27% to 8.48%.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

PassGPT+: Leveraging Linguistic Priors for Password Modeling
Sep 30, 2026Rajneesh Anand, Neeraj Lakshmanan, Masoud YariAutoregressive Language Modeling Diffusion Language Models
Department of Chemical and Biomolecular Engineering, Lehigh University, Bethlehem, PA, 18015, USA · Department of Computer Science & Design, Singapore University of Technology & Design, Singapore · Department of Computer Science & Engineering, Lehigh University, Bethlehem, PA, 18015, USA
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
May 9, 2026Ismail Hossain, Tanzim Ahad, Md Jahangir Alam +3Adversarial Prompt Generation Language Model Safety Evaluation
Department of Computer Science, University of Texas at El Paso, TX, USA 79902 · School of Computing, Southern Illinois University Carbondale, IL, USA 62901 · University of Illinois Urbana-Champaign, IL, USA
SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization
May 8, 2026Houjun Liu, Lisa Einstein, John Yang +5LLM Auditing Prompt Optimization
Stanford University

Sep 30, 2026J/K move · Enter open · S save

PassGPT+: Leveraging Linguistic Priors for Password Modeling

Rajneesh Anand, Neeraj Lakshmanan, Masoud Yari

Department of Chemical and Biomolecular Engineering, Lehigh University, Bethlehem, PA, 18015, USA · Department of Computer Science & Design, Singapore University of Technology & Design, Singapore · Department of Computer Science & Engineering, Lehigh University, Bethlehem, PA, 18015, USA

Autoregressive Language Modeling Diffusion Language Models Text Generation

Open paper AskPDF arXiv

Passwords remain the dominant online authentication mechanism, and understanding how humans choose them is essential for defensive strength estimation and attack simulation alike. Recent learning-based approaches such as PassGAN and PassGPT have shown that deep generative models can learn password structure directly from leaked corpora. However, both train from random initialization on password data alone. The role of linguistic prior knowledge in password modeling, and what it reveals about how humans create secrets, remains largely underexplored. Here, we address this gap with PassGPT+, which adapts the linguistic prior of GPT-2 to password observations through character-aware tokenization. We also introduce PassDiffusion, the first absorbing-state discrete diffusion model for password generation, as a probe of whether non-autoregressive approaches are competitive. On the RockYou benchmark, PassGPT+ recovers 22.53% of held-out passwords at 108 guesses, a 16% relative gain over PassGPT, and retains 79% of this match rate when transferred without retraining to a disjoint 2020 leak dataset, demonstrating that linguistic priors capture persistent regularities of human password generation. PassDiffusion underperforms by two to three orders of magnitude, indicating that autoregressive modeling is substantially better matched than iterative denoising to the discrete, exact-match nature of password generation.

Abstract

Automatic Prompt Optimization

Explore similar work

PassGPT+: Leveraging Linguistic Priors for Password Modeling

The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization