Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization

Shiyan Liu; Qifeng Xia; Qiyun Xia; Yisheng Liu; Xinyu Yu; Rui Qu

cs.AIMar 19, 2026

Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization

Authors: Shiyan Liu, Qifeng Xia, Qiyun Xia, Yisheng Liu, Xinyu Yu, Rui Qu

Organizations: University of California, Berkeley · Huazhong University of Science and Technology · Hefei University of Technology

Abstract

Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025.

Figures & tables

Panel of Figure 1 — Figure 1: A representative failure of GEPA under the defective seed.

Figure 2: Conceptual illustration of optimization trajectories under the defective seed.

Panel of Figure 3 — Figure 3: Four systematic limitations of reflective APO. L1–L3 form a causal chain, and L4 can apply even when optimization succeeds.

Panel of Figure 4 — Figure 4: An illustrative case of attribution blindspot.

Panel of Figure 5 — Figure 5: GEPA attribution distribution on GSM8K (Qwen3-4B base, Qwen3-8B/GPT-4o-mini reflectors).

Panel of Figure 6 — Figure 6: An illustrative case of trajectory opacity.

Panel of Figure 7 — Figure 7: An illustrative case of transfer fragility.

Figure 8: VISTA pipeline overview: a multi-agent framework that decouples hypothesis generation from prompt rewriting via heuristic-guided parallel verification and interpretable semantic trace trees.

	GSM8K			AIME2025
Method	Defective	Repaired	Minimal	Defective	Repaired	Minimal
No Opt.	23.81	85.59	20.67	38.67	40.00	40.00
GEPA	13.50	86.53	21.68	44.00	39.33	42.00
VISTA	87.57	87.34	85.67	46.00	46.67	44.00

Table 1: Accuracy (%) across benchmarks, conditions, and methods.

Panel of Figure 9 — Figure 9: Optimization curves on GSM8K under the defective seed (best score to date).

	Reflector
Method	Qwen3-8B	GPT-4o-mini	Cross-Model
GEPA	13.50	23.43	22.74
VISTA	87.57	87.64	86.05

Table 2: GSM8K accuracy (%) on the defective seed. Columns 1–2: Qwen3-4B base, Qwen3-8B/GPT-4o-mini reflectors. Column 3: trained on GPT-4.1-mini (GPT-4o-mini reflector), evaluated on Qwen3-4B.

Effect of $K$
	Hyper.
Configuration	$K$	$p$	$\varepsilon$	Acc.
VISTA	3	0.2	0.1	87.57
VISTA, K=1	1	0	0.1	75.97
VISTA, K=3	3	0	0.1	86.81
VISTA, K=5	5	0	0.1	83.89

Table 3: Ablation results on GSM8K (%) under the defective seed (Qwen3-4B base, Qwen3-8B reflector).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: GEPA optimization tree under the defective seed. Every edge carries only a question mark; no root-cause label or accuracy gain $\delta$ is recorded for any transition. The structural root cause is never identified and optimization stagnates at 38%.

Panel of Figure 11 — Figure 11: VISTA optimization tree under the defective seed. Every edge indicates a hypothesis annotated with a root-cause label and $\delta$ . Iteration 1 immediately identifies cot_field_ordering as the structural root cause, achieving a $+48$ pp jump to 78% accuracy; iteration 3 further reaches 86% via the same diagnosis.

Panel of Figure 12 — Figure 12: GEPA optimization tree under the repaired seed. The trace follows a single-candidate mutation path; two independent branches successfully reach 88% accuracy, while one subsequent update on the left branch regresses to 86%.

Panel of Figure 13 — Figure 13: VISTA optimization tree under the repaired seed. The run branches early through parallel hypotheses in iterations 3, 4, and 9, reaching 88% peak accuracy via multiple distinct, semantically tagged paths. Unlabeled edges indicate iterations where no failure cases were collected on the minibatch of its parent, falling back to a single-mutation step as in GEPA.

Panel of Figure 14 — Figure 14: GEPA optimization tree under the minimal seed. The main chain reaches a peak of 32% at iteration 13 before regressing to 26% in the next update.

Panel of Figure 15 — Figure 15: VISTA optimization tree under the minimal seed. Iteration 2 immediately identifies cot_field_ordering as a high-impact root cause ( $+50$ pp), then improves further through semantically tagged branches.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

Reflective Prompt Tuning through Language Model Function-Calling
May 20, 2026Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour +1LLM Self-Refinement LLM Prompting
Megagon Labs
MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization
Oct 18, 2025Juhyeon Lee, Wonduk Seo, Junseo Koh +6Automatic Prompt Optimization Multi-Agent LLMs
Peking University · Enhans · Fudan University +1
Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization
Oct 13, 2026Chinmoy Mitra, Tarek MahmudAutomatic Prompt Optimization Language Model Generation Evaluation
Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Texas A&M University–Kingsville, Kingsville, Texas, USA

May 20, 2026J/K move · Enter open · S save

Reflective Prompt Tuning through Language Model Function-Calling

Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour, Estevam Hruschka

Megagon Labs

LLM Self-Refinement LLM Prompting Automatic Prompt Optimization

Open paper AskPDF arXiv

Large language models (LLMs) have become increasingly capable of following instructions and complex reasoning, making prompting a flexible interface for adapting models without parameter updates. Yet prompt design remains labor-intensive and highly sensitive to formatting, phrasing, and instruction order, motivating automated prompt optimization methods that reduce manual effort while preserving inference-time flexibility. However, existing methods often search over prompt candidates or use fixed critique-refine pipelines driven by individual examples or small batches, limiting their ability to capture systematic error patterns and make targeted edits grounded in failure history. We propose Reflective Prompt Tuning (RPT), a framework that uses LLM function calling to simulate the iterative workflow of human prompt engineers. An LLM optimizer calls a diagnostic function that evaluates the target model over an entire optimization set, summarizes recurring failure modes, and returns a structured diagnostic report. The optimizer uses this report, together with an accumulated memory of prior reports, to revise the prompt for the next iteration. RPT further supports confidence-aware optimization by using calibration signals in diagnostic feedback and final prompt selection. Across three reasoning tasks, RPT improves over initial prompts by up to 12.9 points, remains competitive with state of the art, and improves confidence calibration. Our analyses show that RPT is especially effective on multi-hop and mathematical reasoning, producing targeted prompt revisions that align with diagnosed failure patterns and lead to gains in task performance and calibration.

Abstract

Automatic Prompt Optimization

Explore similar work

Reflective Prompt Tuning through Language Model Function-Calling

MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization

Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization