Coding Agents are Strong Prompt Optimizers

Agamdeep Singh; Srishti Gautam; Priyanshu Gupta; Nikita Mehrotra; Tanmay Bakshi; Sumit Gulwani

cs.AIAug 13, 2026

Coding Agents are Strong Prompt Optimizers

Authors: Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, Sumit Gulwani

Organizations: Microsoft

Abstract

Search-based prompt optimizers improve prompts through iterative search: they propose edits, execute fresh rollouts, score the resulting trajectories, and retain only edits that improve a validation metric. We show that this optimization loop is unnecessary. Given only a static corpus of agent trajectories, an off-the-shelf coding agent can directly synthesize an optimized prompt, requiring neither environment access nor validation data. We call this approach Coding-Agent Skill Distillation (CASD). The key insight is reflection scope. Rather than reasoning over a small batch of trajectories at each optimization step, the coding agent writes and executes analysis code to compute corpus-wide statistics, identifies systematic failure modes, inspects representative episodes, and distills the resulting insights into behavioral rules. Across four agentic benchmarks (ALFWorld, $τ^2$ -bench retail and telecom, and SpreadsheetBench-Verified), under matched data access, a single CASD pass outperforms GEPA, a state-of-the-art reflective prompt optimizer, on three of four benchmarks and outperforms validation-gated reflective search (SkillOpt) on all four, improving the unoptimized baseline by 16.6 percentage points on average versus 10.9 for GEPA and 5.3 for SkillOpt. Because CASD performs a single offline analysis pass rather than iterative search, producing an optimized prompt costs approximately $1.60---over $22\times$ cheaper than validation-gated search. Even when competing methods are granted additional validation data and unrestricted environment access, CASD remains ahead on two of four benchmarks. These results suggest that corpus-scale statistical reflection is a viable alternative to iterative search for prompt optimization.

Figures & tables

Figure 1: A coding agent as a prompt optimizer. Left: Search-based optimizers (GEPA, SkillOpt) iteratively refine prompts using environment rollouts and validation feedback. In contrast, CASD performs a single offline pass over a frozen rollout corpus to directly produce the optimized skill file, without iterative search or additional environment interaction. Right: Across four agentic benchmarks under matched data access, CASD achieves the largest average improvement while requiring substantially lower optimization cost.

Figure 2: Inside the distiller. (a) Tool traces across 24 distillation runs show a characteristic workflow: exploration is front-loaded, skill writing is terminal, and the intermediate steps alternate between corpus-statistics analysis and inspection of flagged episodes. (b) Corpus statistics most frequently computed by the executed analysis code. (c) The resulting prompts consistently cite quantitative evidence (fractions, percentages, and measured counts), reflecting the corpus-driven reasoning process.

Benchmark	Baseline	CASD (ours)	SkillOpt	GEPA
ALFWorld	56.7 $\pm$ 1.2	83.3 $\pm$ 2.3	68.0 $\pm$ 2.0 ‡	74.0 $\pm$ 2.0
$\tau^{2}$ retail	32.5 $\pm$ 2.5	40.0 $\pm$ 2.5	38.3 $\pm$ 10.1	39.2 $\pm$ 7.2
$\tau^{2}$ telecom	19.2 $\pm$ 5.2	39.2 $\pm$ 8.0	25.8 $\pm$ 3.8	17.5 $\pm$ 4.3
SSB-Verified	39.3 $\pm$ 3.1	51.3 $\pm$ 3.1	38.7 $\pm$ 2.3	60.7 $\pm$ 3.1
Mean gain vs. base	—	+16.6	+5.3	+10.9

Table 1: Limited-data regime (headline): held-out test accuracy (%), mean over 3 seeds (SD), when all optimizers see only the static rollout pool. ‡ ALFWorld SkillOpt uses its native scaffold (own baseline 58.7%); not directly comparable to column 2.

Benchmark	Baseline	CASD (ours)	SkillOpt	GEPA
ALFWorld	56.7 $\pm$ 1.2	83.3 $\pm$ 2.3	68.0 $\pm$ 2.0 ‡	74.0 $\pm$ 2.0
$\tau^{2}$ retail	32.5 $\pm$ 2.5	40.0 $\pm$ 2.5	38.3 $\pm$ 10.1	46.7 $\pm$ 8.0
$\tau^{2}$ telecom	19.2 $\pm$ 5.2	39.2 $\pm$ 8.0	44.2 $\pm$ 5.2	15.8 $\pm$ 5.2
SSB-Verified	39.3 $\pm$ 3.1	51.3 $\pm$ 3.1	48.0 $\pm$ 2.0	56.7 $\pm$ 1.2

Table 2: Head-to-head regime: GEPA and SkillOpt additionally get a separate held-out validation set and environment access; CASD is unchanged (it uses neither). ALFWorld cells repeat the limited-data runs (no separate head-to-head run exists).

Figure 3: Limited-data regime: held-out test accuracy over 3 seeds (bars: mean; whiskers: SD). A single CASD pass over the static pool beats GEPA on 3/4 benchmarks and SkillOpt on 4/4.

Figure 4: Corpus-level absences are invisible to minibatch reflection. A

\tau^{2}

-telecom “no service” ticket requires repairing two independent root causes. Columns show the prompts distilled by the three optimizers from the same 50-rollout pool and the resulting agent behavior. The device issue is evident in individual trajectories and is captured by all methods. In contrast, the billing issue appears only as a corpus-level absence— make_payment is never invoked in the rollout pool. Only CASD identifies this missing behavior from corpus-level statistics and distills the required billing rule, whereas GEPA and SkillOpt omit it and fail the task.

	ALFWorld	Retail	Telecom	SSB	Total
CASD (ours)	1.6	1.6	1.56	1.6	6.4
GEPA	2.1	1.1	4.5	0.74	8.4
SkillOpt	15.9	13.9	74.7	38.0	142.5

Table 3: Optimizer cost to produce each prompt (USD; with token-caching on). GEPA/SkillOpt telecom and SSB(SkillOpt) costs are from the head-to-head runs. CASD needs no environment rollouts at all.

Benchmark	no-think	+CASD	think	Rec.	tok ( $\downarrow$ )
ALFWorld	56.7	83.3	71.3 $\pm$ 1.2	$>$ 100%	3.7k $\to$ 0.8k
$\tau^{2}$ retail	32.5	40.0	35.0 $\pm$ 6.6	$>$ 100%	1.6k $\to$ 0.6k
$\tau^{2}$ telecom	19.2	39.2	45.0 $\pm$ 2.5	78%	2.1k $\to$ 0.6k
SSB-Verified	39.3	51.3	61.3 $\pm$ 1.2	55%	3.3k $\to$ 0.8k

Table 4: Recovering the reasoning gap without reasoning tokens (GPT-5.4-mini). “Rec.” = fraction of the no-think

\to

think accuracy gap recovered by the skill; tok = mean output tokens per episode (think

\to

skill).

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 99 papers

Aug 7, 2026

The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows

Recent systems for optimizing prompts, programs, and ML workflows typically rely on explicit outer-loop controllers such as evolutionary search, bandits, or textual-gradient methods. We ask a fundamentally different question: how much of this search policy can be internalized by a single tool-using agent? We present ReASearch, a unified framework for reasoning-driven optimization in which the agent autonomously decides what to evaluate, how to diagnose failures, which edits to make, and when to verify or restart. Rather than serving only as a proposal generator guided by hand-designed heuristics, the agent actively analyzes outcomes, allocates budget, and refines its strategy over long horizons through persistent memory. With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML workflows. Across 14 diverse tasks, it is competitive with and mostly better than specialized optimization systems, achieving gains of 2% to 40% over strong domain-specific baselines, and in some cases discovering solutions that improve on prior human best-known results. Crucially, we observe that complex search behaviors, which are typically implemented by explicit controllers, emerge naturally from the agent's reasoning process.

Junbo Li, Boyi Liu, Canwen Xu +5

The University of Texas at Austin · Snowflake

Large Language Model-Guided Optimization LLM Agent Workflow Optimization Automatic Prompt Optimization Automated Machine Learning

Ask PDF

Aug 27, 2026

Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search

Efficiently improving autonomous agents across diverse tasks is central to accelerating recursive self-improvement (RSI) in agentic AI, with prompt optimization emerging as a promising approach capable of delivering performance gains comparable to those achieved by fine-tuning model weights, while reducing computational costs in both optimization and serving. However, recent developments increasingly favor unnecessarily complex prompt optimizers. We introduce Naive Prompt Optimization (NPO), a lightweight single-lineage method that iteratively revises prompts using a teacher model with rollout feedback. NPO achieves comparable or better performance than GEPA with fewer rollouts, and its advantage increases with stronger teacher models, suggesting that stronger teacher reasoning can partially substitute for optimizer-side search complexity. In interactive games, NPO remains broadly competitive with GEPA, while GRPO performs better on some tasks less amenable to prompt optimization. We also show that NPO-optimized prompts elicit similar performance improvements when applied verbatim to other student models, especially across models within the same family. Overall, our preliminary results show that simple, linear prompt optimization can rival substantially more sophisticated and complex search procedures.

Yuan Chang, Xiaoqi Chen

Purdue University

Automatic Prompt Optimization Prompt Optimization Large Language Model-Guided Optimization LLM Agent Self-Improvement

Ask PDF

Aug 2, 2026

Same Task, Different Work: Prompt-Induced Waste in Coding Agents

Two prompts can request the same code change and produce the same correct patch, yet cause a coding agent to perform radically different kinds and amounts of work. We study this effect in a preregistered benchmark spanning 4,644 valid runs, 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses. The central finding is that prompt wording does not merely scale total effort; it changes where that effort is spent. Multiple approaches and deep thinking primarily inflate reasoning. Multiple approaches increases reasoning by 2.4x to 7.4x across all six open models and creates about three elaborated but discarded solution branches, while still yielding only one implemented solution and no success gain. Maximum certainty activates a different pathway: repeated verification propagates into extra test runs, tool calls, turns, latency, and context growth. Runs with high redundant verification cost 18x the clean-run median, execute 2.5x more tool calls, and take 3x longer, again without a success gradient. These mechanisms therefore have distinct cost carriers: some prompts are reasoning-heavy and token-borne, while others are tool-heavy and system-borne. Harness design amplifies both effects and changes cost per successful task by 5x to 30x in our setting. The findings survive a frozen holdout, paraphrase tests, a Kimi-K3 replication, and a first-party Claude Sonnet 5 study. In contrast, bounded-efficiency wording preserves diagnosis and final validation while avoiding the measured waste mechanisms. Prompt engineering for coding agents is therefore work design: it determines what the agent thinks through, what it executes, and when it stops.

Sarel Weinberger, Amir Hozez

PointFive

LLM Inference Efficiency Overthinking in Language Models AI Coding Agents AI Agent Benchmarks

Ask PDF

Abstract

Automatic Prompt Optimization

Explore similar work

The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows

Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search

Same Task, Different Work: Prompt-Induced Waste in Coding Agents