DUET: Co-Evolving Solver and Grader Agents

Fengyu Gao; Sourav Pal; Austin Z. Henley; Arjun Radhakrishna; Gustavo Soares

cs.LGOct 2, 2026

DUET: Co-Evolving Solver and Grader Agents

Authors: Fengyu Gao, Sourav Pal, Austin Z. Henley, Arjun Radhakrishna, Gustavo Soares

Organizations: University of Virginia, Charlottesville, United States · Microsoft, Redmond, United States

Abstract

Agentic workflows are increasingly used across domains such as technology, finance, and enterprise operations. As these agents become more widely deployed, continually improving them becomes increasingly important. This raises an immediate challenge: How should the agent evolve? This evolution requires effective evaluation that can assess outcomes and provide useful feedback for optimization. As the agent evolves, its behaviors and failure modes may also change, making a fixed evaluator increasingly inadequate. Another fundamental question: How should we evaluate an evolving agent? These two challenges are inherently coupled; changes in agent behavior can expose limitations of the current evaluator, while a stronger evaluator provides more informative feedback for improving the agent. Motivated by this interaction, we introduce DUET, a framework that jointly optimizes a solver agent and a grader agent to improve both. DUET iteratively selects training tasks, executes them with the solver, evaluates the resulting outcomes with the grader, and uses a tool-using update module to revise the solver and the grader, alternating between the two across rounds. By updating the grader within the optimization loop, DUET turns evaluation from a fixed source of feedback into a first-class optimization objective that adapts alongside the solver. Experiments across four agent benchmarks show that DUET improves both solver and grader performance and consistently outperforms baselines that optimize the solver with a fixed grader.

Figures & tables

Figure 1: Overview of DUET. At each round, DUET first selects a training batch adaptively that balances exploitation and exploration. The solver then executes the selected tasks, and the grader evaluates the resulting trajectories and outputs to produce scores and critiques. Finally, the update module alternates between revising the grader and the solver, keeping the other fixed in each round.

Figure 2: Representative solver and grader prompt updates produced by DUET during optimization. The solver develops reusable practices for root-cause diagnosis, completeness checking, and efficient tool use, while the grader develops practices for correctness verification, evidence assessment, and score calibration.

Panel of Figure 3 — Figure 3: DUET ablations on Finance Agent Benchmark.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Batch Size	$\tau$	Max. Rounds
SpreadsheetBench	8	20	50
GDPval	4	8	40
$\tau^{2}$ -Bench	8	10	30
Finance Agent Benchmark	8	10	30

Appendix

Table 6: Training hyperparameters used for DUET across benchmarks.

Solver Performance
Method	$\tau_{2}$ -Bench	SpreadsheetBench
$A_{0}$	83.3	62.2
DUET	89.6	62.9
Grader Performance
Method	$\tau_{2}$ -Bench	SpreadsheetBench
$G_{0}$	74.1	45.0

Appendix

Table 7: Solver and grader performance on the test sets using GPT-5.6 Sol, with system prompts as the optimization target. Solver metrics are success rate for

\tau_{2}

-Bench, and task accuracy for SpreadsheetBench. Grader performance is measured by pairwise agreement.

A_{0}

and

G_{0}

denote empty-prompt initialization.

Setting	Solver Performance	Grader Agreement
Validation-selected	$52.2\pm 1.8$	$78.5\pm 6.1$
Fixed-round	$53.1\pm 1.4$	$74.0\pm 2.2$

Appendix

Table 11: Checkpoint selection ablation on GDPval. The fixed-round setting runs for 50 rounds and returns the final checkpoints without validation-based early stopping or checkpoint selection.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 277 papers

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Jun 24, 2026Alex Iacob, Andrej Jovanović, William F. Shen +12Evolutionary Optimization AI Agent Evaluation
University of Cambridge · Flower Labs · NVIDIA +2
VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
Oct 21, 2026Yueshen Li, Kamer Ali Yuksel, Gokhan Tur +2Self-Evolving Agents Tool-Use Evaluation
University of Illinois Urbana-Champaign · aiXplain
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Oct 8, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Automated Evaluation LLM Agent Self-Improvement
AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China

Jun 24, 2026J/K move · Enter open · S save

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Alex Iacob, Andrej Jovanović, William F. Shen, Daniel Burkhardt+11

University of Cambridge · Flower Labs · NVIDIA+2

Evolutionary Optimization AI Agent Evaluation Self-Improving Agents Automated Peer Review Recursive Self-Improvement

Open paper AskPDF arXiv

Self-improving agents are state-of-the-art on agentic coding benchmarks, yet their search methods assume a stationary evaluation criterion. This ignores a central feature of evolution: species adapt as their environments change with them. We introduce the Red Queen Gödel Machine (RQGM), an evolutionary framework for recursive self-improvement under non-stationary utilities. This allows learned evaluators to improve alongside the agents they guide. On DeepSWE, the RQGM improves over its fixed-evaluator baseline by adding a complementary agent-as-a-judge code-review signal: a co-evolved reviewer grades coder patches to guide search. At low reasoning effort, the RQGM coder passes 82.1% of held-out tasks against the baseline's 75.0%, and nearly matches the GPT-6 Astra model at high effort. In scientific paper writing and reviewing, and Olympiad-level proof writing and grading, co-evolved evaluators provide an evaluation criterion. Anchored to human IMO grades, a co-evolved grader writes its own milestone rubric and exceeds static baselines at a 3x lower search cost, driving the prover to the best mean score. Since the RQGM can modify the search objective across epochs, it can regularize the search. For example, the RQGM reduces self-preference bias via an additional adversarial objective to discover reviewers equally stringent on AI and human work. Guided by these calibrated reviewers, co-evolved writers reach 1.78x-1.86x higher acceptance rates than the baseline under an agent-as-a-judge panel. The RQGM enables self-improving systems where agents and evaluators recursively bootstrap each other beyond static evaluation.

Abstract

LLM Agent Self-Improvement

Explore similar work

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

VeriEvolve: Evaluator-First Evolution for Tool-Using Agents

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents