EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

Qianhong Guo; Wei Xie; Xiaofang Cai; Shuoyoucheng Ma; Chengxin Hu; Baosheng Wang

cs.AIOct 14, 2026

EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

Authors: Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu, Baosheng Wang

Organizations: College of Computer Science and Technology, National University of Defense Technology · Institute of Information Engineering, Chinese Academy of Sciences

Abstract

Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8% in mean Outcome and 40.0% in mean Pass\textasciicircum{}3. These results support using fine-grained diagnostics to guide iterative task updates and probe agents' current capabilities and limitations.

Figures & tables

Figure 1: Motivation for EvoProbe. Static tasks face saturation and contamination risks, while coarse outcomes and judge dependence limit diagnostic insight. EvoProbe connects fine-grained diagnostic evaluation with feedback-driven task regeneration, moving beyond result reporting to iteratively probe agents’ current capability boundaries.

Figure 2: Overview of the EvoProbe framework. Generated tasks undergo validation and solver execution, followed by subtask outcome scoring and multi-judge trajectory evaluation. The resulting diagnostics are distilled into Diagnostic Feedback, which guides subsequent task generation, closing the benchmark evolution loop.

Model	Outcome	Pass@3	Pass^3
GPT-5.6-Terra	7.74	81.25	52.08
GPT-5.5	7.44	72.92	52.08
DeepSeek-V4-Flash-0731	6.36	62.50	37.50
GPT-5.6-Luna	6.10	54.17	27.08
Qwen-3.7-Plus	5.28	45.83	29.17
DeepSeek-V4-Pro-Preview	4.97	41.67	16.67

Table 1: Outcome-level evaluation results in R1. Outcome denotes the mean task score (0–10). Pass@3 (%, any run succeeds) and Pass^3 (%, all three runs succeed) report success rates. Models are sorted by Outcome. Best results in each column are bolded.

Model	Recon.	Vuln.	Exploit.	Flag
GPT-5.6-Terra	91.67	81.94	76.66	67.36
GPT-5.5	92.36	81.60	73.01	62.50
DeepSeek-V4-Flash-0731	91.67	72.92	58.62	49.31
GPT-5.6-Luna	86.81	77.43	54.29	40.97
Qwen-3.7-Plus	83.33	58.68	46.70	36.81
DeepSeek-V4-Pro-Preview	84.49	66.32	40.35	29.17

Table 2: Phase-level evaluation results in R1. Scores are normalized by the maximum score for each phase in the corresponding task and then averaged across runs and tasks (%). Models are sorted by Outcome. Best results in each column are bolded.

Model	Info.	Reason.	Tool Use	Path	Recovery	Trajectory Score
GPT-5.6-Terra	17.34	16.71	16.74	16.11	16.08	82.97
GPT-5.5	16.90	16.26	16.10	15.37	15.28	79.90
GPT-5.6-Luna	15.55	14.46	14.59	13.03	13.85	71.49
DeepSeek-V4-Flash-0731	14.55	13.59	13.98	12.47	13.32	67.92
DeepSeek-V4-Pro-Preview	14.41	13.28	13.57	11.52	12.21	65.00
Qwen-3.7-Plus	13.45	12.43	12.97	11.17	11.64	61.66

Table 3: Trajectory-level evaluation results in R1. Dimension scores (0–20) and Trajectory Score (0–100) are averaged across tasks. Models are sorted by Trajectory Score. Best results in each column are bolded.

Figure 3: Failure category distribution in R1. Percentages are computed over the 183 failed trajectories from the selected runs. Each trajectory is assigned one primary failure category through multi-judge voting. Other includes the remaining categories.

Category	R1 Diagnostic Feedback	R2 Task Requirements
Misc	Failure Pattern: Solvers relied on unvalidated protocol assumptions. Success Pattern: Solvers compared response fields and samples to derive and validate rules.	ParcelLoom: The solver must derive the target parcel’s label from public samples in a parcel-routing service, then exploit differences in how the frontend and backend interpret badge fields to escalate privileges.
Reverse	Failure Pattern: The solver split register setup across separate executions, overlooking that each execution reset the state.	Sigil Relay: The service reinitializes its state for each request. The solver must combine the necessary state setup and attestation operations within a single complete frame.
Crypto	Failure Pattern: The solver attempted to recover the private key by solving a discrete logarithm for the public key, overlooking the reuse of nonces.	The Glass Relay: The nonces used in two signatures differ by a disclosed constant. The solver must use this relationship to formulate equations and recover the private key.

Table 4: Representative correspondences between R1 Diagnostic Feedback and R2 task requirements. Each row pairs category-level feedback with key solution requirements summarized from the corresponding task reference.

Figure 4: Outcome comparison between R1-Top3 and R2. Circles denote R1-Top3 and diamonds denote R2, with lines connecting each model’s scores. Models are sorted by Outcome on R1-Top3.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Seed Tasks
Web	LockTalk; chunky; frog-waf
Reverse	Crushing; FlecksOfGold; LootStash
Pwn	Delulu; network-tools
Misc	Unbreakable; Were Pickle Phreaks Revenge; just-another-pickle-jail
Forensics	Data Siege; It Has Begun; eval-me
Crypto	failproof; noisier-crc; randsubware

Appendix

Table 5: Cybench tasks provided as few-shot examples for task generation.

	Mean Outcome $\downarrow$
Examiner	w/o Self	w/ Self
GPT-5.6-Terra	3.23	3.50
GPT-5.6-Luna	3.97	4.28
GPT-5.5	4.20	4.42
Claude-Sonnet-5	6.37	6.48
DeepSeek-V4-Pro-Preview	6.88	6.82

Appendix

Table 6: Mean Outcome on tasks generated by each Examiner in R1. w/o Self excludes the Examiner model’s own Solver results, while w/ Self includes them. Lower values indicate more challenging tasks for the evaluated Solvers. Selected Examiners are bolded.

	Outcome		Pass@3 (%)		Pass^3 (%)
Model	R1-Top3	R2	R1-Top3	R2	R1-Top3	R2
GPT-5.6-Terra	5.87	4.96	50.00	27.78	16.67	16.67
GPT-5.6-Luna	5.09	4.48	27.78	22.22	16.67	5.56
GPT-5.5	5.02	4.98	33.33	38.89	11.11	16.67
DeepSeek-V4-Flash-0731	4.33	4.11	38.89	22.22	11.11	11.11
DeepSeek-V4-Pro-Preview	3.28	2.70	11.11	11.11	5.56	0.00

Appendix

Table 7: Outcome-level comparison between R1-Top3 and R2, each containing 18 tasks. Outcome is scored on a 0–10 scale. Models are ordered by Outcome on R1-Top3.

Category	Attribution Criterion	Subtypes
Tool Execution Failure	Incorrect tool selection or execution despite a reasonable strategy.	tool misuse; parameter error; parsing failure; command error
Reasoning Error	Faulty reasoning or interpretation despite possessing the necessary knowledge.	logical leap; wrong hypothesis; hallucination; evidence misinterpretation; causal reasoning failure
Planning Strategy Error	Ineffective strategy or exploration plan, even when individual reasoning steps are sound.	random exploration; wrong path selection; premature exploitation; insufficient exploration
Information Management Failure	Failure to retain, integrate, or use information already obtained.	context forgetting; information synthesis failure; clue missing
Knowledge Limitation	Missing cybersecurity or vulnerability knowledge needed to identify or understand the solution.	vulnerability identification failed; vulnerability type misclassification; missing vulnerability knowledge
Exploitation Failure	Failure during exploitation or payload execution after correctly identifying the target.	payload failure; authentication bypass failed; privilege escalation failed; flag retrieval failed; exploit chain failure

Appendix

Table 8: Failure categories, attribution criteria, and subtypes used in trajectory evaluation. Underscores in taxonomy identifiers are replaced with spaces for readability.

Difficulty Signals
Phase Reach Rate	Reconnaissance: 96.88%; Vulnerability Analysis: 64.06%; Exploitation: 50.00%; Flag Capture: 48.44%.
Conditional Phase Completion	Reconnaissance: 99.46%; Vulnerability Analysis: 100.00%; Exploitation: 98.44%; Flag Capture: 100.00%.
Primary Bottleneck	Transition into Vulnerability Analysis. Transition bottlenecks: 18; within-phase progression bottlenecks: 0; total: 18.
Success Pattern (Selected)
Vulnerability Analysis	Correlate server-provided fields, repeated samples, and diagnostic clues to derive and validate deterministic encodings, checksums, or token formulas.
Failure Patterns (Selected)

Appendix

Table 9: Example Diagnostic Feedback for Misc. All difficulty signals are shown, together with one success pattern and the two most frequent failure patterns. Failure frequencies are computed over the 33 selected failed trajectories in this category.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
Jul 6, 2026Xingze Gao, Chuanrui Hu, Hongda Chen +9LLM Agent Self-Improvement AI Agent Benchmarks
1Anhui University · 2EverMind, Shanda Group · 3Southeast University
EvalBuilder: Can LLMs Build Efficient Benchmarks?
Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark Construction AI Agent Benchmarks
Meituan · Pennsylvania State University · Allen Institute for AI (AI2) +2
Evo-Bench: Can Language Models Improve Agent Harness?
Aug 10, 2026Lisheng Huang, Chen Yang, Hao Zhou +6AI Agent Evaluation Long-Horizon Agent Evaluation
1Gaoling School of Artificial Intelligence, Renmin University of China · 2BOSS Zhipin, Beijing, China

Jul 6, 2026J/K move · Enter open · S save

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Xingze Gao, Chuanrui Hu, Hongda Chen, Pengfei Yao+8

1Anhui University · 2EverMind, Shanda Group · 3Southeast University

LLM Agent Self-Improvement AI Agent Benchmarks Agent Skill Learning Self-Evolving Agents

Open paper AskPDF arXiv

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

Abstract

LLM Agent Evaluation

Explore similar work

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Evo-Bench: Can Language Models Improve Agent Harness?