Task-Efficient Agent Evaluation Using Cached Traces
Organizations: Jataware · Helivan
Abstract
Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family- and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon ) and from 0.14 to 0.12 (Wilcoxon ), respectively, when provided a single random executed task. Further, given access to the full set of reference systems, our approach dominates baselines for any given number of executed tasks on both benchmarks.
Figures & tables
| SWE-bench Verified ( ) | Terminal-Bench 2.0 ( ) | |||||||||||
| random | adaptive | random | adaptive | |||||||||
| Num. tasks | 1 | 5 | 20 | 1 | 5 | 20 | 1 | 5 | 20 | 1 | 5 | 20 |
| population mean | 0.139 | 0.155 | ||||||||||
| sample score | 0.442 | 0.175 | 0.082 | 0.380 | 0.151 | 0.072 | 0.396 | 0.164 | 0.073 | 0.360 | 0.124 | 0.132 |
| IRT | 0.127 | 0.088 | 0.048 | 0.080 | 0.062 | 0.034 | 0.137 | 0.097 | 0.052 | 0.112 | 0.067 | 0.039 |
| raw trace | 0.099 | 0.093 | 0.089 | 0.096 | 0.083 | 0.081 | 0.135 | 0.126 | 0.120 | 0.128 | 0.125 | 0.113 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| encoder | representation | (i) task | (ii) correctness | (iii) system | (iv) family | (v) harness |
|---|---|---|---|---|---|---|
| text-embedding-3-small | raw trace | 1.00 | 0.82 | 0.89 | 0.97 | 0.96 |
| generic | 0.99 | 0.84 | 0.61 | 0.64 | 0.76 | |
| trubric | 0.99 | 0.84 | 0.20 | 0.36 | 0.29 | |
| nomic-embed-text-v1.5 | raw trace | 0.98 | 0.78 | 0.84 | 0.90 | 0.91 |
| generic | 0.99 | 0.86 | 0.56 | 0.63 | 0.70 | |
| trubric | 0.99 | 0.84 | 0.16 | 0.30 | 0.22 |
| system | family | |
|---|---|---|
| 20240402_sweagent_gpt4 | gpt | 0.224 |
| 20240612_MASAI_gpt4o | gpt | 0.326 |
| 20240620_sweagent_claude3.5sonnet | claude | 0.336 |
| 20240721_amazon-q-developer-agent-20240719-dev | – | 0.388 |
| 20240820_epam-ai-run-gpt-4o | gpt | 0.240 |
| 20240820_honeycomb | – | 0.406 |
| system | harness | model tags | |
|---|---|---|---|
| Ante__Gemini-3-Pro-Preview | ante | gemini-3-pro-preview | 0.647 |
| Ante__Gemini-3.1-Pro-Preview | ante | gemini-3-pro-preview | 0.742 |
| BashAgent__TermiGen-32B | bash-agent | termigen-32b | 0.197 |
| Capy__GPT-5.5 | capy | gpt-5-5 | 0.834 |
| ClaudeCode__GLM-4.7 | claude-code | glm-4-7 | 0.333 |
| CodeBrain-1.5__GPT-5.3-Codex | codebrain-1 | gpt-5-3-codex | 0.816 |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 555 papers