Agent psychometrics: Task-level performance prediction in agentic coding benchmarks

Chris Ge; Daria Kryvosheieva; Daniel Fried; Uzay Girit; Kaivalya Hariharan

cs.AIApr 1, 2026

Agent psychometrics: Task-level performance prediction in agentic coding benchmarks

Authors: Chris Ge, Daria Kryvosheieva, Daniel Fried, Uzay Girit, Kaivalya Hariharan

Organizations: MIT · Fulcrum · Carnegie Mellon University

Abstract

As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult. This is compounded by current practice: agent performance is typically measured by aggregate pass rates on benchmarks, but single-number metrics obscure the diversity of tasks within a benchmark. We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime. Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases, and introduces a novel decomposition of agent ability into LLM and scaffold ability components. This parameterization enables us to aggregate evaluation data across heterogeneous leaderboards and accurately predict task-level performance for unseen benchmarks, as well as unseen LLM-scaffold combinations. Our methods have practical utility for benchmark designers, who can better calibrate the difficulty of their new tasks without running computationally expensive agent evaluations.

Figures & tables

Figure 1: Agent and task features predicting success probability. On the agent side, the features are two categorical variables, the LLM and the scaffold; on the task side, they are vectors extracted from rich task artifacts. From these we estimate the agent’s ability and the task’s difficulty , then apply the logistic model from IRT ( Baker, 2001 ) to predict the probability that the agent succeeds on the task.

Setting	# Bench.	Method	Held-Out	Metric	Objective
New Tasks	Single	IRT with task features	Random subset of tasks (in-distribution)	Mean 5-fold CV AUC-ROC over tasks	Test whether task features explain task difficulty.
New Responses	Single & Multi	IRT with agent features	Random subset of responses (in-distribution)	Mean 5-fold CV AUC-ROC over responses	Test whether LLM and scaffold abilities combine additively.
New Agents	Single	IRT with agent features	Agents whose LLM and scaffold were individually seen	Mean 5-fold CV AUC-ROC over agents with seen LLMs and scaffolds	Test generalization to unseen model–scaffold combinations.
New Benchmarks	Multi	IRT with task & agent features	Entire benchmark (out-of-distribution)	Validation AUC-ROC on held-out benchmark	Test out-of-distribution generalization for early-stage benchmark design.

Table 1: Experimental settings. We evaluate response prediction under four settings.

Method	SWE-bench Verified	SWE-bench Pro	GSO	Terminal-Bench 2.0
Baseline	0.718 $\pm$ 0.001	0.656 $\pm$ 0.001	0.718 $\pm$ 0.007	0.732 $\pm$ 0.002
Embedding	0.824 $\pm$ 0.002	0.752 $\pm$ 0.003	0.756 $\pm$ 0.010	0.767 $\pm$ 0.010
LLM-as-a-judge	0.840 $\pm$ 0.001	0.743 $\pm$ 0.003	0.778 $\pm$ 0.015	0.794 $\pm$ 0.007
Combined	0.841 $\pm$ 0.003	0.763 $\pm$ 0.003	0.787 $\pm$ 0.014	0.792 $\pm$ 0.013
Oracle	0.945 $\pm$ 0.000	0.918 $\pm$ 0.000	0.916 $\pm$ 0.002	0.932 $\pm$ 0.001

Table 2: AUC-ROC (mean

\pm

std of 5-fold cross-validation means across 20 random seeds) on held-out tasks for each of the four benchmarks . All feature vector-based predictors significantly beat the Baseline , which always predicts the agent’s empirical success rate from the training data. Oracle is a standard IRT trained on all tasks, serving as an upper bound on performance.

Feature Source	SWE-bench Verified	SWE-bench Pro	GSO	Terminal-Bench 2.0
LLM-as-a-judge features
Problem statement only	0.786 $\pm$ 0.002	0.718 $\pm$ 0.002	0.710 $\pm$ 0.012	0.789 $\pm$ 0.007
Full w/o repository state	0.840 $\pm$ 0.001	0.732 $\pm$ 0.003	0.775 $\pm$ 0.017	0.797 $\pm$ 0.006
Full w/o tests	0.837 $\pm$ 0.001	0.735 $\pm$ 0.002	0.766 $\pm$ 0.011	0.798 $\pm$ 0.005
Full w/o solution	0.833 $\pm$ 0.002	0.748 $\pm$ 0.002	0.728 $\pm$ 0.010	0.797 $\pm$ 0.005
Full	0.845 $\pm$ 0.002	0.750 $\pm$ 0.003	0.774 $\pm$ 0.016	0.799 $\pm$ 0.006

Table 3: Feature source ablation: AUC-ROC (mean

\pm

std of 5-fold cross-validation means across 20 random seeds) of predictors with some feature sources removed. For LLM-as-a-judge features, we remove each of the three agentic feature sources one at a time. For embedding features, we compare a problem-statement-only predictor (the method of Truong et al. (2025) applied without modification) and one that additionally includes the solution.

Method	SWE-bench Verified	Terminal-Bench 2.0
Baseline	0.852 $\pm$ 0.010	0.839 $\pm$ 0.001
IRT-Agent	0.930 $\pm$ 0.002	0.923 $\pm$ 0.001
Oracle	0.949 $\pm$ 0.002	0.935 $\pm$ 0.001

Table 4: AUC-ROC (mean

\pm

std of 5-fold cross-validation means across 20 random seeds) of predictors fitted on held-out agent data. Baseline predicts the task’s empirical solve rate; Oracle is a standard IRT trained on all data.

Method	SWE-bench Pro	GSO
Baseline	0.571	0.637
Embedding	0.667	0.724
LLM-as-a-judge	0.696	0.743
Combined	0.674	0.719
Oracle	0.913	0.912

Table 5: AUC-ROC of predictors fitted on three benchmarks when evaluated on a fourth held-out benchmark. Baseline predicts the LLM’s empirical success rate ignoring the scaffold; Oracle is an IRT with agent features trained on all data from all four benchmarks.

Figure 2: Adaptive evaluation of a new agent on SWE-bench Verified. Mean absolute error between the running ability estimate

\hat{\theta}_{i}

and the full-benchmark IRT ability

\theta_{i}^{\star}

, averaged over 30 held-out LLM-scaffold pairs. Shaded regions show 95% bootstrap confidence intervals over held-out agents.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Feature source	Extracted by	# extracted	# in default set
Problem statement	Claude Opus 4.6 (main prompt)	15	10
Test cases	Claude Opus 4.6 (main prompt)	3	1
Solution	Claude Opus 4.6 (main prompt)	2	1
Repository state	GPT-5.4 (auditor agent prompt)	8	3
Total		28	15

Appendix

Table 6: LLM-as-a-judge feature inventory. The default set of 15 features is used, unchanged, for every benchmark and every experiment except the feature source ablations.

Ablation (Table 3 )	Ablation (Table 10 )	Sources in predictor	Sources in main prompt
Problem statement only	Problem statement	stmt.	stmt.
	+ Repository state	stmt., repo.	stmt.
Full w/o solution	+ Tests	stmt., tests, repo.	stmt., tests
Full w/o tests		stmt., sol., repo.	stmt., sol.
Full w/o repository state		stmt., tests, sol.	stmt., tests, sol.
Full	+ Solution	stmt., tests, sol., repo.	stmt., tests, sol.

Appendix

Table 7: Configuration of each feature source ablation level. “Sources in predictor” lists the sources whose features enter the predictor at the given level. “Sources in main prompt” lists the sources provided in the main judge prompt (used to extract all features included at the given level except the repository state features, which are extracted via an independent pipeline).

Backbone	AUC-ROC $\pm$ std
Qwen3-VL-4B-Instruct	0.814 $\pm$ 0.017
Qwen3-VL-8B-Instruct	0.816 $\pm$ 0.017
Qwen3-VL-32B-Instruct	0.817 $\pm$ 0.018
Qwen3-8B	0.798 $\pm$ 0.021
Qwen3-14B	0.817 $\pm$ 0.021
Qwen3-32B	0.801 $\pm$ 0.026

Appendix

Table 8: Embedding backbone test . AUC-ROC is averaged over 5 cross-validation folds; standard deviation ( std ) is also shown.

Benchmark	Claude Opus 4.6	GPT-5.4	Claude Sonnet 4.6
SWE-bench Verified	0.842	0.835	0.838
SWE-bench Pro	0.759	0.760	0.758
GSO	0.804	0.701	0.746
Terminal-Bench 2.0	0.810	0.828	0.830

Appendix

Table 9: LLM-as-a-Judge backbone ablation. Grouped Ridge (Embedding + LLM Judge) AUC-ROC across backbones. Bold indicates best per dataset.

Feature Source	SWE-bench Verified	SWE-bench Pro	GSO	Terminal-Bench 2.0
Problem statement	0.786 $\pm$ 0.002	0.718 $\pm$ 0.002	0.710 $\pm$ 0.012	0.789 $\pm$ 0.007
+ Repository state	0.797 $\pm$ 0.002	0.735 $\pm$ 0.001	0.713 $\pm$ 0.012	0.796 $\pm$ 0.008
+ Tests	0.833 $\pm$ 0.002	0.748 $\pm$ 0.002	0.728 $\pm$ 0.010	0.797 $\pm$ 0.005
+ Solution	0.845 $\pm$ 0.002	0.750 $\pm$ 0.003	0.774 $\pm$ 0.016	0.799 $\pm$ 0.006

Appendix

Table 10: Hierarchical information source ablation. AUC-ROC (mean

\pm

std of cross-validation means across 20 random seeds) of LLM-as-a-judge difficulty predictors as feature sources are added in order of increasing access (problem statement, repository state, test cases, solution). Each row adds one source on top of those in the previous row, and the top 15 features are selected by ridge regression coefficient from the cumulative features available at each row, similar to Table 3 . The solution contributes the largest gain on three of four benchmarks—most strikingly on GSO, where adding the solution lifts AUC from 0.728 to 0.774—but it is also the feature source which is the most difficult to obtain. On SWE-bench Verified, the test cases contribute a comparable amount (0.797

\to

0.833).

Benchmark	Baseline	Embedding	LLM-as-a-Judge	Combined	Oracle
SWE-bench Verified	0.718	0.825	0.842	0.843	0.945
SWE-bench Pro	0.657	0.756	0.744	0.750	0.918
GSO	0.714	0.771	0.769	0.781	0.914
Terminal-Bench 2.0	0.734	0.795	0.805	0.807	0.932

Appendix

Table 11: Jointly training the linear difficulty predictor model with the IRT abilities results in slightly lower AUC for held-out tasks on most benchmarks compared to training a ridge regression and using frozen IRT abilities during evaluation.

Functional Form	Formula	AUC-ROC
Sum	$\theta_{m}+\theta_{s}$	0.939 $\pm$ 0.000
Maximum	$\max(\theta_{m},\theta_{s})$	0.937 $\pm$ 0.000
Minimum	$\min(\theta_{m},\theta_{s})$	0.937 $\pm$ 0.000
Product	$\mathrm{sign}(\theta_{m}+\theta_{s})\left\|\theta_{m}\theta_{s}\right\|$	0.933 $\pm$ 0.001
L2 norm	$\mathrm{sign}(\theta_{m}+\theta_{s})\sqrt{\theta_{m}^{2}+\theta_{s}^{2}}$	0.938 $\pm$ 0.000
Harmonic mean	$\mathrm{sign}(\theta_{m}+\theta_{s})\left\|\frac{2\theta_{m}\theta_{s}}{\theta_{m}+\theta_{s}}\right\|$	0.815 $\pm$ 0.004

Appendix

Table 12: Functional form test for the relationship between the LLM and scaffold ability parameters . All runs are conducted with the best backbone ( DeepSeek-R1-Distill-Qwen-32B ). AUC-ROC is the mean (

\pm

std) of 5-fold cross-validation means across 20 random seeds. In the weighted sum,

w_{m}

and

w_{s}

are additional learnable parameters (shared for all LLMs and scaffolds). Sum with variance term aims to capture each LLM’s sensitivity to scaffolds: combined with a good scaffold (positive

\theta_{s}

), an LLM with a high variance term

\lambda_{m}

will do better than predicted by regular sum, and combined with a bad scaffold (negative

\theta_{s}

), worse than predicted by regular sum.

Protocol	Baseline	Embedding	LLM-as-a-Judge	Combined	Oracle
Binary (main paper)	0.732 $\pm$ 0.002	0.767 $\pm$ 0.010	0.794 $\pm$ 0.007	0.792 $\pm$ 0.013	0.932 $\pm$ 0.001
All attempts	0.754 $\pm$ 0.002	0.777 $\pm$ 0.005	0.801 $\pm$ 0.005	0.796 $\pm$ 0.010	0.912 $\pm$ 0.001

Appendix

Table 13: Treating each attempt as an independent trial on Terminal-Bench 2.0 (mean

\pm

std of cross-validation means across 20 random seeds). The Binary row reproduces the Terminal-Bench 2.0 row of Table 2 ; the All attempts row repeats the same experiment but treats each attempt as an independent observation for both training and evaluation. The Oracle drops by about 0.02 AUC under this protocol due to being evaluated on multiple attempts instead of one binary majority outcome; the predictor methods reach comparable AUC under both protocols.

Table 14: Bootstrapping results for the New Tasks experiment. All IRT-based methods significantly beat the naive baseline (two-sided

p

-value below the resolution of bootstrapping with 10,000 samples, which is equal to 0.0002; Holm-adjusted

p

-value below 0.0024).

Table 15: Bootstrapping results for the New Responses experiment. IRT-Agent —the proposed IRT model with LLM-scaffold decomposition—is not expected to (significantly) beat Standard IRT , because its intended use case is generalization to multi-benchmark IRT models.

Table 16: Bootstrapping results for the New Agents experiment. IRT-Agent significantly beats the naive baseline (two-sided

p

-value below the resolution of bootstrapping with 10,000 samples, which is equal to 0.0002; Holm-adjusted

p

-value below 0.0004).

Table 17: Bootstrapping results for the New Benchmarks experiment. All IRT-based methods significantly beat the naive baseline, both before and after the Holm-Bonferroni correction.

	Terminal-Bench 2.0	GSO
domain_knowledge_required (1–5)
1	Basic shell commands anyone could use (ls, cd, cat, echo)	Basic Python performance (list comprehensions, generators)
2	Standard Unix tools (grep, sed, awk, find)	Standard library optimization patterns
3	Specialized tools or configurations (cmake, git internals, network tools)	Library-specific knowledge (numpy, pandas internals)
4	Deep understanding of systems (kernel, filesystems, protocols)	Deep understanding of library implementation
5	Obscure tools, APIs, or highly specialized domain knowledge	Expert knowledge (SIMD, memory layout, CPU caches)

Appendix

Table 18: Per-dataset scale text variants for features that differ across datasets. Five features ( solution_hint , logical_reasoning_required , atypicality , verification_difficulty , and codebase_scope ) use the same scale text across all datasets and are omitted.

	Terminal-Bench 2.0	GSO
reproduction_clarity (1–5)
1	No setup steps, unclear how to begin	No benchmark or test scenario provided
2	Vague environment requirements mentioned	Vague description of slow use case
3	General setup described	General performance scenario described
4	Clear steps but some prerequisites unclear	Clear benchmark but some parameters unclear
5	Exact setup and execution steps with commands provided	Exact benchmark with input sizes and expected speedup

Appendix

Table 19: Per-dataset scale text variants for the 5 additional features (from the information ablation) that differ across datasets.

Benchmark	IRT-Agent	Standard IRT	Oracle
SWE-bench Verified	0.939 $\pm$ 0.000	0.940 $\pm$ 0.000	0.945 $\pm$ 0.000
SWE-bench Pro	0.870 $\pm$ 0.002	0.875 $\pm$ 0.002	0.919 $\pm$ 0.000
GSO	0.810 $\pm$ 0.014	0.814 $\pm$ 0.015	0.920 $\pm$ 0.001
Terminal-Bench 2.0	0.925 $\pm$ 0.000	0.925 $\pm$ 0.001	0.935 $\pm$ 0.000
All benchmarks	0.930 $\pm$ 0.000	—	—

Appendix

Table 20: AUC-ROC (mean

\pm

std of cross-validation means across 20 random seeds) on held-out responses . IRT-Agent —our proposed IRT model with agent features—performs on par with Standard IRT despite being strictly less expressive, while enabling multi-benchmark training and predictions with high performance. Both approach Oracle , which is another standard IRT model trained on all data, including the held-out responses.

Figure 3: Task difficulty histograms. SWE-bench Pro is harder on average than Verified, GSO is the hardest, and Terminal-Bench 2.0 is highly heterogeneous.

Rank	LLM	Ability ( $\theta_{m}$ )
1	Gemini 3.1 Pro	3.660
2	GPT-5.3-Codex	3.440
3	Claude Opus 4.6	3.332
4	GPT-5.2-Codex	2.997
5	GPT-5.2	2.697
6	Claude Opus 4.5	2.607

Appendix

Table 21: Top-15 and bottom-15 LLMs by IRT ability. Teletype font indicates open-weight LLMs.

Rank	Scaffold	Ability ( $\theta_{s}$ )
1	AgentScope	1.453
2	Learn-by-interact	1.207
3	Ante	1.060
4	Junie	0.919
5	AutoCodeRover v20240620	0.896
6	Z.ai	0.805

Appendix

Table 22: Top-15 and bottom-15 scaffolds by IRT ability. Teletype font indicates open-source scaffolds.

Figure 4: Validation of decomposition. Strong correlation (Pearson

r=0.974

) between agent abilities learned on a fixed scaffold (Terminus 2) versus LLM abilities isolated via our decomposition method.

Explore similar work

Code Understanding is a Bottleneck for Coding Agents
Oct 7, 2026Nishant Balepur, Kiran Tomlinson, Tobias SchnabelCoding Agents LLM Agent Evaluation
Microsoft Research · University of Maryland
Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
Oct 7, 2026Tan Yu, Alexander Bukharin, Khushi Bhardwaj +19AI Coding Agents AI Agent Evaluation
NVIDIA · University of Minnesota – Twin Cities · University of California, Berkeley
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1AI Coding Agents AI Agent Benchmarks

Oct 7, 2026J/K move · Enter open · S save

Code Understanding is a Bottleneck for Coding Agents

Nishant Balepur, Kiran Tomlinson, Tobias Schnabel

Microsoft Research · University of Maryland

Coding Agents LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents

Open paper AskPDF arXiv

Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following. We run eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size~grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while a separate study on SWE-bench Verified shows these tool call counts predict agents' accuracy better than lines of code edited, suggesting task difficulty for agents can lie in understanding code to edit, not just in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this finding, as agent accuracy finally falls. More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.

Abstract

Explore similar work

Code Understanding is a Bottleneck for Coding Agents

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?