HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Wanyi Chen; Wanqin Shen; Yuanchen Shi; Zhensu Sun; Fang Kong

cs.AIOct 17, 2026

HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Authors: Wanyi Chen, Wanqin Shen, Yuanchen Shi, Zhensu Sun, Fang Kong

Organizations: Soochow University · Singapore Management University

Abstract

Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets. Code and project resources are publicly available.

Figures & tables

Figure 1: Task-efficient software-agent evaluation. Representative strategies estimate full-benchmark scores from limited target executions. HSPR combines hierarchical response modeling with adaptive querying. The displayed median is the decision before source-selected calibration.

Figure 2: HSPR evaluation pipeline. Source responses provide fitted response factors and source-signature groups. Sequential queries update the posteriors over the target latent state and group success rates. Observed successes and posterior-predictive success counts for unqueried tasks define the full-score posterior used for the final calibrated estimate.

Method	Lite (30/300)			Verified (50/500)			Full (229/2294)			Pro (73/730)
	MAE $\downarrow$	MaxErr $\downarrow$	$\tau_{b}$ $\uparrow$	MAE $\downarrow$	MaxErr $\downarrow$	$\tau_{b}$ $\uparrow$	MAE $\downarrow$	MaxErr $\downarrow$	$\tau_{b}$ $\uparrow$	MAE $\downarrow$	MaxErr $\downarrow$	$\tau_{b}$ $\uparrow$
Uniform raw	6.632	19.300	0.654	5.482	15.460	0.785	2.227	5.342	0.923	3.832	10.233	0.757
Stratified	4.468	13.354	0.723	3.302	10.768	0.843	1.636	3.877	0.938	3.111	7.508	0.837
RSL ( Zhang et al., 2025 )	4.911	16.064	0.683	3.654	14.590	0.823	2.277	8.547	0.930	4.467	12.822	0.714
FAQ-port ( Wu et al., 2026 )	6.681	26.263	0.638	4.951	25.989	0.765	1.807	5.049	0.921	5.433	15.330	0.673
Mid-Range ( Ndzomga, 2026 )	18.448	49.333	0.830	16.449	32.600	0.915	24.709	50.239	0.862	22.798	39.041	0.967

Table 1: Main results at a matched 10% task budget. MAE and MaxErr are in percentage points (pp);

\tau_{b}

is unscaled. Metrics pool complete target cohorts before averaging query seeds. Best and second-best point estimates are bold and underlined, respectively, using unrounded values. Section 4.1 describes the baselines and ports.

A. Hierarchical modeling: source-selected acquisition objective
Ablation / control	Lite $\downarrow$	Verified $\downarrow$	Full $\downarrow$	Pro $\downarrow$	Macro $\downarrow$
HSPR (ours)	1.927	1.915	0.717	2.098	1.664
w/o group uncertainty	1.895	1.970	1.331	2.034	1.808
B. Query adaptivity: frozen HSPR reconstruction
w/o adaptive queries (Uniform)	4.573	3.288	1.413	3.066	3.085
w/o adaptive queries (Mid-Range)	3.961	3.059	2.741	1.975	2.934

Table 2: Ablations at a 10% budget; MAE in pp, with equal-bank Macro. A retains each fold’s acquisition objective, global configuration, and median calibration; only group uncertainty changes, and both variants generate their own query paths. B freezes HSPR reconstruction and calibration, using five Uniform seeds or one deterministic Mid-Range path. C shares HSPR query paths, with decision-specific calibration selected on sources.

Bank	Tasks $N$	Groups $\bar{G}$	$N/\bar{G}$	Source-zero tasks
				Share (%)	Target success (%)
Lite	300	201.5	1.49	24.6	0.87
Verified	500	426.0	1.17	8.3	0.74
Full	2294	275.9	8.32	41.3	1.84
Pro	730	222.0	3.29	36.7	2.07

Table 3: Historical response structure.

\bar{G}

averages source group counts over targets;

N/\bar{G}

divides the task count by this average. Source-zero tasks have no successes among the source agents. Target success rates within these sets are averaged over targets, using complete outcomes after prediction.

Figure 3: Budget sensitivity across four SWE-bench task banks at 5–20% task budgets. HSPR uses fixed 10% source-selected settings and nested query paths; baselines retain budget-specific settings. MAE and MaxErr are in pp; metrics pool target configurations before averaging query seeds.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Bank	Fold	$r$	$\gamma$	$\kappa$	Policy	$\lambda$
Lite	0	1	4	1	Score	0.5
Lite	1	4	4	4	Score	0.5
Lite	2	4	8	0.25	Score	0.5
Lite	3	1	8	4	Latent	0
Verified	0	4	1	1	Score	0
Verified	1	1	8	1	Score	0.5

Appendix

Table 4: Frozen source-selected settings. For

r>0

, the scalar ability precision is .25 and residual precision is

\gamma

; for

r=0

, ability precision is

\gamma

. Score and Latent denote the acquisition rules in Section 3.4 .

Benchmark	Comparator	$\Delta$ MAE (pp)	Paired 95% CI (pp)	Holm $p$
Lite	ATLAS-port	-0.762	$[-1.370,-0.162]$	0.081
Lite	PTA-IRT	-0.797	$[-1.724,+0.006]$	1.000
Lite	Stratified	-2.541	$[-3.205,-1.886]$	$<.001$
Lite	RSL	-2.984	$[-3.812,-2.258]$	$<.001$
Verified	ATLAS-port	+0.055	$[-0.355,+0.448]$	1.000
Verified	PTA-IRT	-0.262	$[-0.825,+0.283]$	1.000

Appendix

Table 5: Paired MAE differences: HSPR minus comparator, so negative favors HSPR. Seed-averaged agent errors are paired. The 95% percentile bootstrap intervals use 10,000 configuration resamples, conditional on the archived task bank, and are unadjusted. Wilcoxon

p

-values are Holm-adjusted across all 16 comparisons.

Bank	$N$	$\bar{G}$	$N/\bar{G}$	MAE $\downarrow$		Group gain $\uparrow$
				w/o groups	with groups
Lite	300	201.5	1.49	1.949	1.941	+0.009
Verified	500	426.0	1.17	1.928	1.927	+0.001
Full	2294	275.9	8.32	0.902	0.722	+0.180
Pro	730	222.0	3.29	2.101	2.108	-0.006

Appendix

Table 6: Historical group structure and matched-path reconstruction at 10% budgets. Both MAE columns use posterior means with identical query paths and calibration.

\bar{G}

is the target-averaged source group count. Group gain is MAE without minus with group uncertainty, computed before rounding (pp). Bold marks the largest

N/\bar{G}

and gain.

Bank	Source groups	Source-zero tasks (%)	Target success on zeros (%)	Queries to zeros (%)
Lite	201.5	24.6	0.87	0.248
Verified	426.0	8.3	0.74	0.000
Full	275.9	41.3	1.84	13.225
Pro	222.0	36.7	2.07	0.020

Appendix

Table 7: Post-hoc support diagnostic at 10% budgets. Each target uses its frozen source fold; entries average targets equally, with queries averaged over five seeds. Source-zero tasks have zero successes among all source agents. Target success rates are computed after prediction for this diagnostic.

Task bank	Runs	Median (s)	P90 (s)
Lite	175	6.19	8.57
Verified	350	20.31	32.35
Full	70	30.96	61.67
Pro	70	7.79	14.12

Appendix

Table 8: HSPR evaluator wall time per target and query seed at the 10% task budget. P90 is the 90th percentile across runs within each bank.

Figure 4: Full-benchmark scores and HSPR estimates by model family at the 10% task budget. Points show mean predictions over five query seeds; bars show one sample standard deviation across those seeds. The dashed line marks exact agreement, and the shaded reference band marks

\pm 2

pp. Panel MAE averages absolute errors over configurations and seeds. Family labels follow archived configuration names; Mixed denotes multiple named families and Unspecified denotes no identified family.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Sep 1, 2026Kefeng Duan, Dewu Zheng, Yanlin Wang +7Software Engineering Benchmarks LLM Agent Evaluation
School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation
May 11, 2026Shanshan Gao, Liyi ZhouComputer-Use Agent Benchmarks Benchmark Design
The University of Sydney
Efficient Benchmarking of AI Agents
Mar 24, 2026Franck NdzomgaAI Agent Evaluation AI Agent Benchmarks

Sep 1, 2026J/K move · Enter open · S save

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang+6

School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Software Engineering Benchmarks LLM Agent Evaluation AI Agent Evaluation Item Response Theory

Open paper AskPDF arXiv

Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.

Abstract

AI Agent Evaluation

Explore similar work

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation

Efficient Benchmarking of AI Agents