LLM-based coding agents are primarily evaluated by their solve rates. However, this metric does not reveal how difficult the solved tasks are or account for differences in task difficulty across benchmarks. Determining the difficulty of individual tasks by repeatedly running multiple agents is costly, while a predicted difficulty score alone provides limited guidance on what support an agent needs. To address these challenges, we propose DARE (Difficulty Analysis for Repair Enhancement), which integrates repair-task difficulty measurement, prediction, feature-group contribution analysis, and task-specific repair support. We define continuous task difficulty as the observed failure rate across multiple coding agents. The DARE difficulty model predicts this difficulty using 151 static features extracted from benchmark artifacts. The framework also analyzes the contributions of seven Repair-Task Feature Groups to identify the characteristics most salient to each task and uses this information to select task-specific agent support. We evaluate DARE on four benchmark partitions comprising 2,594 tasks. The difficulty model's aligned predictions achieve Spearman's ρ = .476-.621 on held-out partitions, and the most salient Feature Groups vary across tasks and benchmarks. Furthermore, coding-agent experiments show that matching support to each task's salient Feature Group raises repair success from 54.7% to 79.6% on nine high-difficulty tasks. These findings demonstrate that difficulty prediction and interpretation can lead to practical improvements in automated program repair (APR), while also providing a basis for difficulty-stratified agent evaluation, cross-benchmark comparisons, and identifying areas of repair capability that warrant further improvement.
Figures & tables
Fig. 1: Overview of the DARE framework. Static repair artifacts yield task features organized into seven Feature Groups. The DARE difficulty model predicts repair difficulty, while its feature contributions are aggregated by Feature Group to guide the selection of task-specific assistance. The displayed contribution values and selected module are illustrative.
Population comparison
Spearman ρ
Provider
.808
Scaffold
.914
Submission era
.921
Performance level
.894
TABLE I: Cross-population consistency of task difficulty on SWE-bench Verified.
Level
Difficulty interval
L1
0≤D^i<0.2
L2
0.2≤D^i<0.4
L3
0.4≤D^i<0.6
L4
0.6≤D^i<0.8
L5
0.8≤D^i≤1.0
TABLE II: Fixed absolute difficulty levels.
Benchmark / partition
#Test
#Train
#Agents
SWE-bench Verified
500
2,094
125
Lite ∖ Verified
207
2,387
84
Full ∖ (Verified ∪ Lite)
1,587
1,007
24
SWE-bench-Live Lite
300
2,294
20
TABLE III: Benchmarks and LODO evaluation partitions.
Held-out partition
ρ [95% CI]
pAcc [95% CI]
MAE
SWE-bench Verified
.621 [.563,.674]
.723 [.698,.748]
.254
Lite ∖ Verified
.541 [.434,.635]
.694 [.658,.737]
.197
Full ∖ (Verified ∪ Lite)
.566 [.530,.601]
.744 [.725,.759]
.123
SWE-bench-Live Lite
.476 [.376,.564]
.675 [.639,.708]
.149
TABLE IV: Difficulty prediction performance under cross-partition holdout evaluation.
Fig. 2: Distribution of observed non-anchor failure rates across five fixed predicted absolute-difficulty levels (L1–L5), assigned using final aligned predictions. Boxes indicate the interquartile range (IQR), center lines indicate medians, whiskers extend to the most extreme observations within 1.5×IQR , and n above each level indicates the number of tasks. Outlier markers are omitted for readability.
Feature group
Importance (%)
Spearman ρ [95% CI]
Transformation ( T )
26.9
.412 [.366, .455]
Validation ( V )
20.1
.338 [.290, .380]
Specification–Solution Gap ( SG )
13.9
.347 [.298, .393]
Localization ( L )
13.5
.255 [.208, .301]
Context ( C )
11.0
.030 [ − .024, .082]
Specification ( S )
10.0
.052 [.001, .105]
TABLE V: Feature-Group Importance and Spearman correlation with observed non-anchor difficulty.
Fig. 3: Distribution of salient Feature Groups across predicted absolute difficulty levels (L1–L5). Each cell reports the percentage of tasks in that level assigned to the indicated salient-group category; n denotes the number of tasks at each level. Gray rows indicate levels containing no tasks.
Feature Group
Task
Baseline
With Module
L
matplotlib-29721
8/25
21/25
L
reflex-4371
16/25
19/25
L
keras-20443
16/25
23/25
T
pybamm-4865
2/25
13/25
T
pylint-10089
18/25
19/25
T
starlette-2812
18/25
22/25
TABLE VI: Repair success with and without Assistant Module.
Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties. Aims. We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks. Method. We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis. Results We found that task difficulty is substantially predictable from static features (AU C = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty. Conclusion. The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.
Ebtesam Al-Haque, Brittany Johnson
Department of Computer Science, George Mason University, Fairfax, VA
Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following. We run eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size~grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while a separate study on SWE-bench Verified shows these tool call counts predict agents' accuracy better than lines of code edited, suggesting task difficulty for agents can lie in understanding code to edit, not just in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this finding, as agent accuracy finally falls. More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.
As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult. This is compounded by current practice: agent performance is typically measured by aggregate pass rates on benchmarks, but single-number metrics obscure the diversity of tasks within a benchmark. We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime. Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases, and introduces a novel decomposition of agent ability into LLM and scaffold ability components. This parameterization enables us to aggregate evaluation data across heterogeneous leaderboards and accurately predict task-level performance for unseen benchmarks, as well as unseen LLM-scaffold combinations. Our methods have practical utility for benchmark designers, who can better calibrate the difficulty of their new tasks without running computationally expensive agent evaluations.