LLM-based coding agents are primarily evaluated by their solve rates. However, this metric does not reveal how difficult the solved tasks are or account for differences in task difficulty across benchmarks. Determining the difficulty of individual tasks by repeatedly running multiple agents is costly, while a predicted difficulty score alone provides limited guidance on what support an agent needs. To address these challenges, we propose DARE (Difficulty Analysis for Repair Enhancement), which integrates repair-task difficulty measurement, prediction, feature-group contribution analysis, and task-specific repair support. We define continuous task difficulty as the observed failure rate across multiple coding agents. The DARE difficulty model predicts this difficulty using 151 static features extracted from benchmark artifacts. The framework also analyzes the contributions of seven Repair-Task Feature Groups to identify the characteristics most salient to each task and uses this information to select task-specific agent support. We evaluate DARE on four benchmark partitions comprising 2,594 tasks. The difficulty model's aligned predictions achieve Spearman's ρ = .476-.621 on held-out partitions, and the most salient Feature Groups vary across tasks and benchmarks. Furthermore, coding-agent experiments show that matching support to each task's salient Feature Group raises repair success from 54.7% to 79.6% on nine high-difficulty tasks. These findings demonstrate that difficulty prediction and interpretation can lead to practical improvements in automated program repair (APR), while also providing a basis for difficulty-stratified agent evaluation, cross-benchmark comparisons, and identifying areas of repair capability that warrant further improvement.
Figures & tables
Fig. 1: Overview of the DARE framework. Static repair artifacts yield task features organized into seven Feature Groups. The DARE difficulty model predicts repair difficulty, while its feature contributions are aggregated by Feature Group to guide the selection of task-specific assistance. The displayed contribution values and selected module are illustrative.
Population comparison
Spearman ρ
Provider
.808
Scaffold
.914
Submission era
.921
Performance level
.894
TABLE I: Cross-population consistency of task difficulty on SWE-bench Verified.
Level
Difficulty interval
L1
0≤D^i<0.2
L2
0.2≤D^i<0.4
L3
0.4≤D^i<0.6
L4
0.6≤D^i<0.8
L5
0.8≤D^i≤1.0
TABLE II: Fixed absolute difficulty levels.
Benchmark / partition
#Test
#Train
#Agents
SWE-bench Verified
500
2,094
125
Lite ∖ Verified
207
2,387
84
Full ∖ (Verified ∪ Lite)
1,587
1,007
24
SWE-bench-Live Lite
300
2,294
20
TABLE III: Benchmarks and LODO evaluation partitions.
Held-out partition
ρ [95% CI]
pAcc [95% CI]
MAE
SWE-bench Verified
.621 [.563,.674]
.723 [.698,.748]
.254
Lite ∖ Verified
.541 [.434,.635]
.694 [.658,.737]
.197
Full ∖ (Verified ∪ Lite)
.566 [.530,.601]
.744 [.725,.759]
.123
SWE-bench-Live Lite
.476 [.376,.564]
.675 [.639,.708]
.149
TABLE IV: Difficulty prediction performance under cross-partition holdout evaluation.
Fig. 2: Distribution of observed non-anchor failure rates across five fixed predicted absolute-difficulty levels (L1–L5), assigned using final aligned predictions. Boxes indicate the interquartile range (IQR), center lines indicate medians, whiskers extend to the most extreme observations within 1.5×IQR , and n above each level indicates the number of tasks. Outlier markers are omitted for readability.
Feature group
Importance (%)
Spearman ρ [95% CI]
Transformation ( T )
26.9
.412 [.366, .455]
Validation ( V )
20.1
.338 [.290, .380]
Specification–Solution Gap ( SG )
13.9
.347 [.298, .393]
Localization ( L )
13.5
.255 [.208, .301]
Context ( C )
11.0
.030 [ − .024, .082]
Specification ( S )
10.0
.052 [.001, .105]
TABLE V: Feature-Group Importance and Spearman correlation with observed non-anchor difficulty.
Fig. 3: Distribution of salient Feature Groups across predicted absolute difficulty levels (L1–L5). Each cell reports the percentage of tasks in that level assigned to the indicated salient-group category; n denotes the number of tasks at each level. Gray rows indicate levels containing no tasks.
Feature Group
Task
Baseline
With Module
L
matplotlib-29721
8/25
21/25
L
reflex-4371
16/25
19/25
L
keras-20443
16/25
23/25
T
pybamm-4865
2/25
13/25
T
pylint-10089
18/25
19/25
T
starlette-2812
18/25
22/25
TABLE VI: Repair success with and without Assistant Module.