Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties. Aims. We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks. Method. We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis. Results We found that task difficulty is substantially predictable from static features (AU C = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty. Conclusion. The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.
Figures & tables
Patch (18)
Repository (14)
Prompt (31)
Edit volume (3)
Scale (5)
Component load (4)
Edit fragmentation (4)
Directory structure (4)
Coordinative complexity (6)
File scope (4)
Naming ambiguity (1)
Coordination ambiguity (5)
Churn distribution (3)
Test infrastructure (3)
Attachment ambiguity (7)
Spatial distribution (4)
Documentation (1)
Referential ambiguity (4)
Cohesion (5)
Table 1: Feature taxonomy overview. Each cell shows the sub-construct and its feature count
Table 6: Held-out test performance. Majority denotes the always-positive baseline. Bold denotes best non-baseline result per metric per target.
Figure 1: Calibration reliability diagrams
any_success
maj_success
pass_rate
Feature set
N
AUC
Brier
AUC
Brier
R2
MAE
Patch only
13
0.846
0.135
0.830
0.154
0.375
0.285
Repo only
14
0.839
0.137
0.820
0.158
0.350
0.291
Prompt only
27
0.599
0.214
0.593
0.231
0.025
0.417
Patch + Repo
27
0.861
0.129
0.841
0.149
0.401
0.275
Patch + Prompt
40
0.850
0.135
0.832
0.154
0.380
0.290
Table 7: Component ablation using XGBoost on the held-out test set. AUC and Brier score are reported for the two binary outcomes; R2 and MAE for the continuous pass_rate outcome. Bold denotes the best value in each column.
Figure 2: Rank–magnitude bump chart for the top four SHAP interaction pairs by mean absolute interaction value averaged across outcome targets.
Figure 3: Rank–magnitude bump chart for the top 20 features by mean absolute SHAP value across three outcome targets (XGBoost, held-out test set). Each dot is one feature–outcome pair; dot area is proportional to mean absolute SHAP. Connecting lines trace rank shifts across outcomes; steep slopes indicate instability. Color encodes feature family: patch (blue), repo (red-brown), prompt (green). The top six ranks are stable across all three outcomes (near-flat lines); prompt features first appear at rank 16 with visibly smaller dots and show greater rank instability below rank 16.
Figure 4: SHAP summary plot for any_success .
Figure 5: SHAP waterfalls for extreme-difficulty tasks (XGBoost, any_success target). All values in log-odds space; E[f(x)]=0.73 for both.
Figure 6: SHAP waterfalls for four representative average-difficulty tasks ( p^≈0.68 , E[f(x)]=0.73 , XGBoost, any_success ). In three of four cases a prompt feature ranks among the top contributors, with magnitudes comparable to the structural features for those tasks.