Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties. Aims. We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks. Method. We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis. Results We found that task difficulty is substantially predictable from static features (AU C = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty. Conclusion. The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.
Figures & tables
Patch (18)
Repository (14)
Prompt (31)
Edit volume (3)
Scale (5)
Component load (4)
Edit fragmentation (4)
Directory structure (4)
Coordinative complexity (6)
File scope (4)
Naming ambiguity (1)
Coordination ambiguity (5)
Churn distribution (3)
Test infrastructure (3)
Attachment ambiguity (7)
Spatial distribution (4)
Documentation (1)
Referential ambiguity (4)
Cohesion (5)
Table 1: Feature taxonomy overview. Each cell shows the sub-construct and its feature count
Table 6: Held-out test performance. Majority denotes the always-positive baseline. Bold denotes best non-baseline result per metric per target.
Figure 1: Calibration reliability diagrams
any_success
maj_success
pass_rate
Feature set
N
AUC
Brier
AUC
Brier
R2
MAE
Patch only
13
0.846
0.135
0.830
0.154
0.375
0.285
Repo only
14
0.839
0.137
0.820
0.158
0.350
0.291
Prompt only
27
0.599
0.214
0.593
0.231
0.025
0.417
Patch + Repo
27
0.861
0.129
0.841
0.149
0.401
0.275
Patch + Prompt
40
0.850
0.135
0.832
0.154
0.380
0.290
Table 7: Component ablation using XGBoost on the held-out test set. AUC and Brier score are reported for the two binary outcomes; R2 and MAE for the continuous pass_rate outcome. Bold denotes the best value in each column.
Figure 2: Rank–magnitude bump chart for the top four SHAP interaction pairs by mean absolute interaction value averaged across outcome targets.
Figure 3: Rank–magnitude bump chart for the top 20 features by mean absolute SHAP value across three outcome targets (XGBoost, held-out test set). Each dot is one feature–outcome pair; dot area is proportional to mean absolute SHAP. Connecting lines trace rank shifts across outcomes; steep slopes indicate instability. Color encodes feature family: patch (blue), repo (red-brown), prompt (green). The top six ranks are stable across all three outcomes (near-flat lines); prompt features first appear at rank 16 with visibly smaller dots and show greater rank instability below rank 16.
Figure 4: SHAP summary plot for any_success .
Figure 5: SHAP waterfalls for extreme-difficulty tasks (XGBoost, any_success target). All values in log-odds space; E[f(x)]=0.73 for both.
Figure 6: SHAP waterfalls for four representative average-difficulty tasks ( p^≈0.68 , E[f(x)]=0.73 , XGBoost, any_success ). In three of four cases a prompt feature ranks among the top contributors, with magnitudes comparable to the structural features for those tasks.
Agentic software engineering benchmarks are typically summarized by nominal category labels such as "bug fix" or "feature implementation," yet benchmarks carrying the same label are built through very different curation pipelines. A label thus reveals little about the engineering work a benchmark demands. We introduce the Spread--Novelty--Centrality (SNC) profile, a three-axis characterization of the demands of repository-level coding tasks, grounded in empirical software engineering research. We apply the profile to five widely used benchmarks and 14,922 trajectories of two model families at three scales, and report three findings. (1) A label is an unreliable proxy for task demands, as every pair of benchmarks is statistically separated on at least two SNC axes, and the separations trace back to specific curation decisions. (2) Agent behaviour reveals demands that the human-written gold solution cannot. Agents produce larger solutions than the gold where problem statements withhold hints and smaller ones where curation inflates the gold. How a task is phrased shapes what an agent produces. (3) Task demands correlate with success uniformly, with resolved runs concentrating in the low-SNC region for every family and scale, whereas the behavioural signatures of success are family-specific. Claude succeeds by matching the scope of the gold solution, and its parity share on files rises from 0.17 at the smallest scale to 0.54 at the largest. Qwen succeeds by exceeding the gold scope at every scale, and editing too little marks failure for both families.
Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult. This is compounded by current practice: agent performance is typically measured by aggregate pass rates on benchmarks, but single-number metrics obscure the diversity of tasks within a benchmark. We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime. Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases, and introduces a novel decomposition of agent ability into LLM and scaffold ability components. This parameterization enables us to aggregate evaluation data across heterogeneous leaderboards and accurately predict task-level performance for unseen benchmarks, as well as unseen LLM-scaffold combinations. Our methods have practical utility for benchmark designers, who can better calibrate the difficulty of their new tasks without running computationally expensive agent evaluations.
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.