Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Figures & tables
Figure 1: Automated action classification pipeline. The rule-based parser pairs each tool call with its execution output and classifies it across four core attributes: primary operation, target file role, mutation type, and read mode.
Figure 2: Workflow differences across outcome-matched agent pairs. Points indicate the median task-level absolute difference across 60 scaffold-matched agent configuration pairs ( ∣Δsolve∣≤1.0pp on ≥50 shared tasks), with bars showing the interquartile range (IQR). The right column reports the count of pairs whose task-paired 95% bootstrap confidence interval excludes zero. Top: primary workflow metrics. Bottom: supporting metrics.
Target Attribute
Consensus Rate
Agreement with Consensus
Macro-F1
Primary operation
94.0%
82.4%
88.8%
Target file role
96.6%
77.0%
57.8%
Mutation type
98.2%
68.8%
73.7%
Read mode
99.0%
96.6%
98.4%
Ordered operations
87.8%
86.0%
71.6%
Read scope
100.0%
100.0%
100.0%
Table 1: Multi-model consensus audit of the rule-based parser. Reference labels require agreement among at least 5 of 6 frontier LLMs across the 500-action sample.
Separated metrics
Split-half null
Matched pairs ( ∣Δ∣≤1 pp)
≥1 metric
30.53%
100.0%
≥2 metrics
6.40%
100.0%
≥3 metrics
0.781%
96.7%
Table 2: Behavioral separation in matched pairs vs. null baseline. Nominal proportion of pairs with a task-paired 95% interval excluding 0 on ≥k of the six workflow metrics.
Claude Opus (SWE)
GPT (SWE)
Opus 4.8 (CC)
Metric
4.6
4.7
4.8
5
5.4
5.5 M
5.5 xH
5.6 Sol
Low
Med
High
xHigh
Total actions
47.2
42.8
33.2
47.6
24.4
24.2
43.4
12.8
31.3
35.5
39.8
54.4
File-read share (%)
32.3
28.4
25.8
27.8
29.2
26.4
27.5
30.3
23.2
24.3
23.9
25.2
File-mutation share (%)
12.7
15.9
16.1
12.9
14.9
15.8
11.2
12.3
19.9
18.0
17.1
15.4
Workspace-inspection share (%)
4.4
3.1
3.9
4.6
6.9
8.5
8.7
11.4
0.7
0.9
1.3
2.2
sed -n reads (% of reads)
6.9
58.8
61.0
57.7
53.6
87.4
79.9
76.4
3.6
3.3
2.8
2.4
Table 3: Generational evolution and reasoning-effort scaling across workflow metrics (110 shared tasks).
Model
Scaffold
Native Edit
Shell Redirection
Python Script
Native Read
Opus 4.7
SWE-agent-like
37.0
24.4
4.6
5.5
Opus 4.7
Claude Code
92.4
5.4
0.2
90.5
GPT-5.5
SWE-agent-like
0.5
14.9
69.0
0.0
GPT-5.5
Codex
30.9
7.1
54.6
0.0
Table 4: Code editing and reading mechanisms across scaffolding harnesses. Values show shares (%) of recognized edit and read events.
Cross-Repository (111 stacks)
Multilingual (16 stacks)
Feature Representation
Top-1
Top-3
Top-1
Top-3
Full behavioral profile
85.1%
96.8%
90.3%
98.8%
Action-coordinate subset
62.6%
86.5%
87.5%
98.4%
Table 5: Exact deployed-stack retrieval across unseen environments. Evaluating cross-repository transfer (111 stacks, chance =0.9% ) with an 8-feature profile and cross-language transfer (16 stacks across 5 languages, chance =6.25% ) with a separate 12-feature profile. Row labels denote within-panel representations, not a shared feature schema.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Parser field or rule
Operational actions
Attempted nonterminal structured or synthetic calls; terminal submission is excluded.
Exploration share
primary_operation in { EXPLORE_REPOSITORY , SEARCH_REPOSITORY , READ_FILE }, divided by semantic_denominator .
File-mutation share
primary_operation == MUTATE_FILE , divided by semantic_denominator .
Workspace-inspection share
primary_operation == INSPECT_WORKSPACE_CHANGES , divided by semantic_denominator .
Non-test repository writes
Executed events containing resolved non-test repository paths in mutation_paths with no_observed_failure (no nonzero exit, failed test, or execution error); qualifying MIXED events count once.
Formal-test invocations
Count of recognized formal-test invocations in contained_operations_ordered ; this is an invocation rate when normalized and can exceed one per semantic event.
Appendix
Table 6: Metric dictionary linking the six workflow metrics and closure diagnostics to parser fields and event rules.
Feature Representation
Top-1
Top-3
Self Similarity
Nearest Non-Self
Full 8-feature profile
85.1%
96.8%
0.972
0.914
Normalized action coordinates only †
62.6%
86.5%
0.971
0.966
Sequence and verification only ‡
56.8%
79.7%
0.960
0.968
Generic harness only (SWE-agent)
85.5%
97.7%
0.973
0.914
Appendix
Table 7: Exact deployed-stack retrieval across held-out repositories (111 stacks, random chance =0.9% ). Ablations use subsets of the 8-dimensional observable profile; the generic-harness subset contains 107 stacks.
Table 8: Cross-language exact-stack retrieval (16 stacks across 5 languages, random chance =6.25% ). Results pool 320 directional queries across all ten 3-vs-2 language splits using task-centered cosine retrieval.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
62.4
15.0
Python script edits (% of edits)
0.0%
100.0%
Shell redirection edits (% of edits)
48.4%
0.0%
Formal-test invocations
12.4
3.4
Appendix
Table 9: Workflow metrics for Case 1 ( pypsa-1653 ) . Values are task-level means across 5 runs.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
67.2
11.6
Exploration and read actions
25.4
4.6
Formal-test invocations
6.6
4.6
Workspace-inspection operations
5.0
5.4
Appendix
Table 10: Workflow metrics for Case 2 ( ytmusicapi-909 ) . Values are task-level means across 5 runs.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
69.8
20.2
Formal-test invocations
9.2
3.6
Workspace-inspection operations
5.8
7.0
Actions after final counted write
8.8
4.4
Appendix
Table 11: Workflow metrics for Case 3 ( astronomy-commons__lsdb-1349 ) . Values are task-level means across 5 runs.
Trajectory Dimension
Low
Medium
xHigh
Difference [95% CI]
Total operational actions
31.3
35.5
54.4
+23.0[16.5,27.8]
Exploration and read actions
14.4
17.8
27.1
+12.7[11.0,14.5]
File-mutation events
5.4
6.1
8.3
+3.0[2.3,3.6]
Formal-test invocations
4.7
4.5
5.4
+0.7[−1.8,2.3]
Rollouts inspecting workspace changes after the final counted write (%)
12.7%
24.1%
66.8%
+54.1 pp[45.9,61.8]
Action-type switch count
19.0
23.0
34.5
+15.6[13.7,17.5]
Appendix
Table 12: Trajectory structural shifts across selected configured effort levels for Claude Opus 4.8 in Claude Code (110 common tasks, 2 rollouts per task). Formal-test volume counts contained invocations. Differences are computed before display rounding.
Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique which is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. ProcGrep is available at https://github.com/hamidahoderinwale/procgrep.
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.