Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Figures & tables
Figure 1: Automated action classification pipeline. The rule-based parser pairs each tool call with its execution output and classifies it across four core attributes: primary operation, target file role, mutation type, and read mode.
Figure 2: Workflow differences across outcome-matched agent pairs. Points indicate the median task-level absolute difference across 60 scaffold-matched agent configuration pairs ( ∣Δsolve∣≤1.0pp on ≥50 shared tasks), with bars showing the interquartile range (IQR). The right column reports the count of pairs whose task-paired 95% bootstrap confidence interval excludes zero. Top: primary workflow metrics. Bottom: supporting metrics.
Target Attribute
Consensus Rate
Agreement with Consensus
Macro-F1
Primary operation
94.0%
82.4%
88.8%
Target file role
96.6%
77.0%
57.8%
Mutation type
98.2%
68.8%
73.7%
Read mode
99.0%
96.6%
98.4%
Ordered operations
87.8%
86.0%
71.6%
Read scope
100.0%
100.0%
100.0%
Table 1: Multi-model consensus audit of the rule-based parser. Reference labels require agreement among at least 5 of 6 frontier LLMs across the 500-action sample.
Separated metrics
Split-half null
Matched pairs ( ∣Δ∣≤1 pp)
≥1 metric
30.53%
100.0%
≥2 metrics
6.40%
100.0%
≥3 metrics
0.781%
96.7%
Table 2: Behavioral separation in matched pairs vs. null baseline. Nominal proportion of pairs with a task-paired 95% interval excluding 0 on ≥k of the six workflow metrics.
Claude Opus (SWE)
GPT (SWE)
Opus 4.8 (CC)
Metric
4.6
4.7
4.8
5
5.4
5.5 M
5.5 xH
5.6 Sol
Low
Med
High
xHigh
Total actions
47.2
42.8
33.2
47.6
24.4
24.2
43.4
12.8
31.3
35.5
39.8
54.4
File-read share (%)
32.3
28.4
25.8
27.8
29.2
26.4
27.5
30.3
23.2
24.3
23.9
25.2
File-mutation share (%)
12.7
15.9
16.1
12.9
14.9
15.8
11.2
12.3
19.9
18.0
17.1
15.4
Workspace-inspection share (%)
4.4
3.1
3.9
4.6
6.9
8.5
8.7
11.4
0.7
0.9
1.3
2.2
sed -n reads (% of reads)
6.9
58.8
61.0
57.7
53.6
87.4
79.9
76.4
3.6
3.3
2.8
2.4
Table 3: Generational evolution and reasoning-effort scaling across workflow metrics (110 shared tasks).
Model
Scaffold
Native Edit
Shell Redirection
Python Script
Native Read
Opus 4.7
SWE-agent-like
37.0
24.4
4.6
5.5
Opus 4.7
Claude Code
92.4
5.4
0.2
90.5
GPT-5.5
SWE-agent-like
0.5
14.9
69.0
0.0
GPT-5.5
Codex
30.9
7.1
54.6
0.0
Table 4: Code editing and reading mechanisms across scaffolding harnesses. Values show shares (%) of recognized edit and read events.
Cross-Repository (111 stacks)
Multilingual (16 stacks)
Feature Representation
Top-1
Top-3
Top-1
Top-3
Full behavioral profile
85.1%
96.8%
90.3%
98.8%
Action-coordinate subset
62.6%
86.5%
87.5%
98.4%
Table 5: Exact deployed-stack retrieval across unseen environments. Evaluating cross-repository transfer (111 stacks, chance =0.9% ) with an 8-feature profile and cross-language transfer (16 stacks across 5 languages, chance =6.25% ) with a separate 12-feature profile. Row labels denote within-panel representations, not a shared feature schema.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Parser field or rule
Operational actions
Attempted nonterminal structured or synthetic calls; terminal submission is excluded.
Exploration share
primary_operation in { EXPLORE_REPOSITORY , SEARCH_REPOSITORY , READ_FILE }, divided by semantic_denominator .
File-mutation share
primary_operation == MUTATE_FILE , divided by semantic_denominator .
Workspace-inspection share
primary_operation == INSPECT_WORKSPACE_CHANGES , divided by semantic_denominator .
Non-test repository writes
Executed events containing resolved non-test repository paths in mutation_paths with no_observed_failure (no nonzero exit, failed test, or execution error); qualifying MIXED events count once.
Formal-test invocations
Count of recognized formal-test invocations in contained_operations_ordered ; this is an invocation rate when normalized and can exceed one per semantic event.
Appendix
Table 6: Metric dictionary linking the six workflow metrics and closure diagnostics to parser fields and event rules.
Feature Representation
Top-1
Top-3
Self Similarity
Nearest Non-Self
Full 8-feature profile
85.1%
96.8%
0.972
0.914
Normalized action coordinates only †
62.6%
86.5%
0.971
0.966
Sequence and verification only ‡
56.8%
79.7%
0.960
0.968
Generic harness only (SWE-agent)
85.5%
97.7%
0.973
0.914
Appendix
Table 7: Exact deployed-stack retrieval across held-out repositories (111 stacks, random chance =0.9% ). Ablations use subsets of the 8-dimensional observable profile; the generic-harness subset contains 107 stacks.
Table 8: Cross-language exact-stack retrieval (16 stacks across 5 languages, random chance =6.25% ). Results pool 320 directional queries across all ten 3-vs-2 language splits using task-centered cosine retrieval.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
62.4
15.0
Python script edits (% of edits)
0.0%
100.0%
Shell redirection edits (% of edits)
48.4%
0.0%
Formal-test invocations
12.4
3.4
Appendix
Table 9: Workflow metrics for Case 1 ( pypsa-1653 ) . Values are task-level means across 5 runs.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
67.2
11.6
Exploration and read actions
25.4
4.6
Formal-test invocations
6.6
4.6
Workspace-inspection operations
5.0
5.4
Appendix
Table 10: Workflow metrics for Case 2 ( ytmusicapi-909 ) . Values are task-level means across 5 runs.
Metric
Claude Opus 5
GPT-5.6 Sol
Total tool actions
69.8
20.2
Formal-test invocations
9.2
3.6
Workspace-inspection operations
5.8
7.0
Actions after final counted write
8.8
4.4
Appendix
Table 11: Workflow metrics for Case 3 ( astronomy-commons__lsdb-1349 ) . Values are task-level means across 5 runs.
Trajectory Dimension
Low
Medium
xHigh
Difference [95% CI]
Total operational actions
31.3
35.5
54.4
+23.0[16.5,27.8]
Exploration and read actions
14.4
17.8
27.1
+12.7[11.0,14.5]
File-mutation events
5.4
6.1
8.3
+3.0[2.3,3.6]
Formal-test invocations
4.7
4.5
5.4
+0.7[−1.8,2.3]
Rollouts inspecting workspace changes after the final counted write (%)
12.7%
24.1%
66.8%
+54.1 pp[45.9,61.8]
Action-type switch count
19.0
23.0
34.5
+15.6[13.7,17.5]
Appendix
Table 12: Trajectory structural shifts across selected configured effort levels for Claude Opus 4.8 in Claude Code (110 common tasks, 2 rollouts per task). Formal-test volume counts contained invocations. Differences are computed before display rounding.