As language-model-based coding agents evolve their programs and reuse code from one another, safety failures can acquire a history that extends beyond the agent in which they are detected. An unsafe change may originate in one agent, propagate through intermediate recipients, and survive subsequent revisions before an audit reveals it. Understanding this history is important for identifying the introducing change and distinguishing inherited behavior from independently occurring failures. We study origin attribution in self-evolving multi-agent coding systems: recovering the agent and program version that introduced an observed unsafe behavior into the target's code lineage, or determining that it arose locally. This task is challenging because rewriting obscures textual similarity, independent occurrences confound temporal ordering, and intermediate recipients can resemble the source. We propose a method that combines behavioral fingerprint matching with historical localization and recursive tracing. It compares candidate implementations beyond the unsafe outcome, retains evidence from multiple historical versions, and follows matching candidates upstream. On 158 queries across 85 controlled scenarios, the method achieves 96.8% exact attribution accuracy, compared with 77.2% for exhaustive fingerprint matching without recursion. Component ablations and evaluation on 34 additional queries using the same method and hyperparameters demonstrate the complementary roles of multiple representatives and recursive tracing. Further checks extend the evaluation to approval-boundary and resource-owner errors. Together, these results show how behavioral and historical evidence can recover the origins of unsafe behavior through code revision and cross-agent propagation.
Figures & tables
Figure 1: Environment and attribution task (illustrative history). D⋅v2 propagates through A⋅v3 to C⋅v4 ; C⋅v5 is audited after further revision. B⋅v2 is an earlier independent occurrence. Orange adoption edges and filled introduction markers in the history are evaluation-only ground truth, unavailable to the method. Vertical position indicates schematic time; version indices are local to each agent.
Figure 2: Attribution on the example in Figure 1 . The matrix illustrates one replacement per candidate agent: B⋅v2 disagrees with the anchor beyond the unsafe outcome, while A⋅v3 and D⋅v2 match. Agreement ties favor the earlier timestamp, selecting D⋅v2 here. Recursive search retains the current attribution when no earlier match is found; an empty accepted set at the initial tracing level returns None .
Code
Condition
Change to the attribution problem
A
Semantic rewriting
Adopted code is rewritten while preserving the unsafe behavior.
B
Independent co-occurrence
An unrelated agent develops the behavior before the true source.
E
Multi-hop propagation
Code reaches the target through an intermediate recipient.
C
Unrelated adoption
Additional code transfers do not carry the unsafe behavior.
D
Evidence drift
The query targets the last flagged version instead of the first.
Table 1: Attribution conditions. A, B, C, and E describe histories; D changes the audited version.
Condition
n
Text
Bisect
Scan
Naive CF
Exh.
Ours
Star ( n=54 )
A: Rewrite
12
66.7
100.0
100.0
33.3
100.0
100.0
A+B: Rewrite + co-occur.
21
66.7
33.3
47.6
52.4
100.0
100.0
A+B+C: All three conditions
21
47.6
14.3
14.3
23.8
85.7
85.7
Chain ( n=33 )
E: Multi-hop
11
0.0
90.9
100.0
0.0
81.8
100.0
Table 2: Exact attribution accuracy (%) by topology and condition. Conditions follow Table 1 ; groups partition the 158 core queries. D re-audits their last flagged versions. Exh.: exhaustive fingerprint matching; —: not separately rerun for D. Blue highlights Ours; pale blue highlights the strongest baseline(s) per row.
Figure 3: Accuracy on 135 external-source queries, grouped by the number of attribution conditions. Table 2 reports the individual combinations.
Table 6
Multiple representatives
Recursive tracing
Star ( n=54 )
Chain ( n=33 )
Tree ( n=48 )
Negative ( n=23 )
Overall ( n=158 )
No
No
94.4
66.7
47.9
100.0
75.3
Yes
No
94.4
66.7
50.0
100.0
75.9
No
Yes
94.4
66.7
47.9
100.0
75.3
Yes
Yes
94.4
97.0
97.9
100.0
96.8
Table 5: Ablation of multiple representatives and recursive tracing (accuracy, %). Historical localization, fingerprint matching, and temporal selection remain enabled in all four configurations. Negative examples have no external source; a correct answer is None .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Structure / audit
Evaluated groups ( n )
Total
Attribution question
Star
A (12); A+B (21); A+B+C (21)
54
Rewritten direct adoption with independent occurrences and unrelated transfers.
Chain
E (11); E+B (11); E+B+A (11)
33
Recovering an upstream source through sequential adoption and revision.
Tree
E (14); E+B (16); E+B+A (18)
48
Recovering a source through branching adoption with the same ambiguity types.
Negative (no source)
Target-local introduction
23
Rejecting an external source despite benign code exchange.
Later audit (D)
Last rather than first flagged version
158
Stability across audited versions of the same histories.
Appendix
Table 6: Coverage of the fixed core evaluation. Counts refer to queries; D re-audits the same 158 queries at their last flagged versions. Condition definitions appear in Table 1 .
Serial evaluation; 30-second timeout; cascade evaluation off
Schedules
Recorded pre-event depths 1–5 rounds and background depths 1–3; intermediate relay evolution uses 3 rounds before republication
Appendix
Table 7: Core environment and attribution configuration.
Figure 4: Fingerprint matching versus unsafe-outcome reproduction on external-source and no-source queries. Localization and temporal selection are fixed.
Localization
Fingerprint
Temporal
External
No source
First
Last
–
–
–
15.6
91.3
26.6
22.8
–
Yes
–
14.8
100.0
27.2
21.5
Yes
–
–
15.6
0.0
13.3
13.3
–
–
Yes
60.0
91.3
64.6
44.3
Yes
Yes
–
15.6
100.0
27.8
27.8
–
Yes
Yes
50.4
100.0
57.6
34.8
Appendix
Table 8: Complete factorial results on the core queries (accuracy in percent). Localization, fingerprint matching, and temporal selection are enabled where marked Yes. The first and last columns refer to the audited target version.
Method
Query accuracy [95% CI]
Scene mean [95% CI]
Text
20.9 [12.8, 29.4]
16.9 [10.2, 24.3]
Text (function)
23.4 [14.8, 32.5]
18.4 [11.4, 26.3]
Bisect
42.4 [32.3, 52.9]
44.2 [34.2, 54.5]
Scan
42.4 [32.0, 53.1]
43.0 [32.9, 53.6]
Naive counterfactual
26.6 [17.2, 35.9]
19.4 [12.4, 27.0]
Ours
96.8 [92.1, 100.0]
97.1 [92.9, 100.0]
Appendix
Table 9: Core accuracy with uncertainty estimated by resampling all queries within each of the 85 scenes together (10,000 bootstrap draws; 95% percentile intervals). Scene means weight each scene equally.
Figure 5: Accuracy and mean program executions per query on the core set. Curves vary K for the two counterfactual methods; other methods use fixed configurations.
Method
K
Accuracy
Executions
Seconds
Text similarity (region level)
—
23.4
0.0
0.218
Pure bisection
—
42.4
12.9
0.028
Exhaustive scan
—
42.4
20.7
0.033
Naive counterfactual
1
27.2
1.0
0.317
2
27.2
2.0
0.317
3
26.6
2.9
0.325
Appendix
Table 10: Candidate-budget sweep on the 158 core queries. Accuracy is in percent; executions and seconds are per query. The text baseline here is region-level (23.4%); the main table and Figure 5 use the whole-program result (20.9%).
Condition
Scenes
n
Text
Bisect
Scan
Naive counterfactual
Ours
Chain: E+C
3/4
3
0
3
3
0
3
Tree: E+C
4/4
8
0
8
8
0
8
Star: B only
2/2
4
4
0
0
2
4
Star: C only
2/2
4
4
4
4
2
4
Overall
11/12
19
8
15
15
4
19
Appendix
Table 11: New GPT-generated coverage checks, separate from the 158-query core set. Entries are correct-query counts; n is the number of verified queries. Scene counts report verified/attempted. B: independent co-occurrence; C: unrelated adoption; E: multi-hop propagation. Blue highlights Ours; pale blue highlights the strongest baseline(s).
Utility inputs
Robustness inputs
Correct / 158
External / 135
No source / 23
0
0
105
105
0
0
7
104
104
0
4
0
130, 146, 140
121, 127, 121
9, 19, 19
4
7
128, 146, 140
119, 127, 121
9, 19, 19
8
0
151, 151, 152
130, 128, 131
21, 23, 21
8
7
151, 151, 152
130, 128, 131
21, 23, 21
Appendix
Table 12: Input-battery sensitivity on the 158 core queries. Triples report correct counts under three fixed input permutations; all rows retain the audit probe.
Fault
History
n
Full
No rec.
Exhaustive
Naive CF
Boundary
Direct
2
2
2
2
2
Multi-hop
2
0
0
0
0
Local
1
1
1
1
1
Field substitution
Direct
2
2
2
2
0
Multi-hop
2
2
0
0
0
Local
2
2
2
2
2
Appendix
Table 13: Additional fault-family checks. Entries are correct-query counts; each valid scene supplies one query. Full: complete method; No rec.: multiple representatives without recursion; Exhaustive: all-version fingerprint matching without recursion.
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.
Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. We believe this work has a range of applications to help developers work with and program coding agents, such as task-aware model routing, agent monitoring, and finer-grained cost analysis.
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains that make it hard to tell when the agent goes off track and why. Existing agent tracing analyses either focus on simple interaction or rely on small-scale manual inspection, which limits their scalability and usefulness for real coding workflows. We present CodeTracer, a tracing architecture that parses heterogeneous run artifacts through evolving extractors, reconstructs the full state transition history as a hierarchical trace tree with persistent memory, and performs failure onset localization to pinpoint the failure origin and its downstream chain. To enable systematic evaluation, we construct CodeTraceBench from a large collection of executed trajectories generated by four widely used code agent frameworks on diverse code tasks (e.g., bug fixing, refactoring, and terminal interaction), with supervision at both the stage and step levels for failure localization. Experiments show that CodeTracer substantially outperforms direct prompting and lightweight baselines, and that replaying its diagnostic signals consistently recovers originally failed runs under matched budgets. Our code and data are publicly available.
Han Li, Yifan Yao, Letian Zhu +13
Nanjing University · Alibaba Group · Renmin University of China +3
Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).