cs.AINov 3, 2026
SaveArtifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment
Organizations: 1Tianjin University · 2Independent Researcher · 3International Digital Economy Academy (IDEA)
Abstract
Self-evolving agents write interaction experience into artifacts and reuse them in later tasks. Most work supports such methods with three proxy signals: artifacts accumulate, behavior changes once they are loaded, and metrics rise. We call this signal-based evaluation. What these signals are meant to indicate, however, has not been defined. To fill this gap, we define capability formation from artifacts as five sequentially dependent properties, two of which form a checkpoint for capability formation, and introduce three-scale diagnosis, which judges these properties at the state, game, and lineage levels. In freestyle Gomoku, Reflexion, Voyager, and DGM evolve with the same base model and all show the three signals, yet each exhibits a distinct capability-formation break. Reflexion shows a legalization gain: its pass rate rises only because more of its moves are legal. Voyager shows a compositional override: components that are sound on their own combine to miss all 20 available one-move wins. DGM shows loss-only drift: after passing the checkpoint, later versions only lose passing states. These counterexamples show that signal-based evaluation cannot determine whether a capability has formed.
Figures & tables
Figure 1: Different reasoning requirements from the same board state.
Figure 2: The self-evolution loop: experience, evaluation, reflection, artifact updates, and feedback.
Figure 3: The three properties of the rule-based environment and the diagnostic scale each one supports: replayable states (state level), rule-judged task effects (state and game level), and fully recorded games and versions (game and lineage level).
| Scale | Properties and criteria | Why this works |
|---|---|---|
| State-level | Compare moves with and without the tested artifact on the same states. Property 2: move changes exceed sampling noise. Property 3: the pass rate among legal moves increases. | The states are identical and the tested artifact is the only difference, so any change in moves can only come from it. A rule program independently specifies the acceptable moves, distinguishing “changed for the better” from “merely changed.” |
| Game-level | Play complete games between the frozen system and a programmatic opponent. Property 4: complete five at every immediate-win state. | At an immediate-win state, “win in one move” is already established and the rules specify the unique correct move. Failing to complete five shows that another part of the system overturned this effect, so no control condition is needed. |
| Lineage-level | Compare adjacent versions state by state on the same states, along with their wins. Property 5: later versions retain prior effects while adding new ones; losing without adding, or barely changing behavior, fails. | Paired state-level comparison shows exactly what each version adds or loses, and whether its modifications still reach behavior. |
Table 1: Three-scale diagnosis and the properties it judges.
| Signals | Three-scale diagnosis | ||||
|---|---|---|---|---|---|
| Mechanism | Accumulation / behavior change / metric gain | State-level: Property 3 | Game-level: Property 4 | Lineage-level: Property 5 | Break |
| Reflexion | Six cells, seven memories each; moves change on 59.8%–78.2% of B states (noise: 46.7%); overall pass rate rises from 8.0% to 9.2%–10.3% on B and from 21.7% to 22.5%–27.5% on A. | Legal pass rate improves in none of 12 combinations; fails . | Not judged. | Not judged. | Property 3 |
| Voyager | Seven skills and a calling script; ablating components changes moves on 89–808 B states; open fours rise from 0/20 to 20/20. | round1 full : 459 newly passing and 73 lost on B; holds. | Completes five at 0/20 immediate-win states; fails . | Not judged. | Property 4 |
| DGM | Six accepted descendants; OB-3 changes moves on at least 661 B states relative to P0; B pass rate rises from 48.7% (P0) to 63.0% (OB-7); evolutionary score rises from 150.4 (OB-1) to 312.6 (OB-7). | OB-3 vs. P0: 522 newly passing and 139 lost; holds. | Completes five at 19/19; holds. | OB-5 and OB-6 only lose; OB-7 barely changes behavior; fails . | Property 5 |
Table 2: Signals and three-scale diagnosis results for the three mechanisms. Behavior change corresponds to Property 2; Property 1 is confirmed from call records for all three mechanisms (Section 5 ); Properties 3 and 4 must both hold.
Figure 4: Voyager against search_2 . Twenty independent games with different random seeds converge to four move sequences (G1–G4); numbers give move order. In every sequence, Voyager forms an open four whose two immediate winning points (blue stars) stay empty. At move 19, a conflict state, auto_player returns early on the defense branch and blocks one of search_2 ’s immediate winning points (orange ring); search_2 completes five at the other on move 20 (orange line). Immediate winning points are recomputed from the rules; Appendix E.1 lists them and the auto_player call order.
Figure 5: Reflexion on set B: change in each rate relative to empty memory, one dot per frozen memory (six in total; each memory keeps the same horizontal position in all three columns). All six raise the overall pass rate and the legality rate, yet all six lower the legal pass rate. The empty-memory rerun changes the three rates by , , and . Full results for both sets are in Table C.1 .
| Version | Wins | B pass rate | Newly passing / lost |
|---|---|---|---|
| P0 | 20/20 | 48.7% | — |
| OB-3 | 19/20 | 70.5% | / (vs. P0) |
| OB-5 | 9/20 | 64.6% | 0 / |
| OB-6 | 0/20 | 63.0% | 0 / |
| OB-7 | 0/20 | 63.0% | 1 / 0 |
Table 3: Wins against search_2 , B pass rate, and paired changes relative to the preceding version along the main DGM lineage.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Set | Legality rate | # Pass | ||
|---|---|---|---|---|---|
| Empty memory | A | 58.3% | 26 | 21.7% | 37.1% |
| Empty memory | B | 49.8% | 140 | 8.0% | 16.0% |
| Empty memory (second run) | A | 65.8% | 26 | 21.7% | 32.9% |
| Empty memory (second run) | B | 51.0% | 144 | 8.2% | 16.1% |
| outcome ( threat-aware ) | A | 75.8% | 33 | 27.5% | 36.3% |
| outcome ( threat-aware ) | B | 67.3% | 164 | 9.3% | 13.9% |
Table C.1: Reflexion call outcomes on sets A and B. Each state is queried once under each condition. The opponent cell associated with each memory condition is shown in parentheses.
| Condition | Move change | Net change | ||
|---|---|---|---|---|
| Empty memory (second run; noise baseline) | 46.7% | 0.73 | 16.0% 16.1% | |
| threat-aware , outcome | 61.3% | 0.0433 | 13.9% | |
| threat-aware , trace | 78.2% | 0.0037 | 12.3% | |
| rule-based , outcome | 62.9% | 0.0033 | 13.0% | |
| rule-based , trace | 72.8% | 0.0478 | 11.6% | |
| search_2 , outcome | 65.5% | 0.0031 | 13.8% |
Table C.2: Results for the six frozen memories on set B ( ). Net changes and -values are based on state-level paired outcomes using the exact McNemar test.
| Feedback | Set | Newly passing | Lost | Illegal pass | Legal non-pass pass | Net change among legal moves | |
|---|---|---|---|---|---|---|---|
| outcome | A | 8 | 1 | 0.039 | 6 | 2 | |
| outcome | B | 77 | 53 | 0.043 | 53 | 24 | |
| trace | A | 7 | 4 | 0.55 | 4 | 3 | |
| trace | B | 111 | 71 | 0.0037 | 72 | 39 |
Table C.3: State-level paired outcomes for the threat-aware memories and the sources of newly passing states. “Net change among legal moves” is computed only over states on which the model makes a legal move both with empty memory and with loaded memory; it equals the number of “legal non-pass pass” transitions minus the number of “pass legal non-pass” transitions.
| Version | search_2 wins/undecided/losses | A | B |
|---|---|---|---|
| P0 | 20/0/0 | 66.7% | 48.7% |
| OB-1 | 0/0/20 | 80.0% | 78.6% |
| OB-2 | 15/0/5 | 74.2% | 70.7% |
| OB-3 | 19/0/1 | 80.0% | 70.5% |
| OB-5 | 9/11/0 | 66.7% | 64.6% |
| OB-6 | 0/11/9 | 66.7% | 63.0% |
Table D.1: Terminal outcomes against search_2 and pass rates on sets A and B for the main DGM lineage. The denominators for A and B are 120 and 1,756, respectively.
| Configuration | Own open four formed | Wins | Mean length (moves) |
|---|---|---|---|
| full | 20/20 | 0/20 | 20.0 |
| -open3 (without create_open_three ) | 20/20 | 20/20 | 129.0 |
| -minimax (without minimax_search ) | 0/20 | 0/20 | 20.0 |
| round1 (first-round tactical skills only) | 0/20 | 0/20 | 18.0 |
Table E.1: Game outcomes against search_2 for the Voyager configurations (20 games per configuration).
| Configuration | A (/120) | B (/1,756) |
|---|---|---|
| full | 88 | 1,076 |
| -open3 (without create_open_three ) | 96; 8/8/0 | 1,345; 330/274/5 |
| -minimax (without minimax_search ) | 88; 0/0/0 | 1,084; 89/18/10 |
| round1 (first-round tactical skills only) | 80; 16/0/8 | 690; 808/73/459 |
Table E.2: Pass counts for the Voyager configurations on sets A and B, with paired changes relative to the full system (moves changed / newly passing / lost).
| Group | Records | Voyager’s immediate winning points | search_2 ’s immediate winning points | Move played |
|---|---|---|---|---|
| G1 | 4 | (4,10), (9,5) | (4,5), (9,10) | (4,5) |
| G2 | 5 | (4,10), (9,5) | (5,4), (10,9) | (5,4) |
| G3 | 5 | (6,4), (11,9) | (6,9), (11,4) | (6,9) |
| G4 | 6 | (4,6), (9,11) | (4,11), (9,6) | (4,11) |
Table E.3: The conflict state at move 19 in each Voyager trajectory group against search_2 . Immediate winning points are recomputed from the rules.
| Work | Artifact source | Pairing or intervention unit | Same-state action pairing | Primary readout | Cross-version |
|---|---|---|---|---|---|
| EvoPathBench ( Lin et al., 2026a ) | Evolutionary artifacts | Artifact on/off on held-out episodes | No (episode-level; trajectories diverge) | Episode risk-adjusted score difference | Yes (successive checkpoints) |
| Agent Plasticity ( Singh et al., 2026 ) | Evolutionary artifacts | Checkpoint-level games; LLM-observed annotations | Single case (appendix only) | Game score; usage annotations | Yes (successive checkpoints) |
| Lin et al. (2026b) | Evolutionary artifacts | Agent evolver cross-product | No | Pass-rate gain; loading and following rates | No |
| Misevolution ( Shao et al., 2026 ) | Hand-constructed (memory experiment) | Same query with and without memory | Yes | Safety label | Yes (before and after evolution) |
| SkillApt ( Guo et al., 2026 ) | Externally provided | Same task state with and without a skill | Yes (single answer) | Answer correctness; token cost | No |
| CMA ( Zhang et al., 2026c ) | Policy-internal history | Two histories crossed on a verified-identical input | Yes | Task branch selected by the action; pairwise reliability | No |
Table G.1: Design elements compared with the closest related work.
More in this topic
Self-Evolving Agents
Self-evolving agents improve their skills, memory, tools, policies, or environments by learning from interaction and evaluation feedback.
See all 441 papers