Audit the Scaffold, Not the Checkpoint: A Stationarity Dichotomy for Recursive Self-Improvement in Agentic Coding

Sebastian Bobadilla-Suarez; Bob Suh; Ryan Fortin

cs.LGSep 28, 2026

Audit the Scaffold, Not the Checkpoint: A Stationarity Dichotomy for Recursive Self-Improvement in Agentic Coding

Authors: Sebastian Bobadilla-Suarez, Bob Suh, Ryan Fortin

Organizations: OnCorps

Abstract

An auditor who checks whether a system's weights are frozen is checking the wrong thing. Our stationarity dichotomy says that iterative self-modification hits strict diminishing returns whenever the agent's reachable set of edits stays fixed, and can escape only if that set expands. Rewriting scaffolding (tools, verifiers, decomposition) expands what an agent reaches without touching a weight, so frozen weights buy an eventual ceiling but no stationarity along the way. The criterion also separates three regimes usually merged: search within a fixed class, test-time training that raises the ceiling itself, and scaffold rewriting between them. Audit the scaffold, not the checkpoint. The same ceiling binds sideways. Best-of- $k$ orchestration realizes the best worker's ceiling exactly: width buys rate, not budget. Re-consulting a fixed pool has a horizon computable in advance, decided by the pool alone, and the one arrangement that would beat it, a weighted vote, needs diversity real workers lack: on 30 same-family workers the failure overlap sits at its maximum, and a majority fails 23/55 (42%) of tasks. We obtain the criterion by reading refinement as gradient boosting on the residual error between draft and target, a patch or git diff, and then measuring where that reading breaks: patches compose instead of standing beside each other to be voted on, and failures overlap. What we measure is saturation. Per-round improvement decays toward zero on SWE-bench, and churn decays geometrically across 401 production sessions, a shape shared with a pre-AI human baseline that establishes the regime without identifying its cause. Both breaks are engineering choices rather than laws about code, so together they specify a harness worth building.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Boosting concept	Agentic coding analog	Concrete example
Input $X$	Specification	Prompts, GitHub tickets
Target $Y$	Optimal Code	The optimal “best” code for $X$
Current State $F_{t-1}(X)$	Current Draft	The code generated by previous agents
Residual $(Y-F_{t-1}(X))$	Code Edit	A git diff or patch
Weak learner $h_{t}$	Agent Patch Generator	LLM call proposing an edit
Loss Function $L(Y,\hat{Y})$	Structured Distance	Token edit distance, AST distance

Appendix

Table 1: Mapping between gradient boosting and agentic coding.

Effect	$F$	df	$p$	partial $\eta^{2}$	Cohen’s $f$
Model (Sonnet vs. Haiku)	53.46	(1, 54)	¡0.0001	0.497	0.99 (huge)
Feedback mode	0.50	(2, 108)	0.61	0.009	0.10 (small)
Model $\times$ mode	0.81	(2, 108)	0.45	0.015	0.12 (small)

Appendix

Table 2: Part B:

2\times 3

repeated-measures ANOVA (model

\times

feedback mode) on final-round quality,

K{=}55

tasks

\times

5 seeds. Summarized in § 5.1 . The model effect’s Cohen’s

f

carries 95% CI

[0.738,\,1.319]

, consistent with the near-complete tier separation in solve rates rather than with a computation artifact.

Commit floor	Pairs	AI higher	Sign $p$	Wilcoxon $p$	Median $\Delta\psi$
none	21	14	0.19	0.022	+0.127
${\geq}15$	14	10	0.18	0.011	+0.094
${\geq}30$	9	6	0.51	0.098	+0.061
${\geq}50$	7	5	0.45	0.11	+0.127

Appendix

Table 3: Part C: within-era AI vs. non-AI survivor ratio, paired within developer, at four minimum-commits-per-side floors. “AI higher” counts developers whose AI-assisted sessions show the higher mean non-pure survivor ratio. The sign test is null at every floor; Wilcoxon reaches significance only where the magnitude asymmetry is largest.

Result	Lean declarations
Thm. 10 , training-error bound	thm1_zero_one_loss_bound (the bound as stated); thm1_training_error_bound for the underlying exponential-loss identity $\mathrm{exp\,loss}=\prod_{t}Z_{t}$ , with z_decomposition , indicator_le_exp , exp_loss_pos
Thm. 13 , forward stagewise equivalence	thm4_fsam_optimal_alpha ; thm4_fsam_global_minimum , exp_half_log
Cor. 11 , exponential decay	cor2_exponential_decay ; z_le_sqrt_edge , sqrt_edge_le_exp , z_bound_from_edge
Thm. 2 , monotone-improvement refinement game	thm5_refinement_game ; thm5_saturation
Cor. 15 , high-probability refinement	corollary7_high_prob_refinement ; prob_inter_bound , sum_telescope . Verified against Mathlib’s measure-theoretic library ( IsProbabilityMeasure and the finite union bound)
Prop. 3 , stationarity dichotomy	prop8_saturation_forces_decay (part i), prop8_sustained_edge_diverges (part ii)

Appendix

Table 4: Core results and the Lean declarations that carry them (principal name first, then supporting lemmas and non-vacuity witnesses). All declarations live in lean_proofs/LeanProofs/ .

Result	Lean declarations
Prop. 5 , best-of- $k$ ceiling equality	prop15_tight_ceiling_orchestration (tight equality); prop15_orchestration_ceiling , prop15_orchestration_dominates for the weaker upper-bound-hypothesis version
Prop. 17 , weighted aggregation beats the weak-learning baseline	cor_soft_aggregation_beats_weak_baseline ; built on the pure-analysis lemma exists_ensemble_size_beating_weak_threshold
Prop. 18 , multi-round best-of- $k$	thm_width_refinement_game ; cor_width_dominates_best_worker , cor_width_saturation , with exists_width_orchestrator for non-vacuity
Prop. 6 , bounded overlap characterizes a correct vote	thm_overlap_iff_zero_error (characterization); cor_tight_overlap_iff ( $m^{\star}$ form), thm_bounded_overlap_zero_error (sufficient direction), thm_overlap_boundary_sharp with exists_boundary_overlap_family (sharpness at the boundary), cor_overlap_markov_bound (counting-only baseline)
Reweighted edge hypothesis via the density ratio	lem_bounded_reweighting_edge takes the concentration cap $\kappa$ as given; D_le_of_margin_bound and cor_rho_from_confidences derive it from logged confidences, and cor_reweighting_edge_from_confidences states the $\kappa$ -free conclusion. The sources spell $\kappa$ as rho and $M$ as B , the names they carried before this paper separated those glyphs
Prop. 19 , immediate reuse zeroes the edge	thm_self_reuse_zero_edge ; rests on the derived reweighting recursion D_succ_eq and reused_worker_err_eq , with exists_zero_edge_reuse_instance for non-vacuity

Appendix

Table 5: Orchestration and diversity results of § C and their Lean declarations. Every row is machine-checked; the one item that is not formalized is recorded in the note below the table.

Explore similar work

Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality
Jul 4, 2026Oussama Ben Sghaier, Hao Li, Bram Adams +1Coding Agents Software Engineering
Queen’s University, Canada
Self Improvement via Fast Tree-search
Sep 17, 2026Xinghong Fu, Aravinth Kulanthaivelu, Yutaro YamadaRecursive Self-Improvement Tree Search
Massachusetts Institute of Technology · Sakana AI
Same Task, Different Work: Prompt-Induced Waste in Coding Agents
Aug 2, 2026Sarel Weinberger, Amir HozezCoding Agents Large Reasoning Models
PointFive

Jul 4, 2026, cs.SEJ/K move · Enter open · S save

Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

Oussama Ben Sghaier, Hao Li, Bram Adams, Ahmed E. Hassan

Queen’s University, Canada

Coding Agents Software Engineering Scaffolds Self-Evolving Agents Evolutionary Optimization Methods Opencode

Open paper AskPDF arXiv

Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these scaffoldings evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after scaffolding updates, yet consistently attribute them to the underlying model rather than the scaffolding itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the scaffolding's contribution. Unlike prior work that fixes the scaffolding and varies the model, we fix the model and vary only the scaffolding, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source scaffoldings (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

Abstract

Explore similar work

Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

Self Improvement via Fast Tree-search

Same Task, Different Work: Prompt-Induced Waste in Coding Agents