Audit the Scaffold, Not the Checkpoint: A Stationarity Dichotomy for Recursive Self-Improvement in Agentic Coding

Sebastian Bobadilla-Suarez; Bob Suh; Ryan Fortin

cs.LGSep 28, 2026

Audit the Scaffold, Not the Checkpoint: A Stationarity Dichotomy for Recursive Self-Improvement in Agentic Coding

Authors: Sebastian Bobadilla-Suarez, Bob Suh, Ryan Fortin

Organizations: OnCorps

Abstract

An auditor who checks whether a system's weights are frozen is checking the wrong thing. Our stationarity dichotomy says that iterative self-modification hits strict diminishing returns whenever the agent's reachable set of edits stays fixed, and can escape only if that set expands. Rewriting scaffolding (tools, verifiers, decomposition) expands what an agent reaches without touching a weight, so frozen weights buy an eventual ceiling but no stationarity along the way. The criterion also separates three regimes usually merged: search within a fixed class, test-time training that raises the ceiling itself, and scaffold rewriting between them. Audit the scaffold, not the checkpoint. The same ceiling binds sideways. Best-of- $k$ orchestration realizes the best worker's ceiling exactly: width buys rate, not budget. Re-consulting a fixed pool has a horizon computable in advance, decided by the pool alone, and the one arrangement that would beat it, a weighted vote, needs diversity real workers lack: on 30 same-family workers the failure overlap sits at its maximum, and a majority fails 23/55 (42%) of tasks. We obtain the criterion by reading refinement as gradient boosting on the residual error between draft and target, a patch or git diff, and then measuring where that reading breaks: patches compose instead of standing beside each other to be voted on, and failures overlap. What we measure is saturation. Per-round improvement decays toward zero on SWE-bench, and churn decays geometrically across 401 production sessions, a shape shared with a pre-AI human baseline that establishes the regime without identifying its cause. Both breaks are engineering choices rather than laws about code, so together they specify a harness worth building.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Boosting concept	Agentic coding analog	Concrete example
Input $X$	Specification	Prompts, GitHub tickets
Target $Y$	Optimal Code	The optimal “best” code for $X$
Current State $F_{t-1}(X)$	Current Draft	The code generated by previous agents
Residual $(Y-F_{t-1}(X))$	Code Edit	A git diff or patch
Weak learner $h_{t}$	Agent Patch Generator	LLM call proposing an edit
Loss Function $L(Y,\hat{Y})$	Structured Distance	Token edit distance, AST distance

Appendix

Table 1: Mapping between gradient boosting and agentic coding.

Effect	$F$	df	$p$	partial $\eta^{2}$	Cohen’s $f$
Model (Sonnet vs. Haiku)	53.46	(1, 54)	¡0.0001	0.497	0.99 (huge)
Feedback mode	0.50	(2, 108)	0.61	0.009	0.10 (small)
Model $\times$ mode	0.81	(2, 108)	0.45	0.015	0.12 (small)

Appendix

Table 2: Part B:

2\times 3

repeated-measures ANOVA (model

\times

feedback mode) on final-round quality,

K{=}55

tasks

\times

5 seeds. Summarized in § 5.1 . The model effect’s Cohen’s

f

carries 95% CI

[0.738,\,1.319]

, consistent with the near-complete tier separation in solve rates rather than with a computation artifact.

Commit floor	Pairs	AI higher	Sign $p$	Wilcoxon $p$	Median $\Delta\psi$
none	21	14	0.19	0.022	+0.127
${\geq}15$	14	10	0.18	0.011	+0.094
${\geq}30$	9	6	0.51	0.098	+0.061
${\geq}50$	7	5	0.45	0.11	+0.127

Appendix

Table 3: Part C: within-era AI vs. non-AI survivor ratio, paired within developer, at four minimum-commits-per-side floors. “AI higher” counts developers whose AI-assisted sessions show the higher mean non-pure survivor ratio. The sign test is null at every floor; Wilcoxon reaches significance only where the magnitude asymmetry is largest.

Result	Lean declarations
Thm. 10 , training-error bound	thm1_zero_one_loss_bound (the bound as stated); thm1_training_error_bound for the underlying exponential-loss identity $\mathrm{exp\,loss}=\prod_{t}Z_{t}$ , with z_decomposition , indicator_le_exp , exp_loss_pos
Thm. 13 , forward stagewise equivalence	thm4_fsam_optimal_alpha ; thm4_fsam_global_minimum , exp_half_log
Cor. 11 , exponential decay	cor2_exponential_decay ; z_le_sqrt_edge , sqrt_edge_le_exp , z_bound_from_edge
Thm. 2 , monotone-improvement refinement game	thm5_refinement_game ; thm5_saturation
Cor. 15 , high-probability refinement	corollary7_high_prob_refinement ; prob_inter_bound , sum_telescope . Verified against Mathlib’s measure-theoretic library ( IsProbabilityMeasure and the finite union bound)
Prop. 3 , stationarity dichotomy	prop8_saturation_forces_decay (part i), prop8_sustained_edge_diverges (part ii)

Appendix

Table 4: Core results and the Lean declarations that carry them (principal name first, then supporting lemmas and non-vacuity witnesses). All declarations live in lean_proofs/LeanProofs/ .

Result	Lean declarations
Prop. 5 , best-of- $k$ ceiling equality	prop15_tight_ceiling_orchestration (tight equality); prop15_orchestration_ceiling , prop15_orchestration_dominates for the weaker upper-bound-hypothesis version
Prop. 17 , weighted aggregation beats the weak-learning baseline	cor_soft_aggregation_beats_weak_baseline ; built on the pure-analysis lemma exists_ensemble_size_beating_weak_threshold
Prop. 18 , multi-round best-of- $k$	thm_width_refinement_game ; cor_width_dominates_best_worker , cor_width_saturation , with exists_width_orchestrator for non-vacuity
Prop. 6 , bounded overlap characterizes a correct vote	thm_overlap_iff_zero_error (characterization); cor_tight_overlap_iff ( $m^{\star}$ form), thm_bounded_overlap_zero_error (sufficient direction), thm_overlap_boundary_sharp with exists_boundary_overlap_family (sharpness at the boundary), cor_overlap_markov_bound (counting-only baseline)
Reweighted edge hypothesis via the density ratio	lem_bounded_reweighting_edge takes the concentration cap $\kappa$ as given; D_le_of_margin_bound and cor_rho_from_confidences derive it from logged confidences, and cor_reweighting_edge_from_confidences states the $\kappa$ -free conclusion. The sources spell $\kappa$ as rho and $M$ as B , the names they carried before this paper separated those glyphs
Prop. 19 , immediate reuse zeroes the edge	thm_self_reuse_zero_edge ; rests on the derived reweighting recursion D_succ_eq and reused_worker_err_eq , with exists_zero_edge_reuse_instance for non-vacuity

Appendix

Table 5: Orchestration and diversity results of § C and their Lean declarations. Every row is machine-checked; the one item that is not formalized is recorded in the note below the table.

Explore similar work

Jul 4, 2026cs.SE

Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these scaffoldings evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after scaffolding updates, yet consistently attribute them to the underlying model rather than the scaffolding itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the scaffolding's contribution. Unlike prior work that fixes the scaffolding and varies the model, we fix the model and vary only the scaffolding, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source scaffoldings (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

Oussama Ben Sghaier, Hao Li, Bram Adams +1

Queen’s University, Canada

Coding Agents Software Engineering Scaffolds Self-Evolving Agents Evolutionary Optimization Methods Opencode

Ask PDF

Sep 17, 2026cs.AI

Self Improvement via Fast Tree-search

Coding agents can recursively modify their own implementations, forming a loop of self-improvement. While prior work shows this can boost performance on coding benchmarks, existing approaches are costly and compute-intensive. We introduce a simple, sample-efficient self-improvement framework that significantly improves coding performance under strict budget constraints. We identify evaluation of candidate self-modifications as the main runtime bottleneck since prior approaches estimate their effectiveness by re-running a subset of benchmark tasks with the modified agent, which is time-consuming. We introduce Recursive Self Improvement via Fast Tree-search (SIFT), which augments these downstream task evaluations with an LLM-as-a-judge signal that performs pairwise comparisons between candidate patches, where the win-loss record is aggregated with a regularized Bradley-Terry model, and the resulting strength scores drive rank-based parent sampling inside a lightweight disaggregated tree search. Expensive downstream task evaluations are reserved only for the most promising nodes. Using a fully disaggregated tree search pipeline, the judge scores provide intermediate signal to guide exploration on promising candidate patches without being bottlenecked by slow evaluation runs. SIFT outperforms existing tree-search based self-evolution frameworks on the full Polyglot benchmark with significantly lower resource requirements in terms of CPU hours, wall clock time, and API cost.

Xinghong Fu, Aravinth Kulanthaivelu, Yutaro Yamada

Massachusetts Institute of Technology · Sakana AI

Recursive Self-Improvement Tree Search Coding Agents Bottlenecks

Ask PDF

Aug 2, 2026cs.CL

Same Task, Different Work: Prompt-Induced Waste in Coding Agents

Two prompts can request the same code change and produce the same correct patch, yet cause a coding agent to perform radically different kinds and amounts of work. We study this effect in a preregistered benchmark spanning 4,644 valid runs, 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses. The central finding is that prompt wording does not merely scale total effort; it changes where that effort is spent. Multiple approaches and deep thinking primarily inflate reasoning. Multiple approaches increases reasoning by 2.4x to 7.4x across all six open models and creates about three elaborated but discarded solution branches, while still yielding only one implemented solution and no success gain. Maximum certainty activates a different pathway: repeated verification propagates into extra test runs, tool calls, turns, latency, and context growth. Runs with high redundant verification cost 18x the clean-run median, execute 2.5x more tool calls, and take 3x longer, again without a success gradient. These mechanisms therefore have distinct cost carriers: some prompts are reasoning-heavy and token-borne, while others are tool-heavy and system-borne. Harness design amplifies both effects and changes cost per successful task by 5x to 30x in our setting. The findings survive a frozen holdout, paraphrase tests, a Kimi-K3 replication, and a first-party Claude Sonnet 5 study. In contrast, bounded-efficiency wording preserves diagnosis and final validation while avoiding the measured waste mechanisms. Prompt engineering for coding agents is therefore work design: it determines what the agent thinks through, what it executes, and when it stops.

Sarel Weinberger, Amir Hozez

PointFive

Coding Agents Large Reasoning Models Waste

Ask PDF