The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase

Yannick Roy

cs.SEMar 26, 2026

The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase

Authors: Yannick Roy

Abstract

Code production is now a commodity; the bottleneck is knowing what to build and proving it works. We present the Kitchen Loop, a framework for autonomous, self-evolving software built on a unified trust model: (1) a specification surface enumerating what the product claims to support; (2) 'As a User x 1000', where an LLM agent exercises that surface as a synthetic power user at 1,000x human cadence; (3) Unbeatable Tests, ground-truth verification the code author cannot fake; and (4) Drift Control, continuous quality measurement with automated pause gates. We validate across two production systems over 285+ iterations, producing 1,094+ merged pull requests with zero regressions detected by the regression oracle (methodology in Section 6.1). We observe emergent properties at scale: multi-iteration self-correction chains, autonomous infrastructure healing, and monotonically improving quality gates. The primitives are not new; our contribution is their composition into a production-tested system with the operational discipline that makes long-running autonomous evolution safe.

Figures & tables

Panel of Figure 1 — Figure 1: The Kitchen Loop: a six-phase autonomous improvement cycle.

Term	Definition
Specification surface	The enumerable set of capabilities a product claims to support — the input to coverage-exhaustion
Unbeatable test	A test that verifies outcomes against ground truth that the code author cannot fake
Regression oracle	A repeatable, bounded test that answers “is the system at least as good as before?”
Coverage-exhaustion mode	An operating regime where the agent systematically exercises every combination in the specification surface until coverage gaps approach zero
UAT gate	Adversarial user-acceptance testing by a fresh evaluator with zero implementation context
Drift control	Continuous measurement of quality trends with automated pause gates that halt the loop when metrics degrade

Table 2

Panel of Figure 2 — Figure 2: The unified trust model: each iteration flows through the full verification stack before proceeding.

	Task-Completion	Ralph Loops	Self-Improving	Kitchen Loop
Unit of work	Issue $\to$ Patch	Commit	Optimization step	Scenario $\to$ experience report
Stopping cond.	Issue resolved	Time/count	Metric plateau	Specification exhausted
Coverage	Reactive (given issues)	Random/greedy	Objective-guided	Three-tier (F/C/F)
Quality gate	CI passes	CI + typecheck	Evaluator function	4-layer + multi-model tribunal
Drift control	None	Basic (progress log)	Implicit (metrics)	Explicit (oracle + pause gates)
Human role	PR review	Manual oversight	None	Shared backlog, ticket parity

Table 2: Comparison of agentic software engineering approaches.

Regime	Unit of Work	Stopping Cond.	Verification Target	Failure Mode
Task-completion	Issue	Issue resolved	Patch correctness	Local fix / global mismatch
Metric-optimization	Objective step	Metric plateau	Proxy metric	Goodharting
Coverage-exhaustion	User scenario	Surface exhaustion	User-visible behavior	Drift / incomplete surface

Table 3: Three operating regimes for agentic software engineering.

Level	What It Tests	Speed	Trust Level	Written By
L1	Isolated logic, calculations	Fast (ms)	Low — proves logic, not integration	Agent or human
L2	API methods, adapters	Medium (s)	Medium — proves API contracts	Agent or human
L3	Full execution pipeline	Slow (min)	High — proves real-world behavior	Loop + agent
L4	Complete user journeys	Slowest	Highest — proves product works	Loop (AaU1000)

Table 8

Layer	Name	What It Catches	How
1	Compilation	Wrong params, missing config, type errors	Build/compile the action, assert success
2	Execution	Runtime failures, permission errors, timeouts	Execute against real env, assert success
3	Output Parsing	Wrong decoding, missing fields, malformed data	Parse output, assert expected data extracted
4	State Deltas	Silent failures, partial exec, wrong outcomes	Measure state before/after, assert exact deltas

Table 9

Tier	Name	Description	Expected	Observed (iter 163)
1	Obviously Bad	Glaring structural or factual errors any gate should catch	100%	100% (0 escapes / 163 iters)
2	Shadow	Factually true but stale, low-novelty, or below threshold	50–80%	100% (from 33% at iter 1, fixed iter 124)
3	Adversarial	Real data with wrong conclusions — fools deterministic and LLM checks	30–60%	100% (from 67% at iter 1)
4	Mixed T/F	Blend of valid and fabricated data in one signal — partial-failure detection	20–50%	100% (added iter $\sim$ 100, fixed iter 124)

Table 10

Verdict	Meaning	Action
PASS	Feature works from user perspective	Proceed, attach evidence to PR
PRODUCT_FAIL	Feature is broken	Keep ticket open, tag PR uat-failed
UAT_SPEC_FAIL	Test card is ambiguous or un-runnable	Log for process improvement, don’t block
EVAL_CHEAT_FAIL	Evaluator modified product files	Serious process issue, flag for human review

Table 11

Mode	Duration	Coverage	When to Use
Full	120-150 min	All scenarios, all platforms	Scheduled weekly
Quick	30-40 min	One scenario per platform	Every iteration

Table 12

Metric	Healthy Trend	Warning Signal
Test count	Growing or stable	Declining
Pass rate	Stable at ${>}$ 95%	Declining over 3+ iterations
Bug discovery rate	Declining (maturity)	Sudden spike (regression)
Oracle pass rate	100%	Any failure correlated with recent changes
Blocked combos	Declining	Growing without corresponding fix tickets
Canary escape rate	0% for Tier 1	Any Tier 1 escape

Table 13

Gate	Trigger	Response	Auto?
Regression Failure	Oracle pass rate drops below threshold	Pause after $N$ consecutive failures (default 3)	Semi
Canary Escape	Tier 1 canary passes quality gate	Warn operator	Advisory
Drift Threshold	Quality metric declines 3+ consecutive iters	Warn operator	Advisory
Backpressure	Open PRs exceed threshold	Enter drain mode (polish-only)	Yes
Starvation	Execute starved $N$ consecutive iters	Monitor-only, alert human	Yes

Table 14

Skill	Input	Output	Must Guarantee
backlog	Ticket state	Promoted + new scenario tickets	No duplicate tickets created
ideate	Scenario criteria	Experience report + scenario	Runs against real test env
triage	Experience report	Prioritized tickets w/ file pointers	Deduplicates against existing
execute	Ranked ticket list	Feature branch + PR per ticket	Every PR includes tests
regress	Full codebase state	Pass/fail report + drift metrics	Runs regression oracle

Table 16

Mode	Purpose	When to Use
Framework modes (domain-independent):
strategy (default)	Full cycle: ideate + execute + regress	Standard operation
user-only	Rapid ideation loops to fill the backlog	Backlog empty, need discovery
dev-only	Implementation-focused loops to drain the backlog	Backlog full, need throughput
drain (auto)	Polish-only with increased PR throughput	Auto-triggered by backpressure
regress-quick	Shortened regression: one scenario per platform	Every iteration (fast feedback)

Table 17

Field	Convention	Why
Labels	bug , feature , improvement , exploration	Picks bugs first, then balances
Priority	critical , high , medium , low	Respects human priority, no override
State	Backlog $\to$ Todo $\to$ In Progress $\to$ In Review $\to$ Done	PRs auto-link to tickets via title
Dependencies	blocks / blockedBy	Skips blocked tickets automatically

Table 18

Dimension	Value
Total discussions	23 unique discussions across 3 codebases
Discussion types	Architecture/PRDs (10), Code reviews (7), Strategy (5), Exploration (1)
Typical participants	3 models (Gemini, Codex, Claude); 1 discussion used 2 models
Typical rounds	3 (range: 1–10)
Convergence	2/23 formally converged; 21/23 hit max rounds
Conclusions	100% “Agreed” — zero “Agreed to Disagree”

Table 20

Criterion	Measured	Paper’s Ideal	Assessment
Disagree. Collapse Rate (DCR)	$\sim$ 0% (all converge)	Low but non-zero	Red flag — universal convergence implausible
Sycophancy Score (SS)	35–50	${<}$ 20	Moderate-high — mitigated by code grounding
Negative Agreement Rate (NAR)	$\sim$ 25%	${<}$ 10%	Moderate — Claude concedes readily
Evidence Quality	85/100	High	Strong — code grounding is best feature
Perspective Diversity	60/100	High heterogeneity	Moderate — good diversity, rigid roles
Round Efficiency	$\sim$ 50% productive	${>}$ 80%	Low — 2–3 wasted rounds per discussion

Table 21

Figure 3: The Almanak SDK repository: a production DeFi strategy framework supporting 14 chains and 30+ protocol connectors.

Dimension	Count
Supported chains	14	(13 EVM + Solana)
Protocol connectors	30+	(13 core + Aave, Compound, GMX, Morpho, Lido, …)
Intent types	21	(Swap, LP Open/Close, Borrow, Supply, Perp, Stake, …)
Cross-combinations	$\sim$ 1,000	(chain $\times$ protocol $\times$ intent)

Table 23

Layer	Implementation
Compilation	compiler.compile(intent) $\rightarrow$ assert status SUCCESS
Execution	orchestrator.execute(bundle) $\rightarrow$ assert success on-chain
Receipt Parsing	Protocol-specific parser extracts amounts, position IDs
State Deltas	get_token_balance() BEFORE and AFTER, assert exact expected changes

Table 24

Metric	Value
Loop iterations completed	122+
Merged pull requests	728+
Unique tickets resolved	350+
Lines of code added	$\sim$ 250,000
Unit tests (start)	$\sim$ 6,400
Unit tests (current)	10,913

Table 25

Representative Finding	Impact	Iter
API method documented but not exposed to users	AttributeError on first call	12
Gas price cap correct for one chain, blocks all txs	All strategies on one chain fail silently	18
Protocol router config missing for entire protocol	Blocks swap compilation everywhere	50
Native token symbol missing for two chains	Blocks ALL strategies on those chains	51

Table 26

Finding	Severity	Impact	Iter
Pendle PT sell uses 1:1 exchange rate, but PT trades at $\scriptstyle\sim$ 19% discount	HIGH	Every PT sell at normal slippage (0.5-10%) would revert with INSUFFICIENT_TOKEN_OUT	89
BTC.b completely missing from Avalanche token resolver	HIGH	Any BTC.b strategy on Avalanche fails at compilation	83
Uniswap V4 adapter silently falls back to 18 decimals	MEDIUM	Incorrect balance calculations for non-18-decimal tokens	80-81

Table 27

Dimension	Count
Detection agents	46+
Signal types	77	(all verified)
Verifier families	22
Cross-combinations	$\sim$ 1,078	(agent $\times$ signal type)

Table 28

Layer	Name	Method	What It Catches
L1	Structural	Schema validation	Format errors, invalid enums, missing metadata
L2	Factual	API cross-reference (DefiLlama, Snapshot, CoinGecko, GeckoTerminal, on-chain RPCs)	False claims, wrong numbers
L3	Temporal	Dedup + timestamp analysis	Stale signals, zombie duplicates
L4	Cognitive	LLM tribunal (3 models)	Bad reasoning, wrong conclusions from correct data

Table 29

Tier	Canaries	Catch Rate (iter 1)	Catch Rate (iter 163)
Tier 1 (Obviously Bad)	6	100%	100% (zero escapes across 163 iters)
Tier 2 (Shadow)	5	33%	100% (fixed iter 124)
Tier 3 (Adversarial)	5	67%	100%
Tier 4 (Mixed True/False)	5	N/A (added $\sim$ iter 100)	100% (fixed iter 124)
API Degradation	3	N/A (added later)	100% (3/3)

Table 30

Metric	Value
Loop iterations completed	163
Tickets created	200+
Pull requests merged	366
Signal type coverage	77/77 (100%) — up from 33/36 at iter 1
Verifier coverage	100% (22 verifier families) — every signal type has a dedicated L2 factual verifier
Non-functional agents identified	18 (managed via exclusion list)

Table 31

Metric	Iter 1-10	Iter 50-60	Iter 128-135	Iter 163
L1 pass rate	85-100%	100%	100%	100%
L2 pass rate	76-91%	90-97%	100%	100%
L2 unverifiable	9-31%	3-10%	0%	0%
L3 pass rate	85-100%	92-100%	100%	100%
Verifier coverage	$\scriptstyle\sim$ 92% (33/36)	$\scriptstyle\sim$ 100%	100% (72/72)	100% (77/77)
PR merge rate	$\scriptstyle\sim$ 0%	$\scriptstyle\sim$ 85%	100%	97%

Table 32

Iter	What Happened
130	Ideate discovers OnChainSentinel hangs the pipeline via WebSocket constructor blocking. Creates ticket.
130	Execute ships PR: 5-second timeout guard around getNetwork() + adds agent to exclusion list.
131	Ideate retests. Discovers the fix was incomplete — the WebSocketProvider constructor itself blocks before getNetwork() is ever called. Reopens ticket.
131	Execute ships PR: wraps entire WS provider init in a single timeout with provider cleanup on expiry. Adds destroy-on-timeout test.
131	Regress confirms: build passes, 27 signals produced, L1/L2/L3 100%, no regressions.

Table 33

Classification	Example	Response
Non-functional (confirmed)	RWARiskAgent: hardcoded stub data (backingRatio=1.0)	Add to exclusion list immediately
Non-functional (suspected)	LPRotationAgent: 0 signals, 1st observation	Monitor — needs 2+ confirmations before excluding
Cold-start	DevActivityAgent: needs warm baseline from prior runs	Don’t exclude — works in scheduled mode
Timing-dependent	BribeTrackingAgent: no active vote rounds	Don’t exclude — market conditions, not a bug
Deprecated	YieldAgent: converted to return-empty stub	Add to exclusion list

Table 34

Finding	Fix	Impact
RESULTS_DIR undefined in pr-manager.sh	Grep prep log instead	Gate rejection memory completely non-functional
gh --jq --arg invalid syntax	Pipe to jq --arg	Shell command silently failing
Missing iter 127 row in loop-state.md	Backfill + verification step	History gap invisible to drift detection
synthesisPipelineDead false positive	localDev flag for drift suppression	9 iterations of wasted CRITICAL escalation
Deployment drift (local main stale)	Pre-flight detection	All merged fixes untested for 5+ iterations
NON_FUNCTIONAL_AGENTS set out of sync	Sync with confirmed findings + expose via API	Ideate kept selecting dead agents

Table 35

Phase	Strategy Framework	Signal Platform	Speedup
Ideate	10-15 min	1-2 min	$\scriptstyle\sim$ 8x
Triage	5-8 min	<1 min	$\scriptstyle\sim$ 8x
Execute	25-60 min	10-30 min	$\scriptstyle\sim$ 2x
Regress	40-150 min	2-3 min	$\scriptstyle\sim$ 30x
Total	80-230 min	$\scriptstyle\sim$ 15-35 min *	$\scriptstyle\sim$ 5-15x

Table 36

Cost Component	Monthly Cost	Notes
Claude Code Max (20x plan)	$200	Primary agent for all 6 phases — unlimited usage within plan
Codex (subscription)	$20	External feasibility checker + tribunal reviewer
Gemini (subscription)	$20	Tribunal reviewer for multi-model audits
CodeRabbit	$\scriptstyle\sim$ $15	Automated PR code review
Anvil / Foundry	$0	Open-source, runs locally
External data APIs	$\scriptstyle\sim$ $0-50	CoinGecko, DefiLlama (free tiers sufficient)

Table 37

Metric	Kitchen Loop	Single Engineer	Ratio
Monthly cost	$\scriptstyle\sim$ $350	$\scriptstyle\sim$ $15,000	$\scriptstyle\sim$ 43x cheaper
PRs merged/month	600+	$\scriptstyle\sim$ 15-25	$\scriptstyle\sim$ 30x more
Coverage scenarios tested/month	150+	$\scriptstyle\sim$ 10-20	$\scriptstyle\sim$ 10x more
Cost per merged PR	$\scriptstyle\sim$ $0.38	$600-1,000	$\scriptstyle\sim$ 1,800x cheaper

Table 38

Metric	Start (iter 1)	Current (iter 122)	Growth Rate
Unit tests	$\scriptstyle\sim$ 6,400	10,913	+70% over 122 iterations
Full regression time	$\scriptstyle\sim$ 90 min	$\scriptstyle\sim$ 150 min	$\scriptstyle\sim$ 0.5 min/iteration growth
Quick regression time	$\scriptstyle\sim$ 25 min	$\scriptstyle\sim$ 40 min	$\scriptstyle\sim$ 0.12 min/iteration growth
Demo strategies	13	62	$\scriptstyle\sim$ 0.40/iteration
Incubating strategies	0	183	$\scriptstyle\sim$ 1.5/iteration

Table 39

Concern	Record	Details
Production downtime	0 incidents	Loop operates on isolated branches + test environments (Anvil forks). No direct production access.
Security incidents	0 incidents	Loop has no access to production keys, wallets, or user data. Test wallets use Anvil defaults.
Data loss	0 incidents	Git worktree isolation prevents branch contamination. All changes are reversible PRs.
Regressions shipped to main	0	Quality gates (multi-model review + regression oracle) catch issues before merge.
Cost overruns	0	Monthly tooling cost is bounded and predictable ( $\scriptstyle\sim$ $305-405/month across both systems,$ \scriptstyle\sim1.50/iteration).

Table 40

Limitation	Impact	Mitigation
Anvil-only testing	Loop has not been exercised with real mainnet transactions	Mainnet mode exists but requires explicit operator approval + real gas
API key dependency	Some scenarios require external API keys not in standard config	Produces PASS(caveat) instead of clean PASS — a known quality ceiling
Single-threaded execution	Gateway port constraint limits to sequential strategy runs	Future: dynamic port allocation for parallel execution
PR merge automation fragility	The Polish phase has been the most failure-prone component	Each failure mode has been fixed through the self-improvement cycle
Cooldown phantom failures (signal platform)	Agents with cooldown periods inflate dead-agent counts	Distinguishing “cooldown active” from “agent broken” is tracked as an open ticket

Table 41

Concern	AI (Kitchen Loop)	Human
Coverage	Exhaustive — 1000x the scenarios a human would test	Strategic — focuses on what matters most given user context
Bug discovery	Bottom-up — finds what’s broken by trying everything	Top-down — knows what users are complaining about
Ticket writing	Precise, file-level, with reproduction steps	Context-rich, business-aware
Implementation	Fast, consistent, follows established patterns	Creative, architectural, can break patterns when needed
Code review	Multi-model parallel tribunals	Domain expertise, security intuition
Backlog curation	Deduplication, severity assessment, dependency graphs	Strategic priority, business value, external signals

Table 42

Domain	Specification Surface	Regression Oracle	Example Adaptation
Web Application	Pages x User Flows x Browsers	Browser automation + visual regression (Playwright, Cypress)	Ideate generates user journeys; Regress runs screenshot diffing against known-good baselines
ML Pipeline	Models x Datasets x Metrics	W&B / MLflow run comparison + statistical tests	Ideate trains model variants; Regress compares against baseline metrics with significance tests
Smart Contracts	Functions x Chains x Edge Cases	Foundry/Anvil fork execution + invariant checks	Ideate writes fuzz test scenarios; Regress runs invariant test suites
Backend API	Endpoints x Methods x Auth Roles	Contract testing + live traffic shadow comparison	Ideate generates API call sequences; Regress replays against shadow traffic
Mobile App	Screens x Gestures x Devices	Appium automation + visual regression	Ideate scripts user flows; Regress runs on device farm
Compiler / Language	Grammar x Optimizations x Targets	Test suite execution + benchmark comparison	Ideate generates programs exercising edge cases; Regress runs conformance suite

Table 43

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	Chain	Swap	LP Open	LP Close	Supply	Borrow	Repay	Withdraw
Aerodrome	Base	P0	P0	P0	-	-	-	-
TraderJoe V2	Avalanche	P0	P0	P0	-	-	-	-
Uniswap V3	Ethereum	P1	P1	P1	-	-	-	-
Uniswap V3	Arbitrum	P1	P1	P1	-	-	-	-
Uniswap V3	Base	P1	P1	P1	-	-	-	-
PancakeSwap V3	BSC	P1	-	-	-	-	-	-

Appendix

Table 45

Verifier Family	Signal Types Covered	Data Source
Protocol TVL	Discovery, TVL migration, undervalued protocol	DeFi analytics API
Pool Yield	Yield opportunity, LP opportunity, cross-DEX arbitrage	Pool analytics API
Lending Rate	Lending arbitrage, utilization spike	Lending analytics API
Governance	Governance catalyst, proposal tracking	Governance API
Exploit	Exploit detection, exploit warning	Security feeds
Price Feed	Whale alert, depeg risk	Price oracle API

Appendix

Table 46

Skill	Phase	Input	Output	Domain-Specific?
backlog	Backlog	Ticket state	Promoted tickets	Partially (labels)
ideate	Ideate	Scenario + spec	Report + scenario	Yes (defines “usage”)
triage	Triage	Experience report	Labeled, deduped tickets	Partially (taxonomy)
execute	Execute	Ranked tickets	Branch + PR + tests	Partially (test patterns)
pr-manager	Polish	Open PR list	Reviewed, CI-passing PRs	No
regress	Regress	Codebase state	Pass/fail + drift	Yes (defines “oracle”)

Appendix

Table 47

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 270 papers

Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Sep 1, 2026Haoyang Yan, Min-le Su, Hangfan Zhang +6Agent Harness Optimization Self-Improving Agents
Shanghai Artificial Intelligence Laboratory
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
May 7, 2026Jingjie Ning, Xiaochuan Li, Ji Zeng +2Multi-Agent Systems Self-Improving Agents
School of Computer Science, Carnegie Mellon University
Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework
Jul 13, 2026Aditya Aggarwal, Nahid Farhady GhalatyLLM Self-Correction Continual Learning for LLM Agents
Microsoft

Sep 1, 2026J/K move · Enter open · S save

Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li+5

Shanghai Artificial Intelligence Laboratory

Agent Harness Optimization Self-Improving Agents Long-Horizon LLM Agents LLM Agent Harnesses

Open paper AskPDF arXiv

This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/

Abstract

Software Engineering

Explore similar work

Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework