SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning

Zhoumin Xie

cs.LGAug 17, 2026

SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning

Authors: Zhoumin Xie

Organizations: Independent Researcher

Abstract

Today, a neural system is almost always used in two phases -- trained, then deployed -- and in that regime it freezes twice: training ends, and the topology itself was never a degree of freedom. We take the opposite premise as an axiom -- total plasticity: no part of a model, including its structure, is ever frozen -- and derive the governance a lifelong learner then requires. The design's target regime is continual, in-service learning: a long-lived model on a non-stationary stream, whose stability comes from governance rather than immobility and whose capacity follows demand. The result is a growable soft model: an algebra of structural operators (width, hierarchy, composition, input interface, grown cycles, attention heads), each exact at application, budgeted, and audited, with adoption decided solely by a held-out reality gate that treats parametric and structural change uniformly. A complete from-scratch system realizes the whole account; its factory surface is operated end-to-end by a production LLM. Two conclusions follow from the axiom by construction: stability under lifelong change becomes an audit property of the lifecycle, and structure that follows demand removes the silent cap a fixed topology places on later capability where the capacity floor binds. A third is measured: in the worlds where this was measured, the marginal value of new capacity was unobservable before adoption, so workable growth governance took its ex-post form. The same governance extends to evaluative signals, and the core method is evaluated on standard continual-learning benchmarks, where governed growth preserves the ability to keep learning along long task sequences. A pre-registered experimental program adjudicates the mechanism and value claims on the tested problems and reports its failures at full prominence; the map -- positive and negative -- is the contribution.

Figures & tables

Panel of Figure 1 — Figure 1: The recursive substrate (Definition 1 ). A composite node adds its inner network’s output to its own activation; inner networks read the same input and may themselves contain composite nodes — structure is grown, not designed.

Figure 2: Schematic (not data): the operator algebra growing a topology over time. New elements (accent) enter with zero-weight dashed connections — silent at application, so

f

is unchanged (Props. 1–3, 5) — and earn participation through training. An inner network reads the same input

\hat{x}

and adds its vector output to the scope’s hidden state through its zero-born attachment at the

+

junction (

h=\varphi(a)+g_{j}(\hat{x})A_{j}

); the

\delta

panel appends a zero-initialized composition block in series on the scope’s hidden state (Prop. 5 ). The final panel shows an inner node refined in turn and the block matured (tree height 3, layer depth 2). Every parameter remains trainable from step one; adoption is always by the gate.

Operator	Axis	Exactness at application	Typical trigger
$\rho$ (refine)	hierarchy	inner net enters at zero (Prop. 2 )	high $u_{j}$ at a node
$\omega$ (widen)	width	zero output-weight block (Prop. 3 )	scope-wide saturation
$\sigma$ (interface)	inputs	zero input column, any value (Prop. 1 )	new observable arrives
$\delta$ (deepen)	composition	block enters at zero (Prop. 5 )	delayed-payoff slope signal (§ 4.3 )
$\Phi$ (re-found)	whole model	candidate; takeover gated	inversion / disposal / flat LP

Table 1: The operator algebra. Every operator is budgeted (parameter/depth caps as policy; refusals are logged outcomes), audited (lineage events), and closed under exact removal (each grown structure carries its inverse); adoption is always by the gate of Algorithm 2 . The additive family (

\omega

\rho

\sigma

, per-head growth, skips) leaves the composition-degree invariant unchanged;

\delta

alone raises it — the boundary that separates multiscale organization from true deepening. The loop operator

\lambda

(§ 5 ) and the per-head attention operators (§ 6 ) join these five under the identical contract — the abstract’s operator list spans all of them. Measured exactness values: App. A .

Figure 3: The two growth directions on one pipeline. Additive operators (

\rho

\omega

) add terms to the scope’s hidden state

H_{0}

; the compositional operator (

\delta

) appends zero-initialized blocks after it. Both directions enter exactly (dashed: silent at application); cross-node aggregation remains a sum, and layer depth grows only inside the scope. Inset: depth is a grown, non-uniform field — each branch evolves its tree height and each scope its layer depth independently.

Figure 4: Direction selection: predict first (under the predictability certificate), probe only to verify (asymptote-read curves), adopt only by the gate. The probe READING is the measured part: single-horizon prices carry no axis signal; the two-horizon slope does (§ 13.5 ). The widen-only mode bypasses the machinery above the replaceable-parts layer (§ 4.4 ).

	Grown cycle ( $\lambda$ )	Self-processing unit
Solved quantity	scope state $z^{*}$	newborn weights
When it runs	every pass on the scope	newborn windows only
At serving	iterates (capped)	absent by construction
Guarantee	enforced contraction	bounded loop; collapse-guarded objective
Cost bound	$K_{\max}$ per pass	$S_{\max}$ per step
Entry	exact ( $L_{\mathrm{out}}=0$ )	n/a — refines weights; adoption gated

Table 2: One principle, two realizations. Both are local solving loops embedded in the global pass; they solve different quantities on different clocks under the same governance contract, and each is enabled by policy, never by fiat.

Panel of Figure 5 — Figure 5: The governed lifecycle. Speculation (learning and all growth operators) is free in the working state; serving changes only through the gate; the held-out stream is quarantined from all training. Teaching by the brain can therefore never degrade the served model (threat model: § 9 ).

Regime	Net effect of growth	Evidence
Capacity binds	higher time-axis value	quadratic staged world: reward-only 3/5 (mean gap $+10.8$ ), alternating regime 3/5 at its registered criterion (mean $+1.1$ ); CartPole after the dynamics change: 3/5 paired lives with one tie (Figure 7 ); LunarLander, gate-level merit only: grown candidates pass the gate in 8/10 lives, trained clones in 0/10 (long-run net effect unresolved, § 8.5 )
Capacity suffices (capacity-sufficient staged world), designed alternating teach–reward regime	cost-neutral	mean AUC gap $-1.9$ (ties dominate; the registered value line missed, 0/5 — the wash itself is the recorded outcome, per the registered clause)
Capacity suffices (same world), reward-only operation (outside the designed regime)	transient cost	mean AUC gap $-33.9$ ; the designed regime removes 94% of it

Table 3: The three-regime law for structural growth under evaluative learning. “Time-axis value” is area under the per-episode score curve over the life; gaps are grown-arm minus protocol-matched twin, and score scales are per-world — magnitudes are not comparable across rows. Registered criteria: at least 3 of 5 per five-life comparison; for the LunarLander gate comparison, an adoption in at least 6 of 10 lives.

Figure 6: The capacity-sufficient staged world (distinct from the capacity-binding quadratic staged world of Table 3 ), designed alternating teach–reward regime: mean score curves over five seeds (bands span the seeds). The growing model and its protocol-matched twin track each other — the cost-neutral regime of Table 3 .

Figure 7: CartPole with a mid-life dynamics change: paired post-change area under the score curve, growing model versus protocol-matched twin, per seed.

Figure 8: Anti-interference instruments. Left: policy drift with a free versus incumbent-anchored reference, paired per seed (lower in 5/5). Right: taught-knowledge probe error under the interleaved versus reward-only regime (log–log; all points below the diagonal, by

5.9

–

95\times

Figure 9: Keep-learning curves at forty full-data tasks: accuracy on each arriving task after its first training epoch (means over ten seeds; bands span the seeds). The fixed control’s first-pass score decays along the sequence; the growing model’s holds.

Lane	Registered criterion	Outcome	Final params / note
Split, full data	above fixed control $\geq 8/10$	9/10	7,117 vs. 8,273 ceiling
	above protocol twin $\geq 6/10$	9/10
	retention not worse $\geq 6/10$	8/10
	governance: identical finals	10/10
Split, pilot	all four lines	met	75% ceiling recovery
Permuted	both lines	met (5/5)	—

Table 4: Continual-learning benchmark results. Every criterion was registered before its run; the two uncontrolled Fashion lines and their dose boundary are described in § 11.5 .

Figure 10: Size adjustability under persistently growing data (moderate round; five verdict seeds faint, medians bold). (a) Stage-exam transfer accuracy of the growable and the lifelong-trained fixed arm; both decline as the class inventory grows to

\sim

400. (b) The growable arm’s size follows demand from parity (12k) to

\sim

57k backbone parameters while the fixed arm’s backbone is constant. (c) Per-seed score gap (growable

-

fixed); markers denote the registered settled crossovers. Stage scores are taken only after both arms consolidate to a measured plateau.

Figure 11: Fidelity on current surfaces over life (five seeds faint, medians bold). Both arms are educated under the same curriculum to graduation (dashed line); afterwards the frozen arm’s fidelity collapses under drift and temptation while the governed plastic arm holds and recovers.

Round	Gate design	Growing track	Stationary regret
1	naive ex-ante	3/5	8/8 regretted
2	silenced ex-ante	1/5	governed, 2 breaches
3	shadow audit (exam world)	C-G1 0/5 †	(no stationary track)
4	incumbent–candidate	1/5 ∗	non-regret 4/5
4b	incumbent–candidate (exam world, X2b)	C-G1 0/5; C-G2 3/5	(no stationary track)

Table 5: The governed-gate arc (details App. B ). Rounds 1–2 and 4 report the generative track’s C-N1 against the lifelong-trained fixed twin; round 3 ran in the open-inventory exam world, whose criterion is C-G1 ( † the gate un-starved there: adoptions

0\to 2

–

3

per seed). ∗ The same round’s transformer comparison is 5/5. Appendix-internal round labels map to rows 2–4 here; X2b shares row 4’s governance form in the exam world. Each round’s observations appear in § 14 .

Figure 12: Depth-separation mapping (three seeds, equal parameter cap; shaded band

=

seed range). The deep/wide final-loss ratio reverses with the target’s composition depth

k

: above parity at

k{=}2

for most seeds (depth wins in 1/3), and below parity for every seed at

k{=}4

and

k{=}6

— with the deepening arm holding 58% of the width arm’s parameters. Data: the depth-mapping campaign’s run directories.

Figure 13: The retention / new-learning frontier under a strict sequential domain switch (both axes log; lower-left is better). Untreated, window-only, and growth-only lives forget at

10^{2.7}

–

10^{3.2}

; review courses move lives to the frontier, with the number of spaced courses acting as the retention dose and course-plus-growth taking the best new-domain corner. Data: the forgetting-family campaigns’ run directories.

Figure 14: Cyclic drift (A

\to

\to

A; seed mean, shaded bands

=

per-arm seed range). During B the old-world probe rises for every arm (the served function must track the drift); when A returns, lives that adapted in service re-recover in 2–3 batches against 20–29 for a fresh model — the relearning-savings measurement of retention. Data: the cyclic-drift campaign’s run directories.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

battery	runs	rows/run	end params	wall total
GGEN2 (+P) [X1]	20+20	40k / 18.2k	58–641	95 s
GGROW2 [X2]	15	30k	1.9k–7.3k	213 s
GGEN3 (+P) a [X1b]	25+25	40k / 18.2k	58–641	127 s
GGROW3 [X2b]	15	30k	1.9k–2.5k	237 s
GUARD4 (+cal.) [r4]	30+4	60k	2.3k–6.4k	22.4 min
X-SAT r1	5 pairs	600k	5.0k $\to$ 40.8k	8.7 h

Appendix

Table 6: Recorded wall-clock (store fields; medians and ranges in the run directories); bracketed labels are the registered round names used in App. B and App. D . a Includes the pre-fix battery kept on record; excludes one anomalous 900.0 s pre-fix wall record (disclosed as recorded).

A-row	claim	verdict
A-M1	additive output identity (concat $=$ block-sum)	exact ( $3.6\times 10^{-15}$ )
A-M2	head-add function-preserving	exact (bitwise $0.0$ )
A-M3	head-widen with absorbed scale	exact ( $\leq 10^{-12}$ ; unabsorbed control shifts $O(1)=1.27$ )
A-M4	two-step trainability property	both instantiations as derived (add: $W^{O}$ only at step 1, generators exactly $0$ ; widen: zero side $W^{K}$ / $W^{O}$ live at step 1 under the task signal, seeded side’s gradient exactly $0$ until step 2)
A-M5	$J_{\mathrm{att}}$ certified gradient	FD agreement $3.0\times 10^{-11}$ (three configurations)
A-M6	locality split (no upstream leak)	exact zeros with positive control

Appendix

Table 7: Attention-component program: mechanics rows (A-M) and prediction rows (A-P), verdicts as registered.

seed	end params	crossover ( $N$ rows $\cdot$ $P$ params)	end grow/fixed
0	57,247	$200{,}000\cdot 27{,}411$	$0.261\,/\,0.201$
1	58,015	censored	$0.126\,/\,0.143$
2	54,943	$350{,}000\cdot 35{,}288$	(final checkpoint unsettled) †
3	54,943	$550{,}000\cdot 51{,}358$	$0.372\,/\,0.264$
4	58,399	$50{,}000\cdot 17{,}614$	$0.179\,/\,0.076$

Appendix

Table 8: X-SAT round 2 (moderate regime), per seed: end size, registered crossover point, and end-of-life loss ratio.

seed	slope diff	TTD g/f (rows)	recovery med. g/f	cost g/f
0	$-0.0094$	$2{,}900\,/\,1{,}000$	$8{,}050\,/\,8{,}000$	$20{,}268\,/\,5{,}400$
1	$-0.0029$	$7{,}000\,/\,1{,}000$	$500\,/\,8{,}000$	$4{,}096\,/\,5{,}400$
2	$+0.0046$	$1{,}000\,/\,1{,}000$	$8{,}000\,/\,32{,}000^{c}$	$20{,}781\,/\,5{,}400$
3	$+0.0007$	$1{,}000\,/\,1{,}000$	$1{,}675\,/\,8{,}000$	$2{,}079\,/\,5{,}400$
4	$+0.0094$	$2{,}250\,/\,1{,}000$	$2{,}675\,/\,7{,}900^{c}$	$9{,}431\,/\,5{,}400$

Appendix

Table 9: GUARD round 4, per-seed grading (grading-twin output, archived): slope difference (growable

-

frozen, fidelity per

10^{3}

rows; positive favors the growable arm; favorable 3/5 by the paired sign), time-to-degradation (drop

0.10

below own graduation; favorable 5/5 under the registered

\geq

rule, two censoring-floor ties), median recovery time per excursion (favorable 4/5; c marks seeds whose frozen arm includes an episode that never recovered — censored; medians are over recovered episodes), and logged remediation work (rows; reported as a frontier, not an adjudication criterion — the frozen arm’s rehearsal cost is fixed by its schedule and never restores its decline).

Figure 15: A governed growth timeline (real trace from the released system; parameters 65

\to

\to

147, final tree height 2). Structural edits (

\omega

\rho

) are exact at application (measured

|\Delta f|

annotated); the gate promotes three times and refuses the final commit — the incumbent already scored 1.0 on the recent held-out slice, so further speculation that did not pay was not adopted.

Figure 16: A real topology growing adaptively under governance (released system, fixed seed): working-state structure at five stages, with structure parameters (tree height, parameters, composite-node count) and the gate’s verdict per stage. Every network — outer and inner — is the standard layered motif (input

\to

hidden column

\to

output); accent squares are composite units, and each inner network hangs beneath its host as the same motif, reads the same input

\hat{x}

, and adds its output to the host unit through the small

+

(parallel correction, not a chain). Note the final stages: structure that did not pay was refused by the gate and never reached the served model.

§	Claim	Campaign / artifact	Verdict	Key number	Status
The axiom and its predictions (§ 2 )
§2	T1: adaptation localizes at the drift scale	E8	MIXED	—	boundary
§2	T2: lock-in release after $\omega$	E11a	PRECOND. NOT MET	3/3 scenario failed to bind	open
§2	T3: unfrozen coarse scale robust to label noise	E10	FAIL branch	—	boundary
§2	T4: $\Phi$ trigger necessity / takeover safety	E11c	safety PASS; necessity NOT SUPP.	—	boundary
§2	size adjustability: born small, ends above a fixed counterpart	X-SAT A-1 (App. B.7 )	PASS	5/5; $5{,}020\to\sim$ 40k past 12,020	supported

Appendix

Table 10: Claims

\to

artifacts

\to

adjudicated verdicts. No sentence in this paper outruns this table. Rows tagged “C” cite the pedagogy appendix (App. C ).

Lane	Per-seed values (paired, seed order)
Stationary control (gap; registered as a no-bar report)	$-60.5$ , $-4.2$ , $+27.9$ , $+10.8$ , $-54.4$
Staged, reward-only (gap)	$-73.8$ , $-8.9$ , $-10.2$ , $-39.0$ , $-37.5$
Staged, alternating (gap)	$0.0$ , $-4.2$ , $-5.4$ , $0.0$ , $0.0$
Quadratic staged, reward-only (gap)	$-14.0$ , $+50.6$ , $-11.5$ , $+1.6$ , $+27.0$
Quadratic staged, alternating (gap)	$+1.9$ , $-4.3$ , $+7.2$ , $-9.8$ , $+10.4$
CartPole post-change AUC (grown/twin)	2,557/2,259, 2,144/2,062, 2,416/1,928, 1,701/2,405, 2,244/2,244

Appendix

Table 11: Per-seed values for the adjudicating lanes of § 8 /§ 11 (gaps are grown-arm minus protocol-matched twin; declines in points; exam-day means are all-task post-review accuracies).

Symbol / name	Kind	One-line meaning	Defined
scope	structure	a network in the inclusion tree (root or inner)	Def. 1
composite node	structure	hidden unit hosting an inner network	Def. 1
inclusion tree	structure	the containment relation of scopes	§ 3
tree height	quantity	height of the inclusion tree (per branch)	§ 3
layer depth	quantity	$1+$ blocks: a scope’s composed stages (per scope)	§ 4.3
$u_{j}$	signal	EMA instability of node $j$ ’s input-weight updates (aims growth)	§ 3.3

Appendix

Table 12: Glossary. Every role is a replaceable part behind a registry (§ 4.3 ); every quantity is recorded in the audit artifacts.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 305 papers

Aug 19, 2026

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Continual Learning for LLM Agents Continual Learning Agent Harness Optimization

Ask PDF

May 14, 2026

On the Stability of Growth in Structural Plasticity

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden-neuron units or growing new ones. Although growth is appealing for adaptive and continual systems, we show that it is not simply the inverse of pruning. Pruning selects among units that have participated in training from the start, whereas growth inserts new units into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks, but becomes clear in harder image-classification settings with a convolutional trunk. In these settings, Grow can achieve high final accuracy during the structural-editing procedure, while Prune is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch. Interventions targeting optimizer state, insertion, selection, and trainability show that improving the integration of newborn units can improve adaptive performance, but does not automatically produce better final subnetworks. In continual-learning benchmarks stressing plasticity loss, Grow becomes competitive mainly when new units have enough time to integrate. Together, these results suggest that Grow should be evaluated not only as an architecture-search operator, but as a time-sensitive optimization process whose success depends on insertion stability.

Lute Lillo, Nick Cheney

University of Vermont

Dynamic Neural Networks Neural Network Growth Neural Network Training Dynamics Neural Network Pruning

Ask PDF

Feb 23, 2026

Representation Stability in a Minimal Continual Learning Agent

Continual learning systems are increasingly deployed in environments where retraining or reset is infeasible, yet many approaches emphasize task performance rather than the evolution of internal representations over time. In this work, we study a minimal continual learning agent designed to isolate representational dynamics from architectural complexity and optimization objectives. The agent maintains a persistent state vector across executions and incrementally updates it as new textual data is introduced. We quantify representational change using cosine similarity between successive normalized state vectors and define a stability metric over time intervals. Longitudinal experiments across eight executions reveal a transition from an initial plastic regime to a stable representational regime under consistent input. A deliberately introduced semantic perturbation produces a bounded decrease in similarity, followed by recovery and restabilization under subsequent coherent input. These results demonstrate that meaningful stability plasticity tradeoffs can emerge in a minimal, stateful learning system without explicit regularization, replay, or architectural complexity. The work establishes a transparent empirical baseline for studying representational accumulation and adaptation in continual learning systems.

Vishnu Subramanian

AI/ML Computational Science Associate Manager, Accenture

Continual Learning Lifelong Learning Agents Stability-Plasticity Dilemma Representation Drift Online Continual Learning

Ask PDF

Abstract

Continual Learning

Explore similar work

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

On the Stability of Growth in Structural Plasticity

Representation Stability in a Minimal Continual Learning Agent