Dynamics of Hidden Learning Structure

Qinyou Wang

cs.LGOct 21, 2026

Dynamics of Hidden Learning Structure

Authors: Qinyou Wang

Abstract

A learner's present predictions do not fully determine how it will learn next. We study the dynamics of hidden learning structure through two related operations. Fiber Regulation Theory (FRT) describes protected repair of existing function and the learning responses left available after repair. Fiber Acquisition Theory (FAT) describes the formation and renewal of trainable structure, distinguishing neutral installation, access to a learning signal, and additional functional directions. The two operations also constrain one another. A repair obligation can force a change in a learning channel, while acquired features can make new protected repairs feasible or reduce the minimum coefficient cost of existing ones. We characterize joint admissibility and response reachability, and give a finite supervised-learning construction in which the two operations together produce a favorable response absent from either operation alone. We implement protected head repair and fixed-capacity LoRA renewal in Qwen2.5-32B with AdamW, under replay and output distillation. Across two training trajectories differing in seed and within-task example order, the coordinated policy achieves higher teacher-forced Exact than either separate policy in all four comparisons. It also has the lowest macro NLL under both learning rules on the second trajectory. The implementations realize structure-conditioned self-regulation within specified information and action constraints: accessible state readouts and arrived evidence guide repairs and structural renewals, and ordinary learning continues from the modified state.

Figures & tables

Panel of Figure 1 — Figure 1: Regulation and acquisition within one evolving learner. The complete state $s_{t}$ is distinct from the available information $i_{t}$ . Regulation repairs existing function; acquisition forms or renews trainable structure. Using $\mathcal{I}_{t}$ , a policy selects an admissible pair $\xi_{t}=(f_{t},a_{t})\in\mathscr{K}(s_{t})$ through available certificates or a valid robust rule. Identity components allow either intervention, both, or no extra write. The state-transition marginal $\bar{K}_{\xi_{t}}$ uses the declared execution order. Ordinary learning consumes the resulting state and supplies readouts to later decisions. The FRT–FAT arrow denotes possible cross-effects in admissibility, response and repair cost.

Panel of Figure 2 — Figure 2: Acquisition-adjusted repair geometry. Two independent exact affine examples of Theorem 5.3 are shown in obligation-response coordinates. The solid old set and dashed enlarged set contain responses attainable with squared coefficient cost at most one. In both examples, $q=1$ , $B=Q=1$ and $W=2$ . (a) Set $C=(1,0)$ , $H=\left(\begin{smallmatrix}1&1\\ 1&0\end{smallmatrix}\right)$ and $\Psi=(1,2)^{\top}$ . With $v=1/2$ , the raw acquired response $S=\Psi v$ receives the compensation response $-HC^{\dagger}Bv$ , yielding $Lv=D$ . The target $D=(0,1/2)^{\top}$ lies outside the old reachable space. (b) Set $C=(1,0,0)$ , $H=\left(\begin{smallmatrix}1&1&0\\ 1&0&1\end{smallmatrix}\right)$ and $\Psi=(2,1)^{\top}$ . Both interfaces span the response plane, yet the minimum cost of $D=(1,0)^{\top}$ falls from $1$ to $2/3$ . The arrows in (a) express vector addition, not a learning trajectory.

		Original trajectory		Changed trajectory
Learning rule	Policy	Exact	Macro NLL	Exact	Macro NLL
Replay	Baseline	102/128	0.279321	110/128	0.249136
	FRT	113/128	0.200758	112/128	0.268962
	FAT	114/128	0.216516	110/128	0.263532
	Joint FRT–FAT	115/128	0.222998	113/128	0.240527
LwF	Baseline	85/128	0.491530	75/128	0.547257

Table 1: Full-policy outcomes on the same 128-item MASSIVE-derived panel. Exact is teacher-forced label-plus-EOS correctness. Bold marks the highest Exact and lowest macro NLL within each learning rule and trajectory. These are complete-policy comparisons, not fixed-action factorial experiments.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Object	Specification
Model	Qwen2.5-32B base; 64 layers; hidden size 5120; vocabulary 152064; BF16 backbone
Revision	1818d35814b8319459f4bd55ed1ac8709630f003
Adapter	FP32 LoRA, rank 8, alpha 16, dropout 0, both factors trainable; q/v/o/down projections in all 64 layers; 30,408,704 trainable adapter parameters
Renewal location	Zero-based layer 32, self-attention o_proj ; eight physical factor slots of width 5120
Optimizer	AdamW, learning rate $10^{-4}$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\epsilon=10^{-8}$ , weight decay 0, clipping norm 1, constant scheduler; foreach/fused disabled
Recorded software	PyTorch 2.10.0, CUDA wheel 12.8, Transformers 4.57.6, PEFT 0.18.1, Accelerate 1.12.0, NumPy 2.3.5; math SDPA, nonreentrant checkpointing, TF32 disabled

Appendix

Table 2: Execution settings of the tested model and learning rules.

Learning rule	Policy	FRT commits	FAT commits	Replay redirections
Replay	Baseline	0	0	0
	Regulation	3	0	95
	Acquisition	0	3	0
	Joint	3	3	96
Distillation	Baseline	0	0	0
	Regulation	5	0	0

Appendix

Table 3: Committed structural operations in the changed training trajectory. Repair consultations may decline to write. Replay redirections count individual examples substituted during subsequent batches.

Policy	Teacher forwards per trajectory	Extra calls
Baseline	771	0
Regulation	771	0
Acquisition	783	12
Joint	803	32

Appendix

Table 4: Teacher forward calls in distillation. Both training trajectories have the same count for each policy. Extra calls are relative to the corresponding baseline.

Ordinary update	Before FRT	After FRT	$\Delta_{F}Q_{C}$
96	0.016320877	0.016660973	$+0.000340097$
160	0.009440793	0.010511544	$+0.001070751$
192	0	0	$0$
224	0.014273338	0.014275268	$+0.000001930$
256	0.004792862	0.004691567	$-0.000101294$

Appendix

Table 5: Matched-physical-proposal finite acquisition opportunity (nats per current example). All entries use arrived training samples and the same teacher within each pair.

Trajectory	Learning rule	Exact contrast (count)	Macro-NLL contrast
Original	Replay	$-10$	$+0.085045$
Original	Distillation	$-1$	$+0.072506$
Changed	Replay	$+1$	$-0.042830$
Changed	Distillation	$0$	$-0.020493$

Appendix

Table 6: Readout-specific complete-policy contrasts. The policies may follow different later states and actions, so these values are not fixed-action causal factorial effects.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 516 papers

Fiber Fingerprints of Hidden Learning-State Dynamics
Aug 17, 2026Qinyou WangHidden-State Probing Predictive Representation Learning
Generalized Residual Closure: General Learning Dynamics for Stability-Plasticity Compatibility
Sep 30, 2026Dongxu Li, Yinuo Zhang, Hongyu Zhang +1Stability-Plasticity Dilemma Continual Learning
Xi’an Jiaotong University · Xi’an University of Architecture and Technology
Intrinsic-Extrinsic Coupling in Learning Dynamics
Sep 24, 2026Qinyou WangContinual Learning

Aug 17, 2026J/K move · Enter open · S save

Fiber Fingerprints of Hidden Learning-State Dynamics

Qinyou Wang

Hidden-State Probing Predictive Representation Learning State Representation Learning

Open paper AskPDF arXiv

A learning system can occupy execution states that are indistinguishable under every declared present-behavior readout yet respond differently to future training. We formalize this through fiber fingerprints: controlled future-learning response laws restricted to present-behavior equivalence classes. Prefix-compatible finite probes induce a predictive quotient functor, a Nerode-type minimal recursively sufficient representation, and a canonical set-level predictive fiber without assuming smoothness, reversibility, finite rank, or a manifold. Under an explicit finite-dimensional Hilbert realization, response decomposes into visible, visible-mode-reuse, and irreducible-new sectors; a history-reachability bridge retains only distinctions generated by natural training histories. Conditional mechanism results then identify a graph-Hodge chronology decomposition, a regular switching class with root-mean-square scale

\sqrt{p}η^{3/2}

and finite-scale corrections, and an exact Adam moment section whose immediate adaptive field is constant while common future gradients can reveal hidden moment differences. Frozen Transformer--LoRA--AdamW studies with Qwen2.5-7B and Mistral-7B-v0.3 support a local action backbone, longer-horizon first-return non-closure, and fresh visible-relative completion with output-range reuse and a low-rank irreducible sector. Stronger claims remain bounded by preregistered negative or mixed results: re-anchored transport is unresolved above its measurement floor; the strict finite-grid Hodge--

3/2

conjunction is unmet despite prospective contraction; Qwen accessibility is not established in the frozen raw moment chart; and Mistral revelation is future-context dependent rather than bank invariant. Within these support-, scale-, metric-, and context-resolved boundaries, present behavior is not a sufficient statistic for declared future learning.

Abstract

Neural Network Training Dynamics

Explore similar work

Fiber Fingerprints of Hidden Learning-State Dynamics

Generalized Residual Closure: General Learning Dynamics for Stability-Plasticity Compatibility

Intrinsic-Extrinsic Coupling in Learning Dynamics