Matched Specialization, Different Adaptation: Second-Moment Structure in Shared Representations

Toru Takahashi

cs.LGOct 11, 2026

Matched Specialization, Different Adaptation: Second-Moment Structure in Shared Representations

Authors: Toru Takahashi

Organizations: Human Informatics and Systems Lab, Doshisha University · Linked Open Data Initiative, NPO · Keio Research Institute at SFC · Stroly Inc.

Abstract

Small Adam-trained shared-representation networks can reach equivalent selected mean readout measurements yet respond differently to the same further learning. We use state interventions to separate second-moment scale and coordinate-pattern effects, then examine their formation and time dependence. In the central two-target experiment, the more precisely readable target switches within ten updates in 67.5% of acquisition-formed checkpoints and 0% of update-formed checkpoints. First-moment exchange leaves the ten-update contrast change within its pre-specified margin, whereas second-moment exchange reverses the contrast. A coordinate-pattern effect remains at common mean adaptive scale. Layer and movement controls locate an effect beyond overall update size. Formation records connect small encoder second moments with stronger cancellation between task gradients. A prospective gradient-projection experiment changes this retained state; transferring the resulting second moment alone reduces the ten-update adaptation response in a fixed recipient. The matched comparison extends to three Gaussian targets, while a room-sensor extension retains component effects but fails endpoint matching. Across tested regimes, component effects survive more widely than reversal or reset equivalence. Effect magnitude and duration depend on learning conditions.

Figures & tables

Prior work	Established comparison	Present comparison
Lyle et al. (2023)	Plasticity, loss geometry, and optimizer instability under changing tasks	Reallocation response after matching selected reconstruction measurements
Asadi et al. (2023) ; Ellis et al. (2024)	Resetting state or time; update-size effects of changed objectives or gradient scales	Component exchange and coordinate pattern at matched mean inverse scaling
Xu (2026)	First-moment transplantation and later transfer, including immediate-output invariance	Second-moment structure and the time course of specialization response

Table 1: Closest prior comparisons and the question tested here. The interventions and measured outcomes specify how the comparisons relate.

Figure 1: A common measured allocation can conceal different adaptation. (a) Acquisition selects specialization with equal loss weights. (b) The matched formation routes receive the same combined opposing pulse. Both move toward the other target, but the acquisition-formed route moves further. Points and intervals summarize 40 seeds; mirrored labels in (b) are averaged within seed. Lines connect endpoints, rather than showing the intervening trajectory.

Pulse	Adam inherited	Adam reset
Acquisition only	$0.1668\ [0.1601,0.1736]$	$0.0011\ [-0.0010,0.0035]$
Update only	$0.1510\ [0.1456,0.1562]$	$0.0120\ [0.0093,0.0149]$
Combined	$0.1951\ [0.1880,0.2021]$	$0.0103\ [0.0079,0.0129]$

Table 2: Route difference in specialization lost after ten opposing updates. Brackets give 95% seed-bootstrap intervals. All inherited-state intervals lie above

+0.05

, and all reset intervals lie inside

\pm 0.02

, meeting their respective pre-specified criteria.

Figure 2: Second-moment exchange reverses two short-horizon response contrasts. (a) Acquisition-formed minus update-formed specialization loss. (b) Constant-minus-variable weight-history contrast at fixed acquisition and mean loss weight. Shaded bands show the respective pre-specified equivalence margins (

\pm 0.02

\pm 0.01

). Intervals are paired seed-bootstrap intervals; each comparison uses 40 disjoint confirmatory seeds.

Figure 3: Formation intervention and a fixed-recipient test. (a) Mean encoder second-moment sum during formation in 40 new seeds. (b) Ten-update specialization loss for the two acquisition recipients with original or projected-history second moments. Within each recipient, weights, first moment, and step count are fixed before the pulse; only

v

changes. Error bars are paired seed-bootstrap 95% intervals.

Figure 4: Scale/pattern interventions and observation horizon resolve different parts of the response. (a) Ten-update route contrast after retaining inherited state, scalarizing

v

, equalizing

\bar{a}_{v}

, or both. (b) Continued common opposition under inherited Adam, fresh Adam, or calibrated SGD. The band at

\pm 0.02

refers to the ten-update equivalence margin; later points have no pre-specified decision rule at that margin. Means and 95% paired seed-bootstrap intervals use 40 new seeds.

Setting	Endpoints	Common-scale pattern effect	Reversal	Reset equivalence
Baseline ( $\beta_{2}=0.999$ , 500)	Y	Y	Y	Y
Width 64	Y	Y	Y	N
ReLU	Y	Y	Y	Y
$\rho=0$	Y	Y	Y	Y
$\beta_{2}=0.9$ , 500	Y	Y	N	N
$\beta_{2}=0.99$ , 500	Y	Y	Y	Y

Table 3: Ten-update claims across 13 distinct extension settings. Y: criterion satisfied; N: an interval establishes failure; ?: unresolved. Endpoint matching requires every measurement interval inside its tolerance. Common-scale pattern effect requires a positive interval; reversal requires own positive and exchanged-

v

negative intervals; reset equivalence requires its interval inside

\pm 0.02

. Individual 95% intervals are used; this inventory is not a success-rate estimate over a population of settings. IV-F has 32 shared seeds; V-B and V-C each have 40, shared across capacities. Baseline occurs only once. Reversal here is a sign criterion; the separate pre-specified requirement that the own-contrast interval exceed

+0.05

is not met by the two-dimensional room-sensor setting.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Component condition	IV-E route contrast	Equal-mean update-history control
Own state	$0.1968\ [0.1910,0.2029]$	$0.0324\ [0.0294,0.0354]$
Exchange $m$	$0.1982\ [0.1912,0.2054]$	$0.0321\ [0.0277,0.0362]$
Exchange $v$	$-0.1265\ [-0.1348,-0.1183]$	$-0.0141\ [-0.0176,-0.0110]$
Reset	$0.0097\ [0.0078,0.0117]$	$0.0046\ [0.0031,0.0062]$

Appendix

Table 4: Combined-pulse response contrasts under component interventions. IV-E compares acquisition-formed minus update-formed routes; the control compares constant minus variable update-weight histories. Each uses 40 disjoint confirmatory seeds.

Intervention	Route contrast [95% CI]
Own	$0.1907\;[0.1837,0.1976]$
First encoder layer	$0.0115\;[0.0060,0.0171]$
Second encoder layer	$0.0611\;[0.0558,0.0665]$
Both heads	$0.1911\;[0.1838,0.1978]$
All parameters	$-0.1183\;[-0.1264,-0.1099]$
Equal movement, own	$0.0303\;[0.0269,0.0338]$

Appendix

Table 5: Layer and movement interventions on 40 saved seed pairs. Layer rows exchange only the named layer’s

v

; equal-movement rows apply the joint norm-control rule.

Figure 5: Formation-time accounting on the original 40 paired seeds. Top: mean second moment per parameter in each encoder layer; dashed curves show the discounted shared ancestor. Bottom: mean accumulated cancellation fraction

-C/S

along each route’s actual observations and loss weights. Curves average both directions within seed and then seeds; interval estimates for the final values appear in the text.

Recipient	Native $v$	Projected-history $v$
Native acquisition	$0.3551\;[0.3469,0.3634]$	$0.3079\;[0.3007,0.3154]$
Projected acquisition	$0.3435\;[0.3358,0.3512]$	$0.2948\;[0.2880,0.3016]$

Appendix

Table 6: Crossed second-moment transplants: specialization loss after ten updates. Each row fixes recipient weights and first moment. Intervals use 40 paired seeds.

$\beta_{2}$	Formation	Own state	Exchange $v$	Reset	Common-scale pattern effect
$0.9$	500	$0.0298$ $[0.0278,0.0318]$	$0.0104$ $[0.0082,0.0127]$	$0.0373$ $[0.0342,0.0406]$	$0.0058$ $[0.0053,0.0064]$
$0.9$	1500	$0.0764$ $[0.0736,0.0794]$	$0.0494$ $[0.0462,0.0526]$	$0.1073$ $[0.1023,0.1123]$	$0.0120$ $[0.0111,0.0131]$
$0.99$	500	$0.1385$ $[0.1329,0.1444]$	$-0.0720$ $[-0.0770,-0.0672]$	$0.0174$ $[0.0150,0.0197]$	$0.0673$ $[0.0625,0.0721]$
$0.99$	1500	$0.2859$ $[0.2789,0.2925]$	$0.0297$ $[0.0204,0.0390]$	$0.0963$ $[0.0913,0.1014]$	$0.1334$ $[0.1204,0.1448]$
$0.999$	500	$0.1962$ $[0.1881,0.2043]$	$-0.1290$ $[-0.1379,-0.1198]$	$0.0073$ $[0.0054,0.0092]$	$0.1028$ $[0.0947,0.1109]$
$0.999$	1500	$0.4341$ $[0.4277,0.4408]$	$-0.1279$ $[-0.1416,-0.1139]$	$0.0574$ $[0.0529,0.0618]$	$0.2528$ $[0.2414,0.2644]$

Appendix

Table 7: Decay and duration grid, 32 paired seeds per setting. Common-scale pattern effect is the route-contrast difference before minus after

v

exchange at common

\bar{a}_{v}

. All settings satisfy endpoint tolerances.

Condition	G, $d_{Z}=2$	G, $d_{Z}=3$	S, $d_{Z}=2$	S, $d_{Z}=3$
Own	$0.1945$ $[0.1906,0.1987]$	$0.4054$ $[0.3951,0.4152]$	$0.0601$ $[0.0456,0.0741]$	$0.1170$ $[0.1024,0.1320]$
Exchanged $v$	$-0.1005$ $[-0.1054,-0.0955]$	$-0.2193$ $[-0.2266,-0.2118]$	$-0.0254$ $[-0.0381,-0.0130]$	$-0.0090$ $[-0.0232,0.0049]$
Reset	$0.0116$ $[0.0101,0.0132]$	$0.0203$ $[0.0178,0.0226]$	$-0.0076$ $[-0.0168,0.0017]$	$0.0046$ $[-0.0062,0.0152]$
Common-scale pattern effect	$0.0825$ $[0.0792,0.0860]$	$0.1541$ $[0.1479,0.1597]$	$0.0103$ $[0.0064,0.0140]$	$0.0257$ $[0.0210,0.0306]$

Appendix

Table 8: Three-target ten-update contrasts. G: Gaussian; S: room sensors. Sensor endpoints are unmatched. All intervals are individual 95% paired-seed intervals.

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 389 papers

Hidden Gauge Controls Feature Specialization in ReLU Networks
Aug 7, 2026Tongxi WangReLU Neural Networks Neural Network Training Dynamics
School of Future Technology Southeast University Nanjing, China
Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum Methods Adaptive Gradient Methods
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
From Latent Space to Training Data: Explainable Specialization in Minimal MLPs
May 25, 2026Enrique Alba, Ezequiel Lopez-RubioMultilayer Perceptrons Neural Network Interpretability
aITIS Software, University of Malaga, Spain

Aug 7, 2026J/K move · Enter open · S save

Hidden Gauge Controls Feature Specialization in ReLU Networks

Tongxi Wang

School of Future Technology Southeast University Nanjing, China

ReLU Neural Networks Neural Network Training Dynamics

Open paper AskPDF arXiv

Training changes a network's predictions while allocating task-relevant structure across its internal units. In an overparameterized ReLU network, several neurons can begin with exactly the same functional role, yet one may acquire a teacher feature while the others become redundant. We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor. In a tractable Gaussian teacher--student model, we fix the complete initial function and vary only a positive-homogeneous scaling gauge. Opposite gauges produce distinct feature trajectories and a sharp

Θ(D^2)

separation in specialization time that no global change of clock can explain. Among any fixed number of initially duplicate students, assigning the favorable gauge to one neuron deterministically selects it as the owner and drives the remaining functional contribution to zero. An exact reaction--transport decomposition attributes the effect to different mobilities for changing a feature's coefficient and direction. We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent, and verify the predicted loss, alignment, pruning, and dissipation trajectories in population and finite-sample training. The initial predictor therefore determines neither when the feature is learned nor which neuron learns it.

Abstract

Multi-Task Learning

Explore similar work

Hidden Gauge Controls Feature Specialization in ReLU Networks

Why β1=β2β_1 = β_2β1​=β2​ Is Dynamically Special in Adam

From Latent Space to Training Data: Explainable Specialization in Minimal MLPs

Why $β_1 = β_2$ Is Dynamically Special in Adam