Optimization Geometry of Equivalent Brownian RKHS Representations

Mahdi Mohammadigohari; Gustau Camps-Valls

cs.LGSep 18, 2026

Optimization Geometry of Equivalent Brownian RKHS Representations

Authors: Mahdi Mohammadigohari, Gustau Camps-Valls

Organizations: Faculty of Engineering Free University of Bozen-Bolzano Bruno Buozzi 1, Bolzano 39100, Italy · Image Processing Laboratory (IPL) Universitat de València Paterna, València 46980, Spain

Abstract

Equivalent finite parameterizations can represent the same functions and intrinsic norm yet induce different optimization algorithms. We study this effect in a controlled finite Brownian RKHS with nodal, increment, and spectral coordinates. Classical finite-element, RKHS-interpolation, Brownian-covariance, and mixed-boundary DCT identities make the shared hypothesis class, Brownian energy, approximation operator, and coordinate maps explicit. Our main results concern the optimization geometry of this fixed model. With mapped initialization, identical scalar steps, and identical minibatches, nodal and spectral GD/SGD have exactly the same mapped trajectories. Increment GD is an explicit Euler step for the constant Brownian/Sobolev metric, with factor $1/h$ . For Brownian-regularized least squares, $κ_2(\mathbf H_{\mathrm{inc}})\le1+A/ρ$ , independently of grid resolution $G$ for fixed $A$ , $ρ>0$ , and the stated normalization. Under the stated standard-Adam convention, the universal orthogonal equivariance group is exactly the signed permutations; the block DCT-VIII transform is not one. Float64 tests over five grids numerically verify the finite identities, mapped one-layer and recursive trajectories, conditioning predictions, and theorem-matched Adam separation. Thus coordinate effects are isolated without changing the represented functions, intrinsic regularizer, or approximation space.

Figures & tables

Figure 1: Direct numerical verification of the coordinate–optimizer theory. (a) Each marker is the maximum over the recorded parameter, function, objective, gradient, and preconditioner diagnostics and over full-batch GD and shared-minibatch SGD. The recursive markers correspond to grid triples

(8,16,32)

and

(32,64,128)

, plotted at their maximum

G

; dashed segments guide the eye. All measured residuals remain far below the predeclared

10^{-8}

tolerance. (b) For the pure Brownian quadratic, nodal/spectral conditioning grows with the mesh whereas increment conditioning is exactly one. Least-squares markers are five-seed means; increment conditioning remains independent of

G

under the stated fixed quantities and below

1+A/\rho=21

. (c) Ordinary GD remains equivariant, whereas standard Adam has a nonzero mapped nodal–spectral first-step discrepancy. Learning-objective markers are five-seed means and bars span the minimum-to-maximum range. The ordinate is a coordinate-mapping discrepancy, not a predictive loss or error. These results establish permitted basis dependence, not a universal performance ordering.

Check	Scope	Maximum residual	Type
$\mathbf{D}_{0}^{\top}\mathbf{D}_{0}=h\mathbf{K}_{0}$	5 grids	$0$	absolute
$\mathbf{K}_{0}=\mathbf{Q}\bm{\Lambda}\mathbf{Q}^{\top}$	5 grids	$9.99\times 10^{-15}$	relative
Closed-form spectrum	5 grids	$1.41\times 10^{-13}$	relative
$\mathbf{K}_{0}^{-1}$ versus Brownian Gram	5 grids	$0$	absolute
Coordinate reconstruction	125 random trials	$3.12\times 10^{-14}$	absolute
Intrinsic Brownian energy	125 random trials	$3.99\times 10^{-15}$	relative

Table 1: Float64 verification of the exact finite-Brownian and mapped-optimizer identities. The last four rows report maxima over the listed diagnostics; these are identity checks, not predictive-performance comparisons.

$G$	$\kappa(\mathbf{K}_{0})$	Pure iters N/S / Inc.	LS $\kappa$ N/S / Inc.	LS iters N/S / Inc.	Adam $\Delta_{1}$ linear	Adam $\Delta_{1}$ learning
$8$	$29.3$	$123.2\,/\,1.0$	$6.94\,/\,4.86$	$31.2\,/\,21.6$	$0.02$	$7.09\times 10^{-3}$
$16$	$113.5$	$467.0\,/\,1.0$	$24.45\,/\,5.11$	$109.6\,/\,22.6$	$0.04$	$0.01$
$32$	$437.7$	$1795.2\,/\,1.0$	$92.15\,/\,5.16$	$413.0\,/\,23.4$	$0.06$	$0.02$
$64$	$1.7\times 10^{3}$	$7004.2\,/\,1.0$	$357.64\,/\,5.17$	$1601.8\,/\,23.4$	$0.09$	$0.03$
$128$	$6.7\times 10^{3}$	$27629.0\,/\,1.0$	$1408.71\,/\,5.17$	$6310.2\,/\,23.4$	$0.13$	$0.04$

Table 2: Conditioning, fixed-step convergence, and standard-Adam basis dependence across meshes. “N/S” denotes identical nodal/spectral quantities and “Inc.” the increment realization. Iteration counts use the analytic optimal scalar step and relative objective-gap tolerance

10^{-8}

; least squares uses

\rho=0.1

, with increment bound

1+A/\rho=21

. The last columns report mapped nodal–spectral first-step discrepancies.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Result	Content
Proposition 1	Classical stiffness identity in the Brownian normalization
Theorem 1	Brownian-specific kernel and covariance–precision formula
Corollaries 2 and 3	Classical interpolation consequences with exact constants
Theorem 2	Exact equivalence of the three realizations
Theorems 3 and 4	Closed-form block DCT-VIII specialization
Propositions 2 and 3	Standard reparameterization and Brownian metric consequence

Appendix

Table 3: Summary of the main theoretical results.

Figure 2: Proof-dependency structure of the theoretical results. The displayed artwork is the supplied dependency graph without redrawing or re-typesetting; panel headings and result nodes are clickable and jump to the corresponding section or statement.

Result	Content	Page
Lemma D1	Structure of the stiffness matrix	page D1
Lemma D2	Kernel of the stiffness matrix	page D2
Lemma D3	Block decomposition of the anchored stiffness matrix	page D3
Lemma D4	Brownian energy in increment coordinates	page D4
Lemma D5	Brownian energy in spectral coordinates	page D5
Lemma D6	Explicit spectral Brownian basis	page D6

Appendix

Table 4: Summary of the auxiliary theoretical results. Their logical dependencies are illustrated in Fig. 2 . For the main theoretical results, see Table 3 .

Stage	Purpose	Data/runs	Mesh	Budget	Precision
E0	Finite Brownian RKHS, spectrum, and coordinate identities	125 random vectors	$G\in\{8,16,32,64,128\}$	4097-point dense grid	float64 CPU
E1A	One-layer mapped-trajectory identities for GD and SGD	256 samples, 5 seeds	$G\in\{8,16,32,64,128\}$	40 GD; 80 SGD; batch 32	float64 CPU
E1B	Conditioning and fixed-step convergence under mesh refinement	256 samples, 5 seeds	$G\in\{8,16,32,64,128\}$	gap $10^{-8}$ ; analytic steps	float64 CPU
E1C	Basis dependence of standard Adam	256 samples, 5 seeds	$G\in\{8,16,32,64,128\}$	200 steps; full batch	float64 CPU
E2	Recursive blockwise mapped-trajectory identities	128 samples, 5 seeds	$(8,16,32)$ and $(32,64,128)$	8 GD; 16 SGD; batch 32	float64 CPU
Real-data audit	Independent metric and statistical reconstruction	390 prediction artifacts	EuroSAT and Salinas	5720 published-value checks	NumPy reconstruction

Appendix

Table 5: Frozen verification and audit protocols. Identity-test arms share one mapped represented initialization, objective normalization, scalar step sequence, and, for SGD, minibatch order; they are never tuned separately.

Model	Accuracy Retention AUC (%)	Macro-F 1 Retention AUC (%)	Worst Accuracy Retention (%)	Worst Macro-F 1 Retention (%)	Maximum ECE
Patch MLP	$37.78$	$32.16$	$21.80$	$13.88$	$0.774$
Spatial Path-Atomic BKL	$\mathbf{46.81}$	$\mathbf{42.10}$	$\mathbf{23.38}$	$\mathbf{17.68}$	$\mathbf{0.629}$

Appendix

Table 6: EuroSAT robustness under nested test-time spectral-band dropout. Values are means over five paired seeds. Retention AUC is normalized over zero to six zeroed bands; higher is better. Maximum ECE is lower-is-better.

Figure 3: EuroSAT accuracy- and macro-F 1 -retention trajectories under nested test-time spectral-band dropout. Lines show paired-seed means and shading one standard deviation.

Figure 4: EuroSAT calibration and likelihood degradation under nested spectral-band dropout; lower is better.

Metric	Mean improvement	Two-sided $p$	Paired $d_{z}$	BKL-favored seeds
Accuracy retention AUC	$0.0903$	$0.0625$	$1.61$	$5/5$
Macro-F 1 retention AUC	$0.0994$	$0.0625$	$2.11$	$5/5$
Worst accuracy retention	$0.0158$	$0.6875$	$0.17$	$3/5$
Worst Macro-F 1 retention	$0.0379$	$0.4375$	$0.46$	$3/5$
Maximum ECE reduction	$0.1449$	$0.0625$	$1.53$	$5/5$

Appendix

Table 7: Exact paired sign-flip analysis over five seeds. Improvements are oriented so that positive values favor Spatial Path-Atomic BKL.

Model	Acc. (%)	Macro-F1 (%)	Bal. Acc. (%)	Worst F1 (%)	ECE	NLL
Patch MLP	$91.87\pm 0.93$	$94.68\pm 1.14$	$94.83\pm 1.28$	$73.93\pm 2.55$	$0.018\pm 0.002$	$0.245\pm 0.013$
Constrained VBKL	$92.18\pm 0.20$	$95.49\pm 0.40$	$95.23\pm 0.60$	$73.83\pm 0.63$	$0.015\pm 0.004$	$0.218\pm 0.008$

Appendix

Table 8: Clean Salinas performance (mean

\pm

standard deviation over five final evaluation seeds).

Figure 5: Salinas accuracy, Macro-F1, and balanced-accuracy retention under nested spectral-band masking. Lines are crossed-design means and shaded regions are 95% crossed-bootstrap intervals.

Model	Accuracy-retention AUC (%)	Macro-F1-retention AUC (%)	Balanced-retention AUC (%)
Patch MLP	$76.30\,[69.82,\,82.13]$	$67.98\,[60.57,\,74.78]$	$72.26\,[66.21,\,77.99]$
Constrained VBKL	$75.12\,[71.53,\,78.39]$	$65.66\,[60.29,\,70.70]$	$69.97\,[65.71,\,74.09]$

Appendix

Table 9: Aggregate Salinas robustness under nested spectral-band masking. Entries are means with 95% crossed-bootstrap intervals. Retention is higher-is-better; ECE and NLL increases are lower-is-better.

Figure 6: Salinas corruption-path comparisons. The upper panels show BKL-minus-MLP differences, with positive values favoring constrained VBKL; for ECE and NLL, positive values denote a smaller corruption-induced increase. The lower panels show the corresponding calibration and negative-log-likelihood degradation.

Metric	Favored model	Mean improvement [.2ex][95% CI]	Cells B/M	Model seeds B/M	$p_{\rm model}$	Mask seeds B/M	$p_{\rm mask}$
Accuracy-retention AUC	Patch MLP	$-1.17\,[-6.86,\,+4.33]$	14/11	2/3	0.6875	3/2	0.6250
Macro-F1-retention AUC	Patch MLP	$-2.32\,[-10.28,\,+5.39]$	12/13	2/3	0.6875	2/3	0.4375
Balanced-accuracy-retention AUC	Patch MLP	$-2.29\,[-8.88,\,+4.27]$	12/13	2/3	0.6875	1/4	0.3125
Level-6 accuracy retention	Constrained VBKL	$+9.74\,[-0.54,\,+18.58]$	19/6	5/0	0.0625	3/2	0.2500
Level-6 Macro-F1 retention	Constrained VBKL	$+6.45\,[-2.64,\,+14.72]$	18/7	4/1	0.1875	4/1	0.1875
ECE-increase AUC	Constrained VBKL	$+0.021\,[-0.032,\,+0.074]$	16/9	4/1	0.4375	3/2	0.3125

Appendix

Table 10: Crossed Salinas robustness comparisons. Improvements are oriented so that positive values favor constrained VBKL. The confidence interval is obtained by crossed bootstrap. The model-seed and mask-seed columns report BKL-favored/MLP-favored marginal means and exact two-sided sign-flip

p

-values.

Level	Bands	MLP Acc. ret.	BKL Acc. ret.	$\Delta$ Acc.	MLP F1 ret.	BKL F1 ret.	$\Delta$ F1
0	0	100.00	100.00	+0.00	100.00	100.00	+0.00
1	16	95.09	94.34	-0.75	91.68	93.27	+1.60
2	31	90.83	86.75	-4.09	85.46	81.60	-3.86
3	47	78.30	73.17	-5.13	68.70	62.62	-6.08
4	63	66.72	63.43	-3.29	55.36	48.91	-6.45
5	78	55.97	57.27	+1.29	42.67	40.28	-2.39

Appendix

Table 11: Severity-by-severity Salinas retention. Means are taken over the full

5\times 5

crossed design at corrupted levels; the clean level uses five model seeds. Differences are BKL minus MLP in percentage points.

Explore similar work

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers
Jul 16, 2026Akhilesh GogikarNeural Network Optimization Adaptive Gradient Methods
1Independent Researcher.
The Loss Does Not See the Basis, but Adam Does
Aug 5, 2026Devender SinghLow-Rank Matrix Decomposition Matrix Optimization
Department of Computer Science Memorial University of Newfoundland
Functional Gradient Descent with Adaptive Representations
Jun 15, 2026Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia +3Representation Learning Gradient Descent
FGV EMAp · IMPA · Stanford University

Jul 16, 2026, cs.LGJ/K move · Enter open · S save

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

Akhilesh Gogikar

1Independent Researcher.

Neural Network Optimization Adaptive Gradient Methods Neural Network Training Dynamics Implicit Regularization

Open paper AskPDF arXiv

Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.

Abstract

Explore similar work

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

The Loss Does Not See the Basis, but Adam Does

Functional Gradient Descent with Adaptive Representations