Non-Interfering Asymmetric AdamW Updates for Constrained Exploration of Equifinality in Hybrid Models
Authors: Ekaterina Gikalo, Zavud Baghirov, Christian Reimers, Markus Reichstein, Martin Jung, Marco Körner
Organizations: Max Planck Institute for Biogeochemistry, Jena, Germany · TUM School of Engineering and Design, Technical University of Munich (TUM), Munich, Germany · ELLIS Unit Jena, Jena, Germany · German Remote Sensing Data Center (DFD), German Aerospace Center (DLR), Weßling, Germany
Hybrid models let neural networks supply the unresolved parameters of a process-based simulator, and are fitted to observations that constrain aggregate fluxes and storages but not the internal partitioning that produces them. Such a fit is equifinal: configurations indistinguishable in the observed aggregates can disagree substantially about the unobserved internals, and therefore about the mechanism they attribute to the same data. We study evapotranspiration, whose total flux is observed while its split into transpiration and evaporation is not. A model ensemble built from random seeds alone spans only a narrow band of the partitionings the data admit. We produce the spread deliberately, training members under auxiliary repulsion on transpiration, subject to the fit to observations not degrading. Non-interference is usually arranged by making the auxiliary gradient orthogonal to the main one, but under AdamW the applied step is a preconditioned momentum direction, so orthogonality between raw gradients no longer implies it. We introduce Non-interfering Asymmetric Network Optimization (NANO), an update whose first-order effect on the main loss is exactly that of the main objective alone, whose auxiliary step is bounded, and which concentrates repulsion in the coordinates the main gradient leaves idle.
Figures & tables
Figure 1: Equifinality in the partitioning of evapotranspiration. Left: ET is the sum of transpiration, soil evaporation and interception evaporation. Right: network is trained against ET alone, so different partitionings can reproduce the same observed target.
Method
median Lmaintest↓
max Lmaintest↓
R std ↑
R spread ↑
Baseline
1.919±0.017
2.130±0.010
0.009±0.003
0.031±0.013
PCGrad
1.887±0.025
2.163±0.026
0.177±0.007
0.621±0.009
Bloop
2.135±0.064
2.429±0.152
0.103±0.015
0.340±0.044
NANO(ours)
1.876±0.016
2.034±0.076
0.143±0.026
0.491±0.093
Table 1: Test-set main loss and spread of R across 10 members on H2CM, as mean and standard deviation over three seeds. A good method keeps the fit unchanged while revealing disagreement in the unobserved partitioning.
Figure 2: Main test loss of 10 members per method over 3 seeds, relative to the baseline median. A method preserves the fit if none of its members exceeds the baseline’s worst test loss.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Per-member mean ET against the main test loss, for the baseline and NANOensembles over three seeds.
Adam and its variants dominate neural network training, but a single run only reveals whether a configuration works well after most of its budget is spent, a poor fit for hyperparameter or architecture search, where configurations must be ranked cheaply and pruned early. We introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via DE, running a short burst of gradient descent only if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and since gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven scalable benchmarks up to one million variables. On three real neural-network tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE finishes the same charged budget 1.7-3.9x faster, at a modest cost in final quality (about one accuracy point on MNIST, 9-11% higher relative test loss on the two fine-tuning tasks; on GSM8K, Adam is about 5 accuracy points more accurate, and fine-tuning lowers accuracy below the base model for both). Inside successive halving on UCI Adult, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, ranking configurations about as consistently with Adam as Adam does with itself across seeds (Kendall's tau 0.66-0.69). EvE is not a total replacement for Adam as a final-stage trainer, but a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up.
Shashank Raj, Kalyanmoy Deb
Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA
Hybrid mechanistic/data-driven models, which combine first-principles with learned components, are increasingly used in process engineering and scientific machine learning. Common hybrid modeling designs are specified primarily through their architectures and training losses, which offers a limited basis for a shared semantic interface to compare or verify them across domains, with comparatively little attention paid to epistemic uncertainty in the mechanistic part. We bridge hybrid modeling and neuro-symbolic (NeSy) AI by reconstructing these designs as instances of NeSy interface. The resulting translation, Hybrid-to-NeSy (H2N), places mechanistic knowledge on the language side, learned modules on the belief side, and validity domains together with constraints on the logic side. For each design, H2N then yields an explicit NeSy inference functional and a logic-belief decomposition. From this decomposition we derive two metrics: structural violation rate (SVR), measuring whether the learned belief respects the mechanistic structure; and belief dispersion (BD), measuring how concentrated the learned plausibility is, serving as a hybrid model's epistemic uncertainty in its mechanistic part. We instantiate H2N on a case study of a structured hybrid model for binary classification under label noise and show that models with higher SVR and BD exhibit greater variability in held-out accuracy. Under structural distribution shift, H2N further quantifies a model's uncertainty during extrapolations, whereas test accuracy reveals the same shift only post hoc.
Moein E. Samadi, Andreas Schuppert
Institute for Computational Biomedicine, RWTH Aachen University, Aachen, Germany. · Center for Computational Life Sciences, RWTH Aachen University, Aachen, Germany.
A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.
Kang Liu, Suyan Li
School of Future Technology, Xi’an Jiaotong University, Xi’an, China, 710049 · Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 119077.