Residual Learning

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 103

Aug 9, 2026cs.LG

Hybrid Neural-Classical Correction for Frozen Time Series Foundation Models: A Comprehensive Ablation Study on High-Frequency Stock Prediction

Foundation models for time series forecasting demonstrate impressive zero-shot generalization but often underperform on specialized domains such as high-frequency finance. We present a comprehensive study of hybrid neural-classical correction for adapting frozen TimesFM (200M parameters) to stock return prediction during the volatile opening trading hour. We compare two neural correction architectures - AttnCorrect (multi-head self-attention, approximately 471K parameters) and GatedLinear (low-rank bilinear projection with gating, approximately 49K parameters) - each augmented with Random Forest residual learning. Through systematic ablation across 10 major technology stocks (NVDA, MSFT, AAPL, GOOG, GOOGL, AMZN, META, AVGO, TSLA, NFLX) spanning 2 million data points, we reveal critical insights: (1) The hybrid neural-classical approach achieves 0.597 pooled correlation and 6.4x mean per-day correlation improvement over frozen TimesFM; (2) Classical residual learning (Random Forest) provides the largest single-component contribution, matching or exceeding the neural correction component; (3) Simpler neural architectures surprisingly outperform complex ones when classical residual learning is removed; (4) Self-attention provides the largest neural-only contribution. GatedLinear+RF achieves best overall performance with 9x fewer neural parameters than AttnCorrect+RF. We report three complementary correlation metrics - mean per-day, cross-day cumulative, and pooled - to provide a complete picture of predictive quality. Our results provide practical guidance: effective foundation model adaptation requires careful integration of neural and classical components, with classical methods playing a crucial complementary role.

Kasun Dewage, Suranadi De Silva, Shankhadeep Mondal

Dept. of Mathematics University of Central Florida Orlando, FL, USA · Dept. of Computer Science University of Central Florida Orlando, FL, USA

Residual Learning Time Series Forecasting Financial Time Series Time-Series Foundation Models

Aug 9, 2026cs.CV

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition

Fine-grained facial expression recognition (FER) hinges on capturing subtle muscular cues that distinguish adjacent emotions. Yet capturing these cues presents a dilemma. Detector-based methods depend on fragile landmark pipelines, whereas we find that directly transferring foundation models such as DINOv3 under conventional global readouts can cause local evidence dilution: early global aggregation washes out sparse muscular signals and leaves persistent confusion between categories such as fear/surprise and sad/neutral. To recover this evidence, we propose SRE-FER, a readout-level regional residual evidence learning framework. Its core module, RERA, adds zero-initialized residual logits that refine class boundaries while preserving the backbone's global prediction. Training-time action unit (AU) guidance steers regional features toward expression-relevant areas using Facial Action Coding System (FACS)-based anatomical priors, without requiring an external facial pipeline at inference. An optional Full setting further routes sample-specific non-redundant tokens. On three benchmarks, SRE-FER attains 92.76% on RAF-DB, 91.32% on FERPlus, and 67.78% on AffectNet-7, demonstrating highly competitive performance compared to existing FER methods.

Jiaye Song, Ruochen Zhang, Yuliang Wang +1

Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China

Residual Learning Facial Expression Recognition

Aug 9, 2026physics.ao-ph

Deep Learning-Based Statistical Downscaling of Sea Surface Temperature Using a Residual Corrective Neural Network

The large-scale oceanic and atmospheric forecasts provided by global climate models typically lack sufficient resolution to accurately capture the response of the coastal ocean to atmospheric forcing and coastal circulation that drive fine-scale SST variability. Dynamical downscaling is computationally prohibitive, when applied to extensive coastlines, predictive ensembles, or long time periods. Therefore, this work presents a statistical downscaling of sea surface temperature (SST) from the seasonal coupled ocean-atmosphere forecast system (ACCESS-S2) using machine learning techniques. This study proposes a novel deep learning framework that uses a U-Net to generate an initial high-resolution SST estimate, which is subsequently refined using a residual corrective approach. The target SST fields are derived from the Regional Ocean Modeling System (ROMS). This two step approach called Residual Corrective Neural Network (RCNN) progressively refines initial U-Net predictions by incorporating dynamically scaled residuals at each step, enabling accurate capture of broad patterns and fine-grained features such as eddies and fronts. We also introduce a custom loss-assisted RCNN variant to improve performance during extreme events, which may be absent from training data due to climate-driven shifts in SST extremes. The framework efficiently downscales SST along the west coast of Australia. A 2011 marine heatwave case study shows that the RCNN improves ACCESS-S2 SST predictions by increasing horizontal resolution from 25 km to 2 km, enabling identification of fine-scale anomalies unresolved in the ACCESS-S2 dataset. This balance between computational efficiency and accuracy supports applications in coastal impact assessment and marine ecosystem studies.

Onkar Jadhav, Tim French, Ivica Janekovic +2

School of Earth and Oceans, UWA Oceans Institute, University of Western Australia, Crawley, WA, Australia. · School of Physics, Maths and Computing, Computer Science and Software Engineering, University of Western Australia, Crawley, WA, Australia. · School of Engineering, UWA Oceans Institute, University of Western Australia, Crawley, WA, Australia.

Climate Science Residual Learning Convolutional Neural Networks Climate Downscaling

Aug 8, 2026cs.LG

TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold-constrained (mHC) variants stabilize training through doubly stochastic residual mixing. However, a generator-level bottleneck remains in existing methods: they use dense, unstructured generators for pre-branch aggregation, residual mixing, and post-branch redistribution, which results in parameter count growing rapidly with the number of streams. To address this issue, we propose \underline{\textbf{T}}ensorized \underline{\textbf{E}}fficient \underline{\textbf{M}}anifold-constrained \underline{\textbf{P}}arameterization for \underline{\textbf{E}}xpressive Residual \underline{\textbf{R}}outing (\textbf{TEMPER}), which represents these generators as multi-way tensors over the input-stream, feature, and output-stream modes, and parameterizes them using tensor networks. Such a structured low-rank formulation is shown to preserve token-dependent manifold-constrained routing interface while substantially reducing parameter growth. It also promotes interpretability and intuition, as: i) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii) the generator approximation errors bound differences in routing logits and, consequently, in the routed-block outputs. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters. At eight residual streams, TEMPER achieves the best CORE score while using about

84\%

fewer additional parameters than mHC, thus showing a stronger performance-parameter efficiency trade-off.

Yuxuan Gu, Wuyang Zhou, Huijun Xing +1

Department of Electrical and Electronic Engineering, Imperial College London

Residual Learning Hyper-Connections

Aug 7, 2026cs.LG

Residual Algebra for Representation-Preserving Learning

Learning from heterogeneous representations is usually reduced to feature concatenation, which erases which representation produced an error. We instead algebraize the residual: a representation is a typed object that owns both a coordinate system and the residual it leaves unresolved, and learning is an ordered composition of operators that preserve or deliberately erase that type. Fold realizes the objects as point-in-time conditional-mean fields on 10x10 rank grids. FPRC-PQ realizes the algebra as relax-aggregate-close: each field is relaxed by a correction fitted to its own residual in its own coordinates; corrected fields meet at a fixed mean that is the sole identity-erasure boundary; and a shared learner closes only the aggregate's fresh residual. The composition telescopes exactly into representation, local residual estimate, and residual-of-residual estimate. Its aggregate is a learned control-variate interface with population variance reduction, while refitting the closer along perturbations of the backbone yields first-order coupled-path mean orthogonality. As an analytical extension, a reflective rumination operator reads the displacement of a global reconstruction from the aggregate anchor, reflects it, and fixes its gain by a unique orthogonal projection rather than return-tuned grid search. On 3.67M Chinese A-share stock-day observations (2023-2026) under a frozen point-in-time protocol, the evaluated base algebra raises net-of-cost return from 13.52% to 19.10% and Sharpe from 1.42 to 2.09. Matched-capacity, unified-residual, identity-free two-stage, and pairwise-only controls all trail it. The gain is therefore not explained by more features or more trees, but by making residual ownership and composition explicit while representation identity is still available.

Yao Wu

Representation Learning Quantitative Finance Residual Learning

Aug 6, 2026cs.SD

Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

Residual connections are fundamental to deep speaker recogni- tion models, such as ECAPA-TDNN and ResNet. However, standard identity mapping limits information flow to a sin- gle path, constraining representation capacity. We introduce Manifold-Constrained Hyper-Connections (mHC), reformulat- ing residual paths as a multi-stream evolution where informa- tion is mixed through a doubly stochastic matrix. By employing Sinkhorn-Knopp iterations, mHC ensures energy conservation by preserving signal intensity and feature mean, which stabi- lizes gradients and mitigates signal degradation in complex net- works. We evaluate mHC by replacing standard residual con- nections in backbones including ECAPA-TDNN, ResNet-34, Res2Net, and E-Res2Net. Extensive experiments on VoxCeleb1 demonstrate that mHC connections consistently enhance per- formance across all architectures, highlighting its effectiveness for robust speaker representation learning.

Zezhong Jin, Xiaoyu Wang, Zhe Li +4

Dept. of EEE, The Hong Kong Polytechnic University · Baidu Inc. · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR

Audio Representation Learning Residual Learning Hyper-Connections Speech Processing

Aug 5, 2026cs.LG

When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasters

Frozen pretrained forecasters often fail in structured, recurring ways that are costly to repair through fine-tuning. We study corrective feature discovery: mining interpretable features of a frozen forecaster's residual to drive a lightweight post-hoc corrector. Prior automated feature engineering models the data-generating process; corrective features instead model the model-failure process. We present CRAFTER (Corrective Residual Agent with Feature-based Temporal Exploration and Reasoning), which keeps the backbone frozen and mines its residual with two complementary generators: a compositional search over the raw input channels, and a large language model (LLM) that proposes named feature combinations, binary flags, and short executable code. A single validation-grounded gate accepts or rejects every candidate regardless of its origin, and a validation-selected corrector applies the accepted features or leaves the forecast unchanged. This source-agnostic pipeline also allows prior feature-engineering systems to be evaluated under identical conditions, making CRAFTER an instrument for attributing forecast improvements to the feature source alone. Across six public datasets and six frozen backbones, CRAFTER surpasses every dedicated feature-engineering system at every feature budget, roughly doubling the improvement achieved by the corrector alone and reducing the error of the weakest backbones by up to 27%. These gains are robust across different LLM backends and persist even when applied on top of fine-tuned backbones.

Fangxin Wang, Ziyi Zhang, Diyi Zhuang +4

University of Illinois Chicago · Texas A&M University · Massachusetts Institute of Technology +1

Residual Learning Time Series Forecasting Automated Feature Engineering

Aug 5, 2026cs.LG

When Proxy Prediction Becomes Equation Reconstruction: Diagnostics and Residual Learning for Factor-Derived Proxy Supervision

Scientific machine learning often relies on proxy targets computed from known domain factors when direct observations are limited. When those same factors are used as model inputs, however, high predictive accuracy may reflect reconstruction of the proxy-generating equation rather than robustness to degraded factor information. We study this problem in RUSLE-derived soil-loss proxy prediction under controlled degradation of the soil-erodibility factor

K

. We introduce a diagnostic framework that combines degraded-formula references, classical tree-based baselines, matched direct and formula-feature predictors, contextual ablations, tail-error analysis, and degradation robustness scoring. We then propose RASPL, a formula-preserving residual framework that retains the degraded formula estimate as the prediction anchor and learns an adaptively gated contextual correction. RASPL substantially outperforms matched direct prediction and provides stronger degradation and tail robustness than treating the formula estimate as an ordinary input feature. Within RASPL, a compact statistical encoder achieves the highest macro-averaged

R^2

and lowest computational cost, whereas a convolutional encoder achieves the strongest degradation robustness and lowest Tail95 mean absolute error (MAE). These results establish formula preservation as the central design principle for robust learning from factor-derived proxy targets.

Chayan Lahiri, Ahmed Shafee, Cody Fehringer

Department of Geosciences, Adams State University, Alamosa, Colorado, USA · Department of Computer Science, Adams State University, Alamosa, Colorado, USA · Devils Staircase Wildland Fire Module, Siuslaw National Forest, US Forest Service

Distribution Shift Robustness Residual Learning Scientific ML

Aug 3, 2026cs.CV

HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams

Much of the recent progress in image and video recognition has come at the cost of memory: larger models, increased resolution, and longer temporal contexts. An inevitable component is the quadratic (or larger) growth of memory and compute based on image resolution, which is a property of the grid sampling used in convolutional networks and vision transformers. In this work we study residual networks whose convolutional blocks have logarithmic-square growth instead, enabling them to process very high-resolution video quickly. The key insight is to use a residual architecture's residual stream as a high-resolution buffer, to which convolutional blocks only read and write via log-polar image warp operations. Layers adaptively focus on different parts of each frame, with very high resolution only near the focus point. A complete high-resolution representation is built up in the residual stream, analogous to eye saccades creating a complete picture in biological vision, and a theoretical construction is presented that eliminates the quadratic dependency of the residual stream resolution. Experiments demonstrate that our proposed HiResNets learn to foveate around scenes similarly to human vision, and have superior performance in difficult egocentric video recognition tasks, especially egocentric video with small objects and fine-grained recognition.

Shivani Mall, Swarnim Jain, Joao F. Henriques

Visual Geometry Group, University of Oxford

Residual Learning Active Visual Perception

Aug 2, 2026cs.LG

Sphere Retraction Normalizations

Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant

\ell_{2}

-norm, confining the residual stream to a hypersphere. The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework. Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions,

p

-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at

p = 1

and

p = 2

, while the identity map and GeoNorm arise only as limits at either end. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite

p

, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum.

Jie Zhang, Cheng-Fang Su, Yi-Jui Huang +1

Department of Computer Science and Information Engineering, National Central University, Taiwan · Department of Applied Mathematics, National Yang Ming Chiao Tung University, Taiwan · Department of Mathematics, National Central University, Taiwan

Residual Learning Riemannian Geometry

Jul 29, 2026cs.LG

TREA-Net: A Transferable Residual Epidemiological Adaptation Network for Dengue Incidence Forecasting

Accurate multi-week dengue forecasting supports timely vector-control interventions, outbreak preparedness, and healthcare resource allocation. However, newly established surveillance systems often lack the historical data needed to train reliable neural forecasting models. Although pretrained time-series models offer promising zero-shot forecasts, their cross-domain training may not capture local epidemiological dynamics. We propose TREA-Net, a Transferable Residual Epidemiological Adaptation Network for dengue forecasting under limited data. TREA-Net augments neural forecasting backbones with projections from an Environmental Time-Series Susceptible-Infected-Recovered model and learns a lightweight gated residual correction transferable from data-rich to data-scarce regions. Its node-invariant design accommodates surveillance systems with different numbers of locations, while target adaptation requires learning only two global parameters. We transfer knowledge from long-running dengue surveillance in Colombia and Nicaragua to 8-week-ahead forecasting in Mexico and Malaysia using only 78 or 104 weeks of target data. Across five neural backbones and ten transfer settings, TREA-Net improves the corresponding backbone in 9 out of 10 settings, with statistically significant gains. When integrated with TiRex, a foundation model for forecasting, it achieves the lowest mean absolute error across all target datasets. Conformal prediction further maintains empirical coverage while reducing 8-week prediction-interval width by 29.6% in Mexico. These results demonstrate TREA-Net's potential as a lightweight and portable early-warning framework for health agencies with limited surveillance data.

Inesh Shukla, Madhurima Panja, Tanujit Chakraborty +1

International Institute of Information Technology Hyderabad, India · SAFIR, Sorbonne University Abu Dhabi, United Arab Emirates · Sorbonne Center for Artificial Intelligence, Sorbonne Universit´e, Paris, France

Epidemic Forecasting Residual Learning Time Series Forecasting

Jul 27, 2026cs.LG

Attribution and Uncertainty Behavior of Learned Residual Gyro Correction for Gyro-Stellar Estimation

This work investigates uncertainty decomposition and explainability in a deep learning-based framework for gyroscope bias correction. A 1-D Convolutional Neural Network is trained to predict residual angular rate corrections from multi-sensor inputs, including gyroscope and star tracker measurements. The bias corrections are sent to a flight-representative Gyro-Stellar Estimator. The network produces both mean corrections and input-dependent (heteroscedastic) aleatoric uncertainty, while epistemic uncertainty is estimated via an ensemble of independently trained models. The proposed approach is trained under nominal conditions and evaluated in both nominal and structured perturbations that include additive and temporally correlated noise. Gradient-based attribution methods are applied to both the correction and uncertainty outputs, enabling a decomposition of the evidence that drives state updates and uncertainty estimates. By aggregating attribution patterns across rotational axes and regimes, we reveal axis-specific behaviors and characterize how structured perturbations influence the collaboration between aleatoric and epistemic uncertainty. Uncertainty analysis shows that aleatoric uncertainty increases with perturbation intensity, but the distributions overlap and the calibration is not consistent across regimes. On the other hand, epistemic uncertainty gives a clear signal that gets clearer as the distributional shift happens, showing that the models disagree more. These results show that aleatoric and epistemic uncertainty work well together and that epistemic uncertainty is better at distinguishing between nominal and perturbed operating conditions. The results provide insight into the behavior of hybrid learning-based state estimation components and motivate the use of uncertainty for downstream monitoring and fault detection.

Mariela De Lucas Álvarez, Melvin Laux, Arthur de Freitas Precht +4

Robotics Innovation Center, German Research Center for AI (DFKI GmbH), Bremen, Germany · Airbus Defence and Space GmbH, Immenstaad, Germany

Residual Learning Aleatoric Uncertainty Inertial Navigation Epistemic Uncertainty

Jul 27, 2026cs.LG

Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Jinhao Zhang, Zeyu Liu, Zicheng Yan +4

Beijing University of Posts and Telecommunications · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China +1

Neural Network Generalization Residual Learning Neural Scaling Laws Gradient Alignment

Jul 23, 2026cs.IR

Probabilistic Residual Learning for Online Recommendations

Modern recommender systems are typically based on deep learning (DL) models, where a dense encoder learns representations of users and items. As a result, these systems often suffer from the black-box nature and computational complexity of the underlying models, making it difficult to systematically enhance their recommendation capabilities. To address this problem, we propose Probabilistic Residual Learning (PRL), a causal Bayesian recommendation model that models the residual between ground-truth and base predictions, enabling targeted refinement of existing systems. Specifically, PRL (1) probabilistically groups users for localized residual modeling, (2) models domain-level confounders that influence user and item representations, and (3) aggregates cluster-specific residual predictions over the confounders using do-calculus. Experiments demonstrate that our plug-and-play PRL is compatible with various base deep learning recommender systems, improving their performance while automatically discovering meaningful user clusters.

Wenyuan Wang, Yusong Zhao, Zihao Xu +11

Rutgers University · Piscataway, New Jersey, United States · Meta +4

Residual Learning Recommender Systems Causal Inference

Jul 22, 2026physics.comp-ph

Dynamical and Optimization Trade-offs of Levi--Civita Coordinates for Learned Close-Encounter Dynamics

Classical regularization removes the binary-collision singularity from the Kepler problem, but its value as a representation for learned Hamiltonian dynamics has not been systematically isolated. We compare Cartesian and planar Levi--Civita formulations of a perturbed Kepler system with a smooth quadrupole potential. With the perturbation supplied analytically, a Levi--Civita Hamiltonian splitting holds the maximum relative energy error near

2.1\times10^{-5}

through eccentricity

e=0.99

, while the Cartesian splitting becomes unstable. This advantage persists at matched physical horizon and force-evaluation budget, where the regularized baseline is

3\times10^{-5}

, about

4.7

--

8.3

orders of magnitude below the Cartesian arm depending on eccentricity. In held-out high-eccentricity tests with matched sampling, regularized models produce finite rollouts in

40/40

runs versus

0/40

for Cartesian. However, the fixed-shell construction supplies the regularized model with the exact initial orbit energy, and survival still carries

\mathcal{O}(1)

energy error. Four neural residual objectives fail to approach the analytic result. Exact-feature controls show that the regularized residual is a four-monomial degree-6 polynomial that a direct least-squares solve fits to the baseline. The remaining exact-feature gap is due to severe raw-basis ill-conditioning: orthogonalization restores baseline fitting for L-BFGS in two iterations. Small MLPs remain at

\mathcal{O}(1)

rollout error even after gauge symmetrization. Levi--Civita coordinates therefore improve dynamical conditioning while worsening raw-basis optimization conditioning; accurate neural residual learning remains unresolved. This is a controlled falsification-plus-trade-off study, not a solution to learned close-encounter dynamics.

Abhishek Shankar

Residual Learning Hamiltonian Dynamics

Jul 21, 2026cs.CL

Dual Attention Residuals

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Xingda Yu, Yining Li, Xinzhang Liu +5

TeleAI

Transformer Cross-Attention Attention Residual Connections Residual Learning

Jul 16, 2026cs.LG

Certifying Residual Architectures from Their Primitives: A Sharp Stability Threshold

Whether a deep residual architecture trains stably is usually determined by training it, which is expensive and answers the question only for the architecture, depth, and floating-point format tested. In practice, stability is secured by heuristics for where to place normalization and which type to use, supported by experiments but lacking a common principle. We show that stability can instead be certified before training, directly from the architectural primitives of a residual block, as an explicit function of depth and floating-point format. The certificate has two elements: (a) a power-law growth bound,

|v(x)|\le c|x|^q+b

, whose exponent

q

is computed from the block's primitives by an arithmetic of exponents (forward); and (b) gradient bounds from a Lipschitz condition on the block over reachable states (backward). The certificate determines when the state can reach the largest finite floating-point value

M

: for

q\le1

, overflow requires at least order

\log M

layers, whereas for

q>1

it can occur within order

\log_q\log M

layers; the threshold

q=1

is sharp. Every normalization and bounded activation sets

q=0

, and the arithmetic identifies minimal modifications that bring a block from

q>1

to

q=1

without normalization. Empirically, in forecasting, only

q>1

blocks blow up, as predicted by the forward certificate. In GPT-2 on OpenWebText,

q=1

blocks without normalization also diverge: the forward certificate holds, while the backward one fails in the attention block with the largest query-key coefficient. Relaxing normalization from

q=0

to

q=1

improves out-of-distribution generalization in operator learning and accuracy in time-series forecasting. As deep models become more complex and costly to train, our results provide a way to assess the trade-off between stability and representational flexibility directly from block primitives, before training.

Hyemin Gu, Michael Tyrrell, Tuhin Sahai +1

University of Massachusetts Amherst · SRI International

Neural Network Verification Residual Learning Neural Network Training Dynamics

Jul 15, 2026cs.LG

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank. We show that skip connections trade off rank collapse against ensemble-like behavior, controlled by the relative scales of the branch and the skip: skip connections route the gradient around the residual branch, where rank is lost, rather than along the long gradient paths that encourage the layers to compose. The placement of the normalization layer controls this same tradeoff by setting the branch-to-skip ratio across depth, unifying much of the normalization placement and depth scaling literature, in particular why rank collapses for Post-Norm but plateaus for Pre-Norm. Other aspects of the architecture, like the two-matrix structure that expands and contracts the width, use additional parameters to preserve the representation or branch Jacobian rank. The second matrix decorrelates a coherent mean spike that would grow across blocks with a single matrix and uncentered activation, preventing the residual representation from collapsing. The width expansion between the two matrices keeps the branch Jacobian full rank: applying the rank-reducing activation in this expanded space leaves enough directions to span the original, at a width that follows a Marchenko--Pastur law. The initialization rank of the input--output Jacobian predicts which networks train on CIFAR-10. Taken together, we recast architecture design for deep networks as navigating an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.

Katie Everett

MIT CSAIL

Transformer FFNs Transformer Neural Network Initialization Residual Learning Layer Normalization

Jul 15, 2026cs.LG

DeepLoop: Depth Scaling for Looped Transformers

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient

κ_R

. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from

1/4

to

1/2

as loop count grows at fixed physical depth. The resulting method, \textbf{DeepLoop}, keeps the Post-LN DeepNorm architecture and sets

α=(2N)^{1/2}

and

β=(8N)^{-1/2}

for unrolled depth

N

. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.

Shuzhen Li, Yifan Zhang, Jiacheng Guo +2

Princeton University · University of California, Los Angeles

Residual Learning Recurrent Transformers

Jul 15, 2026cs.RO

Learning Physics-Guided Residual Dynamics for Deformable Object Simulation

Simulating deformable objects is essential for a wide range of robotic manipulation applications, yet accurately predicting their dynamics remains challenging. We propose Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches. Specifically, PGRD combines an optimizable spring-mass simulator as a backbone with a learned neural network that predicts residual corrections to the physics-based predictions. We adopt a velocity-based formulation to ensure stable simulation and a sliding-window transformer architecture to capture temporal dependencies. We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects. We further demonstrate the utility of PGRD in two applications: manipulation planning via Model Predictive Control, including a language-conditioned setting with a generated goal image; and interactive simulation via action-conditioned video prediction by 3D Gaussian Splatting.

Shivansh Patel, Kaifeng Zhang, Sanjay Pokkali +2

UIUC · Columbia University

Residual Learning Residual Dynamics Learning Deformable Object Manipulation Physics-Based Simulation Deformable Object Simulation

Jul 13, 2026cs.AI

Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learning

Decoding continuous three-dimensional (3D) motor imagery (MI) using non-invasive electroencephalography (EEG)-based brain--computer interfaces (BCIs) remains challenging due to signal variability and residual decoding errors. Deep learning architectures such as convolutional neural network--long short-term memory (CNN--LSTM) models can capture spatial and temporal dynamics for continuous kinematic decoding; however, systematic residual errors persist in predicted trajectories. We propose a two-stage decoding framework that applies reinforcement learning (RL) to perform residual kinematic correction on the outputs of a CNN--LSTM decoder (CNN--LSTM--RL). The RL agent is trained offline without direct EEG input and instead operates on predicted kinematic trajectories to optimize movement accuracy relative to target trajectories. Decoding performance was quantified using Pearson correlation coefficients (

r

) and Root Mean Square Errors (RMSE) along the

x, y

, and

z

axes. Compared to CNN--LSTM applied alone, CNN--LSTM--RL improved the mean correlation from

0.5076

to

0.7181

(

p = 0.0005

) in 2D and from

0.6420

to

0.7780

(

p = 0.0059

) in VR, with relative gains of

41.5\%

and

21.2\%

, respectively. Correspondingly, RMSE was reduced from

0.0890

to

0.0532

(2D,

p < 0.0001

) and from

0.0714

to

0.0441

(VR,

p < 0.0001

), representing relative reductions of

40.2\%

and

38.2\%

. These findings demonstrate that this scalable framework enhances 3D BCI MI decoding by correcting kinematic errors via offline residual RL without extra neural data, advancing neurorehabilitation, prosthetics, and virtual interaction.

Jiamian Li, Niall McShane, Attila Korik +6

Department of Computer Science, University of Bath, Bath, BA2 7AY · Intelligent Systems Research Centre, University of Ulster, BT48 7JL · Bath Institute for the Augmented Human, University of Bath, Bath, BA2 7AY +1

Brain-Computer Interfaces Neural Decoding Residual Learning EEG Decoding RL Post-Training

Jul 11, 2026cs.RO

Diffusion-Residual Model Predictive Steering Control for Vehicle Stabilization at the Limit of Handling under Model Uncertainty

At the limit of handling, a stabilizing MPC depends on the yaw-rate reference it tracks and the stable-handling envelope it enforces, both operating-point-dependent and unknown a priori, so fixed or worst-case settings are either too conservative or unsafe. We learn this uncertainty with a conditional diffusion residual model and apply it to the controller's reference and constraints rather than its control law. Conditioned on the steering command, the model returns the residual's mean and a predictive spread: the mean re-sizes the tracked yaw reference, while the spread, propagated over the prediction horizon, tightens the stable-handling envelope through a one-sided chance back-off. Together these form the proposed diffusion-residual MPC (D-res), so caution is anticipated ahead of the tracking error rather than corrected after it by a high-gain loop. Because only two moments per command are needed, the generator is tabulated offline and the online controller adds a single table lookup to the baseline MPC, with no in-loop diffusion; it runs within the 100 Hz budget on an NVIDIA Jetson AGX Xavier (worst-case 4.08 ms per step). Across a 7-DOF model and high-fidelity CarMaker co-simulation spanning vehicle, tire, road, and maneuver diversity, D-res reduces peak side-slip where the fixed bicycle model is least accurate and restores directional stability on low-friction maneuvers, where the fixed reference over-commands the available grip.

Bongsob Song

Department of Mobility Engineering, Ajou University, Suwon, South Korea

Residual Learning Model Predictive Control Vehicle Dynamics Feedback Control

Jul 8, 2026cs.LG

Physics-Informed Machine Learning Under Small-Data Constraints: Lessons from Abrasive Waterjet Milling

In physically dominated machining processes, experimental datasets are small, expensive, and material-specific; in this regime, data curation, evaluation design, and the form of physics integration can matter as much as the learning algorithm. Using an abrasive waterjet milling dataset (

n{=}155

, Inconel,718), we make three methodological contributions. First, we separate physics-based data \emph{cleaning} from statistical \emph{curation} and treat the latter as competing modelling hypotheses rather than silent preprocessing. Second, we find that model rankings from a 15-point hold-out set can be unstable: the single-split winner drops from rank~~1 to rank~~7 under 10-fold cross-validation, while Gaussian Process (GP) variants occupy the top ranks. Third, we study a spectrum of physics integration levels and find that residual learning on a compact physics baseline is competitive for GP, yielding lower variance and an interpretable decomposition, but degrades tree-based models. Bayesian hyper parameter tuning improves parameter-sensitive baselines such as gradient boosting and SVR, yet harms multi-stage hybrid pipelines at this sample size. GP uncertainty intervals are approximately calibrated (

86\%

empirical coverage at nominal

90\%

). The resulting picture is methodological: for small, expensive process datasets, our results suggest that, in this setting, reliable model comparison benefits from explicit curation hypotheses, robust evaluation, and careful choices about how physics enters the model.

Sarah Grewe, Jörg Frochte

Bochum University of Applied Sciences Germany

Surrogate Modeling Residual Learning Physics-Informed ML Gaussian Process Regression Data-Efficient Learning

Jul 6, 2026stat.ML

Wasserstein Residuals: Learning Gradient Flows from Population Dynamics

Reconstructing population dynamics is a central problem in the physical and data sciences. Often, the dynamics are modeled as a Wasserstein gradient flow (WGF): a curve of distributions driven by an energy functional. Though there are multiple mathematical characterizations of a WGF, the dominant algorithmic approach relies on the Jordan--Kinderlehrer--Otto (JKO) scheme. JKO-based methods are inflexible to time discretisation and require solving costly optimal transport problems. We take a residual approach, enforcing the continuity equations via a non-negative loss function whose minimum is the WGF. Combined with a data-fitting divergence, this gives a single global objective. This perspective unifies several existing methods and leads to a new particle-based method, stitching, that is simulation-free and robust to large gaps between observations. We demonstrate that the stitching method achieves state-of-the-art performance across trajectory inference benchmarks. For code see github.com/BasisResearch/wasserstein-residuals.

Markus Heinonen, Yair Shenfeld, Ricardo Baptista +4

Basis Research Institute · Aalto University · Brown University +2

Dynamical Systems Wasserstein Gradient Flows Residual Learning Trajectory Inference Optimal Transport

Jul 6, 2026cs.LG

Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions

d_x

and

d_y

, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve

L^p

approximation

(1\leq p <+\infty)

on any compact domain, we show that the exact minimum block width is

\max\{d_x,d_y\}

when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width

\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}

can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than

\max\{d_x, d_y\}

, both in the

L^p

sense and the uniform sense, regardless of inner width.

Qi Zhou, Xuan Zhou, Xiao-Song Yang

School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.

ReLU Neural Networks Residual Learning Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Jul 2, 2026cs.CV

Interpretation-Oriented Cloud Removal via Observation-Anchored Residual Flow with Geo-Contextual Alignment

Cloud removal (CR) is essential for optical remote sensing, serving as a prerequisite for reliable downstream interpretation, such as semantic segmentation and change detection. However, existing CR approaches often prioritize visual realism while overlooking their impact on subsequent analytical tasks, leading to semantic drift and degraded downstream performance. To address this issue, we propose Geo-Anchored Cloud Removal (GACR), a unified framework that jointly ensures faithful reconstruction and robust interpretability. At its core, GACR incorporates Observation-Anchored Residual Flow (OAR-Flow), which reformulates CR as a physically grounded residual inversion process. By anchoring the generative trajectory to the cloudy observation rather than pure noise, OAR-Flow enables fast, stable, and faithful reconstruction. To further preserve semantic structures critical for downstream interpretation, GACR integrates Geo-Contextual Prior Alignment (GCPA) to constrain the reconstruction within a semantic manifold induced by a Vision Foundation Model (VFM). Consequently, GACR strictly maintains the spatial-semantic integrity of complex landscapes. Extensive experiments across six CR datasets and twelve downstream tasks demonstrate that GACR produces superior reconstruction quality while consistently improving downstream task accuracy. The code is available at https://github.com/wzy6055/GACR.

Ziyao Wang, Maonan Wang, Yucheng He +5

School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China · Shanghai Ai Lab, Shanghai, China · Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu,2026 China

Residual Learning Image Restoration

Jun 30, 2026cs.LG

Review Residuals: Update-Conditioned Residual Gating for Transformers

Residual connections add every sublayer's proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it. Drawing on the human-factors principle of independent verification, we introduce Review Residuals, which scale each update by a learned, input-dependent gate conditioned on both the current state and the proposed update: h_l = h_{l-1} + r_l * u_l with r_l = sigmoid(W[RMSNorm(h_{l-1}), RMSNorm(u_l)]). Conditioning the gate on the update is the property that distinguishes it from prior gated and scaled residuals. We report two findings. First, a depth-stability result: a convex (Highway-style) form of the gate reintroduces vanishing gradients and fails to train beyond ~20 layers, whereas the additive, identity-preserving form trains stably at all depths we tested. Second, an emergence-with-scale result: trained from scratch across five sizes (60M-1B parameters, multi-seed), Review Residuals show no advantage at small scale but at 590M significantly outperform both a parameter-matched Highway gate and a parameter-matched standard residual (p<0.05), with a larger advantage at 1B. The benefit grows with model size rather than shrinking.

Kyle Kramer

NeraTech LLC

Transformer Residual Learning Neural Network Training Dynamics

Jun 24, 2026cs.LG

Black-Box Assisted Regression: Phase Transitions and Minimax Optimality

Foundation models are often used as fixed black-box predictors for downstream tasks with limited labeled data, but their predictions may be biased and unsafe to trust blindly. We study this setting through black-box assisted nonparametric regression: a learner observes labeled samples and can query a fixed predictor

f_0

, while the target

f^*

is close to

f_0

in

L_2(P_X)

up to an unknown radius

δ

. We give a finite-sample minimax characterization showing a phase transition at

δ_c(n) \asymp n^{-β/(2β+d)}

, with leading risk

\min\{δ^2, n^{-2β/(2β+d)}\}

. We then analyze a Safe Residual Estimator: it learns a correction around

f_0

, initializes the residual head at zero so the initial predictor equals

f_0

, and uses holdout selection to revert to

f_0

when the learned correction is not supported by validation data. Here, "safe" means avoiding negative transfer, i.e., performing worse than the black-box predictor alone. The estimator matches the leading minimax term up to an additive validation-selection cost. Synthetic regression experiments verify the predicted phase transition, while CIFAR-100 with CLIP and AG News with Qwen3-8B provide practice-facing evidence that the same residual-correction tradeoff is useful beyond the formal squared-loss regression setting.

Yan Zhou

School of Mathematics and Statistics, Changsha University of Science and Technology.

Residual Learning Nonparametric Regression Minimax Estimation Learning-Augmented Algorithms

Jun 23, 2026cs.LG

Semantic Allocation in Ordered Bottlenecks: Predictive Residual Inference for Visual Representation Learning

Ordered bottlenecks aim to provide utility at flexible budgets by assigning coarse information to early tokens and task-relevant detail to later ones. Prior work, including tail dropping (TD), typically enforces ordering by means of a masking-based ordering pressure (MBOP): Late tokens are masked more frequently than early tokens and are therefore encouraged to store less essential fine details. We introduce predictive residual inference for ordered representations (PRIOR), a framework designed to address inherent weaknesses of MBOP. MBOP is prone to weak late-token utility because it lacks an explicit refinement objective and uses gradient exposure as a proxy for importance. Furthermore, representations may become particularly brittle in optimization-sensitive settings, such as when using discrete or quantized token representations. PRIOR replaces activation-rate control with log2-scaled levels and level-wise predictors. These predictors separate already explained from unexplained information, focusing each level on residual error. We compare PRIOR against MBOP-TD and independent tail-biased dropout (MBOP-ITD) in contrastive learning and image reconstruction tasks. Unlike the baselines, PRIOR learns well-ordered representations across experiments: low budgets provide coarse descriptors, while high budgets add refinements. Simultaneously, full-budget performance with PRIOR is higher in all but one experimental setting, where performance remains comparable. MBOP baselines are severely limited in discrete and quantized settings, while PRIOR approaches the performance of continuous counterparts. Taken together, these findings establish PRIOR as an effective framework for ordered representation learning.

Erik Ayari, Manuel Traub, Martin V. Butz

Neuro-Cognitive Modeling Group, University of Tübingen, Tübingen, Germany

Representation Learning Visual Representation Learning Residual Learning Predictive Representation Learning

Jun 19, 2026cs.LG

Low-Rank Attention Residuals

Attention Residuals (AttnRes) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes the cost of computing depth-routing scores scale with hidden width

d

. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using

r

-dimensional keys, with

r < d

, for routing. LR-AttnRes uses the last

r

dimensions of each value as the routing key, reducing total residual-side FLOPs while still improving performance. Comprehensive sweeps across the number of blocks (

N

) and

r

show that depth-wise routing can be effective with far fewer dimensions than the model width. At both

1

B and

4

B parameters with

r = d/4

, LR-AttnRes achieves lower final validation loss, higher average downstream accuracy, and higher measured training-step throughput than standard AttnRes. We also provide a fused kernel supporting standard and low-rank routing. We release all code, the kernel, and all trained models to facilitate future research.

Jonathan Su

Independent Researcher

Low-Rank Attention Decoder-Only Language Models Attention Residual Connections Residual Learning Attention Mechanisms