Residual Learning

Ask about this topic

Overview

Residual learning studies how models can build on an existing representation or prediction by adding learned updates, rather than learning the entire transformation from scratch. In deep networks, skip connections and newer approaches such as attention-based routing and multi-stream hyper-connections shape information and gradients across layers; in hybrid models, learned corrections refine physics-based, statistical, or frozen-model predictions. These methods are evaluated in settings including image classification, language modeling, diffusion, time-series forecasting, scientific simulation, and control. A central challenge is determining when residual pathways improve depth, stability, and generalization—and when their complexity, added compute, or poorly transferable corrections instead hurt performance.

Latest papers

CardsList

158 papers32 papers in the last four weeks, up 23% on the four weeks before. 0.2% of all new papers.

  1. The EΔΔ-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

    May 7, 2026Arash ShahmansooriTransformerResidual Learning

    Independent Researcher

  2. SATFormer: Learning When to Use Early Value Residuals in Transformers

    May 5, 2026Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1TransformerResidual Learning

    UC Santa Cruz · UC Berkeley

  3. RPBA-Net: An Interpretable Residual Pyramid Bilateral Affine Network for RAW-Domain ISP Enhancement

    May 5, 2026Yucheng Xin, Wu Chen, Xiang Chen +7Image EnhancementResidual Learning

    School of Information Science and Engineering, Shandong Normal University, Jinan, China · National University of Defense Technology, Changsha, China · Nanjing University of Science and Technology, Nanjing, China +2

  4. Robust Path Tracking for Vehicles via Continuous-Time Residual Learning: An ICODE-MPPI Approach

    May 5, 2026Shugen Song, Wenjie Mei, Chengyan ZhaoNonlinear MPCResidual Learning

    School of Automation, Southeast University, Nanjing 210096, China · School of Automation and the Key Laboratory of MCCSE of the Ministry of Education, Southeast University, Nanjing 210096, China · Graduate School of Computer Science and Systems Engineering, Kyushu Institute of Technology, 680-4 Kawazu, Iizuka-shi, Fukuoka, Japan

  5. Forecasting Medium-Horizon Alzheimer's Disease Progression: Residual Gap-Aware Transformers for 24-Month CDR-SB Change from ADNI Clinical and Biomarker Histories

    May 4, 2026Ran Tong, Tong Wang, Lanruo Wang +1Residual LearningAlzheimer's Disease

    Department of Mathematical Sciences, University of Texas at Dallas, Richardson, TX 75080, United States · Department of Plant Science and Landscape Architecture, University of Connecticut, Storrs, CT, United States · Naveen Jindal School of Management, University of Texas at Dallas, Richardson, TX 75080, United States +1

  6. Geometric and Spectral Alignment for Deep Neural Network I

    May 4, 2026Ziran Liu, Wei Wang, Jinhao Wang +5Residual LearningSpectral Methods

    Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai 200433, China · Futurewei Technologies, Inc., San Jose, CA 95131 · Dept. of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95050 +2

  7. Progressive Approximation in Deep Residual Networks: Theory and Validation

    Apr 27, 2026Wei Wang, Xiao-Yong Wei, Qing LiEfficient Neural Network InferenceResidual Learning

    Department of Computing, the Hong Kong Polytechnic University.

  8. Can an MLP Absorb Its Own Skip Connection Exactly?

    Apr 26, 2026Antonij Mijoski, Marko KarbevskiMultilayer PerceptronsResidual Learning

    IRMA, Universit´e de Strasbourg, France · Independent Researcher

  9. Dissipative Latent Residual Physics-Informed Neural Networks for Modeling and Identification of Electromechanical Systems

    Apr 20, 2026Youyuan Long, Gokhan Solak, Arash AjoudaniDynamical SystemsResidual Learning

    Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, 16163 Genoa, Italy

  10. Rethinking Cross-Dose PET Denoising: Mitigating Averaging Effects via Residual Noise Learning

    Apr 18, 2026Yichao Liu, Zongru Shao, Yueyang Teng +1Residual LearningPET/CT Imaging

    IWR, Heidelberg University, 69120, Heidelberg, Germany · Silicon Austria Labs, 4040, Linz, Austria · College of Medicine and Biological Information Engineering, Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, 110169, Shenyang, China +1

  11. SMART Fine-tuning Factor Augmented Neural Lasso

    Apr 14, 2026Jinhang Chai, Jianqing Fan, Cheng Gao +1Fine-TuningTransfer Learning

    Department of Operations Research and Financial Engineering, Princeton University

  12. Multi-Head Residual-Gated DeepONet for Coherent Nonlinear Wave Dynamics

    Apr 13, 2026Zhiwei Fan, Yiming Pan, Daniel CocaDeep Operator NetworksNeural Operators

    School of Engineering, Newcastle University, NE1 7RU, United Kingdom · School of Physical Science and Technology, ShanghaiTech University, Shanghai 200031, China

  13. Beyond Logit Adjustment: A Residual Decomposition Framework for Long-Tailed Reranking

    Apr 2, 2026Zhanliang Wang, Hongzhuo Chen, Quan Minh Nguyen +2Long-Tailed ClassificationLearning to Rank

    University of Pennsylvania, Philadelphia, PA, USA · Children’s Hospital of Philadelphia, Philadelphia, PA, USA

  14. Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

    Mar 30, 2026Christian Kuehn, Sara-Viola Kuntz, Tobias WöhrerUniversal ApproximationNeural Network Approximation Theory

    Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany +1

  15. GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

    Mar 23, 2026Soudeep Ghoshal, Himanshu BuckchashConvolutional Neural NetworksAttention Residual Connections

    Kalinga Institute of Industrial Technology, Bhubaneswar, India · IMC University of Applied Sciences Krems, Krems, Austria

  16. Spectral-Sphere-Constrained Hyper-Connections

    Mar 21, 2026Zhaoyi Liu, Haichuan Zhang, Ang LiHyper-ConnectionsResidual Learning

    University of Maryland, College Park · University of Utah

  17. Deep learning and the rate of approximation by flows

    Mar 16, 2026Jingpu Cheng, Qianxiao Li, Ting Lin +1Neural Network Approximation TheoryFunction Approximation

    Department of Mathematics, National University of Singapore, 117543, Singapore · Department of Mathematics and Institute for Functional Intelligent Materials, National University of Singapore, 117543, Singapore · School of Mathematical Sciences, Peking University, 100871, China

  18. Attention Residuals

    Mar 16, 2026Kimi Team, Guangyu Chen, Yu Zhang +34Attention Residual ConnectionsResidual Learning

  19. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

    Mar 16, 2026William Peng, Josheev Rai, Kevin Tseng +2TransformerHyper-Connections

    Stanford University · Georgia Institute of Technology · University of California, Berkeley +2

  20. Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators

    Mar 12, 2026Samaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh +3Aerial RoboticsTest-Time Adaptation

    1Robotics Research Center, IIIT Hyderabad, India. · Department of Computer Science, University of Manchester, UK. · 3Newcastle University, UK.

  21. SCORE: Replacing Layer Stacking with Contractive Recurrent Depth

    Mar 11, 2026Guillaume GodinResidual LearningRecurrent Neural Networks

    1Osmo Labs PBC New York, USA

  22. Mitigating Frequency Learning Bias in Quantum Models via Multi-Stage Residual Learning

    Mar 10, 2026Ammar DaskinQuantum Machine LearningQuantum Neural Networks

    Department of Computer Engineering, Istanbul Medeniyet University, Istanbul, Turkiye, 34000

  23. Observationally Informed Adaptive Causal Experimental Design

    Mar 4, 2026Erdun Gao, Liang Zhang, Jake Fawkes +5Active Experimental DesignCausal Inference

    Australian Institute for Machine Learning, Adelaide University · Hong Kong University of Science and Technology (Guangzhou) · University College London +3

  24. Bi-cLSTM: Residual-Corrected Bidirectional LSTM for Aero-Engine RUL Estimation

    Feb 28, 2026Rafi Hassan Chowdhury, Nabil Daiyan, Faria Ahmed +2RUL EstimationPredictive Maintenance

    Department of Computer Science and Engineering, Rajshahi University of Engineering and Technology, Rajshahi, Bangladesh · Department of Aeronautical Engineering, Military Institute of Science and Technology, Dhaka 1216, Bangladesh

  25. On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective

    Feb 24, 2026Jinshu Huang, Mingfei Sun, Chunlin WuNeural Network GeneralizationGeneralization Bounds

    School of Mathematical Sciences, Nankai University, Tianjin, China

  26. Scientific Knowledge-Guided Machine Learning for Vessel Power Prediction: A Comparative Study

    Feb 20, 2026Orfeas Bourchas, George PapalambrouPhysics-Informed MLResidual Learning

    Laboratory of Marine Engineering N.T.U.A.

  27. JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

    Feb 20, 2026Biswa Sengupta, Jinhua Wang, Leo BrunswicHyper-ConnectionsResidual Learning

    LLM Suite Team, JP Morgan Chase & Co.

  28. Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

    Feb 20, 2026Mohan Tang, Sidi LuTransformerLarge Reasoning Models

    UCLA.

  29. HPMixer: Hierarchical Patching for Multivariate Time Series Forecasting

    Feb 18, 2026Jung Min Choi, Vijaya Krishna Yalavarthi, Lars Schmidt-ThiemeMultivariate Time Series ForecastingLong-Term Time Series Forecasting

    ISMLL, University of Hildesheim

  30. Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers

    Feb 16, 2026Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4TransformerResidual Learning

    IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France · Sony Europe Ltd. Stuttgart Technology Center, EUREC, Germany