Mode Connectivity

Follow this topic Ask about this topic

Mode connectivity asks whether neural networks trained to different solutions can be joined by a path through parameter space without a large increase in loss. Because straight-line interpolation often encounters barriers, methods build curved paths using Bézier curves, architecture-aware constructions, or local geometry such as Hessian null spaces; theory also studies connectivity in wide ReLU networks. Researchers use these paths to examine optimizer effects and to support applications such as model merging, unlearning, and robust interpolation across classifiers, generative models, and vision-language models. A key tension is that a low-loss connection does not guarantee similar representations, privacy behavior, or robustness, and connectivity can depend on the architecture, optimizer, and task.

Latest papers

11 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Traversing the solution space of neural networks with Hessian Null Space Continuation
Sep 29, 2026Ann Huang, Mitchell Ostrow, Zhouyang Lu +3Mode Connectivity Neural Network Interpretability
Harvard University · Harvard Medical School · Kempner Institute +3
SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation
Sep 15, 2026Alejandro Calatrava, Xu Zhang, Ren WangMode Connectivity Neural Network Robustness
Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
Aug 31, 2026Chengzheyi Yao, Yongzhao Zhang, Yongding TianMode Connectivity
University of Electronic Science and Technology of China · Delft University of Technology
From the Loss Landscape to Diverse Feature Learning in Neural Networks
Aug 28, 2026David Aram YunisLoss Landscape Mode Connectivity
TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO Chicago, Illinois
Understanding Machine Unlearning Through the Lens of Mode Connectivity
Jul 27, 2026Jiali Cheng, Hadi AmiriLoss Landscape Mode Connectivity
University of Massachusetts Lowell
On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference
May 28, 2026Daniel Dold, Emanuel Sommer, Julius Kobialka +2Mode Connectivity Bayesian Inference
HTWG Konstanz · LMU Munich · Munich Center for Machine Learning (MCML)
Approximate Machine Unlearning through Manifold Representation Forgetting Guided by Self Mode Connectivity
May 20, 2026Weiqi Wang, Zhiyi Tian, Chenhan Zhang +2Mode Connectivity Machine Unlearning
Xi’an Jiaotong University Xi’an, Shaanxi, China · Southeast University Nanjing, Jiangsu, China · University of Technology Sydney Sydney, NSW, Australia
Unlocking the Potential of Continual Model Merging: An ODE Perspective
May 19, 2026Lihong Lin, Haidong KangMode Connectivity Ordinary Differential Equations
Northeastern University, Shenyang, China
Optimizer-Induced Mode Connectivity: From AdamW to Muon
May 11, 2026Fangzhao Zhang, Sungyoon Kim, Erica Zhang +2Mode Connectivity Neural Network Optimization
Stanford University
FreeMOCA: Memory-Free Continual Learning for Malicious Code Analysis
May 10, 2026Zahra Asadi, Haeseung Jeon, Sohyun Han +3Class-Incremental Learning Mode Connectivity
Division of Artificial Intelligence & Software Ewha Womans University · Department of Computer Science University of Texas at El Paso
From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks
Feb 19, 2026Saveliy BaturinShallow Neural Networks ReLU Neural Networks
Independent Researcher, Russia.

Sep 29, 2026J/K move · Enter open · S save

Traversing the solution space of neural networks with Hessian Null Space Continuation

Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman+2

Harvard University · Harvard Medical School · Kempner Institute+3

Mode Connectivity Neural Network Interpretability

Open paper AskPDF arXiv

On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network than any independently trained model with a different architecture or objective. In reinforcement-learning agents, HNC uncovers a distinct navigation strategy at comparable return and exposes reward hacking in an AI Safety Gridworld. Finally, HNC measures the local geometry of the solution set, showing how model size and task complexity shape its dimension and functional sensitivity. Our results show that a surprisingly large amount of representational diversity exists near a single trained solution, unseen by standard gradient-based optimization. HNC identifies and quantifies this diversity, opening new possibilities for mechanistic understanding of solution spaces and for model merging, editing, and fine-tuning.