Model Re Basin

Model re-basin focuses on the surprising observation that, despite the non-convex nature of deep learning optimization, solutions often converge to a single basin of attraction when considering the permutation symmetries of hidden units. Current research investigates efficient algorithms, including those based on Sinkhorn differentiation and novel cost functions, to find optimal permutations aligning different models within this basin, facilitating model averaging and incremental learning. This work is significant because it provides insights into the structure of deep learning loss landscapes and offers potential improvements to model training and generalization, particularly through techniques like model merging and pruning.