Orthogonal Procrustes Alignment

Latest papers

11 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning
Oct 12, 2026Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha +2Orthogonal Procrustes Alignment Continual LoRA
Sony Research India
SuperAI: Merging Capabilities Across Heterogeneous Foundation Models
Oct 11, 2026Jingtong Dou, Chuancheng Shi, Wenhua Wu +4Orthogonal Procrustes Alignment Model Merging
University of Sydney Sydney, Australia · National University of Singapore Singapore
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2
Geometric Data Perturbation with Noisy-Anchor Alignment for Privacy-Preserving Collaborative Learning
Aug 19, 2026Keiyu Nosaka, Yamato Suetake, Yuichi Takano +2Orthogonal Procrustes Alignment Training Data Reconstruction Attacks
Graduate School of Science and Technology, University of Tsukuba, 1-1-1, Tennodai, Tsukuba, 305-8573, Ibaraki, Japan · Institute of Systems and Information Engineering, University of Tsukuba, 1-1-1, Tennodai, Tsukuba, 305-8573, Ibaraki, Japan · Center for Artificial Intelligence Research, Tsukuba Institute for Advanced Research (TIAR), University of Tsukuba, 1-1-1, Tennodai, Tsukuba, 305-8577, Ibaraki, Japan
High-Dimensional Procrustes Matching via Tree Counts
Jul 9, 2026Xiaochun Niu, Tselil Schramm, Jiaming XuOrthogonal Procrustes Alignment Low-Degree Polynomial Method
The Fuqua School of Business, Duke University, Durham NC, USA · Department of Statistics, Stanford University, Stanford, CA, USA
Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders
Jul 9, 2026Bendegúz Váradi, Zoltán KmettyOrthogonal Procrustes Alignment Transformer Interpretability
Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. · Department of Sociology, Faculty of Social Sciences, Eötvös Loránd University, Budapest, Hungary.
Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning
Jun 2, 2026Aditya Sharma, Christopher J. Pal, Amal ZouaqOrthogonal Procrustes Alignment Neural Representation Geometry
Polytechnique Montréal · Mila - Quebec AI Institute · LAMA-WeST Lab +1
Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer
May 27, 2026Jungyong Son, Jinwook Jung, Minhee Park +1Orthogonal Procrustes Alignment Task Vectors
Dept. of Artificial Intelligence, 2 Dept. of Data Science Hanyang University
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannOrthogonal Procrustes Alignment Transformer Interpretability
Independent Researcher
DiffATS: Diffusion in Aligned Tensor Space
May 10, 2026Jinhua Lyu, Tianmin Yu, Brian Kim +3Orthogonal Procrustes Alignment Tensor Decomposition
Northwestern University
COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
Feb 16, 2026Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip +3Orthogonal Procrustes Alignment Model Compression
Fundamental Research Center MWS AI · ITMO

Oct 12, 2026J/K move · Enter open · S save

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala+1

Sony Research India

Orthogonal Procrustes Alignment Continual LoRA Continual Learning for LLMs Low-Rank Adaptation Parameter-Efficient Fine-Tuning Orthogonal Representation Learning

Open paper AskPDF arXiv

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.