cs.LGNov 3, 2026

SaMA: Morpho Adaptation via Asymmetric Expansion of Kronecker Product

Authors: An Nguyen, Anh Tong

Organizations: Department of Artificial Intelligence, Korea University

Abstract

Low-rank adaptation (LoRA) has attracted significant attention in the parameter-efficient fine-tuning (PEFT) landscape. However, its design suffers from two fundamental limitations. First, its invariance manifold induces flat directions in the loss landscape, creating an optimization bottleneck for adaptive optimizers. Second, its effective rank scales only linearly with the parameter budget. In this paper, we propose Scalable Morpho Adaptation (SaMA), a PEFT method grounded in a principled generalization of the Kronecker product. By relaxing the shared-block constraint in the perfect-shuffle factorization, SaMA spans from low to full rank and achieves a practical rank that scales quadratically with the parameter budget, up to full-rank. Furthermore, we show that its invariance manifold collapses to a diagonal group, making it structurally more optimizer-friendly. Empirically, SaMA consistently outperforms strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while being more parameter-efficient.

Explore similar work

CardsList
  1. SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

    May 20, 2026Yongkang Liu, Xing Li, Mengjie Zhao +7LLM Fine-TuningAdapter Tuning

  2. Parameter-Efficient Fine-Tuning with Learnable Rank

    Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRALow-Rank Adaptation

  3. Low-Rank Adaptation Redux for Large Models

    Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisAdapter TuningLow-Rank Adaptation