cs.LGSep 25, 2026

New LoRA Skills Should Read but Never Write

Authors: Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu

Organizations: Shanghai Jiao Tong University · Xi’an Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou) · Jinan University

Abstract

Low-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult: merging the updates in weight space causes interference, retraining on all task data is expensive, and routing between separate adapters gives up the goal of a single combined model. We trace the difficulty to two choices that every composition method makes implicitly. A LoRA update admits infinitely many equivalent factorizations; the choice among them is invisible while an adapter serves alone, but it determines what a learned interaction between adapters can see. A coupling between an old skill and a new one can likewise point in either direction, and the direction decides whether the old skills keep computing what they computed before. We introduce READ (Read-only Expansion of Adapter Deltas), which fixes both choices: each adapter is rewritten into a balanced canonical form that preserves its update exactly, and the coupling grows in one direction only, so a new skill can read the input subspaces of old skills but cannot write into their output subspaces. The only trainable object at each append is the new skill's row of the coupling matrix, and the composed update folds into the base weights with no inference cost, routing, or task-specific rules. We evaluate READ across four benchmark suites and two model families, adding skills one at a time. Across several families, READ improves every suite average over the strongest published baselines built from the same adapters---by more than twenty points on SuperGLUE and more than seven points on the domain suite---and nearly all complete addition sequences end above every direct baseline. Factor coordinates and coupling direction, which a lone adapter never exposes, are what decide whether composed skills survive.

Figures & tables

Explore similar work

CardsList
  1. Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters

    Oct 22, 2026Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov +1Low-Rank AdaptationLanguage Model Merging

  2. LoRAtorio: An intrinsic approach to LoRA Skill Composition

    Aug 15, 2025Niki Foteinopoulou, Ignas Budvytis, Stephan LiwickiT2I Diffusion ModelsLow-Rank Adaptation

  3. Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

    Jun 30, 2026Seyed Alireza Molavi, Zhan Su, Yan Hu +3Mixture-of-Experts Language ModelsLow-Rank Adaptation