Organizations: University of Illinois Urbana-Champaign, IL, USA · Meta AI Menlo Park, CA, USA · Washington University St. Louis, WA, USA · Meta AI Sunnyvale, CA, USA
Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we observe that the routing weights are typically extremely imbalanced across LoRAs in practice, where only one or two LoRAs often dominate the routing weights. This essentially limits the number of effective LoRAs and thus severely hinders the expressive power of existing Mixture-of-LoRAs models. In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix). Our key idea is using non-learnable routing weights to ensure all active LoRAs to be equally effective, with no LoRA dominating the routing weights. However, our routers cannot be trained directly via gradient descent due to our non-learnable routing weights. Hence, we further propose an unbiased gradient estimator for the router by employing the reinforce leave-one-out (RLOO) technique, where we regard the supervision loss as the reward and the router as the policy in reinforcement learning. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperform state-of-the-art parameter-efficient finetuning methods under a comparable number of activated parameters.
Figures & tables
Figure 1: Finetuning procedure of our proposed ReMix.
Figure 2: We often observe that only one LoRA has a dominating routing weight that is close to one at deeper layers even when we activate k>1 LoRAs during finetuning. Consequently, the computation of the other k−1 LoRAs are essentially wasted because using k>1 would have similar accuracy to k=1 . (The dominating LoRA can differ on different inputs .)
Figure 3: Routing weight collapse even worsens as finetuning progresses . The effective support size ESS(π(l)) often drops to 1 quickly during finetuning.
Type
Method
GSM8K
HumanEval
ARC-c
Average
Accuracy
Params
Pass@1
Params
Accuracy
Params
Accuracy
Params
No Tuning
Zero-Shot
04.78
N/A
13.41
N/A
22.03
N/A
13.41
N/A
Few-Shot
55.95
N/A
17.68
N/A
81.36
N/A
51.66
N/A
Prefix Injection
Prefix Tuning
02.65
0.034B
00.00
0.034B
28.47
0.004B
10.37
0.024B
Prompt Tuning
04.70
0.000B
26.22
0.000B
23.73
0.000B
18.22
0.000B
P-Tuning
34.19
0.001B
27.44
0.001B
43.05
0.001B
34.89
0.001B
Table 1: Comparison with existing parameter-efficient finetuning methods. Our ReMix consistently outperforms all baseline methods while maintaining strong parameter efficiency. We use the same parameter count budget for all methods to ensure fair comparison; for each method, we search for the best parameter count and report the accuracy under the best parameter count.
Figure 4: Both of our proposed RLOO and top- k selection contribute significantly to the strong performance of our ReMix.
Method
k=1
k=2
k=4
Rank- kr LoRA
56.10
54.51
59.21
Ours: k rank- r LoRAs
56.18
59.67
64.22
Table 2: The fact that our method significantly outperforms rank- kr LoRA clearly shows that our ReMix can activate diverse LoRA subsets: If the activated subset were always the same subset, then the results would be the same as rank- kr LoRA.
Method
LoRA-type ω
rsLoRA-type ω
ReMix (Ours)
53.30
55.72
Table 3: Our ReMix is not sensitive to routing weight ω .
Method
Per-Step Time
Total Time
Accuracy
MixLoRA
8.95 s
1:12:56
50.34
ReMix (Ours)
9.87 s
1:28:21
58.38
Table 4: Our ReMix significantly outperforms MixLoRA under similar training time.
Method
k=1
k=2
k=3
k=4
ReMix (Ours)
56.18
59.67
61.33
64.22
Table 5: Our ReMix consistently improves as we scale up the number k of activated LoRAs.
Figure 5: Our proposed ReMix can further benefit from scaling up the training compute. In contrast, HydraLoRA and MixLoRA have fixed training compute and thus cannot benefit from scaling up training compute.
Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater representational plasticity, Low-Rank Adaptation (LoRA) can match or surpass FFT performance while constraining updates to a low-rank space and potentially benefiting from additional regularization. Through empirical evaluation across diverse tasks (SQL, Medical QA, and Counterfactual Knowledge) and varying language models (Gemma-3-1B, Qwen2.5-1.5B, and Qwen2.5-3B), we observe both trends and find that the better static architecture depends on the task and model. Spectral and truncation analyses further show that endpoint compressibility alone does not explain these task differences, suggesting task-score sensitivity and constrained optimization trajectories as possible explanations. To address this challenge, we propose a Mixture of LoRA and Full (MoLF) Fine-Tuning, a unified framework that enables continuous navigation between both training regimes. MoLF dynamically routes updates between FFT and LoRA at the optimizer level to ensure that exact gradient signals are available to both experts throughout training, while only selected experts update their weights. For memory-constrained environments, we also introduce MoLF-Efficient, which freezes base weights and only routes updates among a pair of LoRA experts of potentially varying rank. Our evaluations show that MoLF either improves on or stays within 1.5 percentage points of the better of FFT and LoRA across the nine tested settings, while MoLF-Efficient outperforms both AdaLoRA and AdaMix in eight of nine settings, with gains over the stronger baseline of up to 11.70 percentage points on Fact, 3.13 on Med, and 2.98 on SQL.
Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3
Carnegie Mellon University · Tsinghua University · Infinigence AI
Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose Hard-Routed MoR-LoRA, a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition.
Seyed Alireza Molavi, Zhan Su, Yan Hu +3
1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA experts. However, existing MoE-augmented LoRA methods assume that experts operate independently, often leading to unstable routing, expert dominance. In this paper, we propose TalkLoRA, a communication-aware MoELoRA framework that relaxes this independence assumption by introducing expert-level communication prior to routing. TalkLoRA equips low-rank experts with a lightweight Talking Module that enables controlled information exchange across expert subspaces, producing a more robust global signal for routing. Theoretically, we show that expert communication smooths routing dynamics by mitigating perturbation amplification while strictly generalizing existing MoELoRA architectures. Empirically, TalkLoRA consistently outperforms vanilla LoRA and MoELoRA across diverse language understanding and generation tasks, achieving higher parameter efficiency and more balanced expert routing under comparable parameter budgets. These results highlight structured expert communication as a principled and effective enhancement for MoE-based parameter-efficient adaptation. Code is available at https://github.com/why0129/TalkLoRA.
Lin Mu, Haiyang Wang, Li Ni +4
Anhui University · Hefei University · University of Science and Technology of China