JevSoup: System-One Routing for Training-Free LoRA Composition

Xiuying Wang; Jiahua Cheng; Shuotian Li; Yufan Cheng; Bowen Deng; Zhexuan Bai; Yichen Li

cs.AISep 25, 2026

JevSoup: System-One Routing for Training-Free LoRA Composition

Authors: Xiuying Wang, Jiahua Cheng, Shuotian Li, Yufan Cheng, Bowen Deng, Zhexuan Bai, Yichen Li

Organizations: Beijing University of Posts and Telecommunications · University of Malaya · Georgian College · Huazhong University of Science and Technology

Abstract

Building adaptable AI systems requires effective coordination of specialized capabilities across diverse tasks. Low-rank adaptation (LoRA) enables modular expertise, but existing routing approaches may require auxiliary data, additional training, or autoregressive decoding. We propose JevSoup, a training-free framework separating System One expert routing from System Two execution. Using only the input and expert descriptions, Jev selects two experts through structured probabilities. JevSoup retains the leading expert's update, projects the second onto the orthogonal complement of the first update's row space, and combines them with equal weights. Across 14 PorTAL tasks and three Qwen3 scales, JepSoup achieves absolute gains of up to 1.19% in task-macro and 1.21% in sample-micro accuracy over the strongest evaluated external baselines. Our code is available at https://github.com/Leowang980/JevSoup.

Figures & tables

Figure 1: Overview of JevSoup. Top (System One): Jev uses the input and expert descriptions to rank the experts and select an ordered Top-2 pair. Bottom (System Two): The second LoRA update is orthogonally projected relative to the first, and the two complete updates are equally combined in the frozen language model to produce the answer. Probability bars are schematic.

	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
Method	Macro	Micro	Macro	Micro	Macro	Micro
Base	58.05	61.79	64.05	69.76	68.15	74.41
Adaptive Minds [ 12 ]	69.79	67.15	74.11	72.49	77.07	78.54
LoGo [ 10 ]	68.74	64.96	71.43	69.75	75.94	77.01
AdapterSoup [ 2 ]	69.18	66.85	73.61	72.60	76.86	77.87
Arrow [ 11 ]	69.12	65.81	72.65	71.27	76.45	77.11

Table 1: Main comparison on PorTAL (accuracy, %). Macro and Micro denote task- and sample-averaged accuracy. Best and second-best values in each column are bold and underlined, respectively. At each model scale, all routing methods share the same backbone, expert library, and evaluation protocol.

Selection	Weights	Proj.	Macro	Micro
Random Top-2	Equal	No	72.43 $\pm$ 0.34	71.77 $\pm$ 0.16
Jev Top-1	Single	–	73.24	71.83
Jev Top-2	Prob.	No	74.45	72.33
Jev Top-2	Prob.	Yes	74.86	72.41
AM Top-2	Equal	No	74.11	72.49
Jev Top-2	Equal	No	74.64	73.49

Table 2: 4B ablation (accuracy, %). Random Top-2 reports mean

\pm

standard deviation over three routing seeds. Prob. denotes routing probabilities renormalized over the selected pair. Proj. indicates whether full projection (

\lambda=1

) is applied to the second expert’s update.

Figure 2: 4B sensitivity to projection strength. Point-estimate accuracy changes relative to the unprojected control (

\lambda=0

, dashed line), with fixed routing and equal weights.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
Apr 7, 2026Lin Mu, Haiyang Wang, Li Ni +4Mixture-of-Experts Language Models Low-Rank Adaptation
Anhui University · Hefei University · University of Science and Technology of China
MoLoRA: Composable Specialization via Per-Token Adapter Routing
Mar 16, 2026Shrey Shah, Justin WagleLLM Routing Low-Rank Adaptation
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Jun 30, 2026Seyed Alireza Molavi, Zhan Su, Yan Hu +3Mixture-of-Experts Language Models Low-Rank Adaptation
1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China

Apr 7, 2026J/K move · Enter open · S save

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

Lin Mu, Haiyang Wang, Li Ni, Lei Sang+3

Anhui University · Hefei University · University of Science and Technology of China

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning LLM Routing

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA experts. However, existing MoE-augmented LoRA methods assume that experts operate independently, often leading to unstable routing, expert dominance. In this paper, we propose TalkLoRA, a communication-aware MoELoRA framework that relaxes this independence assumption by introducing expert-level communication prior to routing. TalkLoRA equips low-rank experts with a lightweight Talking Module that enables controlled information exchange across expert subspaces, producing a more robust global signal for routing. Theoretically, we show that expert communication smooths routing dynamics by mitigating perturbation amplification while strictly generalizing existing MoELoRA architectures. Empirically, TalkLoRA consistently outperforms vanilla LoRA and MoELoRA across diverse language understanding and generation tasks, achieving higher parameter efficiency and more balanced expert routing under comparable parameter budgets. These results highlight structured expert communication as a principled and effective enhancement for MoE-based parameter-efficient adaptation. Code is available at https://github.com/why0129/TalkLoRA.

Abstract

LLM Routing

Explore similar work

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

MoLoRA: Composable Specialization via Per-Token Adapter Routing

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs