MoLoRA: Composable Specialization via Per-Token Adapter Routing

Shrey Shah; Justin Wagle

cs.CLMar 16, 2026

MoLoRA: Composable Specialization via Per-Token Adapter Routing

Authors: Shrey Shah, Justin Wagle

Abstract

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.

Figures & tables

Panel of Figure 1 — Figure 1: Per-sequence routing (left) sends all tokens in a sequence to the same adapter, even when tokens have different modalities (T=text, I=image, A=audio, V=video). Per-token routing (right) routes each token to its modality-specialized adapter within a single forward pass.

Panel of Figure 2 — Figure 2: MoLoRA: logical vs. physical execution. Left: Per-token view—a router selects top- $k$ adapters (here $k$ =2), which are combined via weighted sum. Right: Batched execution—tokens are grouped by their selected adapter, enabling parallel grouped GEMM using identical infrastructure to MoE systems.

Panel of Figure 3 — Figure 3: MoLoRA enables small models to exceed larger ones. Qwen3-1.7B with four specialized LoRA adapters and learned routing (blue) exceeds Qwen3-8B (red) on all four reasoning benchmarks, while being 4.7 $\times$ smaller.

Identical Infrastructure (Reused)
Component	Vocabulary Routing	MoLoRA
Routing decision	$\mathcal{R}_{\text{vocab}}(v)$	$\text{TopK}(g_{\theta}(x))$
Routing cost	$\mathcal{O}(1)$	$\mathcal{O}(64d+64K)$
Histogram computation	✓	✓
Pointer arrays	✓	✓
Grouped GEMM dispatch	✓	✓

Table 1: Infrastructure reuse between vocabulary routing and MoLoRA. Only the routing decision differs; post-routing computation is identical.

Panel of Figure 4 — Figure 4: System architecture comparison. Paging (left) copies adapters on-demand from CPU, causing variable latency and preventing CUDA graph capture. Our hot-set architecture (right) pre-allocates GPU-resident adapters with fixed addresses, enabling graph capture and predictable latency.

Adapters	Indirect	Direct	Speedup
4	0.215ms	0.196ms	1.10 $\times$
8	0.228ms	0.194ms	1.17 $\times$
16	0.227ms	0.201ms	1.13 $\times$
32	0.232ms	0.207ms	1.12 $\times$
64	0.258ms	0.218ms	1.18 $\times$
Average			1.14 $\times$

Table 2: Memory layout comparison using identical operations.

Panel of Figure 5 — Figure 5: Kernel performance comparison. Scalar implementations (S-LoRA’s BGMV) excel in the memory-bound regime (batch $<$ 160), while our tensor-core kernel with CUDA graph capture dominates in the compute-bound regime. The crossover occurs at batch size $\approx$ 160, with our kernel achieving 1.98 $\times$ speedup at batch 512.

Routing	Passes	Latency	Latency/Pass
Per-sequence ( $K$ =4)	4	5.88ms	1.47ms
Per-token	1	1.43ms	1.43ms
Speedup	4 $\times$	4.1 $\times$	—

Table 3: Per-token routing reduces

K

forward passes to 1. Per-sequence routing requires

K

passes for

K

-modality workloads; per-token routing requires exactly one. Configuration:

K

=4 modalities, 2048 tokens,

d

=4096,

r

=64.

Configuration	Latency	vs Baseline	Incremental	Source
Per-seq + Paging	7.48ms	1.0 $\times$	—	Baseline
Per-seq + Hot-set	5.88ms	1.3 $\times$	1.3 $\times$	Paging eliminated
Per-token + Hot-set	1.43ms	5.2 $\times$	4.1 $\times$	Passes: $K\to 1$
Per-token + Graph	1.36ms	5.5 $\times$	1.05 $\times$	Launch overhead

Table 4: Incremental improvements from each architectural choice. Same configuration as Table 3 .

Scenario	Batch	Seq	Per-Seq	Per-Token	Speedup
Diverse adapters	4	128	179.0ms	41.5ms	4.3 $\times$
Diverse adapters	8	128	178.7ms	43.4ms	4.1 $\times$
Uniform (best S-LoRA)	8	256	178.1ms	79.8ms	2.2 $\times$
Single adapter	8	256	80.3ms	80.0ms	1.0 $\times$

Table 5: Qwen3-4B benchmark comparing per-sequence (S-LoRA style) vs per-token routing. Speedup depends on adapter diversity.

Distribution	Per-Token	Per-Sequence	Speedup
Interleaved, text-heavy	0.23ms	5.94ms	26.0 $\times$
Interleaved, balanced	0.24ms	12.79ms	52.7 $\times$
Separated (control)	0.24ms	1.81ms	7.5 $\times$
Average			28.7 $\times$

Table 6: Speedup varies with modality distribution. “Interleaved” requires sequence splitting; “separated” allows efficient batching.

Panel of Figure 6 — Figure 6: Latency comparison across batch sizes. Per-token routing (blue) maintains sub-millisecond latency while per-sequence routing (red) requires sequence splitting, incurring 5.8–112 $\times$ higher latency. The gap narrows at larger batches as per-sequence overhead amortizes.

Panel of Figure 7 — Figure 7: P99 latency with variance bands derived from coefficient of variation (CV). CUDA graph capture (blue) achieves 42–108 $\times$ lower latency than paging (red). Shaded regions show $\pm$ 1 std.

Panel of Figure 8 — Figure 8: P99 latency under different access patterns. Hot-set latency remains stable ( $\sim$ 0.09ms) regardless of workload, while paging varies 3 $\times$ between best-case (bursty) and worst-case (adversarial).

Config	Batch	Per-Token	Per-Seq Sim	Speedup
$d$ =1024, $L$ =2	16 $\times$ 512	14.5ms	57.7ms	4.0 $\times$
$d$ =2048, $L$ =4	4 $\times$ 512	17.3ms	68.8ms	4.0 $\times$
$d$ =4096, $L$ =4	16 $\times$ 512	121.6ms	488.0ms	4.0 $\times$

Table 7: End-to-end transformer latency with per-token routing. Per-sequence simulation runs

K

forward passes (one per adapter) to handle mixed-modality sequences. Speedup is

\sim

K

\times

where

K=4

adapters.

Method	Passes	Latency	Speedup
Per-Sequence ( $K$ =2)	2	5.18ms	1.0 $\times$
Per-Token	1	3.14ms	1.65 $\times$

Table 8: Chameleon-style interleaved generation benchmark. Full transformer with per-token LoRA (

d

=2048,

L

=4,

K

=2 modalities). Per-sequence routing requires

K

full forward passes.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Distribution	Fixed (64 $\times$ 64)	Adaptive	Speedup
Uniform (25% each)	0.412ms	0.398ms	1.04 $\times$
Skewed (80/10/5/5)	0.456ms	0.389ms	1.17 $\times$
Extreme (95/2/2/1)	0.521ms	0.372ms	1.40 $\times$

Appendix

Table 9: Adaptive vs. fixed tiling. Adaptive selection provides up to 1.4

\times

improvement for skewed distributions.

Panel of Figure 9 — Figure 9: Kernel comparison: Tensor Core (with CUDA graph) vs BGMV scalar kernel. Our kernel achieves near-constant latency ( $\sim$ 0.016ms) while BGMV scales linearly. Crossover at batch 160; at production sizes (512–1024), we achieve 1.98–2.69 $\times$ speedup. Benchmarked on H100 NVL with $d$ =4096, $r$ =64, 8 adapters.

Concept	MoE	Multi-Adapter
Routing function	$g_{\theta}(x)=\text{softmax}(Wx+b)$	$j:[B]\to[K]$ or $\mathcal{R}_{\text{vocab}}(v)$
Routing cost	$\mathcal{O}(E\cdot d)$ (learned)	$\mathcal{O}(1)$ (deterministic)
Routing decision	Per-token, learned	Per-sequence or per-token
Targets	Experts $\{1,\ldots,E\}$	Adapters $\{1,\ldots,K\}$
Weight shape	$(E,d_{\text{in}},d_{\text{out}})$	$(K,d,r)$ for $A$ ; $(K,r,d)$ for $B$
Capacity	Expert capacity $C$	Adapter memory budget

Appendix

Table 10: Complete correspondence between MoE and multi-adapter serving.

Batch	Eager CV	Graph CV	Paging CV
32	0.52	0.03	0.63
64	0.68	0.33	0.32
128	0.34	0.12	0.12
256	0.01	0.07	0.18

Appendix

Table 11: Latency variance comparison (CV = std/mean).

Scenario	Vocab Routing	MoLoRA
Chameleon-style (disjoint vocab)	✓	✓
Encoder-based multimodal (LLaVA, Flamingo)	✗	✓
Semantic specialization (code/math/prose)	✗	✓
Sub-modality granularity (photo/diagram/chart)	✗	✓
Multi-attribute routing (modality $\times$ domain)	✗	✓

Appendix

Table 12: MoLoRA use case taxonomy. Vocabulary routing requires modality-encoding vocabulary structure. MoLoRA handles all scenarios through learned routing.

Model	Adapters	top- $k$	Train Loss	Val Loss
Single Adapter	1	–	3.008	3.435
Fixed (Oracle)	3	1	2.588	3.819
MoLoRA	4	2	2.861	3.555
MoLoRA-L	8	3	2.783	3.599

Appendix

Table 13: MoLoRA training results on synthetic multimodal task. MoLoRA learns routing without labels, achieving 35% of oracle improvement.

Epoch	ARI	NMI	Routing Entropy
0 (random)	0.27	0.37	1.35
20	0.72	0.86	0.09
99 (converged)	0.71	0.84	0.18

Appendix

Table 14: MoLoRA discovers modality structure without supervision. ARI (Adjusted Rand Index) and NMI (Normalized Mutual Information) measure alignment between learned routing and true modality labels. Higher is better; 1.0 indicates perfect clustering.

Domain	Adapter 0	Adapter 1	Adapter 2
Code	0.0%	100%	0.0%
Math	0.0%	0.0%	100%
Prose	100%	0.0%	0.0%

Appendix

Table 15: MoLoRA semantic routing: perfect domain specialization (ARI=1.0) despite shared vocabulary. Each domain routes exclusively to a single adapter.

Model	Latency (ms)	Throughput (M tok/s)	Overhead
Single Adapter	0.082	49.7	1.0 $\times$
Fixed Routing	0.894	4.6	10.9 $\times$
MoLoRA $k=1$	0.995	4.1	12.1 $\times$
MoLoRA $k=2$	2.267	1.8	27.6 $\times$
MoLoRA $k=4$	4.001	1.0	48.8 $\times$

Appendix

Table 16: MoLoRA inference overhead at

d=4096

, batch=32, seq=128.

Content Type	Adapter 0	Adapter 1	Adapter 2	Adapter 3
Code	98.6%	0.0%	0.0%	1.4%
Math	0.0%	96.6%	0.5%	3.0%
Creative	0.0%	0.0%	98.8%	1.2%
Technical	0.3%	0.3%	0.2%	99.2%

Appendix

Table 17: MoLoRA routing specialization on Qwen3-4B. Each content type routes predominantly to a single adapter, achieving near-perfect specialization.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

TokenRouter: Efficient Serving System for Token-Level LLM Routing
Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM Serving LLM Routing
Tsinghua University · Carnegie Mellon University
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Jun 30, 2026Seyed Alireza Molavi, Zhan Su, Yan Hu +3Mixture-of-Experts Language Models Low-Rank Adaptation
1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China
Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Jul 21, 2026Atahan Dokme, Larry HeckRetrieval-Augmented Generation Selective SSMs
AI Virtual Assistant (AVA) Lab Georgia Institute of Technology

Oct 8, 2026J/K move · Enter open · S save

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong+3

Tsinghua University · Carnegie Mellon University

LLM Serving LLM Routing LLM Inference Scheduling

Open paper AskPDF arXiv

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Abstract

LLM Routing

Explore similar work

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

Selective State-Space Adaptation and Retrieval for Language Model Reasoning