Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation

Suraj Yadav

cs.CVAug 17, 2026

Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation

Authors: Suraj Yadav

Organizations: Indraprastha Institute of Information Technology Delhi

Abstract

Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules. Parameter-efficient fine-tuning adapts frozen visual foundation models efficiently, but standard low-rank adapters use a fixed subspace for all inputs, which can be restrictive when domains differ in style, background, and semantic context. MoE-based adapters improve specialization through multiple expert pathways, but often rely on external routers and large expert banks, adding parameters and separating routing from adaptation. We propose Self-Routed Tensor Adapters, a compact framework for multi-domain visual adaptation. SRTA projects each input into a low-rank space, computes routing weights from this representation using a learnable domain matrix, and uses these weights to blend slices of a shared Tucker core. This produces a sample-specific adaptation matrix without an external gating network, allowing shared visual factors to be reused while supporting domain-aware specialization. To strengthen pathway learning, we introduce a progressive depth-weighted routing objective that supervises routing decisions across adapter layers. Across five heterogeneous multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines while using substantially fewer trainable parameters. At rank 64, SRTA uses 2.77M parameters in the 4-domain setting compared with 9.52M for MoLoRA, and 3.00M in the 6-domain setting compared with 14.31M. Overall, SRTA offers an effective accuracy-parameter trade-off for adapting visual foundation models toward universal multi-domain representations. GitHub

Figures & tables

Figure 1: Overview of the Self-Routed Tensor Adapter pipeline . The input projection

V

and domain-coordinate matrix

C

generate intrinsic routing probabilities, which are used to blend slices of the shared Tucker core tensor

\mathcal{G}

. This removes the need for an external MoE gating network.

Method	PACS	VLCS	Office-Home	Digits-DG	NICO++	Average
LoRA	$\underline{95.1}{\scriptstyle\pm 0.1}$	$82.8{\scriptstyle\pm 0.7}$	$77.3{\scriptstyle\pm 0.3}$	$\mathbf{93.8}{\scriptstyle\pm 0.3}$	$77.6{\scriptstyle\pm 0.3}$	$85.3$
DoRA	$95.0{\scriptstyle\pm 0.2}$	$82.8{\scriptstyle\pm 0.3}$	$77.3{\scriptstyle\pm 0.3}$	$\mathbf{93.8}{\scriptstyle\pm 0.4}$	$77.4{\scriptstyle\pm 0.3}$	$85.3$
MALoRA	$94.3{\scriptstyle\pm 0.5}$	$82.8{\scriptstyle\pm 0.1}$	$85.0{\scriptstyle\pm 0.0}$	$90.2{\scriptstyle\pm 0.4}$	$82.3{\scriptstyle\pm 0.3}$	$86.9$
MOELoRA	$94.5{\scriptstyle\pm 0.3}$	$84.2{\scriptstyle\pm 0.5}$	$\underline{85.2}{\scriptstyle\pm 0.0}$	$88.9{\scriptstyle\pm 1.3}$	$\mathbf{83.3}{\scriptstyle\pm 1.0}$	$87.2$
MoLoRA	$\underline{95.1}{\scriptstyle\pm 0.3}$	$\underline{84.4}{\scriptstyle\pm 0.9}$	$\mathbf{85.3}{\scriptstyle\pm 0.6}$	$91.7{\scriptstyle\pm 0.5}$	$\mathbf{83.3}{\scriptstyle\pm 0.3}$	$\underline{88.0}$
SRTA	$\mathbf{95.2}{\scriptstyle\pm 0.7}$	$\mathbf{84.7}{\scriptstyle\pm 0.3}$	$84.8{\scriptstyle\pm 0.7}$	$\underline{92.7}{\scriptstyle\pm 0.6}$	$\underline{83.0}{\scriptstyle\pm 0.4}$	$\mathbf{88.1}$

Table 1: Accuracy (%) across heterogeneous multi-domain visual benchmarks (Rank 64).

Method	4 Domains				6 Domains
Method	$r$ =8	$r$ =16	$r$ =32	$r$ =64	$r$ =8	$r$ =16	$r$ =32	$r$ =64
LoRA	0.29M	0.59M	1.18M	2.36M	0.29M	0.59M	1.18M	2.36M
DoRA	0.31M	0.61M	1.20M	2.38M	0.31M	0.61M	1.20M	2.38M
MALoRA	0.53M	0.98M	1.90M	3.82M	0.75M	1.35M	2.59M	5.17M
MOELoRA	0.30M	0.59M	1.18M	2.36M	0.44M	0.89M	1.77M	3.54M
MoLoRA	1.26M	2.44M	4.80M	9.52M	1.93M	3.70M	7.23M	14.31M

Table 2: Parameter comparison across ranks for 4-domain and 6-domain settings. Parameter counts exclude the classifier.

$\lambda_{\mathrm{route}}$	PACS	VLCS	Office-Home	Digits-DG	NICO++	Average
0.0	$\mathbf{95.2}{\scriptstyle\pm 0.3}$	$82.3{\scriptstyle\pm 0.9}$	$83.5{\scriptstyle\pm 0.3}$	$92.6{\scriptstyle\pm 0.4}$	$81.7{\scriptstyle\pm 0.6}$	$87.1$
1.0	$\mathbf{95.2}{\scriptstyle\pm 0.7}$	$\mathbf{84.7}{\scriptstyle\pm 0.4}$	$\mathbf{84.8}{\scriptstyle\pm 0.8}$	$\mathbf{92.7}{\scriptstyle\pm 0.7}$	$\mathbf{83.0}{\scriptstyle\pm 0.4}$	$\mathbf{88.1}$

Table 3: Ablation of SRTA loss components at rank

64

Method	Rank-1	Rank-2	Extra Rank	PACS	VLCS	Office-Home	Digits-DG	NICO++
LoRA	8	–	–	$93.4{\scriptstyle\pm 0.6}$	$81.1{\scriptstyle\pm 0.7}$	$71.7{\scriptstyle\pm 1.2}$	$91.6{\scriptstyle\pm 0.7}$	$67.9{\scriptstyle\pm 1.4}$
LoRA	16	–	–	$94.1{\scriptstyle\pm 0.5}$	$81.5{\scriptstyle\pm 0.6}$	$75.9{\scriptstyle\pm 1.4}$	$92.2{\scriptstyle\pm 0.9}$	$71.5{\scriptstyle\pm 0.8}$
LoRA	32	–	–	$94.7{\scriptstyle\pm 0.4}$	$82.4{\scriptstyle\pm 0.3}$	$77.1{\scriptstyle\pm 0.6}$	$92.9{\scriptstyle\pm 0.4}$	$75.2{\scriptstyle\pm 0.9}$
LoRA	64	–	–	$95.1{\scriptstyle\pm 0.1}$	$82.8{\scriptstyle\pm 0.7}$	$77.3{\scriptstyle\pm 0.3}$	$93.8{\scriptstyle\pm 0.3}$	$77.6{\scriptstyle\pm 0.3}$
DoRA	8	–	–	$93.5{\scriptstyle\pm 0.7}$	$80.8{\scriptstyle\pm 1.1}$	$72.1{\scriptstyle\pm 1.2}$	$91.4{\scriptstyle\pm 0.7}$	$67.1{\scriptstyle\pm 1.2}$
DoRA	16	–	–	$93.9{\scriptstyle\pm 0.3}$	$81.9{\scriptstyle\pm 0.8}$	$76.1{\scriptstyle\pm 1.4}$	$92.2{\scriptstyle\pm 0.5}$	$71.7{\scriptstyle\pm 0.5}$

Table 4: Rank ablation across all PEFT and MoE-PEFT methods. Results are reported as mean

\pm

standard deviation across multiple runs. For SRTA,

r_{1}

and

r_{2}

denote Tucker ranks, while

r_{3}

is set to the number of domains.

Figure 2: Intrinsic routing probabilities ( $\alpha$ ) across core tensor slices.

Explore similar work

Mar 16, 2026

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune

Salim Khazem

Talan Research Center, Paris, France

Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Vision Transformer

Ask PDF

Aug 4, 2026

MuRA: Multi-Rank Adaptation for Efficient and Effective Test-Time Vision-Language Generalization

Vision-language models exhibit remarkable zero-shot capabilities but suffer significant performance degradation under distribution shifts. While test-time adaptation (TTA) via Low-Rank Adaptation offers a parameter-efficient solution, we identify a fundamental bottleneck in current methods: the reliance on static rank configurations. Because visual inputs inherently possess varying information densities, a fixed rank forces an inevitable optimization compromise, leading to underfitting on complex scenes and overfitting on simple ones. To bridge this gap, we propose Multi-Rank Adaptation (MuRA), a novel framework that dynamically selects and fuses adaptation modules of varying capacities based on token-level visual complexity. MuRA synergizes Multi-Rank Orthogonal Decomposition to provide a superior, knowledge-preserving initialization, and Unified Component Fusion with Continuous Router Updating to sustainably learn semantic-to-rank mappings. Furthermore, we provide rigorous theoretical justifications mathematically proving the necessity and gradient stability of this adaptive mechanism. Crucially, MuRA's dynamic design uniquely thrives at the deepest visual layer, capitalizing on the shortest gradient backpropagation path. Extensive experiments demonstrate that MuRA achieves state-of-the-art accuracy across extensive domain generalization and cross-dataset benchmarks while significantly reducing both computational and memory overhead.

Gengyuan Liu, Nanzhou Wang, Chang Liu +5

Tsinghua University Shenzhen, China · Tsinghua University Beijing, China · Huazhong University of Science and Technology Wuhan, China +1

Vision-Language Models VLM Adaptation Test-Time Adaptation of VLMs Low-Rank Adaptation

Ask PDF

Oct 7, 2026

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.

Josh McGiff, Salma Mekaoui, Robert Shanahan +1

University of Limerick, Ireland

Domain Adaptation for LLMs

Ask PDF