Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

Yunzhi Zhuge; Xinzhuo Yu; Sitong Gong; Lu Zhang; Pingping Zhang; Huchuan Lu

cs.CVNov 2, 2026

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

Authors: Yunzhi Zhuge, Xinzhuo Yu, Sitong Gong, Lu Zhang, Pingping Zhang, Huchuan Lu

Organizations: the School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China · the School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China

Abstract

Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.

Figures & tables

Fig. 1: Motivation of D 2 MoE . (a) SAK distills from multiple VFMs into a shared student, coupling heterogeneous task gradients and incurring multi-teacher overhead. (b) D 2 MoE instead freezes a single VFM and enforces early task decoupling through a Mixture of Task Adapter (MoTA), allocating sparse, task-conditioned low-rank update routes for specialization without eroding pretrained invariances. (c) Correlation probes (Pearson maps around sampled points) confirm that model with MoTA induces more task-consistent feature organization for both semantic segmentation and depth, improving boundary adherence without sacrificing long-range structural coherence.

Panel of Figure 2 — Fig. 2: Performance vs. Model Complexity . Comparison of D 2 MoE against state-of-the-art methods on NYUD-v2 [ 48 ] . D 2 MoE achieves superior performance (higher mIoU and lower RMSE) despite a substantial reduction in the total number of trainable parameters. The trainable-parameter count varies with the task configuration, with 45M on NYUD-v2 (shown here) and 54M on Pascal-Context (as shown in Tab. I .)

Fig. 3: Overall architecture of D 2 MoE .(§ III-A ) Built upon a frozen DINOv3 backbone, D 2 MoE organizes MoTA across four stages to preserve shared priors through task-generic adaptation and introduce localized task-specific adaption at stage endings. SPI fuses prior and task features, followed by lightweight heads for multi-task dense predictions.

Fig. 4: Illustration of the key components in D²MoE . (§ III-B and III-C ) (a) Task-Generic Adaptation inserts TG-LoRA into attention projections to provide shared, low-rank adaptation while keeping the DINOv3 backbone frozen. (b) Task-Specific Adaptation employs TA-MoE, where task routers select Top-k low-rank experts to produce sparse, task-conditioned updates for attention projections. (c) The Semantic Prior Injector fuses multi-level task features with frozen semantic priors to enhance semantic consistency and structural coherence for dense prediction tasks.

Method		Backbone	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$	Para.	Train. Para.
Method		Backbone	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\%)\uparrow$	(M) $\downarrow$	(M) $\downarrow$
MTL Baseline		ViT-L	82.61	76.38	77.09	17.63	74.00	0.00	328	25
PAD-Net [ 58 ]	[CVPR2018]	HRNet-18	53.60	59.60	65.80	15.30	72.50	-12.11	81	81
MTI-Net [ 60 ]	[ECCV2020]	HRNet-18	61.70	60.18	84.78	14.23	70.80	-4.32	128	128
ATRC [ 64 ]	[ICCV2021]	HRNet-18	67.67	62.93	82.29	14.24	72.42	-2.37	96	96
ASTMT [ 81 ]	[ECCV2024]	ResNet-50	68.00	61.12	65.71	14.68	72.40	-7.57	49	49

TABLE I: Performance comparison on the PASCAL-Context datasets.

\uparrow

\downarrow

indicates that the higher/lower results represent better performance.

Methods		Backbone	Sem. Seg.	Dep.	Norm.	Edge	$\Delta_{m}$
Methods		Backbone	mIoU $\uparrow$	RMSE $\downarrow$	mErr $\downarrow$	odsF $\uparrow$	$(\%)\uparrow$
MTL Baseline		ViT-L	63.15	0.5119	18.81	75.10	0.00
MTI-Net [ 60 ]	[ECCV2024]	HRNet-18	45.97	0.5365	20.27	77.84	-9.03
PAD-Net [ 58 ]	[CVPR2018]	HRNet-18	36.61	0.6270	20.85	76.38	-18.41
ATRC [ 64 ]	[ICCV2021]	HRNet-18	46.33	0.5363	20.18	77.94	-8.73
ASTMT [ 81 ]	[ECCV2024]	ResNet-50	32.16	0.5700	23.18	74.50	-21.11

TABLE II: Performance comparison on the NYUDv2 dataset.

\uparrow

\downarrow

indicates that the higher/lower results represent better performance.

Methods	Backbone	Sem. Seg.	Dep.	$\Delta_{m}$
Methods	Backbone	mIoU $\uparrow$	RMSE $\downarrow$	$(\textit{\%})\uparrow$
MTL Baseline	ViT-L	76.45	4.36	0
MTI-Net [ 60 ]	HRNet-18	53.19	5.05	-23.13
PAD-Net [ 58 ]	HRNet-18	59.85	5.06	-18.88
MTMamba [ 37 ]	Swin-L	78.00	4.66	-2.43
MTMamba++ [ 38 ]	Swin-L	79.13	4.63	-1.34

TABLE III: Performance comparison on the Cityscapes dataset.

Fig. 5: Qualitative Comparison on Pascal-Context. Compared with TaskDiffusion [ 35 ] and MTMamba [ 37 ] , D 2 MoE produces crisper boundaries and fewer cross-task inconsistencies (e.g., aligned silhouettes and object extents across parsing/segmentation/saliency and normals), indicating improved shared structure modeling and reduced task interference in multi-objective dense prediction.

Fig. 6: Qualitative Comparison on NYUD-v2 . Compared with TaskDiffusion [ 35 ] and MTMamba [ 37 ] , D 2 MoE better preserves depth/normal discontinuities at object boundaries and yields structurally coherent geometry that agrees with segmentation, suggesting stronger cross-task geometric consistency across heterogeneous dense objectives.

Fig. 7: PCA Visualizations on Pascal-Context (Left) and NYUD-v2 (Right). PCA visualizations are provided to analyze feature representations learned with TA-MoE, TG-LoRA, and the Semantic Prior Injector (SPI). As shown, features extracted from different network stages exhibit progressively clearer task-specific structures, demonstrating the effectiveness of the proposed modules in enhancing task-aware representation learning across both datasets.

Fig. 8: Hyperparameter analysis of TG-LoRA and TA-MoE. The results demonstrate clear performance trends across different LoRA ranks and expert configurations, highlighting an effective trade-off between model capacity and multi-task performance.

MoTA	SPI	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$
MoTA	SPI	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\textit{\%})\uparrow$
✗	✗	82.61	76.38	77.09	17.63	74.00	0.00
✗	✓	83.27	77.21	82.51	15.24	78.30	5.66
✓	✗	82.44	74.90	84.12	14.01	76.20	6.10
✓	✓	84.48	78.05	85.55	13.57	78.40	8.88

TABLE IV: Overall ablation of the proposed components (MoTA and SPI) under the unified multi-task setting.

Modules	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$
Modules	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\textit{\%})\uparrow$
MTL Baseline	82.61	76.38	77.09	17.63	74.00	0.00
w/o TG-LoRA	84.07	77.92	84.32	14.15	78.30	7.74
w/o TA-MoE	83.31	77.60	84.35	13.65	78.00	7.97
Ours	84.48	78.05	85.55	13.57	78.40	8.88

TABLE V: Component ablation of Mixture of Task Adapters (MoTA), isolating the contributions of TG-LoRA and TA-MoE under the unified multi-task setting.

QKV	Proj	FFN	Sem. Seg	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$
QKV	Proj	FFN	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\textit{\%})\uparrow$
✗	✗	✗	83.31	77.60	84.35	13.65	78.00	7.97
✓	✗	✗	83.74	77.80	85.18	13.57	78.70	8.62
✗	✓	✗	83.31	77.67	84.86	13.73	78.80	8.24
✗	✗	✓	83.34	77.24	84.63	13.58	78.00	8.03
✓	✓	✗	84.48	78.05	85.55	13.57	78.40	8.88

TABLE VI: Effect of inserting the MoE module at different locations within the backbone transformer block.

Fig. 9: 3D t-SNE Visualizations of Feature Embeddings . We compare the MTL baseline (top) with our D 2 MoE framework (bottom) across three benchmarks. (a) Semantic Segmentation: Points are colored by class label. D 2 MoE achieves significantly higher intra-class compactness and inter-class separability than the baseline. (b) Depth Estimation: Points are colored by discretized depth intervals (see legend). Our model yields more structured, monotonic groupings of scene geometry. Axes denote the three primary t-SNE components.

Fig. 10: Router visualization of TA-MoE across backbone stages. We visualize the task-wise expert selection patterns of TA-MoE inserted at different backbone stages and positions. For each task, we aggregate expert activation values over all samples in the Pascal-Context dataset and compute the cosine similarity between task-specific activation vectors. The resulting similarity matrices reveal how task relationships evolve across stages and insertion locations.

Fig. 11: Corruption Robustness Evaluation on Pascal-Context. Across five tasks, performance decreases with corruption severity, with zoom blur and gaussian noise inducing the largest degradations (e.g., Sem. Seg. drops from 84.48 on clean to 47.50 at Level 5). Weather/digital corruptions are comparatively mild at Levels 1–3, while normal error rises primarily under noise.

Fig. 12: Corruption Robustness Comparison with TaskDiffusion [ 35 ] . All examples are drawn from the Pascal-Context dataset under level-5 corruptions. Our method demonstrates stronger robustness to severe degradations, yielding cleaner object boundaries and more stable predictions.

Backbone	Para.	Sem. Seg.	Pars.	Sal.	Norm.	Edge
Backbone	(M)	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$
ViT-S	62.0	80.62	73.36	85.28	13.79	76.40
ViT-B	126.6	85.25	75.66	84.95	13.66	77.50
ViT-L	357.6	84.48	78.05	85.55	13.57	78.40
ViT-H	911.3	84.80	78.87	83.17	13.62	79.60

TABLE VII: Comparison of backbone variants under the same multi-task setting.

Modules	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$	Para.	Train. Para.	Flops	Latency
Modules	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\textit{\%})\uparrow$	$(\textit{M})\downarrow$	$(\textit{M})\downarrow$	( G ) $\downarrow$	( ms ) $\downarrow$
MTL Baseline	82.61	76.38	77.09	17.63	74.00	0.00	328	25	759.7	198.9
MLoRE [ 34 ]	83.85	78.05	84.98	15.51	78.10	6.30	407	407	829.9	203.2
D 2 MoE	84.48	78.05	85.55	13.57	78.40	8.88	358	54	1029.9	224.2

TABLE VIII: Comparison with MLoRE using a DINOv3 backbone.

Method	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge	$\Delta_{m}$	Backbone	Decoder	Train.	Para.
Method	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$	$(\%)\uparrow$	Train. (M) $\downarrow$	Train. (M) $\downarrow$	Para. (M) $\downarrow$	(M) $\downarrow$
MTL Baseline	82.61	76.38	77.09	17.63	74.00	0.00	25.2	0	25.2	328.4
Shared LoRA	83.60	77.52	85.19	13.51	78.30	8.48	25.2	29.2	54.4	357.6
Task LoRA	84.25	77.95	84.65	13.77	78.30	8.31	25.2	29.2	54.4	357.6
Decoder MoE	83.22	78.00	82.46	15.16	78.20	5.90	0	54.4	54.4	357.6
D 2 MoE	84.88	78.05	85.55	13.57	78.40	8.88	25.2	29.2	54.4	357.6

TABLE IX: Comparison with shared LoRA, task-specific LoRA and decoder-side MoE under matched trainable-parameter budgets.

Method	Sem. Seg.	H. Pars.	Sal.	Norm.	Edge
Method	mIoU $\uparrow$	mIoU $\uparrow$	maxF $\uparrow$	mErr $\downarrow$	odsF $\uparrow$
MTL Baseline	65.36	64.54	70.08	22.45	63.07
TaskDiffusion	57.01	49.48	67.42	22.81	59.04
D 2 MoE	68.08	67.03	72.77	21.47	64.96

TABLE X: Averaged robustness comparison on PASCAL-Context across all corruptions.

Fig. 13: Clean-normalized Performance Retention under Corruption. Task-wise retention for the MTL baseline, D 2 MoE, and TaskDiffusion. Avg. $\Delta$ denotes mean relative degradation from clean performance.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 334 papers

Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning
Jun 14, 2026Donghyun Han, Yuseok Bae, Jung Uk Kim +1Vision Foundation Model Adaptation Multi-Task Learning
Electronics and Telecommunications Research Institute (ETRI), Korea · Kyung Hee University, Korea · Chonnam National University, Korea
Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction
Aug 28, 2026Yangyang Xu, Haobo Yuan, Yuzhu Wang +4Multi-Task Learning Vision Foundation Model Adaptation
Tsinghua University · University of California, Merced · Shanghai Jiao Tong University
Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks
May 16, 2026Yachan Guo, JoseLuis Gomez Zurita, Danna Xue +2Image Segmentation Vision Foundation Models
1Universitat Aut`onoma de Barcelona · 2Computer Vision Center · 3Harbin Institute of Technology, Shenzhen

Jun 14, 2026J/K move · Enter open · S save

Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning

Donghyun Han, Yuseok Bae, Jung Uk Kim, Hyung-Il Kim

Electronics and Telecommunications Research Institute (ETRI), Korea · Kyung Hee University, Korea · Chonnam National University, Korea

Vision Foundation Model Adaptation Multi-Task Learning Expert Routing Mixture-of-Experts Models

Open paper AskPDF arXiv

Vision foundation models (VFMs) have demonstrated strong robustness and transferability across a wide range of visual tasks. However, each model typically encodes strong inductive biases shaped by its pre-training objective and data domain, resulting in fragmented yet complementary visual knowledge. As a result, a single model often struggles to capture the diverse visual representations required across multiple dense prediction tasks. To address this limitation, we propose TIGER (Task-Instruction-Guided Expert Routing), a framework that coordinates multiple heterogeneous VFMs for multi-task dense prediction. Instead of naively aggregating expert features, TIGER leverages natural-language task instructions to guide a routing network that assigns token-level expert weights conditioned on task semantics, enabling adaptive integration of complementary expert features. TIGER further introduces a counterfactual loss that aligns routing decisions with each expert's causal contribution by measuring prediction changes when experts are excluded, encouraging more reliable and interpretable routing. We evaluate TIGER on two multi-task dense prediction benchmarks, NYUD-v2 and Pascal Context, where it consistently outperforms recent multi-task learning baselines while keeping all VFMs frozen. These results demonstrate that combining instruction-guided expert routing with counterfactual causal alignment enables effective coordination of heterogeneous vision foundation models.

Abstract

Vision Foundation Model Adaptation

Explore similar work

Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning

Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction

Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks