Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters
Authors: Yunzhi Zhuge, Xinzhuo Yu, Sitong Gong, Lu Zhang, Pingping Zhang, Huchuan Lu
Organizations: the School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China · the School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China
Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.
Figures & tables
Fig. 1: Motivation of D 2 MoE . (a) SAK distills from multiple VFMs into a shared student, coupling heterogeneous task gradients and incurring multi-teacher overhead. (b) D 2 MoE instead freezes a single VFM and enforces early task decoupling through a Mixture of Task Adapter (MoTA), allocating sparse, task-conditioned low-rank update routes for specialization without eroding pretrained invariances. (c) Correlation probes (Pearson maps around sampled points) confirm that model with MoTA induces more task-consistent feature organization for both semantic segmentation and depth, improving boundary adherence without sacrificing long-range structural coherence.
Fig. 2: Performance vs. Model Complexity . Comparison of D 2 MoE against state-of-the-art methods on NYUD-v2 [ 48 ] . D 2 MoE achieves superior performance (higher mIoU and lower RMSE) despite a substantial reduction in the total number of trainable parameters. The trainable-parameter count varies with the task configuration, with 45M on NYUD-v2 (shown here) and 54M on Pascal-Context (as shown in Tab. I .)
Fig. 3: Overall architecture of D 2 MoE .(§ III-A ) Built upon a frozen DINOv3 backbone, D 2 MoE organizes MoTA across four stages to preserve shared priors through task-generic adaptation and introduce localized task-specific adaption at stage endings. SPI fuses prior and task features, followed by lightweight heads for multi-task dense predictions.
Fig. 4: Illustration of the key components in D²MoE . (§ III-B and III-C ) (a) Task-Generic Adaptation inserts TG-LoRA into attention projections to provide shared, low-rank adaptation while keeping the DINOv3 backbone frozen. (b) Task-Specific Adaptation employs TA-MoE, where task routers select Top-k low-rank experts to produce sparse, task-conditioned updates for attention projections. (c) The Semantic Prior Injector fuses multi-level task features with frozen semantic priors to enhance semantic consistency and structural coherence for dense prediction tasks.
Method
Backbone
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
Δm
Para.
Train. Para.
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
(M) ↓
(M) ↓
MTL Baseline
ViT-L
82.61
76.38
77.09
17.63
74.00
0.00
328
25
PAD-Net [ 58 ]
[CVPR2018]
HRNet-18
53.60
59.60
65.80
15.30
72.50
-12.11
81
81
MTI-Net [ 60 ]
[ECCV2020]
HRNet-18
61.70
60.18
84.78
14.23
70.80
-4.32
128
128
ATRC [ 64 ]
[ICCV2021]
HRNet-18
67.67
62.93
82.29
14.24
72.42
-2.37
96
96
ASTMT [ 81 ]
[ECCV2024]
ResNet-50
68.00
61.12
65.71
14.68
72.40
-7.57
49
49
TABLE I: Performance comparison on the PASCAL-Context datasets. ↑ / ↓ indicates that the higher/lower results represent better performance.
Methods
Backbone
Sem. Seg.
Dep.
Norm.
Edge
Δm
mIoU ↑
RMSE ↓
mErr ↓
odsF ↑
(%)↑
MTL Baseline
ViT-L
63.15
0.5119
18.81
75.10
0.00
MTI-Net [ 60 ]
[ECCV2024]
HRNet-18
45.97
0.5365
20.27
77.84
-9.03
PAD-Net [ 58 ]
[CVPR2018]
HRNet-18
36.61
0.6270
20.85
76.38
-18.41
ATRC [ 64 ]
[ICCV2021]
HRNet-18
46.33
0.5363
20.18
77.94
-8.73
ASTMT [ 81 ]
[ECCV2024]
ResNet-50
32.16
0.5700
23.18
74.50
-21.11
TABLE II: Performance comparison on the NYUDv2 dataset. ↑ / ↓ indicates that the higher/lower results represent better performance.
Methods
Backbone
Sem. Seg.
Dep.
Δm
mIoU ↑
RMSE ↓
(%)↑
MTL Baseline
ViT-L
76.45
4.36
0
MTI-Net [ 60 ]
HRNet-18
53.19
5.05
-23.13
PAD-Net [ 58 ]
HRNet-18
59.85
5.06
-18.88
MTMamba [ 37 ]
Swin-L
78.00
4.66
-2.43
MTMamba++ [ 38 ]
Swin-L
79.13
4.63
-1.34
TABLE III: Performance comparison on the Cityscapes dataset.
Fig. 5: Qualitative Comparison on Pascal-Context. Compared with TaskDiffusion [ 35 ] and MTMamba [ 37 ] , D 2 MoE produces crisper boundaries and fewer cross-task inconsistencies (e.g., aligned silhouettes and object extents across parsing/segmentation/saliency and normals), indicating improved shared structure modeling and reduced task interference in multi-objective dense prediction.
Fig. 6: Qualitative Comparison on NYUD-v2 . Compared with TaskDiffusion [ 35 ] and MTMamba [ 37 ] , D 2 MoE better preserves depth/normal discontinuities at object boundaries and yields structurally coherent geometry that agrees with segmentation, suggesting stronger cross-task geometric consistency across heterogeneous dense objectives.
Fig. 7: PCA Visualizations on Pascal-Context (Left) and NYUD-v2 (Right). PCA visualizations are provided to analyze feature representations learned with TA-MoE, TG-LoRA, and the Semantic Prior Injector (SPI). As shown, features extracted from different network stages exhibit progressively clearer task-specific structures, demonstrating the effectiveness of the proposed modules in enhancing task-aware representation learning across both datasets.
Fig. 8: Hyperparameter analysis of TG-LoRA and TA-MoE. The results demonstrate clear performance trends across different LoRA ranks and expert configurations, highlighting an effective trade-off between model capacity and multi-task performance.
MoTA
SPI
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
Δm
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
✗
✗
82.61
76.38
77.09
17.63
74.00
0.00
✗
✓
83.27
77.21
82.51
15.24
78.30
5.66
✓
✗
82.44
74.90
84.12
14.01
76.20
6.10
✓
✓
84.48
78.05
85.55
13.57
78.40
8.88
TABLE IV: Overall ablation of the proposed components (MoTA and SPI) under the unified multi-task setting.
Modules
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
Δm
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
MTL Baseline
82.61
76.38
77.09
17.63
74.00
0.00
w/o TG-LoRA
84.07
77.92
84.32
14.15
78.30
7.74
w/o TA-MoE
83.31
77.60
84.35
13.65
78.00
7.97
Ours
84.48
78.05
85.55
13.57
78.40
8.88
TABLE V: Component ablation of Mixture of Task Adapters (MoTA), isolating the contributions of TG-LoRA and TA-MoE under the unified multi-task setting.
QKV
Proj
FFN
Sem. Seg
H. Pars.
Sal.
Norm.
Edge
Δm
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
✗
✗
✗
83.31
77.60
84.35
13.65
78.00
7.97
✓
✗
✗
83.74
77.80
85.18
13.57
78.70
8.62
✗
✓
✗
83.31
77.67
84.86
13.73
78.80
8.24
✗
✗
✓
83.34
77.24
84.63
13.58
78.00
8.03
✓
✓
✗
84.48
78.05
85.55
13.57
78.40
8.88
TABLE VI: Effect of inserting the MoE module at different locations within the backbone transformer block.
Fig. 9: 3D t-SNE Visualizations of Feature Embeddings . We compare the MTL baseline (top) with our D 2 MoE framework (bottom) across three benchmarks. (a) Semantic Segmentation: Points are colored by class label. D 2 MoE achieves significantly higher intra-class compactness and inter-class separability than the baseline. (b) Depth Estimation: Points are colored by discretized depth intervals (see legend). Our model yields more structured, monotonic groupings of scene geometry. Axes denote the three primary t-SNE components.
Fig. 10: Router visualization of TA-MoE across backbone stages. We visualize the task-wise expert selection patterns of TA-MoE inserted at different backbone stages and positions. For each task, we aggregate expert activation values over all samples in the Pascal-Context dataset and compute the cosine similarity between task-specific activation vectors. The resulting similarity matrices reveal how task relationships evolve across stages and insertion locations.
Fig. 11: Corruption Robustness Evaluation on Pascal-Context. Across five tasks, performance decreases with corruption severity, with zoom blur and gaussian noise inducing the largest degradations (e.g., Sem. Seg. drops from 84.48 on clean to 47.50 at Level 5). Weather/digital corruptions are comparatively mild at Levels 1–3, while normal error rises primarily under noise.
Fig. 12: Corruption Robustness Comparison with TaskDiffusion [ 35 ] . All examples are drawn from the Pascal-Context dataset under level-5 corruptions. Our method demonstrates stronger robustness to severe degradations, yielding cleaner object boundaries and more stable predictions.
Backbone
Para.
Sem. Seg.
Pars.
Sal.
Norm.
Edge
(M)
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
ViT-S
62.0
80.62
73.36
85.28
13.79
76.40
ViT-B
126.6
85.25
75.66
84.95
13.66
77.50
ViT-L
357.6
84.48
78.05
85.55
13.57
78.40
ViT-H
911.3
84.80
78.87
83.17
13.62
79.60
TABLE VII: Comparison of backbone variants under the same multi-task setting.
Modules
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
Δm
Para.
Train. Para.
Flops
Latency
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
(M)↓
(M)↓
( G ) ↓
( ms ) ↓
MTL Baseline
82.61
76.38
77.09
17.63
74.00
0.00
328
25
759.7
198.9
MLoRE [ 34 ]
83.85
78.05
84.98
15.51
78.10
6.30
407
407
829.9
203.2
D 2 MoE
84.48
78.05
85.55
13.57
78.40
8.88
358
54
1029.9
224.2
TABLE VIII: Comparison with MLoRE using a DINOv3 backbone.
Method
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
Δm
Backbone
Decoder
Train.
Para.
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%)↑
Train. (M) ↓
Train. (M) ↓
Para. (M) ↓
(M) ↓
MTL Baseline
82.61
76.38
77.09
17.63
74.00
0.00
25.2
0
25.2
328.4
Shared LoRA
83.60
77.52
85.19
13.51
78.30
8.48
25.2
29.2
54.4
357.6
Task LoRA
84.25
77.95
84.65
13.77
78.30
8.31
25.2
29.2
54.4
357.6
Decoder MoE
83.22
78.00
82.46
15.16
78.20
5.90
0
54.4
54.4
357.6
D 2 MoE
84.88
78.05
85.55
13.57
78.40
8.88
25.2
29.2
54.4
357.6
TABLE IX: Comparison with shared LoRA, task-specific LoRA and decoder-side MoE under matched trainable-parameter budgets.
Method
Sem. Seg.
H. Pars.
Sal.
Norm.
Edge
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
MTL Baseline
65.36
64.54
70.08
22.45
63.07
TaskDiffusion
57.01
49.48
67.42
22.81
59.04
D 2 MoE
68.08
67.03
72.77
21.47
64.96
TABLE X: Averaged robustness comparison on PASCAL-Context across all corruptions.
Fig. 13: Clean-normalized Performance Retention under Corruption. Task-wise retention for the MTL baseline, D 2 MoE, and TaskDiffusion. Avg. Δ denotes mean relative degradation from clean performance.
Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.
Vision foundation models (VFMs) have demonstrated strong robustness and transferability across a wide range of visual tasks. However, each model typically encodes strong inductive biases shaped by its pre-training objective and data domain, resulting in fragmented yet complementary visual knowledge. As a result, a single model often struggles to capture the diverse visual representations required across multiple dense prediction tasks. To address this limitation, we propose TIGER (Task-Instruction-Guided Expert Routing), a framework that coordinates multiple heterogeneous VFMs for multi-task dense prediction. Instead of naively aggregating expert features, TIGER leverages natural-language task instructions to guide a routing network that assigns token-level expert weights conditioned on task semantics, enabling adaptive integration of complementary expert features. TIGER further introduces a counterfactual loss that aligns routing decisions with each expert's causal contribution by measuring prediction changes when experts are excluded, encouraging more reliable and interpretable routing. We evaluate TIGER on two multi-task dense prediction benchmarks, NYUD-v2 and Pascal Context, where it consistently outperforms recent multi-task learning baselines while keeping all VFMs frozen. These results demonstrate that combining instruction-guided expert routing with counterfactual causal alignment enables effective coordination of heterogeneous vision foundation models.
Donghyun Han, Yuseok Bae, Jung Uk Kim +1
Electronics and Telecommunications Research Institute (ETRI), Korea · Kyung Hee University, Korea · Chonnam National University, Korea
Vision foundation backbones provide strong representations for dense prediction, yet a single shared feature still needs to support tasks with different, image-dependent adaptation requirements. We propose MemMTL, a multi-task dense prediction framework that estimates a compact task state from global visual context and refines it through a learnable task-state prototype memory. The refined state is converted into task-conditioned expert logits and combined with token-level logits before sparse top-k selection over a local expert bank shared by all tasks. A separate task-agnostic residual bank provides a common adaptation path, and both paths are added once to the backbone feature before task-specific prediction. We specify a matched evaluation protocol on NYUD-v2 and PASCAL-Context with SAM 3 and ViT-L backbones to measure predictive quality, computational cost, and the contributions of task-state conditioning, prototype retrieval, and sparse routing. The numerical record in the present working draft predates this canonical implementation and must be regenerated before it can support empirical claims.
Yangyang Xu, Haobo Yuan, Yuzhu Wang +4
Tsinghua University · University of California, Merced · Shanghai Jiao Tong University
Although large-scale visual foundation models (VFMs) achieve remarkable performance in semantic understanding, they still underperform in instance-aware dense prediction tasks. They exhibit different biases in representation: for instance, promptable segmentation models (e.g., SAM2) focus on fine-grained region boundaries, while self-supervised models (e.g., DINOv3) emphasize object-level structure. This observation highlights the potential of combining complementary features from different VFMs to enhance downstream dense prediction tasks. However, naive multi-VFM fusion seldom leads to reliable gains, and interpretable principles for leveraging their complementary features are still underexplored. In this work, we propose a metric-guided approach that effectively selects and aggregates complementary features from different VFMs based on explicit assessment scores. Specifically, we design a suite of label-free metrics in feature space across two aspects, Structural Coherence and Edge Fidelity, to assess features of VFM encoders. Guided by these scores, we identify complementary edge-strong and structure-strong encoder pairs, and integrate them via a master-auxiliary fusion scheme. This feature fusion requires no complex architectural changes and is trained only in a single stage. Our model shows consistent performance gains across multiple dense prediction tasks compared with the baselines, with better object-level semantics and more accurately localized boundaries. The code is available at {https://github.com/gyc-code/metric-guided-fusion}.
Yachan Guo, JoseLuis Gomez Zurita, Danna Xue +2
1Universitat Aut`onoma de Barcelona · 2Computer Vision Center · 3Harbin Institute of Technology, Shenzhen