Multi-task scene understanding models typically rely on modified vision encoders or complex decoders for cross-task interaction, which limits their flexibility and compatibility with off-the-shelf vision foundation models (VFMs). We introduce Virtual Task Prompting (VTP), which frames multi-task prompting as retrieving task-adaptive compositions from a shared virtual state, where both the state and routing process are optimized in an end-to-end manner. Concretely, VTP maintains an expanded set of virtual tasks alongside the backbone. At each layer, learnable routing matrices read task-specific compositions as input to interact with image tokens and write the processed results back, forming explicit pathways that capture the intricacies among tasks outside the encoder. Since VTP keeps the backbone clean, it integrates seamlessly with pretrained VFMs, advancing the state of the art on Pascal-Context and NYUD-v2. Stress test on Taskonomy with 13 heterogeneous tasks further shows VTP's robustness in extreme settings. Ablations and analysis indicate that VTP's gains stem from adaptive routing rather than increased capacity. The project page is available at https://jacky1128.github.io/VTP/.
Figures & tables
Figure 1: Overview of Virtual Task Prompting (VTP). (a) TaskPrompter [ 71 ] modifies ViT encoder to compensate for the limited capacity. (b) VTP instead maintains T+V virtual states alongside the encoder, where the model read them to T task vectors to interact with image tokens, and then write the results back to T+V bypass, enriching task representations while maintaining a vanilla ViT block. (c) VTP keeps the prompt-guided decoding pipeline but unifies the spatial-channel parts into one.
Figure 2: Qualitative results on Pascal-Context [ 9 ] . VTP timm produces precise fine-grained object segmentation (bags), body-part parsing (human arms) with tighter saliency (suppress the spurious background). Key regions are with red circles. VTP vit-5 and VTP dinov3 and more results on Figure 5 .
Figure 3: Qualitative results on Taskonomy [ 77 ] with 9 dense tasks. VTP timm produces apparently more accurate predictions than the baseline. VTP vit-5 and VTP dinov3 and more results on Figure 8 .
Method
Semseg
Depth
Normal
Edge
Δp
mIoU ↑
RMSE ↓
mErr ↓
odsF ↑
(%) ↑
CNN-based
Cross-Stitch [ 43 ]
36.34
0.6290
20.88
76.38
− 17.94
PAP [ 79 ]
36.72
0.6178
20.82
76.42
− 17.13
PSD [ 81 ]
36.69
0.6246
20.87
76.42
− 17.54
PAD-Net [ 63 ]
36.61
0.6270
20.85
76.38
− 17.68
Table 1: State-of-the-art comparison on NYUD-v2 [ 51 ] (left) and PASCAL-Context [ 9 ] (right). All Transformer-based models use a ViT-L backbone. VTP timm is with ImageNet-pretrained weights, while VTP dinov3 and VTP vit-5 employ DINOv3 [ 52 ] and ViT-5 [ 57 ] without any modification. Δp (%) denotes the average MTL gains relative to TaskPrompter [ 71 ] ( Δp=0 ). ↑(↓) indicates higher (lower) values are better. The best and second-best results per task are in bold and underline , respectively.
Method
Obj.Cls.
Scn.Cls.
Dep.Euc.
Dep.Zbu.
Normal
Curv.
Reshad.
Edge 2D
Edge 3D
Keyp. 2D
Keyp. 3D
Seg. 2D
Seg. 2.5D
Δp
Acc ↑
Acc ↑
RMSE ↓
RMSE ↓
mErr ↓
L2 ↓
L1 ↓
L1 ↓
L1 ↓
L1 ↓
L1 ↓
L1 ↓
L1 ↓
(%) ↑
InvPT [ 69 ]
57.01
65.50
6.251
6.384
29.850
0.9120
0.1810
0.0985
0.0835
0.1150
0.0835
0.2035
0.1590
− 2.34
TaskPrompter [ 71 ]
58.37
66.76
6.150
6.268
29.046
0.8947
0.1754
0.0961
0.0819
0.1107
0.0814
0.1991
0.1566
0.00
VTP timm
58.78
70.96
5.757
5.823
27.505
0.8650
0.1750
0.0804
0.0948
0.0834
0.0805
0.1927
0.1570
+4.50
VTP vit-5
61.73
69.90
5.460
5.620
33.060
0.8494
0.1703
0.0795
0.0862
0.0710
0.0796
0.2020
0.1608
+5.53
VTP dinov3
60.21
68.08
5.440
5.630
28.356
0.8288
0.1447
0.0683
0.0859
0.0719
0.0747
0.1992
0.1515
+9.58
Table 2: Taskonomy results over 13 tasks. Δp (%) denotes the average MTL performance gains relative to TaskPrompter [ 71 ] baseline ( Δp=0 ). ↑(↓) indicates the higher (lower) values are better. The best and second-best results for each task are highlighted in bold and underline , respectively.
Figure 4: Task correlation and UMAP analysis. Pairwise cosine similarity among task vectors at selected layers for (a) TaskPrompter collapses into uniformly high correlation with negligible variation in deep layers. (b) UMAP of VTP’s task vectors shows clear layer-wise clusters, confirming that vectors form an internally cohesive latent space encoding layer-wise discriminative information. (c) VTP exhibits structured task affinities, capturing adaptive inter-task differentiation across depths.
Configuration
Δpascal
Δnyu
(a) Capacity controls
TaskPrompter
0.00
0.00
+ 2 × decoder parameters
− 0.18
− 0.07
+ 2 × vectors per task
+0.09
+0.05
+ 4 × vectors per task
− 0.14
− 0.02
(b) Virtual task number V
Table 3: Ablation (left) and efficiency (right) results. Left: Δpascal and Δnyu denote the MTL gains Δp (%) on Pascal-Context and NYUD-v2 relative to TaskPrompter [ 71 ] . V : virtual task number. VTPs share similar parameters and FLOPs. Throughput results are measured on a single L40 GPU.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Euclidean Depth (Dep. Euc.)
Z-buffer Depth (Dep. Zbu.)
Method
RMSE ↓
Abs ↓
Rel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
Abs ↓
Rel ↓
δ1↑
δ2↑
δ3↑
InvPT [ 69 ]
6.251
4.873
0.412
34.52
60.18
75.24
6.384
4.965
0.420
33.87
59.43
74.61
TaskPrompter [ 71 ]
6.150
4.784
0.398
36.10
61.85
76.43
6.268
4.871
0.406
35.42
61.06
75.78
VTP timm
5.757
4.402
0.353
39.85
65.23
79.16
5.823
4.456
0.374
39.17
64.52
78.48
VTP vit-5
5.460
4.118
0.358
42.73
67.95
81.30
5.620
4.262
0.352
41.55
66.88
80.42
VTP dinov3
5.440
4.093
0.337
43.15
68.40
81.72
5.630
4.240
0.348
41.92
67.25
80.85
Appendix
Table 4: Full depth estimation metrics on Taskonomy. We report RMSE, absolute error (Abs), relative error (Rel), and threshold accuracies ( δ1 , δ2 , δ3 ) for both Euclidean depth and z-buffer depth. ↓ indicates lower is better; ↑ indicates higher is better. Best and second-best results are marked in bold and underline , respectively.
Figure 5: Qualitative results on Pascal-Context [ 9 ] . VTP timm produces precise fine-grained object segmentation (bags), body-part parsing (human arms) with tighter saliency (suppress the spurious background). VTP vit-5 and VTP dinov3 further sharpen the details. Key regions are with red circles.
Figure 6: Additional qualitative results on Pascal-Context (Set 1). Two multi-person outdoor scenes that stress-test inter-individual disambiguation. Left: a man holding an infant against a cluttered natural background. TaskPrompter [ 71 ] (red circles) lets semantic labels bleed across the person with background contour in segmentation, fragments body-part assignments at the overlapping arms in parsing, and over-highlights background clutter in boundary. Right: a crowded group where closely spaced individuals share similar appearance. TaskPrompter’s parsing labels entangle across adjacent persons, with leg assignments bleeding from one individual into the next. In both scenes, VTP timm recovers coherent per-person segmentation masks, disentangled body-part labels, and tighter saliency focus. VTP vit-5 and VTP dinov3 [ 57 , 52 ] further resolve fine-grained inter-person boundaries, saliency, and produce geometrically smoother surface normals, approaching the ground-truth.
Figure 7: Additional qualitative results (Set 2). A challenging street scene that stress-tests two orthogonal challenges simultaneously: extremely small and thin semantic objects (distant poles, a tiny person occupying only a handful of pixels) and dense edge structures from road markings, building contours, and lampposts. TaskPrompter [ 71 ] (red circles) drops these tiny objects entirely from its segmentation and parsing maps, and produces a large spurious saliency blob on the background sky. VTP timm begins to recover the thin structures in segmentation and parsing while suppressing the false saliency response. VTP vit-5 and VTP dinov3 further sharpen the objects (the blue and green masks in Semseg) and produce boundaries that capture both macro building contours and micro pole edges.
Figure 8: Qualitative results on Taskonomy [ 77 ] . An indoor scene with 9 dense prediction tasks under the extreme 13-task joint training regime. From top to bottom, predictions become progressively finer and more accurate. TaskPrompter [ 71 ] produces over-smoothed depth maps, flat curvature, and imprecise edges, while VTP timm through VTP dinov3 successively recover depth discontinuities, surface details, and region coherence, converging toward the ground truth. This progressive improvement requires no additional computational cost. Only the backbone weights are swapped among VFMs.
Method
Backbone Setup
Semseg
Parsing
Saliency
Normal
Edge
Δp
mIoU ↑
mIoU ↑
maxF ↑
mErr ↓
odsF ↑
(%) ↑
(1) Standard PEFT on Frozen ViT
LoRA [ 20 ] (r=16)
ViT-L, frozen + LoRA
74.62
63.18
82.45
14.85
71.20
− 6.84
LoRA [ 20 ] (r=64)
ViT-L, frozen + LoRA
76.18
64.92
83.10
14.52
71.85
− 5.21
(2) Hard Parameter Sharing with VFM
DINOv3 [ 52 ] + HPS heads
ViT-L, fully tunable + per-task heads
81.15
70.35
84.80
13.92
73.65
+0.23
Appendix
Table 5: Comparisons to further ablating VTP’s effectiveness and contribution on Pascal-Context [ 9 ] . We compare VTP against (1) standard PEFT (LoRA) on a frozen ViT, (2) HPS with DINOv3 and task-specific heads, (3) TaskPrompter forcibly adapted to DINOv3, (4) multi-task PEFT methods (Polyhistor, VMT-Adapter, MTLoRA) with their original frozen-backbone setting, (5) the same multi-task PEFT methods extended with a fully tunable ViT-L backbone, and (6) our VTP timm . Δp is computed relative to TaskPrompter [ 71 ] . The results demonstrate that the effectiveness of VTP is attributed more to its prompting design, rather than the VFMs or the added capacity. Although it shares similar technical formation with those PEFT methods, their goals are different. VTP as a multi-task model architecture jointly trains the base model and the VTP modules, surpassing PEFT methods on both fully and partially fine-tune scenarios.