Recent image generators exhibit rich visual priors that can support dense visual perception. However, existing approaches typically encode multiple perception tasks into a shared parameterization, making joint training sensitive to the mixture and weighting of heterogeneous data and prone to cross-task interference. To address this issue, we introduce ReVision, an instruction-conditioned parameter generation framework built on a frozen multimodal understanding model and a frozen image generator. Given an image and a free-form instruction, the understanding model jointly encodes them into a multimodal image--instruction representation, which a trainable parameter generator maps to transient low-rank updates for the generator. Each image--instruction pair induces a distinct set of transient prediction parameters, enabling instance-specific adaptation while both pretrained backbones remain unchanged. ReVision requires no task-specific heads, explicit task identifiers, or stored adapters. A shared RGB output interface unifies segmentation, depth estimation, and surface-normal prediction. Across diverse dense prediction benchmarks, ReVision achieves competitive performance with state-of-the-art methods while supporting flexible task specification through natural-language instructions. Our analysis further reveals substantial gradient disagreement across tasks, highlighting the conflict introduced when heterogeneous objectives share a single parameterization. Meanwhile, the generated updates exhibit structured task- and input-dependent organization in weight space, further validating instance-level parameter generation for multi-task perception.
Figures & tables
Figure 1: Parameter-space decoupling for visual perception. (a) Fine-tuning repurposes a pretrained generator through a shared set of optimized parameters. (b) Unified models encode understanding, generation, and perception within one parameterization. (c) ReVision keeps both backbones frozen and generates instruction-conditioned low-rank updates for each input.
Figure 2: Overview of ReVision. A frozen multimodal model extracts an image–instruction representation that conditions a parameter generator to produce input-dependent low-rank updates for a frozen image generator. Dense perception tasks are unified as RGB generation, with only the parameter generator trained using a flow-matching objective.
Method
Zero-shot
mIoU ↑
SegMan-L ( Fu et al., 2025b )
✗
84.2
SenseNova-Vision ( Han et al., 2026 )
✗
71.2
APE-D ( Shen et al., 2024 )
✓
44.2
OpenSeeD ( Zhang et al., 2023 )
✓
47.8
X-Decoder ( Zou et al., 2023 )
✓
52.0
SAM 3 ( Carion et al., 2026 )
✓
65.2
Table 1: Semantic segmentation on Cityscapes.
Model
Zero-shot
cgF1↑
IL_MCC ↑
pmF1↑
SAM 3 ( Carion et al., 2026 )
✗
54.1
0.82
66.1
SAM 3 ( Carion et al., 2026 ) + Llama 3.2 (ft)
✗
61.2
0.86
70.8
gDino-T ( Liu et al., 2024 )
✓
3.3
0.15
16.2
LLMDet-L ( Fu et al., 2025a )
✓
6.5
0.21
27.3
Gemini 2.5 ( Comanici et al., 2025 )
✓
13.0
0.29
46.1
APE-D ( Shen et al., 2024 )
✓
16.4
0.40
36.9
Table 2: Instance segmentation on SA-Co/Gold.
Model
Zero-shot
cIoU ↑
HyperSeg-Phi2-2.7B ( Wei et al., 2025 )
✗
79.4
X-SAM-Phi3-3.8B ( Wang et al., 2026a )
✗
83.8
SenseNova-Vision ( Han et al., 2026 )
✗
80.3
HybridGL ( Liu and Li, 2025 )
✓
51.3
LocalizationHeads-LLaVA-1.5-13B ( Kang et al., 2025 )
✓
67.7
SAM 3 ( Carion et al., 2026 ) + Gemini 2.5 Pro
✓
73.4
Table 3: Referring expression segmentation results on RefCOCOg val (UMD).
Method
Zero-shot
AbsRel ↓ / δ1↑
NYUv2
KITTI
ETH3D
ScanNet
DIODE
iBims1
Marigold ( Ke et al., 2024 )
✓
5.5 / 96.4
9.9 / 91.6
6.5 / 95.9
6.4 / 95.2
30.8 / 77.3
- / -
DICEPTION ( Zhao et al., 2025 )
✓
6.1 / 96.0
6.9 / 94.9
5.0 / 97.5
7.2 / 94.4
28.9 / 72.2
- / -
FE2E ( Wang et al., 2026b )
✓
4.1 / 97.7
6.6 / 96.0
3.8 / 98.7
4.4 / 97.5
22.8 / 81.2
- / -
Lotus-2 ( He et al., 2025a )
✓
4.1 / 97.6
6.7 / 94.5
4.6 / 98.1
4.2 / 97.6
22.1 / 75.2
- / -
Vision Banana ‡ ( Gabeur et al., 2026 )
✓
8.1 / 94.8
10.7 / 91.5
10.3 / 93.5
- / -
- / -
7.8 / 93.4
Table 4: Comparison on monocular depth estimation. ∗ SenseNova-Vision is not strictly zero-shot due to dense geometry supervision and possible evaluation-source overlap. ‡ Vision Banana reports metric-depth without per-image affine alignment and is included for reference.
Methods
Zero-shot
Indoor
Outdoor
Average
NYUv2
DIODE-indoor
ScanNet
VKitti
mean ↓
median ↓
mean ↓
median ↓
mean ↓
median ↓
mean ↓
median ↓
mean ↓
median ↓
Marigold ( Ke et al., 2024 )
✓
19.606
11.828
20.864
11.134
16.671
12.084
21.284
12.268
–
–
DSINE ( Bae and Davison, 2024 )
✓
17.017
10.190
16.4
8.4
18.453
13.871
16.2
8.3
28.9
9.9
StableNormal ( Ye et al., 2024 )
✓
17.168
10.028
19.707
10.527
13.701
9.46
18.098
10.097
–
–
Lotus-2-Normal ( He et al., 2025a )
✗ ∗
16.558
–
16.9
N/A
18.575
N/A
14.2
N/A
28.894
9.677
Table 5: Comparison on surface normal estimation. We report mean and median angular errors ( ↓ ). ∗ Lotus-2-Normal is trained on Virtual KITTI 2, so its VKitti result is not zero-shot. ‡ SenseNova-Vision includes ScanNetV2 in training and is not strictly benchmark-disjoint.
Figure 3: Gradient geometry across perception tasks. (a) Mean pairwise cosine similarity between gradients of the generated LoRA parameters. (b) Conflict ratio, defined as the fraction of gradient pairs with negative cosine similarity. (c) Cosine-similarity distributions for within-task and cross-task pairs. Dashed lines mark zero and the distribution means.
Method
Cityscapes
SA-Co/Gold
RefCOCOg
Depth
Normal
mIoU ↑
mAcc ↑
pixAcc ↑
pmF1↑
cIoU ↑
AbsRel ↓
δ1↑
mean ∘↓
med. ∘↓
Base Image Generator
20.29
36.79
59.91
4.32
7.64
0.5522
0.4662
105.84
109.21
Shuffled Text (within task)
67.74
79.19
94.20
-
46.9
-
-
-
-
Shuffled Image (within task)
68.03
79.34
94.21
48.5
51.7
0.062
0.950
18.83
11.83
Shuffled condition (cross task)
22.16
42.65
51.71
19.6
15.7
5.85
0.286
93.61
96.00
ReVision (Ours)
70.27
81.36
94.79
51.1
54.2
0.049
0.971
17.99
11.02
Table 6: Condition-routing ablation. Only the parameter-generator condition is perturbed; image-generator inputs remain unchanged. We shuffle text or images within a task to break sample-level correspondence, and route conditions across tasks to break task-level correspondence. SA-Co/Gold, depth, and normals are omitted due to similar text conditions.
Figure 4: Generated updates in parameter space. The same t-SNE embedding of 7,168-dimensional LoRA update sketches is colored by (a) CLIP image-semantic clusters ( K=7 ) and (b) task identity. Silhouette scores are shown below each panel.
Sem.
Inst.
Ref.
Depth
Normal
All
Centroid norm
359.9
380.2
392.8
352.7
330.2
316.9
Mean distance
8.6
25.1
13.4
75.2
92.6
181.6
Dispersion
0.024
0.066
0.034
0.213
0.281
0.573
Offset / spread
14.8
5.5
11.9
2.8
2.5
–
Participation ratio
2.12
3.08
5.85
2.48
1.84
1.91
Effective rank (mean)
13.20
13.20
13.14
13.61
13.65
13.36
Table 7: Geometry of the generated update family. Centroid norm measures the magnitude of the mean update, and mean distance measures the spread around it. Their ratio defines dispersion , the normalized update spread. Offset/spread measures task separation from the global centroid relative to within-task variation, while participation ratio estimates the effective dimensionality. We also report the effective rank and relative Frobenius norm ∥ΔWs∥F/∥Ws∥F of the generated LoRA updates. Detailed definitions are provided in the Appendix E .
Figure 5: Layerwise and module-wise profile of the generated LoRA updates. Relative update magnitude ∥ΔWs∥F/∥Ws∥F across (a) double-stream and (b) single-stream blocks, with dashed lines indicating the all-site mean. (c) Mean relative update magnitude for each projection type.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
stage 1
stage 2
resolution
256×256
512×512
steps
10 000
20 000
micro-batch / GPU
24
8
gradient accumulation
1
1
optimizer
AdamW, β=(0.9,0.999) , weight decay 10−4
peak learning rate
2×10−5
2×10−5
Appendix
Table 8: Training configuration. The stage transition loads only the ReVision weights: global step, optimizer state, learning-rate schedule, EMA and data-order RNG are all reset, so stage 2 performs a full warmup to the same peak learning rate rather than inheriting stage 1 ’s step counter.
Table 9: Training data sources. Datasets and sample counts used in the 2562 and 5122 training stages.
Figure 6: Examples from the datasets used for different segmentation tasks.
Figure 7: Examples from the depth and surface-normal datasets.
Module
Shape
Params
Instruction bridge
Linear(4096→1536)
6.3M
Learnable queries Q
3584×1536
5.5M
Transformer blocks
10×[Linear(d→9d)+Linear(4d→d)]
306.7M
Per-site decode heads
112×[2Linear(d→m)+LoRAdecoding]
599.8M
Total
918.3M
Appendix
Table 10: Parameter budget of the ReVision parameter generator. Parameter counts omit linear-layer biases. The per-site decode heads dominate the budget because they produce LoRA factors at the native widths of the generator backbone.
Figure 8: Qualitative results across dense perception tasks. ReVision follows free-form image–instruction inputs to produce pixel-space outputs for semantic segmentation, instance segmentation, referring segmentation, depth estimation, and surface-normal prediction. Different colors and output conventions are specified directly by the instruction.