Task Adaptation Through Guidance in Frozen Feature Upsampling

Pingan Wang

cs.CVNov 2, 2026

Task Adaptation Through Guidance in Frozen Feature Upsampling

Authors: Pingan Wang

Abstract

Adapting features before upsampling can improve dense prediction, but the gains may arise from changes in feature values, spatial aggregation, or both. We investigate this ambiguity by comparing matched residual adapters before and after a frozen JAFAR upsampler under a shared parent. We construct a readout-null intervention that preserves a frozen linear classifier's coarse logits while allowing spatial attention weights to adapt. Guidance/value exchanges on trained models and retrained fixed-guidance controls test the utility of this adaptation route. In three-seed experiments on PASCAL Visual Object Classes (VOC) and Cityscapes, adaptation before upsampling improves mean intersection over union (mIoU) by 0.807 and 1.413 percentage points, respectively, over adaptation after upsampling. Across both datasets and the JAFAR and AnyUp upsamplers, readout-null adaptation produces mean gains of 1.379-1.987 points over the corresponding frozen linear baselines. Retrained controls further support the benefit of adaptive spatial guidance. These findings explain how dense predictions can improve despite unchanged coarse classifier outputs and identify spatial guidance as a useful target when adapting frozen feature upsampling systems.

Figures & tables

Fig. 1: A fixed coarse readout need not fix the dense prediction. The equality is a schematic consequence of $\Delta B=0$ ; the masks are retained JAFAR predictions for the original Cityscapes q90 example, revisited in Fig. 10 .

Fig. 2: Local placement of the pointwise residual relative to the frozen image-conditioned upsampler

U_{I}

. PreMLP adapts coarse features and PostMLP adapts dense features; both use a common frozen nonlinear parent. Grids are schematic.

Fig. 3: Readout-null adaptation with an illustrative Cityscapes input. The adapted coarse field conditions the keys and supplies values, while the fixed projector preserves the affine coarse logits. Feature image fields are PCA views; grid glyphs are schematic.

Fig. 4: Guidance and value paths in ordinary Pre and FGPre. Ordinary Pre uses $G=V=T_{o}(F)$ ; FGPre computes $A(I,F)$ under no_grad and mixes differentiable $V=T_{f}(F)$ . The paired VOC control uses separately trained residuals with identical initialization and data streams.

Setting	Comparison	Replication	Question
JAFAR: VOC, Cityscapes	PreMLP vs. PostMLP	3 seeds per dataset	Adapter placement
JAFAR: VOC, Cityscapes	PreNull vs. A20	3 seeds per dataset	Readout-null adaptation
JAFAR: VOC	ordinary Pre vs. FGPre	3 paired seeds	Adaptive guidance
JAFAR: VOC	Six checkpoint-exchange conditions	3 retained models	Guidance/value routes
JAFAR: Cityscapes	Bilinear swap; PreNull vs. FGPre	3 training seeds	Operator and strategy controls
JAFAR: VOC	PreNull vs. ordinary Pre	3 paired seeds	Attention and training risk

TABLE I: Experimental comparisons. Checkpoint exchanges and bilinear swaps use already trained models.

Strategy	Frozen parent	VOC total	City total
A20	None (parent)	8,085	7,315
MLP142/141	None (parent)	57,673	56,983
PreMLP	MLP142/141	107,273	106,583
PostMLP	Same MLP	107,273	106,583
ordinary Pre	A20	57,685	Not fitted
PreNull	A20	57,685	56,915

TABLE II: JAFAR strategies and task-head-plus-adapter parameter counts. “Parent” is trained first and then frozen; the trainable residual has 49,600 parameters in every residual branch. VOC/City counts reflect 21/19 output classes.

Setting	Strategy	Seed 1	Seed 2	Seed 3	Mean $\pm$ SD
VOC confirmation	A20	84.300	84.339	84.390	$84.343\pm 0.045$
VOC confirmation	MLP142	84.562	85.060	84.914	$84.845\pm 0.256$
VOC confirmation	PreMLP	85.355	85.896	85.612	$85.621\pm 0.270$
VOC confirmation	PostMLP	84.571	85.034	84.839	$84.814\pm 0.232$
VOC confirmation	PreNull	85.712	85.736	85.787	$85.745\pm 0.039$
VOC guidance	ordinary Pre	85.411	85.487	85.492	$85.463\pm 0.045$

TABLE III: Final pooled mIoU (%) for the new training cohorts. Each column is one paired training seed;

\pm

denotes sample SD over three seeds. VOC guidance controls use the same A20 parents but fresh paired residual training. Cityscapes includes a zero-fit bilinear swap.

Fig. 5: Paired JAFAR endpoints across three training seeds. Text reports mean differences and 97.5% seed-

t

intervals for the two primary comparisons within each dataset.

Condition	Guidance	Values	Mean
NativeA20	Original	Original	84.343330
SplitA20	Original	Original	84.343330
FixedGuideNull	Original	Null-adapted	84.343323
NativeFullNull	Adapted	Null-adapted	85.744981
SplitFullNull	Adapted	Null-adapted	85.744981
NullGuideOnly	Adapted	Original	85.744980

TABLE IV: VOC frozen-checkpoint exchanges: mean mIoU (%) over the same three readout-null checkpoints. Six decimal places show numerical differences. All conditions are zero-fit forward interventions.

Strategy	Time (s)	Peak allocation (GiB)
ordinary Pre	$1640.1\pm 1.2$	24.80
FGPre	$1071.1\pm 0.6$	13.30

TABLE V: Measured JAFAR VOC guidance-training cost: 7,320 updates per branch on the same RTX 4090 cloud instance (47.4 GiB reported device memory). Time is training-loop wall time (mean

\pm

sample SD); memory is mean peak PyTorch allocation.

Fig. 6: Cityscapes class-IoU effects for PreMLP minus PostMLP, computed from pooled class counts. All three seeds and 19 official classes are shown on one centered colour scale.

Setting	Seed	A20	PreNull	Ordinary Pre	FGPre
VOC	1	84.145	85.530	84.912	84.417
	2	84.403	85.773	85.134	84.618
	3	84.340	85.720	85.004	84.507
	Mean $\pm$ SD	$84.296\pm 0.135$	$85.674\pm 0.127$	$85.017\pm 0.112$	$84.514\pm 0.101$
Cityscapes	1	61.605	63.057	62.777	61.600
	2	61.565	63.001	62.804	61.576

TABLE VI: AnyUp pooled mIoU (%): all training seeds and mean

\pm

sample standard deviation. Each residual freezes its own seed-matched A20 parent.

Fig. 7: Paired AnyUp improvements. Small markers show three training seeds; diamonds and bars show means and 98.75% seed-

t

intervals, adjusted across the four contrasts.

Fig. 8: Attention mass by distance and ground-truth key class. Cells are equal-image means; adapted panels average three seeds, while each shared reference appears once. Distance is in 14-pixel units. All panels use the same color scale.

Fig. 9: Validation gains versus canonical training cross-entropy changes (PreNull minus ordinary Pre). Each point shows one paired training-seed contrast.

Fig. 10: Real outputs of both frozen operators on the original JAFAR-selected q10/q90 cases and crops. Row annotations are whole-image PreNull–A20 differences; q10/q90 refer only to JAFAR selection. The first seed of each operator is used, with a shared class palette within each dataset.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 427 papers

DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models
Mar 13, 2026Xiaoqiong Liu, Heng FanVision Foundation Model Adaptation
Department of Computer Science and Engineering, University of North Texas
PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks
Aug 3, 2026Deepank Singh, Anurag Nihal, Vedhus HoskereDepth Estimation Vision Foundation Model Adaptation
University of Houston
RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation
Jun 22, 2026Yuchuan Ding, Linfei Li, Lin Zhang +1Dense Prediction
School of Computer Science and Technology, Tongji University, Shanghai, China

Mar 13, 2026J/K move · Enter open · S save

DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

Xiaoqiong Liu, Heng Fan

Department of Computer Science and Engineering, University of North Texas

Vision Foundation Model Adaptation

Open paper AskPDF arXiv

Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp

Abstract

Semantic Segmentation

Explore similar work

DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks

RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation