SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition

Le Yang; Yucheng Huang; Junyu Chen; Qi Dang; Xin Kong; Haifeng Huo; Qiang Zhao

cs.CVOct 27, 2026

SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition

Authors: Le Yang, Yucheng Huang, Junyu Chen, Qi Dang, Xin Kong, Haifeng Huo, Qiang Zhao

Organizations: Huawei Technologies Co., Ltd.

Abstract

When adapting self-supervised Vision Transformers (ViTs) to dense prediction, existing methods often treat the ViT as a passive feature extractor and may underuse the rich priors encoded in its representations, including saliency, global context, and high-frequency details. We propose SAGE (Saliency-Annealed Gated Encoder), an enhanced ViT-Adapter backbone that explicitly exploits these priors through three complementary innovations: (1) a saliency-guided architecture that extracts DINO-style class-token (CLS) saliency to guide feature gating and high-frequency enhancement, together with dynamic multi-layer aggregation and a local-detail branch; (2) an annealing-based training paradigm designed to improve convergence and feature quality during early training while reducing feature drift later in training; and (3) a two-stage, quality-aware training mechanism that smoothly transitions from focal loss to Varifocal Loss (VFL) within a single training run. Across the evaluated tasks (Fig.~\ref{fig:fig_com}), SAGE achieves state-of-the-art (SOTA) or competitive results on COCO (62.8 box AP and 55.3 mask AP), COCO keypoint estimation (78.5 AP), ADE20K (60.3 mIoU for semantic segmentation and 44.9 mask AP for instance segmentation), TinyImageNet classification (94.76% Top-1 accuracy), and DIOR-R oriented object detection (78.8 AP $_{50}$ ). The same backbone is used across tasks, with task-specific decoders. To the best of our knowledge, SAGE is the first framework to achieve multi-task state-of-the-art results on Huawei Ascend NPUs. Code and pretrained models will be released after internal review.

Figures & tables

Fig. 1: Cross-task performance comparison of SAGE across six vision tasks on seven benchmarks, normalized to the strongest non-SAGE baseline (100%). For fair comparison, we include baselines with comparable backbone sizes and evaluate all methods without test-time augmentation (TTA).

Fig. 2: Overview of the SAGE architecture. The input image is processed in three parallel paths: (1) SPM CNN pyramid generating multi-scale features

\mathbf{C}_{1:4}

; (2) ViT backbone producing all layer outputs (patch/CLS/register tokens), from which the saliency prior

\mathbf{S}

and multi-depth aggregated features

\hat{\mathbf{X}}_{1:4}

are derived; (3) a 2

\times

2 local-detail branch extracting

\mathbf{L}

. The aggregated features are per-scale enhanced (HF/FiLM), interact with SPM queries via MSDeformAttn, and are fused through residual soft gating (

\mathbf{F}_{i}=\mathbf{F}_{spm,i}+\mathbf{X}_{vit,i}\cdot\mathbf{G}_{i}

) and difference-aware local fusion, followed by SyncBN and a task-specific decoder. (a) CLS–Patch similarity prior:

\mathbf{S}

from 2/3-depth tokens via L2-norm cosine similarity, standardization, sigmoid. (b) Multi-depth aggregation: sampled layers fused into

\hat{\mathbf{X}}_{1:4}

via temperature-softmax weights. (c) Difference-aware local fusion:

\mathbf{Y}_{i}=\mathbf{F}_{i}+\sigma(\text{GateNet}([\mathbf{F}_{i},\mathbf{L}^{\prime},|\mathbf{F}_{i}-\mathbf{L}^{\prime}|]))\odot\mathbf{L}^{\prime}

for high-resolution scales.

Method	Backbone	Params	Pretrain Data	bbox AP	bbox [email protected]	segm AP	segm [email protected]
DINO [ 3 ]	ResNet-50	25M	IN-1K	49.0	67.0	—	—
MaskDINO [ 4 ]	ResNet-50	25M	IN-1K	51.7	70.5	45.0	68.0
D-FINE-X [ 6 ]	HGNetv2-B5	30M	O365	59.3	78.0	—	—
RT-DETRv4-X [ 17 ]	HGNetv2-B4	22M	O365	57.0	75.0	—	—
DEIMv2-X [ 7 ]	HGNetv2-B4	22M	O365	57.8	76.0	—	—
DINO [ 3 ]	Swin-L	197M	IN-22K	58.5	78.0	—	—

TABLE I: Comparison with state-of-the-art methods on COCO val2017. All results use single-scale inference without TTA. Parameter counts refer to the backbone. The best results are shown in bold, and the second-best results are underlined.

Configuration	bbox AP	segm AP	$\Delta$ bbox
Baseline	57.70	51.05	—
+ Saliency modulation	58.90	52.25	+1.20
+ Annealing	60.07	53.46	+1.17
+ Dyn. agg. + local detail	60.91	54.23	+0.84
+ Two-stage VFL + sem.	61.10	54.40	+0.19
Full SAGE	61.10	54.40	+3.40

TABLE II: Module contributions (COCO val2017, ViT-B).

Strategy	bbox	[email protected]	$\Delta$ bbox
No anneal ( $\lambda=1$ )	60.15	53.62	—
2-stage (no ramp-up)	59.21	52.61	-0.94
Linear anneal	60.09	53.64	-0.06
3-stage + FREEZE	60.91	54.23	+0.76

TABLE III: Annealing schedule ablation (COCO val2017, ViT-B).

Strategy	Anneal	bbox AP	segm AP	$\Delta$ AP
VFL from scratch	$\times$	60.69	53.80	—
Two-stage VFL	$\times$	60.15	53.62	$-0.54$
VFL from scratch	✓	60.09	53.64	$-0.60$
Two-stage VFL	✓	60.91	54.23	$+0.22$

TABLE IV: Two-stage VFL training strategy ablation (COCO val2017, ViT-B).

Task & Dataset	Method	Backbone	Backbone Params.	Pre-training	Metric	Baseline	SAGE-B	SAGE-L
Object Detection (COCO)	Co-DETR [ 5 ]	Swin-L	197M	IN-22K	Box AP	60.7	61.1	62.8
	RF-DETR-2XL [ 8 ]	ViT-B	86M	O365		60.1
	D-FINE-X [ 6 ]	HGNetv2-B5	30M	O365		59.3
Instance Segmentation (COCO)	MaskDINO [ 4 ]	Swin-L	197M	IN-22K	Mask AP	52.1	54.4	55.3
	ViTDet [ 12 ]	ViT-H	632M	IN-22K		50.9
	ViT-Adapter-L [ 9 ]	ViT-L	304M	IN-22K		51.1

TABLE V: Cross-task comparison of SAGE with representative methods. The best result is shown in bold, and the second-best result is underlined. All reported results are single-modality (vision-only), and single-scale without test-time augmentation (TTA).

Method	Backbone	Params	Pre-training Data	Box AP	Box AP 50	Mask AP	Mask AP 50
CNN Backbones
DINO [ 3 ]	ResNet-50	25M	IN-1K	49.0	67.0	–	–
MaskDINO [ 4 ]	ResNet-50	25M	IN-1K	51.7	70.5	45.0	68.0
D-FINE-X [ 6 ]	HGNetv2-B5	30M	O365	59.3	78.0	–	–
RT-DETRv4-X [ 17 ]	HGNetv2-B4	22M	O365	57.0	75.0	–	–
DEIMv2-X [ 7 ]	HGNetv2-B4	22M	O365	57.8	76.0	–	–

TABLE VI: Detailed comparison on COCO object detection and instance segmentation.

Method	Backbone	Params	Pre-training Data	mIoU (%)
Mask2Former [ 21 ]	Swin-T	28M	IN-1K	47.7
Mask2Former [ 21 ]	Swin-L	197M	IN-22K	56.1
ViT-Adapter-L + UPerNet [ 9 ]	ViT-L	304M	IN-22K	58.0
OneFormer [ 20 ]	Swin-L	219M	IN-22K	58.3
Mask2Former + ViT-Adapter-L [ 9 ]	ViT-L	304M	IN-22K	59.4
MaskDINO [ 4 ]	Swin-L	223M	IN-22K+O365	59.5

TABLE VII: Detailed comparison on ADE20K semantic segmentation.

Method	Backbone	Params	Pre-training	Mask AP (%)
MP-Former [ 31 ]	Swin-L	197M	IN-22K	35.3
Mask2Former + DiNAT	DiNAT-L	200M	IN-22K	35.4
OneFormer	DiNAT-L	200M	IN-22K	37.1
OneFormer	Swin-L	197M	IN-22K	37.8
OneFormer + ViT-P	DiNAT-L + ViT-P	300M	IN-22K	37.8
OneFormer	DiNAT-L	200M	IN-22K	40.2

TABLE VIII: Detailed comparison on ADE20K instance segmentation.

Method	Backbone	Params	Pre-training Data	AP 50 (%)
RiO-DETR-X [ 19 ]	HGNetV2-B5	33M	IN-1K	77.43
RT-DETRv2-OBB-X [ 23 ]	HGNetV2-B5	33M	IN-1K	76.52
YOLO26x-OBB [ 24 ]	YOLO26x	56M	–	76.48
Oriented R-CNN + RingMoE-KC [ 32 ]	RingMoE-KC	1000M	RingMOSS	76.04
RT-DETRv2-OBB-L [ 23 ]	HGNetV2-B4	13.6M	IN-1K	75.69
Oriented R-CNN + SkySense V2 [ 33 ]	MoE	1994M	–	75.29

TABLE IX: Detailed comparison on DIOR-R oriented object detection.

Method	Backbone	Params	Pre-training Data	Top-1 Acc. (%)	Top-5 Acc. (%)
ViT-L/16 [ 34 ]	ViT-L/16	304M	IN-22K	86.43	—
CaiT-S/36 [ 35 ]	CaiT-S/36	68M	IN-22K	86.74	—
DeiT-B/16-D [ 27 ]	DeiT-B/16-D	87M	IN-1K	87.29	—
Swin-L/4 [ 26 ]	Swin-L/4	196M	IN-22K	91.35	—
OCD (single) [ 36 ]	DeiT-B/16-D	87M	IN-1K	90.80	—
OCD (ensemble $\times$ 5) [ 36 ]	DeiT-B/16-D	87M $\times$ 5	IN-1K	92.00	—

TABLE X: Detailed comparison on TinyImageNet image classification.

Method	Backbone	Params	Pre-training Data	AP (%)
RTMPose [ 28 ]	CSPNeXt-L	–	–	75.8
ViTPose [ 29 ]	ViT-B	86M	IN-1K	75.8
ViTPose [ 29 ]	ViT-L	306M	IN-1K	78.3
ViTPose [ 29 ]	ViT-H	631M	IN-1K	79.1
SAGE (ours)	ViT-B	86M	O365	76.0
SAGE (ours)	ViT-L	304M	O365	78.5

TABLE XI: Detailed comparison on COCO 2017 keypoint detection.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 365 papers

Jun 12, 2026

ViT-Up: Faithful Feature Upsampling for Vision Transformers

Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 [email protected], demonstrating that ViT-Up scales favorably with backbone capacity.

Krispin Wandel, Jingchuan Wang, Hesheng Wang

Shanghai Jiao Tong University, Shanghai, China

Semantic Correspondence Vision Transformer Semantic Segmentation

Ask PDF

May 22, 2026

Vision Transformers Need Better Token Interaction

Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.

Linxiang Su

University of Szeged

Visual Representation Learning Vision Transformer Sparse Attention Semantic Segmentation

Ask PDF

May 21, 2026

ASAP: Attention Sink Anchored Pruning

Vision Transformers (ViTs) face severe computational bottlenecks due to the quadratic complexity of self-attention at high resolutions. Existing token reduction methods rely on local metrics - such as single-layer attention scores - that are inherently vulnerable to the attention sink phenomenon, where uninformative tokens are paradoxically preserved over salient foreground objects. We propose ASAP (Attention Sink Anchored Pruning), a training-free framework that recasts this sink as a feature. Modeling ViT information flow as a Lazy Random Walk, ASAP identifies the sink as a dominant accumulator of probability mass. By computing the diffusion distance to the sink within the cumulative transition matrix, ASAP partitions tokens via Radial Diffusion Clustering and compresses background redundancy through Transition Weight Pooling in a single shot. Extensive experiments across image, video, and vision-language tasks demonstrate ASAP outperforms state-of-the-art methods, accelerating throughput by up to 48% while maintaining - or even exceeding - baseline accuracy.

Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1

Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea

Self-Attention Efficient ViTs Attention Sinks Visual Token Pruning Visual Token Compression

Ask PDF

Abstract

Vision Transformer

Explore similar work

ViT-Up: Faithful Feature Upsampling for Vision Transformers

Vision Transformers Need Better Token Interaction

ASAP: Attention Sink Anchored Pruning