From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding

Wenzhao Xiang; Yue Wu; Hongyang Yu; Feng Gao; Fan Yang; Xilin Chen

cs.CVMar 10, 2026

From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding

Authors: Wenzhao Xiang, Yue Wu, Hongyang Yu, Feng Gao, Fan Yang, Xilin Chen

Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China · Pengcheng Laboratory, Shenzhen 518108, China · School of Arts, Peking University, Beijing 100871, China

Abstract

Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.

Figures & tables

Fig. 1 : Attention maps from different methods, highlighting their representational focus. DINO excels at capturing high-level semantics, while MAE and MultiMAE’s attention is directed toward low-level features. In contrast, our C2FMAE effectively captures features across all levels, successfully building a more robust hierarchical representation.

Fig. 2 : C2FMAE pre-training framework. Multi-granular data (RGB, Instance, Semantic masks) is first masked by Progressive Masking and then concatenated, and fed to a transformer encoder. Encoded tokens subsequently flow into a cascaded decoder with three task-specific blocks. Each block is a standard Transformer decoder block, composed of self-attention, cross-attention, and feed-forward network layers. We use linear layers as the final predictor. As training progresses, the masking strategy transitions from semantic-guided masking to instance-guided masking, and finally to random masking to build hierarchical visual representations.

Fig. 3 : The variation curves of

\alpha_{I}

and

\alpha_{S}

during the training process.

Method	Model	Modality	Masking	PT Epoch	PT Cost	Acc.
Scratch	ViT-B	$-$	$-$	$-$	$-$	82.3
MoCo v3 [ 43 ]	ViT-B	RGB	$-$	300	$-$	83.2
DINO [ 3 ]	ViT-B	RGB	$-$	300	$-$	82.8
BEiT [ 4 ]	ViT-B	RGB	Random	800	$\sim$ 7.0 $\times$	83.2
MAE [ 5 ] *	ViT-B	RGB	Random	400	$\sim$ 1.0 $\times$	82.9
MAE [ 5 ]	ViT-B	RGB	Random	1600	$\sim$ 4.0 $\times$	83.6

TABLE I : Performance comparison of self-supervised methods on ImageNet-1K. We report fine-tuning Top-1 accuracy (%) of ViT-B. The input size is

224\times 224

. PT Cost is the relative time to MAE (400 epochs), which is taken as 1.0. * indicates results reproduced using the official code.

\dagger

means using multi-granular data as input for fine-tuning.

Method	Model	PT Epoch	AP b	AP m
MoCo v3	ViT-B	300	47.3	42.2
BEiT	ViT-B	800	35.6	32.6
MAE	ViT-B	1600	48.3	42.5
CAE	ViT-B	800	49.8	43.9
iBOT	ViT-B	1600	48.3	42.7
ConMIM	ViT-B	800	47.8	42.5

TABLE II : Object detection and instance segmentation results on the COCO dataset, with evaluation metrics of AP b (

\%

) and AP m (

\%

). The pre-trained ViT-B backbone is integrated into the Mask R-CNN framework for end-to-end fine-tuning.

Method	Model	PT Epoch	mIoU
MoCo v3	ViT-B	300	47.3
BEiT	ViT-B	800	47.1
MAE	ViT-B	1600	48.1
MaskFeat	ViT-B	1600	48.8
CAE	ViT-B	800	48.8
SemMAE	ViT-B	800	46.3

TABLE III : Semantic segmentation results on the ADE20K dataset, with evaluation metrics of mIoU. The pre-trained ViT-B backbone is integrated with UperNet for end-to-end fine-tuning.

Method	IN-A	IN-R	IN-S	IN-C $\downarrow$	Score
MAE	35.9	48.3	34.5	51.7	41.8
MFM	32.7	48.6	34.8	47.5	42.2
MultiMAE	33.2	49.9	37.2	49.6	42.7
C2FMAE	35.2	50.6	37.4	48.8	43.6

TABLE IV : Robustness results of ViT-B on various ImageNet OOD variants. Top-1 accuracy is reported for all datasets, except for ImageNet-C, where the mean corruption error (mCE) is the evaluation metric. And (1-mCE) is used to calculate the average score.

Configuration	Top-1
Baseline (MultiMAE)	82.7
+ Our Dataset (R+I+S)	83.0(+0.3)
+ Cascaded Decoder	83.3(+0.3)
+ Progressive Masking (Ours)	83.7 (+0.4)

TABLE V : Component analysis of C2FMAE on ImageNet-1K. Starting from a MultiMAE baseline, we incrementally add our proposed components.

TABLE VI : C2FMAE ablation experiments on ImageNet-1K. The fine-tuning Top-1 accuracy (%) is reported. The default settings include 98 for the number of input tokens, progressive masking with the order of SG→IG→RD, a cascaded decoder with cross-attention and task sequence of S→I→R, R+I+S for the input modality, and

\lambda_{S}=\lambda_{I}=\lambda_{R}=1

for loss weight. The selected settings are underlined .

Fig. 4 : Predictions of C2FMAE on masked multi-granular data. All the tested images are from the ImageNet-1K validation set and masked with the random masking strategy.

Fig. 5 : Single-modal prediction of C2FMAE and MultiMAE on the ImageNet-1K validation set.

Fig. 6 : Single-modal prediction of C2FMAE with instance masks and semantic masks.

Self-Supervised Visual Representation Learning

Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.

See all 235 papers

AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Feb 18, 2026David Smerkous, Zian Wang, Behzad NajafianMasked Autoencoders Self-Supervised Pre-Training
Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA
Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding
Sep 25, 2026Enzhi Zhang, Du Wu, Rui Zhong +18Masked Autoencoders Masked Image Modeling
Hokkaido University, Japan · RIKEN Center for Computational Science (R-CCS), Japan · Oak Ridge National Laboratory, USA +4
Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM
Mar 9, 2026Junyuan Mao, Qiankun Li, Linghao Meng +5Large Vision-Language Models Fine-Grained Visual Perception
National University of Singapore · Nanyang Technological University

Feb 18, 2026J/K move · Enter open · S save

AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware

David Smerkous, Zian Wang, Behzad Najafian

Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA

Masked Autoencoders Self-Supervised Pre-Training Efficient ViTs Microscopy Image Analysis Visual Token Compression

Open paper AskPDF arXiv

Self-supervised pretraining has transformed computer vision by enabling data-efficient fine-tuning, yet high-resolution pretraining typically requires server-scale infrastructure, limiting custom in-domain training for many research laboratories. Masked Autoencoders (MAE) reduce computation by encoding only visible tokens, but combining MAE with hierarchical downsampling architectures has remained structurally challenging due to dense grid priors and mask-aware design compromises. We introduce AFFMAE, a masking-friendly hierarchical pretraining framework built on adaptive, off-grid token merging. AFFMAE removes dense-grid assumptions while preserving hierarchical scalability during pre-training and fine-tuning. To support this architecture, we developed numerically stable mixed-precision Triton kernels and a lightweight, point-based decoder that can be directly repurposed as a segmentation head. On high-resolution microscopy segmentation, AFFMAE matches MAE finetuning performance on foot process width estimation with ViT backbone at equal parameter counts while being 2x faster during pre-training and halving peak memory usage. Furthermore, AFFMAE achieves up to 5x throughput speedups fine-tuning at the 1024px resolution, providing high-resolution model training on desktop hardware. Code available at https://github.com/najafian-lab/affmae.

Abstract

Self-Supervised Visual Representation Learning

Explore similar work

AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware

Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding

Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM