Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China · Pengcheng Laboratory, Shenzhen 518108, China · School of Arts, Peking University, Beijing 100871, China
Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.
Figures & tables
Fig. 1 : Attention maps from different methods, highlighting their representational focus. DINO excels at capturing high-level semantics, while MAE and MultiMAE’s attention is directed toward low-level features. In contrast, our C2FMAE effectively captures features across all levels, successfully building a more robust hierarchical representation.
Fig. 2 : C2FMAE pre-training framework. Multi-granular data (RGB, Instance, Semantic masks) is first masked by Progressive Masking and then concatenated, and fed to a transformer encoder. Encoded tokens subsequently flow into a cascaded decoder with three task-specific blocks. Each block is a standard Transformer decoder block, composed of self-attention, cross-attention, and feed-forward network layers. We use linear layers as the final predictor. As training progresses, the masking strategy transitions from semantic-guided masking to instance-guided masking, and finally to random masking to build hierarchical visual representations.
Fig. 3 : The variation curves of αI and αS during the training process.
Method
Model
Modality
Masking
PT Epoch
PT Cost
Acc.
Scratch
ViT-B
−
−
−
−
82.3
MoCo v3 [ 43 ]
ViT-B
RGB
−
300
−
83.2
DINO [ 3 ]
ViT-B
RGB
−
300
−
82.8
BEiT [ 4 ]
ViT-B
RGB
Random
800
∼ 7.0 ×
83.2
MAE [ 5 ] *
ViT-B
RGB
Random
400
∼ 1.0 ×
82.9
MAE [ 5 ]
ViT-B
RGB
Random
1600
∼ 4.0 ×
83.6
TABLE I : Performance comparison of self-supervised methods on ImageNet-1K. We report fine-tuning Top-1 accuracy (%) of ViT-B. The input size is 224×224 . PT Cost is the relative time to MAE (400 epochs), which is taken as 1.0. * indicates results reproduced using the official code. † means using multi-granular data as input for fine-tuning.
Method
Model
PT Epoch
AP b
AP m
MoCo v3
ViT-B
300
47.3
42.2
BEiT
ViT-B
800
35.6
32.6
MAE
ViT-B
1600
48.3
42.5
CAE
ViT-B
800
49.8
43.9
iBOT
ViT-B
1600
48.3
42.7
ConMIM
ViT-B
800
47.8
42.5
TABLE II : Object detection and instance segmentation results on the COCO dataset, with evaluation metrics of AP b ( % ) and AP m ( % ). The pre-trained ViT-B backbone is integrated into the Mask R-CNN framework for end-to-end fine-tuning.
Method
Model
PT Epoch
mIoU
MoCo v3
ViT-B
300
47.3
BEiT
ViT-B
800
47.1
MAE
ViT-B
1600
48.1
MaskFeat
ViT-B
1600
48.8
CAE
ViT-B
800
48.8
SemMAE
ViT-B
800
46.3
TABLE III : Semantic segmentation results on the ADE20K dataset, with evaluation metrics of mIoU. The pre-trained ViT-B backbone is integrated with UperNet for end-to-end fine-tuning.
Method
IN-A
IN-R
IN-S
IN-C ↓
Score
MAE
35.9
48.3
34.5
51.7
41.8
MFM
32.7
48.6
34.8
47.5
42.2
MultiMAE
33.2
49.9
37.2
49.6
42.7
C2FMAE
35.2
50.6
37.4
48.8
43.6
TABLE IV : Robustness results of ViT-B on various ImageNet OOD variants. Top-1 accuracy is reported for all datasets, except for ImageNet-C, where the mean corruption error (mCE) is the evaluation metric. And (1-mCE) is used to calculate the average score.
Configuration
Top-1
Baseline (MultiMAE)
82.7
+ Our Dataset (R+I+S)
83.0(+0.3)
+ Cascaded Decoder
83.3(+0.3)
+ Progressive Masking (Ours)
83.7 (+0.4)
TABLE V : Component analysis of C2FMAE on ImageNet-1K. Starting from a MultiMAE baseline, we incrementally add our proposed components.
TABLE VI : C2FMAE ablation experiments on ImageNet-1K. The fine-tuning Top-1 accuracy (%) is reported. The default settings include 98 for the number of input tokens, progressive masking with the order of SG→IG→RD, a cascaded decoder with cross-attention and task sequence of S→I→R, R+I+S for the input modality, and λS=λI=λR=1 for loss weight. The selected settings are underlined .
Fig. 4 : Predictions of C2FMAE on masked multi-granular data. All the tested images are from the ImageNet-1K validation set and masked with the random masking strategy.
Fig. 5 : Single-modal prediction of C2FMAE and MultiMAE on the ImageNet-1K validation set.
Fig. 6 : Single-modal prediction of C2FMAE with instance masks and semantic masks.
Self-supervised pretraining has transformed computer vision by enabling data-efficient fine-tuning, yet high-resolution pretraining typically requires server-scale infrastructure, limiting custom in-domain training for many research laboratories. Masked Autoencoders (MAE) reduce computation by encoding only visible tokens, but combining MAE with hierarchical downsampling architectures has remained structurally challenging due to dense grid priors and mask-aware design compromises. We introduce AFFMAE, a masking-friendly hierarchical pretraining framework built on adaptive, off-grid token merging. AFFMAE removes dense-grid assumptions while preserving hierarchical scalability during pre-training and fine-tuning. To support this architecture, we developed numerically stable mixed-precision Triton kernels and a lightweight, point-based decoder that can be directly repurposed as a segmentation head. On high-resolution microscopy segmentation, AFFMAE matches MAE finetuning performance on foot process width estimation with ViT backbone at equal parameter counts while being 2x faster during pre-training and halving peak memory usage. Furthermore, AFFMAE achieves up to 5x throughput speedups fine-tuning at the 1024px resolution, providing high-resolution model training on desktop hardware. Code available at https://github.com/najafian-lab/affmae.
David Smerkous, Zian Wang, Behzad Najafian
Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA
Self-supervised pre-training with Vision Transformers, including Masked Autoencoders (MAE), is difficult to apply to gigapixel scientific images. Random masking is poorly matched to the structured, multi-scale morphology of scientific data, while uniform tokenization produces prohibitively long sequences that make O(N2) attention impractical. We propose SGMA, a structure-guided masked autoencoding framework for ultra-high-resolution scientific images. SGMA couples two components: a content-adaptive quadtree tokenizer that compresses gigapixel images into a fixed-length sequence, and a structure-conditioned masking process that biases reconstruction toward spatially informative regions. To stabilize this process across scales, we introduce Damped Accumulation (DA), which aggregates signal-dependent responses across the tree into a structure canvas used to guide masking. The resulting pre-training task preserves fine microstructure while remaining compatible with standard ViT encoders and MAE-style reconstruction. Across electron microscopy, whole-slide optical microscopy, and X-ray CT datasets, SGMA consistently outperforms MAE baselines. It achieves 95.68% Dice on the 8K x 8K x 28K SpringXCT dataset, improving over the same-architecture MAE baseline by +13.00 points, and 83.21% Dice on the 32K^2 WSI PAIP dataset, improving by +16.84 points, while providing up to a 24.8x inference speedup.
Enzhi Zhang, Du Wu, Rui Zhong +18
Hokkaido University, Japan · RIKEN Center for Computational Science (R-CCS), Japan · Oak Ridge National Laboratory, USA +4
Recent advances in multimodal large language models largely rely on CLIP-based visual encoders, which emphasize global semantic alignment but struggle with fine-grained visual understanding. In contrast, DINOv3 provides strong pixel-level perception yet lacks coarse-grained semantic abstraction, leading to limited multi-granularity reasoning. To address this gap, we propose Granulon, a novel DINOv3-based MLLM with adaptive granularity augmentation. Granulon introduces a text-conditioned granularity Controller that dynamically adjusts the visual abstraction level according to the semantic scope of the textual input, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering to produce compact, semantically rich visual tokens. This design enables unified "pixel-to-fine-to-coarse" reasoning within a single forward pass. Extensive and interpretable experiments demonstrate that Granulon improves accuracy by ~30% and reduces hallucination by ~20%, outperforming all visual encoders under identical settings.
Junyuan Mao, Qiankun Li, Linghao Meng +5
National University of Singapore · Nanyang Technological University