Organizations: Hokkaido University, Japan · RIKEN Center for Computational Science (R-CCS), Japan · Oak Ridge National Laboratory, USA · A*STAR, Singapore · Institute of Science Tokyo, Japan · Japan Synchrotron Radiation Research Institute (JASRI), Japan · RIKEN SPring-8 Center, Japan
Self-supervised pre-training with Vision Transformers, including Masked Autoencoders (MAE), is difficult to apply to gigapixel scientific images. Random masking is poorly matched to the structured, multi-scale morphology of scientific data, while uniform tokenization produces prohibitively long sequences that make O(N2) attention impractical. We propose SGMA, a structure-guided masked autoencoding framework for ultra-high-resolution scientific images. SGMA couples two components: a content-adaptive quadtree tokenizer that compresses gigapixel images into a fixed-length sequence, and a structure-conditioned masking process that biases reconstruction toward spatially informative regions. To stabilize this process across scales, we introduce Damped Accumulation (DA), which aggregates signal-dependent responses across the tree into a structure canvas used to guide masking. The resulting pre-training task preserves fine microstructure while remaining compatible with standard ViT encoders and MAE-style reconstruction. Across electron microscopy, whole-slide optical microscopy, and X-ray CT datasets, SGMA consistently outperforms MAE baselines. It achieves 95.68% Dice on the 8K x 8K x 28K SpringXCT dataset, improving over the same-architecture MAE baseline by +13.00 points, and 83.21% Dice on the 32K^2 WSI PAIP dataset, improving by +16.84 points, while providing up to a 24.8x inference speedup.
Figures & tables
Figure 1: Overview of SGMA . A structure-guided quadtree decomposes the image and accumulates per-level spatial responses by superimposing scale-dependent Gaussian noise. The resulting structure-conditioned noise field determines where masks are placed on the image. When applied to fine-tuning SAM [ 31 ] on HydrogelTEM-1K electron microscopy data, SGMA compresses the sequence length by a large margin while maintaining key spatial structures, enabling stronger fine-tuning performance.
Figure 2: Adaptive structural patches and spatial information from transmission electron microscopy: HydrogelTEM-1K [ 34 ] , X-ray CT: SpringXCT-8K [ 56 ] , and WSI microscopy: PAIP-32K [ 30 ] datasets. These datasets span centimeter to nanometer scales and include biomedical, macroscopic, and microscopic samples. SGMA adapts to the multi-scale micro-structure of each domain for MAE pre-training.
Figure 3: We compared the segmentation difference between MAE-SAM [ 31 ] and SGMA -SAM on TEM HydrogelTEM-1K, X-ray CT SpringXCT-8K, and WSI PAIP-32K datasets. At the same GPU compute budgets, SGMA -SAM can go down to patch size of 8×8 (vs. 1,024×1,024 at best for SAM before going OOM). As a result, SGMA -SAM can extract and express mask details better than MAE-SAM, with the gap in accuracy favoring SGMA -SAM as the resolution gets higher.
Dataset
Model
Patch Size
Sequence Length
Dice Score (%)
Performance Improvement
HydrogelTEM-1K
SAM 2 [ 44 ]
8
16384
69.87
-
UNet [ 46 ]
-
-
71.03
-
UNETR [ 20 ]
16
4096
70.59
-
FT-MAE-SAM
8
16384
72.31
(Baseline)
FT-MAE-SAM 2
8
16384
73.12
(Baseline)
FT-MAE-UNETR
16
4096
74.56
(Baseline)
Table 1: Quantitative segmentation results (Dice Score % ↑ ) on all datasets. Our SGMA pre-training method is compared against standard baselines and MAE pre-training on identical architectures. The "Improvement" column shows the Dice Score gain of our SGMA model over its equivalent MAE-trained baseline (e.g. FT-SGMA-UNETR vs. FT-MAE-UNETR).
Datasets
GPUs
Model
Time (s/img)
Seq. Len.
Dice (%)
Speedup ( × )
HydrogelTEM-1K
4
MAE-SAM-8 [ 22 , 31 ]
0.38261
16384
72.31
1 ×
4
SGMA-SAM (ours)
0.09913
8194
72.56
3.86 ×
SpringXCT-8K
128
MAE-SAM-128 [ 22 , 31 ]
2.5168
4096
82.68
1 ×
128
SGMA-SAM (ours)
0.3512
1024
89.37
7.17 ×
PAIP-32K
512
MAE-SAM-1024 [ 22 , 31 ]
8.9812
16384
62.34
1 ×
512
SGMA-SAM (ours)
0.3663
2048
76.08
24.8 ×
Table 2: Inference efficiency under the same compute budget (identical GPU count per row) for MAE-SAM and our SGMA-SAM. The MAE row uses the longest sequence that fits within its GPU budget before OOM, while SGMA uses a deliberately shorter sequence (e.g. N=2,048 on PAIP-32K vs. N=16,384 in Table 1 ) to match wall-clock compute. This enables a clear comparison of SGFT speedup. SGMA still achieves higher Dice and faster inference, so the reported speedups (up to 24.8 × ) are conservative lower bounds.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Masking ratio ablation. Each row fixes a ratio r∈{0.25,0.50,0.75} and shows the input, the quadtree partition, the visible patches after structure-guided masking, and the SGMA reconstruction. Higher masking ratios yield a harder pre-training task while the adaptive quadtree keeps structurally important regions visible.
Figure 5: Sequence length ablation. The SGFT tokenizer concentrates more tokens in detail-rich regions as the budget N grows from 1024 to 8192 , while keeping uniform background coarsely partitioned.
Figure 6: Model size ablation. Reconstruction quality improves from ViT-L to ViT-XL on a HydrogelTEM-1K patch under the same structure-guided masking and quadtree tokenization. The larger backbone recovers finer network topology, which motivates our use of larger backbones on higher-resolution datasets.
Figure 7: Segmentation comparison on a SpringXCT-8K simulated slice (per-sample Dice reported). UNet [ 46 ] loses small-pore topology; the Adaptive Patching baseline [ 65 ] at N=8,194 tokens improves but still drops thin channels; MAE-pretrained SAM 2 [ 44 ] at the smallest feasible 1282 patch on an 8K image captures coarse structure only; our SGMA -SAM 2 at a 2×2 effective patch size with N=8,194 SGFT tokens recovers the fine pore topology. The aggregate test-set Dice for each model is reported in Table 1 ; the numbers here are for this specific sample.
Figure 8: Segmentation on a real SpringXCT-8K sample with zero-shot inference. The pixel-level microstructure (e.g. void area) can be precisely extracted, which is hard for human experts.
Figure 9: Example of SpringXCT-8K downstream task: connected component analysis [ 67 ] , separation, and mapping the stones distribution in the 3D image.
Figure 10: Example of HydrogelTEM-1K downstream task: a hydrogel sample processed through zero-shot segmentation and skeletonization, followed by junction detection and degree-wise node annotation, illustrating the structural complexity of the hydrogel network.
Self-supervised pretraining has transformed computer vision by enabling data-efficient fine-tuning, yet high-resolution pretraining typically requires server-scale infrastructure, limiting custom in-domain training for many research laboratories. Masked Autoencoders (MAE) reduce computation by encoding only visible tokens, but combining MAE with hierarchical downsampling architectures has remained structurally challenging due to dense grid priors and mask-aware design compromises. We introduce AFFMAE, a masking-friendly hierarchical pretraining framework built on adaptive, off-grid token merging. AFFMAE removes dense-grid assumptions while preserving hierarchical scalability during pre-training and fine-tuning. To support this architecture, we developed numerically stable mixed-precision Triton kernels and a lightweight, point-based decoder that can be directly repurposed as a segmentation head. On high-resolution microscopy segmentation, AFFMAE matches MAE finetuning performance on foot process width estimation with ViT backbone at equal parameter counts while being 2x faster during pre-training and halving peak memory usage. Furthermore, AFFMAE achieves up to 5x throughput speedups fine-tuning at the 1024px resolution, providing high-resolution model training on desktop hardware. Code available at https://github.com/najafian-lab/affmae.
David Smerkous, Zian Wang, Behzad Najafian
Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA
Camera-trap imagery presents substantial variation in animal appearance, illumination, background clutter, and visual degradation. Masked autoencoders (MAEs) offer a label-free framework for modeling such data through reconstruction of randomly hidden image patches. We present WildMAE, a decoder-side extension of MAE that combines local structure-aware refinement with masked-to-visible cross-attention. A zero-gated convolutional residual path introduces a two-dimensional neighborhood prior, while gated cross-attention allows masked positions to retrieve contextual information from visible-position features without replacing the original Transformer decoder. The associated training recipe progressively introduces normalized-pixel, Sobel-edge, and perceptual objectives. In the original Snapshot Serengeti study, covering 4,744 test images at 75% masking, the reported reconstruction scores were 27.90 versus 24.33 dB PSNR, 0.840 versus 0.780 SSIM, and 0.160 versus 0.320 LPIPS for WildMAE and vanilla MAE, respectively. Additional historical records document favorable average differences under four synthetic degradation conditions and describe component variants. We present the method, experimental observations, and their scope together: the retained metrics are historical protocol-dependent results, rather than a newly reproduced, harmonized comparison or a validation of restoration from genuinely unavailable pixels. WildMAE provides a practical architectural case study for combining global token mixing with spatial and targeted contextual refinement in wildlife masked-image modeling.
Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.
Wenzhao Xiang, Yue Wu, Hongyang Yu +3
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China · Pengcheng Laboratory, Shenzhen 518108, China +1