cs.CVAug 18, 2026

GSToken: Geometry-Structured Gaussian Tokens for Compact 3D Medical Image Representation

Authors: Xiaoduo Li, Quan Gu

Organizations: Taiyuan University of Technology · independent researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Effective segmentation of multi-modal MRI is central to improving neural network accuracy in brain tumor recognition. Existing methods typically compress 3D volumes into token sequences via fixed patch encoding or learned attention pooling (e.g., TokenLearner). However, these compression schemes discard explicit spatial shape information; the resulting tokens convey no notion of lesion morphology or spatial extent. Meanwhile, end-to-end evaluation entangles a tokenizer's information retention with the reconstruction capacity of the downstream decoder, and the lack of a unified capacity contract across methods makes performance differences difficult to attribute. In this paper, we introduce Gaussian tokens to multi-modal brain tumor segmentation for the first time: each token carries not only a semantic feature but also a learned 3D center, anisotropic scale, and orientation, endowing the representation with explicit geometric support at negligible parameter cost. We further propose a frozen-token utility evaluation protocol: the trained tokenizer is frozen, its output is cast into a fixed-capacity serialized contract, and a shared lightweight Transformer probe independently measures each tokenizer's retained information under strictly matched conditions. Multi-seed paired statistical testing shows that GSToken consistently and substantially outperforms capacity-matched adaptive baselines under frozen probing, with uniform advantages across all tumor sub-regions, surface, and distance metrics. These results demonstrate that explicitly encoding spatial geometry within tokens significantly improves the information density of volumetric representations, offering a new design principle for compact 3D medical image representation and downstream reading.

Figures & tables

Figure 1: Overall framework of the proposed GSToken method.
Figure 2: Framework of the GSToken block.
Figure 3: Cross-model comparison of the main experimental results.
Figure 4: Token-to-image mapping visualization.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

FieldValues per tokenData typeReader input
Content feature96float32Yes
Center3float32Yes
Anisotropic scale3float32Yes
Rotation matrix9float32Yes
Importance1float32Yes
Semantic logits4float32No
Appendix
Table 1: Fields in the serialized token packet. “Reader input” indicates whether a field is provided to the frozen feature-only Transformer reader.

Explore similar work

CardsList
Jun 17, 2026

BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation

Three-dimensional (3D) brain MRI is central to clinical neurology and neuro-oncology, where generative models could augment under-represented cohorts, simulate disease trajectories, and support privacy-preserving data sharing. Latent diffusion has been the go-to solution for modeling imaging data, but it places two competing demands on the tokenizer: encoder embeddings must retain the clinical information that downstream tasks act on, and the decoder must reconstruct anatomically faithful volumes. Existing reconstruction-driven tokenizers achieve the second at the expense of the first. To address this, we introduce a fully volumetric masked-autoencoder (MAE) based tokenizer for 3D brain MRI latent diffusion, decoupling encoder and decoder: a frozen 3D MAE encoder produces clinically informative embeddings, while a dedicated CNN decoder reconstructs voxels from a linear projection of those embeddings. We pretrain the encoder on 35,309 volumes from 18 public cohorts spanning four modalities, ten disease categories, and 200+ acquisition sites, and demonstrate its dual utility in two settings. First, on a 23-task linear-probing benchmark, the encoder outperforms or matches SOTA models (i.e., BrainIAC, BrainSegFounder, and MedicalNet) on 21 of 23 tasks. Second, a conditional diffusion transformer (DiT) trained on these clinically informative embeddings supports both conditional generation across six variables and patient-specific longitudinal forecasting. Together these results establish a single 3D brain-MRI embedding space capable of both downstream clinical tasks and controllable generation.

Max Van Puyvelde, Ibrahim Gulluk, Wim Van Criekinge +1

Department of Biomedical Data Science, Stanford University School of Medicine · Department of Mathematical Modelling, Statistics & Bioinformatics, Ghent University · Department of Electrical Engineering, Stanford University

Masked AutoencodersLatent Diffusion ModelsMedical Image SynthesisDiffusion MRIMedical Image Representation Learning
AskPDF
Apr 16, 2026

TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens

In this work, we revisit several key design choices of modern Transformer-based approaches for feed-forward 3D Gaussian Splatting (3DGS) prediction. We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal, and instead propose to directly regress 3D mean coordinates using only a self-supervised rendering loss. This formulation allows us to move from the standard encoder-only design to an encoder-decoder architecture with learnable Gaussian tokens, thereby unbinding the number of predicted primitives from input image resolution and number of views. Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies, while naturally supporting efficient test-time optimization in token space without degrading learned priors. TokenGS achieves state-of-the-art feed-forward reconstruction performance on both static and dynamic scenes, producing more regularized geometry and more balanced 3DGS distribution, while seamlessly recovering emergent scene attributes such as static-dynamic decomposition and scene flow.

Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang +1

NVIDIA

3D Gaussian Splatting3D ViTsDynamic Scene ReconstructionFeed-Forward 3D Reconstruction
AskPDF
Feb 23, 2026

Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation

We present Token-UNet, adopting the TokenLearner and TokenFuser modules to encase Transformers into UNets. While Transformers have enabled global interactions among input elements in medical imaging, current computational challenges hinder their deployment on common hardware. Models like (Swin)UNETR adapt the UNet architecture by incorporating (Swin)Transformer encoders, which process tokens that each represent small subvolumes (838^3 voxels) of the input. The Transformer attention mechanism scales quadratically with the number of tokens, which is tied to the cubic scaling of 3D input resolution. This work reconsiders the role of convolution and attention, introducing Token-UNets, a family of 3D segmentation models that can operate in constrained computational environments and time frames. To mitigate computational demands, our approach maintains the convolutional encoder of UNet-like models, and applies TokenLearner to 3D feature maps. This module pools a preset number of tokens from local and global structures. Our results show this tokenization effectively encodes task-relevant information, yielding naturally interpretable attention maps. The memory footprint, computation times at inference, and parameter counts of our heaviest model are reduced to 33%, 10%, and 35% of the SwinUNETR values, with better average performance (86.75% ±0.19%\pm 0.19\% Dice score for SwinUNETR vs our 87.21% ±0.35%\pm 0.35\%). This work opens the way to more efficient trainings in contexts with limited computational resources, such as 3D medical imaging. Easing model optimization, fine-tuning, and transfer-learning in limited hardware settings can accelerate and diversify the development of approaches, for the benefit of the research community.

Louis Fabrice Tshimanga, Andrea Zanola, Federico Del Pup +1

3D Medical Image SegmentationHybrid CNN-Transformer ArchitecturesEfficient ViTsNeuroimaging
AskPDF