cs.CVMar 1, 2026

The Texture-Shape Dilemma: Boundary-Safe Synthetic Generation for 3D Medical Transformers

Authors: Jiaqi Tang, Weixuan Xu, Shu Zhang, Fandong Zhang, Qingchao Chen

Organizations: Peking University · Central South University · Deepwise Ltd.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Transformers (ViTs) have revolutionized medical image analysis, yet their data-hungry nature clashes with the scarcity and privacy constraints of clinical archives. Formula-Driven Supervised Learning (FDSL) has emerged as a promising solution to this bottleneck, synthesizing infinite annotated samples from mathematical formulas without utilizing real patient data. However, existing FDSL paradigms rely on simple geometric shapes with homogeneous intensities, creating a substantial gap by neglecting tissue textures and noise patterns inherent in modalities like CT and MRI. In this paper, we identify a critical optimization conflict termed boundary aliasing: when high-frequency synthetic textures are naively added, they corrupt the image gradient signals necessary for learning structural boundaries, causing the model to fail in delineating real anatomical margins. To bridge this gap, we propose a novel Physics-inspired Spatially-Decoupled Synthesis framework. Our approach orthogonalizes the synthesis process: it first constructs a gradient-shielded buffer zone based on boundary distance to ensure stable shape learning, and subsequently injects physics-driven spectral textures into the object core. This design effectively reconciles robust shape representation learning with invariance to acquisition noise. Extensive experiments on the BTCV and MSD datasets demonstrate that our method significantly outperforms previous FDSL, as well as SSL methods trained on real-world medical datasets, by 1.43% on BTCV and up to 1.51% on MSD task, offering a scalable, annotation-free foundation for medical ViTs. The code will be made publicly available upon acceptance.

Figures & tables

Figure 1: Overview of the proposed physics-inspired spatially-decoupled synthesis framework. From left to right: (A) Existing FDSL relies on homogeneous geometric primitives, resulting in a texture gap to real clinical data. (B) Naive texture injection leads to boundary aliasing, where stochastic gradients corrupt structural cues; a gradient-shielded buffer is constructed to preserve boundary signals. (C) Physics-driven textures are injected into a geometrically decoupled core via Dirichlet-weighted spectral mixing. (D) The composed volumes enable synthetic pre-training followed by downstream fine-tuning on real datasets.
Pre-trainingPT NumTypeAvg.SplRKidLKidGallEsoLivStoAorIVCVeinsPanradlad
UNETR
Scratch0–75.8693.0289.7990.6051.3471.1295.0272.9082.9576.7169.0576.2962.0255.33
PrimGeoSeg5KFDSL78.9093.1993.4793.3554.4968.9195.8878.8387.8282.4772.7380.0865.0659.35
Ours5KFDSL79.6193.4793.1092.8660.5069.0895.8280.0990.2483.4271.4979.3963.9361.49
SwinUNETR
Scratch0–80.1295.3294.0193.5161.1773.8394.4071.1988.1385.2073.1280.6067.1563.88
Table 1: Comparison of segmentation performance on the BTCV dataset.
MethodMSD Dataset
Task 02Task 06Task 09
UNETRSwinUNETRUNETRSwinUNETRUNETRSwinUNETR
Scratch95.1495.3470.6876.1492.5295.92
PrimGeoSeg95.6195.7778.8180.3996.3096.96
Ours95.9196.2979.2081.4796.6398.47
Table 2: Comparison of segmentation performance on several MSD datasets.
MethodPre-train DataData AmountAvg. Dice
SwinMM [ 18 ]Real (CT)5,00076.72
SwinUNETR [ 16 ]Real (CT)5,00080.56
OursSynthetic5,00081.51
Table 3: Comparison with SSL methods pre-trained on RadGenome Dataset [ 20 ]
Dataset ScaleTexture TypeGap Width
ScaleAvg. DiceTypeAvg. DiceWidthAvg. Dice
50080.16fruit80.39w=0w=080.02
500081.51trabecular80.56w=1w=180.22
1500082.26fibrous81.10w=5w=580.55
5000082.40granular80.41w=9w=980.59
––ours81.51w=13w=1380.15
Table 4: Analysis on the BTCV dataset with the SwinUNETR under different settings, including pre-training dataset scales, texture types, and gap widths.
More in this topic

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 453 papers

Explore similar work

CardsList
Mar 1, 2026

Fake It Right: Injecting Anatomical Logic into Synthetic Supervised Pre-training for Medical Segmentation

Vision Transformers (ViTs) excel in 3D medical segmentation but require massive annotated datasets. While Self-Supervised Learning (SSL) mitigates this using unlabeled data, it still faces strict privacy and logistical barriers. Formula-Driven Supervised Learning (FDSL) offers a privacy-preserving alternative by pre-training on synthetic mathematical primitives. However, a critical semantic gap limits its efficacy: generic shapes lack the morphological fidelity, fixed spatial layouts, and inter-organ relationships of real anatomy, preventing models from learning essential global structural priors. To bridge this gap, we propose an Anatomy-Informed Synthetic Supervised Pre-training framework unifying FDSL's infinite scalability with anatomical realism. We replace basic primitives with a lightweight shape bank with de-identified, label-only segmentation masks from 5 subjects. Furthermore, we introduce a structure-aware sequential placement strategy to govern the patch synthesis process. Instead of random placement, we enforce physiological plausibility using spatial anchors for correct localization and a topological graph to manage inter-organ interactions (e.g., preventing impossible overlaps). Extensive experiments on BTCV and MSD datasets demonstrate that our method significantly outperforms state-of-the-art FDSL baselines and SSL methods by 1.74% and up to 1.66%, while exhibiting a robust scaling effect where performance improves with increased synthetic data volume. This provides a data-efficient, privacy-compliant solution for medical segmentation. The code will be made publicly available upon acceptance.

Jiaqi Tang, Mengyan Zheng, Shu Zhang +2

Peking University · Beihang University · Deepwise Ltd.

Medical Image SegmentationSynthetic Data Pretraining3D Medical Image Segmentation3D ViTs
AskPDF
Mar 26, 2026

VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely transformer-based 3D Diffusion Transformer for volumetric medical image synthesis. Our approach extends diffusion transformers to native 3D data through volumetric patch embeddings and global self-attention operating directly over 3D tokens. To enable structured control, we propose a timestep-gated control adapter that maps segmentation masks into learnable control tokens that modulate transformer layers during denoising. This token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformerbased diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.

Marvin Seyfarth, Salman Ul Hassan Dar, Yannik Frisch +4

Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University, Heidelberg, Germany · Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital, Heidelberg, Germany · DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany +1

Medical Image Synthesis3D Diffusion ModelsDiffusion TransformerConditional Image Generation
AskPDF
Mar 11, 2026

Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement

Medical image synthesis is crucial for alleviating data scarcity and privacy constraints. However, fine-tuning general text-to-image (T2I) models remains challenging, mainly due to the significant modality gap between complex visual details and abstract clinical text. In addition, semantic entanglement persists, where coarse-grained text embeddings blur the boundary between anatomical structures and imaging styles, thus weakening controllability during generation. To address this, we propose a Visually-Guided Text Disentanglement framework. We introduce a cross-modal latent alignment mechanism that leverages visual priors to explicitly disentangle unstructured text into independent semantic representations. Subsequently, a Hybrid Feature Fusion Module (HFFM) injects these features into a Diffusion Transformer (DiT) through separated channels, enabling fine-grained structural control. Experimental results in three datasets demonstrate that our method outperforms existing approaches in terms of generation quality and significantly improves performance on downstream classification tasks. The source code is available at https://github.com/hx111/VG-MedGen.

Xin Huang, Junjie Liang, Qingshan Hou +4

Computer Science and Engineering, Northeastern University, Shenyang, China · Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China · National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China +2

Medical Image SynthesisControllable Image GenerationT2I Diffusion ModelsMultimodal Disentangled Representation LearningDiffusion Model Fine-Tuning
AskPDF