TransNSNet: A Transformer-Enhanced Multi-Scale Adversarial Framework for CT-to-PET Image Synthesis

Shaojun Li; Xiaoyu Deng; Ikuo Keshi

eess.IVOct 16, 2026

TransNSNet: A Transformer-Enhanced Multi-Scale Adversarial Framework for CT-to-PET Image Synthesis

Authors: Shaojun Li, Xiaoyu Deng, Ikuo Keshi

Organizations: Fukui University of Technology, Fukui Campus, 3-6-1 Gakuen, Fukui City, Fukui Prefecture 910-8505, JAPAN · University of Fukui, Bunkyo Campus, 3-9-1 Bunkyo, Fukui City, Fukui Prefecture 910-8507, JAPAN

Abstract

This study investigates CT-to-PET image synthesis using TransNSNet, a Transformer-enhanced multi-scale adversarial framework. A paired CT-PET dataset was curated from the public non-small cell lung cancer (NSCLC) Radiogenomics collection through patient screening, slice-level matching, preprocessing, and patient-level train-test separation. The proposed framework generates PET-like predictions at multiple spatial scales and applies coarse-to-fine reconstruction supervision, while TransPatchGAN introduces contextual feature interaction into adversarial evaluation. A progressive ablation study compares Pix2Pix, NSNet-Base, NSNet-D, and TransNSNet. Image-level performance was summarized over a fixed convergence window (epochs 80-101) from one training run per model. TransNSNet achieved the best mean value in nine of ten metrics, whereas NSNet-Base achieved the highest visual saliency-induced index. For fixed-fraction high-intensity regions, NSNet-D was marginally higher in slice-level recall and balanced accuracy, but the equal-weight patient-level values were nearly identical and the highest Dice values shifted to TransNSNet. These differences were small relative to between-patient variability. In the separate adaptive patient-level analysis (n=10), no pairwise model difference remained statistically significant after Holm correction. TransNSNet also had a substantially smaller parameter footprint than the original next-scale models. These findings indicate promising image-quality trends, but they do not establish statistical superiority or clinical validity; synthesized PET-like images should not be interpreted as substitutes for clinically acquired PET.

Figures & tables

Figure 1: Architecture of the proposed TransNSNet framework. The generator selects the middle CT channel, produces one full-resolution and five auxiliary PET predictions, and applies reconstruction supervision at six scales. TransPatchGAN evaluates real [CT, PET truth] and fake [CT, predicted PET] pairs in separate forward passes with shared weights.

Figure 2: Study workflow for paired CT-PET dataset construction and experimental splitting.

	AMC		R01		Total
Split	Patients	CT-PET pairs	Patients	CT-PET pairs	Patients	Images
Train	32	8,590	105	30,369	137	77,918
Test	3	765	7	2,140	10	5,810
Total	35	9,355	112	32,509	147	83,728

Table 1: Constructed paired CT-PET dataset after screening and patient-level splitting.

Figure 3: Progressive ablation design from the Pix2Pix baseline to the final TransNSNet framework.

Model	Generator	Discriminator	Multi-scale supervision	Transformer in generator	Transformer in discriminator	Ablation purpose
Pix2Pix	U-Net generator	PatchGAN discriminator	–	–	–	Baseline conditional GAN for paired CT-to-PET translation
NSNet-Base	Next-scale generator	PatchGAN-style discriminator	$\checkmark$	–	–	Compare next-scale generation and multi-scale PET supervision with the baseline
NSNet-D	Next-scale generator	TransPatchGAN	$\checkmark$	–	$\checkmark$	Compare TransPatchGAN with the PatchGAN-style discriminator
TransNSNet	Transformer-enhanced next-scale generator	TransPatchGAN	$\checkmark$	$\checkmark$	$\checkmark$	Final proposed model; evaluate the complete Transformer-enhanced next-scale adversarial framework

Table 2: Model definitions for the progressive ablation study.

Metric	Pix2Pix	NSNet-Base	NSNet-D	TransNSNet
SSIM $\uparrow$	0.8888 $\pm$ 0.0011	0.8883 $\pm$ 0.0010	0.8894 $\pm$ 0.0006	0.8918 $\pm$ 0.0015
MS-SSIM $\uparrow$	0.8584 $\pm$ 0.0014	0.8588 $\pm$ 0.0002	0.8593 $\pm$ 0.0004	0.8649 $\pm$ 0.0011
PSNR $\uparrow$	26.6841 $\pm$ 0.0339	26.8227 $\pm$ 0.0184	26.8493 $\pm$ 0.0156	27.1390 $\pm$ 0.0994
MSE $\downarrow$	0.0034 $\pm$ 0.0000	0.0034 $\pm$ 0.0000	0.0034 $\pm$ 0.0000	0.0032 $\pm$ 0.0001
MAE $\downarrow$	0.0165 $\pm$ 0.0001	0.0165 $\pm$ 0.0000	0.0165 $\pm$ 0.0000	0.0162 $\pm$ 0.0001
GMSD $\downarrow$	0.1140 $\pm$ 0.0012	0.1121 $\pm$ 0.0002	0.1114 $\pm$ 0.0002	0.1097 $\pm$ 0.0004

Table 3: Main image-level comparison over the fixed convergence window (epochs 80–101; 22 evaluations per model). Values are mean

\pm

epoch-to-epoch standard deviation from one training run. The standard deviation is not a between-patient or between-seed estimate and is not a confidence interval. The best mean value in each row is shown in bold.

Metric	Pix2Pix	NSNet-Base	NSNet-D	TransNSNet
HU-Dice $\uparrow$	0.4063 $\pm$ 0.2187	0.4059 $\pm$ 0.2141	0.4230 $\pm$ 0.2156	0.4223 $\pm$ 0.2196
HU-IoU $\uparrow$	0.2796 $\pm$ 0.1818	0.2784 $\pm$ 0.1790	0.2931 $\pm$ 0.1844	0.2932 $\pm$ 0.1860
HU-Precision $\uparrow$	0.4011 $\pm$ 0.2225	0.4032 $\pm$ 0.2183	0.4145 $\pm$ 0.2193	0.4165 $\pm$ 0.2244
HU-Recall $\uparrow$	0.4176 $\pm$ 0.2187	0.4150 $\pm$ 0.2134	0.4402 $\pm$ 0.2171	0.4359 $\pm$ 0.2198
HU-Specificity $\uparrow$	0.9687 $\pm$ 0.0115	0.9695 $\pm$ 0.0110	0.9689 $\pm$ 0.0116	0.9693 $\pm$ 0.0118
HU-Balanced Acc. $\uparrow$	0.6932 $\pm$ 0.1147	0.6922 $\pm$ 0.1118	0.7046 $\pm$ 0.1140	0.7026 $\pm$ 0.1154

Table 4: Slice-level fixed-fraction high-intensity-region metrics from the saved model-specific checkpoints. Masks are defined by the top-5% intensity percentile unless otherwise specified. Values are mean

\pm

slice-level standard deviation over 2,905 slices. The best mean value in each row is shown in bold.

Figure 4: Qualitative comparison of adaptive LU contour consistency in selected test cases. Ground-truth and predicted LU masks were independently defined using 50% of the corresponding slice maximum. Blue, yellow, and pink denote the ground-truth LU contour, predicted LU contour, and their overlap, respectively; the lower row of each case enlarges the boxed region.

Figure 5: Qualitative comparison of absolute-error maps in adaptive LU regions for selected test cases. Ground-truth and predicted LU masks were independently defined using 50% of the corresponding slice maximum, and errors were calculated as $\lvert I_{\mathrm{pred}}-I_{\mathrm{GT}}\rvert$ . Brighter colors indicate larger errors; the lower row of each case enlarges the boxed region, and the rightmost PET column provides the ground-truth reference.

Metric	Pix2Pix	NSNet-Base	NSNet-D	TransNSNet
LU-Dice $\uparrow$	0.4232 $\pm$ 0.1267	0.4209 $\pm$ 0.1270	0.4277 $\pm$ 0.1245	0.4307 $\pm$ 0.1282
LU-IoU $\uparrow$	0.3053 $\pm$ 0.1023	0.3034 $\pm$ 0.1029	0.3108 $\pm$ 0.1016	0.3154 $\pm$ 0.1047
LU-Precision $\uparrow$	0.4059 $\pm$ 0.1186	0.4101 $\pm$ 0.1089	0.4000 $\pm$ 0.1030	0.4155 $\pm$ 0.1015
LU-Recall $\uparrow$	0.6215 $\pm$ 0.1747	0.6282 $\pm$ 0.1824	0.6953 $\pm$ 0.1790	0.6435 $\pm$ 0.2017
LU-MAE $\downarrow$	0.2573 $\pm$ 0.0750	0.2601 $\pm$ 0.0747	0.2500 $\pm$ 0.0779	0.2458 $\pm$ 0.0824

Table 5: Five patient-level adaptive lesion-like uptake metrics. Ground-truth and predicted regions were independently defined at or above 50% of their respective normalized slice maxima. All 4-connected components containing at least 20 pixels were retained without an empty-mask fallback, and slices were included only when the filtered ground-truth mask contained at least 30 pixels. Values are mean

\pm

between-patient standard deviation across ten equally weighted patients after within-patient slice averaging (2,744 evaluable slices; 161/2,905 excluded). The best mean value in each row is shown in bold.

Model	Generator parameters	Discriminator parameters	Total parameters
Pix2Pix	30.94M	2.77M	33.70M
NSNet-Base	83.06M	2.77M	85.83M
NSNet-D	83.06M	3.82M	86.88M
TransNSNet	19.84M	3.82M	23.66M

Table 6: Model parameter comparison for the four compared architectures. Values are rounded independently to two decimal places in millions from the raw parameter counts; totals were calculated before rounding.

Figure 6: Qualitative visual comparison of CT input, Pix2Pix, NSNet-Base, NSNet-D, TransNSNet, and ground-truth PET images using the saved model-specific checkpoints.

Figure 7: Absolute error maps for Pix2Pix, NSNet-Base, NSNet-D, and TransNSNet on selected cases using the saved model-specific checkpoints.

Metric	Pix2Pix	NSNet-Base	NSNet-D	TransNSNet
HU-Dice $\uparrow$	0.4114 $\pm$ 0.0983	0.4103 $\pm$ 0.1009	0.4261 $\pm$ 0.0991	0.4281 $\pm$ 0.1000
HU-IoU $\uparrow$	0.2837 $\pm$ 0.0775	0.2817 $\pm$ 0.0798	0.2958 $\pm$ 0.0810	0.2978 $\pm$ 0.0806
HU-Precision $\uparrow$	0.4075 $\pm$ 0.0959	0.4091 $\pm$ 0.0985	0.4195 $\pm$ 0.0951	0.4239 $\pm$ 0.0980
HU-Recall $\uparrow$	0.4206 $\pm$ 0.0982	0.4169 $\pm$ 0.1001	0.4397 $\pm$ 0.1002	0.4389 $\pm$ 0.0991
HU-Specificity $\uparrow$	0.9689 $\pm$ 0.0050	0.9696 $\pm$ 0.0051	0.9691 $\pm$ 0.0051	0.9696 $\pm$ 0.0053
HU-Balanced Acc. $\uparrow$	0.6948 $\pm$ 0.0515	0.6932 $\pm$ 0.0526	0.7044 $\pm$ 0.0526	0.7042 $\pm$ 0.0522

Table S1: Patient-level evaluation of fixed-fraction high-intensity regions defined by the top-5% intensity percentile. Values are mean

\pm

between-patient standard deviation across ten patients. Each metric was first averaged over all slices within each patient, after which patient-level means were averaged with equal weights. All 2,905 slices were included.

Medical Image Synthesis

Medical image synthesis creates, translates, reconstructs, and edits clinical images to fill missing data and support analysis.

See all 310 papers

Domain-Adapted Latent Diffusion Models for CT-to-PET Image Translation
Oct 26, 2026Yann Bourdé, Lea Bogensperger, Arnela Hadzic +1Medical Image Synthesis Image-to-Image Translation
Department of Quantitative Biomedicine, University of Zurich · Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz
Cross Modality Image Translation In Medical Imaging Using Generative Frameworks
May 13, 2026Giulia Romoli, Filippo Ruffini, Francesco Di Feola +20Image-to-Image Translation Medical Image Synthesis
Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Universitetstorget, 4, Umeå, 90187, Sweden · Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo, 21, Roma, 00128, Italy · Università Cattolica del Sacro Cuore, Largo Francesco Vito, 1, Roma, 00168, Italy +12
Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
May 6, 2026Aneesh Rangnekar, Joao Miranda, Natally Horvat +13Cross-Modal Learning Cross-Modality Medical Image Segmentation
Department of Medical Physics, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA · Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA · Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA +2

Oct 26, 2026J/K move · Enter open · S save

Domain-Adapted Latent Diffusion Models for CT-to-PET Image Translation

Yann Bourdé, Lea Bogensperger, Arnela Hadzic, Michael Krauthammer

Department of Quantitative Biomedicine, University of Zurich · Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz

Medical Image Synthesis Image-to-Image Translation PET/CT Imaging Diffusion Model Adaptation Latent Diffusion Models

Open paper AskPDF arXiv

Positron Emission Tomography (PET) provides important metabolic information for tumor detection and staging but remains significantly less accessible than Computed Tomography (CT) due to its high cost, infrastructure requirements, and reliance on injected radioactive tracers. CT-to-PET image translation using generative models aims to address this disparity, yet state-of-the-art diffusion-based methods are limited by the scarcity of paired training data. To overcome this, we propose a domain-adapted latent diffusion framework that transfers natural-image priors using a relatively small dataset of 21,930 paired CT/PET slices, incorporating a signed power transform to resolve extreme intensity sparsity in PET data. Compared to CPDM, a state-ofthe-art baseline for CT-to-PET image translation, our approach yields superior performance in pixel-wise reconstruction and localized tumor fidelity, demonstrating effective transfer of natural-image priors to the medical domain. We further provide a systematic analysis of evaluation metric reliability for sparse PET data, showing that standard full-image metrics can be misleading indicators of translated image quality, and propose background-filtered and mask-restricted metrics as more clinically meaningful alternatives.

Abstract

Medical Image Synthesis

Explore similar work

Domain-Adapted Latent Diffusion Models for CT-to-PET Image Translation

Cross Modality Image Translation In Medical Imaging Using Generative Frameworks

Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images