StaX-Net: Stackelberg Interleaved Asymmetric Network for Cross-Modality Medical Image Translation

Amin Golnari; Saeid Sanei

cs.CVOct 25, 2026

StaX-Net: Stackelberg Interleaved Asymmetric Network for Cross-Modality Medical Image Translation

Authors: Amin Golnari, Saeid Sanei

Organizations: College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam · College of Health Science, VinUniversity, Hanoi, Vietnam · Institute of Psychiatry, Psychology & Neuroscience, King’s College London, UK

Abstract

Cross-modality medical image translation is challenged by anatomically inconsistent synthesis that can compromise clinical diagnosis. We propose StaX-Net, a Stackelberg interleaved asymmetric network that incorporates a partially coupled encoder-discriminator structure, and a 2D feature aggregation and spatial transformation (FAST) layer that distills target-supervised priors under the learning using privileged information paradigm. The shared blocks between the generator and discriminator enhance architectural feature alignment. To stabilize the leader-follower training, a Stackelberg-inspired asymmetric optimization scheme employs explicit shared-parameter freezing and asymmetric learning rates. The FAST layer distills target-domain priors during training and applies a source-only transform at inference. This constrains the synthesis process through target-informed feature representations relative to standard generative architectures. The code is available on GitHub for public use.

Figures & tables

Figure 1: Overall architecture of the proposed StaX-Net framework. The generator (left) translates input CT slices into reconstructed MRI representations, while the discriminator (right) evaluates real versus generated MRI slices. A defining characteristic of this architecture is the tight coupling at each spatial resolution stage via identical shared blocks (blue). Structurally, the generator employs dual routing pathways at each scale, incorporating multi-scale MRI priors into the FAST layers (purple) exclusively during training, and utilizing skip connections that directly link FAST outputs to the decoding pathway. Under the proposed Stackelberg-inspired asymmetric optimization, the shared weights are explicitly updated by the discriminator during the first training phase (leader). In the second phase (follower), the generator utilizes these exact, newly updated shared weights as fixed (frozen) feature extractors, optimizing only its dedicated modules.

Figure 2: Detailed architecture of the FAST layer. The internal loss (

\mathcal{L}_{\mathrm{FAST}}

in 11 ) computed by this module during the forward pass is injected into the total optimization objective.

Data	Method	PSNR (2D) $\uparrow$	SSIM (2D) $\uparrow$	MAE (2D) $\downarrow$	VGG-MSE $\downarrow$	FFD $\downarrow$	PSNR (3D) $\uparrow$	SSIM (3D) $\uparrow$	MAE (3D) $\downarrow$	RMSE (3D) $\downarrow$	Z-MAE $\downarrow$
AB	UNet-AE	17.71 (2.29)	0.674 (0.102)	0.1281 (0.0502)	1687.9 (673.6)	369,420	17.48 (1.87)	0.653 (0.106)	0.1299 (0.0462)	0.2737 (0.0597)	0.0460 (0.0133)
	Vanilla cGAN	15.87 (2.73)	0.635 (0.102)	0.1695 (0.0851)	4626.8 (5702.4)	1,033,639	15.41 (2.22)	0.613 (0.096)	0.1742 (0.0773)	0.3520 (0.1057)	0.0435 (0.0149)
	Pix2Pix	17.06 (2.17)	0.652 (0.098)	0.1371 (0.0499)	991.2 (318.5)	168,908	16.76 (1.65)	0.632 (0.098)	0.1394 (0.0435)	0.2957 (0.0571)	0.0522 (0.0139)
	CycleGAN	14.28 (1.93)	0.593 (0.107)	0.2014 (0.0647)	1074.5 (772.0)	109,309	14.10 (1.37)	0.575 (0.104)	0.2021 (0.0564)	0.3995 (0.0615)	0.0832 (0.0175)
	MedViT	17.14 (2.04)	0.661 (0.100)	0.1352 (0.0481)	1164.8 (473.4)	197,237	16.98 (1.78)	0.635 (0.106)	0.1366 (0.0441)	0.2892 (0.0591)	0.0643 (0.0201)
	StaX-Net	17.37 (2.13)	0.669 (0.099)	0.1299 (0.0497)	642.1 (413.7)	86,534	17.14 (1.74)	0.651 (0.102)	0.1322 (0.0452)	0.2837 (0.0579)	0.0517 (0.0131)

Table 1: Comprehensive quantitative comparison across the SynthRAD 2025 Abdomen/Pelvis (AB), Head & Neck (HN), and Thorax (TH) datasets. Values are presented as mean (std).

Panel of Figure 3 — Figure 3: Qualitative comparison of visual fidelity and localized structural preservation. The rows display axial slices of different anatomical regions: Head & Neck (top), Thorax (middle), and Abdomen/Pelvis (bottom). Columns from left to right show the input CT, the GT MRI, followed by the predicted MRI for StaX-Net, MedViT, CycleGAN, Pix2Pix, Vanilla cGAN, and UNet-AE. Zoomed-in insets (top-right) highlight local structures.

Figure 4: 1D radially averaged Fourier power spectrum analysis for (a) Head & Neck, (b) Thorax, and (c) Abdomen/Pelvis datasets. The x-axis represents spatial frequency, while the y-axis shows the logarithmic power magnitude.

Image-to-Image Translation

Image-to-image translation maps images between visual domains or modalities while preserving relevant content, structure, and semantics.

See all 113 papers

Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching
Aug 8, 2026Daniele Molino, Alessio Zoboli, Camillo Maria Caruso +2Image-to-Image Translation Multi-Task Learning
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · UniCamillus – Saint Camillus International University of Health Sciences, Rome, Italy · Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden
PredRA: Fast Medical Image Translation by Deterministic Component Extraction and Controlled Stochastic Refinement
Sep 25, 2026Jianhai Zhang, Pattarawut Charatpangoon, Donghao Zhang +4Image-to-Image Translation Medical Image Synthesis
Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. · Ron and Rene Ward Centre for Healthy Brain Aging Research, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada. · Department of Biomedical Engineering, School of Life Science and Technology, Huazhong University of Science and Technology, Wuhan, Hubei, China. +1
Posterior Samplings are Missing Modalities Generators for Medical Image Translation
Jul 21, 2026Jonghun KimMissing-Modality Learning Image-to-Image Translation
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

Aug 8, 2026J/K move · Enter open · S save

Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching

Daniele Molino, Alessio Zoboli, Camillo Maria Caruso, Valerio Guarrasi+1

Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · UniCamillus – Saint Camillus International University of Health Sciences, Rome, Italy · Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden

Image-to-Image Translation Multi-Task Learning Medical Image Synthesis

Open paper AskPDF arXiv

Cross-modality medical image translation can reduce the burden of multi-modal acquisitions, yet the field remains constrained by two coupled limitations: methods operate on 2D slices or 3D patches rather than whole volumes, and train a separate model for each translation task. Both stem from a single cause, the absence of a sufficiently strong volumetric prior, which forces generative models to learn anatomical appearance and cross-modality mapping simultaneously, an ill-posed problem at the scale of available paired datasets. We propose to decouple these objectives. A large-scale pretrained 3D variational autoencoder provides a compact latent representation of volumetric appearance, reducing translation to a conditional flow-matching problem. This compression makes whole-volume processing tractable, while a resolution-aware sampling strategy preserves native anatomical scale. We train a single model jointly across inter-modality (MRI

\to

CT, CBCT

\to

CT) and intra-modality (MRI

\to

MRI) tasks over three multi-center datasets. Across all tasks, whole-volume processing outperforms its patch-based counterpart, and the multi-task model matches task-specific baselines while replacing

N

networks with one. Crucially, joint training unlocks capabilities inaccessible to task-specific approaches: zero-shot generalization to anatomical regions unseen during training, within 0.15 SSIM of the fully supervised model, and compositional cross-dataset translation along paths never directly supervised. These results suggest that combining a strong volumetric prior with multitask training is a scalable route toward synthesis systems that generalize beyond their training distribution. Code is available at https://github.com/arco-group/Whole-Volume-Latent-FM.

Abstract

Image-to-Image Translation

Explore similar work

Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching

PredRA: Fast Medical Image Translation by Deterministic Component Extraction and Controlled Stochastic Refinement

Posterior Samplings are Missing Modalities Generators for Medical Image Translation