Registration is a Representation Problem: Solving Whole-Body PET/CT Registration

Neel Dey; Adrian V. Dalca

cs.CVOct 13, 2026

Registration is a Representation Problem: Solving Whole-Body PET/CT Registration

Authors: Neel Dey, Adrian V. Dalca

Organizations: A. A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital · Department of Radiology, Harvard Medical School · Computer Science and AI Lab, Massachusetts Institute of Technology

Abstract

Raw image intensities are a poor surrogate for similarity across images in correspondence problems. Further, while auxiliary segmentation labels can provide semantic identification and improve matching, they can assign only coarse identities to fine-grained anatomy and often cover only a subset of structures, particularly in whole-body settings. We present an approach that instead establishes correspondence using complementary representations of anatomy and demonstrate its utility for longitudinal whole-body PET/CT registration in the Learn2Reg 2026 PSMAReg Challenge. Our 3D foundation model, anatomix, is used to derive fine-grained, per-voxel semantic representations that are consistent across volumes, and we further employ off-the-shelf segmentation models to obtain coarse anatomical identities. Together, these representations drive a GPU-accelerated iterative registration solver (FireANTs) in a hybrid multistage framework: the first stage estimates alignment from dense multi-channel features, after which a second refines the deformation using both anatomix- and segmentation-based losses. Lastly, because registration should not introduce non-biological tumor volume changes across time, we develop lesion-specific component-wise volume preservation schemes. Without any task-specific training, our system won first place in the PSMAReg Challenge leaderboard. Code and interactive visualizations are available at http://www.neeldey.com/reps-for-reg.

Figures & tables

Figure 1: Given a pair of PET/CT volumes to register ( a ), we extract both fine-grained representations with the learned anatomix foundation model [ 5 ] and the hand-crafted MIND-SSC descriptor [ 8 ] and coarse-grained representations as in PSMA lesion and anatomical segmentation maps, obtained using LION [ 19 ] and TotalSegmentator [ 25 ] , respectively ( b ). Using these complementary representations, we design a hybrid multi-stage registration framework that uses a fast registration solver (FireANTs) to align the representations ( c ). Most residual misalignment after stage-1 is in small abdominal organs and are rectified after stage-2 ( d ).

Figure 2: Constructing a recall-biased lesion mask . False negative PET lesion predictions would reduce PET biomarker preservation in our work and we therefore bias towards recall. Given the PET scan ( a ), we threshold it to find areas with high uptake ( b ) and subtract organs with non-pathological uptake ( c and d ) from this mask to arrive at a morphologically constructed mask ( e ). Finally, we add the segmentation prediction of a pretrained PSMA lesion segmentor ( f ) to the morphologically constructed mask to obtain the final recall-biased lesion mask ( g ).

Figure 3: Parallelizing feature extraction and segmentation network passes . (a) Serially running all pretrained networks used in our framework at their default hyperparameters would exceed the 90 s limit. (b) We use an optimized schedule and vary the inference hyperparameters of each network to run networks in parallel and deliver segmentations before they are needed by Stage 2. (c) FP16 feature extraction and accumulation, plus the removal of a redundant segmentor, bring peak vRAM under the 80GB maximum defined by the challenge.

Figure 4: Qualitative results . On randomly selected input pairs ( a ), our framework increases visual, label, and cross-modal agreement ( b , c , and d , respectively) across large deformations (rows 1 and 2), only failing in regions without topological correspondence (e.g., the arms in row 3).

Team	CT Dice	CT HD95	PET Dice	PET HD95	MTV Error (%)	TLG Error (%)	NDV (%)	Rank
Ours	0.830	4.6	0.610	19.4	1.78	1.39	0.0008	1
heyufan1995	0.832	4.9	0.628	18.9	5.37	5.35	0.0032	2
tinymilky	0.832	4.6	0.598	20.0	3.41	3.24	0.0009	3

Table 1: Performance of the top three methods on the test set of the challenge by rank, with rankings and evaluation provided by the challenge organizers. Standard deviations were not provided.

Configuration	CT DSC $\uparrow$	CT HD95 $\downarrow$	%NDV $\downarrow$	$\|$ MTV% $\|$ $\downarrow$	$\|$ TLG% $\|$ $\downarrow$	CT MAE $\downarrow$
No registration	0.114(0.011)	41.72(2.98)	–	–	–	302.1(12.0)
Stage 1, CT+PET, default FireANTs	0.586(0.027)	20.07(3.43)	0.00025(0.00015)	15.49(2.15)	16.67(2.31)	118.1(14.5)
Stage 1, CT+PET, tuned FireANTs	0.686(0.012)	11.09(0.94)	0.00113(0.00086)	15.56(2.08)	18.47(2.39)	57.3 (3.3)
+ anatomix + MIND-SSC	0.775(0.007)	6.54(0.37)	0.00125(0.00085)	6.31(1.68)	6.93(1.76)	58.6(3.8)
+ volume penalty	0.773(0.007)	6.57(0.36)	0.00145(0.00090)	4.95(1.28)	5.36(1.48)	60.0(3.8)
+ volume projection	0.772(0.007)	6.57(0.36)	0.00147(0.00092)	3.41(1.18)	4.14(1.49)	60.5(3.8)

Table 2: Ablation study on 65 training set pairs (50 held-out subjects) with mean and std. error.

Explore similar work

CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration
Apr 24, 2026Xiangcen Wu, Ruohua Chen, Sichun Li +4Medical Image Registration PET/CT Imaging
Institute of Medical Technology, Peking University Health Science Center, Beijing, China · Ren Ji Hospital, Shanghai Jiao Tong University, Shanghai, China · University of Oxford, Oxford, United Kingdom
SAMReg: SAM-enabled Image Registration with ROI-based Correspondence
Oct 17, 2024Shiqi Huang, Tingfa Xu, Ziyi Shen +4Medical Image Registration Deformable Image Registration
University College London, London WC1E 6BT, UK · Beijing Institute of Technology, Beijing 100081, China
Polaffini: A feature-based approach for robust affine and polyaffine image registration
Feb 19, 2026Antoine Legouhy, Cosimo Campo, Ross Callaghan +2Medical Image Registration Deformable Image Registration
Hawkes Institute & Department of Computer Science, University College London, London, UK · Institut Pasteur, Universit´e Paris Cit´e, Unit´e de Neuroanatomie Appliqu´ee et Th´eorique, France · AINOSTICS ltd., Manchester, UK

Apr 24, 2026J/K move · Enter open · S save

CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration

Xiangcen Wu, Ruohua Chen, Sichun Li, Qianye Yang+3

Institute of Medical Technology, Peking University Health Science Center, Beijing, China · Ren Ji Hospital, Shanghai Jiao Tong University, Shanghai, China · University of Oxford, Oxford, United Kingdom

Medical Image Registration PET/CT Imaging Deformable Image Registration

Open paper AskPDF arXiv

Whole-body Positron Emission Tomography (PET) registration is essential for multi-parametric tumor characterization and assessment of metastatic disease progression. In deep learning-based deformable registration, the dense displacement field (DDF) regularizer is crucial for stabilizing optimization and preventing unrealistic deformations in large 3D volumes. A key challenge in whole-body deformable registration is anatomical heterogeneity, rigid structures (e.g., bones) should undergo stronger regularization, whereas soft tissues require more flexible deformation and weaker constraints. In this work, we propose a simple yet effective CT-guided spatially-varying regularization strategy for whole-body cross-tracer deformable PET registration. The key idea is to use the paired CT volume from the PET/CT acquisition to construct a voxel-wise regularization map for the DDF, replacing the conventional single global regularization weight. This yields anatomy-adaptive regularization strength across rigid and soft tissues. The proposed method is evaluated on a real clinical cross-tracer PET/CT dataset of 296 patients involving 18F-PSMA and 18F-FDG, showing that the proposed method achieves statistically significant improvements over weakly-supervised registration baseline in both whole-body registration performance and organ-wise alignment.

Abstract

Explore similar work

CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration

SAMReg: SAM-enabled Image Registration with ROI-based Correspondence

Polaffini: A feature-based approach for robust affine and polyaffine image registration