A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy
Authors: Caiwen Jiang, Yuzhen Ding, Mi Jia, Samir H. Patel, Terence T. Sio, Jonathan B. Ashman, Lisa A. McGee, Jean-Claude M. Rwigema, William G. Rule, Sameer R. Keole, Sujay A. Vora, William W. Wong, Nathan Y. Yu, Michele Y. Halyard, Steven E. Schild, Dinggang Shen, Wei Liu
Organizations: Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA · School of Biomedical Engineering, ShanghaiTech University, Shanghai, China · Shanghai United Imaging Intelligence Co., Ltd., Shanghai 200230, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China · Shanghai Clinical Research and Trial Center, Shanghai 201210, China
Proton therapy offers superior organ-at-risk sparing but is highly sensitive to anatomical changes, making accurate deformable image registration (DIR) across longitudinal CT scans essential. Conventional DIR methods are often too slow for emerging online adaptive workflows, while existing deep learning-based approaches are primarily designed for generic benchmarks and underutilize clinically relevant information beyond images. To address this gap, we propose a clinically scalable coarse-to-fine deformable registration framework that integrates multimodal information from the proton radiotherapy workflow to accommodate diverse clinical scenarios. The model employs dual CNN-based encoders for hierarchical feature extraction and a transformer-based decoder to progressively refine deformation fields. Beyond CT intensities, clinically critical priors, including target and organ-at-risk contours, dose distributions, and treatment planning text, are incorporated through anatomy- and risk-guided attention, text-conditioned feature modulation, and foreground-aware optimization, enabling anatomically focused and clinically informed deformation estimation. We evaluate the proposed framework on a large-scale proton therapy DIR dataset comprising 1,222 paired planning and repeat CT scans across multiple anatomical regions and disease types. Extensive experiments demonstrate consistent improvements over state-of-the-art methods, enabling fast and robust clinically meaningful registration.
Figures & tables
Figure 1: Multimodal information available in the proton radiotherapy workflow for each CT scan, including body mask, OAR and CTV contours, dose distribution, and treatment planning texts (diagnosis and treatment planning notes).
Figure 2: Overview of the proposed multimodal clinically informed coarse-to-fine registration framework, highlighting anatomy- and risk-guided attention, text-conditioned clinical priors, and foreground-aware optimization.
Figure 3
Metric
Comparison
Ablation
Anatomix
VoxelMorph
TransMorph
SPR
CorrMLP
Base
Base-A
Base-R
Base-A-R
Base-A-R-T
NCC(%) ↑
80.2 (10.64)
85.92 (5.16)
90.1 (6.24)
93.67 (5.76)
94.71 (4.41)
95.43 (3.61)
96.64 (2.25)
96.57 (2.56)
96.72 (1.96)
96.82 (1.69)
MSE (%) ↓
0.89 (0.0400)
0.53 (0.0196)
0.30 (0.0100)
0.11 (0.0036)
0.11 (0.0036)
0.13 (0.0049)
0.12 (0.0036)
0.11 (0.0036)
0.11 (0.0025)
0.11 (0.0025)
SSIM(%) ↑
71.0 (9.32)
79.7 (6.25)
78.99 (8.04)
85.89 (5.31)
86.79 (4.29)
88.34 (5.76)
88.74 (4.84)
88.55 (5.29)
89.06 (4.00)
89.13 (3.61)
Table 1: Quantitative comparison and ablation results across NCC, MSE, and SSIM.
Figure 6: Qualitative comparison of registration results for a representative head case across axial and sagittal views, with the CTV highlighted in red and zoomed regions showing local deformation differences.
Medical image registration aligns scans across time, patients, or imaging modalities to track anatomy and support diagnosis, treatment, and image-guided procedures.
Adaptive radiotherapy must account for anatomical changes during treatment, but doing so depends on reliably mapping targets, organs at risk and dose across serial images. Existing registration methods treat each image pair independently, without using radiotherapy-specific context or the patient's preceding anatomical trajectory. We developed Clinical-context-Aware and Patient-adaptive Deformable Image Registration (CAP-DIR), integrating source contours, dose, clinical text and prior-registration features. We evaluated 16,361 CT examinations from 3,473 patients across four institutional cohorts and five public benchmarks. CAP-DIR increased internal-test organ-at-risk-region normalized cross-correlation from 84.76% to 96.92% relative to the strongest learning-based comparator, with low deformation folding, and remained superior across seven external cohorts. In a blinded review of 20 longitudinal cases, clinically acceptable contours increased from 45.0% to 82.5% for clinical target volumes and from 22.5% to 62.5% for organs at risk, while estimated editing time decreased. In a controlled benchmark, CAP-DIR reduced global dose-mapping error by 34.5% relative to the strongest comparator. These findings show that clinically informed, longitudinal registration can improve contour usability and dose mapping for adaptive radiotherapy.
Caiwen Jiang, Songlin Zhao, Yuzhen Ding +23
Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA · School of Biomedical Engineering, ShanghaiTech University, Shanghai, China · Department of Radiation Oncology, Mayo Clinic, Rochester, Minnesota, USA +2
Head-and-neck (HN) proton therapy is highly sensitive to anatomical change over a 4-to-6-week course, as tumor shrinkage, weight loss, and setup variation can misposition the Bragg peak near critical organs such as the parotids, oral cavity, brainstem, and spinal cord, leading to target underdosing or organ-at-risk overdosing. Online adaptive proton therapy replans on the anatomy of the day, yet standard workflows rely on offline replanning that requires repeated CT acquisition and roughly a week of preparation, adding burden, cost, and delay. We investigate whether a patient's treatment-day anatomy can be predicted before image acquisition by transferring longitudinal change from a population database. We propose a digital-twin framework built on a pretrained foundation-model deformable registration network used without patient-specific training. A first registration aligns a prior patient's planning CT to the target and carries the prior's during-treatment quality assurance CT (QACT) into the target frame; a second registration estimates the prior's planning-to-QACT change, which is then applied to the target's own planning CT to synthesize predicted CTs (pdCTs) with propagated contours. Using 88 HN patients, each with a planning CT and three QACTs, we show that pdCTs better match treatment-day anatomy than the static planning CT. Compared with the planning CT alone, normalized cross-correlation improves by 22.8%, Dice for organs-at-risk by 20.2%, and CT-number error decreases by 23.4%. Gains are largest for patients with major anatomical change and negligible when anatomy is stable. This cross-patient motion transfer leverages the digital-twin concept to anticipate treatment-day anatomy, enabling personalized online adaptive proton therapy without repeated imaging.
Yizhou Wu, Yuheng Li, Xiaofeng Yang +1
Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, USA · Department of Radiation and Cellular Oncology, University of Chicago, Chicago, IL 60637, USA
Deformable image registration (DIR) is widely used in radiotherapy for dose propagation and accumulation, but uncertainty in the underlying deformation can substantially affect clinically relevant dose estimates. We present a practical probabilistic framework for propagating DIR uncertainty to voxel-wise dose statistics and dose-volume histograms (DVHs). The method models the mapped correspondence at each voxel as a random variable governed by a transparent local certainty map that can be defined by simple safety margins, structure-boundary mismatch, or structure-wise conservative uncertainty values. This yields interpretable quantities such as dose probabilities, expected dose, confidence bounds, and induced DVH envelopes. The framework is designed to remain lightweight and interpretable: it avoids complex biomechanical or ensemble-based uncertainty models and instead emphasizes simple parameterization, computational feasibility, and transparent dose metrics. We further introduce a structure-guided in/out strategy as an optional refinement that restricts mapping probabilities to anatomically plausible target regions. The approach is demonstrated on a prostate radiotherapy case study and used to compare different certainty-map strategies and probability kernels. The experiments show that the certainty-map design has a stronger effect on resulting dose and DVH uncertainty bounds than the specific kernel choice, while the additional benefit of the in/out strategy is case-dependent and modest in the present example. Overall, the proposed framework provides a transparent way to incorporate DIR uncertainty into radiotherapy dose assessment and to study how modelling choices affect propagated dose metrics.
Stefan Heldmann, Sven Kuckertz, Nasim Givehchi +4
Fraunhofer Institute for Digital Medicine MEVIS, Lübeck, Germany · Varian, a Siemens Healthineers company, Dättwil, Switzerland · Icon Group, South Brisbane, QLD, Australia