Diffusion models have emerged as powerful priors for solving inverse problems in computed tomography (CT). In certain applications, such as neutron CT, it can be expensive to collect large amounts of measurements even for a single scan, leading to sparse data sets from which it is challenging to obtain high quality reconstructions even with diffusion models. One strategy to mitigate this challenge is to leverage a complementary, easily available imaging modality; however, such approaches typically require retraining the diffusion model with large datasets. In this work, we propose incorporating an additional modality without retraining the diffusion prior, enabling accelerated imaging of costly modalities. We further examine the impact of imperfect side modalities on cross-modal guidance. Our method is evaluated on sparse-view neutron computed tomography, where reconstruction quality is substantially improved by incorporating X-ray computed tomography of the same samples.
Figures & tables
Figure 1: The cross-modal approach decouples fitting main modality measurements from enforcing cross-modal consistency. In the first step, we fine-tune the diffusion model weights to better fit the data consistency loss, then obtain the reconstruction estimate via any diffusion-based inverse problem solver. After that stage, cross-modal consistency is enforced via a lightweight image translation model. The cross-modal consistency block can be seamlessly incorporated as a separate module without modifying the prior.
Figure 2: Paired degraded NCT/XCT samples (left) and their high-quality reconstructions (right). These examples illustrate the training data used for cross-modal translation.
Figure 3: Comparison between XCT images (top row) and corresponding ideal NCT images (bottom row). XCT quality varies widely relative to NCT.
Steps
#Views
Unimodal
Cross-modal
Δ
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR
SSIM
5
8
20.10
0.374
21.17
0.375
+1.07
+0.001
16
23.34
0.455
23.97
0.471
+0.63
+0.016
32
25.82
0.490
27.45
0.620
+1.63
+0.130
64
28.62
0.648
29.12
0.762
+0.50
+0.114
128
28.74
0.662
28.54
0.777
-0.20
+0.115
Table 1: Comparison of reconstruction quality (PSNR [dB] / SSIM) for Unimodal vs. Cross-modal for different numbers of views, no measurement noise, and at T=10 alternating steps between enforcing the prior and data consistency. Δ=Cross−Uni .
Figure 4: Qualitative comparison between D3IP and our cross-modal guided algorithm across different numbers of projection views ( 8 , 16 , 32 , and 64 views, in rows 1–4, respectively). Within each row, the first column shows Unimodal (D3IP) reconstructions, the second column shows our cross-modal guided algorithm reconstructions, and the third column shows the ground truth samples.