Prior-Guided Physics-Informed Deep Learning for Fast 4D MRI Reconstruction
Authors: Mojtaba Safari, Parag J. Parikh, Shansong Wang, Mingzhe Hu, Philip Connell, Stanley L. Liauw, Ralph R. Weichselbaum, Zhen Tian, Xiaofeng Yang
Organizations: Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA · Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA
Background and purpose: We propose PGPI4MR (prior-guided, physics-informed 4D MRI reconstruction), a deep learning method that leverages information from other respiratory phases or a setup scan, the first 30 s of the same scan, to improve 4D MRI reconstruction without extra scan time. Materials and methods: PGPI4MR is an unrolled network that reconstructs 16 respiratory phases from 60, 30, or 15 s of data, and single 3D volumes, each from 0.29-0.49 s of data, using a setup-scan image. The networks were trained once, on other volunteers, and were not retrained for new volunteers. We evaluated PGPI4MR on measured k-space from three datasets: free-breathing MR-linac (1.5T) and abdominal (3T) scans, and thigh scans with cuff compression (1.5T). All methods, including comparative models, received identical samples and were scored similarly. Results: In vivo, PGPI4MR had the lowest held-out error for 4D MRI at every scan time, and its 16 phases were ready a median of 7.5-13 s after the scan on the 2 GPUs tested. Coupling the 16 respiratory phases reduced the error at 60 s by 9.1% compared with reconstructing each phase alone. For sub-second MRI, the error at 0.49 s was 15.5% lower than that of the best comparison method, classical reconstruction toward the same prior, and on the faster GPU, every 0.29 s window was acquired and reconstructed within 0.5 s. Without retraining, PGPI4MR matched the best methods on the 3T abdominal scans but not on the thigh scans; after retraining on thigh data, it reduced the held-out error by 6.6% relative to the best comparison method. Conclusion: Prior information allowed PGPI4MR to reconstruct 4D and sub-second MRI more accurately than compressed sensing and a published 4D network, within seconds and without extra scan time. This could support 4D MRI for treatment planning and 3D motion monitoring during MR-guided radiotherapy.
Figures & tables
In vivo: held-out error
Simulation: NRMSE
Question
Applied to
Tested
Compared with
Analysis
% lower
95% CI
% lower
95% CI
Coupling the phases (4D MRI)
PGPI4MR, 60 s †
Phases coupled (4D)
Each phase alone (3D)
Primary
9.1 ∗
8.1 to 10.2
28.8 ∗
24.9 to 32.5
PGPI4MR, 30 s
Phases coupled (4D)
Each phase alone (3D)
Registered
9.9 ∗
8.9 to 10.8
35.9 ∗
32.4 to 39.3
PGPI4MR, 15 s
Phases coupled (4D)
Each phase alone (3D)
Registered
7.8 ∗
6.2 to 9.4
33.2 ∗
29.6 to 36.7
No prior information
4D MRI, 60 s
3D network
BART
Exploratory
− 7.3 ∗
− 9.2 to − 5.4
− 14.2 ∗
− 19.6 to − 9.1
Sub-second MRI
PGPI4MR without prior
BART
Exploratory
− 8.2 ∗
− 13.1 to − 3.5
− 8.6 ∗
− 16.5 to − 1.3
Table 2: Ablation: the effect of each design choice in vivo (MR-linac) and in physics-based simulation. Each value is the percentage by which the tested setting lowered the error of the setting it is compared with (negative, a higher error), with its 95% confidence interval.
Figure 2Figure 3Figure 4Figure 5
Figure S1: Organs below the heart in sub-second 3D MRI of one volunteer (axial slice 93 mm below the liver dome). (A) In-vivo data, the first 0.29, 0.39, and 0.49 s windows after the setup scan (top to bottom); the box on the MOTUS reference marks the region enlarged in B. (B) The boxed region. Red arrows, the pancreas, identified visually on the MOTUS reference; amber arrows, the aorta, from the contour provided with the data. The arrows mark the same positions in every column of a row. The MOTUS reference is shown for anatomy only.
Figure S2: Held-out k-space error against image NRMSE on the UMC thigh data (real k-space with fully sampled references; 0.69 s windows). Each point is the mean of one method over the 9 volunteers. Open circles, methods without a prior; filled circles, methods given the setup prior. Networks not labeled as trained on UMC volunteers were applied zero-shot. E2E-VarNet lies outside the plotted range (Table S4 ).
60 s
30 s
15 s
Data
Volunteer
ZF
CG-S
BART
3D
4D
ZF
CG-S
BART
3D
4D
ZF
CG-S
BART
3D
4D
4D MRI, real data: held-out k-space error ↓
1
1.84
1.59
1.20
1.35
1.24
1.90
1.70
1.34
1.43
1.31
1.93
1.76
1.50
1.52
1.45
2
1.83
1.61
1.34
1.40
1.26
1.88
1.68
1.46
1.48
1.35
1.94
1.79
1.62
1.60
1.53
3
1.85
1.60
1.28
1.37
1.27
1.92
1.75
1.48
1.52
1.37
1.97
1.88
1.65
1.66
1.50
4
1.84
1.64
1.33
1.40
1.26
1.92
1.77
1.48
1.50
1.33
1.95
1.82
1.60
1.62
1.50
5
1.84
1.60
1.27
1.38
1.27
1.91
1.71
1.48
1.53
1.41
1.95
1.80
1.61
1.61
1.53
Table S1: Per-volunteer results of every method. Arrows mark the better direction; bold, best; underlined, second best.
MODEST, 4D MRI
Data
Volunteer
60 s (simulation: 61 s)
30 s
15 s
Real data: held-out k-space error ↓
6
1.86
1.85
1.88
9
1.94
1.93
1.80
10
1.90
1.89
1.81
Simulation: NRMSE ↓
6
0.234
0.287
0.471
9
0.300
0.379
0.608
Table S1. Per-volunteer results of every method (continued)
Simulation
Motion
Metric
Method
0.49 s
0.39 s
0.29 s
Normal breathing
NRMSE ↓
CG-SENSE
0.731
0.766
0.799
Unmoved setup prior alone
0.320
0.319
0.313
PGPI4MR, unmoved prior
0.261
0.267
0.284
Prior moved along the registration-based field, alone
0.310
0.309
0.308
PGPI4MR, prior with registration-based field
0.251
0.258
0.277
Table S1. Results of sub-second MRI in physics-based simulation under 3 kinds of motion (continued). Values are means over 10 volunteers; bold, best; underlined, second best.
60 s
30 s
15 s
Task
Comparison
Unit
% lower
95% CI
% lower
95% CI
% lower
95% CI
4D MRI
PGPI4MR (4D, phases coupled) vs PGPI4MR without coupling (3D), primary end point
Volunteers ( n = 10)
9.1 ∗
8.1 to 10.2
9.9 ∗
8.9 to 10.8
7.8 ∗
6.2 to 9.4
Folds ( n = 5)
9.1 ∗
7.8 to 10.4
9.9 ∗
8.9 to 10.8
7.8 ∗
6.4 to 9.3
Without the 2 development volunteers ( n = 8)
9.0 ∗
7.7 to 10.3
9.7 ∗
8.6 to 10.9
7.6 ∗
5.5 to 9.7
PGPI4MR (4D, phases coupled) vs BART
Volunteers ( n = 10)
2.5 ∗
0.1 to 4.9
6.5 ∗
5.0 to 8.0
5.9 ∗
4.5 to 7.3
Folds ( n = 5)
2.5
− 0.7 to 5.6
6.5 ∗
4.3 to 8.6
5.9 ∗
3.9 to 7.9
Table S2: Sensitivity analyses in vivo: the main comparisons with the fold as the unit and without the development volunteers. Each value is the percentage by which the first method lowered the held-out k-space error of the second (negative, a higher error), with its 95% confidence interval.
Compute, ms ↓ : median / 95th percentile
Latency of the slowest window, s ↓ : from the middle / from the start
Window
Method
Ada
Blackwell
Ada
Blackwell
Windows (volunteers)
MR-linac, 3 frames (0.29 s)
Zero-filled
< 1 / < 1
< 1 / < 1
0.150 / 0.297
0.149 / 0.297
1,015 (10)
CG-SENSE
31 / 39
20 / 22
0.190 / 0.338
0.170 / 0.318
1,015 (10)
BART
673 / 705
586 / 660
0.870 / 1.018
0.825 / 0.973
405 (4)
E2E-VarNet
204 / 288
132 / 173
0.462 / 0.610
0.321 / 0.469
1,015 (10)
PGPI4MR without prior
145 / 169
109 / 130
0.318 / 0.465
0.277 / 0.425
1,015 (10)
Table S3: Compute time and image latency. Values are the compute time per window or volume and the latency of the slowest window, with the statistic given in each header.
Stanford abdomen, 6 scans with normal breathing:
held-out k-space error ↓
Scan
ZF
CG-S
BART
E2E
NP
CGP
PGPI4MR
101
1.92
1.35
1.15
2.56
1.67
1.17
1.18
102
1.93
1.68
1.45
2.44
1.92
1.42
1.41
104
1.95
1.75
1.45
2.59
1.85
1.46
1.46
106
1.92
1.58
1.29
2.47
1.67
1.30
1.31
Table S4: External data sets. Values are the held-out k-space error of each Stanford scan, correlations ( r ) with the scanner navigator, and mean ± standard deviation over the UMC volunteers. Paired effects are the percentage by which the first method lowered the error of the second. Arrows mark the better direction; bold, best; underlined, second best.
UMC thigh, 9 volunteers, 0.69 s windows (exploratory)
CG toward the
PGPI4MR, trained on
Paired result
Scan
Score
moved prior
other UMC volunteers
% lower
95% CI
Dynamic scan
Held-out k-space error ↓
1.15 ± 0.04
1.08 ± 0.06
6.6 ∗
3.8 to 9.3
Image NRMSE, all windows ↓
0.267 ± 0.02
0.256 ± 0.03
4.4
− 0.4 to 9.0
Image NRMSE, 80 mm Hg ↓
0.281 ± 0.03
0.269 ± 0.03
4.4 ∗
0.2 to 8.3
Shading-insensitive NRMSE ↓
0.356 ± 0.04
0.328 ± 0.04
8.1 ∗
4.0 to 12.0
Table S4. External data sets (continued)
Comparison (first vs second)
Data
n
Effect (95% CI)
P
Table 1: the best value of a column against the second best
PGPI4MR (4D) vs BART, 60 s
Real
10
2.52 (0.06 to 4.91)
0.045
PGPI4MR (4D) vs BART, 30 s
Real
10
6.49 (4.95 to 8.01)
6.5×10−6
PGPI4MR (4D) vs BART, 15 s
Real
10
5.92 (4.51 to 7.30)
6.4×10−6
PGPI4MR (4D) vs BART, 61 s
Simulated
10
18.67 (16.86 to 20.44)
5.3×10−9
PGPI4MR (4D) vs 3D network, 30 s
Simulated
10
35.95 (32.36 to 39.35)
1.8×10−8
Table S5: Exact P values of the comparisons in Tables 1, 2, S2, and S4 and of the registered analyses.
Radiology Department, New York University Grossman School of Medicine, New York, NY 10016 USA · Electrical and Computer Engineering Department, New York University Tandon School of Engineering, Brooklyn, NY 11201 USA
Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN, USA · Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA
Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN, USA · Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA · Department of Medicine (Cardiology), University of Minnesota, Minneapolis, MN, USA