cs.CVMar 4, 2026
SavesFRC for assessing hallucinations in medical image restoration
Organizations: Center for Devices and Radiological Health in FDA, Silver Spring, MD, 20993, USA
Abstract
Deep learning (DL) methods are currently being explored to restore images from sparse-view-, limited-data-, and undersampled-based acquisitions in medical applications. Although outputs from DL may appear visually appealing based on likability/subjective criteria (such as less noise, smooth features), they may also suffer from hallucinations. This issue is further exacerbated by a lack of easy-to-use techniques and robust metrics for the identification of hallucinations in DL outputs. In this work, we propose performing Fourier Ring Correlation (FRC) analysis over small patches and concomitantly (s)canning across DL outputs and their reference counterparts to detect hallucinations (termed as sFRC). We describe the rationale behind sFRC and provide its mathematical formulation. The parameters essential to sFRC may be set using predefined hallucinated features annotated by subject matter experts or using imaging theory-based hallucination maps. We use sFRC to detect hallucinations for three undersampled medical imaging problems: CT super-resolution, CT sparse view, and MRI subsampled restoration. In the testing phase, we demonstrate sFRC's effectiveness in detecting hallucinated features for the CT problem and sFRC's agreement with imaging theory-based outputs on hallucinated feature maps for the MR problem. Finally, we quantify the hallucination rates of DL methods on in-distribution versus out-of-distribution data and under increasing subsampling rates to characterize the robustness of DL methods. Beyond DL-based methods, sFRC's effectiveness in detecting hallucinations for a conventional regularization-based restoration method and a state-of-the-art unrolled method is also shown.
Figures & tables
Fig. 1: A range of artifacts, including (a) patient implant-based artifact [ 1 ] ), (b) missing wedge-based acquisition artifacts (described in SI section S2), and (c) hallucinations due to the application of an AI-assisted SRGAN model (described in IV-B1 ) to upsample the low-resolution counterpart of (d) by a factor of . As opposed to artifacts in (a) and (b), hallucinations in (c) in the yellow box (with the two loops of bowel instead of one contiguous loop) and in the green box (an unwarranted plaque-like structure), only become concretely apparent after a thorough comparison against its normal-resolution reference in (d) . Display windows for (a) is (W: L: ), (b) is (W: L: , and (c-d) is (W: L: ).
| Data Fidelity Analysis | Physical Quality Analysis | Task-based Image Quality Approach | |
|---|---|---|---|
| Examples | Image fidelity, visual perceptual quality, distance, and distribution metrics (e.g., PSNR, SSIM, RMSE, FID, KL divergence, etc.). | Traditional Bench testing approaches (e.g., NPS, MTF, HU accuracy, geometric accuracy, signal ghosting, position accuracy, etc.) [ 2 , 3 ] . | Numerical model observers (e.g., Channelized Hotelling Observer, Non-Prewhitening Matched Filter); forced-choice experiments (e.g., 2AFC, 4AFC); multireader multicase (MRMC) studies [ 4 , 5 ] . |
| Task dependent | No | No | Yes |
| Designed for or effective in analyzing non-linear methods | No | No | Yes |
| Purpose | Inference of generic, task-agnostic performance; commonly used for system appraisal [ 6 ] . | Inference of generic, task-agnostic performance; commonly used for system appraisal [ 6 ] . | Inference of human or model performance for specific tasks (e.g., detection or discrimination of subtle lesions or signals) [ 7 , 8 ] . |
| Hallucinatory inference | No | No | No or not explored |
TABLE I: Performance testing methodologies, their primary purposes, and their ability to assess hallucinations in image restoration problems
Fig. 2: Decomposition of a patch containing a hallucinated structure in (g) - indicated by a red arrow - into its frequency-based components in (h-l). (m) is a normal-resolution CT patch. (g) is obtained by upsampling (using SRGAN described in section IV-B1 ) the low-resolution ( downsampled four times) counterpart of (m). (a) is the Fourier transform of (g). (b-f) are different bandpass filters . (h-l) are the image components of (g) obtained by convolving the bandpass filters, as indicated in (b-f), with (g). Similarly, (n-r) are the image components of (m) obtained by convolving the bandpass filters, as shown in (b-f), with (m).
Fig. 3: A visual depiction of our sFRC analysis to detect hallucinated regions. Complimentary pairs of patches (i.e., over the same x-y coordinate) of the same dimension are scanned across the image pairs from the two methods (reference and deep learning images) in (a). FRC is calculated in (b) across all patch pairs from (a). In addition to the FRC threshold used in a typical FRC calculation, we introduce a new threshold called , a line parallel to axis. Patches corresponding to the deep learning method whose FRC comparison against their reference counterparts lead to are labeled as candidates to exhibit hallucinations. As shown in fig. 4 , is set a priori using hallucinated ROIs/maps estimated by an imaging theory or annotated by subject matter experts using tuning or developmental set.
Fig. 4: An illustration for setting as an upper bound of s using patches that are labeled as hallucinations by subject matter experts or using an imaging theory.
Fig. 5: An illustration of the imaging theory-based approach to set . In our sFRC analysis of the MRI output post-processed, using a U-Net in (c), is set as its FRC threshold and as its such that the patches labeled as hallucinations from our analysis (indicated by the red bounding boxes in (d)) overlap with the false structured obtained using a theoretical approach proposed by Bhadra et al. in [ 15 ] .
Fig. 6: The purple ROIs indicate hallucinated patches in the SRGAN outputs that were initially missed by our sFRC analysis in the top right scan (000077.IMA). The regions corresponding to the missed hallucinated patches get flagged in the subsequent CT scans in the center right (000080.IMA) and the bottom right (000082.IMA) within the abdominal CT volume of patient L [ 1 ] . Display window is (W: L: ).
| Model (test set kernel) | No. of patches detected as | sFRC hallucination |
|---|---|---|
| hallucinations | rate (in ) | |
| SRGAN (sharp) | ||
| SRGAN (smooth) | ||
| SR-WGAN (sharp) | ||
| SR-WGAN (smooth) |
TABLE II: sFRC-detected hallucinations on sharp and smooth test sets for SRGAN and SR-WGAN models. Both test sets consist of CT images (i.e., patches)
Fig. 7: An explicit illustration of the decay in performance of the SRGAN super-resolution model - trained using smooth data - when analyzed against (b) sharp and (d) smooth kernel-based normal-resolution patches using our sFRC analysis. The smoothly trained SRGAN generated hallucinated indentations in (c), which were detected by our sFRC analysis. In contrast, the same SRGAN upsampled smooth test data more faithfully in (e) and is not flagged by our sFRC analysis. The patches in (b-e) correspond to an ROI of an abdominal CT scan in (a). Display window is (W: L: ).
Fig. 8: Plaque-like hallucination that is wrongly added along a bordering region (displayed using the arrow (1) and underfitting of fatty attenuation to air (displayed using the arrow (2)) in the SRGAN-based output as compared to its normal-resolution FBP counterpart. Radiologists are trained to look for even small amounts of air. Such air-based hallucinations may mislead a radiologist’s clinical decision. Display window is (W: and L: ).
Fig. 9: A ground truth (in (a)) was used to perform subsampled MRI acquisition by an acceleration factor of 3. The subsampled output is post-processed by a U-Net (in (b), (d)) and reconstructed using PLS-TV (in (c), (e)). Bhadra et. al’s linear operator theory-based approach (in (b), (c)) and our sFRC analysis (in (d), (e)) are applied on the U-Net and PLS-TV outputs to identify hallucinations. In fig. (f), the first column depicts hallucinated ROIs missed (false negative) by our and Bhadra et al.’s [ 15 ] approaches; the second column depicts an ROI missed by our approach on the PLS-TV output; and the third depicts an ROI missed by Bhadra et al’s approach on the U-Net.
Fig. 10: (a) Red bounding boxes as hallucinations detected by our sFRC analysis on a U-Net output. It was restored from a subsampled MRI data acquired using an acceleration factor of three ( x). (b) Zoomed view of the purple dotted region from (a). The first two rows in (b) correspond to the ground truth and iFFT ( x) counterparts of the purple dotted region from (a).
Fig. 11: (a) Ground truth image and its corresponding (b) iFFT, (c) the U-Net, and (d) PLS-TV outputs restored using a subsampled acquisition (i.e., accelerated by a factor three ( x)). PSNR and SSIM were calculated over signal-only regions. Plot (c) exhibits a high PSNR and SSIM values, and visually it looks as good as plot (a). Still, plot (c) is contaminated with hallucinations, as shown in fig. 10 .
| Method | No. of patches | sFRC hallucination |
|---|---|---|
| detected as hallucinations | rate (in ) | |
| iFFT ( x) | ||
| iFFT ( x) | ||
| iFFT ( x) | ||
| U-Net ( x) | ||
| PLS-TV ( x) |
TABLE III: sFRC analysis-based hallucinations relative to different subsampling rates and restoration methods from the four MRI test scans (totaling patches)
Fig. 12: The reference CT image in (a) and its 36 view–based reconstruction from a state-of-the-art (SOTA) model in (c) are processed using sFRC to detect potentially hallucinated local regions, shown as red bounding boxes in (b). Display window is (W: L: )
| Restoration | Training | Test | No. of Testing | PSNR | SSIM | Hellinger | sFRC |
| Method | set | set | [Images Patches] | (std) | (std) | Distance (std) | hallucination rate |
| CT Super-resolution problem | |||||||
| Bicubic (x ) | - | smooth | |||||
| SRGAN (x ) | smooth | smooth | |||||
| SR-WGAN (x ) | smooth | smooth | |||||
| Bicubic (x ) | - | sharp | |||||
TABLE IV: Variation in restoration performance across different evaluation metrics. Higher values indicate better performance for PSNR ( ) and SSIM ( ), whereas lower values indicate better performance for the Hellinger distance ( ) and normalized sFRC ( ).
Fig. 13: A visual depiction of how our patch-wise (scanned) FRC (sFRC) plot enables estimation of the hallucination rate as a function of the sFRC’s hallucination threshold ( ). Concretely, the minimum value is set based on the sampling rate for a given restoration problem, while the maximum value is determined by considering the resolving power of the imaging modality and the high-frequency bands where confirmed hallucinations have been isolated.
Fig. 14: Hallucination Operating Characteristic (HOC) curve for the SRGAN model - trained on smooth data - evaluated using smooth and sharp test sets.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| No. of patients used | Testing | PSNR | SSIM |
|---|---|---|---|
| to train (training kernel) | kernel | (std) | (std) |
| 6 patients (smooth) | smooth | ||
| 8 patients (smooth) | smooth | ||
| 6 patients (smooth) | sharp | ||
| 8 patients (smooth) | sharp |
TABLE I: SSIM and PSNR of SRGAN trained using six and eight patient data and tested using CT images
Fig. S1: An illustration of user-defined hallucinated ROIs (as bounding boxes) and markers (as arrows) annotated in the SRGAN upsampled output in (d) by comparing the image against its normal-resolution reference in (c). The comparison was assisted using the complementary components (e, f) of the two CT images (c, d) retrieved by convolving a bandpass filter ( to ) in (b) to (c, d), respectively. Prior information that the normal-resolution CT reference and low-resolution test data correspond to contrast-enhanced abdominal CT examinations in the portal-venous phase of enhancement was also utilized when comparing. The normal-resolution CT scan in (c) corresponds to the axial scan 000039.IMA of patient L (acquired at a mm slice thickness using a sharp kernel). This normal-resolution CT scan was provided in the LDGC repository [ 1 ] . The display window for CT images in (c, d) is (W: L: ).
Fig. S2: An illustration of user-defined hallucinated ROIs (as bounding boxes) and a marker (as an arrow) annotated in the SRGAN upsampled output in (d) by comparing the image against its normal-resolution reference in (c). The comparison was assisted using the complementary components (e, f) of the two CT images (c, d) retrieved by convolving a bandpass filter ( to ) in (b) to (c, d), respectively. Prior information that the normal-resolution CT reference and low-resolution test data correspond to contrast-enhanced abdominal CT examinations in the portal-venous phase of enhancement was also utilized when comparing. The normal-resolution CT scan in (c) corresponds to the axial scan 000069.IMA of patient L (acquired at a mm slice thickness using a sharp kernel). This normal-resolution CT scan was provided in the LDGC repository [ 1 ] . The display window for CT images in (c, d) is (W: L: ).
Fig. S3: An illustration of user-defined hallucinated ROIs (as bounding boxes) annotated in the SRGAN upsampled output in (d) by comparing the image against its normal-resolution reference in (c). The comparison was assisted using the complementary components (e, f) of the two CT images (c, d) retrieved by convolving a bandpass filter ( to ) in (b) to (c, d), respectively. Prior information that the normal-resolution CT reference and low-resolution test data correspond to contrast-enhanced abdominal CT examinations in the portal-venous phase of enhancement was also utilized when comparing. The normal-resolution CT scan in (c) corresponds to the axial scan 000149.IMA of patient L (acquired at a mm slice thickness using a sharp kernel). This normal-resolution CT scan was provided in the LDGC repository [ 1 ] . The display window for CT images in (c, d) is (W: L: ).
Fig. S4: An illustration of user-defined hallucinated ROIs (as bounding boxes) and markers (as arrows) annotated in the SRGAN upsampled output in (d) by comparing the image against its normal-resolution reference in (c). The comparison was assisted using the complementary components (e, f) of the two CT images (c, d) retrieved by convolving a bandpass filter ( to ) in (b) to (c, d), respectively. Prior information that the normal-resolution CT reference and low-resolution test data correspond to contrast-enhanced abdominal CT examinations in the portal-venous phase of enhancement was also utilized when comparing. The normal-resolution CT scan in (c) corresponds to the axial scan 000189.IMA of patient L (acquired at a mm slice thickness using a sharp kernel). This normal-resolution CT scan was provided in the LDGC repository [ 1 ] . The display window for CT images in (c, d) is (W: L: ).
Fig. S5: An illustration of user-defined hallucinated ROIs (as bounding boxes) and markers (as arrows) annotated in the SRGAN upsampled output in (d) by comparing the image against its normal-resolution reference in (c). The comparison was assisted using the complementary components (e, f) of the two CT images (c, d) retrieved by convolving a bandpass filter ( to ) in (b) to (c, d), respectively. Prior information that the normal-resolution CT reference and low-resolution test data correspond to contrast-enhanced abdominal CT examinations in the portal-venous phase of enhancement was also utilized when comparing. The normal-resolution CT scan in (c) corresponds to the axial scan 000217.IMA of patient L (acquired at a mm slice thickness using a sharp kernel). This normal-resolution CT scan was provided in the LDGC repository [ 1 ] . The display window for CT images in (c, d) is (W: L: ).
Fig. S6: Illustration of hallucination threshold ( ) getting stricter (i.e., label more patches to be hallucinated) as approaches Nyquist frequency ( ). For instance, in plot (a) is very relaxed, as sFRC will only consider very low-frequency bands (i.e., blurred image components) when performing correlation analysis between Deep learning and reference image pairs. In contrast, in plot (d) is strict, as sFRC considers image components across low- to high-frequency bands when performing the correlation analysis.
Fig. S7: A visual depiction of how the FRC threshold ( ) targets different spatial frequencies in sFRC analysis between deep learning and reference images, with Y set in the range .
Fig. S8: Hallucinations identified in the red bounding boxes from our sFRC analysis in the SRGAN upsampled outputs. The five SRGAN upsampled CT images correspond to images used to tune the sFRC parameters. The sFRC parameters comprised of as the patch size, as the FRC threshold, and as the hallucination threshold ( ). At , almost all the hallucinated ROIs - labeled in figs. S1 through S5 - were missed by the sFRC. The display window for all images is (W: L: ).
Fig. S9: Hallucinations identified in the red bounding boxes from our sFRC analysis in the SRGAN upsampled outputs. The five SRGAN upsampled CT images correspond to images used to tune the sFRC parameters. The sFRC parameters comprised of as the patch size, as the FRC threshold, and as the hallucination threshold ( ). At , almost all the hallucinated ROIs - labeled in figs. S1 through S5 - were captured by the sFRC. The display window for all images is (W: L: ).
Fig. S10: Hallucinations identified in the red bounding boxes from our sFRC analysis in the SRGAN upsampled outputs. The five SRGAN upsampled CT images correspond to images used to tune the sFRC parameters. The sFRC parameters comprised of as the patch size, as the FRC threshold, and as the hallucination threshold ( ). At , most of the hallucinated ROIs - labeled in figs. S1 through S5 - were captured by the sFRC. The display window for all images is (W: L: ).
Fig. S11: Hallucinations identified in the red bounding boxes from our sFRC analysis in the SRGAN upsampled outputs. The five SRGAN upsampled CT images correspond to images used to tune the sFRC parameters. The sFRC parameters comprised of as the patch size, as the FRC threshold, and as the hallucination threshold ( ). At , the sFRC was still able to detect all types of hallucinations (related to plaques, perturbations, and indentations) that were annotated in figs. S1 through S5 . The display window for all images is (W: L: ).
Fig. S12: An illustration of imaging errors spanning from low- to high-frequency components of restored images plagued with Missing Wedge and Distortion artifacts. The second, third, and fourth rows indicate plots corresponding to reference, missing wedge, and distorted images. Imaging filters corresponding to each component of an image (in the first column) are depicted in the second through sixth columns in the top row. Zoomed views of the ROI corresponding to the yellow bounding box are provided for all the image components. Display windows corresponding to each column are provided in the bottom row.
Fig. S13: An illustration of a reference CT image in (a) that was used to simulate different types of conventional CT artifacts. (b-e) plots outcomes from sFRC, as a red-bounded box, on the CT images with artifacts. (b) Missing wedge is due to a limited number of projection angles ( at a stepsize). (c) Distortion is due to a mismatch in the forward projection and backprojection phases by . (d) Blurring artifact is due to interpolation-based pixel upsampling and downsampling by a factor of . (e) Noise is due to CT physics-based noise insertion for an acquisition corresponding to a dose of the reference CT image in (a). The display window for all the CT images is (W: L: ).
Fig. S14: An illustration of linear system–based improvements achieved by progressively decreasing the missing-wedge gap. These linear system–based improvements also translate into a human observer’s increased ability to discern dark lesions—shown using yellow circular regions in plots (b–d)—as well as improvements in full-image PSNR and SSIM values in plots (b–d), and Fourier Ring Correlation (FRC) curves in plot (e). The display window for all the CT images in (a-d) is (W: L: ).
Fig. S15: An illustration of the lack of efficacy of the FRC, PSNR, and SSIM determined using the entire image to evaluate SRGAN, a post-processing non-linear method, accurately. The FRC curves and PSNR and SSIM values indicate apparent improvement with the use of SRGAN for a CT super-resolution problem. However, the second column reveals hallucinations in the SRGAN-based outputs, including the appearance of two bowel loops instead of a single contiguous loop within the yellow-boxed region, as well as the addition of a plaque-like structure in the green-boxed region. The display window for all the CT images in the first two columns is (W: L: ).
Fig. S16: The purple ROIs indicate hallucinated patches in the SRWGAN outputs that were initially missed by our sFRC analysis in the leftmost scan (000077.IMA). The regions corresponding to these missed hallucinations are subsequently flagged in the center (000080.IMA) and rightmost (000082.IMA) scans within the abdominal CT volume of patient L067 [ 1 ] . This plot is similar to that obtained from the SRGAN model shown in fig. 6 of the main paper. Display window is (W: L: ).
Fig. S17: An explicit illustration of the decay in performance of the SRWGAN super-resolution model—trained using smooth data—when analyzed using our sFRC approach against (b) sharp- and (d) smooth-kernel–based normal-resolution patches. The smoothly trained SRWGAN generates hallucinated indentations in (c), which are detected by our sFRC analysis. In contrast, the same SRWGAN upsamples (x ) smooth test data more faithfully in (e) and is not flagged by sFRC. The patches in (b–e) correspond to an ROI from an abdominal CT scan shown in (a). This plot is similar to that obtained from the SRGAN model shown in fig. 7 of the main paper. Display window is (W: L: ).
Fig. S18: Red bounding boxes indicate hallucinations identified by our sFRC analysis in an SRWGAN output. A plaque-like hallucination is incorrectly added along a bordering region (indicated by arrow (1)), and an ROI with fatty attenuation is underfitted as air (indicated by arrow (2)) in the SRWGAN-based output compared with its normal-resolution FBP counterpart. Radiologists are trained to identify even small amounts of air; therefore, such air-based hallucinations can mislead clinical decision-making. This plot is similar to that obtained from the SRGAN model shown in fig. 8 of the main paper. However, compared with fig. 8, sFRC flags two additional ROIs in the SRWGAN results here, as shown in the third column. Red arrow (3) indicates a cut-like hallucination, while marking (4) appears to be a false positive. Display window is (W: L: ).
Fig. S19: Sparse view sinogram data is shown in (a). The corresponding FBP reconstruction is shown in (b), and the PAIL reconstruction is shown in (f). The fully sampled FBP reconstruction is provided in (e). sFRC is performed using the fully sampled FBP and PAIL reconstructions in (e) and (f), with and a patch size of . The subsequent hallucinations, highlighted by red bounding boxes in (d), were confirmed and labeled in coordination with a medical officer and are shown in (f). Display window in (b-f) is (W: L: ).
More in this topic
Hallucination Detection
Hallucination detection identifies unsupported, fabricated, or evidence-inconsistent content in language and multimodal model outputs.
See all 78 papers