Evaluating the Safety of Deep Learning-Based Brain MRI Reconstruction

Dat Tat Mai; Thai Viet Pham; Thu Nguyen Thi Dang; James Jin Kang

eess.IVAug 28, 2026

Evaluating the Safety of Deep Learning-Based Brain MRI Reconstruction

Authors: Dat Tat Mai, Thai Viet Pham, Thu Nguyen Thi Dang, James Jin Kang

Organizations: School of Science, Engineering & Technology, RMIT University Vietnam, Ho Chi Minh City, Vietnam · School of Computing Technologies, RMIT University, Melbourne, VIC 3000, Australia · School of Health and Biomedical Science, RMIT University, Melbourne, VIC 3000, Australia

Abstract

Objective: Deep learning accelerates brain MRI four- to tenfold, but models can erase lesions or synthesize false tissue - failures pixel-averaged metrics like PSNR and SSIM miss. We review whether current evaluation practices detect this blind spot. Methods: Following PRISMA 2020, we searched seven databases without date limits, including 263 studies (1995-2026), appraised them using QUADAS-2 and matched instruments, and synthesized narratively. Categories were derived from titles, abstracts, and controlled vocabulary; reported prevalence figures represent floors. Duplicate screening achieved high agreement (Fleiss kappa = 0.877), as did appraisal (0.788; 0.390 where observable). Extraction is unaudited. Results: Only 18 of 263 studies (6.8%) recorded both a fidelity metric and reader assessment on identical data, leaving the central surrogate unmeasured. Reader studies mostly measured inter-reader agreement, which was weak: fastMRI 2020 concordance reached 0.457 and 0.386 (Kendall W), improving only where SSIM diverged. Erasing a 100 mm3 lacunar infarct shifts global PSNR by 0.03 dB under the stated error model. As the corpus grew fivefold, reader assessments dropped from 32% to 18%, recovering to 21%. Generative models - most associated with hallucination (39%) - were among the least reader-evaluated (11.3%), while self-supervised models reached 47% with zero reader evaluation. Only 5% released code and ran reader studies; none evaluated a model observer; no named dataset covered acute stroke or hemorrhage. Conclusions: On these floors, current evaluation practices cannot certify diagnostic safety. We derive five requirements safety-oriented evaluations must meet.

Figures & tables

Database	Executed Query Structure	Controlled Vocabulary	Fields Searched	Hits ( $N$ )
PubMed / MEDLINE	(B1 ${}_{\text{ft}}$ [TIAB] OR B1 ${}_{\text{mesh}}$ ) AND (B2 ${}_{\text{ft}}$ [TIAB] OR B2 ${}_{\text{mesh}}$ ) AND (B3 ${}_{\text{ft}}$ [TIAB] OR B3 ${}_{\text{mesh}}$ ) NOT (X[TIAB])	Yes : 9 MeSH headings OR-ed into the three blocks	Title/Abstract $+$ MeSH	$24{,}693$
SpringerLink	(B1) AND (B2) AND (B3) NOT (X) ; quoted phrases, no truncation operators	No	Full record	$1{,}598$
IEEE Xplore	(B1) AND (B2) AND (B3) NOT (X) ; wildcard truncation retained	No	Metadata	$778$
Scopus	TITLE-ABS-KEY(B1) AND TITLE-ABS-KEY(B2) AND TITLE-ABS-KEY(B3) AND NOT TITLE-ABS-KEY(X)	No	TITLE-ABS-KEY	$127$
Semantic Scholar	(B1) + (B2) + (B3) , where \| denotes OR and + denotes AND	No	Title/Abstract	$66$
OpenAlex	title_and_abstract.search:B1, title_and_abstract.search:B2, title_and_abstract.search:B3	No	Title/Abstract	$34^{*}$

TABLE I: Query structure executed in each database and record hit breakdown.

Element	Definition	Application in this Review
Population	Target imaging modalities, clinical domains, and scanner physics	Human brain MRI acquisitions (acute ischemic stroke, neurovascular lesions, intracranial haemorrhages, tumours, and structural T1w, T2w, FLAIR, DWI scans across $1.5\text{T}$ , $3\text{T}$ , and $7\text{T}$ field strengths).
Intervention	AI-based and accelerated reconstruction methods, and their safety/fidelity evaluation	Deep learning and accelerated reconstruction methods (unrolled physics-based networks, GANs, diffusion priors, self-supervised, denoising, super-resolution), together with methods for evaluating their fidelity, artifacts, robustness, or failure modes.
Comparison	Baseline metrics, observers, and reference standards	Conventional pixel-wise metrics (PSNR, SSIM, NRMSE), perceptual/learned metrics (LPIPS, FID), model observers, fully sampled reference images, or radiologist reader assessments.
Outcome	Reconstruction quality, fidelity, and failure characterization	Image-quality/fidelity measures, artifact and failure-mode characterization, metric–radiologist agreement, uncertainty estimates, robustness to distribution shift, and diagnostic/observer outcomes.
Study Design	Methodological criteria and empirical validation settings	Primary empirical studies, algorithmic benchmarking, counterfactual interventional stress-testing, and human or model observer evaluations.

TABLE II: PICOS elements and their application in this review.

Indicator / Domain	Value / Risk of Bias Breakdown
Publication period	1995–2026; 229/263 ( $87.1\%$ ) published in 2020 or later, and 174/263 ( $66.2\%$ ) in 2023 or later, indicating a predominantly recent evidence base.
Basis of assessment, current	Full text available for 258/263 ( $98.1\%$ ); partial or truncated full text for 1/263 ( $0.4\%$ ); title, abstract and MeSH only for 4/263 ( $1.5\%$ ).
Basis of assessment, at appraisal	The first appraisal round was made on the text obtainable at the time, which for 18/263 studies was abstract and MeSH only (6 of them within the QUADAS-2 subset). A later library sweep recovered full text for 14 of those 18 (including 5 of the 6), and all 14 have since been re-appraised on the recovered full text, so appraisal basis now stands at 257/263 full text, 2/263 partial text and 4/263 abstract and MeSH only (within the QUADAS-2 subset, 56/57 on full text and 1/57 [S001] on abstract-level text). The two rows now differ by a single study (S200), appraised on truncated text whose full version has since become available.
Leading venues	The eight most frequent of the 87 distinct venue titles represented in the corpus, accounting for 128/263 ( $48.7\%$ ) of included studies: Magnetic Resonance in Medicine (40), IEEE Transactions on Medical Imaging (17), AJNR (15), Magnetic Resonance Imaging (14), JMRI (12), Scientific Reports (12), NeuroImage (9), Medical Image Analysis (9).
Predominant design	Algorithmic / methodological reconstruction studies ( $n=206$ ), alongside a verified observer/reader subset ( $n=57$ ) [ 9 , 20 ] . Extraction recorded 61 reader designs; appraisal of their full text found 4 with no human reader, and those 4 were reclassified as algorithmic ( $202+4=206$ ; $61-4=57$ ). Of the 57: 34 are appraised and verified by all three reviewers; 18 were appraised by all three in a second round and await final adjudication of 6 split domain judgements; and 5 (S004, S020, S027, S034, S115) were re-appraised by the reviewers on full text recovered after the first round, with domain judgements verified against the source text. Every reader-derived statistic in this article uses 57.
Reference standard	Expert reader consensus (70), an unspecified stated ground truth (62), a fully sampled acquisition (69), a conventional or standard-of-care acquisition (15); not stated in 47/263 ( $17.9\%$ ).

TABLE III: Corpus-level provenance and methodological quality indicators across the

263

included studies, with QUADAS-2 domain-wise risk of bias and applicability concerns for the full appraised diagnostic/reader subset (

n=57

, rated across seven domains; independently by all three reviewers except for the five studies re-appraised on recovered full text). The

39

studies of the first appraisal round and the

18

of the second are pooled here; the remaining

206

studies were appraised on reproducibility and robustness instead (

57+206=263

). Six of the

399

domain judgements split three ways with no majority and are recorded as Unclear (Section III ).

Indicator / Domain	Value / Risk of Bias Breakdown
QUADAS-2 Risk of Bias ( $n=57$ Reader / Diagnostic Accuracy Studies)
Patient Selection	Low: 15/57 ( $26.3\%$ ); Unclear: 19/57 ( $33.3\%$ ); High: 23/57 ( $40.4\%$ ). The weakest domain: sampling is predominantly purposive or pathology-enriched, and most of the appraised studies state no enrolment rule.
Index Test (AI Reconstruction)	Low: 26/57 ( $45.6\%$ ); Unclear: 9/57 ( $15.8\%$ ); High: 22/57 ( $38.6\%$ ). Low risk falls below half once the second round is included. High ratings arise where readers scored accelerated reconstructions with the reference visible, or where test parameters were set using reference-derived labels.
Reference Standard	Low: 26/57 ( $45.6\%$ ); Unclear: 15/57 ( $26.3\%$ ); High: 16/57 ( $28.1\%$ ). High ratings arise where no artifact-free reference exists, or where lesion truth is defined by the comparator images themselves (incorporation bias). Reading the recovered full text moved four of the re-appraised studies from Low to High here, each because the comparator images, or software applied to them, supplied the truth against which they were scored.
Flow and Timing	Low: 45/57 ( $78.9\%$ ); Unclear: 7/57 ( $12.3\%$ ); High: 5/57 ( $8.8\%$ ). The strongest domain, since retrospective undersampling of the same raw $k$ -space gives a zero interval between index test and reference.

TABLE IV: (Table III continued.) QUADAS-2 risk of bias for the appraised reader/diagnostic subset (

n=57

Indicator / Domain	Value / Risk of Bias Breakdown
QUADAS-2 Applicability Concerns ( $n=57$ )
Patient Selection	Low: 36/57 ( $63.2\%$ ); Unclear: 4/57 ( $7.0\%$ ); High: 17/57 ( $29.8\%$ ), chiefly healthy-volunteer, lesion-free, or non-clinical cohorts.
Index Test	Low: 30/57 ( $52.6\%$ ); Unclear: 4/57 ( $7.0\%$ ); High: 23/57 ( $40.4\%$ ), where the evaluated method is image-domain post-processing, cross-field synthesis, harmonization, or a quality index rather than reconstruction from undersampled $k$ -space.
Reference Standard	Low: 39/57 ( $68.4\%$ ); Unclear: 3/57 ( $5.3\%$ ); High: 15/57 ( $26.3\%$ ).
Across all 228 risk-of-bias judgements: 112 ( $49.1\%$ ) Low, 50 ( $21.9\%$ ) Unclear, 66 ( $28.9\%$ ) High. Across all 171 applicability judgements: 105 ( $61.4\%$ ) Low, 11 ( $6.4\%$ ) Unclear, 55 ( $32.2\%$ ) High. Over the 364 judgements that rest on three independent ratings Fleiss’ $\kappa=0.788$ (95% CI $0.739$ – $0.833$ ), but agreement is not uniform across the two appraisal rounds: the 238 retained first-round judgements were unanimous, the 126 of the second round reach $\kappa=0.390$ (95% CI $0.293$ – $0.484$ ); the 35 judgements of the 5 re-appraised studies are excluded, having been conducted subsequently on recovered full text. Section III gives the decomposition and Section V-B what it bounds. The high proportion of Unclear ratings reflects unreported blinding and sampling procedures, compounded by the one first-round study for which only abstract-level text is available. Six second-round judgements split three ways with no majority and are recorded as Unclear under the rule stated in Section III . Per-study ratings, each with the verbatim supporting quotation, are released as included_characteristics_supplement.csv .

TABLE V: (Table III continued.) QUADAS-2 applicability concerns (

n=57

), with the pooled judgement counts.

Demographic / Method Dimension	Category Breakdown	( $n$ )	Percentage / Key Methodological Notes
Publication Period	1995–2019	$n=34$	$12.9\%$ (Classical CS/PI & early CNNs)
	2020–2022	$n=55$	$20.9\%$ (Unrolled physics & GAN priors)
	2023–2026	$n=174$	$66.2\%$ (Diffusion priors & 3D/4D foundation models)
Reconstruction Paradigm (multi-label; $339$ labels recorded)	Compressed Sensing / Parallel Imaging	$n=93$	Iterative sparsity & coil sensitivity encoding
	Generative (GAN/Diffusion/VAE)	$n=80$	Adversarial perception & posterior sampling
	Motion Correction	$n=57$	Intra-scan motion artifact suppression

TABLE VI: Characteristics of the 263 included studies.

Panel of Figure 7 — Fig. 1: PRISMA 2020 flow diagram for study selection across the two search arms. Multi-tier deduplication reduced $27{,}327$ identified records to $26{,}924$ unique records; title and abstract screening advanced $1{,}149$ candidates. Full texts were obtained for $1{,}068$ of them ( $93.0\%$ ); $81$ could not be retrieved. Following PRISMA 2020, exclusion and background retention are recorded as distinct dispositions: of the $1{,}068$ reports assessed in full, $787$ were excluded ( $785$ on explicit PICOS grounds and $2$ as duplicate reports of included studies) and a further $18$ were retained as background material rather than excluded, leaving $263$ primary studies in the analytical set ( $263+787+18=1{,}068$ ).

Study	Readers and design	Comparison performed	Reported quantity (as defined in the study)
fastMRI 2020 challenge [ 43 ]	6 radiologists ranking submissions for quality of pathology depiction	Radiologist concordance across accelerations; SSIM-based ranking vs reader ranking	Kendall’s $W$ $=0.457$ ( $4\times$ ), $0.386$ ( $8\times$ ), $0.781$ (transfer track); concordance improved as SSIM scores diverged
Tang et al. [18]	7 radiologists, rank-based reading; 2 also Likert scoring. Rankings were collected on synthetically corrupted image pairs, not on reconstructions of undersampled $k$ -space	Quality networks trained on radiologist rankings vs MSE/SSIM as optimisation targets	Ranking accuracy $75.2\%\pm 1.3\%$ and $79.2\%\pm 1.7\%$ with reference image; radiologist interobserver agreement $61.5\%\pm 5.5\%$ ; ranking vs Likert agreement $70.4\%$ vs $25\%$
Radmanesh et al. [9]	Retrospective; neuroradiologist scoring of diagnostic quality at accelerations up to $100\times$	Reader-assigned acceleration thresholds for diagnostic and screening quality	$65/69$ volumes ( $94\%$ ) sufficient for potential screening at $\leq 14\times$ ; interreader ICC $0.875$ ; mean absolute interreader difference in threshold acceleration factor $0.7\times$ (level 1) and $5.4\times$ (level 2)
Sommer et al. [12]	2 neuroradiologists, blinded, 28 clinical cases with real motion	Artifact severity (5-point scale) before vs after network correction	Significant reduction in mean artifact scores ( $p<.03$ ); reference-based metrics inapplicable, as the reference itself is motion-corrupted

TABLE VII: What the reader studies in this corpus actually measured about metric–reader agreement (RQ1). Every value is quoted from the named study, with the reported coefficient and its definition stated. No included study reports a metric–reader rank correlation for deep learning-based brain reconstruction; the rows below are the closest measurements that exist, and the second column states for each row what the readers were actually shown, because two of the four rows were not collected on reconstructions of undersampled

k

-space.

Acc. ( $R$ )	Studies	Reader	M $+$ R	Halluc.	Artifacts	Sequences most often evaluated
$2\leq R<4$	$32$	$7$ ( $21.9\%$ )	$1$ ( $3.1\%$ )	$7$ ( $21.9\%$ )	$25$ ( $78.1\%$ )	T1w 15, T2w 12, FLAIR 11
$4\leq R<8$	$17$	$5$ ( $29.4\%$ )	$1$ ( $5.9\%$ )	$2$ ( $11.8\%$ )	$12$ ( $70.6\%$ )	T1w 9, T2w 9, FLAIR 7
$R\geq 8$	$14$	$5$ ( $35.7\%$ )	$4$ ( $28.6\%$ )	$2$ ( $14.3\%$ )	$8$ ( $57.1\%$ )	T1w 9, T2w 8, FLAIR 8
All	$\mathbf{63}$	$\mathbf{17}$ ( $\mathbf{27.0\%}$ )	$\mathbf{6}$ ( $\mathbf{9.5\%}$ )	$\mathbf{11}$ ( $\mathbf{17.5\%}$ )	$\mathbf{45}$ ( $\mathbf{71.4\%}$ )	T1w 33, T2w 29, FLAIR 26
The $R$ -reporting subset is close to the whole corpus on reader assessment ( $27.0\%$ against $21.7\%$ for all $263$ ), so it is not a systematically better-evaluated stratum. Reader assessment rises with acceleration ( $21.9\%\rightarrow 29.4\%\rightarrow 35.7\%$ ) and paired metric–reader reporting concentrates in the most accelerated band ( $4/14$ against $1/32$ in the $2\leq R<4$ band), but the counts are small and most studies in every band (including $9$ of the $14$ with $R\geq 8$ ) report no reader assessment at all. Artifact description is the most frequently recorded failure mode at every acceleration level, whereas hallucination or fidelity instability is recorded in $11$ of $63$ .

TABLE VIII: Evaluation practice by the highest acceleration factor (

R

) each study reports, among the

63

263

included studies that state one. Bands are half-open, so a study reporting exactly

R=4

falls in the second band and one reporting exactly

R=8

in the third. Counts are studies; percentages are within band. Reader , any observer or reader assessment recorded; M

+

R , a pixel-wise fidelity metric and a reader assessment recorded on the same data; Halluc. , hallucination or fidelity instability among the recorded failure modes; Artifacts , aliasing, ghosting or Gibbs artifacts among them. Acceleration factors were extracted from full text after the registered extraction round, each with a verbatim supporting quotation and excluding matches inside reference lists and citation contexts; the

407

retained quotations come from the

63

studies that state a factor, a mean of

6.5

quotations per such study, and are released as acceleration_factors_evidence.csv . The extraction is screening-level, so counts are lower bounds. No study at any acceleration factor reports a rank correlation between a fidelity metric and reader judgement.

Mechanism	Formal statement	Clinical failure it permits	Why the score does not move
(a) Spatial error averaging	$L_{2}$ error is averaged uniformly over all $H\times W\times D$ voxels, so a lesion contributes with weight $f=V_{\text{lesion}}/V_{\text{total}}$	Erasure of an acute lacunar infarct on DWI or FLAIR, or of a punctate microhaemorrhage on SWI or $T_{2}^{*}$	By ( 2 ), erasing a $100\text{ mm}^{3}$ infarct at $r=100$ moves global PSNR by $0.027$ – $0.036\text{ dB}$ ; a $5\text{ mm}^{3}$ microhaemorrhage by $1.3$ – $1.8\times 10^{-3}\text{ dB}$
(b) Reference dependence	Every reference-based score presupposes a registered, artifact-free reference image	A valid motion-corrected reconstruction is penalised against a corrupted reference, and genuine correction is scored as error	The comparison is no longer between reconstruction and truth; PSNR and SSIM become uninterpretable rather than merely noisy
(c) Loss symmetry	$(\hat{X}_{i}-X_{i})^{2}=(X_{i}-\hat{X}_{i})^{2}$ : the objective is indifferent to the sign of the error	A suppressed hyperintense lesion is scored identically to added background noise of the same magnitude	Diagnosis is asymmetric where the metric is not: a missed finding and a spurious brightening carry the same penalty
(d) Realism–reward trade-off	Adversarial and perceptual objectives reward outputs typical of the training distribution, $\log D_{\phi}(\hat{X})$ , not outputs faithful to the measurement	Synthesis of plausible vessels, sulci or texture that was never acquired	Perceptual scores can rise as measurement fidelity falls, so the objective pays for the hallucination it should penalise

TABLE IX: The four mechanisms by which a whole-volume fidelity score fails to register a focal diagnostic error (RQ2). The third column names a failure the mechanism permits, not one this review has observed at a stated frequency; the fourth states why the score does not move. Numerical entries are computed from Equation ( 2 ) under the stated error model, not measured in any primary study.

Fig. 2: The PSNR cost of erasing a focal lesion, from Equation ( 2 ). Curves give

|\Delta\mathrm{PSNR}|

against erased lesion volume for three local error ratios

r

at a

1.4\text{ L}

brain; the shaded band around

r=100

spans the

1.2

–

1.6\text{ L}

range. Markers show the three clinical anchors used in the text. The horizontal line at

0.01\text{ dB}

is the two-decimal precision at which PSNR is conventionally reported, and the shaded left-hand region is the volume below a single voxel on a clinical DWI acquisition, where erasure is not defined because the finding is not resolved. Every value is computed from the identity under the stated error model; nothing in this figure is measured.

Evaluation Paradigm	Core Mechanism	What It Can and Cannot Establish	Key References
1. Pixel-wise fidelity metrics	$L_{2}$ / SSIM whole-image voxel averaging	Aggregate agreement with a reference only. Cannot resolve focal change: by ( 2 ), erasing a resolvable $100\text{ mm}^{3}$ lacunar infarct at $r=100$ costs $0.03\text{ dB}$ . Requires a registered reference	Recorded in $84$ of the $144$ evaluation-labelled studies
2. Perceptual & learned metrics	Networks trained on radiologist rankings	Tracks reader preference better than MSE/SSIM as an optimisation target ( $75.2\%\pm 1.3\%$ and $79.2\%\pm 1.7\%$ against readers’ own $61.5\%\pm 5.5\%$ ). Inherits training-reader variability, and was calibrated on synthetically corrupted data rather than clinically undersampled acquisitions, so transfer is assumed (Section IV-D )	Tang et al. [18] ( $n=18$ in corpus)
3. Uncertainty estimation	Posterior sampling / Bayesian variance $p(X\mid Y)$	Flags out-of-distribution inputs and regions of low confidence. Does not attribute missing structure to prior vs sampling operator; high confidence in hallucinated regions is possible	Khawaled and Freiman [19] ( $n=18$ in corpus)
4. Task-based observer assessment	Human reader performance on a diagnostic task; model observers are its automated form, unrepresented here [ 27 , 28 , 29 ]	Establishes diagnostic sufficiency directly; the gold standard in assessment theory. Costly and reader-variable ( $61.5\%\pm 5.5\%$ , falling to $25\%$ under absolute Likert scoring), and rarely paired with a metric: only $18/263$ report both	Radmanesh et al. [9] , Lang et al. [20] ( $n=57$ in corpus)
5. Measurement-space residual check	$\\|\mathbf{A}\hat{X}-Y\\|_{2}/\\|Y\\|_{2}$ against tolerance	Cannot detect prior-induced edits : the residual is invariant to anything in $\mathcal{N}(\mathbf{A})$ . A two-sided measurement-fidelity band, not a safety test	Common diagnostic check in the reviewed studies
6. Null-space / hallucination mapping	Structure written into the null-space component $(\mathbf{I}-\mathbf{A}^{+}\mathbf{A})\hat{X}$	Isolates prior-supplied content, where erasure and hallucination must reside. Requires $\mathbf{A}$ , hence raw $k$ -space and coil sensitivities	Bhadra et al. [31] , Gottschling et al. [11] ; formalization for reconstruction auditing is author-derived

TABLE X: What each evaluation paradigm can and cannot establish about prior-induced diagnostic error (RQ3). Rows 1–4 summarize evaluation practice and performance as reported in the included studies; quoted values are representative results from the cited primary studies, not pooled estimates. Rows 5–6 state analytical limits derived by the authors from the acquisition physics (our own analysis; Section IV ) and are labelled as such rather than attributed to any primary study.

Fig. 3: Reader assessment against hallucination awareness, by method family (recorded extraction labels; lower bounds). Bars are the share of each family’s studies that record a hallucination or instability failure mode (orange) and the share that carry any reader assessment (blue), using the verified reader subset (

n=57

). Bar labels are rounded to the nearest integer; the one-decimal values quoted in the text are the same quantities (for example

15

against

15.1\%

39

against

38.8\%

). Generated from the released per-study file ( included_characteristics.csv ) by 5_Code/make_figures.py ; every value is recomputable from the deposit.

Fig. 4: Evaluation practice across publication eras. Grey bars give the number of included studies per era (34, 55, 174); lines give the share with any reader assessment and the share co-reporting a fidelity metric and a reader assessment on the same data, using the verified reader subset (

n=57

). What recovery there is in 2023–2026 comes from subscription-walled clinical journals (

52.0\%

reader assessment in that stratum against

15.4\%

in the open remainder; Section V-B ). Generated from the released per-study file by 5_Code/make_figures.py ; counts are lower bounds from recorded extraction labels.

Method family (multi-label)
Era	n	Reader	Uncert.	Co-report
1995–2019	34	$32.4\%$ (11/34)	$5.9\%$	$5.9\%$
2020–2022	55	$18.2\%$ (10/55)	$5.5\%$	$7.3\%$
2023–2026	174	$20.7\%$ (36/174)	$7.5\%$	$6.9\%$
Family		n	Reader share
Compressed sensing / parallel imaging		93	$25.8\%$ (24/93)

TABLE XI: Evaluation practice across publication eras and method families (recorded labels; lower bounds). Reader-assessment share fell as the corpus grew and recovers only partially in the latest era, and that recovery is carried by subscription-walled clinical journals. Among the four largest method families the generative group, the one most associated with hallucination, is one of the two least reader-evaluated and is indistinguishable from physics-based unrolled work; the small self-supervised family (

n=17

) records no reader assessment at all. Reader shares use the verified reader subset (

n=57

; Section III ), so counts sum to 57 across the era rows and are multi-label across the family rows.

Medical Imaging

Medical imaging uses computational methods to analyze, reconstruct, and interpret images for diagnosis, treatment planning, and monitoring.

See all 251 papers

Feb 20, 2026

Triggering hallucinations in model-based MRI reconstruction via adversarial perturbations

Generative models are increasingly used to improve the quality of medical imaging, such as reconstruction of magnetic resonance images and computed tomography. However, it is well-known that such models are susceptible to hallucinations: they may insert features into the reconstructed image which are not actually present in the original image. In a medical setting, such hallucinations may endanger patient health as they can lead to incorrect diagnoses. In this work, we aim to quantify the extent to which state-of-the-art generative models suffer from hallucinations in the context of magnetic resonance image reconstruction. Specifically, we craft adversarial perturbations resembling random noise for the unprocessed input images which induce hallucinations when reconstructed using a generative model. We perform this evaluation on the brain and knee images from the fastMRI data set using UNet and end-to-end VarNet architectures to reconstruct the images. Our results show that these models are highly susceptible to small perturbations and can be easily coaxed into producing hallucinations. This fragility may partially explain why hallucinations occur in the first place and suggests that a carefully constructed adversarial training routine may reduce their prevalence. Moreover, these hallucinations cannot be reliably detected using traditional image quality metrics. Novel approaches will therefore need to be developed to detect when hallucinations have occurred.

Suna Buğday, Yvan Saeys, Jonathan Peck

Faculty of Engineering and Architecture/Computer Engineering, İzmir Bakırçay University, İzmir, Turkey · Department of Computer Science, Mathematics and Statistics, Ghent University, Ghent, Belgium · Data Mining and Modeling for Biomedicine Group, VIB Inflammation Research Center, Ghent, Belgium

Adversarial Examples

Ask PDF

Aug 3, 2026

Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI

Vision-language models (VLMs), including medical specialists, are increasingly proposed for medical imaging, yet their stated confidence is rarely evaluated separately from correctness. We use brain MRI as a controlled, high-stakes testbed for a broader failure mode in frontier multimodal systems: models can appear competent while lacking reliable self-knowledge. We present an automatically graded behavioral audit and pilot study of six instruction-tuned VLMs (five general-purpose and one medical specialist) on 4,102 images (4,032 axial/coronal/sagittal MRI slices from 250 subjects plus 70 non-brain/noise controls), with labels derived from public metadata and released expert segmentation masks rather than new human annotation. Across models, answer coverage is near-complete, but verbalized-confidence calibration is poor: ECE ranges from 0.27 to 0.40, mean confidence on incorrect answers ranges from 0.82 to 0.97, and 33-46% of answered items are high-confidence errors. The most accurate model is also the most confident on its errors, while a base/specialist family contrast suggests that medical adaptation improves tumor-presence detection without improving confidence reliability. Open-ended diagnostics further show that hallucination and abstention vary separately from multiple-choice accuracy. These findings argue that medical-image VLM evaluation should report verbalized-confidence reliability, confident error, hallucination, and abstention alongside accuracy.

Amir Sabbaghziarani, Mohammadsajad Abavisani, Sergey Plis

TReNDS Center Atlanta, GA, USA

VLM Evaluation VLM Hallucination Medical VLMs VLM Calibration

Ask PDF

May 10, 2026

Rethinking Evaluation of Multiple Sclerosis (MS) Lesion Segmentation Models

Multiple Sclerosis (MS) is a chronic autoimmune disease that can significantly reduce the quality of life of a patient. Existing treatment options can only help slow down the progression of the disease. Therefore, early detection and precise monitoring of disease progression are important. Deep learning offers state-of-the-art models for detecting and segmenting MS lesions in brain MRI scans. However, most of these models are evaluated using the Dice score, without accounting for lesion-wise detection and segmentation performance or other metrics that quantify model performance in cases that are complex or confusing for human annotators, or in cases that are essential for disease detection and progression monitoring. In this paper, we highlight the need to rethink the evaluation of MS lesion segmentation models. In this context, we first present problem fingerprinting in detail to highlight what neurologists look for in brain MRI scans for MS detection and progression monitoring, and which metrics are required to properly quantify model performance in these contexts. Additionally, we present an analysis of state-of-the-art models on two open-source datasets using these metrics to highlight their usability for real-world deployment in hospitals.

Abdul Basit, Ashir Rashid, Muhammad Abdullah Hanif +1

eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi, Abu Dhabi, UAE

Medical Image Segmentation Brain MRI

Ask PDF

Abstract

Medical Imaging

Explore similar work

Triggering hallucinations in model-based MRI reconstruction via adversarial perturbations

Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI

Rethinking Evaluation of Multiple Sclerosis (MS) Lesion Segmentation Models