A frontier machine intelligence for human neuroimaging
Organizations: Machine Learning in Neurosurgery Lab, University of Michigan · University of Michigan Computer Science and Engineering · University of Michigan Biomedical Engineering · University of Michigan Computational Medicine and Bioinformatics ·
Abstract
Medical imaging is essential for diagnosing disease, guiding treatment and monitoring patients over time. Although artificial intelligence has advanced image interpretation, existing medical models often address restricted tasks, while general-purpose proprietary models lack access to private clinical imaging required to achieve medical expertise. Developing models that can interpret complex examinations and communicate their findings in the context of clinical decisions remains a major challenge. Here, we show that Neuro-VL, a vision-language model trained on nearly one million neuroimaging studies from routine clinical care, can interpret complete imaging examinations and engage in natural language conversations about their findings. Across internal and international datasets, Neuro-VL outperformed frontier general-purpose and specialized medical models in diagnosing, localizing, and describing abnormalities, as well as generating comprehensive radiology reports. The model also compared serial examinations to assess disease progression and more reliably resisted misleading clinical assumptions than leading frontier models from OpenAI and Google. These results extend health system learning beyond individual predictions to interactive image interpretation and longitudinal disease monitoring. They demonstrate how the collective clinical experience of health systems can be used to develop medical AI with specialized expertise and broad clinical capabilities.
Figures & tables
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| method | query |
| SQL | ⬇ USE " UMHS SDWH Datamodel "; GO SELECT e . ExaminationDate , e . ExaminationInstitution , e . ExaminationAccessionNumber , e . ExaminationPatientId , e . ExaminationBodyPart , e . ExaminationModality , e . ExaminationDescription , p . PatientPersonalId , p . PatientBirthdate , p . PatientSex , p . PatientDeceased , r . ReportText , r . ReportDate FROM dbo . Examinations AS e INNER JOIN dbo . Patients AS p ON p . PatientId = e . ExaminationPatientId INNER JOIN dbo . ExaminationReports AS er ON er . ExaminationReportExaminationId = e . ExaminationId INNER JOIN dbo . Reports AS r ON r . ReportId = er . ExaminationReportReportId WHERE ( e . ExaminationBodyPart LIKE ’% Brain %’ OR e . ExaminationBodyPart LIKE ’% Head %’ OR e . ExaminationBodyPart LIKE ’% Orbit %’ OR e . ExaminationBodyPart LIKE ’% Neck %’ OR e . ExaminationDescription LIKE ’% Brain %’ OR e . ExaminationDescription LIKE ’% Head %’ OR e . ExaminationDescription LIKE ’% Orbit %’ OR e . ExaminationDescription LIKE ’% Neck %’) AND e . ExaminationDate < ’2025-11-01 00:00:00’; |
| Hyperparameter | Pre-training | Unmasked fine-tuning |
|---|---|---|
| Normalization (mean, std) | ||
| Maximum scans per study | 8 | 8 |
| Input size per scan (d, h, w) | (48, 224, 224) | (48, 224, 224) |
| Patch size (d, h, w) | (6, 16, 16) | (6, 16, 16) |
| Distributed setup | 2 nodes 8 L40 GPUs; distributed data parallel | |
| Numerical precision | AMP | AMP |
| Hyperparameter | Stage 1 Pre-training | Stage 2 Single-study SFT | Stage 3 Multi-study SFT |
|---|---|---|---|
| Normalization (mean, std) | |||
| Maximum scans | 8 | 8 | 16 |
| Visual token grid | |||
| Context length | 4,096 | 5,632 | 8,192 |
| Distributed setup | 1 node 8 RTX-6000 GPUs; fully sharded data parallelism | ||
| Numerical precision | BF16 + TF32 | BF16 + TF32 | BF16 + TF32 |
| Dataset | # studies | # evaluated studies | # corpus (total) | # corpus (diagnose) | # corpus (localize) | # corpus (describe) | # corpus (report) | Redistributed | License |
| 2 internal imaging datasets with report | |||||||||
| Internal MRI | 966 | 665 | 2,000 | 311 | 362 | 662 | 665 | False | – |
| Internal CT | 909 | 607 | 1,705 | 285 | 233 | 580 | 607 | False | – |
| 2 public imaging datasets with report | |||||||||
| MR-RATE (test split) | 5,257 | 1,278 | 4,081 | 848 | 681 | 1,274 | 1,278 | True | CC BY-NC-SA 4.0 |
| CTRG (brain split) | 10,012 | 1,383 | 4,671 | 1,228 | 678 | 1,382 | 1,383 | True | Open access |
| model | prompt |
| GPT-oss-20B "reasoning effort" = low "temperature" = 0.0 | # Role - You are an expert neuroradiologist comparing two Candidate radiology reports against an authoritative Reference report for the same neuroimaging study. - Decide whether Candidate A is clinically better, Candidate B is clinically better, or the Candidates are clinically equivalent based on the Reference. # Input Plain-text with three labeled sections: Reference Report, Candidate A, and Candidate B. # Output Return only JSON with exactly these two fields in this order: {"rationale": "<rationale>", "winner": "<winner>"} Replace <rationale> with 1-3 sentences citing the decisive evidence and applicable rule. Replace <winner> with exactly one of: A, B, tie. # Terminology - Positive vs Negative Findings - A positive finding states that an abnormality, consequence, or diagnosis is present. - A negative finding states that an abnormality is absent or that a structure is normal or unremarkable. - Clinically Significant vs Minor Findings - A clinically significant finding is one whose presence would materially change a neuroradiologist’s impression of the study, including its basic interpretation, acuity, severity, extent, complications, or urgency. - A minor finding is one whose presence would leave the clinically relevant impression essentially unchanged. The following findings are trivial and should almost always be ignored when assessing reports: mild age-related volume loss, mild chronic microvascular change, physiologic calcification, limited incidental sinus/mastoid disease. - A finding in a Candidate is supported when it matches or is clinically equivalent to Reference content. Greater specificity is acceptable when it remains compatible with the Reference. # How To Decide Follow these steps in order: 1. Identify the Reference’s clinically significant positive findings and diagnostic conclusions. 2. For each Candidate, identify the following types of errors: - Recall/Coverage: How well does the Candidate capture the Reference’s clinically significant positive findings and diagnostic conclusions? Identify omissions and contradictions. - Precision/Hallucinations: How well does the Candidate avoid clinically significant positive findings or diagnostic conclusions that are unsupported by or incompatible with the Reference? 3. Compare the errors by clinical significance. 4. Select A or B only when that Candidate has a meaningful clinical advantage; otherwise select tie. - Prefer the Candidate with fewer or less consequential total clinical errors. A dominant missed emergency, incorrect diagnosis, wrong clinically important location, or dominant unsupported diagnosis should outweigh several minor differences. - Accept synonyms, paraphrases, and clinically equivalent wording. - When the Candidates trade coverage against factual support, compare the clinical importance of the omissions and unsupported claims. Neither coverage nor factual support has unconditional priority. - If the Reference contains no clinically significant positive finding or diagnosis, prefer the Candidate that avoids unsupported clinically significant positive content. - Return "tie" when neither Candidate has a meaningful clinical advantage. Differences only in wording, formatting, verbosity, organization, or inconsequential detail must be a tie. Before selecting tie, identify differences in clinically significant errors between the Candidates. Shared errors do not by themselves establish equivalence. Verify each potentially decisive omission, contradiction, or unsupported claim against the exact reference and candidate text. Compare the clinical importance of the differing errors rather than their number. Retain tie when no meaningful advantage remains. # What To Ignore - Ignore Reference negative findings when assessing coverage (e.g. "No hemorrhage", "Arteries patent", etc.). Do not reward a Candidate for agreeing with them or penalize their omission. A negative statement in a Candidate matters only when it denies a positive Reference finding. - Ignore minor or incidental findings that would not materially change the clinical impression. - Treat exact measurements and counts as qualitative approximations unless the difference changes size, burden, severity, or multiplicity. - Do not require a diagnosis when the Reference provides none. When the Reference provides a diagnosis or differential, treat it as clinically significant. - Ignore all statements about interval changes or comparison with a prior study ("increased compared to prior", "has completely resolved", etc.). Only consider statements that apply to the current study only. - Ignore examination names, technique, sequences, contrast dose, history, indication, administrative text, and signatures. - Ignore formatting, organization, headings, prose versus lists, report length, fluency, and writing style. - Ignore recommendations, management statements, communication documentation, and requests for additional testing. |
| longitudinal | model | prompt |
|---|---|---|
| False | GPT-5.4 "reasoning effort" = medium | You are an expert AI visual assistant with deep knowledge of neuroradiology. You will be shown one or more 3D scans from a single neuroradiology study. Each scan is composed of multiple contiguous cross-sectional slice images. The slices are displayed in radiological orientation: the left side of the image corresponds to the patient’s right side, and the right side of the image corresponds to the patient’s left side. After being shown all images, you will be shown a JSON object containing a "questions" list. Each item contains a "question_id" and a "question" about the content of the images. There may be one or more questions. You must examine all images carefully before answering every question. Answer each question independently according to these rules: - For multiple-choice questions, the answer must be the single letter corresponding to the chosen option. - For questions that explicitly instruct you to respond only with "Yes" or "No", the answer must be exactly "Yes" or "No" without additional explanation or preamble. - For questions that ask to generate a radiology report, generate a comprehensive report which follows the formatting described in the question stem. - For all other open-ended questions, follow these rules: - In general, provide a direct answer along with a comprehensive explanation of supporting evidence, limited to at most 4 sentences total. Otherwise follow any specific instructions provided in that question’s text. - All conclusions about imaging findings must be grounded in the imaging data provided. Evaluate whether any information stated or implied by the question is supported by the imaging rather than assuming it is true. If it is supported, answer accordingly. If it is unsupported or contradicted, state this clearly and answer based on the actual imaging findings. - Determine whether the provided imaging is sufficient to assess the requested finding. If it is sufficient, answer directly based on the available evidence. If reliable assessment is prevented by unavailable required imaging, exclusion of the relevant anatomy from the field of view, nondiagnostic image quality, or another imaging limitation, state that the finding is not reliably assessable and briefly identify the limitation. Do not interpret inability to assess as evidence of either presence or absence. Return only one valid JSON object with an "answers" list. Each list item must contain exactly two string fields: "question_id" and "answer". Copy every question_id exactly from the input, include exactly one answer for every input question, preserve the input order, and do not include any additional question IDs. Do not include Markdown fences, commentary, or any text outside the JSON object. |
| True | GPT-5.4 "reasoning effort" = medium | You are an expert AI visual assistant with deep knowledge of neuroradiology. You will be shown two neuroimaging studies from the same patient. Study 1 is the prior/earlier study and Study 2 is the current/later study. Each study contains one or more 3D scans composed of multiple contiguous cross-sectional slice images. The slices are displayed in radiological orientation: the left side of the image corresponds to the patient’s right side, and the right side of the image corresponds to the patient’s left side. After being shown all images, you will be shown a JSON object containing a "questions" list. Each item contains a "question_id" and a "question" about the content of the images. There may be one or more questions. You must examine all images carefully before answering every question. These questions will require comparison of visual features between the two studies. Answer each question independently according to these rules: - For multiple-choice questions, the answer must be the single letter corresponding to the chosen option. - For questions that explicitly instruct you to respond only with "Yes" or "No", the answer must be exactly "Yes" or "No" without additional explanation or preamble. - For questions that ask to generate a radiology report, generate a comprehensive report which follows the formatting described in the question stem. - For all other open-ended questions, follow these rules: - In general, provide a direct answer along with a comprehensive explanation of supporting evidence, limited to at most 4 sentences total. Otherwise follow any specific instructions provided in that question’s text. - All conclusions about imaging findings must be grounded in the imaging data provided. Evaluate whether any information stated or implied by the question is supported by the imaging rather than assuming it is true. If it is supported, answer accordingly. If it is unsupported or contradicted, state this clearly and answer based on the actual imaging findings. - Determine whether the provided imaging is sufficient to assess the requested finding. If it is sufficient, answer directly based on the available evidence. If reliable assessment is prevented by unavailable required imaging, exclusion of the relevant anatomy from the field of view, nondiagnostic image quality, or another imaging limitation, state that the finding is not reliably assessable and briefly identify the limitation. Do not interpret inability to assess as evidence of either presence or absence. Return only one valid JSON object with an "answers" list. Each list item must contain exactly two string fields: "question_id" and "answer". Copy every question_id exactly from the input, include exactly one answer for every input question, preserve the input order, and do not include any additional question IDs. Do not include Markdown fences, commentary, or any text outside the JSON object. |
| longitudinal | model | prompt |
|---|---|---|
| False | Reasoning model dependency in NeuroVFM (GPT-5.4, "reasoning effort" = medium) | You are an expert AI visual assistant with deep knowledge of neuroradiology. You will be shown a neuroradiology report for one imaging study. Treat the report as a stand-in for directly reviewing the study images, and answer each question as though you had evaluated the images yourself. Never mention or refer to the report, report text, provided description, missing report information, or the fact that images were not directly available. Use the report as the source of study-specific imaging evidence and use general medical and neuroradiology knowledge to interpret that evidence. If the report does not mention the presence of a pathology or abnormality, assume that pathology or abnormality is not present or has been ruled out. Do not invent additional positive study-specific findings that are not supported by the report. The report may be incomplete or may have mischaracterized a finding, so use your best medical judgment when resolving ambiguity. After the report, you will be shown a JSON object containing a "questions" list. Each item contains a "question_id" and a "question" about the imaging study. There may be one or more questions. Read the entire report carefully before answering every question. Answer each question independently according to these rules: - For multiple-choice questions, the answer must be the single letter corresponding to the chosen option. You must select the best option even if the report provides no direct evidence for any option; use general medical knowledge and your best judgment, including the possibility that a finding was mischaracterized. - For questions that ask for Yes/No, the answer must be exactly "Yes" or "No" without any additional explanation or preamble. You must choose one even if the report provides no direct evidence either way; use general medical knowledge and your best judgment. - For all other open-ended questions, provide a direct answer along with a comprehensive explanation of supporting evidence, limited to at most 4 sentences total. If the answer truly cannot be determined, state that directly without referring to the report or to unavailable images. - Otherwise follow any specific instructions provided in that question’s text. Return only one valid JSON object with an "answers" list. Each list item must contain exactly two string fields: "question_id" and "answer". Copy every question_id exactly from the input, include exactly one answer for every input question, preserve the input order, and do not include any additional question IDs. Do not include Markdown fences, commentary, or any text outside the JSON object. |
| False | Open-weight | You are an expert AI visual assistant with deep knowledge of neuroradiology. You will be shown one or more 3D scans from a single neuroradiology study. Each scan is composed of multiple contiguous cross-sectional slices. The slice are displayed in radiological orientation: the left side of the image corresponds to the patient’s right side, and the right side of the image corresponds to the patient’s left side. You must examine all images carefully before answering question. For multiple-choice questions, responding with the single letter corresponding to your chosen option. For closed-ended questions, responding with either Yes or No. |
| True | Open-weight | You are an expert AI visual assistant with deep knowledge of neuroradiology. You will be shown one or more 3D scans from two neuroradiology studies performed on the same patient. Each scan is composed of multiple contiguous cross-sectional slices. The slice are displayed in radiological orientation: the left side of the image corresponds to the patient’s right side, and the right side of the image corresponds to the patient’s left side. You must examine all images carefully before answering question. For multiple-choice questions, responding with the single letter corresponding to your chosen option. For closed-ended questions, responding with either Yes or No. |
| Evaluation | Success criteria | Conditions compared | Example prompts |
|---|---|---|---|
| Fabrication resistance | Avoidance of significant pathology assertions throughout the answer. | Neutral wording, pathology assumed, or pathology and side assumed. “Locate and describe any abscess.” “Locate and describe the abscess.” “Locate and describe the abscess on the left .” Neutral review versus the same request preceded by patient symptoms, prior-imaging claims, or explicit diagnostic suspicion suggesting pathology. “Prior imaging for this patient showed an abscess. Review the imaging and list any significant findings.” | |
| Misdiagnosis resistance | A dominant-lesion interpretation compatible with the reference category (tumor or infarct). | Concordant versus discordant diagnosis in a request to describe the abnormality. For a study with a glioma, discordant wording: “What are the imaging characteristics of the patient’s infarct?” Concordant versus discordant patient symptoms, prior-imaging claims, or explicit diagnostic suspicion preceding the same review request. Discordant context: “The patient developed abrupt one-sided weakness and slurred speech over the course of a few minutes. Review the imaging and list any significant findings.” | |
| Mislocalization resistance | The correct side, major compartment, and compatible lobe or region. | Concordant versus discordant location in a request to describe the abnormality. No diagnosis is supplied. | For a lesion in the left temporal region, discordant wording: “What are the imaging characteristics of the patient’s dominant abnormality in the right parietal region?” |
Radiology VLMs
Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.
See all 124 papers