cs.AIOct 23, 2026
SaveMedCaseAgent: From Multimodal Clinical Evidence to Rare Disease Case Reports
Organizations: PediaMed AI · University of Chinese Academy of Sciences · University of Illinois Urbana-Champaign · Carnegie Mellon University ·
Abstract
Although Large Language Models (LLMs) have demonstrated potential in clinical documentation, generating the rigorous, evidence-based reasoning required for unprecedented, atypical, and rare disease case reports remains a significant challenge. We present MedCaseAgent, an end-to-end agentic reasoning paradigm that bridges this gap by synthesizing publication-quality reports from unstructured clinical notes. MedCaseAgent leverages a hierarchical structure, integrating patient-level data mining with iterative, tool-integrated reasoning. By dynamically querying medical literature and knowledge bases, the agent maintains high clinical fidelity and assesses case novelty. The framework is inherently modular, supporting various backbone LLMs in a plug-and-play fashion. To facilitate this research, we release MedCase-150K, a large-scale multimodal clinical case-report corpus, and MedCase-Bench, a robust temporally held-out evaluation suite. We also introduce an open-source model post-trained from Qwen3.5-9B and optimized specifically for this task. Under both LLM-judged and rule-based evaluation, MedCaseAgent outperforms existing general-purpose and specialized medical LLMs and agents. Blinded expert reviews and ablations further support the system's clinical relevance and evidence grounding.
Figures & tables
Figure 1: Overview of the MedCaseAgent report-generation workflow.
Figure 2: Illustration of fragmented first-hand clinical observations used as multimodal input to MedCaseAgent. The notes are patient-specific and incomplete, resembling the partial facts available before a formal academic case report is written.
Figure 3: Planner phase from initial multimodal evidence to a generated case-report plan.
Figure 4: Training phase for the MedCaseAgent planner.
Figure 5: MedCaseAgent configuration comparison on MedCase-Bench across five rubric metrics and their average. Each legend entry is a MedCaseAgent variant, with the centered LLM shown in parentheses. Bars show means; error bars show standard deviations.
| Method | Centered LLM | COVID-19 | Small-Cell Lung Cancer | Erdheim-Chester Disease | Rare Genetic Disorder | Ultra-Rare Disease | Average |
| Human Publication | - | 82.96 | 76.84 | 73.75 | 73.28 | 71.07 | 75.58 |
| Qwen3.5-9B ( Yang et al., 2025 ) | - | 62.46 | 55.84 | 54.35 | 53.44 | 53.79 | 55.98 |
| Qwen3.5-397B-A17B | - | 78.28 | 74.64 | 66.21 | 66.15 | 64.11 | 69.88 |
| Gemma-4-31B | - | 72.34 | 69.26 | 64.25 | 63.03 | 62.45 | 66.27 |
| Claude Opus 4.6 | - | 82.05 | 75.18 | 74.62 | 73.58 | 67.75 | 74.64 |
| GPT-5.4 | - | 84.41 | 79.41 | 74.34 | 70.65 | 73.36 | 76.43 |
Table 1: Disease-subset evaluation on MedCase-150K, comparing tool-equipped general MLLMs and medical agentic systems across selected disease categories. Each disease column samples 100 cases, except Erdheim-Chester disease (ECD), where all 53 available MedCase-150K cases are used. Each score averages citation depth, patient-history nuance, clinical coherence, image integration, and readability.
| Method | Exact DOI | Novel Rec. | Fig. Seq. | Auth. Indist. | P0 Recall | P1 Recall |
| Qwen3.5-9B | 22.00 | 38.12 | 54.75 | 31.95 | 62.63 | 31.54 |
| Qwen3.5-397B-A17B | 36.27 | 57.39 | 66.93 | 45.53 | 66.70 | 35.32 |
| Gemma-4-31B | 31.11 | 52.88 | 63.36 | 38.92 | 65.87 | 34.87 |
| Claude Opus 4.6 | 43.22 | 66.75 | 73.45 | 53.79 | 68.60 | 35.62 |
| GPT-5.4 | 46.90 | 69.01 | 75.92 | 50.89 | 69.15 | 36.13 |
| Gemini 3.1 Pro | 44.50 | 68.01 | 76.56 | 52.76 | 68.83 | 35.59 |
Table 2: Comparison of rule-based and structural evaluation metrics across different models. All entries are percentages. Abbreviations: Exact DOI Retrieval (Exact DOI), Novelty Recognition (Novel Rec.), Figure Sequence Accuracy (Fig. Seq.), Authorship Indistinguishability (Auth. Indist.), and citation recall for must-cite (P0) and good-to-cite (P1) references.
| Report | Quality | Timeline | Omission control |
| Raw records | 7.4 | 7.5 | 7.8 |
| Atoms | 7.6 | 7.6 | 7.6 |
| Clinician | 7.5 | 7.8 | 7.9 |
Table 4: Mean expert scores (1–10; ).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B1 : Clinical atom compression example. A polished source case report is compressed into fragmented, patient-specific atoms that resemble real-world shorthand notes. The prompt-output view illustrates how background prose and final diagnostic exposition are removed while preserving clinical history, findings, tests, interventions, outcomes, and image-relevant details.
Figure B2 : Detailed view of the MedCaseAgent planning phase. The planner organizes fragmented multimodal case evidence, retrieves relevant medical context, and produces a structured plan that guides downstream report writing and editing.
Figure B3 : Prompt stack used by MedCaseAgent across the agentic report-generation workflow. The stack separates role instructions, case-specific evidence, retrieved context, output constraints, and audit requirements so that each phase receives grounded inputs for clinical case report synthesis.
| Review stage | Reviewed ( ) | Excluded (%) | Retained (%) | Pre-adjudication agreement |
| Clinical-atom fidelity | 632 | 30 (4.75%) | 602 (95.25%) | 94% |
| Rarity/novelty label | 602 | 18 (2.99%) | 584 (97.01%) | 87% |
Table C1 : Sequential human verification of clinical atoms and rarity/novelty labels during MedCase-Bench construction. Cases failing either stage were excluded rather than edited or relabeled; exclusion and retention percentages use the number reviewed at that stage. Each case was reviewed by at least two reviewers. Agreement is measured before adjudication.
Figure C1 : Rarity and novelty stratification for the curated case-report corpus. Bars summarize rarity labels assigned during data construction, separating rare diseases in the strict epidemiological sense, unprecedented presentations, atypical presentations, and relatively standard cases.
Figure C2 : Temporal coverage of the MedCase-150K case-report corpus. Bars show annual case-report counts and the line shows cumulative coverage.
Figure C3 : Representative visual complexity in MedCase-Bench. Rows group examples by imaging or figure modality, illustrating the breadth of radiology, clinical photography, ophthalmology, dentistry, pathology, and structured graphical evidence that the benchmark requires multimodal agents to interpret.
Figure C4 : Source breadth across journals. The left panel reports the leading source venues, while the right panel shows that coverage remains broadly distributed across the long tail of journals.
| Raw documentation | Corresponding report-derived atom | Information compressed |
| Dated entries separately record fever, marrow findings, impaired consciousness, visual loss, and later motor milestones. | Visual dysfunction persisting for over four months after EBV-associated encephalitis. | The central presentation is retained, while earlier events, care transitions, and temporal context are compressed. |
| Visual findings occur in narrative text and photographed reports. Several BAEP, P-VEP, and OCT screenshots contain demographic details inconsistent with the index patient. | P-VEP showed bilateral NPN waveforms with prolonged latencies; fundoscopy showed no obvious structural abnormality. | Summarized findings are retained, while provenance and possible wrong-patient contamination may be lost. |
| A summary specifies occipital rTMS at 90–100% motor threshold and 5–6 Hz. The contemporaneous plan records 20% intensity, a 25% threshold, 5 Hz, and 750 stimuli. | Occipital rTMS at 90–100% resting motor threshold, 5–6 Hz, and 20 minutes per day. | A single-version atom may obscure conflicting treatment parameters across sources. |
| Left/right optic-radiation streamline counts are 20/496, 314/21, and 483/10 across scans. The final diffusion scan changes from 3 T to 7 T. | Left optic-radiation fiber count increased, whereas the right decreased. | A direction-only atom may omit acquisition differences between scans. |
Table F1 : Same-patient example of atomization for visual dysfunction following encephalitis. The atoms were derived from a clinician-written manuscript, not directly from the original clinical files. The abbreviated comparisons show source context compressed by the corresponding report-derived atoms.
| Evaluation dimension | Agreement | Reported correlation / exact agreement | Weighted | ICC |
| Factuality | 23/30 | 0.74 | 0.74 | |
| Clinical coherence | 24/30 | 0.76 | 0.77 | |
| Omission | 26/30 | 0.69 | 0.69 | |
| Hallucination | 30/30 | 1.00 | N/A | N/A |
| Citation accuracy | 24/30 | 0.73 | 0.74 |
Table F2 : LLM–clinician agreement in the initial paired calibration (30 clinician–case assessments). Agreement on scored dimensions allows an absolute difference of at most two points on a 10-point scale. Hallucination uses binary pass/fail ratings and exact agreement.
Figure F1 : Expert web interface for the blinded A/B user study. Clinical reviewers inspect paired reports with shared medical images, guess which report was generated by an LLM, choose the stronger report overall, and may provide rubric-aligned scores and comments.
| Method | Identified as Generated (%) | Identified as Human (%) | Preference vs. Ground Truth (%) |
| Qwen3.5-9B | 86.53 | 13.47 | 24.84 |
| Gemma-4-31B | 82.31 | 17.69 | 30.58 |
| Claude Opus 4.6 | 72.18 | 27.82 | 45.28 |
| GPT-5.4 | 70.24 | 29.76 | 53.70 |
| Gemini 3.1 Pro | 71.07 | 28.93 | 50.48 |
| MedCaseAgent-Base | 79.67 | 20.33 | 36.29 |
Table G1 : LLM-judged blind A/B testing averaged across Gemini 3.1 Pro, GPT-5.4, Qwen3.5-397B-A17B, and Claude Opus 4.6 judges. All entries are reported as percentages.
| System | Qwen | Gemini | Claude | GPT | Average |
| Human Publication | 76.74 | 78.35 | 74.85 | 72.13 | 75.52 |
| GPT-5.4 | 77.65 | 78.32 | 75.84 | 74.78 | 76.65 |
| Gemini 3.1 Pro | 76.14 | 78.83 | 74.96 | 72.75 | 75.67 |
| Claude Opus 4.6 | 75.60 | 77.02 | 75.10 | 70.84 | 74.64 |
| MedCaseAgent-Base | 66.24 | 67.70 | 64.36 | 62.42 | 65.18 |
| MedCaseAgent-Max | 84.10 | 85.48 | 82.34 | 80.76 | 83.17 |
Table G2 : General-quality scores under individual judges and the four-judge average.
Figure H1 : Example 1 of a case report generated by MedCaseAgent. The full-page sheet shows the generated manuscript structure, image placement, citation formatting, and clinical timeline organization.
Figure H2 : Example 2 of a case report generated by MedCaseAgent. The full-page sheet shows the generated manuscript structure, image placement, citation formatting, and clinical timeline organization.
Figure H3 : Example 3 of a case report generated by MedCaseAgent. The full-page sheet shows the generated manuscript structure, image placement, citation formatting, and clinical timeline organization.
More in this topic
Medical Report Generation
Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.
See all 82 papers