Rethinking Next-Visit Diagnosis Prediction Beyond Aggregate Metrics: Retrieval--Memory Multi-Agent Adjudication
Organizations: University of Illinois Urbana-Champaign
Abstract
Next-visit diagnosis prediction forecasts the diagnoses documented at a patient's next encounter, turning retrospective electronic health records into prospective clinical prediction. However, conventional aggregate evaluation obscures its multifaceted nature: distributionally, models must handle variable-size diagnosis sets and long-tailed labels; temporally, they must distinguish persistent, newly recorded, and recurrent diagnoses; and both vary with observed history depth. We introduce an evaluation framework along these dimensions and apply it to rule-based heuristics, LLM-based prompting, retrieval, memory, and multi-agent approaches. This reveals that repeating the last visit outperforms most LLM-based approaches overall despite predicting no new or recurrent diagnoses, and that retrieval and memory exhibit distinct profiles whose relative strengths vary across temporal transitions and backbones. Motivated by this, we propose LONGINUS, a retrieval-memory multi-agent adjudication framework combining evidence retrieved from patient records with structured memory of the patient trajectory. Using cross-view agreement as a reliability heuristic, it preserves shared predictions and adjudicates disagreements without revealing their source. Across MIMIC-III and MIMIC-IV with four LLM backbones, LONGINUS achieves the best transition-specific performance among LLM-based methods in nearly all settings while remaining competitive with them on aggregate metrics.
Figures & tables
| Prior Work | Overall | Distribution | Temporal Transitions | History Depth | |||
| Agg. | Co | LT | Pers. | New | Rec. | ||
| GRAM Choi et al. (2017) | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✘ |
| TCoN Sun et al. (2021) | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ |
| NECHO Koo (2024) | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| TAKECare Wu et al. (2025a) | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ |
| RAVEN Rajamohan et al. (2026) | ✔ | ✘ | ✘ | ✘ | ✔ | ✘ | ✘ |
| Dataset | History | Patients (%) | New | Persistent | Recurrent |
| MIMIC-III | Visit 1 | 78.6 | 45.64 | 54.36 | – |
| Visit 2+ | 21.4 | 27.52 | 59.31 | 13.17 | |
| MIMIC-IV | Visit 1 | 74.4 | 41.37 | 58.63 | – |
| Visit 2+ | 25.6 | 25.05 | 59.14 | 15.81 |
| Backbone | Method | Visit 1 | Visit 2+ | |||||
| Agg. | Temp. | Agg. | Temp. | |||||
| Mic. | Pers. | New | Mic. | Pers. | New | Rec. | ||
| Rule | LVCF | .582 | .732 | .000 | .586 | .734 | .000 | .000 |
| OHU | .582 | .732 | .000 | .582 | .734 | .000 | .444 | |
| Gemini-2.5 -Flash-Lite | NECHOv3 | .493 | .722 | .203 | .521 | .741 | .148 | .484 |
| EHR-RAG | .564 | .731 | .247 | .574 | .737 | .157 | .437 | |
| Backbone | Type | Method | MIMIC-III | MIMIC-IV | ||||||||||
| Agg. | Dist. | Temp. | Agg. | Dist. | Temp. | |||||||||
| Mic. | Mac. | Jac. | Pers. | New | Rec. | Mic. | Mac. | Jac. | Pers. | New | Rec. | |||
| - | Rule | LVCF | .558 | .510 | .397 | .719 | .000 | .000 | .582 | .479 | .439 | .732 | .000 | .000 |
| OHU | .553 | .510 | .391 | .719 | .000 | .383 | .582 | .492 | .436 | .732 | .000 | .444 | ||
| Gemini-3.1 -Flash-Lite | Single-Prompting | NECHOv3 | .514 | .450 | .353 | .715 | .194 | .393 | .552 | .459 | .392 | .756 | .236 | .453 |
| Multi-agent | MedAgents | .506 | .407 | .347 | .695 | .217 | .383 | .547 | .444 | .390 | .736 | .264 | .416 | |
| Backbone | Type | Method | Visit 1 | Visit 2+ | |||||||||
| Agg. | Dist. | Temp. | Agg. | Dist. | Temp. | ||||||||
| Mic. | Mac. | Jac. | Pers. | New | Mic. | Mac. | Jac. | Pers. | New | Rec. | |||
| - | Rule | LVCF | .582 | .470 | .440 | .732 | .000 | .586 | .402 | .437 | .734 | .000 | .000 |
| OHU | .582 | .470 | .440 | .732 | .000 | .582 | .455 | .426 | .734 | .000 | .444 | ||
| Gemini-3.1 -Flash-Lite | Single-Prompting | NECHOv3 | .552 | .435 | .391 | .752 | .254 | .552 | .411 | .396 | .768 | .153 | .453 |
| Multi-agent | MedAgents | .552 | .426 | .397 | .736 | .279 | .532 | .388 | .372 | .734 | .195 | .416 | |
| Component | Variant | Agg. | Dist. | Temp. | |||
| Mic. | Mac. | Jac. | Pers. | New | Rec. | ||
| Gemini-3.1-Flash-Lite | |||||||
| Full | LONGINUS | .583 | .490 | .418 | .773 | .314 | .529 |
| Retrieval | w/ SQ | .585 | .495 | .421 | .774 | .313 | .520 |
| w/o Current | .582 | .488 | .419 | .772 | .313 | .524 | |
| w/o Past-only | .584 | .489 | .420 | .773 | .315 | .527 | |
| Type | Variant | Agg. | Dist. | Temp. | |||
| Mic. | Mac. | Jac. | Pers. | New | Rec. | ||
| Gemini-3.1-Flash-Lite | |||||||
| Single view | Retrieval | .578 | .489 | .411 | .753 † | .308 | .483 † |
| Memory | .556 † | .457 † | .389 † | .767 | .296 † | .529 | |
| No adjud. | Union | .552 † | .472 † | .382 † | .755 † | .300 † | .490 |
| Intersection | .587 | .474 | .426 | .765 | .303 | .523 | |
| Stage | Measure (%) | Gemini-3.1 -Flash-Lite | Llama-3.3 -70B |
| Candidate precision | Consensus | 54.9 | 66.7 |
| Disagreement (before) | 21.6 | 38.8 | |
| Disagreement (after) | 27.8 | 49.2 | |
| Within disagreement | TP retained | 64.1 | 82.2 |
| FP rejected | 54.3 | 46.3 | |
| Final FN share | Missed by both | 83.1 | 89.3 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | MIMIC-III | MIMIC-IV |
| Patients | 1,000 | 1,000 |
| Observed visit rows | 1,308 | 1,453 |
| Mean target labels / patient | 7.745 | 6.888 |
| Parent Category | Child Diagnosis Label | MIMIC-III | MIMIC-IV | ||
|---|---|---|---|---|---|
| Count | Dist. (%) | Count | Dist. (%) | ||
| Diseases of the blood and blood-forming organs | |||||
| Acute post-hemorrhagic anemia | 94 | 1.21 | 63 | 0.91 | |
| Anemia | 301 | 3.89 | 292 | 4.24 | |
| Coagulation and hemorrhagic disorders | 137 | 1.77 | 126 | 1.83 | |
| White blood cell disorders | 49 | 0.63 | 87 | 1.26 | |
| Type | Method | LLM calls per patient |
| Single-Prompting | NECHOv3 | |
| Multi-agent | MedAgents | |
| MDAgents | ||
| KAMAC | ||
| Retrieval | Summary-RAG | |
| RGAR |
| Method | Gemini-2.5 -Flash-Lite | Gemini-3.1 -Flash-Lite | GPT-OSS -120B | Llama-3.3 -70B | ||||
| In | Out | In | Out | In | Out | In | Out | |
| MIMIC-III | ||||||||
| KAMAC | 49.87 | 3.35 | 50.35 | 1.52 | 61.32 | 9.22 | 62.83 | 1.46 |
| MedAgents | 40.47 | 3.95 | 45.37 | 1.48 | 46.72 | 5.74 | 44.96 | 1.75 |
| MDAgents | 65.97 | 4.09 | 71.38 | 1.18 | 67.44 | 5.10 | 73.68 | 0.55 |
| Traj-CoA | 14.33 | 2.02 | 13.38 | 0.90 | 12.39 | 3.53 | 12.76 | 0.83 |
| Backbone | Type | Method | MIMIC-III | MIMIC-IV | ||||||||||
| Agg. | Dist. | Temp. | Agg. | Dist. | Temp. | |||||||||
| Mic. | Mac. | Jac. | Pers. | New | Rec. | Mic. | Mac. | Jac. | Pers. | New | Rec. | |||
| - | Rule | LVCF | .558 | .510 | .397 | .719 | .000 | .000 | .582 | .479 | .439 | .732 | .000 | .000 |
| OHU | .553 | .510 | .391 | .719 | .000 | .383 | .582 | .492 | .436 | .732 | .000 | .444 | ||
| Gemini-2.5 -Flash-Lite | Single-Prompting | NECHOv3 | .473 | .404 | .319 | .693 | .172 | .391 | .500 | .440 | .353 | .726 | .193 | .484 |
| Multi-agent | MedAgents | .436 | .338 | .285 | .606 | .176 | .314 | .484 | .383 | .336 | .649 | .227 | .400 | |
| Backbone | Type | Method | Visit 1 | Visit 2+ | |||||||||
| Agg. | Dist. | Temp. | Agg. | Dist. | Temp. | ||||||||
| Mic. | Mac. | Jac. | Pers. | New | Mic. | Mac. | Jac. | Pers. | New | Rec. | |||
| - | Rule | LVCF | .548 | .489 | .389 | .712 | .000 | .591 | .467 | .426 | .741 | .000 | .000 |
| OHU | .548 | .489 | .389 | .712 | .000 | .566 | .467 | .397 | .741 | .000 | .383 | ||
| Gemini-2.5 -Flash-Lite | Single-Prompting | NECHOv3 | .461 | .391 | .309 | .684 | .179 | .508 | .384 | .355 | .717 | .140 | .391 |
| Multi-agent | MedAgents | .446 | .345 | .294 | .623 | .185 | .404 | .269 | .251 | .549 | .133 | .314 | |
| Backbone | Type | Method | Visit 1 | Visit 2+ | |||||||||
| Agg. | Dist. | Temp. | Agg. | Dist. | Temp. | ||||||||
| Mic. | Mac. | Jac. | Pers. | New | Mic. | Mac. | Jac. | Pers. | New | Rec. | |||
| - | Rule | LVCF | .582 | .470 | .440 | .732 | .000 | .586 | .402 | .437 | .734 | .000 | .000 |
| OHU | .582 | .470 | .440 | .732 | .000 | .582 | .455 | .426 | .734 | .000 | .444 | ||
| Gemini-2.5 -Flash-Lite | Single-Prompting | NECHOv3 | .493 | .409 | .344 | .722 | .203 | .521 | .415 | .377 | .741 | .148 | .484 |
| Multi-agent | MedAgents | .489 | .374 | .340 | .656 | .241 | .465 | .321 | .323 | .627 | .156 | .400 | |
| Backbone | Type | Method | MIMIC-III | MIMIC-IV | ||||||
| Head | Mid | Tail | All | Head | Mid | Tail | All | |||
| - | Rule | LVCF | .562 | .433 | .536 | .510 | .599 | .472 | .434 | .479 |
| OHU | .566 | .428 | .537 | .510 | .606 | .476 | .454 | .492 | ||
| Gemini-3.1 -Flash-Lite | Single-Prompting | NECHOv3 | .511 | .383 | .465 | .450 | .574 | .430 | .428 | .459 |
| Multi-agent | MedAgents | .503 | .359 | .397 | .407 | .570 | .418 | .408 | .444 | |
| MDAgents | .514 | .379 | .453 | .443 | .574 | .438 | .454 | .474 | ||
| Scope | Stage | Gemini-3.1 -Flash-Lite | Llama-3.3 -70B | ||
| P | R | P | R | ||
| Subset | Consensus | 54.9 | 63.1 | 66.7 | 39.9 |
| Disagreement (before) | 21.6 | 13.3 | 38.8 | 24.1 | |
| Disagreement (after) | 27.8 | 8.6 | 49.2 | 19.8 | |
| Overall | Raw candidate union | 43.2 | 76.4 | 52.4 | 64.1 |
| Final prediction | 49.2 | 71.6 | 59.7 | 59.8 | |
| Backbone | Transition | Precision | Recall | ||
| Union | Final | Union | Final | ||
| Gemini-3.1-Flash-Lite | Persistent | 62.5 | 66.8 | 95.3 | 91.7 |
| New | 22.2 | 25.8 | 46.2 | 40.0 | |
| Recurrent | 34.3 | 40.6 | 85.7 | 76.1 | |
| Llama-3.3-70B | Persistent | 66.0 | 68.1 | 88.1 | 85.8 |
| New | 26.8 | 34.3 | 26.7 | 20.0 | |
| Patient A: MIMIC-III, Gemini-2.5-Flash-Lite; one observed visit, predicting Visit 2. |
| Patient summary. A male patient aged 76+ recovering from CABG, prolonged ventilation, and dialysis-requiring renal disease was admitted with fever, jaundice, biliary obstruction, and pleural effusion. ERCP was followed by pancreatitis. His observed medical history explicitly lists peripheral vascular disease and prostate cancer treated with radiotherapy, although neither appeared in the observed diagnosis-code set. |
| Reference diagnoses (11). Persistent (7): Adult respiratory failure; Anemia; Biliary disease; Cardiac dysrhythmia; Coronary atherosclerosis; Hypertension complications; Thyroid disorders. New (4): Fluid and electrolyte disorders; Peripheral atherosclerosis; Prostate cancer; Substance-related disorders. |
| Patient B: MIMIC-IV, Gemini-3.1-Flash-Lite; two observed visits, predicting Visit 3. |
| Patient summary. A male patient aged 41–60 with ulcerative colitis, prior bowel perforation, and a ventral hernia underwent staged abdominal-wall reconstruction. Approximately 69 days after the first discharge, he underwent proctectomy, ileal pouch construction, diverting ileostomy, and hernia repair. The observed notes describe recurrent C. difficile colitis, although Intestinal infection was absent from the observed coding history. |
| Reference diagnoses (6). Persistent (2): Hypertension; Ulcerative colitis. Recurrent (1): Abdominal hernia. New (3): Anemia; Cardiac dysrhythmia; Intestinal infection. |
| Patient C: MIMIC-III, GPT-OSS-120B; two observed visits, predicting Visit 3. |
| Patient summary. A male patient aged 61–75 with lung squamous-cell carcinoma following right lower lobectomy, bronchopleural fistula/empyema, invasive aspergillosis, HIV, cardiac comorbidities, anemia, and malnutrition. The second admission involved hypoxemic respiratory failure and septic shock attributed to pneumonia, requiring prolonged ventilation, tracheostomy, and gastrostomy. |
| Reference diagnoses (12). Persistent (10): Adult respiratory failure; Congestive heart failure, non-hypertensive; Coronary atherosclerosis; Fluid and electrolyte disorders; HIV infection; Bronchus/lung cancer; Mycoses; Septicemia; Shock; Pleurisy. Recurrent (1): Anemia. New (1): Aspiration pneumonia. |
| Patient D: MIMIC-IV, Llama-3.3-70B; four observed visits, predicting Visit 5. |
| Patient summary. A male patient aged 76+ with insulin-treated diabetes, severe peripheral vascular disease, coronary artery disease after CABG, and hypothyroidism had repeated vascular admissions. Earlier visits involved ischemic foot disease, infection, revascularization, bypass-graft infection and removal, non-healing ulcers, and bypass revision. The fourth observed admission concerned left-leg ischemia and planned bypass surgery. Gangrene was coded in Visit 3 but not Visit 4. |
| Reference diagnoses (6). Persistent (5): Conduction disorders; Coronary atherosclerosis; Diabetes mellitus without complications; Skin ulcer; Thyroid disorders. Recurrent (1): Gangrene. New (0). |
| Temporal and tail signal. Gangrene coding across observed Visits 1–4: 0, 0, 1, 0; target: 1. Gangrene is a Tail label. These states denote diagnosis coding, not necessarily clinical resolution or reactivation. |
Clinical Prediction
Clinical prediction uses patient records, signals, and images to estimate diagnoses, risks, outcomes, and treatment responses for care.
See all 228 papers