Case2Flow: Bridging Patient Cases and Guideline Flowcharts through Multimodal Retrieval
Authors: Jiale Wei, Yufan Chen, Alexander Jaus, Zdravko Marinov, Julian Friedrich, Simon Reiß, Jens Kleesiek, Rainer Stiefelhagen
Organizations: Karlsruhe Institute of Technology, Karlsruhe, Germany · Helmholtz Information and Data Science School for Health (HIDSS4Health), Karlsruhe/Heidelberg, Germany · University Hospital Essen, Essen, Germany
Medical guidelines encode rich, evidence-based decision logic, yet the specific decision artifact a clinician needs is hard to locate within a guideline, let alone across guidelines covering plausible diseases and treatments. While guideline passages have supported end-to-end question answering, flowcharts remain largely underused in decision support despite their ability to encode actionable clinical pathways. We therefore introduce Case2Flow, a task designed to retrieve the most relevant guideline flowchart for a given patient case from a collection of guideline documents. To support it, we construct FlowAtlas, a curated corpus of 202 flowcharts extracted from 2,080 medical guidelines, together with a pipeline that synthesises 1,911 aligned case-flowchart pairs. Our evaluation of multimodal retrieval methods reveals systematic failure modes, including overreliance on keywords and spurious token-patch matches induced by uninformative background regions in flowcharts. Motivated by this, we propose CRISP, a training-free scoring method that sharpens late-interaction retrieval by suppressing uninformative patches, discounting ambiguous token matches, and incorporating bidirectional query-image alignment. CRISP improves Recall@1 by up to 18.71 percentage points, while a blinded physician assessment on published case narratives provides preliminary feasibility evidence beyond synthetic queries.
Figures & tables
Figure 1: The task Case2Flow : given a textual patient case (history, symptoms, key findings), retrieve the most relevant guideline flowchart from a large cross-provider candidate pool. We build FlowAtlas from 2,080 guidelines by extracting and digitising 202 flowcharts and synthesising 1,911 aligned case-flowchart pairs through DFS path enumeration, case generation, and clinical rewriting.
Figure 2: CRISP sharpens late-interaction retrieval by reducing the influence of uninformative background patches, subtracting a soft uniform-match baseline from each query token to demote generic matches, and adding a reverse image-to-query coverage term. All three components are training-free and operate only at scoring time.
Method
Size
Type
CDC
WHO
ESMO
Onko.
NCCN
Mixed
RANDOM
–
–
7.69
1.92
1.12
2.08
1.82
0.39
Textual Pipeline-Based Flowchart Retrieval
Tesseract OCR + BM25
–
Original
83.82
79.55
77.65
88.70
50.59
62.55
Rewritten
75.00
72.16
68.94
64.81
N/A
58.55
VLM Captioning + BM25
2B
Original
89.71
84.66
80.56
86.12
71.28
71.83
Rewritten
83.82
76.42
75.00
67.38
N/A
68.62
Table 1: Recall@1 (%) across guideline sources and query types. Per-source : candidate pool from one provider. NCCN : external MedGUIDE Li et al. (2025b) flowcharts. Mixed : all 257 flowcharts pooled. MedGUIDE supplies its own case-chart pairs and has no rewriting pass, so the Rewritten row is not applicable (N/A) in the NCCN column. Best results per column in bold .
Mixed
NCCN
Method
orig
rew
orig
OCR + BGE-M3
56.74
54.23
57.45
OCR hybrid
70.25
66.90
68.41
Graph-text + BGE-M3
55.50
53.46
59.06
Graph-text hybrid
67.60
65.82
70.12
OCR hybrid → BGE-reranker
51.34
49.53
54.30
Table 2: Recall@1 (%) of stronger text, reranking, and fusion baselines on the full query sets. Hybrid : equal-weight fusion of BM25 and BGE-M3 scores after per-query z -normalisation; graph-text : the digitised flowchart graph serialised instead of OCR text.
ColPali
ColQwen3
Scoring variant
R@1
MRR
R@1
MRR
MaxSim (baseline)
54.67
66.91
79.91
87.55
+ PW
62.31
72.86
80.77
88.25
+ Adj
57.93
68.70
76.30
84.49
+ Bi
64.85
75.65
82.59
89.74
+ PW + Adj
59.27
69.69
75.24
84.12
Table 3: All combinations of the CRISP components on MedGUIDE ( 7,747 cases, 55 flowcharts; Recall@1 and MRR in %). PW : patch weighting; Adj : adjusted forward scoring; Bi : bidirectional coverage.
ColPali
ColQwen3
orig
rew
orig
rew
Per-query rank change vs. MaxSim (%)
Improved
37.4
39.9
14.5
15.2
Unchanged
57.1
53.8
81.4
80.2
Hurt
5.4
6.3
4.0
4.6
Recovered to rank 1 †
40.9
39.6
35.8
33.5
Table 4: Behavioural analysis of CRISP on the Mixed pool. Top : per-query rank change of the ground-truth flowchart relative to MaxSim; † Recovery is a fraction of the queries MaxSim ranks below top- 1 , not of all queries. Bottom : the top- 1 errors remaining under CRISP, classified by the deepest level at which the prediction matches the ground truth.
Top- 1 system
Accept. ( ≥2 )
Applic. ( =3 )
ColPali (MaxSim)
9/50 ( 18% )
6/50 ( 12% )
+ CRISP
13/50 ( 26% )
6/50 ( 12% )
ColQwen3 (MaxSim)
22/50 ( 44% )
14/50 ( 28% )
+ CRISP
24/50 ( 48% )
14/50 ( 28% )
Table 5: Blinded physician rating of the top- 1 chart on 50 published PMC-CaseReport narratives Wu et al. (2025) , scored 0 (irrelevant) to 3 (directly applicable).
Figure 3: Per-patch similarity heatmaps on a CDC flowchart (ColPali). Top: standard MaxSim assigns high scores almost uniformly. Bottom: CRISP suppresses background and highlights decision nodes.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Guide.
Flow.
Cases
Path
Words
Jacc.
(avg)
(o/r)
CDC
776
13
68
5.3
113/102
0.45
WHO
1,159
52
352
6.3
114/104
0.44
ESMO
91
89
792
5.8
103/103
0.44
Onkopedia
54
48
699
7.7
106/105
0.45
FlowAtlas
2,080
202
1,911
6.4
109/104
0.44
Appendix
Table 6: FlowAtlas dataset statistics per guideline source. Guide. : number of crawled guideline PDFs; Flow. : retained flowcharts; Cases : synthesised case-flowchart pairs; Path : mean number of decision nodes per case; Words : mean word count of (case + question) in original / rewritten mode; Jacc. : mean token-level Jaccard similarity between the two modes. MedGUIDE is included for the Mixed and external evaluations but is not part of FlowAtlas.
Figure 4: One representative flowchart from each of the five candidate pools (CDC, ESMO, Onkopedia, WHO and MedGUIDE).
CDC
WHO
ESMO
Method
R@1
R@3
R@5
R@10
MRR
R@1
R@3
R@5
R@10
MRR
R@1
R@3
R@5
R@10
MRR
Original queries
OCR + BM25
83.82
95.59
100.00
100.00
90.00
79.55
91.48
93.18
93.18
85.52
77.65
87.75
89.14
91.16
83.12
VLM Cap. + BM25
89.71
94.12
100.00
100.00
92.89
84.66
95.45
96.59
96.59
90.10
80.56
90.03
90.28
91.54
85.30
CLIP
45.59
83.82
92.65
98.53
65.68
50.57
76.42
88.92
95.74
65.94
17.68
26.89
32.20
43.06
26.08
Qwen3-VL-Emb
80.88
98.53
100.00
100.00
89.58
85.80
95.17
96.59
96.59
90.47
82.83
91.54
92.42
93.43
87.38
Appendix
Table 7: Per-source Recall@ {1,3,5,10} and MRR (%) on CDC, WHO and ESMO. Best score per column among the direct-image rows in bold . Continued in Table 8 .
Onkopedia
MedGUIDE
Method
R@1
R@3
R@5
R@10
MRR
R@1
R@3
R@5
R@10
MRR
Original queries
OCR + BM25
88.70
93.85
96.42
98.14
91.92
50.59
68.40
76.20
88.00
62.49
VLM Cap. + BM25
86.12
92.27
93.56
95.28
89.69
71.28
89.14
92.89
96.48
81.02
CLIP
13.16
21.89
28.18
37.63
22.29
18.86
35.52
46.44
62.68
32.46
Qwen3-VL-Emb
92.70
96.57
98.86
100.00
95.16
79.24
94.45
98.36
99.56
87.10
Appendix
Table 8: Per-source Recall@ {1,3,5,10} and MRR (%) on Onkopedia and the MedGUIDE pool. MedGUIDE does not provide separate rewritten cases, so its rewritten rows are marked N/A. Continued from Table 7 .
Original queries
Rewritten queries
Method
R@1
R@3
R@5
R@10
MRR
R@1
R@3
R@5
R@10
MRR
OCR + BM25
62.55
77.74
83.23
88.75
71.71
58.55
74.64
80.90
86.85
68.40
VLM Cap. + BM25
71.83
87.71
91.20
93.86
80.50
68.62
85.69
89.86
92.97
78.07
CLIP
15.17
28.32
35.78
46.02
25.48
14.73
27.57
34.96
45.05
24.85
Qwen3-VL-Emb
74.39
90.33
93.95
96.75
82.90
73.05
89.98
93.69
96.57
82.03
ColPali
55.48
73.75
80.61
86.43
66.22
52.09
71.00
78.44
84.71
63.36
Appendix
Table 9: Recall@ {1,3,5,10} and MRR (%) on the Mixed pool ( 257 candidates, 9,658 queries per mode).
Figure 5: Per-patch decomposition of CRISP; each panel isolates one ingredient from the ablation table ( Table 3 ). (a) MaxSim : maxtqt⊤ps saturates over whitespace. (b) Patch weights : ws from Eq. ( 1 ); query-independent, concentrated on decision and outcome boxes. (c) Patch weights + Adjusted : per-token contrast at from Eq. ( 2 ) credited to its argmaxs patch; sparse by construction. (d) Complete CRISP : image-side proxy ws⋅maxtqt⊤ps for Eq. ( 3 ); the additive reverse term contributes a scalar per candidate with no separate per-patch signature. Each panel is normalised to its minimum and maximum values independently.
Biomedical information retrieval finds and ranks relevant evidence in scientific literature, clinical records, databases, and biomedical knowledge resources.
Clinical practice guidelines are long, multimodal documents whose branching recommendations are difficult to convert into executable clinical decision support (CDS), and one-shot parsing often breaks cross-page continuity. Recent LLM/VLM extractors are mostly local or text-centric, under-specifying section interfaces and failing to consolidate cross-page control flow across full documents into one coherent decision graph. We present a decomposition-first pipeline that converts full-guideline evidence into an executable clinical decision graph through topology-aware chunking, interface-constrained chunk graph generation, and provenance-preserving global aggregation. Rather than relying on single-pass generation, the pipeline uses explicit entry/terminal interfaces and semantic deduplication to preserve cross-page continuity while keeping the induced control flow auditable and structurally consistent. We evaluate on an adjudicated prostate-guideline benchmark with matched inputs and the same underlying VLM backbone across compared methods. On the complete merged graph, our approach improves edge and triplet precision/recall from 19.6%/16.1% in existing models to 69.0%/87.5%, while node recall rises from 78.1% to 93.8%. These results support decomposition-first, auditable guideline-to-CDS conversion on this benchmark, while current evidence remains limited to one adjudicated prostate guideline and motivates broader multi-guideline validation.
Onur Selim Kilic, Yeti Z. Gurbuz, Cem O. Yaldiz +4
Georgia Institute of Technology · MetaDialog · Infuse Inc.
Clinical decision-making requires synthesizing heterogeneous evidence, including patient histories, clinical guidelines, and trajectories of comparable cases. While large language models (LLMs) offer strong reasoning capabilities, they remain prone to hallucinations and struggle to integrate long, structured medical documents. We present MED-COPILOT, an interactive clinical decision-support system designed for clinicians and medical trainees, which combines guideline-grounded GraphRAG retrieval with hybrid semantic-keyword similar-patient retrieval to support transparent and evidence-aware clinical reasoning. The system builds a structured knowledge graph from WHO and NICE guidelines, applies community-level summarization for efficient retrieval, and maintains a 36,000-case similar-patient database derived from SOAP-normalized MIMIC-IV notes and Synthea-generated records. We evaluate our framework on clinical note completion and medical question answering, and demonstrate that it consistently outperforms parametric LLM baselines and standard RAG, improving both generation fidelity and clinical reasoning accuracy. The full system is available at https://huggingface.co/spaces/Cryo3978/Med_GraphRAG , enabling users to inspect retrieved evidence, visualize token-level similarity contributions, and conduct guided follow-up analysis. Our results demonstrate a practical and interpretable approach to integrating structured guideline knowledge with patient-level analogical evidence for clinical LLMs.
Guideline documents are published as unstructured PDFs whose evidence is locked in visual structures---tables, flowcharts, and graded recommendations---that standard retrieval pipelines flatten into fixed-size text chunks. We cast evidence access as a document image analysis problem: parse each page image into typed structural elements, then retrieve structure-aware evidence units that follow the document's own layout (sections, table rows, flowchart paths, graded recommendations), each keeping its structural context so a result points to a specific element rather than a page. On 26 clinical practice guidelines from 9 sources (3,619 pages, Chinese and English) with 199 evidence queries, structure-aware units rank the gold element first under BM25, dense, and hybrid retrieval (hybrid Element Hit@1 of 0.382), with a significant element-level ranking gain over per-element OCR text (MRR_e +0.107, p=0.002; the Hit@5 gain is directional, p=0.17), while matching page-level recall (Page Hit@5 0.879 vs. 0.889, p=0.75) at 3.8x less context and clearly outperforming a ColPali visual-RAG baseline (PH@5 0.497).
Xingyu Lin, Dehui Du
Software Engineering Institute, East China Normal University Shanghai, China