ALLMTS: Automated LLM-based Medical Triage System

Michał Remigiusz Janiszewski; Artur Stopa; Łukasz Lenkiewicz; Wojciech Achtelik; Bartosz Gonczarek

cs.AIOct 23, 2026

ALLMTS: Automated LLM-based Medical Triage System

Authors: Michał Remigiusz Janiszewski, Artur Stopa, Łukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek

Organizations: Vstorm, R&D, Poland · Wrocław University of Science and Technology, Wrocław, Poland · Faculty of Computer Science, AGH University of Kraków, Kraków, Poland · University of Technology and Arts in Applied Sciences in Warsaw, Wrocław, Poland

Abstract

Telephone triage is a safety critical task: from a brief intake note, a nurse must decide whether a caller needs an ambulance, an emergency department, an appointment, or self care at home. Queried directly, large language models (LLMs) are unreliable: they over-escalate routine cases and cannot justify their recommendations against an auditable clinical standard. We present the Automated LLM-based Medical Triage System (ALLMTS), which grounds every disposition in the same commercially maintained telehealth guidelines nurses use. Rather than asking an LLM to triage, ALLMTS makes it execute the guideline: it selects the appropriate guideline and evaluates its triage assessment questions. The disposition follows deterministically from the first positive answer, using the same traversal rule a nurse follows. We evaluate ALLMTS on 566 synthetic, expert-validated scenarios spanning 14 dispositions and designed to test specific guideline questions. ALLMTS achieves 0.95 accuracy, compared with 0.47 for a raw-LLM baseline. On a curated 50-scenario subset released as an open benchmark, ALLMTS reaches 0.98 accuracy, on par with the 0.944 correct disposition rate that a five nurse expert panel recorded on those same scenarios. Critically, on that subset the baselines have over-triage rates of up to 0.50 (0.44 on the full dataset), whereas ALLMTS reduces the over-triage rate to at most 0.02 while maintaining low under-triage. These results support coupling LLM language understanding with executable clinical guidelines for triage from intake notes.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of the ALLMTS pipeline. Grey nodes are input/output, white nodes are the core sequential stages, blue nodes are the pre-classification stages that feed classification, amber nodes form the concurrent TAQ evaluation block, the dashed red box marks conditional SMAG re-routing. Intermediate data artifacts are shown as labels on the arrows rather than as separate boxes. The TAQ evaluation stage runs all disposition levels concurrently.

Ad-hoc
	ALLMTS		Guideline Ctx		Raw LLM		Web Search
View	Acc/F1	UT/OT	Acc/F1	UT/OT	Acc/F1	UT/OT	Acc/F1	UT/OT
Disposition	.98/.98	.02/.00	.94/.94	.00/.06	.42/.40	.14/.44	.38/.35	.14/.48
Acuity	–	–	.94/.94	.02/.04	.66/.66	.04/.30	.72/.72	.06/.22
Urgency	–	–	.84/.81	.04/.12	.52/.50	.14/.34	.46/.43	.16/.38
Urgency Group	–	–	.76/.76	.04/.20	.58/.58	.12/.30	.58/.59	.16/.26

Table 1: 50-scenario set: accuracy / weighted F1 (Acc/F1, higher is better) and under-/over-triage rates (UT/OT, lower is better). In the ad-hoc panel the model predicts directly in each view’s label space. In the post-hoc panel each coarse-view label is projected from the fine-grained disposition prediction. Only the baselines and the ablation populate the ad-hoc coarse views, since ad-hoc ALLMTS predicts on the disposition space directly. Discordance is the sum of the two triage-error rates.

Ad-hoc
	ALLMTS		Guideline Ctx		Raw LLM		Web Search
View	Acc/F1	UT/OT	Acc/F1	UT/OT	Acc/F1	UT/OT	Acc/F1	UT/OT
Disposition	.95/.96	.04/.02	.90/.90	.05/.04	.47/.48	.10/.44	.50/.51	.11/.39
Acuity	–	–	.94/.94	.02/.04	.67/.67	.08/.25	.70/.70	.08/.22
Urgency	–	–	.89/.88	.04/.07	.56/.55	.15/.29	.54/.53	.18/.28
Urgency Group	–	–	.86/.86	.06/.09	.60/.61	.16/.25	.61/.62	.16/.23

Table 2: Full 566-scenario dataset (superset of the 50-scenario set), scored and laid out as in Table 1 .

	vs. reference			self-agreement		scenarios
Method	Acc	$\kappa_{\text{ref}}$	$\kappa_{\text{ref}}^{w}$	$\kappa_{\text{run}}$ / Cons.	Flip	$10/10$	$0/10$
ALLMTS	$.966\pm.018$	$.955$	$.961$	$.958$ / $.974$	$.06$	47	0
Guideline Ctx	$.918\pm.019$	$.890$	$.932$	$.942$ / $.970$	$.12$	43	1
Raw LLM	$.658\pm.024$	$.541$	$.707$	$.892$ / $.946$	$.20$	28	12
Web Search	$.706\pm.031$	$.605$	$.747$	$.844$ / $.922$	$.26$	28	9

Table 3: Run-to-run stability on the acuity view, 10 runs over the same 50 scenarios. Acc is the per-run accuracy (mean

\pm

s.d.).

\kappa_{\text{ref}}

and

\kappa_{\text{ref}}^{w}

are unweighted and linearly weighted Cohen’s

\kappa

against the reference label, averaged over runs.

\kappa_{\text{run}}

is Fleiss’

\kappa

across the ten runs (self-agreement, independent of correctness). Cons. is mean modal prediction consistency and Flip the fraction of scenarios whose prediction is not identical on all ten runs.

10/10

and

0/10

count scenarios correct on every run and on no run.

Method	Acc	F1	Under	Over	Disc
Levine	0.70	0.70	0.10	0.20	0.30
Gaber	0.58	0.55	0.18	0.24	0.42
ALLMTS	0.98	0.98	0.02	0.00	0.02

Table 4: Reproduced prior methods on the 50-scenario STCC set with gpt-5.2 , scored in each method’s native label space. Levine uses its four-level taxonomy while Gaber uses ESI.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Disposition ( $\mathcal{D}$ )	Acuity	Urgency	Urgency Group
Call EMS 911 Now	EMS 911	911 Now	Very High
Go to ED Now	Emergent	Emergent Now	High
Go to L&D Now	Emergent	Emergent Now	High
Go to ED/UCC Now (or PCP Triage)	Emergent	Within 1 Hour	High
Go to L&D Now (or PCP Triage)	Emergent	Within 1 Hour	High
Call PCP Now	Urgent	Within 1 Hour	High

Appendix

Table 5: Deterministic label view mappings. Each fine-grained disposition projects independently into the acuity, urgency, and urgency-group label spaces used for grouped evaluation. The table lists the 14 reference dispositions exercised by the full 566-scenario dataset; the 50-scenario subset exercises 10 of them. No positive TAQ found is an ALLMTS-only outcome with no analogue in the fixed baseline label space

\mathcal{D}

and is kept verbatim rather than mapped.

Hyperparameter	Value
Embedding model	text-embedding-3-large
Embedding dimension	3072
Guideline representation	Title
Number of candidates	$k=50$
Query augmentation	Primary reason for visit
Similarity function	Cosine similarity

Appendix

Table 6: Retriever configuration used in the experiments.

Disposition	Acuity	Levine 4 level	Gaber ESI
EMS 911 now	1	Emergent	1
ED now	2	Emergent	2
ED/UCC now (PCP triage)	2	1 day	2
HCP ${\leq}4$ h (PCP triage)	3	1 day	3
PCP now	3	1 day	3
See PCP ${\leq}24$ h	3	1 day	3

Appendix

Table 7: Post-hoc projection of STCC ground-truth dispositions into the label spaces of the reproduced methods. Rows are the 10 dispositions exercised by the 50-scenario set. Each disposition is routed through the STCC published acuity (1–5) and urgency (10-label) views, then into Levine’s four-level taxonomy and Gaber’s ESI (1–5). Levine is scored via the urgency path; Gaber via the via-acuity path, a near-identity alignment in which

\text{ESI}\,N=\text{acuity rating}\,N

Figure 3: Ground-truth vs. predicted label distributions for the two reproduced methods and ALLMTS ( gpt-5.2 , 50 scenarios). Both compress predictions toward the middle of their scale: Gaber under-uses the extreme ESI 1 and ESI 4 bins while over-populating ESI 2, and Levine over-triages into Emergent .

Dataset	N	Acc	F1	Under	Over	Disc
Levine et al.	43	0.41	0.46	0.34	0.25	0.59
Semigran et al.	37	0.58	0.64	0.29	0.13	0.42

Appendix

Table 8: ALLMTS ( gpt-5.2 , low reasoning) run directly on the non-pediatric subset of external vignette sets, scored via-acuity in each set’s native label space.

Panel of Figure 4 — Figure 4: Reference label distribution of the 566-scenario dataset across the four label views, severity ordered (most urgent first) and coloured as in the confusion matrices. The distribution is head heavy with a long thin tail in the fine-grained disposition space; coarsening collapses the tail into populated bands.

Figure 5: Scenario-length distribution (words per scenario), with the median marked.

Figure 6: Run-to-run stability of ALLMTS. Nearly every cell is green: ALLMTS reproduces the reference acuity band on all ten runs for 47 of 50 scenarios, and no scenario is wrong on every run.

Figure 7: Run-to-run stability of the guideline-context ablation. Reading the selected guideline holistically is nearly as consistent as executing it ( $0.970$ versus $0.974$ for ALLMTS) but less accurate: $43$ of $50$ scenarios are correct on every run and one is wrong on all ten.

Figure 8: Run-to-run stability of the Raw LLM, shown for comparison with ALLMTS (Figure 6 ) and Web Search (Figure 9 ).

Figure 9: Run-to-run stability of the Web Search baseline. Web Search is the least consistent of the four methods (flip rate 0.26), with 13 of 50 scenarios changing prediction across runs.

Panel of Figure 10 — Figure 10: Confusion matrices on the 50-scenario set. The guideline grounded methods stay tightly concentrated along the diagonal, whereas both baselines exhibit an upward spread toward more urgent predictions, indicating a tendency to over-triage.

Panel of Figure 11 — Figure 11: Confusion matrices on the full 566-scenario dataset. ALLMTS’s mass stays on the diagonal across the wider 14 disposition vocabulary; the trailing No positive TAQ found class collects cases where the pipeline found no positive TAQ. Both baselines again spread upward into more urgent predictions.

Clinical Triage

Clinical triage uses patient symptoms, records, and measurements to prioritize urgency, referral, and care when time and resources are limited.

See all 48 papers

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts
Sep 29, 2026Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2Clinical Triage Clinical Decision-Making
Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science
Auditable Emergency Triage for Maternal and Newborn Care in India
Sep 8, 2026Shobhit Jagga, Aman Dalmia, Niharika Priyadarshini +7Healthcare LLM Auditing
Noora Health · The Agency Fund
Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries
May 15, 2026Liqi Zhou, Jiafu LiHealthcare Clinical Triage

Sep 29, 2026J/K move · Enter open · S save

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych+1

Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science

Clinical Triage Clinical Decision-Making Language Model Bias Evaluation Clinical Language Model Evaluation Language Model Robustness

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.

Abstract

Clinical Triage

Explore similar work

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Auditable Emergency Triage for Maternal and Newborn Care in India

Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries