Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes

Yang Zhong; Ling Luo; Xuanlyu Zhu; Yanhua Wang; Yuanyuan Sun; Jian Wang; Hongfei Lin

cs.AIOct 15, 2026

Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes

Authors: Yang Zhong, Ling Luo, Xuanlyu Zhu, Yanhua Wang, Yuanyuan Sun, Jian Wang, Hongfei Lin

Organizations: College of Computer Science and Technology, Dalian University of Technology, Dalian, China · Air Force Communications NCO Academy, Dalian, China

Abstract

Medical calculators translate raw patient observations into clinically actionable values and scores. However, applying these calculators directly to free-form patient notes requires precise variable extraction combined with the exact execution of complex formulas, scoring rules, and output conventions. Relying solely on direct large language model (LLM) prompting is unreliable in this context, as rule recall, parameter extraction, arithmetic reasoning, and formatting become entangled within a single generative step. To address these limitations, we propose a tool-augmented LLM framework for reliable medical calculation from patient notes. The framework retrieves and structures calculator specifications, generates reusable executable tools, and uses a constrained extractor to map each patient note to calculator-ready inputs. The tools then perform deterministic calculation, separating clinical language understanding from rule execution and arithmetic. Evaluated on the MedCalc-Bench dataset, our method achieves 86.73% final-answer accuracy, outperforming the one-shot prompting and supervised fine tuning-only baselines by 24.0 and 17.27 percentage points. Ablation and error analyses show that these performance gains are primarily driven by rule-grounded tool synthesis and schema-constrained extraction, whereas remaining failures are concentrated in clinical-semantic extraction and benchmark-specific output contracts. By grounding calculator logic in executable tools, the proposed approach alleviates arithmetic hallucinations and improves the computational reliability of the final clinical outputs.

Figures & tables

Fig. 1: Overview of the proposed method. It is illustrated with the Cockcroft–Gault creatinine-clearance calculator. The workflow retrieves and structures calculator rules, synthesizes and validates an executable calculator tool, extracts calculator inputs from a patient note, and executes the validated tool deterministically.

Panel of Figure 2 — Fig. 2: Condensed prompts used for calculator rule curation, executable tool synthesis, and schema-constrained parameter extraction.

Item	Setting
Base models	Qwen3-8B
Tuning method	LoRA adapter on frozen base model
LoRA rank / alpha / dropout	16 / 32 / 0.05
Learning rate	$5{\times}10^{-5}$
Epochs	5
global batch size	8

TABLE I: Main supervised fine-tuning configuration for the parameter-extraction model.

Method	Model	Score (%)
MedRaC * [ 23 ]	LLaMA3.1-8B-Instruct	64.68
Open-book guided prompting [ 14 ]	GLM-4.6V	82.00
Open-book prompting * [ 14 ]	GLM-4.7	85.50
Code agent (Python REPL) [ 31 ]	Claude Sonnet 4.6	86.24
Direct prompting	Qwen3-8B	14.00
Zero-shot CoT	Qwen3-8B	31.55

TABLE II: Main results on MedCalc-Bench Verified v1.0.7.

Tool setting	Model	Runtime	Tool acc.	Final acc.
		errors	(%)	(%)
Direct generation	GPT-5.5	0	74.45	73.27
Retrieval + repair	Qwen3.5-Plus	21	84.45	76.18
Retrieval + repair	GPT-5.3-Codex	4	89.00	82.82
Retrieval + repair	GPT-5.5	0	94.91	86.73
Note: The last column pairs each tool set with the same Qwen3-8B-SFT extractor.

TABLE III: Generated Tool Quality and End-to-End Calculation Performance.

Fig. 3: Parameter matching rate and final-answer accuracy across base extractor families with a fixed GPT-5.5-generated tool set.

Fig. 4: Parameter matching rate and final-answer accuracy by calculator category for the best configuration.

Variant	Final acc. (%)	$\Delta$
Full workflow	86.73	-
No extractor SFT	83.73	-3.00
No schema/few-shot	79.18	-7.55
No tool	69.46	-17.27

TABLE IV: Component ablation results.

Builder	Calls	Input	Output	Total
Qwen3.5-Plus	144	243,571	147,350	390,921
GPT-5.3-Codex	159	272,884	124,773	397,657
GPT-5.5	147	227,454	371,758	599,212

TABLE V: Construction calls and token usage.

Tool library	Total (min)	Seconds/case
Qwen3.5-Plus: retrieval + repair	48.15	2.626
GPT-5.3-Codex: retrieval + repair	48.47	2.644
GPT-5.5: retrieval + repair	47.17	2.573
GPT-5.5: direct generation	47.28	2.579

TABLE VI: Inference time with the same SFT extractor.

	Case A	Case B	Case C
Calculator	Creatinine Clearance, Cockcroft-Gault (lab test)	HEART Score (risk)	Steroid Conversion Calculator (dosage)
Patient note (key facts)	87-year-old man; height 163 cm; weight 48 kg; serum creatinine 1.4 mg/dL; the question asks Cockcroft-Gault clearance with weight selection.	47-year-old man in the ED; diabetes, hypertension, atorvastatin use, normal ECG, troponin $>3\times$ upper limit, no prior atherosclerotic disease, slightly suspicious history.	Patient receives Hydrocortisone PO 162.935 mg; the question asks for the equivalent Dexamethasone PO dose, with the oral route preserved.
Question	Creatinine clearance, mL/min, using adjusted body weight if it exceeds ideal.	HEART Score (0–10) at first admission, before treatment.	Equivalent Dexamethasone PO dose in mg.
Generated tool pseudocode (written once by GPT-5.5)	ibw = ideal_weight(sex, height) ; w = actual if actual < ibw else adjusted ; crcl = ((140-age)w)/(72scr) .	risk_count = diabetes + hypertension + hypercholesterolemia + ... ; risk_points = 2 if risk_count >= 3 else 1 if risk_count >= 1 else 0 .	if "cortisone" in name: return cortisone ; elif "hydrocortisone" in name: return hydrocortisone ; then convert by steroid-equivalence table.
Extracted parameters (Qwen3-8B-SFT)	sex=Male, age=87 y, weight=48 kg, height=163 cm, creatinine=1.4 mg/dL.	age=47, hypertension=True, diabetes=True, hypercholesterolemia=False , ECG=Normal, troponin= $>3\times$ , atherosclerotic=False, suspicion=Slightly suspicious.	input steroid=Hydrocortisone PO, dose=162.935 mg; target steroid=Dexamethasone PO.
Tool computation	weight 48 $<$ ideal 59.6, so uses actual weight; $(140-87)\times 48/(72\times 1.4)=25.24$ .	only 2 risk factors counted, so risk block scores 1 and total = 4 .	Hydrocortisone is misread as cortisone: $162.935\times 0.75/25={\color[rgb]{1,0,0}4.88805}$ mg.

TABLE VII: Representative cases of our method for medical calculation.

Fig. 5: Error analysis of the best run. Extraction errors come from incorrectly extracted calculator fields, while Generated-tool errors occur after the required fields have been extracted correctly.

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 295 papers

Jul 3, 2026

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.

Siran Zhao, Ruihui Hou, Ziyue Huai +2

East China University of Science and Technology, Shanghai, China

LLM Agent Evaluation LLM Tool Use Medical LLMs

Ask PDF

Sep 25, 2026

Improving Medical Calculation of LLMs with Embedded Coding

Large Language Models (LLMs) perform well on medical examinations and question-answering benchmarks, but remain unreliable on medical calculation tasks that require exact numerical outputs. These calculations support high-stakes decisions such as medication dosing, organ-function assessment, and prognostic scoring, for which even small errors can have serious clinical consequences. We introduce MedCode, a framework that improves medical calculation by training LLMs to generate embedded executable code. Given a clinical context, the model identifies the relevant calculator, extracts its input variables, and produces a script that delegates arithmetic operations to a deterministic interpreter. Executing the script returns the calculated value together with an explanation and the appropriate unit. We construct supervised fine-tuning (SFT) and preference datasets from the MedCalc benchmark and additionally curate a dataset for calculation tasks in Intensive Care Unit (ICU) scenarios. We further propose weighted Direct Preference Optimization (wDPO), which adaptively emphasizes preference pairs that are difficult for the model to distinguish. Experiments with LLaMA3-8B, Qwen2.5-7B, and Mistral-7B show absolute accuracy gains of 20--30 percentage points, demonstrating the effectiveness of embedded code generation for medical calculation.

Tianshi Ming, Yingying Zhang, Xian Wu

Carnegie Mellon University · Tencent YouTu Lab

Medical LLMs Numerical Reasoning in Language Models LLM Code Generation

Ask PDF

Sep 9, 2026

Towards a Deterministic Math Solver for Clinical Language Models

Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. We test an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it. We evaluate this Program-Solve interface on MedCalc-Bench Verified (1,100 cases, 55 calculators) against direct model arithmetic and a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ, after auditing the benchmark's formulas against current clinical guidelines and flagging 16 of 55 with version, use or coefficient concerns. With formulas and gold variables supplied and both routes reading the whole note, handing off to the solver is not a reliable advantage at 7B (75.31% against 72.02%, a paired +3.29 points with a 95% calculator-cluster interval of [-3.49, 10.38]) but is one at 32B (90.53% against 83.47%, +7.05 [0.47, 14.60], clear of zero). The hand-written library is exact on its 440 supported cases but abstains elsewhere (40.0% overall). Adding an executor thus helps some open-weight models more than others even under matched formula, variable and note access, and is not a substitute for verified formulas or reliable variable extraction either way.

Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange +5

MIT Critical Data · UNC Chapel-Hil · University of Pavia +1

Numerical Reasoning in Language Models LLM Reliability LLM Tool Use Clinical Language Model Evaluation LLM Code Generation

Ask PDF

Abstract

LLM Tool Use

Explore similar work

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

Improving Medical Calculation of LLMs with Embedded Coding

Towards a Deterministic Math Solver for Clinical Language Models