Improving Medical Calculation of LLMs with Embedded Coding

Tianshi Ming; Yingying Zhang; Xian Wu

cs.AISep 25, 2026

Improving Medical Calculation of LLMs with Embedded Coding

Authors: Tianshi Ming, Yingying Zhang, Xian Wu

Organizations: Carnegie Mellon University · Tencent YouTu Lab

Abstract

Large Language Models (LLMs) perform well on medical examinations and question-answering benchmarks, but remain unreliable on medical calculation tasks that require exact numerical outputs. These calculations support high-stakes decisions such as medication dosing, organ-function assessment, and prognostic scoring, for which even small errors can have serious clinical consequences. We introduce MedCode, a framework that improves medical calculation by training LLMs to generate embedded executable code. Given a clinical context, the model identifies the relevant calculator, extracts its input variables, and produces a script that delegates arithmetic operations to a deterministic interpreter. Executing the script returns the calculated value together with an explanation and the appropriate unit. We construct supervised fine-tuning (SFT) and preference datasets from the MedCalc benchmark and additionally curate a dataset for calculation tasks in Intensive Care Unit (ICU) scenarios. We further propose weighted Direct Preference Optimization (wDPO), which adaptively emphasizes preference pairs that are difficult for the model to distinguish. Experiments with LLaMA3-8B, Qwen2.5-7B, and Mistral-7B show absolute accuracy gains of 20--30 percentage points, demonstrating the effectiveness of embedded code generation for medical calculation.

Figures & tables

Figure 1: An example medical calculation task. Mistral-7B and GPT-4 produce incorrect answers because of formula-selection or arithmetic errors. MedCode instead generates an embedded code script and executes it with a deterministic interpreter. The execution output contains both the calculation procedure and the numerical answer. This pipeline requires one LLM forward pass for script generation, followed by code execution without an additional model call.

> Equation-based Calculator Example : Modification of Diet in Renal Disease Glomerular Filtration Rate (MDRD GFR)
Patient Note	A 73-year-old Caucasian male presented with longstanding chronic kidney disease (CKD) G2 related to arterial hypertension stage II. Past medical history: prostate cancer and monoclonal gammopathy of undetermined significance (MGUS) IgG k, rosacea dermatitis, right hand arthritis. …
Question	What is the patient’s Glomerular Filtration Rate (GFR) in terms of mL/min/1.73 m 2 ?
Generated Code	# GFR = 175 * creatinineˆ(-1.154) * ageˆ(-0.203) * race_coefficient * gender_coefficient race_coefficient = 1.212 if ’Black’ == ’Black’ else 1.0 gender_coefficient = 0.742 if ’Male’ == ’Female’ else 1.0 gfr = 175 * 10.6*(-1.154) 49*(-0.203) race_coefficient * gender_coefficient gfr = round(gfr, 3) print(’The MDRD GFR is ’, gfr, ’ mL/min/1.73 m 2 ’)
Code Execution Result	The MDRD GFR is 49.677 mL/min/1.73 m 2
> Rule-based Calculator Example : Centor Score (Modified/McIsaac) for Strep Pharyngitis
Patient Note	A 15-year-old female presents to your office for evaluation of a painful neck mass. The family reports the patient has had a midline neck mass for many years, but had become painful over the past week with erythema at the overlying skin. …

Table 1: Examples of embedded code generation for equation-based and rule-based medical calculation tasks.

Figure 2: Overview of MedCode. 1) Data generation : We convert MedCalc calculators into an embedded code dataset for SFT and pair erroneous scripts produced by the fine-tuned model with reference scripts to construct preference data. 2) Fine-tuning and preference training : We first perform SFT and then apply wDPO, which adaptively weights the preference objective. 3) One-pass inference : The model generates a script containing values extracted from the patient note, and a deterministic interpreter executes the script to produce the final answer and calculation explanation.

Model	Equation-based				Rule-based			Avg.
	Lab	Phys.	Date	Dosage	Risk	Sev.	Diag
Zero-shot
LLaMA3-8B	10.70	19.17	3.33	5.00	12.50	8.75	25.00	13.09
Mistral-7B	10.70	18.33	3.33	0.00	4.58	3.75	13.33	9.8
GPT-3.5	17.13	35.00	13.33	5.00	12.92	6.25	18.33	18.82
GPT-4	14.37	34.58	38.33	15.00	14.58	15.00	20.00	20.82

Table 2: Results on MedCalc-Bench. We evaluate LLaMA3-8B-Instruct, Mistral-7B-Instruct-v0.2, GPT-3.5-Turbo, and GPT-4. CoT denotes Chain-of-Thought prompting. Zero-shot, zero-shot CoT, and one-shot CoT results are taken from MedCalc-Bench ( Khandekar et al., 2024 ) . *For code generation with GPT-3.5 and GPT-4, we use one in-context example.

Model	Methods	extr.	fcnt.
LLaMA3-8B	One-shot Direct	39.22	0.0
	SFT-Direct	52.39	75.10
	SFT-Code	82.10	97.60
	SFT-Code-wDPO	83.20	98.00
Qwen2.5-7B	One-shot Direct	47.90	0.0
	SFT-Direct	56.70	77.90

Table 3: Results on the ICU dataset. “extr.’ denotes extraction accuracy and “fcnt.’ denotes frequency-count accuracy. We evaluate LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct.

Model	extr.	fcnt.
LLaMA3-8B-Instruct (wDPO)	83.20	98.00
Qwen2.5-7B-Instruct (wDPO)	62.10	97.50
Baichuan2-7B-Chat (wDPO)	79.13	96.54
DeepSeek-R1	85.87	99.53
OpenAI-o3	90.10	99.10

Table 4: Comparison with reasoning-focused models on the ICU dataset. “extr.’ denotes extraction accuracy and “fcnt.’ denotes frequency-count accuracy.

Strategy	Lab	Phys.	Date	Dos.	Risk	Sev.	Diag.	Avg.
Zero-shot	10.70	19.17	3.33	5.00	12.50	8.75	25.00	13.09
Zero-shot CoT	16.51	25.00	1.67	7.50	11.25	13.75	26.67	16.43
One-shot CoT	34.86	35.42	3.33	2.50	20.00	11.25	41.67	27.13
Zero-shot SFT Code	19.27	85.00	33.33	28.57	11.25	10.00	41.67	33.37
One-shot SFT Code	20.38	89.85	35.25	30.24	11.92	10.59	44.08	34.56
3-shot SFT Code	21.49	94.72	37.17	31.89	12.55	11.15	46.49	36.77

Table 5: Comparison of zero and few-shot strategies.

Method	Lab	Phys.	Date	Dosage	Risk	Sev.	Diag.	Avg.
rDPO ( Chowdhury et al., 2024 )	68.45	79.20	30.12	59.33	25.10	12.55	51.33	50.84
ROPO ( Liang et al., 2024 )	70.80	82.50	34.00	63.75	23.90	11.30	53.45	54.25
WPO ( Zhou et al., 2024 )	69.20	81.00	31.45	61.25	24.70	11.00	47.70	52.33
wDPO (ours)	71.78	83.82	33.33	65.00	22.82	10.00	58.33	55.34

Table 6: Comparison of preference optimization methods on MedCalc-Bench using LLaMA3-8B-Instruct. All methods use the same SFT-Code initialization and preference dataset.

Loss	Reg.	Eq.	Rl.	Avg.
$\mathcal{L}_{\text{SFT}}$	-	70.77	25.72	54.39
$\mathcal{L}_{\text{DPO}}$	-	60.17	5.23	41.36
	0.5 $\mathcal{L}_{\text{SFT}}$	62.82	8.66	43.13
	1.0 $\mathcal{L}_{\text{SFT}}$	66.87	10.24	46.28
$\mathcal{L}_{\text{wDPO}}$	-	66.50	15.37	47.68
	0.5 $\mathcal{L}_{\text{SFT}}$	67.32	20.27	52.17

Table 7: Results of regularization strategies on MedCalc-Bench using LLaMA3-8B-Instruct. “adpt.’ denotes the adaptive coefficient; Eq. and Rl. denote equation-based and rule-based tasks; “–’ indicates that no additional SFT regularization is used.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category		Embedded Coding Script Design (Explanation printings omitted)
Equation-based	Single-step	# BMI = weight / (height * height) bmi = 76.0 / ((183.0 / 100) * (183.0 / 100)) bmi = round(bmi, 3) print(’The Body Mass Index (BMI) is ’, bmi, ’kg/m²’)
	Multi-step	# GFR = 175 * creatinineˆ(-1.154) * ageˆ(-0.203) * race_coefficient * gender_coefficient race_coefficient = 1.212 if ’Black’ == ’Black’ else 1.0 gender_coefficient = 0.742 if ’Male’ == ’Female’ else 1.0 gfr = 175 * 10.6*(-1.154) 49*(-0.203) race_coefficient * gender_coefficient gfr = round(gfr, 3) print(’The MDRD GFR is ’, gfr, ’mL/min/ 1.73 m²’)
	Conversion	# converted dose = (from mass) * (to reletive mass) / (from reletive mass) reletive_mass = {’Betamethasone IV’: 1, ’Cortisone PO’: 33.33, ’Dexamethasone IV’: 1, ’Dexamethasone PO’: 1,’Hydrocortisone IV’: 26.67,’Hydrocortisone PO’: 26.67, ’MethylPrednisoLONE IV’: 5.33,’MethylPrednisoLONE PO’: 5.33,’PrednisoLONE PO’: 6.67, ’PredniSONE PO’: 6.67, ’Triamcinolone IV’: 5.33} converted_dose = 190.936 * reletive_mass[’MethylPrednisoLONE IV’] / reletive_mass[’Hydrocortisone PO’] converted_dose = round(converted_dose, 3) print(’The Steroid Conversion is ’, converted_dose, ’mg’)
	Date	import datetime # estimated due date = input_date + 40 weeks future_date = datetime.datetime.strptime(’11/13/2020’, ’%m/%d/%Y’) + datetime.timedelta(weeks=40) cycle_length_gap = (-1)⁢⁢(23 < 28) ⁢ datetime.timedelta(days=abs(23 - 28)) estimated_due_date = (future_date + cycle_length_gap).strftime(’%m/%d/%Y’) print(’The Estimated Due Date is ’, estimated_due_date, ”)
	Duration	import datetime # estimated gestational age = the number of weeks and days apart today’s date is from the patient’s last menstrual period date current_date = datetime.datetime.strptime(’09/01/2009’, ’%m/%d/%Y’) menstrual_date = datetime.datetime.strptime(’05/21/2009’, ’%m/%d/%Y’) weeks = abs(current_date - menstrual_date).days // 7 days = abs(current_date - menstrual_date).days % 7 print(’The Estimated Gestational Age is’, weeks, ’weeks’, days, ’days’,”)
Rule-based	Single-sym.	# Centor Score for Strep Pharyngitis criteria: # age, temperature, tonsil swelling, lymph nodes # age thresholds: (45, inf), (3, 15); temperature > 38°C score = 0 score += -1 * (45 <= 48 < float(’inf’)) + 1 * (3 <= 48 < 15) score += 1 * (38 < ((99.0 - 32) * 5/9) < float(’inf’)) has_cough_absent = False; score += 1 if has_cough_absent else 0 has_tender_lymph_nodes = False; score += 1 if has_tender_lymph_nodes else 0 has_exudate_swelling_tonsils = False; score += 1 if has_exudate_swelling_tonsils else 0 print(’The Centor Score (Modified/McIsaac) for Strep Pharyngitis is’, score)

Appendix

Table 8: Examples of embedded scripts for medical calculation. “single-sym.’ denotes a criterion containing one symptom, whereas “multi-sym.’ denotes a criterion combining multiple symptoms (e.g., has_pre_pe or has_pre_dvt ). Explanatory output statements are omitted for brevity.

Hyperparameters	Value
batch size	4
learning rate	5e-5
epochs	2.0
optimizer	AdamW
cutoff length	2048
data type	bfloat16

Appendix

Table 9: Supervised fine-tuning and preference training hyperparameters.

Hyperparameters	Value
max length	8192/ 4096
temperature	0.0
data type	float16
do sample	False
truncation	True

Appendix

Table 10: Inference hyperparameters. The maximum length is 8,192 for LLaMA3 and GPT-4 and 4,096 for GPT-3.5, following the MedCalc-Bench configuration.

Method	Minor	Substantial
w/o wDPO	360	87
w/ wDPO	312	144

Appendix

Table 11: Coding error types before and after wDPO. Minor errors comprise incorrect extracted values, coefficients, or constants, with at most two such errors per script. Substantial errors comprise missing or unnecessary steps and execution failures.

Category	Metric	Unit	Range
Oxygen Saturation	Blood gas analysis oxygen saturation	%	[80, 100]
Oxygen Saturation	Vital sign oxygen saturation (SpO 2 )	%	[80, 100]
Blood Pressure	Vital sign invasive diastolic pressure	mmHg	[40, 90]
	Vital sign non-invasive diastolic pressure	mmHg	[40, 90]
	Vital sign invasive systolic pressure	mmHg	[90, 140]
	Vital sign non-invasive systolic pressure	mmHg	[90, 140]

Appendix

Table 12: Categories, metrics, units, and clinically plausible ranges in our ICU dataset.

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 397 papers

Jul 3, 2026

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.

Siran Zhao, Ruihui Hou, Ziyue Huai +2

East China University of Science and Technology, Shanghai, China

LLM Agent Evaluation LLM Tool Use Medical LLMs

Ask PDF

Oct 15, 2026

Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes

Medical calculators translate raw patient observations into clinically actionable values and scores. However, applying these calculators directly to free-form patient notes requires precise variable extraction combined with the exact execution of complex formulas, scoring rules, and output conventions. Relying solely on direct large language model (LLM) prompting is unreliable in this context, as rule recall, parameter extraction, arithmetic reasoning, and formatting become entangled within a single generative step. To address these limitations, we propose a tool-augmented LLM framework for reliable medical calculation from patient notes. The framework retrieves and structures calculator specifications, generates reusable executable tools, and uses a constrained extractor to map each patient note to calculator-ready inputs. The tools then perform deterministic calculation, separating clinical language understanding from rule execution and arithmetic. Evaluated on the MedCalc-Bench dataset, our method achieves 86.73% final-answer accuracy, outperforming the one-shot prompting and supervised fine tuning-only baselines by 24.0 and 17.27 percentage points. Ablation and error analyses show that these performance gains are primarily driven by rule-grounded tool synthesis and schema-constrained extraction, whereas remaining failures are concentrated in clinical-semantic extraction and benchmark-specific output contracts. By grounding calculator logic in executable tools, the proposed approach alleviates arithmetic hallucinations and improves the computational reliability of the final clinical outputs.

Yang Zhong, Ling Luo, Xuanlyu Zhu +4

College of Computer Science and Technology, Dalian University of Technology, Dalian, China · Air Force Communications NCO Academy, Dalian, China

LLM Tool Use Clinical Decision Support

Ask PDF

May 2, 2026

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks

Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32

Sophont · MedARC · Prime Intellect

LLM Evaluation Healthcare Clinical Language Model Evaluation Medical LLMs

Ask PDF

Abstract

Medical LLMs

Explore similar work

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks