Using reasoning LLMs to extract SDOH events from clinical notes

Ertan Dogan; Kunyu Yu; Yifan Peng

cs.CLApr 15, 2026

Using reasoning LLMs to extract SDOH events from clinical notes

Authors: Ertan Dogan, Kunyu Yu, Yifan Peng

Organizations: Department of Population Health Sciences, Weill Cornell Medicine, New York, USA

Abstract

Social Determinants of Health (SDOH) refer to environmental, behavioral, and social conditions that influence how individuals live, work, and age. SDOH have a significant impact on personal health outcomes, and their systematic identification and management can yield substantial improvements in patient care. However, SDOH information is predominantly captured in unstructured clinical notes within electronic health records, which limits its direct use as machine-readable entities. To address this issue, researchers have employed Natural Language Processing (NLP) techniques using pre-trained BERT-based models, demonstrating promising performance but requiring sophisticated implementation and extensive computational resources. In this study, we investigated prompt engineering strategies for extracting structured SDOH events utilizing LLMs with advanced reasoning capabilities. Our method consisted of four modules: 1) developing concise and descriptive prompts integrated with established guidelines, 2) applying few-shot learning with carefully curated examples, 3) using a self-consistency mechanism to ensure robust outputs, and 4) post-processing for quality control. Our approach achieved a micro-F1 score of 0.866, demonstrating competitive performance compared to the leading models. The results demonstrated that LLMs with reasoning capabilities are effective solutions for SDOH event extraction, offering both implementation simplicity and strong performance.

Figures & tables

Panel of Figure 1 — Figure 1: A sample of SDOH annotations in the 2022 n2c2/UW shared task.

	MIMIC-III	UW	Total
Clinical notes	1,880	2,600	4,480
Avg number of words	39.15	36.32	37.52
Annotations
Alcohol	1,424	1,870	3,294
Drug	940	1,913	2,853
Employment	1,012	826	1,838

Table 1: Characteristics of the dataset.

Figure 2: Our proposed SDOH extraction pipeline.

	Precision	Recall	Micro-F1
Microsoft	0.891	0.887	0.889
Children’s Hospital of Philadelphia	0.874	0.888	0.881
University of Texas at San Antonio	0.880	0.841	0.860
Philips Research North America	0.885	0.780	0.829
Our method (o4-mini)	0.902	0.833	0.866
Alcohol	0.918	0.891	0.904

Table 2: Comparison of our results with n2c2 shared task teams.

Single-prompt
Prompt Type	Precision	Recall	Micro-F1
SHAC annotation guidelines	0.806	0.806	0.806
Alcohol	0.830	0.846	0.838
Drug	0.829	0.707	0.763
Employment	0.708	0.791	0.747
Living Status	0.803	0.845	0.824

Table 3: Comparing methods of structuring guided prompts on a subset of 30 samples from the MIMIC-III test set.

$n$ -shot	Precision	Recall	Micro-F1
0	0.764	0.810	0.786
10	0.827	0.821	0.824
30	0.866	0.879	0.872
50	0.888	0.875	0.881
100	0.744	0.768	0.756

Table 4: Comparison of few-shot learning on a subset of 30 from the MIMIC-III test set.

Prompt	Precision	Recall	Micro-F1
Few-shot prompt	0.802	0.885	0.842
+ self-consistency	0.886	0.920	0.903
+ self-consistency & post-processing	0.948	0.911	0.929
Generating all SDOH types at once	0.884	0.896	0.890

Table 5: Comparison of self-consistency and post-processing on a subset of 30 from the MIMIC-III and UW test sets.

Error Type	Freq
Errors by the model	55
Only included one event of many with the same SDOH type	21
Non-specific type, frequency, history, duration, amount	8
Missing an inferable but not explicit event	7
Missing values in an extracted event	6
Missing method / type for drug event	5

Table 6: Categories of errors in a sample of 100 predictions (50 MIMIC-III / 50 UW)

Panel of Figure 3 — Figure 3: A sample of reasonably correct annotations that were marked as errors because they differed from the gold annotations.

	Precision	Recall	Micro-F1
Our method (llama-3.1-8b)	0.577	0.605	0.591
Alcohol	0.616	0.635	0.626
Drug	0.568	0.452	0.503
Employment	0.382	0.537	0.446
Living Status	0.690	0.757	0.722
Tobacco	0.566	0.598	0.582

Supplementary Table I : Performance of Llama-3.1-8b model on n2c2 shared task.

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

May 5, 2026

Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction

Clinical named entity recognition from dental progress notes is challenging because documentation is highly unstructured, domain-specific, and often privacy-sensitive. We developed a locally deployable framework that enables small language models to self-generate, verify, refine, and evaluate entity-specific prompts for extracting multiple clinical entities from dental notes. Using 1,200 annotated notes, we evaluated candidate open-weight models with multi-prompt ensemble inference and further adapted selected models using QLoRA-based supervised fine-tuning and direct preference optimization. Model performance varied substantially, highlighting the need for task-specific evaluation rather than reliance on generic benchmarks. Qwen2.5-14B-Instruct achieved the strongest baseline performance. After DPO, Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct achieved micro/macro F1 scores of 0.864/0.837 and 0.806/0.797, respectively. These findings suggest that automated prompt optimization combined with lightweight preference-based post-training can support scalable clinical information extraction using locally deployed small language models.

Yao-Shun Chuang, Tushti Mody, Uday Pratap Singh +6

McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX 77054, United States +1

Named Entity Recognition Healthcare LLM Prompting Small Language Models Clinical Information Extraction Clinical Language Model Evaluation

Ask PDF

Jun 19, 2026

Clinical Term Extraction using Open-Source Small Language Models

Clinical information for amyotrophic lateral sclerosis (ALS) care documented in unstructured clinical notes limits downstream analysis without extraction into structured formats. Open-source small language models with few-shot prompting for detecting the presence of ALS-relevant clinical terms in patient documentation were evaluated without task-specific training data. The detection task targeted 17 categories spanning functional scores, respiratory measures, medications, and related clinical and non-clinical attributes. Clinical note content was normalized from JSON-encoded discharge summaries and processed with a prompt template having structured JSON outputs. We compared 26 open-source models using aggregate, label-level, and manual-validation multilabel classification metrics. Manual validation showed that a regex rule baseline had higher overall micro-F1 and lower Hamming loss than any single SLM or TF-IDF baseline, while Qwen3-4B-Instruct-2507 was the highest-performing SLM by micro-F1. Model rankings varied by metric and label category, with the TF-IDF baseline showing high recall but low precision, some SLMs showing higher precision but lower recall, and Hammer2.1-7b showing strong performance for ALSFRS-R subscore detection. These findings support targeted hybrid extraction workflows rather than replacement of existing rule-based methods.

Noah Marchal, William E. Janes, Mihail Popescu +1

University of Missouri, Columbia, Missouri, USA · Department of Occupational Therapy; University of Missouri, Columbia, Missouri, USA

Small Language Models Clinical Information Extraction Classification Clinical NLP

Ask PDF

Apr 22, 2026

Benchmarking Large Language Models for Safety Data Extraction

Accurate extraction of structured information from Safety Data Sheets (SDS) remains challenging in industrial safety due to heterogeneous document formats and the limitations of traditional rule-based methods. This study benchmarks state-of-the-art Large Language Models (LLMs) for automated SDS data extraction, comparing text-based and multimodal processing pipelines. We systematically evaluate four models: Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, and Llama 3.1-70B, across three prompting strategies: zero-shot, few-shot, and chain-of-thought. The evaluation framework assessed accuracy, latency, and cost across more than 50,000 extracted data fields. Results show that text-based extraction consistently outperforms multimodal processing across all metrics. Gemini 1.5 Pro combined with a Chain-of-Thought prompt achieved the highest accuracy (84%), outperforming GPT-4o (81%) and Claude 3.7 Sonnet (79%). However, no model surpassed the 90% accuracy threshold commonly required for reliable real-world deployment. These findings indicate that general-purpose LLMs are not yet robust enough for unsupervised industrial use, though performance suggests strong potential with task-specific fine-tuning. Future research should focus on domain-adapted training, model calibration, and the integration of Human-in-the-Loop verification to ensure safety-critical reliability.

Jonas Grill, Thomas Bayer, Sören Berlinger

1SAP SE, Germany. · Institute for Digital Transformation, Ravensburg-Weingarten University.

LLM Evaluation LLM Prompting Document Information Extraction Multimodal Large Language Models LLM Information Extraction

Ask PDF

Abstract

Clinical Information Extraction

Explore similar work

Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction

Clinical Term Extraction using Open-Source Small Language Models

Benchmarking Large Language Models for Safety Data Extraction