Retrieval-Augmented Generation Based Nurse Observation Extraction

Kyomin Hwang; Nojun Kwak

cs.CLMar 27, 2026

Retrieval-Augmented Generation Based Nurse Observation Extraction

Authors: Kyomin Hwang, Nojun Kwak

Organizations: GSCST, Seoul National University · AIIS, Seoul National University

Abstract

Recent advancements in Large Language Models (LLMs) have played a significant role in reducing human workload across various domains, a trend that is increasingly extending into the medical field. In this paper, we propose an automated pipeline designed to alleviate the burden on nurses by automatically extracting clinical observations from nurse dictations. To ensure accurate extraction, we introduce a method based on Retrieval-Augmented Generation (RAG). Our approach demonstrates effective performance, achieving an F1-score of 0.796 on the MEDIQA-SYNUR test dataset.

Figures & tables

Figure 1: Full illustration of Retrieval-Augmented Generation (RAG) Based Nurse Observation Extraction Pipeline

Figure 2: Prompt for nurse dictation segmentation

Figure 3: Prompt for description generation

Figure 4: Prompt for segment generation for train dataset

Figure 5: Segment Example for Train Dataset.

Figure 6: Prompt for observation generation

Method	Precision	Recall	F1
+ Few-shot Ex	0.789	0.694	0.739
+ Few-shot Ex + Schema	0.812	0.847	0.829

Table 1: Performance comparison of the proposed methods. Best results are marked in bold .

Base Model	Shots	LLM Backbone		Metrics
Base Model	Shots	Generator	Segmenter	Precision	Recall	F1
BlueBert	3	GPT-5-mini	GPT-5-mini	0.739	0.810	0.773
	5	GPT-5-mini	GPT-5-mini	0.744	0.817	0.779
	10	GPT-5-mini	GPT-5-mini	0.744	0.818	0.779
	10	GPT-5.1	GPT-5-mini	0.785	0.805	0.795
	10	GPT-5.2	GPT-5-mini	0.766	0.825	0.794

Table 2: Performance comparison with different configurations on the Test Dataset. †denotes the usage of refined prompt.

Method	Precision	Recall	F1
ClinicalBert	0.811	0.845	0.828
BlueBert	0.807	0.860	0.833
PubMedBert	0.805	0.842	0.823

Table 3: Ablation on retrieval model.

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction
May 14, 2026A H M Rezaul Karim, Ozlem UzunerRetrieval-Augmented Generation Healthcare
George Mason University, VA, USA
Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents
Oct 6, 2026Lasse B. Strand, Robert Jakob, Kevin O'Sullivan +1Retrieval-Augmented Generation Large Language Model-Guided Optimization
ETH Zurich
LLMs can construct powerful representations and streamline sample-efficient supervised learning
Mar 12, 2026Ilker Demirel, Lawrence Shi, Zeshan Hussain +1Large Language Model-Based Feature Engineering Data-Efficient Learning
MIT · Harvard Medical School

May 14, 2026J/K move · Enter open · S save

Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction

A H M Rezaul Karim, Ozlem Uzuner

George Mason University, VA, USA

Retrieval-Augmented Generation Healthcare Schema-Constrained Generation Clinical Information Extraction LLM Information Extraction Text Generation

Open paper AskPDF arXiv

Conversational nurse-patient transcripts contain actionable observations, but converting these transcripts into structured representations at scale remains challenging. Documentation burden is substantial, with prior studies showing clinicians spend large portions of their workday on documentation and related desk work rather than direct patient care. MEDIQA-SYNUR focuses on observation extraction from conversational nurse-patient transcripts, requiring systems to normalize these narratives into a predefined schema with value-type constraints. We propose a modular retrieval-augmented generation (RAG) pipeline that uses the training set as an exemplar corpus, combines schema-constrained prompting (full schema vs. pruned candidate schema), deterministic schema-based postprocessing, and a second-pass audit, with two LLM backbones: Llama-4-Scout-17B-16E-Instruct and GPT-5.2 with corresponding embedding models for RAG. Our best configuration uses GPT-5.2 with full schema, RAG, and a second-pass auditing, achieving 80.36% F1 score. Overall, our results show that RAG consistently improves performance, while the optimal degree of schema constraint depends on the model, and second-pass auditing yields modest additional gains by correcting residual schema-adherence errors.

Abstract

Clinical Information Extraction

Explore similar work

Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

LLMs can construct powerful representations and streamline sample-efficient supervised learning