From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

Bayram Ayadi; Florian Geissler; Johanna Schmidhuber

cs.CLOct 22, 2026

From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

Authors: Bayram Ayadi, Florian Geissler, Johanna Schmidhuber

Organizations: University of Passau, Passau, Germany · Fraunhofer Institute for Cognitive Systems IKS, Munich, Germany

Abstract

As wearable and digital health systems increasingly generate longitudinal physiological records, communicating these observations in patient-accessible language remains a significant challenge. Large Language Models (LLMs) offer a promising mechanism for translating structured clinical information into readable narratives, but this process may introduce semantic drift, where clinically precise meaning is distorted during simplification. We investigate evidence-grounded health summarization for menstrual health monitoring using a constrained generation framework that combines structured evidence conditioning with Retrieval-Augmented Generation (RAG) using a prompt-based constrained generation framework. We evaluate semantic alignment using Natural Language Inference (NLI), atomic factuality metrics, and readability measures. Across three LLMs and 4,500 generated summaries, explicit grounding constraints consistently improve clinical alignment, particularly by improving uncertainty preservation and, once faithfulness is evaluated against retrieved evidence, generally preserving or improving faithfulness, while effects on omission remain model-dependent. We also find that improved clinical alignment comes at the cost of increased reading difficulty and medical terminology. These findings highlight the need to jointly evaluate faithfulness, uncertainty preservation, and readability when developing patient-facing medical LLM systems.

Figures & tables

Figure 1: Overview of the pipeline and five-category semantic alignment evaluation framework.

Alignment Condition	Definition	Example	Metric	Related Work
Faithfulness Violation	Unsupported physiological claims generated beyond the scope of the structured evidence	Stating elevated HRV when only temperature data is present	Eq. 2	Faithfulness and factuality Maynez et al. (2020)
Epistemic misalignment	Amplification of uncertain observations into definitive statements	“Your cycle is irregular” instead of “your cycle may be irregular”	Eq. 3	Clark et al. (2025)
Diagnostic inference overreach	Implication of pathology unsupported by the underlying evidence	Suggesting endometriosis from mild cramp annotations	Eq. 4	Zhu et al. (2026)
Omission Drift	Removal of clinically relevant nuance during simplification	Dropping confidence scores or phase uncertainty from the summary	Eq. 5	Oukelmoun et al. (2025)
Urgency Misalignment	Unnecessarily alarming or over-reassuring language	“This is completely normal” for an ambiguous HRV drop	Eq. 6

Table 1: Semantic alignment taxonomy covering five clinical alignment conditions, their definitions, illustrative examples, formal metrics, and connections to related literature concepts.

Model	Setting	$M_{\text{faith}}\downarrow$	$M_{\text{epist}}\downarrow$	$M_{\text{omit}}\downarrow$	$M_{\text{urg}}$
GPT-4o	Base	0.264	0.293	0.219	0.170
	Constrained	0.186	0.243	0.195	0.227
	RAG-Constrained	0.152	0.237	0.130	0.179
Qwen2.5-14B	Base	0.190	0.299	0.176	0.247
	Constrained	0.165	0.275	0.147	0.270
	RAG-Constrained	0.162	0.251	0.167	0.199

Table 2: Clinical alignment metrics by model and generation setting, ordered by model scale (large to small).

M_{\text{faith}}

: faithfulness violation rate (RAG-constrained: entailed by

X

or retrieved context);

M_{\text{epist}}

: epistemic misalignment;

M_{\text{diag}}

: diagnostic inference overreach (corrected, excluding pregnancy/ectopic pregnancy mentions);

M_{\text{omit}}

: omission drift;

M_{\text{urg}}

: urgency misalignment. All metrics range

[0,1]

with lower values indicating better alignment.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Phase	Poor	Moderate	Good
Menstrual	0.35	0.40	0.25
Follicular	0.08	0.32	0.60
Ovulatory	0.12	0.40	0.48
Luteal	0.32	0.43	0.25

Appendix

Table 3: Conditional probability table:

P(\text{sleep quality}\mid\text{phase})

Phase	$\mu$	$\sigma$	Min	Max
Menstrual	63.0	3.0	57	71
Follicular	60.0	2.5	55	66
Ovulatory	64.0	3.0	58	72
Luteal	67.0	3.5	61	78

Appendix

Table 4: RHR baseline parameters by phase (bpm)

Profile	$N$	Mean Fields
Minimal tracker	75	$\sim$ 2–3
App logger	100	$\sim$ 4–5
Basic wearable	125	$\sim$ 6–8
Advanced wearable	125	$\sim$ 9–12
Inconsistent user	75	$\sim$ 3–5

Appendix

Table 5: User profile distribution and data access characteristics

Model	Metric	Base	Constrained	RAG-constrained
GPT-4o	FKGL	7.38	9.93	10.31
GPT-4o	ARI	8.35	10.97	10.83
GPT-4o	Dale-Chall	9.67	10.60	10.54
Phi-4-mini	FKGL	10.63	12.18	12.32
Phi-4-mini	ARI	12.63	14.13	14.29
Phi-4-mini	Dale-Chall	9.84	10.44	10.38

Appendix

Table 6: Mean grade-level readability scores by model and generation setting.

Panel of Figure 2 — Figure 2: Distribution of jargon density, raw medical entity counts, and entities per sentence across generation conditions (base, constrained, rag_constrained) for all evaluated models.

Figure 3: Pooled pairwise correlations ( $r$ ) across seven readability, lexical, and syntactic-complexity metrics computed over all $4,500$ generated summaries. Grade-level indices are strongly intercorrelated ( $r\geq 0.87$ ).

Text Summarization

Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.

See all 142 papers

Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
Jul 10, 2026Robert WilliamsHallucination in Language Models Text Summarization
University of Texas at Austin Computer & Data Science Online Austin, Texas, USA
Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation
Sep 11, 2026Paul Landes, Sitara Rao, Aaron Jeremy Chaise +1Medical Report Generation LLM Grounding
Department of Computer Science University of Illinois Chicago Chicago, USA · College of Medicine University of Illinois Chicago Chicago, USA
CARE: A Conformal Safety Layer for Medical Summarization
Jun 8, 2026Suhana Bedi, Bridget Lin, Anson Y. Zhou +5Text Summarization Conformal Risk Control
Department of Biomedical Data Science, Stanford University · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University · Department of Computer Science, Stanford University +2

Jul 10, 2026J/K move · Enter open · S save

Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences

Robert Williams

University of Texas at Austin Computer & Data Science Online Austin, Texas, USA

Hallucination in Language Models Text Summarization RAG Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large language models are increasingly used to summarize clinical trial results for healthcare providers, patients, and payers, but their tendency to hallucinate poses significant risks in this high-stakes context. This study introduces a benchmark evaluation framework for measuring the faithfulness of LLM-generated clinical trial summaries across three stakeholder audiences. The framework consists of 200 stratified trials drawn from the Aggregate Analysis of ClinicalTrials.gov database, evaluated using audience-specific prompt templates and a six-dimension faithfulness annotation schema. Baseline measurements were established for GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Flash across 1,800 generated summaries scored using a cross-encoder natural language inference (NLI) model. Unsupported Claims was identified as the dominant failure mode across all three models, with a mean annotation score of 1.55 out of three. A knowledge-graph-augmented retrieval system was developed and evaluated against the baseline, producing statistically significant improvements in NLI-based faithfulness scores (entailment +0.0125, faithfulness +0.0130, p < 0.0001). Improvement pathways were model-dependent, with GPT-4o improving primarily through contradiction reduction while Claude Sonnet 4.6 and Gemini 2.5 Flash improved through increased entailment.

Abstract

Text Summarization

Explore similar work

Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences

Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation

CARE: A Conformal Safety Layer for Medical Summarization