A Proactive EMR Assistant for Doctor-Patient Dialogue: Streaming ASR, Belief Stabilization, and Preliminary Controlled Evaluation

Zhenhai Pan; Yan Liu; Jia You

cs.CLMar 18, 2026

A Proactive EMR Assistant for Doctor-Patient Dialogue: Streaming ASR, Belief Stabilization, and Preliminary Controlled Evaluation

Authors: Zhenhai Pan, Yan Liu, Jia You

Organizations: The Hong Kong Polytechnic University, Hong Kong

Abstract

Most dialogue-based electronic medical record (EMR) systems still behave as passive pipelines: transcribe speech, extract information, and generate the final note after the consultation. That design improves documentation efficiency, but it is insufficient for proactive consultation support because it does not explicitly address streaming speech noise, missing punctuation, unstable diagnostic belief, objectification quality, or measurable next-action gains. We present an end-to-end proactive EMR assistant built around streaming speech recognition, punctuation restoration, stateful extraction, belief stabilization, objectified retrieval, action planning, and replayable report generation. The system is evaluated in a preliminary controlled setting using ten streamed doctor-patient dialogues and a 300-query retrieval benchmark aggregated across dialogues. The full system reaches state-event F1 of 0.84, retrieval Recall@5 of 0.87, and end-to-end pilot scores of 83.3% coverage, 81.4% structural completeness, and 80.0% risk recall. Ablations further suggest that punctuation restoration and belief stabilization may improve downstream extraction, retrieval, and action selection within this pilot. These results were obtained under a controlled simulated pilot setting rather than broad deployment claims, and they should not be read as evidence of clinical deployment readiness, clinical safety, or real-world clinical utility. Instead, they suggest that the proposed online architecture may be technically coherent and directionally supportive under tightly controlled pilot conditions. The present study should be read as a pilot concept demonstration under tightly controlled pilot conditions rather than as evidence of clinical deployment readiness or clinical generalizability.

Figures & tables

Panel of Figure 1 — Figure 1: Online execution pipeline of the proactive EMR assistant.

Method	Coverage (%)	Structural Completeness (%)	Action Utility	Redundancy (%)	Risk Recall (%)	$T_{\mathrm{goal}}$
Baseline A: Direct generation	55.6	59.3	0.41	N/A	30.0	N/A
Baseline B: Chunk-only RAG	69.4	71.4	0.54	24.8	60.0	6.8
Baseline C: Rule-template interaction	77.8	78.6	0.61	28.4	70.0	7.4
Full system	83.3	81.4	0.69	15.8	80.0	5.8

Table 1: End-to-end pilot results in the controlled setting. Higher is better for Coverage, Structural Completeness, Action Utility, and Risk Recall. Lower is better for Redundancy and

T_{\mathrm{goal}}

. Action Utility is a supportive internal control metric and should be interpreted together with Risk Recall, Redundancy, and

T_{\mathrm{goal}}

rather than as a standalone benchmark metric. N/A indicates metrics not applicable to the non-interactive baseline.

Method	Precision	Recall	F1
Baseline B: Chunk-only RAG	0.76	0.72	0.74
Baseline C: Rule-template interaction	0.81	0.79	0.80
Full system	0.85	0.83	0.84

Table 2: Extraction results on gold stateful-event annotations.

Metric	Chunk-only RAG	Hybrid Retrieval
Recall@5	0.770	0.870
MRR@5	0.560	0.710
nDCG@5	0.680	0.790
Object hit rate	0.71	0.83
Path hit rate	0.54	0.73

Table 3: Retrieval results on the 300-query benchmark aggregated across dialogues.

Variant	Volatility	EIG variance	Wrong actions
Raw LLM belief	0.214	0.161	8
+ Temperature scaling	0.173	0.129	6
+ Smoothing	0.141	0.101	5
+ Conservative mode	0.128	0.092	4
Full stabilization	0.118	0.084	4

Table 4: Belief stabilization ablation. Lower is better for all metrics.

Setting	Boundary F1	State F1	Retrieval nDCG	Action Acc.
No punctuation	0.52	0.74	0.66	0.63
Pause-only recovery	0.74	0.80	0.73	0.71
Pause + lexical cues	0.83	0.84	0.79	0.76

Table 5: Impact of punctuation restoration.

Stage	Target latency (ms)	Notes
Streaming ASR chunk emission	250–600	After chunk close; depends on chunk size
Boundary recovery / punctuation	20–60	Lightweight lexical and pause cues
Stateful extraction	120–250	Includes schema normalization
Belief update + gap derivation	20–50	Deterministic plus light probabilistic logic
Hybrid retrieval + reranking	80–180	Single-turn object retrieval budget
Action planning / response generation	150–350	Candidate scoring and short suggestion generation

Table 6: Engineering design budget for single-consultation online latency on the target

3\times

V100 workstation. Values are design ranges for a single active consultation.

Electronic Health Records

Electronic Health Records (EHRs) are longitudinal digital patient records studied for clinical data integration, analysis, and decision support.

See all 111 papers

Mar 18, 2026

Proactive Knowledge Inquiry in Doctor-Patient Dialogue: Stateful Extraction, Belief Updating, and Path-Aware Action Planning

Most automated electronic medical record (EMR) pipelines remain output-oriented: they transcribe, extract, and summarize after the consultation, but they do not explicitly model what is already known, what is still missing, which uncertainty matters most, or what question or recommendation should come next. We formulate doctor-patient dialogue as a proactive knowledge-inquiry problem under partial observability. The proposed framework combines stateful extraction, sequential belief updating, gap-aware state modeling, hybrid retrieval over objectified medical knowledge, and a POMDP-lite action planner. Instead of treating the EMR as the only target artifact, the framework treats documentation as the structured projection of an ongoing inquiry loop. To make the formulation concrete, we report a controlled pilot evaluation on ten standardized multi-turn dialogues together with a 300-query retrieval benchmark aggregated across dialogues. On this pilot protocol, the full framework reaches 83.3% coverage, 80.0% risk recall, 81.4% structural completeness, and lower redundancy than the chunk-only and template-heavy interactive baselines. These pilot results do not establish clinical generalization; rather, they suggest that proactive inquiry may be methodologically interesting under tightly controlled conditions and can be viewed as a conceptually appealing formulation worth further investigation for dialogue-based EMR generation. This work should be read as a pilot concept demonstration under a controlled simulated setting rather than as evidence of clinical deployment readiness. No implication of clinical deployment readiness, clinical safety, or real-world clinical utility should be inferred from this pilot protocol.

Zhenhai Pan, Yan Liu, Jia You

The Hong Kong Polytechnic University, Hong Kong

State Tracking Electronic Health Records Task-Oriented Dialogue Systems Spoken Dialogue Systems

Ask PDF

Apr 8, 2026

EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agents

Conversational diagnosis prediction requires models to track evolving evidence in streaming clinical conversations and decide when to commit to a diagnosis. Existing medical dialogue corpora are largely dyadic or lack the multi-party workflow and annotations needed for this setting. We introduce an ePCR-grounded, topic-flow-based multi-agent generation pipeline that iteratively plans, generates, and self-refines dialogues with rule-based factual and topic flow checks. The pipeline yields EMSDialog, a dataset of 4,414 synthetic multi-speaker EMS conversations based on a real-world ePCR dataset, annotated with 43 diagnoses, speaker roles, and turn-level topics. Human and LLM evaluations confirm high quality and realism of EMSDialog using both utterance- and conversation-level metrics. Results show that EMSDialog-augmented training improves accuracy, timeliness, and stability of EMS conversational diagnosis prediction. Our datasets and code are publicly available at https://uva-dsa.github.io/EMSDialog

Xueren Ge, Sahil Murtaza, Anthony Cortez +1

School of Engineering and Applied Sciences, University of Virginia, Charlottesville, VA, USA · School of Medicine, University of Virginia, Charlottesville, VA, USA

Synthetic Data Generation for Language Models Text Generation

Ask PDF

Jul 19, 2026

Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge

This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), which requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription. After screening open-weight speech LLMs for long-audio robustness, we adapted Voxtral Mini (lightweight track) and Voxtral Small (heavyweight track) with LoRA supervised fine-tuning followed by DAPO reinforcement learning that uses the challenge metric, Open Medical Concept F1, as its reward. Our systems ranked first in both tracks, and an independent LLM-as-a-judge evaluation showed the lowest hallucination rate among all submissions, indicating that reinforcement learning against a concept-matching metric need not compromise factual reliability. We also find that fine-tuning on text transcripts transfers well to speech input and appears to improve robustness on out-of-domain real recordings.

Aivo Olev, Tanel Alumäe

Tallinn University of Technology Tallinn, Estonia

LLM Hallucination Mitigation Text Summarization Speech Language Models Medical LLMs

Ask PDF

Abstract

Electronic Health Records

Explore similar work

Proactive Knowledge Inquiry in Doctor-Patient Dialogue: Stateful Extraction, Belief Updating, and Path-Aware Action Planning

EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agents

Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge