Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals

Jun-Min Lee; Suyoung Yoo; Gyubok Lee; Edward Choi; Meong Hi Son

cs.AIOct 31, 2026

Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals

Authors: Jun-Min Lee, Suyoung Yoo, Gyubok Lee, Edward Choi, Meong Hi Son

Organizations: Department of Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Seongnam-si, 13558, Gyeonggi-do, Republic of Korea. · Research Institute for Future Medicine, Samsung Medical Center, Seoul, 06351, Republic of Korea. · Department of Emergency Medicine, Samsung Medical Center, Seoul, 06351, Republic of Korea. · Office of AI Transformation, Samsung Medical Center, Seoul, 06351, Republic of Korea.

Abstract

Conversational text-to-SQL interfaces could give healthcare professionals natural-language access to electronic health record (EHR) data, but evidence for their real-world usability remains limited: prior evaluations have relied largely on curated benchmarks with fully specified questions or on simulated users. We report a user study in which 14 physicians who regularly work with EHR data used both a single-turn and a conversational multi-turn text-to-SQL interface over the MIMIC-IV database in a within-subjects design; dialogue analyses additionally drew on interaction logs from 21 healthcare professionals across the main study and two pilot studies. The conversational interface was consistently favored: mean System Usability Scale scores rose from 68.2, near the industry average, to 85.9 (p<0.001), with significant advantages across technology-acceptance constructs including performance expectancy, effort expectancy, self-efficacy, and behavioral intention. In post-study forced-choice comparisons, 93% of participants reported greater overall satisfaction with the conversational interface, 86% preferred it for real-world adoption, and 71% trusted its results more; no participant preferred or trusted the single-turn interface more. Turn-level analysis of 425 logged user turns showed that freely composed queries triggered clarification requests in 24.8% of cases versus 1.8% for predefined, fully specified questions, and 64.3% of clarification episodes stemmed from EHR- or clinical-domain knowledge. Conversational interaction is a practical prerequisite, rather than a convenience, for natural-language EHR access. The frequent need for clarification in realistic clinical queries further underscores the importance of conversational interfaces, while benchmark-style evaluation can substantially underrepresent the ambiguity of real clinical use.

Figures & tables

Figure 1: Overview of the user-study design. Participants completed a within-subject comparison of single-turn and conversational multi-turn text-to-SQL interfaces, consisting of a controlled comparison on pre-defined questions (Phase 1) followed by open-ended querying (Phase 2).

Figure 2: Distribution of participant background characteristics ( $n=14$ ), including EHR (or EMR)/CDW usage frequency, SQL proficiency, statistical software proficiency, and conversational AI usage frequency.

Figure 3: System usability, technology acceptance, and interface preference for the single-turn versus conversational multi-turn interfaces (

n=14

). (a) SUS scores (0–100); the dashed line indicates the industry-average score of approximately 68. (b) UTAUT construct scores (1–5): Performance Expectancy (PE), Effort Expectancy (EE), Self-Efficacy (SE), Anxiety (ANX), Behavioral Intention (BI), and the average of the four positively oriented constructs, Avg. UTAUT (P). Bars represent the mean and error bars indicate the standard deviation. Statistical significance is based on the primary paired test for each metric (Section 4.5.2 ):

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

; n.s., not significant. Higher ANX scores indicate greater anxiety. (c) Distribution of participant responses to the three post-study forced-choice comparison questions (

n=14

): overall satisfaction, preferred interface for real-world adoption, and trust in the returned results.

Occurrence of Clarification Requests (C-ANS)
Preceding Turn Type	Turn Sequence	C-ANS / Total	Rate
Question (Q)	Q $\rightarrow$ C-ANS	40 / 161	24.8%
Fixed Question (FX-Q)	FX-Q $\rightarrow$ C-ANS	1 / 56	1.8%
Suggested Follow-up Question (SF-Q)	SF-Q $\rightarrow$ C-ANS	1 / 60	1.7%
Clarification Response (CL-Q)	CL-Q $\rightarrow$ C-ANS	6 / 46	13.0%
Total		48 / 323	14.9%

Table 1: Clarification occurrence and resolution in the conversational multi-turn interface, and user acceptance of agent-proposed follow-up suggestions.

Figure 4: Distribution of the 42 clarification exchanges by dominant trigger category. Categories shaded in blue indicate triggers requiring EHR- or clinical-domain knowledge.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Single-turn Mean (SD)	Multi-turn Mean (SD)	Shapiro–Wilk $\boldsymbol{p_{\mathrm{SW}}}$	Paired $t$ -test			Wilcoxon
Metric	Single-turn Mean (SD)	Multi-turn Mean (SD)	Shapiro–Wilk $\boldsymbol{p_{\mathrm{SW}}}$	$t$ (13)	$p$	$d_{z}$	$Z$	$p$	$r$
SUS	68.2 (16.7)	85.9 (11.8)	0.176	$-5.09$	$<0.001$	$-1.36$	$-3.30$	0.001	$-0.88$
PE	3.16 (1.24)	4.82 (0.32)	0.045 †	$-5.03$	$<0.001$	$-1.34$	$-3.30$	0.001	$-0.88$
EE	4.04 (0.87)	4.52 (0.44)	0.017 †	$-2.82$	0.014	$-0.75$	$-2.55$	0.010	$-0.85$
SE	3.91 (0.96)	4.55 (0.38)	0.002 †	$-2.68$	0.019	$-0.72$	$-2.37$	0.018	$-0.89$
ANX	1.89 (0.94)	1.70 (0.65)	0.052	$+1.36$	0.196 (n.s.)	-	$+1.27$	0.201 (n.s.)	-

Appendix

Table S2 : Paired comparison of SUS and UTAUT outcomes between the single-turn and conversational multi-turn interfaces (

n=14

). SD denotes standard deviation.

p_{\mathrm{SW}}

denotes the Shapiro–Wilk test

p

-value for the paired differences.

d_{z}

and

r

denote Cohen’s

d_{z}

and the Wilcoxon effect size, respectively. n.s., not significant.

Figure S5 : Distribution of self-reported Query Detail Level (QDL) ratings across the three clinical information-seeking scenarios ( $n=14$ ). Ratings ranged from 1 (vague example phrasing) to 5 (detailed example phrasing). Detailed scenarios and example phrasings are provided in Table 6.3 .

Origin	Type	Definition
User	Question (Q)	A query freely composed by the participant, without predefined constraints on its format or scope.
	Fixed Question (FX-Q)	A predefined, highly specified question with an expected answer established in advance by the study investigators.
	Suggested Follow-up Question (SF-Q)	A user turn selecting one of the follow-up analysis options suggested by the agent after presenting a result.
	Clarification Response (CL-Q)	A user turn providing additional information in response to an agent clarification request, thereby refining the preceding query.
Agent	Answer (ANS)	An agent response containing neither a clarification request nor a follow-up suggestion.
	Answer with Suggestion (S-ANS)	An agent response presenting a result together with one or more suggested follow-up analyses. If a suggestion is selected, the subsequent user turn is labeled SF-Q.

Appendix

Table S3 : Definitions of the seven turn types used to annotate dialogue turns.

User Turn
Turn Type	Single-turn	Conversational Multi-turn
Question (Q)	40 (39.2%)	161 (49.8%)
Fixed Question (FX-Q)	62 (60.8%)	56 (17.3%)
Suggested Follow-up Question (SF-Q)	–	60 (18.6%)
Clarification Response (CL-Q)	–	46 (14.2%)
Total	102 (100.0%)	323 (100.0%)

Appendix

Table S4 : Distribution of annotated user-turn and agent-response types by interface.

Category	Coding criterion
Cohort/population definition	The criteria defining the target patient population ( e.g., admission timing, inclusion/exclusion criteria, diagnostic scope) can be interpreted in multiple ways ( WHERE -clause, row-wise ambiguity).
Operational definition of a measure	An everyday-language expression ( e.g., “regional distribution,” “cause of death,” “increasing trend”) requires a concrete computational rule to translate into a SQL expression ( SELECT -clause, column-wise ambiguity).
Aggregation unit/grouping criterion	The unit of analysis ( e.g., patient vs. visit) or grouping criterion is unclear.
Feasibility of the requested DB/analytic method	The requested analysis cannot be performed using the available database, or the proposed statistical method requires revision.
Measurement source/data provenance	The same clinical concept can be derived from multiple data sources or database tables.
Anaphora/contextual reference	The query refers to an entity mentioned earlier in the conversation and cannot be interpreted without dialogue history.

Appendix

Table S5 : Coding criteria for the six qualitative categories of clarification triggers. Categories shown in italics were considered related to EHR- or clinical-domain knowledge.

Category	Count (%)	Representative User Query	Source of Ambiguity
Cohort/population definition	15 (35.7%)	“Patients most recently admitted to the ICU for pneumonia…”	What defines “most recently,” and whether the lookback window should be anchored to the admission date or the current date.
Operational definition of a measure	10 (23.8%)	“I want to know the regional distribution of patients who visited the ED.”	What “regional distribution” refers to, such as residence, race, referral pathway, or insurance, and whether counts should be patient- or visit-based.
Aggregation unit/grouping criterion	8 (19.0%)	“Recalculate this for in-hospital deaths, and organize the distribution of the time interval between admission and death by age group.”	The ambiguity lies in the aggregation level: whether to report summary statistics by age group, a distribution across time intervals, or individual-level values.
Feasibility of requested DB/analytic method	5 (11.9%)	“Which country’s microbial frequency is most similar to that of this ICU?”	Country-level comparison data are unavailable in the database, requiring clarification of the intended comparison strategy or external reference source.
Measurement source/data provenance	2 (4.8%)	“What percentage of patients received a potassium chloride prescription (administration) within six hours of a reported potassium level of 3.5 mEq/L?”	Whether “prescription” refers to a medication order ( prescriptions ) or actual administration ( emar ).
Anaphora/contextual reference	2 (4.8%)	“Test whether the association above is statistically significant.”	Which previously discussed association is referenced by “above.”

Appendix

Table S6 : Qualitative classification of the 42 clarification exchanges by the dominant trigger of the initial clarification request, with representative user queries and ambiguity details. Categories shown in italics were considered related to EHR- or clinical-domain knowledge.

Model	Avg. F1	Pass@1	Pass@5	Pass^5	Avg. Latency (s)	Max Latency (s)
GPT-5.2	65.4%	42.2%	55.6%	22.2%	5.74	10.86
GPT-5.4	70.4%	48.0%	57.8 %	33.3%	3.21	4.95

Appendix

Table S7 : Comparison of GPT-5.2 and GPT-5.4 on the 45 Phase 1 fixed questions (FXQ; Table ). Each question was executed five times under the single-turn interface (225 trials per model). Accuracy was evaluated by comparing the generated SQL execution results with the corresponding gold result tables.

Category	Total Turns	Correct Turns	Accuracy
FXQ-turns	45	28	62.2%
Others (Q + SFQ + CLQ)	141	106	75.2%
All judgable turns	186	134	72.0%

Appendix

Table S8 : Human-labeled correctness of GPT-5.4’s conversational multi-turn responses, judged against the full preceding dialogue context, across all real Phase 2 sessions collected from the 14 main-study participants.

Figure S6 : Web-based interface used in the user study. The left panel provides a schema browser listing database tables by category. The top-right panel displays the selected table’s description, column-level schema, keyword search, and paginated raw data. The bottom-right panel provides the query interface, including model selection, interaction mode (single-turn or conversational multi-turn), and the conversation history.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 185 papers

CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain
Mar 5, 2026Hung Nguyen, Hans Moen, Pekka MarttinenCase-Based Reasoning Text-to-SQL
Department of Computer Science Aalto University · Department of Clinical Medicine Aalborg University
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
Jun 22, 2026Yitong Qiao, Lei Liu, Yue Shen +4AI Agent Benchmarks Clinical Reasoning
1Zhejiang University · †Work done during an internship at Ant Group. · 2Ant Group
HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering
Apr 29, 2026Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud +3LLM Grounding Healthcare
University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh

Mar 5, 2026J/K move · Enter open · S save

CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain

Hung Nguyen, Hans Moen, Pekka Marttinen

Department of Computer Science Aalto University · Department of Clinical Medicine Aalborg University

Case-Based Reasoning Text-to-SQL Retrieval-Augmented Generation Electronic Health Records

Open paper AskPDF arXiv

Extracting insights from Electronic Health Record (EHR) databases often requires SQL expertise, creating a barrier for clinical decision-making and research. A promising approach is to use Large Language Models (LLMs) to translate natural language questions into SQL through Retrieval-Augmented Generation (RAG), where relevant question-SQL examples are retrieved to generate new queries via few-shot learning. However, adapting this method to the medical domain is non-trivial, as effective retrieval requires examples that align with both the logical structure of the question and its referenced entities (e.g., drug names, procedure titles). Standard single-step RAG struggles to optimize both aspects simultaneously and often relies on near-exact matches to generalize effectively. This issue is especially severe in healthcare, as questions often contain noisy and inconsistent medical jargon. To address this, we present CBR-to-SQL, a framework inspired by Case-based Reasoning theory that decomposes RAG's single-step retrieval into two explicit stages: one that focuses on retrieving structurally relevant examples, and one that aligns entities with the target database schema. Evaluated on two clinical benchmarks, CBR-to-SQL achieves competitive accuracies compared to fine-tuned methods. More importantly, it demonstrates considerably higher sample efficiency and robustness than the standard RAG approach, particularly under data scarcity and retrieval perturbations.

Abstract

Text-to-SQL

Explore similar work

CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering