Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals
Authors: Jun-Min Lee, Suyoung Yoo, Gyubok Lee, Edward Choi, Meong Hi Son
Organizations: Department of Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Seongnam-si, 13558, Gyeonggi-do, Republic of Korea. · Research Institute for Future Medicine, Samsung Medical Center, Seoul, 06351, Republic of Korea. · Department of Emergency Medicine, Samsung Medical Center, Seoul, 06351, Republic of Korea. · Office of AI Transformation, Samsung Medical Center, Seoul, 06351, Republic of Korea.
Conversational text-to-SQL interfaces could give healthcare professionals natural-language access to electronic health record (EHR) data, but evidence for their real-world usability remains limited: prior evaluations have relied largely on curated benchmarks with fully specified questions or on simulated users. We report a user study in which 14 physicians who regularly work with EHR data used both a single-turn and a conversational multi-turn text-to-SQL interface over the MIMIC-IV database in a within-subjects design; dialogue analyses additionally drew on interaction logs from 21 healthcare professionals across the main study and two pilot studies. The conversational interface was consistently favored: mean System Usability Scale scores rose from 68.2, near the industry average, to 85.9 (p<0.001), with significant advantages across technology-acceptance constructs including performance expectancy, effort expectancy, self-efficacy, and behavioral intention. In post-study forced-choice comparisons, 93% of participants reported greater overall satisfaction with the conversational interface, 86% preferred it for real-world adoption, and 71% trusted its results more; no participant preferred or trusted the single-turn interface more. Turn-level analysis of 425 logged user turns showed that freely composed queries triggered clarification requests in 24.8% of cases versus 1.8% for predefined, fully specified questions, and 64.3% of clarification episodes stemmed from EHR- or clinical-domain knowledge. Conversational interaction is a practical prerequisite, rather than a convenience, for natural-language EHR access. The frequent need for clarification in realistic clinical queries further underscores the importance of conversational interfaces, while benchmark-style evaluation can substantially underrepresent the ambiguity of real clinical use.
Figures & tables
Figure 1: Overview of the user-study design. Participants completed a within-subject comparison of single-turn and conversational multi-turn text-to-SQL interfaces, consisting of a controlled comparison on pre-defined questions (Phase 1) followed by open-ended querying (Phase 2).
Figure 2: Distribution of participant background characteristics ( n=14 ), including EHR (or EMR)/CDW usage frequency, SQL proficiency, statistical software proficiency, and conversational AI usage frequency.
Figure 3: System usability, technology acceptance, and interface preference for the single-turn versus conversational multi-turn interfaces ( n=14 ). (a) SUS scores (0–100); the dashed line indicates the industry-average score of approximately 68. (b) UTAUT construct scores (1–5): Performance Expectancy (PE), Effort Expectancy (EE), Self-Efficacy (SE), Anxiety (ANX), Behavioral Intention (BI), and the average of the four positively oriented constructs, Avg. UTAUT (P). Bars represent the mean and error bars indicate the standard deviation. Statistical significance is based on the primary paired test for each metric (Section 4.5.2 ): ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 ; n.s., not significant. Higher ANX scores indicate greater anxiety. (c) Distribution of participant responses to the three post-study forced-choice comparison questions ( n=14 ): overall satisfaction, preferred interface for real-world adoption, and trust in the returned results.
Occurrence of Clarification Requests (C-ANS)
Preceding Turn Type
Turn Sequence
C-ANS / Total
Rate
Question (Q)
Q → C-ANS
40 / 161
24.8%
Fixed Question (FX-Q)
FX-Q → C-ANS
1 / 56
1.8%
Suggested Follow-up Question (SF-Q)
SF-Q → C-ANS
1 / 60
1.7%
Clarification Response (CL-Q)
CL-Q → C-ANS
6 / 46
13.0%
Total
48 / 323
14.9%
Table 1: Clarification occurrence and resolution in the conversational multi-turn interface, and user acceptance of agent-proposed follow-up suggestions.
Figure 4: Distribution of the 42 clarification exchanges by dominant trigger category. Categories shaded in blue indicate triggers requiring EHR- or clinical-domain knowledge.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Single-turn Mean (SD)
Multi-turn Mean (SD)
Shapiro–Wilk pSW
Paired t -test
Wilcoxon
t (13)
p
dz
Z
p
r
SUS
68.2 (16.7)
85.9 (11.8)
0.176
−5.09
<0.001
−1.36
−3.30
0.001
−0.88
PE
3.16 (1.24)
4.82 (0.32)
0.045 †
−5.03
<0.001
−1.34
−3.30
0.001
−0.88
EE
4.04 (0.87)
4.52 (0.44)
0.017 †
−2.82
0.014
−0.75
−2.55
0.010
−0.85
SE
3.91 (0.96)
4.55 (0.38)
0.002 †
−2.68
0.019
−0.72
−2.37
0.018
−0.89
ANX
1.89 (0.94)
1.70 (0.65)
0.052
+1.36
0.196 (n.s.)
-
+1.27
0.201 (n.s.)
-
Appendix
Table S2 : Paired comparison of SUS and UTAUT outcomes between the single-turn and conversational multi-turn interfaces ( n=14 ). SD denotes standard deviation. pSW denotes the Shapiro–Wilk test p -value for the paired differences. dz and r denote Cohen’s dz and the Wilcoxon effect size, respectively. n.s., not significant.
Figure S5 : Distribution of self-reported Query Detail Level (QDL) ratings across the three clinical information-seeking scenarios ( n=14 ). Ratings ranged from 1 (vague example phrasing) to 5 (detailed example phrasing). Detailed scenarios and example phrasings are provided in Table 6.3 .
Origin
Type
Definition
User
Question (Q)
A query freely composed by the participant, without predefined constraints on its format or scope.
Fixed Question (FX-Q)
A predefined, highly specified question with an expected answer established in advance by the study investigators.
Suggested Follow-up Question (SF-Q)
A user turn selecting one of the follow-up analysis options suggested by the agent after presenting a result.
Clarification Response (CL-Q)
A user turn providing additional information in response to an agent clarification request, thereby refining the preceding query.
Agent
Answer (ANS)
An agent response containing neither a clarification request nor a follow-up suggestion.
Answer with Suggestion (S-ANS)
An agent response presenting a result together with one or more suggested follow-up analyses. If a suggestion is selected, the subsequent user turn is labeled SF-Q.
Appendix
Table S3 : Definitions of the seven turn types used to annotate dialogue turns.
User Turn
Turn Type
Single-turn
Conversational Multi-turn
Question (Q)
40 (39.2%)
161 (49.8%)
Fixed Question (FX-Q)
62 (60.8%)
56 (17.3%)
Suggested Follow-up Question (SF-Q)
–
60 (18.6%)
Clarification Response (CL-Q)
–
46 (14.2%)
Total
102 (100.0%)
323 (100.0%)
Appendix
Table S4 : Distribution of annotated user-turn and agent-response types by interface.
Category
Coding criterion
Cohort/population definition
The criteria defining the target patient population ( e.g., admission timing, inclusion/exclusion criteria, diagnostic scope) can be interpreted in multiple ways ( WHERE -clause, row-wise ambiguity).
Operational definition of a measure
An everyday-language expression ( e.g., “regional distribution,” “cause of death,” “increasing trend”) requires a concrete computational rule to translate into a SQL expression ( SELECT -clause, column-wise ambiguity).
Aggregation unit/grouping criterion
The unit of analysis ( e.g., patient vs. visit) or grouping criterion is unclear.
Feasibility of the requested DB/analytic method
The requested analysis cannot be performed using the available database, or the proposed statistical method requires revision.
Measurement source/data provenance
The same clinical concept can be derived from multiple data sources or database tables.
Anaphora/contextual reference
The query refers to an entity mentioned earlier in the conversation and cannot be interpreted without dialogue history.
Appendix
Table S5 : Coding criteria for the six qualitative categories of clarification triggers. Categories shown in italics were considered related to EHR- or clinical-domain knowledge.
Category
Count (%)
Representative User Query
Source of Ambiguity
Cohort/population definition
15 (35.7%)
“Patients most recently admitted to the ICU for pneumonia…”
What defines “most recently,” and whether the lookback window should be anchored to the admission date or the current date.
Operational definition of a measure
10 (23.8%)
“I want to know the regional distribution of patients who visited the ED.”
What “regional distribution” refers to, such as residence, race, referral pathway, or insurance, and whether counts should be patient- or visit-based.
Aggregation unit/grouping criterion
8 (19.0%)
“Recalculate this for in-hospital deaths, and organize the distribution of the time interval between admission and death by age group.”
The ambiguity lies in the aggregation level: whether to report summary statistics by age group, a distribution across time intervals, or individual-level values.
Feasibility of requested DB/analytic method
5 (11.9%)
“Which country’s microbial frequency is most similar to that of this ICU?”
Country-level comparison data are unavailable in the database, requiring clarification of the intended comparison strategy or external reference source.
Measurement source/data provenance
2 (4.8%)
“What percentage of patients received a potassium chloride prescription (administration) within six hours of a reported potassium level of 3.5 mEq/L?”
Whether “prescription” refers to a medication order ( prescriptions ) or actual administration ( emar ).
Anaphora/contextual reference
2 (4.8%)
“Test whether the association above is statistically significant.”
Which previously discussed association is referenced by “above.”
Appendix
Table S6 : Qualitative classification of the 42 clarification exchanges by the dominant trigger of the initial clarification request, with representative user queries and ambiguity details. Categories shown in italics were considered related to EHR- or clinical-domain knowledge.
Model
Avg. F1
Pass@1
Pass@5
Pass^5
Avg. Latency (s)
Max Latency (s)
GPT-5.2
65.4%
42.2%
55.6%
22.2%
5.74
10.86
GPT-5.4
70.4%
48.0%
57.8 %
33.3%
3.21
4.95
Appendix
Table S7 : Comparison of GPT-5.2 and GPT-5.4 on the 45 Phase 1 fixed questions (FXQ; Table ). Each question was executed five times under the single-turn interface (225 trials per model). Accuracy was evaluated by comparing the generated SQL execution results with the corresponding gold result tables.
Category
Total Turns
Correct Turns
Accuracy
FXQ-turns
45
28
62.2%
Others (Q + SFQ + CLQ)
141
106
75.2%
All judgable turns
186
134
72.0%
Appendix
Table S8 : Human-labeled correctness of GPT-5.4’s conversational multi-turn responses, judged against the full preceding dialogue context, across all real Phase 2 sessions collected from the 14 main-study participants.
Figure S6 : Web-based interface used in the user study. The left panel provides a schema browser listing database tables by category. The top-right panel displays the selected table’s description, column-level schema, keyword search, and paginated raw data. The bottom-right panel provides the query interface, including model selection, interaction mode (single-turn or conversational multi-turn), and the conversation history.
Extracting insights from Electronic Health Record (EHR) databases often requires SQL expertise, creating a barrier for clinical decision-making and research. A promising approach is to use Large Language Models (LLMs) to translate natural language questions into SQL through Retrieval-Augmented Generation (RAG), where relevant question-SQL examples are retrieved to generate new queries via few-shot learning. However, adapting this method to the medical domain is non-trivial, as effective retrieval requires examples that align with both the logical structure of the question and its referenced entities (e.g., drug names, procedure titles). Standard single-step RAG struggles to optimize both aspects simultaneously and often relies on near-exact matches to generalize effectively. This issue is especially severe in healthcare, as questions often contain noisy and inconsistent medical jargon. To address this, we present CBR-to-SQL, a framework inspired by Case-based Reasoning theory that decomposes RAG's single-step retrieval into two explicit stages: one that focuses on retrieving structurally relevant examples, and one that aligns entities with the target database schema. Evaluated on two clinical benchmarks, CBR-to-SQL achieves competitive accuracies compared to fine-tuned methods. More importantly, it demonstrates considerably higher sample efficiency and robustness than the standard RAG approach, particularly under data scarcity and retrieval perturbations.
Hung Nguyen, Hans Moen, Pekka Marttinen
Department of Computer Science Aalto University · Department of Clinical Medicine Aalborg University
Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.
Yitong Qiao, Lei Liu, Yue Shen +4
1Zhejiang University · †Work done during an internship at Ant Group. · 2Ant Group
Patient portals now give individuals direct access to their electronic health records (EHRs), yet access alone does not ensure patients understand or act on the complex clinical information contained in these records. The ArchEHR-QA 2026 shared task addresses this challenge by focusing on grounded question answering over EHRs, and this paper presents the system developed by the HealthNLP_Retrievers team for this task. The proposed approach uses a multi-stage cascaded pipeline powered by the Gemini 2.5 Pro large language model to interpret patient-authored questions and retrieve relevant evidence from lengthy clinical notes. Our architecture comprises four integrated modules: (1) a few-shot query reformulation unit which summarizes verbose patient queries; (2) a heuristic-based evidence scorer which ranks clinical sentences to prioritize recall; (3) a grounded response generator which synthesizes professional-caliber answers restricted strictly to identified evidence; and (4) a high-precision many-to-many alignment framework which links generated answers to supporting clinical sentences. This cascaded approach achieved competitive results. Across the individual tracks, the system ranked 1st in question interpretation, 5th in answer generation, 7th in evidence identification, and 9th in answer-evidence alignment. These results show that integrating large language models within a structured multi-stage pipeline improves grounding, precision, and the professional quality of patient-oriented health communication. To support reproducibility, our source code is publicly available in our GitHub repository