A prospective randomized study of patient-centered AI for clinical visit preparation
Authors: Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan, Roma Ruparel, Michael A. Sanchez, Sean Chang, Joseph Xu, Bhavna Daryani, Preeti Singh, Pinal Bavishi, Kejia Chen, Fan Zhang, Priya Gupta, Ernest Lim, Christina Chen, Bob Lou, Dimitrios Antos, Zach Wasson, Ray Luo, Fadi Yousif, Florence Thng, Mark Malhotra, Joseph Breda, Daniel McDuff, Kavi Goel, Katherine Chou, Dale R. Webster, Avinatan Hassidim, Yossi Matias, Po-Hsuan Cameron Chen, Yun Liu, Mike Schaekermann, Quang Duong, Jackie M. Barr
Patients often arrive at brief, time-constrained clinical visits unsure of what to expect or ask, and millions of adults now turn to general-purpose AI chatbots to prepare beforehand. Yet whether conversational AI improves visit readiness remains unclear. We designed a large language model (LLM)-based visit preparation agent to guide patients before an appointment by gathering clinical context, setting visit expectations, providing relevant health information, building a prioritized list of questions for their clinician, and generating a summary. In a prospective consented randomized study in the Google Health app, 802 U.S. adults held 866 health-related conversations with the visit preparation agent (standard or concise variant) or a minimally prompted control chatbot built on the same LLM. The visit preparation agent significantly outperformed the control across blinded clinician evaluations and patient self-reports. In clinician reviews of completed sessions (n = 272), the agent achieved a 16.6% higher rate of comprehensive preparation (67.2% vs. 50.6%; P < 0.05) and an 8.5% higher rate of alignment with patients' health context (98.9% vs. 90.4%; P < 0.01), while maintaining high medical safety and appropriateness across all study arms. Patient post-surveys (n = 460) showed 16.8% higher visit preparedness (73.8% vs. 57.0%; P < 0.01), 14.3% higher confidence in knowing what to ask (81.4% vs. 67.1%; P < 0.01), and 12.0% higher overall satisfaction (74.0% vs. 62.0%; P < 0.05). Clinician and participant metrics were not significantly correlated, indicating that expert review and patient self-assessment capture complementary dimensions of patient-facing AI. These findings suggest that LLM chatbots can safely help patients prepare for clinical visits, with greater benefit when following a structured workflow, and motivate future studies on in-visit communication and clinical outcomes.
Figures & tables
Figure 1: Overview of Contributions. This work provides the first prospective assessment of AI-based visit preparation in a consumer health application, enabled by three key contributions: (A) A structured visit preparation agent powered by Gemini designed to guide patients through a structured pre-visit preparation flow (the user interface shown is trimmed for brevity and clarity; see for more details); (B) A prospective randomized study comparing the visit preparation agent with a control LLM agent; (C) Clinical evaluation suggesting increased clinical utility while maintaining high clinical reliability, with increases in patient-reported visit preparedness and visit self-efficacy.
Figure 2: Study Design. Step 1: Participants were randomized 1:2:2 to a minimally prompted base LLM (Control) or to one of two structured visit preparation agents (standard or concise) for a pre-visit conversation. All three agents used the same LLM and interface, differed only in their system prompts, and each maintained a pre-visit summary (full prompts in ). Step 2: After the session, participants rated their visit preparedness, visit self-efficacy, satisfaction, and likeliness of reuse in a post-survey, and 14 days later were invited to a follow-up survey on whether they had a visit, whether the visit matched the expectations discussed, and whether they used the prep topics. Step 3: In a separate retrospective evaluation, a random sample of completed sessions (conversation and summary), stratified by arm, was assigned at random in a balanced manner to three PCP clinicians blinded to the study design and LLM associated with the conversation, who rated each agent task on four clinical rubrics.
Figure 3: CONSORT Flow Diagram and Participant Characteristics by Arm. Top: flow of participants (n) and conversations (N) from exposure to the study entry point, through 1:2:2 randomization to Control or to the standard or concise visit preparation agent, to exclusions and inclusion in the analysis. Excluded conversations comprised empty or non-health dialogues, conversations entered outside the intended study pathway, and conversations centered on ineligible chief complaints identified by an LLM and verified via clinician review. In the primary analysis, the two visit preparation intervention arms were pooled. Within each study arm, the analytic cohort reflects conversations that completed clinical evaluation, immediate post-survey, and 14-day follow-up reports. Bottom: demographics (age range and sex), visit type, and disease area of the included conversations, shown as N (%) by arm. Visit type and disease area were parsed via LLM from each conversation.
Figure 4: Clinical Evaluation Results. Top: Share of sessions rated favorably (rating of 4 or 5) on each clinical dimension, overall (all applicable steps rated favorably) and for each preparation step. Bars to the left of 0% show ratings of 1–3. The visit preparation arm combines the concise and standard arms. Error bars indicate 95% Wilson confidence intervals. Asterisks mark significant differences between the visit preparation and Control arms ( ∗P<0.05 ; ∗∗P<0.01 ): for the overall rows, in the share of sessions rated favorably (two-sided two-proportion z test); for individual preparation steps, in how ratings were distributed across the 1–5 scale (Mann–Whitney U test). P values are BH-adjusted within each clinical dimension. Bottom: Sessions grouped by clinical reliability (medical safety and appropriateness) and clinical utility (comprehensiveness and patient alignment), where high indicates that all ratings in that category were 4 or higher. P values compare the two arms on the share of sessions that were both highly reliable and useful, and on the share that were highly reliable but not highly useful (two-sided two-proportion z tests, unadjusted).
Figure 5: Patient-reported Outcomes. Distribution of ratings in the immediate post-survey (left and middle) and the exploratory 14-day follow-up survey (right; limited to respondents who had a doctor visit: Control, 13; Visit preparation, 46); the visit preparation arm combines the concise and standard arms. Bars to the right of 0% show positive responses (rating of 4 or 5; for agenda topic use, the share of participants who reported using them), with the total share of positive responses labeled; bars to the left show neutral and negative ratings. Error bars indicate 95% Wilson CI. ∗P<0.05 ; ∗∗P<0.01 (two-proportion z tests with BH correction across the five immediate survey items).
Figure 6: Factors Associated with Positive Visit Readiness and Confidence. Adjusted odds ratios (log scale) with 95% CIs from multivariable logistic regression of a positive response (rating of 4 or 5) for visit preparedness, confidence in knowing what to ask, and confidence in making the most of the visit. Each row is compared with the reference category in italics, or, for visit expectation and agenda topic themes, with conversations that did not cover the theme. Visit type was included in all models but is collapsed into a single row because no category reached significance for these outcomes. Green markers indicate P<0.05 (Wald test, unadjusted for multiple comparisons). N , number of respondents; N4+ , number of positive responses. Full estimates, including satisfaction and likeliness of reuse, are reported in .
Large language model (LLM)-based AI systems have shown promise for patient-facing diagnostic and management conversations in simulated settings. Translating these systems into clinical practice requires assessment in real-world workflows with rigorous safety oversight. We report a prospective, single-arm feasibility study of an LLM-based conversational AI, the Articulate Medical Intelligence Explorer (AMIE), conducting clinical history taking and presentation of potential diagnoses for patients to discuss with their provider at urgent care appointments at a leading academic medical center. 100 adult patients completed an AMIE text-chat interaction up to 5 days before their appointment. We sought to assess the conversational safety and quality, patient and clinician experience, and clinical reasoning capabilities compared to primary care providers (PCPs). Human safety supervisors monitored all patient-AMIE interactions in real time and did not need to intervene to stop any consultations based on pre-defined criteria. Patients reported high satisfaction and their attitudes towards AI improved after interacting with AMIE (p < 0.001). PCPs found AMIE's output useful with a positive impact on preparedness. AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy. Blinded assessment of AMIE and PCP DDx and management (Mx) plans suggested similar overall DDx and Mx plan quality, without significant differences for DDx (p = 0.6) and appropriateness and safety of Mx (p = 0.1 and 1.0, respectively). PCPs outperformed AMIE in the practicality (p = 0.003) and cost effectiveness (p = 0.004) of Mx. While further research is needed, this study demonstrates the initial feasibility, safety, and user acceptance of conversational AI in a real-world setting, representing crucial steps towards clinical translation.
Consumer-facing health chatbots powered by large language models (LLMs) are increasingly used for symptom assessment. However, chatbot development and evaluation often rely on cooperative, articulate, simulated patients. We analysed 2,053 real patient-chatbot conversations and found that communication patterns and expression of emotions vary widely across users. We developed a patient simulator that separately models clinical content, emotional state, conversational strategy, and communication style. In a Turing-inspired evaluation of realism with 15 human graders, simulated conversations were nearly indistinguishable from real ones, with human graders achieving an accuracy of 55%. We used five distinct patient personae, across 1,164 clinician-graded cases, to evaluate the performance of four LLMs in urgency assessment. We found that communication style can significantly alter triage outcomes. Patient-centred conversational artificial intelligence must accommodate communication diversity: systems designed for idealised, rather than realistic, interactions risk underperforming and amplifying health disparities when deployed in the real world.
João Matos, Olivia Buege, Donny Cheung +8
Verily Health, Dallas, TX, USA · Centre for Statistics in Medicine, University of Oxford, Oxford, UK · Department of Applied Health Sciences, School of Health Sciences, University of Birmingham, Birmingham, UK +2
AI applications are increasingly being introduced into digital health. While technical performance has advanced rapidly, successful deployment mainly depends on consumer attitudes, especially to patient-facing applications. However, most existing research examines consumer attitudes towards healthcare AI at an abstract level rather than in response to concrete artefacts. We report a mixed-methods survey study in Australia (N=275) examining consumer readiness, acceptance, trust, and risk perceptions of healthcare AI, combined with a scenario-based evaluation of an AI-generated versus clinician-written consultation summary. Participants expressed moderate optimism and strong perceived usefulness and ease of use, but also substantial concerns about accuracy, safety, and data use. In the scenario task, the AI-generated summary was strongly preferred for quality, empathy, and overall usefulness, yet identification of the AI summary was near chance. Findings show that consumers judge AI through concrete communication quality and visible human governance, underscoring the need for clinically supervised deployment frameworks beyond technical performance alone.
Wei Zhou, Rashina Hoda, Joycelyn Ling
1*Faculty of Information Technology, Monash University, Wellington Rd, Clayton, 3800, VIC, Australia. · 2Digital Health CRC, George Street, Sydney, 2000, NSW, Australia.