A prospective randomized study of patient-centered AI for clinical visit preparation
Authors: Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan, Roma Ruparel, Michael A. Sanchez, Sean Chang, Joseph Xu, Bhavna Daryani, Preeti Singh, Pinal Bavishi, Kejia Chen, Fan Zhang, Priya Gupta, Ernest Lim, Christina Chen, Bob Lou, Dimitrios Antos, Zach Wasson, Ray Luo, Fadi Yousif, Florence Thng, Mark Malhotra, Joseph Breda, Daniel McDuff, Kavi Goel, Katherine Chou, Dale R. Webster, Avinatan Hassidim, Yossi Matias, Po-Hsuan Cameron Chen, Yun Liu, Mike Schaekermann, Quang Duong, Jackie M. Barr
Patients often arrive at brief, time-constrained clinical visits unsure of what to expect or ask, and millions of adults now turn to general-purpose AI chatbots to prepare beforehand. Yet whether conversational AI improves visit readiness remains unclear. We designed a large language model (LLM)-based visit preparation agent to guide patients before an appointment by gathering clinical context, setting visit expectations, providing relevant health information, building a prioritized list of questions for their clinician, and generating a summary. In a prospective consented randomized study in the Google Health app, 802 U.S. adults held 866 health-related conversations with the visit preparation agent (standard or concise variant) or a minimally prompted control chatbot built on the same LLM. The visit preparation agent significantly outperformed the control across blinded clinician evaluations and patient self-reports. In clinician reviews of completed sessions (n = 272), the agent achieved a 16.6% higher rate of comprehensive preparation (67.2% vs. 50.6%; P < 0.05) and an 8.5% higher rate of alignment with patients' health context (98.9% vs. 90.4%; P < 0.01), while maintaining high medical safety and appropriateness across all study arms. Patient post-surveys (n = 460) showed 16.8% higher visit preparedness (73.8% vs. 57.0%; P < 0.01), 14.3% higher confidence in knowing what to ask (81.4% vs. 67.1%; P < 0.01), and 12.0% higher overall satisfaction (74.0% vs. 62.0%; P < 0.05). Clinician and participant metrics were not significantly correlated, indicating that expert review and patient self-assessment capture complementary dimensions of patient-facing AI. These findings suggest that LLM chatbots can safely help patients prepare for clinical visits, with greater benefit when following a structured workflow, and motivate future studies on in-visit communication and clinical outcomes.
Figures & tables
Figure 1: Overview of Contributions. This work provides the first prospective assessment of AI-based visit preparation in a consumer health application, enabled by three key contributions: (A) A structured visit preparation agent powered by Gemini designed to guide patients through a structured pre-visit preparation flow (the user interface shown is trimmed for brevity and clarity; see for more details); (B) A prospective randomized study comparing the visit preparation agent with a control LLM agent; (C) Clinical evaluation suggesting increased clinical utility while maintaining high clinical reliability, with increases in patient-reported visit preparedness and visit self-efficacy.
Figure 2: Study Design. Step 1: Participants were randomized 1:2:2 to a minimally prompted base LLM (Control) or to one of two structured visit preparation agents (standard or concise) for a pre-visit conversation. All three agents used the same LLM and interface, differed only in their system prompts, and each maintained a pre-visit summary (full prompts in ). Step 2: After the session, participants rated their visit preparedness, visit self-efficacy, satisfaction, and likeliness of reuse in a post-survey, and 14 days later were invited to a follow-up survey on whether they had a visit, whether the visit matched the expectations discussed, and whether they used the prep topics. Step 3: In a separate retrospective evaluation, a random sample of completed sessions (conversation and summary), stratified by arm, was assigned at random in a balanced manner to three PCP clinicians blinded to the study design and LLM associated with the conversation, who rated each agent task on four clinical rubrics.
Figure 3: CONSORT Flow Diagram and Participant Characteristics by Arm. Top: flow of participants (n) and conversations (N) from exposure to the study entry point, through 1:2:2 randomization to Control or to the standard or concise visit preparation agent, to exclusions and inclusion in the analysis. Excluded conversations comprised empty or non-health dialogues, conversations entered outside the intended study pathway, and conversations centered on ineligible chief complaints identified by an LLM and verified via clinician review. In the primary analysis, the two visit preparation intervention arms were pooled. Within each study arm, the analytic cohort reflects conversations that completed clinical evaluation, immediate post-survey, and 14-day follow-up reports. Bottom: demographics (age range and sex), visit type, and disease area of the included conversations, shown as N (%) by arm. Visit type and disease area were parsed via LLM from each conversation.
Figure 4: Clinical Evaluation Results. Top: Share of sessions rated favorably (rating of 4 or 5) on each clinical dimension, overall (all applicable steps rated favorably) and for each preparation step. Bars to the left of 0% show ratings of 1–3. The visit preparation arm combines the concise and standard arms. Error bars indicate 95% Wilson confidence intervals. Asterisks mark significant differences between the visit preparation and Control arms ( ∗P<0.05 ; ∗∗P<0.01 ): for the overall rows, in the share of sessions rated favorably (two-sided two-proportion z test); for individual preparation steps, in how ratings were distributed across the 1–5 scale (Mann–Whitney U test). P values are BH-adjusted within each clinical dimension. Bottom: Sessions grouped by clinical reliability (medical safety and appropriateness) and clinical utility (comprehensiveness and patient alignment), where high indicates that all ratings in that category were 4 or higher. P values compare the two arms on the share of sessions that were both highly reliable and useful, and on the share that were highly reliable but not highly useful (two-sided two-proportion z tests, unadjusted).
Figure 5: Patient-reported Outcomes. Distribution of ratings in the immediate post-survey (left and middle) and the exploratory 14-day follow-up survey (right; limited to respondents who had a doctor visit: Control, 13; Visit preparation, 46); the visit preparation arm combines the concise and standard arms. Bars to the right of 0% show positive responses (rating of 4 or 5; for agenda topic use, the share of participants who reported using them), with the total share of positive responses labeled; bars to the left show neutral and negative ratings. Error bars indicate 95% Wilson CI. ∗P<0.05 ; ∗∗P<0.01 (two-proportion z tests with BH correction across the five immediate survey items).
Figure 6: Factors Associated with Positive Visit Readiness and Confidence. Adjusted odds ratios (log scale) with 95% CIs from multivariable logistic regression of a positive response (rating of 4 or 5) for visit preparedness, confidence in knowing what to ask, and confidence in making the most of the visit. Each row is compared with the reference category in italics, or, for visit expectation and agenda topic themes, with conversations that did not cover the theme. Visit type was included in all models but is collapsed into a single row because no category reached significance for these outcomes. Green markers indicate P<0.05 (Wald test, unadjusted for multiple comparisons). N , number of respondents; N4+ , number of positive responses. Full estimates, including satisfaction and likeliness of reuse, are reported in .
Verily Health, Dallas, TX, USA · Centre for Statistics in Medicine, University of Oxford, Oxford, UK · Department of Applied Health Sciences, School of Health Sciences, University of Birmingham, Birmingham, UK +2