A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic
Authors: Peter Brodeur, Jacob M. Koshy, Anil Palepu, Khaled Saab, Ava Homiar, Roma Ruparel, Charles Wu, Ryutaro Tanno, Joseph Xu, Amy Wang, David Stutz, Wei-Hung Weng, Hannah M. Ferrera, David Barrett, Lindsey Crowley, Jihyeon Lee, Spencer E. Rittner, Ellery Wulczyn, Selena K. Zhang, Elahe Vedadi, Christine G. Kohn, Kavita Kulkarni, Vinay Kadiyala, Sara Mahdavi, Wendy Du, Jessica M. Williams, David Feinbloom, Renee Wong, Tao Tu, Petar Sirkovic, Alessio Orlandi, Christopher Semturs, Yun Liu, Juraj Gottweis, Dale R. Webster, Joëlle Barral, Katherine Chou, Pushmeet Kohli, Avinatan Hassidim, Yossi Matias, James Manyika, Rob Fields, Jonathan X. Li, Marc L. Cohen, Vivek Natarajan, Mike Schaekermann, Alan Karthikesalingam, Adam Rodman
Large language model (LLM)-based AI systems have shown promise for patient-facing diagnostic and management conversations in simulated settings. Translating these systems into clinical practice requires assessment in real-world workflows with rigorous safety oversight. We report a prospective, single-arm feasibility study of an LLM-based conversational AI, the Articulate Medical Intelligence Explorer (AMIE), conducting clinical history taking and presentation of potential diagnoses for patients to discuss with their provider at urgent care appointments at a leading academic medical center. 100 adult patients completed an AMIE text-chat interaction up to 5 days before their appointment. We sought to assess the conversational safety and quality, patient and clinician experience, and clinical reasoning capabilities compared to primary care providers (PCPs). Human safety supervisors monitored all patient-AMIE interactions in real time and did not need to intervene to stop any consultations based on pre-defined criteria. Patients reported high satisfaction and their attitudes towards AI improved after interacting with AMIE (p < 0.001). PCPs found AMIE's output useful with a positive impact on preparedness. AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy. Blinded assessment of AMIE and PCP DDx and management (Mx) plans suggested similar overall DDx and Mx plan quality, without significant differences for DDx (p = 0.6) and appropriateness and safety of Mx (p = 0.1 and 1.0, respectively). PCPs outperformed AMIE in the practicality (p = 0.003) and cost effectiveness (p = 0.004) of Mx. While further research is needed, this study demonstrates the initial feasibility, safety, and user acceptance of conversational AI in a real-world setting, representing crucial steps towards clinical translation.
Figures & tables
Figure 1: Study design. For each patient, the study flow consisted of three distinct steps: (1) The patient conversed with AMIE through a synchronous text chat interface, while an AI supervisor observed the patient-AMIE chat on a video-call with screen-sharing to ensure patient safety; (2) the patient saw a primary care physician (PCP) in person or via telehealth up to five days after the AMIE chat, with the PCP having access to the AMIE chat transcript and summary; (3) eight weeks after the provider visit, the final diagnosis was extracted from the patient’s chart and three independent clinical evaluators rated the chat conversation quality as well as the differential diagnosis and management plan from both AMIE and PCP in a blinded and randomized manner.
Table 1: Patient Participation Statistics. The CONSORT diagram provides statistics regarding the flow of patient participants through our study (A) . For the 98 patients who completed both the AMIE encounter and the PCP encounter, we provide patient characteristics based on survey responses collected from patients before interacting with AMIE. (B) . Counts for ‘Prefer not to say’ and ‘N/A’ include the eight patients who did not complete the pre-AMIE survey.
Figure 2: Clinical Reasoning Performance. Clinical evaluators rated the quality of management plans and differential diagnoses of both AMIE and PCPs in a blinded randomized manner. For each patient case, ratings were aggregated as the median rating across a panel of three independent clinical evaluators. (A) Comparative ratings from clinical evaluators assessed the quality of two candidate management plans and differential diagnoses in a side-by-side manner relative to each other; error bars for comparative ratings represent 95% confidence intervals for binomial proportions (N=98) for expressing a preference (‘slightly better’ or ‘much better’) for either of the two candidates. (B) Pointwise ratings from clinical evaluators assessed the quality of each management plan and differential diagnosis individually on a 5-point Likert scale. For pointwise ratings, asterisks represent statistical significance per two-sided Wilcoxon signed-rank tests with Bonferroni correction ( ∗∗:p<0.01 , n.s.: not significant). In addition to ratings from clinical evaluators, we measured (C) AMIE’s Top-k diagnostic accuracy as compared to the final diagnosis extracted for each patient via chart review eight weeks after their PCP visit. In addition to overall accuracy across all patients (N=98), we provide accuracy for the subset of patients where the final diagnosis was confirmed by a diagnostic test such as imaging, microbiology, laboratory, pathology, EKG (N=46), and the subset where this was not the case, i.e., where the diagnosis was presumptive without a diagnostic test, irrespective of whether the diagnosis was made by a PCP or specialist (N=52). Error bars for diagnostic accuracy correspond to 95% confidence intervals for binomial proportions.
Figure 3: Effect on patient attitudes towards AI. Patients completed the General Attitudes towards AI Scale (GAAIS) prior to interacting with AMIE (Pre-AI), after interacting with AMIE (Post-AI), and after the urgent care consultation with the provider (Post-Provider). The GAAIS scale includes two sub-scales corresponding to (1) perceived utility and (2) concerns around AI. Attitudes shifted more positive after interacting with AMIE and remained at an elevated level after seeing the PCP. This change was statistically significant for both sub-scales and the overall scale as confirmed by a Friedman omnibus test followed by pairwise Wilcoxon post-hoc tests (p < 0.001 for omnibus and pairwise tests).
Figure 4: AMIE Conversation Quality. The quality of AMIE conversations was rated from patient and clinician perspectives. Patient perspectives were collected through surveys immediately after patients completed their interaction with AMIE and the safety debrief with the AI supervisor. Clinician perspectives were rated post-hoc by a panel of three independent clinical evaluators per patient case, and aggregated using the median across the three ratings per case.
Language models excel at diagnostic assessments on curated medical case-studies and vignettes, performing on par with, or better than, clinical professionals. However, existing studies focus on complex scenarios with rich context making it difficult to draw conclusions about how these systems perform for patients reporting symptoms in everyday life. We deployed SymptomAI, a set of conversational AI agents for end-to-end patient interviewing and differential diagnosis (DDx), via the Fitbit app in a study that randomized participants (N=13,917) to interact with five AI agents. This corpus captures diverse communication and a realistic distribution of illnesses from a real world population. A subset of 1,228 participants reported a clinician-provided diagnosis, and 517 of these were further evaluated by a panel of clinicians during over 250 hours of annotation. SymptomAI DDx were significantly more accurate (OR = 2.56, p < 0.001) than those from independent clinicians given the same dialogue in a blinded randomized comparison. Moreover, agentic strategies which conduct a dedicated symptom interview that elicit additional symptom information before providing a diagnosis, perform substantially better than baseline, user-guided conversations (p < 0.001). An auxiliary analysis on 1,509 conversations from a general US population panel validated that these results generalize beyond wearable device users. We used SymptomAI diagnoses as labels for all 13,917 participants to analyze over 500,000 days of wearable metrics across nearly 400 unique conditions. We identified strong associations between acute infections and physiological shifts (e.g., OR > 7 for influenza). While limited by self-reported ground truth, these results demonstrate the benefits of a dedicated and complete symptom interview compared to a user-guided symptom discussion, which is the default of most consumer LLMs.
Joseph Breda, Fadi Yousif, Beszel Hawkins +30
Google Research · Work done while at Google Research · Google DeepMind
Clinical AI evaluation should encompass diagnosis and management after adaptive information gathering. We compared Doctorina, eight physicians and four standalone frontier language models in 150 synthetic Polish-language primary-care consultations. Doctorina achieved 82.0% Top-1 concordance versus 57.0% for physicians (difference, 25.0 percentage points; 95% confidence interval, 17.7-32.7) and 97.3% versus 85.0% primary-or-reference-differential concordance. Across 149 case pairs, normalized workup and treatment scores were 89.4 versus 66.9 and 83.7 versus 61.2. Doctorina had the highest diagnostic point estimates among all six groups; Kimi K3 ranked next, while Claude Opus 5 led the closely spaced management estimates of Opus, Doctorina and Kimi. A second Doctorina execution reproduced the advantages over physicians across all outcomes. Doctorina's advantage over physicians therefore extended from primary-diagnosis selection to higher-rated diagnostic workup and initial treatment after adaptive consultation.
Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei +6
1A.I. Doctor Medical Assist LTD, Limassol, Cyprus. · 2Family Medicine Specialist, Kraków, Poland.
Clinical decision-making in emergency medicine demands rapid, accurate diagnoses under uncertainty. Despite benchmark progress, evidence for LLMs as interactive aids in live physician workflows remains sparse. MedSyn lets physicians iteratively query an LLM provided with the full clinical record while initially viewing only the chief complaint. Seven physicians (three seniors, four residents) completed baseline and AI-assisted sessions across 52 MIMIC-IV cases stratified by difficulty. Blinded evaluation showed residents' Hard-case correctness rose from 0.589 to 0.734; difficulty-standardised completely-correct rates confirmed a medium effect (Δ = 0.092; p = 0.071; d = 0.47). Automated metrics corroborated these gains: standardised any-match accuracy improved by 0.156 (p < 0.0001), and residents showed the largest F1 gain (Δ = 0.138; p < 0.0001). Dialogue analysis revealed expertise-dependent strategies (seniors asked targeted, hypothesis-driven questions; residents relied on broader queries) and cross-expertise concordance increased (Δ = 0.145; p < 0.0001). Interactive LLM support meaningfully enhances diagnostic reasoning.
Burcu Sayin, Ngoc Vo Hong, Ipek Baris Schlicht +8
Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Medicine, Azienda Sanitaria Universitaria Integrata del Trentino (ASUIT), Trento, Italy · Universitat Politècnica de València, Spain +2