Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare
Organizations: Johns Hopkins University, USA · National Institute of Standards and Technology, USA
Abstract
Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or standardized metrics for assessing their performance, making it difficult to compare findings or cumulatively synthesize evidence across studies. Across 54 recent evaluation studies, we identify ten evaluation dimensions and 40 distinct metrics, operationalized in 96 different ways-35 of them (36%) used by only a single study. Assessment is dominated by documentation efficiency and time, measured in 44 of 54 studies. The other dimensions appear at broadly similar rates (16-24 studies each), and safety and clinical outcome least consistently of all. Ripple effects on the broader care system, such as impact on patients' care outcomes, redistribution of work, changes in billing and coding intensity, and downstream use of notes by other clinicians are rarely assessed. Moreover, the metrics used rely heavily on self-report and are fragmented and inconsistently operationalized, which limits comparability and hinders synthesis across studies. As an effort to support more rigorous and cumulative evaluation, in this paper we review the study designs, data sources, and metrics commonly used to evaluate ambient AI scribes in healthcare settings, discuss the limitations of current evaluation practice, and argue for a more holistic approach grounded in a standardized, multi-dimensional set of metrics.
Figures & tables
| PICO(S) Elements | Elements in this Review |
|---|---|
| Participants | All healthcare providers engaged in clinical documentation |
| Intervention | AI tools designed to streamline clinical documentation, including: • Real-time transcription and notetaking • Automated data entry into EHRs • Natural Language Processing (NLP) for clinical summarization, contextual understanding, and structured documentation • Tools transforming spoken interactions into organized clinical notes |
| Comparison | Comparison not required. |
| Outcomes | All outcomes |
| Setting | All clinical settings |
| Specifications for the studies | • Eligible studies: Included evaluation of a deployed ambient AI scribe in a clinical setting. • Studies, pre-post studies, observational, qualitative-only studies, and mixed-method studies. • Excluded studies: Not AI scribe, lack of real-world deployment, simulated patients, simulated setting, veterinary setting. |
| Category | Studies | |
|---|---|---|
| Department / setting | ||
| Primary care / general practice /family medicine /internal medicine | 13 | ( Toussi et al., 2026 ; Ballantyne et al., 2025 ; Shah et al., 2025 ; Moura et al., 2026 ; North et al., 2025 ; Guo et al., 2025 ; Castro et al., 2026 ; Alpert et al., 2025 ; van Linschoten et al., 2026 ; Waken et al., 2026 ; Memon et al., 2026 ; Duggan et al., 2025 ; Husa et al., 2026 ) |
| Ambulatory & multiple specialties (can include nurses, dentists etc.) | 25 | ( Shah et al., 2025 ; Schneider et al., 2026 ; Rotenstein et al., 2026 ; Adler-Milstein et al., 2026 ; Holmgren et al., 2026 ; Patel et al., 2026 ; Wright et al., 2025 ; Harvey et al., 2025 ; Wright et al., 2026 ; Pearlman et al., 2025 ; Lukac et al., 2025 ; Olson et al., 2025 ; Chowdhury et al., 2026 ; Haberle et al., 2024 ; Rui et al., 2025 ; Guo et al., 2025 ; Evans et al., 2025 ; Tierney et al., 2025 ; Tan et al., 2026 ; Harvey et al., 2026 ; Wojda et al., 2026 ; Sanmark et al., 2026 ; Alcázar-Peral et al., 2026a ; Husa et al., 2026 ; Lawrence et al., 2026 ) |
| Pediatrics | 6 | ( Shin et al., 2025 ; Pelletier et al., 2025 ; Rabbani et al., 2025 ; North et al., 2025 ; Talman et al., 2026 ; Kapoor et al., 2026 ) |
| Emergency medicine | 7 | ( Wright et al., 2026 ; Preiksaitis et al., 2026b ; Marquis et al., 2026 ; Akhlaghi et al., 2026 ; Dutta et al., 2026 ; Alcázar-Peral et al., 2026b ; Preiksaitis et al., 2026a ) |
| Oncology / cancer care | 5 | ( Toussi et al., 2026 ; Nguyen et al., 2023 ; Baldwin-Medsker et al., 2025 ; Baldwin et al., 2025 ; Talman et al., 2026 ) |
| Metric | Measurement / instrument | Studies | |
|---|---|---|---|
| A. Efficiency & time ( 44 studies) | |||
| Time in notes / documentation time (n= 30 ) | EHR / Epic Signal “time in notes” (per note, appointment, day, weekly, or monthly average) | 20 | ( Shin et al., 2025 ; North et al., 2025 ; Guo et al., 2025 ; Cao et al., 2024 ; Wright et al., 2025 ; Harvey et al., 2025 ; Rabbani et al., 2025 ; Preiksaitis et al., 2026b ; Tierney et al., 2025 ; Pearlman et al., 2025 ; Lukac et al., 2025 ; Waken et al., 2026 ; Chowdhury et al., 2026 ; Duggan et al., 2025 ; Talman et al., 2026 ; Harvey et al., 2026 ; Husa et al., 2026 ; Lawrence et al., 2026 ; Dutta et al., 2026 ; Preiksaitis et al., 2026a ) |
| DocTime (EHR minutes per scheduled hour) | 2 | ( Rotenstein et al., 2026 ; Adler-Milstein et al., 2026 ) | |
| Average documentation time per patient / consultation (bespoke timing or self-report) | 8 | ( Haberle et al., 2024 ; van Linschoten et al., 2026 ; Evans et al., 2025 ; Pelletier et al., 2025 ; Akhlaghi et al., 2026 ; Carmichael et al., 2026 ; Sanmark et al., 2026 ; Tan et al., 2026 ) | |
| Total EHR / EMR time (n= 9 ) | Aggregate EHR minutes per day or appointment (Epic Signal; EHR-8 in Rotenstein) | 8 | ( Rotenstein et al., 2026 ; Cao et al., 2024 ; Rabbani et al., 2025 ; Preiksaitis et al., 2026b ; Pearlman et al., 2025 ; Tierney et al., 2025 ; Vallabhaneni et al., 2026 ; Preiksaitis et al., 2026a ) |
| Time in specific EHR task categories (in-basket, orders, clinical review, per appt) | 2 | ( Tierney et al., 2025 ; Lawrence et al., 2026 ) | |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Database | Search strategy |
|---|---|
| Search period: November 2024–July 2026 | |
| Google Scholar | “Ambient scribe” OR “Clinical scribe” OR “Medical scribe” OR “automated scribe” OR “virtual scribe” OR “Digital scribe” OR “AI scribe” OR “E scribe” |
| arXiv | “Ambient scribe” OR “Clinical scribe” OR “Medical scribe” OR “automated scribe” OR “virtual scribe” OR “Digital scribe” OR “AI scribe” OR “E scribe” |
| Web of Science | “Ambient scribe” OR “Clinical scribe” OR “Medical scribe” OR “automated scribe” OR “virtual scribe” OR “Digital scribe” OR “AI scribe” OR “E scribe” |
| MEDLINE (Ovid) | ((Ambient OR Clinical OR Medical OR automated OR virtual OR Digital OR AI OR “Artificial Intelligence” OR E) adj1 scrib*).ti,ab,kf |
| Embase (Embase.com) | ((Ambient OR Clinical OR Medical OR automated OR virtual OR Digital OR AI OR “Artificial Intelligence” OR E) NEAR/1 scrib*):ti,ab,kw |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 422 papers