Challenges of Auditing: Variability in Outputs of Large Language Models for Health

Yuan Pu; Yewon Chang; Furong Jia; Xunjian Yin; Jessica Ma; Ayman Ali; Monica Agrawal

cs.CLSep 15, 2026

Challenges of Auditing: Variability in Outputs of Large Language Models for Health

Authors: Yuan Pu, Yewon Chang, Furong Jia, Xunjian Yin, Jessica Ma, Ayman Ali, Monica Agrawal

Organizations: Department of Computer Science, Duke University, 308 Research Drive, Durham, 27708, NC. · Department of Medicine, Duke University, 40 Duke Medicine Circle, Durham, 27710, NC. · Geriatrics and Extended Care, Durham VA Health System, 508 Fulton Street, Durham, 27705, NC. · Department of Surgery, Duke University, 2301 Erwin Road, Durham, 27710, NC. · Department of Biostatistics and Bioinformatics, Duke University, 2424 Erwin Road, Durham, 27710, NC.

Abstract

People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying settings. Here, we find systematic differences across access modes. Because evaluations typically rely on APIs while consumers interact through chatbot interfaces, these discrepancies limit evaluation validity. Our findings underscore an urgent need for model providers to enable faithful replication of consumer experiences and settings for rigorous audits.

Explore similar work

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs
Jun 7, 2026Rahul Gorijavolu, Kaushik Madapati, Pritika Vig +7Sycophancy Responses
HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats
Apr 30, 2026Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim +13Benchmarking Large Language Models Physical Activities
Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20Large Language Model Safety Red-Teaming

Jun 7, 2026 · cs.AIJ/K move · Enter open · S save

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

Rahul Gorijavolu, Kaushik Madapati, Pritika Vig, Rawan Abulibdeh+6

Background: Consumer-facing large language models are now a common source of health information, and they interpret and personalize responses rather than retrieve them. Whether their responses vary across users is a clinical, equity, and governance question, sharpened by evidence that sycophantic responses can alter judgment and increase trust. Objective: To evaluate response variation and sycophancy in consumer-facing health LLMs under conditions resembling ordinary patient use. Methods: We constructed simulated user profiles differing in geography, browsing context, expressed beliefs, and social determinants of health, drawing on literature linking social context to health attitudes. We adapted validated instruments, including the Vaccination Attitudes Examination scale and reproductive attitudes scales, into multi-turn prompts designed to elicit clinically meaningful variation across users. Results: The evaluation encountered five linked barriers. Factual prompts produced stable responses that masked sycophancy emerging over multi-turn conversation. Browser-based interfaces did not disclose which signals influence outputs and could not be reset to a clean baseline. Large-scale testing was restricted by terms of service, rate limits, and bot detection. Accuracy-based criteria could not capture tone, framing, or omission, and LLM-as-judge methods risked shared alignment bias. Models changed without traceable version identifiers, preventing reliable replication. Conclusions: No reliable independent evaluation framework yet exists for examining how consumer-facing health LLMs behave in ordinary use. Oversight requires disclosure of personalization signals, stable version identifiers, researcher safe harbor programs, and post-deployment monitoring of health-related outputs.

Sycophancy Responses Vaccination Opinion Dynamics Black Box Personalization Barriers

Open paper Ask about it PDF

Abstract

Explore similar work

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming