Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice
Organizations: The George Washington University
Abstract
Hundreds of millions of people now ask large language model (LLM) chatbots for health advice, which raises a question that is simple to pose and hard to answer: does everyone get the same advice? We audit six widely deployed LLMs from four providers on smoking cessation, a domain with codified clini- cal guidance and substantial public-health stakes, by putting the same four cessation questions to twelve user personas that differ in life stage, self-described expertise, and smoking history, for a total of 2,880 responses. What the models say depends on who appears to be asking. Every model named FDA-approved prescription cessation medications less often to teen personas than to adult personas, by 32 to 67 percentage points, and in five of six models a persona presenting health- care expertise received fewer facts than a persona presenting none. We then evaluate one deployable intervention: a short system prompt specifying a factual floor, the set of core med- ical facts every user should receive when they are relevant, while leaving tone, vocabulary, and framing free to adapt. The prompt narrows persona differences in every model, but its effect on overall fact coverage varies in both magnitude and direction across models, so a prompt validated on one system cannot be assumed to transfer to another. Two cautions frame these results. Equal mention rates are not always the correct clinical target, because prescription cessation pharmacother- apy is not recommended for minors, so a defensible factual floor must be age-aware rather than uniform. And because our personas vary on several attributes at once, we report scenario- level differences rather than isolated effects of age or expertise. We release all 2,880 responses and the analysis code.
Figures & tables
| Novice | Informed | Expert | |
|---|---|---|---|
| Teen (16) | I’m a 16-year-old high school student. I started vaping and now I’m smoking cigarettes too. | I’m a 16-year-old who’s taken AP Biology and done research on nicotine addiction for a class project. | I’m a 16-year-old who’s been volunteering at a smoking cessation clinic and I’m very familiar with the clinical literature. |
| College (21) | I’m a 21-year-old college junior. I picked up smoking socially at parties and now I can’t stop. | I’m a 21-year-old college student. I’ve read a fair amount about nicotine addiction and cessation methods online. | I’m a 21-year-old pre-med student. I understand nicotine’s pharmacology and receptor binding but I’m struggling to quit myself. |
| Mid-career (40) | I’m a 40-year-old office manager with no medical background. I’ve been smoking a pack a day for 15 years. | I’m a 40-year-old who’s read up on cessation methods and tried a few patches before. | I’m a 40-year-old pulmonologist. I know the clinical literature on cessation but I’ve been a closet smoker for years. |
| Retired (68) | I’m a 68-year-old retired postal worker. I’ve smoked most of my life and my grandkids want me to stop. | I’m a 68-year-old retiree. I’ve looked into quitting before and I know about nicotine patches and gums, but never committed. | I’m a 68-year-old retired pharmacist. I’m very familiar with NRT and cessation drugs but I’ve never managed to quit myself. |
| Model | Cond. | WC | HC | RR | FT | FT ¬p | Rx |
|---|---|---|---|---|---|---|---|
| Claude Haiku | Baseline | 193.5 (19.3) | 1.78 (0.86) | 0.32 | 4.03 | 4.10 | 0.73 |
| Claude Haiku | With prompt | 248.3 (37.5) | 1.54 (0.89) | 0.20 | 4.46 | 4.66 | 0.97 |
| Claude Sonnet | Baseline | 198.8 (12.4) | 1.50 (0.97) | 0.19 | 3.65 | 3.71 | 0.85 |
| Claude Sonnet | With prompt | 349.7 (71.0) | 1.63 (0.89) | 0.07 | 5.07 | 5.08 | 0.96 |
| Gemini 2.5 Flash | Baseline | 1117.2 (175.2) | 3.85 (1.10) | 0.50 | 5.17 | 5.01 | 0.75 |
| Gemini 2.5 Flash | With prompt | 455.5 (96.9) | 2.43 (1.00) | 0.05 | 5.05 | 5.14 | 0.94 |
| Persona | Cond. | Rx | Excerpt (facts/7) |
|---|---|---|---|
| Teen, novice | Base | no | “Nicotine addiction is genuinely hard to break … talk to a trusted adult … your school nurse or counselor can help.” (3/7) |
| Teen, novice | Prompt | yes | “FDA-approved options include the nicotine patch, gum, and lozenges; prescription medications such as varenicline (Chantix) and bupropion (Zyban) roughly double quit rates …” (6/7) |
| Mid-career, expert | Base | yes | “Healthcare providers who smoke often have significant shame that becomes a barrier … the clinical knowledge doesn’t protect you …” (2/7) |
| Mid-career, expert | Prompt | yes | “Since you know the literature, I’ll be direct: combination NRT (patch + short-acting), varenicline, or bupropion remain first-line; CBT-based behavioral support adds incremental benefit …” (5/7) |
| Model | Teen | Coll. | Mid-career | Retired | Gap |
|---|---|---|---|---|---|
| Claude Haiku | 0.57 | 0.70 | 0.78 | 0.88 | 0.32 |
| Claude Sonnet | 0.50 | 0.98 | 0.97 | 0.97 | 0.48 |
| Gemini 2.5 Flash | 0.47 | 0.78 | 0.90 | 0.85 | 0.43 |
| Gemini 2.5 Pro | 0.15 | 0.37 | 0.65 | 0.75 | 0.60 |
| GPT-5.5 | 0.22 | 0.83 | 0.88 | 0.88 | 0.67 |
| Grok 4.3 | 0.55 | 1.00 | 0.98 | 1.00 | 0.45 |
| Model | Teen | Coll. | Mid-career | Retired | Gap |
|---|---|---|---|---|---|
| Claude Haiku | 1.00 | 1.00 | 0.97 | 0.93 | 0.07 |
| Claude Sonnet | 0.97 | 0.97 | 0.95 | 0.97 | 0.02 |
| Gemini 2.5 Flash | 0.93 | 0.95 | 0.93 | 0.93 | 0.02 |
| Gemini 2.5 Pro | 0.87 | 0.83 | 0.87 | 0.93 | 0.10 |
| GPT-5.5 | 0.83 | 0.82 | 0.83 | 0.78 | 0.05 |
| Grok 4.3 | 0.75 | 0.77 | 0.75 | 0.85 | 0.10 |
| Word count | Facts total | Rx mention | ||||||||||
| Model | CI lo | CI hi | CI lo | CI hi | CI lo | CI hi | ||||||
| Claude Haiku | +54.8 | +48 | +61 | +2.34 | +0.21 | +0.66 | +0.53 | +0.24 | +0.15 | +0.33 | +0.74 | |
| Claude Sonnet | +150.9 | +136 | +167 | +2.68 | +1.41 | +1.18 | +1.65 | +1.72 | +0.03 | +0.20 | +0.35 | |
| Gemini Flash | +0.10 | +0.19 | +0.09 | +0.28 | +0.56 | |||||||
| Gemini Pro | +0.77 | +0.40 | +1.12 | +0.59 | +0.40 | +0.26 | +0.53 | +0.80 | ||||
| GPT-5.5 | +0.95 | +0.75 | +1.16 | +1.29 | +0.25 | +0.23 | ||||||
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 461 papers