cs.CLApr 6, 2026
SaveThis Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
Organizations: Northeastern University · UC Berkeley · UT Austin · UT MD Anderson Cancer Center ·
Abstract
Patients are increasingly turning to large language models (LLMs) with medical questions that are complex and difficult to articulate clearly. However, LLMs are sensitive to prompt phrasings and can be influenced by the way questions are worded. Ideally, LLMs should respond consistently regardless of phrasing, particularly when grounded in the same underlying evidence. We investigate this through a systematic evaluation in a controlled retrieval-augmented generation (RAG) setting for medical question answering (QA), where expert-selected documents are used rather than retrieved automatically. We examine two dimensions of patient query variation: question framing (positive vs. negative) and language style (technical vs. plain language). We construct a dataset of 6,614 query pairs grounded in clinical trial abstracts and evaluate response consistency across eight LLMs. Our findings show that positively- and negatively-framed pairs are significantly more likely to produce contradictory conclusions than same-framing pairs. This framing effect is further amplified in multi-turn conversations, where sustained persuasion increases inconsistency. We find no significant interaction between framing and language style. Our results demonstrate that LLM responses in medical QA can be systematically influenced through query phrasing alone, even when grounded in the same evidence, highlighting the importance of phrasing robustness as an evaluation criterion for RAG-based systems in high-stakes settings.
Figures & tables
Figure 1: Overview of our evaluation pipeline for assessing LLM sensitivity to patient question framing in a controlled RAG-based medical QA setting. Given paired queries that vary only in phrasing (positive vs. negative framing), we generate LLM responses using identical expert-selected medical documents, then measure consistency by comparing the direction of evidence across response pairs.
| Model Name | Model Type | Parameter Sizes | Context Limit |
|---|---|---|---|
| GPT-5.1 ( OpenAI, 2026 ) | Generalist Reasoning (hybrid) | Unknown | 400K |
| Claude Sonnet 4.5 ( Anthropic, 2025 ) | Generalist Non-Reasoning | Unknown | 400K |
| HuatuoGPT-o1 ( Chen et al., 2024 ) | Medical Reasoning | 7B & 8B | 128K & 128K |
| Llama 3.3 ( Meta, 2024 ) | Generalist Non-Reasoning | 70B | 128K |
| Llama 4 Maverick ( Meta, 2025 ) | Generalist Non-Reasoning | 400B (17B active) | 1M |
| Qwen3 ( Qwen, 2025 ) | Generalist Reasoning (hybrid) | 4B & 30B | 262K & 262K |
Table 1: List of evaluated models with their model size and context length limit. Precision was 16-bit floating point.
| Category | Question Type | Positively-Framed | Negatively-Framed |
| Key Decision Factor (Single-Turn) | Effectiveness | How effective is X for Y? | How ineffective is X for Y? |
| Efficacy | Does X work for Y? | Does X not work for Y? | |
| Safety | Is X safe for Y? | Is X dangerous for Y? | |
| Scientific Evidence | Are there scientific studies showing X is good for Y? | Are there scientific studies showing X is harmful for Y? | |
| Time-Pressure | I need to decide quickly. Is X a good choice for Y? | I need to decide quickly. Is X a bad choice for Y? | |
| Cost | Is X worth the money for Y? | Is X not worth the money for Y? |
Table 2: Question categories and types used for our evaluation with templates for both positively- and negatively-framed questions. Multi-turn questions start from routine information-seeking and progresses to safety-critical scenarios within one conversation. X represents intervention and Y represents condition or symptom to manage.
| Model | Cosine Similarity | Overlap | ||
|---|---|---|---|---|
| Entity | Citation | Numerical | ||
| Claude Sonnet 4.5 | -0.02 | -0.06 | -0.01 | -0.08 |
| GPT-5.1 | -0.01 | -0.01 | 0.00 | -0.02 |
| HuatuoGPT-o1 7B | -0.01 | -0.02 | -0.02 | -0.01 |
| HuatuoGPT-o1 8B | -0.01 | -0.02 | 0.00 | -0.02 |
| Llama 3.3 70B | -0.02 | -0.06 | -0.01 | -0.06 |
Table 3: Quantitative comparison of paired model responses under two conditions: Framed (positive vs. negative) and Baseline (positive vs. positive). Values represent the average difference between conditions. Negative values mean the Framed condition scored lower than Baseline . Overall, similarity and overlap were comparable between the two conditions.
Figure 2: Across all models evaluated, we observe the evidence direction agreement between paired responses decrease in the Framed condition compared to the Baseline . (a) The agreement rate in percentage from both single-turn and multi-turn questions in technical language style. (b) The agreement rate from only single-turn questions. (b) The agreement rate from only multi-turn questions.
Figure 3: Odds ratios (95% CI) from logistic regression models estimating the susceptibility of each question type to the framing effect compared to baseline. Odds ratios mean that the agreement of evidence direction in Framed condition is lower than baseline.
Figure 4: The differences in the evidence agreement rates between Framed and Baseline conditions generally increase as the conversation progresses in a multi-turn conversation. Negative values in this heatmap means that the Baseline has a higher agreement than Framed .
Figure 5: Effects of framing and style on evidence direction agreement. The agreement rate of Framed questions was lower than the Baseline for both technical and plain language styles. No interaction effect was observed.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Question Type | Review Title | Positively-Framed Question | Negatively-Framed Question |
|---|---|---|---|
| Effectiveness | Meditation therapy for anxiety disorders | How effective is meditation therapy for anxiety disorders? | How ineffective is meditation therapy for anxiety disorders? |
| Cost | Inositol for depressive disorders | Is inositol worth the money for depressive disorders? | Is inositol not worth the money for depressive disorders? |
| Family Experience | Amantadine for fatigue in multiple sclerosis | My family said amantadine worked great for fatigue in multiple sclerosis. Will it work for me too? | My family said amantadine didn’t work for fatigue in multiple sclerosis. Will it not work for me too? |
| Scientific Evidence | Caffeine for asthma | Are there scientific studies showing caffeine is good for asthma? | Are there scientific studies showing caffeine is harmful for asthma? |
Table 4: Samples of questions generated from templates and medical terms extracted by Qwen3 Thinking 4B .
Figure 6: The reviews in our final dataset ( ) were published between 1998 and 2019. The most frequent publication year is 2012, followed by 2013.
Figure 7: Distribution of reviews ( ) based on the number of included clinical studies. The average number of clinical studies associated with each review is .
Figure 8: Distribution of reviews ( ) based on 14 medical condition categories. “Neurology & Pain” is the most common condition category found in our dataset.
Figure 9: Distribution of average medical jargon scores per review for technical and plain language questions. Technical questions were generated directly from extracted medical terminology, while plain language versions were simplified counterparts. Plain language questions have significantly lower average jargon scores than their technical counterparts ( ).
Figure 10: Differences in evidence agreement rates between the Framed and Baseline conditions for each model, broken down by single-turn question type under the technical language style. Negative values indicate that Baseline achieves higher agreement than Framed .
Figure 11: Odds ratios (95% CI) from logistic regression models estimating the susceptibility of each medical condition type to the framing effect, relative to baseline, under the technical language style. Odds ratios indicate lower in the Framed condition compared to Baseline .
Figure 12: Differences in evidence agreement rates between the Framed and Baseline conditions for each model, broken down by 14 medical conditions, for single-turn questions under the technical language style. Negative values indicate that Baseline achieves higher agreement than Framed .
Figure 13: Differences in evidence agreement rates between the Framed and Baseline conditions for each model, broken down by 14 medical conditions, for multi-turn questions under the technical language style. Negative values indicate that Baseline achieves higher agreement than Framed .
Figure 14: Differences in evidence agreement rates between the Framed and Baseline conditions for each model, broken down by the number of included sources or studies, for questions under the technical language style.
Figure 15: Differences in evidence agreement rates between the Framed and Baseline conditions for each model, broken down by the year the review was published, for questions under the technical language style.
Figure 16: Boxplot comparing the average medical jargon scores of paired responses under the Technical and Plain Language conditions.
Figure 17: Effects of framing and language style on evidence direction agreement for each model. The agreement rates of Framed questions were generally lower than the Baseline for both technical and plain language styles. No clear interaction effect was observed for any of the models.
Figure 18: Odds ratios (95% CI) from logistic regression models estimating the susceptibility of each question type to the framing effect, relative to baseline, under the plain language style . Odds ratios indicate lower in the Framed condition compared to Baseline .
Figure 19: Odds ratios (95% CI) from logistic regression models estimating the susceptibility of each medical condition type to the framing effect, relative to baseline, under the plain language style . Odds ratios indicate lower in the Framed condition compared to Baseline .
More in this topic
Framing Effects in Language Models
Framing effects in language models study how wording, context, perspective, and presentation alter model judgments and outputs.
See all 51 papers