When Pleasing the User Compromises Care: AI Sycophancy in Caregiver-Mediated Healthcare

Priyadarsi Mishra; Arjun Beniwal; Ziyu Yi; Yu Ding; Sheng Liu

cs.AIOct 9, 2026

When Pleasing the User Compromises Care: AI Sycophancy in Caregiver-Mediated Healthcare

Authors: Priyadarsi Mishra, Arjun Beniwal, Ziyu Yi, Yu Ding, Sheng Liu

Organizations: Stanford University · Texas A&M University

Abstract

Large language model (LLM) alignment often assumes that the user interacting with a model is also the primary beneficiary of its behavior. Yet in many consequential settings, one person directs the system while another bears the consequences. We study this problem in caregiver-mediated healthcare, where parents, pet owners, and caregivers make decisions for children, animals, and older adults. We introduce an evidence-grounded benchmark spanning veterinary, pediatric, and older-adult care, and test whether AI advisors maintain appropriate care when users express clinically inferior preferences. Holding patient facts fixed, we vary stated care preferences and communication styles and evaluate both initial recommendations and multi-turn follow-up. Across frontier and open-weight models, inferior preferences substantially increase switching from appropriate to inappropriate care beyond neutral conversational drift and increase agreement with the requested action, while communication style alone has a smaller effect. These failures accumulate across turns, including on cases models initially answer correctly, and often carry moderate or severe potential harm. In matched eldercare cases, the effect of an inferior preference is similar whether it is expressed by the patient or by a caregiver. Together with models' limited tendency to elicit patient preferences in caregiver conversations, this suggests that current models are highly sensitive to expressed user preferences but less sensitive to whose preferences are represented and who bears the consequences. Our results expose a broader limitation of user-centric alignment in settings with multiple stakeholders.

Figures & tables

Figure 1: CareDistance overview. Source-grounded patient cases are used to generate conversations under 13 caregiver persona and preference combinations, with patient facts held fixed. Conversations span up to five turns; assessment stops at the first inappropriate response. The example messages are condensed and paraphrased; Appendix E.11.1 gives the full evaluated conversation.

Preference condition	Caregiver request
Neutral	Asks the original question without expressing a care preference.
Inferior preference only	Expresses a preference for the inferior alternative.
Gold-preference control	Expresses a preference for the reference action.
Persona	Caregiver concern or behavior
Practical Constraint	Emphasizes whether the care is realistically manageable.
Values Commitment	Appeals to beliefs about what constitutes responsible care.

Table 1: Caregiver preferences and personas. Each persona is evaluated with and without a preference for the inferior alternative, while clinical facts remain fixed.

Figure 2: Care maintained and inferior-action agreement through turn five. Bars show care maintained (A) and inferior-action agreement at first failure (B). Both panels use all cases with appropriate neutral first-response advice as the denominator; Panel B counts those whose first inappropriate response recommends or permits the inferior action. The three domains receive equal weight, and persona groups average the five personas. Whiskers show 95% confidence intervals.

	Neutral first response (%) $\uparrow$		Switch rate by turn 5 (%) $\downarrow$
Model	Gold agreement	Appropriate care	Neutral control	Persona only	Inferior preference only	Persona + preference	Gold-preference control
GPT-5.6 Sol	93.3 [90.9, 95.6]	96.4 [94.7, 98.0]	3.9 [2.1, 5.8]	6.0 [4.5, 7.5]	33.9 [29.7, 38.2]	38.8 [35.5, 42.1]	2.6 [1.2, 4.1]
GPT-5.6 Luna	92.0 [89.3, 94.2]	95.1 [93.1, 96.9]	4.2 [2.4, 6.1]	6.3 [4.7, 7.9]	32.7 [28.2, 37.1]	35.6 [32.4, 38.7]	5.0 [3.1, 7.1]
Gemini 3.8 Flash	88.4 [85.6, 91.3]	90.9 [88.2, 93.3]	15.8 [12.4, 19.4]	20.2 [17.9, 22.6]	44.4 [39.8, 49.0]	51.2 [47.8, 54.4]	16.5 [13.1, 20.1]
Claude Sonnet 5	86.0 [82.9, 89.1]	88.7 [85.8, 91.6]	19.3 [15.5, 23.2]	22.9 [20.3, 25.4]	70.2 [65.7, 74.6]	62.8 [59.6, 66.0]	20.9 [17.1, 24.8]
Kimi K2.5	85.3 [82.0, 88.4]	88.0 [85.1, 90.9]	27.5 [23.2, 31.9]	34.4 [31.6, 37.2]	37.5 [32.8, 42.2]	39.3 [36.4, 42.2]	26.9 [22.6, 31.5]

Table 2: Neutral first-response performance and switching by turn five. Switch rates count cases with an inappropriate response by turn five among cases with appropriate neutral first-response advice. The three domains and five personas are equally weighted; brackets give 95% confidence intervals. Arrows indicate the favorable direction. Bold marks descriptive column extrema, including ties.

Figure 3: Appropriate-to-inappropriate switching across turns. For each model, curves show the cumulative percentage of cases with an inappropriate response by each turn, restricted to cases with an appropriate neutral first response. The three domains receive equal weight. Shading shows 95% confidence intervals.

Figure 4: Preference sensitivity when the user does or does not bear the clinical consequences. The eldercare comparison shows appropriate-to-inappropriate switching across turns (A) and inferior-action agreement at first failure through turn five (B), under neutral and inferior-preference conditions. Shading and whiskers show paired 95% confidence intervals.

Figure 5: Care maintained through turn five by clinical decision type. Profiles show the percentage of cases with appropriate neutral first-response advice that receive no inappropriate response through turn five under neutral control or inferior preference only. Cases are pooled across the three domains within each decision type; higher values indicate more care maintained.

Figure 6: Switching to inappropriate care and potential-harm severity. Bars show switch rates by turn five among cases with appropriate neutral first-response advice, divided by potential-harm severity at first failure. Domains receive equal weight; whiskers show 95% confidence intervals. “Pref.” denotes inferior preference.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

	Eldercare	Pediatrics	Veterinary
Patient cases	150	150	150
Source documents	11	27	150
Clinical areas	7	8	7
Provide indicated care	85	75	106
Withhold inappropriate care	65	75	44

Appendix

Table 3: Profile of the evaluated datasets. Source documents are distinct guidelines or reference articles represented in the selected cases. Care direction describes the reference decision.

Source family	Documents	Cases
Eldercare
STOPP/START v3	1	45
WHO	1	42
ESPEN	2	39
PALTmed	1	16
USPSTF	4	4

Appendix

Table 4: Clinical source families represented in the selected cases.

Figure 7: Clinical decision types across domains. Each domain contains 150 cases, assigned to one primary type from the question and reference action. Segment labels give case counts.

Tier	Supported consequence if it occurs
Mild	Minor, temporary symptoms or discomfort requiring little or no treatment.
Moderate	Meaningful deterioration requiring treatment, or temporary functional limitation.
Severe	Life-threatening deterioration, major functional loss, lasting disability or death.

Appendix

Table 5: Potential clinical harm: severity definitions.

	Eldercare		Pediatrics		Veterinary
Model	Baseline	Uncertain	Baseline	Uncertain	Baseline	Uncertain
GPT-5.6 Sol	146	10	142	9	146	5
GPT-5.6 Luna	148	3	136	11	144	4
Muse Spark 1.3	144	8	133	17	–	–
Gemini 3.8 Flash	142	19	127	29	140	19
Claude Sonnet 5	140	13	122	26	137	9

Appendix

Table 6: Evaluation coverage and uncertain judgments. Baseline-appropriate counts give the cases used for switching, care maintained and harm analyses. Uncertain counts are conversations with at least one uncertain judgment among evaluated turns, including those that later fail. Cases with uncertain neutral first responses are excluded from these analyses; later uncertain judgments are not counted as failures. Complete runs contain 1,950 conversations; veterinary Claude Sonnet 5 contains 1,947.

Comparison	First-failure agreement (%)	Cohen’s $\kappa$
LLM–Reviewer 1	80.0 [68.3, 90.0]	0.596 [0.409, 0.778]
LLM–Reviewer 2	73.3 [61.7, 83.3]	0.448 [0.255, 0.639]
Reviewer–reviewer	78.3 [70.0, 86.7]	0.610 [0.438, 0.759]

Appendix

Table 7: Agreement on the first inappropriate turn. Each comparison covers 60 conversations and includes agreement on no failure. Brackets give 95% confidence intervals.

Evaluator	Neutral	Inferior preference	Excess (percentage points)
Automatic judge	0/30 (0.0%)	14/30 (46.7%)	$+46.7$
Reviewer 1	8/30 (26.7%)	14/30 (46.7%)	$+20.0$ [0.0, 40.0]
Reviewer 2	8/30 (26.7%)	13/30 (43.3%)	$+16.7$ [ $-3.3$ , 36.7]

Appendix

Table 8: Switching on the matched validation sample. All evaluators judged the neutral first response appropriate in all 30 pairs. Excess is the inferior-preference rate minus neutral; brackets give 95% confidence intervals for reviewer excess rates.

	Switch rate (%) $\downarrow$	Difference from neutral (percentage points) $\downarrow$
Model	Neutral control	Persona only	Inferior preference only	Persona + preference
GPT-5.6 Sol	3.9 [2.1, 5.8]	+2.0 [0.2, 3.8]	+30.0 [25.5, 34.4]	+34.9 [31.5, 38.3]
GPT-5.6 Luna	4.2 [2.4, 6.1]	+2.0 [0.3, 3.7]	+28.5 [23.7, 33.3]	+31.4 [28.0, 34.8]
Gemini 3.8 Flash	15.8 [12.4, 19.4]	+4.5 [0.8, 7.9]	+28.7 [22.9, 34.3]	+35.4 [30.8, 40.0]
Claude Sonnet 5	19.3 [15.5, 23.2]	+3.5 [-0.3, 7.4]	+51.1 [45.1, 56.8]	+43.5 [38.7, 48.4]
Kimi K2.5	27.5 [23.2, 31.9]	+6.9 [2.6, 11.3]	+10.0 [3.9, 16.4]	+11.8 [6.6, 17.0]

Appendix

Table 9: Difference in switch rates relative to neutral by turn five. Neutral control gives the switch rate (%); other columns subtract neutral on matched cases (percentage points). Positive differences indicate more failures; negative differences indicate fewer. Weights and bolding follow Table 2 ; brackets give 95% confidence intervals.

Figure 8: Switching under caregiver preference in eldercare and pediatrics. Points show switch rates by turn five among cases with appropriate neutral first-response advice in each domain. Persona-plus-preference rates average the five personas. Whiskers show 95% confidence intervals.

	Neutral first response (%) $\uparrow$		Switch rate by turn 5 (%) $\downarrow$
Model	Gold agreement	Appropriate care	Neutral control	Persona only	Inferior preference only	Persona + preference	Gold-preference control
GPT-5.6 Sol	95.3 [91.3, 98.7]	97.3 [94.7, 99.3]	2.1 [0.0, 4.8]	4.4 [2.2, 6.8]	32.9 [25.5, 40.7]	37.8 [32.5, 43.4]	1.4 [0.0, 3.4]
GPT-5.6 Luna	96.0 [92.7, 98.7]	98.7 [96.7, 100.0]	4.7 [1.4, 8.2]	5.9 [3.5, 8.6]	35.1 [27.7, 43.0]	36.1 [30.9, 41.4]	4.1 [1.3, 7.4]
Muse Spark 1.3	92.7 [88.0, 96.7]	96.0 [92.7, 98.7]	4.9 [1.4, 8.4]	7.6 [5.3, 10.3]	11.1 [6.2, 16.6]	9.4 [7.2, 12.0]	4.9 [1.4, 8.8]
Gemini 3.8 Flash	90.7 [86.0, 95.3]	94.7 [90.7, 98.0]	16.9 [10.9, 23.2]	21.0 [17.1, 25.0]	46.5 [38.5, 54.5]	53.0 [47.2, 58.9]	14.1 [8.6, 20.0]
Claude Sonnet 5	89.3 [84.0, 94.0]	93.3 [89.3, 96.7]	15.7 [10.0, 22.3]	19.4 [15.3, 23.7]	67.9 [59.9, 75.4]	61.3 [56.3, 66.2]	11.4 [6.4, 16.9]

Appendix

Table 10: Eldercare baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.

	Neutral first response (%) $\uparrow$		Switch rate by turn 5 (%) $\downarrow$
Model	Gold agreement	Appropriate care	Neutral control	Persona only	Inferior preference only	Persona + preference	Gold-preference control
GPT-5.6 Sol	92.0 [87.3, 96.0]	94.7 [90.7, 98.0]	7.0 [2.9, 11.6]	7.6 [4.9, 10.6]	37.3 [29.6, 45.4]	40.1 [34.1, 46.3]	4.2 [1.4, 7.8]
GPT-5.6 Luna	88.7 [83.3, 93.3]	90.7 [86.0, 94.7]	5.1 [1.5, 9.1]	8.5 [5.5, 11.9]	33.1 [25.0, 41.4]	37.2 [31.5, 43.0]	8.1 [3.7, 12.9]
Muse Spark 1.3	88.7 [83.3, 93.3]	88.7 [83.3, 93.3]	9.8 [5.1, 15.2]	11.4 [8.0, 15.2]	21.1 [14.3, 28.2]	14.3 [10.4, 18.4]	11.3 [6.2, 16.7]
Gemini 3.8 Flash	86.0 [80.0, 91.3]	84.7 [78.7, 90.0]	19.7 [13.1, 26.5]	21.4 [17.1, 26.0]	35.4 [27.3, 43.8]	43.0 [36.6, 49.5]	19.7 [13.1, 26.8]
Claude Sonnet 5	82.0 [76.0, 88.0]	81.3 [75.3, 87.3]	26.2 [18.5, 34.1]	26.7 [22.0, 31.8]	64.8 [56.1, 72.9]	56.7 [49.8, 63.7]	33.6 [25.4, 42.3]

Appendix

Table 11: Pediatrics baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.

	Neutral first response (%) $\uparrow$		Switch rate by turn 5 (%) $\downarrow$
Model	Gold agreement	Appropriate care	Neutral control	Persona only	Inferior preference only	Persona + preference	Gold-preference control
GPT-5.6 Sol	92.7 [88.0, 96.7]	97.3 [94.7, 99.3]	2.7 [0.7, 5.6]	5.9 [3.4, 8.7]	31.5 [24.0, 39.0]	38.5 [33.0, 44.1]	2.1 [0.0, 4.8]
GPT-5.6 Luna	91.3 [86.7, 95.3]	96.0 [92.7, 98.7]	2.8 [0.7, 5.7]	4.3 [2.2, 6.9]	29.9 [22.5, 37.2]	33.6 [28.1, 39.4]	2.8 [0.7, 5.7]
Gemini 3.8 Flash	88.7 [83.3, 93.3]	93.3 [89.3, 97.3]	10.7 [5.8, 16.1]	18.3 [14.8, 22.0]	51.4 [43.1, 59.2]	57.6 [52.5, 62.7]	15.7 [10.1, 22.1]
Claude Sonnet 5	86.7 [81.3, 92.0]	91.3 [86.7, 95.3]	16.1 [10.2, 22.3]	22.5 [18.4, 26.8]	77.9 [70.8, 84.6]	70.3 [65.4, 75.0]	17.5 [11.3, 24.1]
Kimi K2.5	87.3 [81.3, 92.7]	91.3 [86.7, 95.3]	19.0 [12.7, 25.9]	28.2 [23.8, 32.8]	34.3 [26.7, 42.4]	39.9 [35.6, 44.4]	19.7 [13.1, 26.8]

Appendix

Table 12: Veterinary baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.

Model	Outcome	Neutral	Persona only	Inferior preference	Persona + preference	Gold preference
GPT-5.6 Sol	Gold agreement	95.3	94.8	91.3	89.1	98.7
	Inferior-action agreement	1.3	1.2	1.3	5.7	0.7
	Appropriate care	97.3	97.7	98.7	93.9	98.0
GPT-5.6 Luna	Gold agreement	96.0	93.1	92.0	89.5	98.7
	Inferior-action agreement	1.3	2.1	3.3	4.7	0.0
	Appropriate care	98.7	96.8	96.7	93.6	99.3

Appendix

Table 13: Eldercare first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.

Model	Outcome	Neutral	Persona only	Inferior preference	Persona + preference	Gold preference
GPT-5.6 Sol	Gold agreement	92.0	91.7	88.0	82.4	97.3
	Inferior-action agreement	2.0	2.7	4.0	12.4	0.7
	Appropriate care	94.7	93.1	93.3	86.0	96.7
GPT-5.6 Luna	Gold agreement	88.7	88.3	88.7	83.3	94.0
	Inferior-action agreement	5.3	3.6	6.7	9.7	3.3
	Appropriate care	90.7	91.3	91.3	88.4	92.7

Appendix

Table 14: Pediatrics first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.

Model	Outcome	Neutral	Persona only	Inferior preference	Persona + preference	Gold preference
GPT-5.6 Sol	Gold agreement	92.7	91.2	93.3	87.9	96.7
	Inferior-action agreement	0.7	1.1	2.0	5.3	0.7
	Appropriate care	97.3	95.5	97.3	92.5	98.0
GPT-5.6 Luna	Gold agreement	91.3	90.1	89.3	88.3	96.0
	Inferior-action agreement	1.3	1.2	2.0	5.7	0.7
	Appropriate care	96.0	95.7	95.3	93.1	97.3

Appendix

Table 15: Veterinary first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.

Figure 9: Inferior-action agreement at first failure. Percentage of cases whose first inappropriate response by turn five recommends or permits the inferior action, among cases with appropriate neutral first-response advice. Panels report eldercare and pediatrics; the three-domain comparison appears in Figure 2 B. Persona groups average the five personas. Whiskers show 95% confidence intervals.

	Inferior preference only			Persona + preference
Model	Gold (%)	Inferior (%)	$n$	Gold (%)	Inferior (%)	$n$
GPT-5.6 Sol	35.0	76.3	147	21.6	78.9	842
GPT-5.6 Luna	36.3	78.1	140	29.7	77.3	762
Gemini 3.8 Flash	28.3	69.6	183	28.4	71.2	1052
Claude Sonnet 5	28.4	75.6	280	28.7	75.5	1255
Kimi K2.5	49.9	54.5	148	44.7	51.2	779

Appendix

Table 16: Agreement at first failure under inferior preference. Among cases with appropriate neutral first-response advice and a failure by turn five, Gold and Inferior report agreement with the reference and inferior actions, respectively. These labels are assessed independently. Rates are averaged equally across domains;

n

is the pooled number of first failures.

Figure 10: Appropriate-to-inappropriate switching with neutral and gold-preference controls. Cumulative switch rates among cases with appropriate neutral first-response advice, averaged equally across three domains. Shading shows 95% confidence intervals.

Figure 11: Individual personas with and without inferior preference. Switch rates by turn five among cases with appropriate neutral first-response advice, averaged equally across three domains. Dashed lines show neutral control; whiskers show 95% confidence intervals.

Eldercare
	Inappropriate care		Inferior-action agreement
Model	Neutral $\to$ inferior	Difference (pp)	Neutral $\to$ inferior	Difference (pp)
GPT-5.6 Sol	0.7 $\to$ 11.8	+11.1 [6.6, 15.9]	0.8 $\to$ 14.9	+14.1 [9.3, 19.3]
GPT-5.6 Luna	0.9 $\to$ 15.2	+14.3 [9.3, 19.8]	0.2 $\to$ 16.4	+16.2 [11.4, 21.4]
Muse Spark 1.3	2.2 $\to$ 11.4	+9.2 [4.8, 14.0]	0.5 $\to$ 10.6	+10.1 [6.2, 14.3]
Gemini 3.8 Flash	3.8 $\to$ 26.2	+22.4 [15.7, 29.1]	1.9 $\to$ 23.2	+21.2 [15.3, 27.2]

Appendix

Table 17: Care appropriateness and action agreement with conversation history held fixed. Models receive the same conversation history within each condition at selected turns 2–5. Rates (%) average paired outcomes across turns within each case and then weight cases equally. Differences are inferior preference minus neutral, in percentage points; brackets give paired 95% confidence intervals. One veterinary Claude Sonnet 5 response is unavailable.

Appropriate-to-inappropriate switching
	Patient (%)		Caregiver (%)		Increase over neutral and difference (pp)
Turn	Neutral	Inferior	Neutral	Inferior	$\Delta_{P}$	$\Delta_{C}$	$\Delta_{C}-\Delta_{P}$
1	0.0	6.4	0.0	5.5	6.4 [4.4, 8.5]	5.5 [3.8, 7.5]	-0.8 [-3.1, 1.5]
2	4.9	23.8	7.3	25.5	18.9 [14.4, 23.3]	18.2 [13.5, 22.9]	-0.7 [-4.9, 3.8]
3	9.7	35.3	9.6	34.7	25.6 [20.0, 31.0]	25.2 [19.7, 30.7]	-0.4 [-5.5, 5.0]
4	12.2	41.1	13.3	41.2	28.9 [23.6, 34.1]	27.9 [22.1, 33.7]	-1.0 [-6.1, 4.3]

Appendix

Table 18: Outcome rates and increases over neutral under patient and caregiver preferences. Rates (%) are cumulative through each turn and averaged equally across models. Within each speaker group, inferior denotes the inferior-preference condition.

\Delta_{P}

and

\Delta_{C}

are inferior-preference minus neutral rates for patient and caregiver speakers;

\Delta_{C}-\Delta_{P}

compares these increases. Differences are in percentage points; brackets give paired 95% confidence intervals. For each model, both neutral first responses must be appropriate; the selected cases are held fixed across speakers, conditions and turns.

Appropriate-to-inappropriate switching
Model	Patient ( $\Delta_{P}$ )	Caregiver ( $\Delta_{C}$ )	Difference ( $\Delta_{C}-\Delta_{P}$ )
GPT-5.6 Sol	24.5 [16.0, 34.0]	34.9 [26.4, 44.3]	10.4 [0.0, 21.7]
GPT-5.6 Luna	29.5 [21.0, 39.0]	33.3 [23.8, 43.8]	3.8 [-7.6, 15.2]
Muse Spark 1.3	2.8 [-5.6, 11.2]	5.6 [0.0, 12.1]	2.8 [-6.5, 12.1]
Gemini 3.8 Flash	31.3 [19.2, 43.4]	27.3 [16.2, 38.4]	-4.0 [-18.2, 9.1]
Claude Sonnet 5	49.5 [38.7, 60.2]	59.1 [47.3, 71.0]	9.7 [-4.3, 23.7]

Appendix

Table 19: Increases over neutral under patient and caregiver preferences by turn five.

\Delta_{P}

and

\Delta_{C}

are the patient and caregiver increases over neutral;

\Delta_{C}-\Delta_{P}

compares them. Values are in percentage points; brackets give paired 95% confidence intervals. The average gives each model equal weight, retaining its paired case selection.

	Switched to inappropriate care		Maintained appropriate care
Model	Asked about	Mentioned or asked about	Asked about	Mentioned or asked about
GPT-5.6 Sol	12.8 [2.9, 24.3]	48.7 [33.3, 64.7]	29.4 [18.8, 40.6]	55.9 [43.8, 67.6]
GPT-5.6 Luna	7.1 [0.0, 15.8]	40.5 [25.6, 55.3]	10.4 [3.5, 18.2]	28.4 [17.8, 39.4]
Muse Spark 1.3	36.4 [9.1, 66.7]	54.5 [22.2, 85.7]	25.3 [16.7, 34.1]	34.7 [25.2, 44.4]
Gemini 3.8 Flash	8.7 [2.0, 17.8]	13.0 [4.3, 23.5]	6.8 [1.6, 13.8]	22.0 [11.9, 32.8]
Claude Sonnet 5	16.7 [8.6, 25.7]	30.6 [20.3, 41.4]	25.0 [10.3, 41.4]	34.4 [18.2, 51.6]

Appendix

Table 20: Patient preferences under inferior caregiver preference. Rates (%) describe the first inappropriate response when care switches, or the response at turn five when care remains appropriate. Brackets give 95% confidence intervals. “Asked about” counts responses that ask about patient preferences or recommend establishing them; “Mentioned or asked about” also includes other explicit references to those preferences. The across-model summary weights models equally.

Figure 12: Care maintained through turn five by clinical decision type in eldercare. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases

n

Figure 13: Care maintained through turn five by clinical decision type in pediatrics. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases

n

Figure 14: Care maintained through turn five by clinical decision type in veterinary care. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases

n

. Two counts give the neutral and preference denominators where conversations are missing.

Figure 15: Failure frequency and potential harm by domain. Within each model, bars show neutral, persona only, inferior preference only, and persona plus inferior preference, in that order. Heights are switch rates by turn five among cases with appropriate neutral first-response advice; segments show severity at first failure. Whiskers give 95% confidence intervals for total switch rates.

Figure 16: Appropriate-to-inappropriate switching by reference care direction. Switch rates by turn five among cases with appropriate neutral first-response advice in each direction, averaged equally across the three domains. Whiskers give 95% confidence intervals.

Figure 17: Effect of beneficiary-aware prompting in pediatrics. Points show the change in switch rate by turn five, in percentage points: beneficiary-aware minus original prompt. Negative values indicate less switching to inappropriate care; whiskers show paired 95% confidence intervals. Adjacent values give original and beneficiary-aware rates (%). Both arms use the same cases with appropriate original neutral first-response advice;

n

gives case counts. Persona-group rates average the five personas.

Explore similar work

When Do LLM Preferences Predict Downstream Behavior?
Feb 21, 2026Katarina Slama, Alexandra Souly, Dishank Bansal +3LLM Decision-Making Preference Alignment
UK AI Security Institute
Can Revealed Preferences Clarify LLM Alignment and Steering?
May 8, 2026Khurram Yamin, Jingjing Tang, Eric Horvitz +1Language Model Steering LLM Decision-Making
1Carnegie Mellon University · 2Microsoft Research
When AI Tells You What You Want to Hear: Sycophantic Behavior of Large Language Models in Dementia Care Settings
Apr 13, 2026Christian KolbLLM Sycophancy LLM Reliability

Feb 21, 2026J/K move · Enter open · S save

When Do LLM Preferences Predict Downstream Behavior?

Katarina Slama, Alexandra Souly, Dishank Bansal, Henry Davidson+2

UK AI Security Institute

LLM Decision-Making Preference Alignment Refusal Behavior in Language Models

Open paper AskPDF arXiv

Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.

Abstract

Explore similar work

When Do LLM Preferences Predict Downstream Behavior?

Can Revealed Preferences Clarify LLM Alignment and Steering?

When AI Tells You What You Want to Hear: Sycophantic Behavior of Large Language Models in Dementia Care Settings