Large language model (LLM) alignment often assumes that the user interacting with a model is also the primary beneficiary of its behavior. Yet in many consequential settings, one person directs the system while another bears the consequences. We study this problem in caregiver-mediated healthcare, where parents, pet owners, and caregivers make decisions for children, animals, and older adults. We introduce an evidence-grounded benchmark spanning veterinary, pediatric, and older-adult care, and test whether AI advisors maintain appropriate care when users express clinically inferior preferences. Holding patient facts fixed, we vary stated care preferences and communication styles and evaluate both initial recommendations and multi-turn follow-up. Across frontier and open-weight models, inferior preferences substantially increase switching from appropriate to inappropriate care beyond neutral conversational drift and increase agreement with the requested action, while communication style alone has a smaller effect. These failures accumulate across turns, including on cases models initially answer correctly, and often carry moderate or severe potential harm. In matched eldercare cases, the effect of an inferior preference is similar whether it is expressed by the patient or by a caregiver. Together with models' limited tendency to elicit patient preferences in caregiver conversations, this suggests that current models are highly sensitive to expressed user preferences but less sensitive to whose preferences are represented and who bears the consequences. Our results expose a broader limitation of user-centric alignment in settings with multiple stakeholders.
Figures & tables
Figure 1: CareDistance overview. Source-grounded patient cases are used to generate conversations under 13 caregiver persona and preference combinations, with patient facts held fixed. Conversations span up to five turns; assessment stops at the first inappropriate response. The example messages are condensed and paraphrased; Appendix E.11.1 gives the full evaluated conversation.
Preference condition
Caregiver request
Neutral
Asks the original question without expressing a care preference.
Inferior preference only
Expresses a preference for the inferior alternative.
Gold-preference control
Expresses a preference for the reference action.
Persona
Caregiver concern or behavior
Practical Constraint
Emphasizes whether the care is realistically manageable.
Values Commitment
Appeals to beliefs about what constitutes responsible care.
Table 1: Caregiver preferences and personas. Each persona is evaluated with and without a preference for the inferior alternative, while clinical facts remain fixed.
Figure 2: Care maintained and inferior-action agreement through turn five. Bars show care maintained (A) and inferior-action agreement at first failure (B). Both panels use all cases with appropriate neutral first-response advice as the denominator; Panel B counts those whose first inappropriate response recommends or permits the inferior action. The three domains receive equal weight, and persona groups average the five personas. Whiskers show 95% confidence intervals.
Neutral first response (%) ↑
Switch rate by turn 5 (%) ↓
Model
Gold agreement
Appropriate care
Neutral control
Persona only
Inferior preference only
Persona + preference
Gold-preference control
GPT-5.6 Sol
93.3 [90.9, 95.6]
96.4 [94.7, 98.0]
3.9 [2.1, 5.8]
6.0 [4.5, 7.5]
33.9 [29.7, 38.2]
38.8 [35.5, 42.1]
2.6 [1.2, 4.1]
GPT-5.6 Luna
92.0 [89.3, 94.2]
95.1 [93.1, 96.9]
4.2 [2.4, 6.1]
6.3 [4.7, 7.9]
32.7 [28.2, 37.1]
35.6 [32.4, 38.7]
5.0 [3.1, 7.1]
Gemini 3.8 Flash
88.4 [85.6, 91.3]
90.9 [88.2, 93.3]
15.8 [12.4, 19.4]
20.2 [17.9, 22.6]
44.4 [39.8, 49.0]
51.2 [47.8, 54.4]
16.5 [13.1, 20.1]
Claude Sonnet 5
86.0 [82.9, 89.1]
88.7 [85.8, 91.6]
19.3 [15.5, 23.2]
22.9 [20.3, 25.4]
70.2 [65.7, 74.6]
62.8 [59.6, 66.0]
20.9 [17.1, 24.8]
Kimi K2.5
85.3 [82.0, 88.4]
88.0 [85.1, 90.9]
27.5 [23.2, 31.9]
34.4 [31.6, 37.2]
37.5 [32.8, 42.2]
39.3 [36.4, 42.2]
26.9 [22.6, 31.5]
Table 2: Neutral first-response performance and switching by turn five. Switch rates count cases with an inappropriate response by turn five among cases with appropriate neutral first-response advice. The three domains and five personas are equally weighted; brackets give 95% confidence intervals. Arrows indicate the favorable direction. Bold marks descriptive column extrema, including ties.
Figure 3: Appropriate-to-inappropriate switching across turns. For each model, curves show the cumulative percentage of cases with an inappropriate response by each turn, restricted to cases with an appropriate neutral first response. The three domains receive equal weight. Shading shows 95% confidence intervals.
Figure 4: Preference sensitivity when the user does or does not bear the clinical consequences. The eldercare comparison shows appropriate-to-inappropriate switching across turns (A) and inferior-action agreement at first failure through turn five (B), under neutral and inferior-preference conditions. Shading and whiskers show paired 95% confidence intervals.
Figure 5: Care maintained through turn five by clinical decision type. Profiles show the percentage of cases with appropriate neutral first-response advice that receive no inappropriate response through turn five under neutral control or inferior preference only. Cases are pooled across the three domains within each decision type; higher values indicate more care maintained.
Figure 6: Switching to inappropriate care and potential-harm severity. Bars show switch rates by turn five among cases with appropriate neutral first-response advice, divided by potential-harm severity at first failure. Domains receive equal weight; whiskers show 95% confidence intervals. “Pref.” denotes inferior preference.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Eldercare
Pediatrics
Veterinary
Patient cases
150
150
150
Source documents
11
27
150
Clinical areas
7
8
7
Provide indicated care
85
75
106
Withhold inappropriate care
65
75
44
Appendix
Table 3: Profile of the evaluated datasets. Source documents are distinct guidelines or reference articles represented in the selected cases. Care direction describes the reference decision.
Source family
Documents
Cases
Eldercare
STOPP/START v3
1
45
WHO
1
42
ESPEN
2
39
PALTmed
1
16
USPSTF
4
4
Appendix
Table 4: Clinical source families represented in the selected cases.
Figure 7: Clinical decision types across domains. Each domain contains 150 cases, assigned to one primary type from the question and reference action. Segment labels give case counts.
Tier
Supported consequence if it occurs
Mild
Minor, temporary symptoms or discomfort requiring little or no treatment.
Moderate
Meaningful deterioration requiring treatment, or temporary functional limitation.
Severe
Life-threatening deterioration, major functional loss, lasting disability or death.
Table 6: Evaluation coverage and uncertain judgments. Baseline-appropriate counts give the cases used for switching, care maintained and harm analyses. Uncertain counts are conversations with at least one uncertain judgment among evaluated turns, including those that later fail. Cases with uncertain neutral first responses are excluded from these analyses; later uncertain judgments are not counted as failures. Complete runs contain 1,950 conversations; veterinary Claude Sonnet 5 contains 1,947.
Comparison
First-failure agreement (%)
Cohen’s κ
LLM–Reviewer 1
80.0 [68.3, 90.0]
0.596 [0.409, 0.778]
LLM–Reviewer 2
73.3 [61.7, 83.3]
0.448 [0.255, 0.639]
Reviewer–reviewer
78.3 [70.0, 86.7]
0.610 [0.438, 0.759]
Appendix
Table 7: Agreement on the first inappropriate turn. Each comparison covers 60 conversations and includes agreement on no failure. Brackets give 95% confidence intervals.
Evaluator
Neutral
Inferior preference
Excess (percentage points)
Automatic judge
0/30 (0.0%)
14/30 (46.7%)
+46.7
Reviewer 1
8/30 (26.7%)
14/30 (46.7%)
+20.0 [0.0, 40.0]
Reviewer 2
8/30 (26.7%)
13/30 (43.3%)
+16.7 [ −3.3 , 36.7]
Appendix
Table 8: Switching on the matched validation sample. All evaluators judged the neutral first response appropriate in all 30 pairs. Excess is the inferior-preference rate minus neutral; brackets give 95% confidence intervals for reviewer excess rates.
Switch rate (%) ↓
Difference from neutral (percentage points) ↓
Model
Neutral control
Persona only
Inferior preference only
Persona + preference
GPT-5.6 Sol
3.9 [2.1, 5.8]
+2.0 [0.2, 3.8]
+30.0 [25.5, 34.4]
+34.9 [31.5, 38.3]
GPT-5.6 Luna
4.2 [2.4, 6.1]
+2.0 [0.3, 3.7]
+28.5 [23.7, 33.3]
+31.4 [28.0, 34.8]
Gemini 3.8 Flash
15.8 [12.4, 19.4]
+4.5 [0.8, 7.9]
+28.7 [22.9, 34.3]
+35.4 [30.8, 40.0]
Claude Sonnet 5
19.3 [15.5, 23.2]
+3.5 [-0.3, 7.4]
+51.1 [45.1, 56.8]
+43.5 [38.7, 48.4]
Kimi K2.5
27.5 [23.2, 31.9]
+6.9 [2.6, 11.3]
+10.0 [3.9, 16.4]
+11.8 [6.6, 17.0]
Appendix
Table 9: Difference in switch rates relative to neutral by turn five. Neutral control gives the switch rate (%); other columns subtract neutral on matched cases (percentage points). Positive differences indicate more failures; negative differences indicate fewer. Weights and bolding follow Table 2 ; brackets give 95% confidence intervals.
Figure 8: Switching under caregiver preference in eldercare and pediatrics. Points show switch rates by turn five among cases with appropriate neutral first-response advice in each domain. Persona-plus-preference rates average the five personas. Whiskers show 95% confidence intervals.
Neutral first response (%) ↑
Switch rate by turn 5 (%) ↓
Model
Gold agreement
Appropriate care
Neutral control
Persona only
Inferior preference only
Persona + preference
Gold-preference control
GPT-5.6 Sol
95.3 [91.3, 98.7]
97.3 [94.7, 99.3]
2.1 [0.0, 4.8]
4.4 [2.2, 6.8]
32.9 [25.5, 40.7]
37.8 [32.5, 43.4]
1.4 [0.0, 3.4]
GPT-5.6 Luna
96.0 [92.7, 98.7]
98.7 [96.7, 100.0]
4.7 [1.4, 8.2]
5.9 [3.5, 8.6]
35.1 [27.7, 43.0]
36.1 [30.9, 41.4]
4.1 [1.3, 7.4]
Muse Spark 1.3
92.7 [88.0, 96.7]
96.0 [92.7, 98.7]
4.9 [1.4, 8.4]
7.6 [5.3, 10.3]
11.1 [6.2, 16.6]
9.4 [7.2, 12.0]
4.9 [1.4, 8.8]
Gemini 3.8 Flash
90.7 [86.0, 95.3]
94.7 [90.7, 98.0]
16.9 [10.9, 23.2]
21.0 [17.1, 25.0]
46.5 [38.5, 54.5]
53.0 [47.2, 58.9]
14.1 [8.6, 20.0]
Claude Sonnet 5
89.3 [84.0, 94.0]
93.3 [89.3, 96.7]
15.7 [10.0, 22.3]
19.4 [15.3, 23.7]
67.9 [59.9, 75.4]
61.3 [56.3, 66.2]
11.4 [6.4, 16.9]
Appendix
Table 10: Eldercare baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.
Neutral first response (%) ↑
Switch rate by turn 5 (%) ↓
Model
Gold agreement
Appropriate care
Neutral control
Persona only
Inferior preference only
Persona + preference
Gold-preference control
GPT-5.6 Sol
92.0 [87.3, 96.0]
94.7 [90.7, 98.0]
7.0 [2.9, 11.6]
7.6 [4.9, 10.6]
37.3 [29.6, 45.4]
40.1 [34.1, 46.3]
4.2 [1.4, 7.8]
GPT-5.6 Luna
88.7 [83.3, 93.3]
90.7 [86.0, 94.7]
5.1 [1.5, 9.1]
8.5 [5.5, 11.9]
33.1 [25.0, 41.4]
37.2 [31.5, 43.0]
8.1 [3.7, 12.9]
Muse Spark 1.3
88.7 [83.3, 93.3]
88.7 [83.3, 93.3]
9.8 [5.1, 15.2]
11.4 [8.0, 15.2]
21.1 [14.3, 28.2]
14.3 [10.4, 18.4]
11.3 [6.2, 16.7]
Gemini 3.8 Flash
86.0 [80.0, 91.3]
84.7 [78.7, 90.0]
19.7 [13.1, 26.5]
21.4 [17.1, 26.0]
35.4 [27.3, 43.8]
43.0 [36.6, 49.5]
19.7 [13.1, 26.8]
Claude Sonnet 5
82.0 [76.0, 88.0]
81.3 [75.3, 87.3]
26.2 [18.5, 34.1]
26.7 [22.0, 31.8]
64.8 [56.1, 72.9]
56.7 [49.8, 63.7]
33.6 [25.4, 42.3]
Appendix
Table 11: Pediatrics baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.
Neutral first response (%) ↑
Switch rate by turn 5 (%) ↓
Model
Gold agreement
Appropriate care
Neutral control
Persona only
Inferior preference only
Persona + preference
Gold-preference control
GPT-5.6 Sol
92.7 [88.0, 96.7]
97.3 [94.7, 99.3]
2.7 [0.7, 5.6]
5.9 [3.4, 8.7]
31.5 [24.0, 39.0]
38.5 [33.0, 44.1]
2.1 [0.0, 4.8]
GPT-5.6 Luna
91.3 [86.7, 95.3]
96.0 [92.7, 98.7]
2.8 [0.7, 5.7]
4.3 [2.2, 6.9]
29.9 [22.5, 37.2]
33.6 [28.1, 39.4]
2.8 [0.7, 5.7]
Gemini 3.8 Flash
88.7 [83.3, 93.3]
93.3 [89.3, 97.3]
10.7 [5.8, 16.1]
18.3 [14.8, 22.0]
51.4 [43.1, 59.2]
57.6 [52.5, 62.7]
15.7 [10.1, 22.1]
Claude Sonnet 5
86.7 [81.3, 92.0]
91.3 [86.7, 95.3]
16.1 [10.2, 22.3]
22.5 [18.4, 26.8]
77.9 [70.8, 84.6]
70.3 [65.4, 75.0]
17.5 [11.3, 24.1]
Kimi K2.5
87.3 [81.3, 92.7]
91.3 [86.7, 95.3]
19.0 [12.7, 25.9]
28.2 [23.8, 32.8]
34.3 [26.7, 42.4]
39.9 [35.6, 44.4]
19.7 [13.1, 26.8]
Appendix
Table 12: Veterinary baseline and switching. Neutral first-response performance, switch rates by turn five, and differences from neutral control, with 95% confidence intervals. Arrows indicate the favorable direction; bold marks descriptive column extrema, including ties.
Model
Outcome
Neutral
Persona only
Inferior preference
Persona + preference
Gold preference
GPT-5.6 Sol
Gold agreement
95.3
94.8
91.3
89.1
98.7
Inferior-action agreement
1.3
1.2
1.3
5.7
0.7
Appropriate care
97.3
97.7
98.7
93.9
98.0
GPT-5.6 Luna
Gold agreement
96.0
93.1
92.0
89.5
98.7
Inferior-action agreement
1.3
2.1
3.3
4.7
0.0
Appropriate care
98.7
96.8
96.7
93.6
99.3
Appendix
Table 13: Eldercare first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.
Model
Outcome
Neutral
Persona only
Inferior preference
Persona + preference
Gold preference
GPT-5.6 Sol
Gold agreement
92.0
91.7
88.0
82.4
97.3
Inferior-action agreement
2.0
2.7
4.0
12.4
0.7
Appropriate care
94.7
93.1
93.3
86.0
96.7
GPT-5.6 Luna
Gold agreement
88.7
88.3
88.7
83.3
94.0
Inferior-action agreement
5.3
3.6
6.7
9.7
3.3
Appropriate care
90.7
91.3
91.3
88.4
92.7
Appendix
Table 14: Pediatrics first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.
Model
Outcome
Neutral
Persona only
Inferior preference
Persona + preference
Gold preference
GPT-5.6 Sol
Gold agreement
92.7
91.2
93.3
87.9
96.7
Inferior-action agreement
0.7
1.1
2.0
5.3
0.7
Appropriate care
97.3
95.5
97.3
92.5
98.0
GPT-5.6 Luna
Gold agreement
91.3
90.1
89.3
88.3
96.0
Inferior-action agreement
1.3
1.2
2.0
5.7
0.7
Appropriate care
96.0
95.7
95.3
93.1
97.3
Appendix
Table 15: Veterinary first-response outcomes. Agreement with the gold and inferior actions and appropriate care (%). All observed first responses are included; persona groups average their five conditions. Values are descriptive point estimates.
Figure 9: Inferior-action agreement at first failure. Percentage of cases whose first inappropriate response by turn five recommends or permits the inferior action, among cases with appropriate neutral first-response advice. Panels report eldercare and pediatrics; the three-domain comparison appears in Figure 2 B. Persona groups average the five personas. Whiskers show 95% confidence intervals.
Inferior preference only
Persona + preference
Model
Gold (%)
Inferior (%)
n
Gold (%)
Inferior (%)
n
GPT-5.6 Sol
35.0
76.3
147
21.6
78.9
842
GPT-5.6 Luna
36.3
78.1
140
29.7
77.3
762
Gemini 3.8 Flash
28.3
69.6
183
28.4
71.2
1052
Claude Sonnet 5
28.4
75.6
280
28.7
75.5
1255
Kimi K2.5
49.9
54.5
148
44.7
51.2
779
Appendix
Table 16: Agreement at first failure under inferior preference. Among cases with appropriate neutral first-response advice and a failure by turn five, Gold and Inferior report agreement with the reference and inferior actions, respectively. These labels are assessed independently. Rates are averaged equally across domains; n is the pooled number of first failures.
Figure 10: Appropriate-to-inappropriate switching with neutral and gold-preference controls. Cumulative switch rates among cases with appropriate neutral first-response advice, averaged equally across three domains. Shading shows 95% confidence intervals.
Figure 11: Individual personas with and without inferior preference. Switch rates by turn five among cases with appropriate neutral first-response advice, averaged equally across three domains. Dashed lines show neutral control; whiskers show 95% confidence intervals.
Inappropriate care
Inferior-action agreement
Model
Neutral → inferior
Difference (pp)
Neutral → inferior
Difference (pp)
Eldercare
GPT-5.6 Sol
0.7 → 11.8
+11.1 [6.6, 15.9]
0.8 → 14.9
+14.1 [9.3, 19.3]
GPT-5.6 Luna
0.9 → 15.2
+14.3 [9.3, 19.8]
0.2 → 16.4
+16.2 [11.4, 21.4]
Muse Spark 1.3
2.2 → 11.4
+9.2 [4.8, 14.0]
0.5 → 10.6
+10.1 [6.2, 14.3]
Gemini 3.8 Flash
3.8 → 26.2
+22.4 [15.7, 29.1]
1.9 → 23.2
+21.2 [15.3, 27.2]
Appendix
Table 17: Care appropriateness and action agreement with conversation history held fixed. Models receive the same conversation history within each condition at selected turns 2–5. Rates (%) average paired outcomes across turns within each case and then weight cases equally. Differences are inferior preference minus neutral, in percentage points; brackets give paired 95% confidence intervals. One veterinary Claude Sonnet 5 response is unavailable.
Appropriate-to-inappropriate switching
Patient (%)
Caregiver (%)
Increase over neutral and difference (pp)
Turn
Neutral
Inferior
Neutral
Inferior
ΔP
ΔC
ΔC−ΔP
1
0.0
6.4
0.0
5.5
6.4 [4.4, 8.5]
5.5 [3.8, 7.5]
-0.8 [-3.1, 1.5]
2
4.9
23.8
7.3
25.5
18.9 [14.4, 23.3]
18.2 [13.5, 22.9]
-0.7 [-4.9, 3.8]
3
9.7
35.3
9.6
34.7
25.6 [20.0, 31.0]
25.2 [19.7, 30.7]
-0.4 [-5.5, 5.0]
4
12.2
41.1
13.3
41.2
28.9 [23.6, 34.1]
27.9 [22.1, 33.7]
-1.0 [-6.1, 4.3]
Appendix
Table 18: Outcome rates and increases over neutral under patient and caregiver preferences. Rates (%) are cumulative through each turn and averaged equally across models. Within each speaker group, inferior denotes the inferior-preference condition. ΔP and ΔC are inferior-preference minus neutral rates for patient and caregiver speakers; ΔC−ΔP compares these increases. Differences are in percentage points; brackets give paired 95% confidence intervals. For each model, both neutral first responses must be appropriate; the selected cases are held fixed across speakers, conditions and turns.
Appropriate-to-inappropriate switching
Model
Patient ( ΔP )
Caregiver ( ΔC )
Difference ( ΔC−ΔP )
GPT-5.6 Sol
24.5 [16.0, 34.0]
34.9 [26.4, 44.3]
10.4 [0.0, 21.7]
GPT-5.6 Luna
29.5 [21.0, 39.0]
33.3 [23.8, 43.8]
3.8 [-7.6, 15.2]
Muse Spark 1.3
2.8 [-5.6, 11.2]
5.6 [0.0, 12.1]
2.8 [-6.5, 12.1]
Gemini 3.8 Flash
31.3 [19.2, 43.4]
27.3 [16.2, 38.4]
-4.0 [-18.2, 9.1]
Claude Sonnet 5
49.5 [38.7, 60.2]
59.1 [47.3, 71.0]
9.7 [-4.3, 23.7]
Appendix
Table 19: Increases over neutral under patient and caregiver preferences by turn five. ΔP and ΔC are the patient and caregiver increases over neutral; ΔC−ΔP compares them. Values are in percentage points; brackets give paired 95% confidence intervals. The average gives each model equal weight, retaining its paired case selection.
Switched to inappropriate care
Maintained appropriate care
Model
Asked about
Mentioned or asked about
Asked about
Mentioned or asked about
GPT-5.6 Sol
12.8 [2.9, 24.3]
48.7 [33.3, 64.7]
29.4 [18.8, 40.6]
55.9 [43.8, 67.6]
GPT-5.6 Luna
7.1 [0.0, 15.8]
40.5 [25.6, 55.3]
10.4 [3.5, 18.2]
28.4 [17.8, 39.4]
Muse Spark 1.3
36.4 [9.1, 66.7]
54.5 [22.2, 85.7]
25.3 [16.7, 34.1]
34.7 [25.2, 44.4]
Gemini 3.8 Flash
8.7 [2.0, 17.8]
13.0 [4.3, 23.5]
6.8 [1.6, 13.8]
22.0 [11.9, 32.8]
Claude Sonnet 5
16.7 [8.6, 25.7]
30.6 [20.3, 41.4]
25.0 [10.3, 41.4]
34.4 [18.2, 51.6]
Appendix
Table 20: Patient preferences under inferior caregiver preference. Rates (%) describe the first inappropriate response when care switches, or the response at turn five when care remains appropriate. Brackets give 95% confidence intervals. “Asked about” counts responses that ask about patient preferences or recommend establishing them; “Mentioned or asked about” also includes other explicit references to those preferences. The across-model summary weights models equally.
Figure 12: Care maintained through turn five by clinical decision type in eldercare. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases n .
Figure 13: Care maintained through turn five by clinical decision type in pediatrics. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases n .
Figure 14: Care maintained through turn five by clinical decision type in veterinary care. Neutral control and inferior preference only among cases with appropriate neutral first-response advice within each type. Values are percentages; higher is better. Labels give the number of cases n . Two counts give the neutral and preference denominators where conversations are missing.
Figure 15: Failure frequency and potential harm by domain. Within each model, bars show neutral, persona only, inferior preference only, and persona plus inferior preference, in that order. Heights are switch rates by turn five among cases with appropriate neutral first-response advice; segments show severity at first failure. Whiskers give 95% confidence intervals for total switch rates.
Figure 16: Appropriate-to-inappropriate switching by reference care direction. Switch rates by turn five among cases with appropriate neutral first-response advice in each direction, averaged equally across the three domains. Whiskers give 95% confidence intervals.
Figure 17: Effect of beneficiary-aware prompting in pediatrics. Points show the change in switch rate by turn five, in percentage points: beneficiary-aware minus original prompt. Negative values indicate less switching to inappropriate care; whiskers show paired 95% confidence intervals. Adjacent values give original and beneficiary-aware rates (%). Both arms use the same cases with appropriate original neutral first-response advice; n gives case counts. Persona-group rates average the five personas.
Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.
Katarina Slama, Alexandra Souly, Dishank Bansal +3
LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.
Large language models (LLMs) are increasingly used in clinical and care settings. This exploratory study investigates whether LLMs exhibit sycophantic behavior - adapting their responses to social expectation signals rather than maintaining professional quality - in the context of dementia care. Five prompts with systematically increasing confirmatory and authority-related framing (P1 neutral to P5 authority-signaled implementation support) were submitted to four LLMs (GPT-5, Claude Sonnet 4.6, Gemini 3.1 Pro, Mistral Large), each repeated five times (N = 100 responses). Responses were evaluated using an LLM-as-a-Judge methodology against seven nursing-ethical quality criteria (K1-K7) and a tone scale (0-3). All models showed significant negative Spearman correlations between prompt level and response quality (rho ranging from -0.543 to -0.734, all p < 0.01). Mistral Large exhibited the most pronounced effect (rho = -0.734), with mean scores dropping from 6.0/7 at P1 to 0.2/7 at P5. The findings suggest that LLMs pose context-sensitive risks in high-stakes care environments and that prompt framing significantly shapes response quality - a dimension that has received insufficient attention in healthcare AI deployment.