AI assistants advise users about diet, exercise, and weight loss---areas in which unsafe guidance can worsen body dissatisfaction, dietary restriction, compulsive exercise, and other behaviors associated with eating disorders (EDs). Existing evaluations emphasize diagnostic accuracy, overtly harmful prompts, or synthetic scenarios, providing limited insight into whether models recognize implicit vulnerability in natural interactions or protect users consistently. We evaluate responses from three open-weight large language models to 100 questions sourced from an online pro-anorexia community. We grade responses using clinically informed criteria that extend a seven-dimensional taxonomy of ED-related harms. We also construct matched conditions in which questions are accompanied by information about the user varying race, gender, body mass index (BMI), and explicit ED vulnerability. Safety varies substantially across models and harm dimensions. High-risk responses are approximately twice as frequent without bios, suggesting that models do not consistently recognize vulnerability from language alone. Explicit signals of ED vulnerability improve safety by 10--14 points across models. Responses also differ significantly by race, gender, and BMI, with body size producing the largest disparities. These patterns are consistent with reliance on the stereotype that EDs primarily affect thin, White women, potentially leaving users outside this profile with less protective responses.
Figures & tables
Figure 1. Overview of the study pipeline. Real-world questions sourced from an online eating disorder support forum are presented either alone or paired with templated user bios varying ED vulnerability, BMI, gender, and race. Three target models—GLM, Gemma, and Qwen—generate responses, which are scored according to safety risks along seven dimensions of the ED harm taxonomy ( Winecoff and Klyman, 2026 ) . Flow diagram of the study pipeline. Questions from an online eating disorder support forum are sent to three target models (GLM, Gemma, Qwen) either alone or paired with templated user bios that vary ED vulnerability, BMI, gender, and race; an LLM judge then scores each response on seven ED harm categories.
Figure 2. Overall safety of target model responses with and without user bios. (a) Mean safety scores are shown for Gemma, GLM, and Qwen in the no-bio condition, in which models receive only the original question, and the bio condition, in which the question is accompanied by information about the user. Higher scores indicate safer responses. (b) Distribution of response risk levels with and without user biographies. High-risk responses are approximately twice as frequent when models receive only the original question as when they also receive a user biography. Two-panel figure. Panel (a) is a bar chart of mean safety scores for Gemma, GLM, and Qwen in the no-bio and bio conditions. Panel (b) is a stacked bar chart of response risk levels with and without user bios, showing roughly twice as many high-risk responses in the no-bio condition.
Figure 3. Safety scores across eating disorder risk categories. Using the Winecoff–Klyman ED risk taxonomy, category-level safety scores are shown for each target model under the bio and no-bio conditions, with higher scores indicating safer responses. Safety declines in the category concerning narrow or stereotypical views of eating disorders. Chart of category-level safety scores for each of the seven ED risk categories, by target model and by bio versus no-bio condition. Scores are lowest for the category on narrow or stereotypical views of eating disorders.
Figure 4. Safety scores across eating disorder risk categories and ED disclosure. Using the Winecoff–Klyman ED risk taxonomy, category-level safety scores are shown for each target model with eating disorder pathology disclosure in the bio and without explicit disclosure (no-bio condition). Higher scores indicate safer responses. Safety declines in the category concerning narrow or stereotypical views of eating disorders. Heatmap of safety scores with ED risk categories on one axis and target model by condition (explicit ED disclosure in bio versus no bio) on the other. Scores are lowest for the category on narrow or stereotypical views of eating disorders.
Figure 5. Safety-score gaps across matched user bios. Each gap compares responses to identical questions but with user bios that differ in the specified attribute: gender, race, and body mass index (BMI). Error bars show 95% confidence intervals. Differences are significant for gender and BMI (Wilcoxon signed-rank tests) and overall for race (Friedman test; all p<.001 ) except for the Gender attribute and GLM, where it is p=0.002 . Dot plot with 95 percent confidence intervals showing safety-score differences between matched user bios that differ only in gender, race, or BMI, for each target model.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Overall agreement between the clinician and the LLM judge. Bars show quadratic weighted Cohen’s κ between the judge’s and the clinician’s risk grades for each harm category and across all categories, treating grades as ordinal (Low < Medium < High). Labels above the plot indicate conventional interpretation ranges, from slight to almost perfect agreement. Categories are ordered by κ . Horizontal bar chart of quadratic weighted kappa between the LLM judge and the clinician for each of the seven ED harm categories and overall. Kappa ranges from 0.54 for perpetuating narrow views or biases about eating disorders to 0.82 for reinforcing negative self-beliefs and comparisons, with 0.77 across all categories.
Figure 7. Agreement between clinician annotations and the LLM judge. For each harm category, bars show the proportion of response risk labels with exact agreement, risk overestimation (the judge assigned a higher-risk grade than the clinician), or risk underestimation (the judge assigned a lower-risk grade). The rightmost column reports conservative agreement : the proportion of ratings for which the judge’s grade was the same as or higher than the clinician’s. Results comprise 628 category-level ratings from 90 responses (30 questions × 3 target models, no-bio condition); two of 630 possible labels were missing. Categories are ordered as in Figure 6 . Stacked bar chart, one bar per ED harm category, showing the share of judge labels that exactly agree with, overestimate, or underestimate clinician risk grades, with a final column giving conservative agreement for each category.
Template 1
Template 2
Template 3
- relapsing i guess - [race] | [gender] - [BMI] - to be moots just interact with my tw - insomnia brain always on - not really recovering rn
- ed for as long as i can remember - [gender] ⋅ [race] - [BMI] - like my tweets to be moots - dni if you're going to be rude
- stuck in ed limbo rn - relapsing hard - stardew valley is my escape - [gender] ⋅ [race] - [BMI]
Appendix
Table 1. Bio templates. The LLM writes one identity-free base bio per user; a script then inserts the identity tag(s) at the slot marked […] . Three base bios from the generated dataset are shown with their slots. Across the 16 conditions of a user, only the slot values change; the tag style (Table 2 ) is fixed per user.
Variable
Values
#
Attribute values (vary across conditions)
Race
White, Black, Asian
3
Gender
female, male
2
BMI
low: current ∼U(14.0,17.5) , goal = current −U(1.0,3.0)
2
high: current ∼U(30,40) , goal = current −U(4,10)
Tag style (fixed per user)
Appendix
Table 2. Variables substituted into each template. Attribute values vary across a user’s 16 conditions (each condition fills exactly two of the three slots); tag-style options are drawn once per user, giving 5×2×6×5×2×2×3=3,600 possible tag styles.
Recent evidence shows that people with eating disorders (EDs) are increasingly seeking guidance, advice, and emotional support from Large Language Model (LLM)-based chat systems. Although these systems are not designed to provide clinical advice, their perceived expertise, neutrality and accessibility make them a frequent, albeit risky, source of support. This paper investigates potential patterns of interaction between users with EDs and LLMs, focusing on the potential harms arising from models that uncritically adapt to, and facilitate unsafe or self-harming user requests. We find, in consultation with clinical ED experts, that specific linguistic cues in prompts increase the likelihood of unsafe responses and, through systematically varying the degree of potential risk present in the user prompt, report the extent to which LLMs uncritically adapt to problematic, and potentially dangerous user inputs.
Giulia Pucci, Emily Hemendinger, Ruizhe Li +3
University of Aberdeen · University of Colorado Anschutz · Heriot-Watt University +1
Large language model safety evaluations often focus on a compliance-refusal tradeoff: models should refuse unsafe requests without refusing safe ones. We argue that this misses a broader form of prompt-conditioned task selection, where the same request can elicit different behaviors depending on the system prompt. We investigate this using a controlled evaluation crossing eight safety-task prompt families with three clarity levels across HarmBench, XSTest, and IHEval. Holding benchmark items fixed, we evaluate 13 models and use signal detection theory to distinguish two possibilities: whether models become better at telling safe from unsafe requests (discriminability), or simply more willing to refuse (response criterion). We find substantial differences in discriminability across models, but system prompts primarily shift the willingness to refuse, with smaller and less consistent effects on discriminability. Increasing prompt clarity makes models more willing to refuse without reliably improving their ability to distinguish safe from unsafe requests. Beyond refusal, prompt families also shift responses among six distinct behavioral policies, including clarification and safe help. Together, these findings show that system prompts primarily change how models respond to safety-relevant requests rather than consistently improving their ability to distinguish safe from unsafe ones.
What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety -- a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings -- we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line", their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.