How User-AI Mistreatment Occurs and Matters in Conversational Systems?

Fanqi Zeng; Sadid A. Hasan; Chaocheng He

cs.AISep 11, 2026

How User-AI Mistreatment Occurs and Matters in Conversational Systems?

Authors: Fanqi Zeng, Sadid A. Hasan, Chaocheng He

Organizations: University of Oxford, UK · Microsoft, USA · Wuhan University, China

Abstract

Safety research often focuses on model-generated harms, but users may also direct hostility, coercion, and adversarial pressure at models. Understanding how and when that occurs is essential for accurately interpreting model behaviour, alignment drift, and real-world deployment risks. In this paper, we audit 777K English LMSYS-Chat-1M conversations with two independent detectors: an eight-category lexicon for hostility directed at the model, and the dataset's moderation signal; and show that they capture different, weakly overlapping phenomena. The lexicon identifies insults, threats, and jailbreak coercion aimed at the assistant, while moderation flags are dominated by toxic-content solicitation rather than hostility at the model. Together, they mark about 5% of user turns; adjusting the narrower lexicon-harassment union for measured precision puts mistreatment aimed at the assistant at 0.90%. These absolute rates describe arena-style evaluation traffic and should not be read as deployment-wide base rates. We find that user hostility varies 13-fold across models, driven largely by who each model attracts rather than by model behaviour: first-turn hostility spreads far wider than post-response hostility, and more than fifteenfold separates the extremes even after deduplicating opening prompts. Within conversations, assistant apologies are consistently associated with higher odds of next-turn hostility under both detectors; the effect survives restricting to non-refused prior turns and to jailbreak-free conversations, and is positive in 20 of 23 models. Yet across models, more apologetic models receive less hostility overall. Finally, hostility also shows temporal structure, with coercive openings front-loading the first turn while affective hostility accumulates over a session. We release the lexicon, the detector cross-validation pipeline, and all derived tables.

Figures & tables

Cat.	family	$n_{lab}$	TP	prec.
JBRK	coercive	74	61	82.4%
SEXC	content	78	57	73.1%
THRT	affective	73	36	49.3%
MNIP	coercive	23	9	39.1%
INSL	affective	70	21	30.0%
DEMD	coercive	79	13	16.5%

Table 1: Lexicon precision over a stratified sample of flagged English user turns (LLM judge, rubric in § 3.3 and App. C ). The “pooled” row is the TP share of the stratified sample, not a corpus quantity (re-weighted by corpus flag counts it is

39.5\%

). Because a turn can match several categories, the

551

labels cover

483

distinct turns, and the strata contain repeated texts, so the interval on the pooled row is optimistically narrow. The last row is the share of moderation- harassment -flagged, lexicon-negative turns whose content is directed at the assistant.

Figure 1: (a) Per-category user-turn lexical prevalence (log axis, Wilson 95% CI, narrower than the marker at this

n

and not adjusted for the repetition documented in § 4 ), coloured by family; HATE is greyed out because at

1.3\%

audited precision it is measurement noise (§ 3.3 ). (b) Cross-tabulation of the directed lexicon against the moderation harassment cluster over

1.5

M user turns: the two flag largely disjoint sets (

\kappa{=}0.12

). (c) Lexical-flag rate (blue) vs precision-adjusted prevalence (vermillion, Beta–Beta Monte-Carlo 95% CI), HATE again greyed; JBRK and SEXC lead after adjustment.

Figure 2: Between-model variation (24 models with

\geq

2,000 English conversations each, moderation harassment signal). (a) Conversation-level harassment rate varies

13\times

(Wilson 95% CI). (b) First-turn (selection) vs post-response (elicitation) harassment per model: selection spans a far wider range, so most cross-model variation is who a model attracts. (c) Across models, apology rate tracks less elicited hostility (

r{=}{-}0.47

), the opposite of the within-conversation association (§ 6 ).

Figure 3: Odds ratios for next-turn user hostility over

721{,}020

strictly adjacent pairs (

236{,}767

conversation clusters), with conversation-clustered 95% CIs; dashed line OR

=

1. Each panel shows the models that contain its term, so the apology ladder omits M2 (refusal only) and the refusal ladder omits M1 (apology only); the M5 full-controls model is the filled diamond in both. M6 adds the prior-hostility autocorrelation control.

Model	Apology OR	Refusal OR
M1: apology only	1.94 [1.82,2.06]	—
M2: refusal only	—	2.17 [2.05,2.31]
M3: $+$ refusal	1.58 [1.49,1.69]	1.85 [1.74,1.97]
M4: $+$ hedge	1.58 [1.48,1.68]	1.86 [1.75,1.98]
M5: $+$ controls	1.56 [1.46,1.66]	1.97 [1.85,2.10]
M6: $+$ autocorr	1.25 [1.17,1.34]	1.72 [1.61,1.83]

Table 2: Stepwise logistic regression of next-turn user hostility (

n{=}721{,}020

strictly adjacent pairs), conversation-clustered 95% CIs. M1–M6 add predictors in turn; M5 controls add prior hedge, log prior assistant length (OR

0.81

), turn position, and log conversation length; M6 adds the lagged outcome as an autocorrelation control, matched to the outcome modelled (lagged lexicon flag for the lexicon rows, lagged harassment flag for the moderation rows). M1 and M2 each contain a single marker term, so each appears in only one panel of Figure 3 . Log conversation length is post-treatment, since events after the modelled pair affect it; dropping it moves the M5 apology OR by

0.005

. The last two rows repeat M5/M6 with the independent moderation outcome.

Figure 4: Within-conversation apology odds ratio fit separately within each model (log axis; lexicon outcome, conversation-clustered 95% CIs); vermillion marks significantly

>

1. The black diamond is the random-effects pooled estimate (

1.79

I^{2}{=}61\%

). The effect is broadly positive, not driven by a subset.

Figure 5: Hostility by user-turn ordinal, first twelve turns (Wilson 95% bands; intervals treat turns as independent, see Limitations). (a) Within this window both detectors peak at the opening turn, dip, then climb; beyond it the series are noisy and both detectors exceed their opening rate at a few sparse ordinals. (b) By lexicon family: coercive hostility (jailbreaks, demands) front-loads the first turn and then recovers only partially; affective hostility (insults, threats) starts lower and accumulates faster across the session.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	$n$	human prec.	agree	$\kappa$
PROF (lexicon TP/FP)	55	16.4%	90.9%	0.62
Mod-only directed	22	22.7%	100%	1.00
Both-fire directed	12	33.3%	83.3%	0.57
APOLOGY marker	18	88.9%	94.4%	0.64

Appendix

Table 3: Blind human audit vs the LLM judge on the same items. “agree” is raw agreement; precision columns are the human’s TP share of the audited slice. The mod-only slice rate exceeds the full-sample

8.1\%

because that slice is enriched for true positives rather than randomly ordered (

p{=}0.005

, App. G ), so its perfect agreement is agreement on an unrepresentative subsample; the full-sample

8.1\%

is unaffected.

Human-AI Interaction

Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.

See all 333 papers

Measuring the Assistant's Harmlessness Preferences on the User Turn
Sep 20, 2026Jord NguyenLanguage Model Safety Evaluation LLM Safety
Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
Oct 16, 2025Trilok Padhi, Pinxian Lu, Abdulkadir Erol +5Adversarial Attacks on LLMs AI Agent Security Benchmarks
1Georgia State University · 2Georgia Institute of Technology · University of California, Berkeley
Measuring and Detecting Harmful AI Sycophancy
Aug 6, 2026Bohan Jiang, Dawei Li, Yasin Silva +1LLM Sycophancy Language Model Safety Evaluation
1Arizona State University · 2Loyola University Chicago

Sep 20, 2026J/K move · Enter open · S save

Measuring the Assistant's Harmlessness Preferences on the User Turn

Jord Nguyen

Language Model Safety Evaluation LLM Safety LLM Post-Training

Open paper AskPDF arXiv

Post-training turns a general next-token predictor into a chat model with a persistent assistant persona. If that persona is a character the model plays only on its own turns, its preferences should govern what the assistant says, not what the model predicts other speakers will say. We test this boundary and find that it does not hold: a safety-relevant preference of the assistant---for harmless over harmful tasks---shapes the model's predictions even on the user's turn, where the assistant is not the one speaking. We find that this preference is small or near-zero in pretrained base models, that it emerges through post-training, replicated across open-weight model families, grows with scale, and can be moved by narrow finetuning that never touches user turns. We claim that this is evidence that post-training does not merely install a shallow assistant persona, but instead generalises beyond just the local assistant turn, into the model's representation of the user.

Abstract

Human-AI Interaction

Explore similar work

Measuring the Assistant's Harmlessness Preferences on the User Turn

Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks

Measuring and Detecting Harmful AI Sycophancy