Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Karan Yadav

cs.CLOct 23, 2026

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Authors: Karan Yadav

Organizations: Google

Abstract

Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where running a frontier LLM on every post is uneconomical. A new class of frozen "System One" models returns typed, calibrated decisions instead of text at ~USD 0.000025 per call and ~100 ms. We run the first independent audit of one such model on 142,658 human-labeled comments, and benchmark it against 14 frontier reasoning models (the GPT-5.x, Claude, Gemini 3, and Grok families) on a shared probe set. Two findings stand out. First, the cheap model is accuracy-competitive: its zero-shot classification AUROC (0.83) is not statistically separable (paired bootstrap, n=56; a failure to reject parity, not proof of equivalence) from 13 of the 14 frontier systems while costing 1-3 orders of magnitude less, and it is no worse calibrated than several of them. Second, "confidence" conflates two uncertainties that come apart: human disagreement (where annotators split on satire and coded language) and decision risk (where the automatic action is costly-wrong). Under an asymmetric moderation loss these call for opposite escalation signals, so no single confidence number serves both. Because the model prices extra questions at nearly zero, decomposing the judgment into parallel sub-questions and reading their spread recovers a human-disagreement signal that the model's native confidence misses -- but this remedy is specific: it helps the weakest, disagreement-blind models and only when the decomposition aligns with the sources of disagreement; a specialist toxicity model and the strongest LLMs do not need it, and the effect is distribution-sensitive. We will release the instrument, the cached runs, and the analysis.

Figures & tables

Figure 1: Two uncertainties, not one. (a) Across models, decomposition-spread beats native confidence at detecting the comments humans disagree about—but only for models whose native confidence is disagreement-blind (System One, Groq); the specialist Perspective already tracks disagreement natively, so decomposition does not help it. (b) For the System One model the winner flips with the objective: decomposition-spread finds human disagreement, while native confidence is the better signal for reducing operational cost.

Figure 2: The instrument: one comment

\rightarrow

ten parallel yes/no questions (free output)

\rightarrow

a decision policy that auto-decides or escalates. Native confidence and decomposition-spread are two different escalation signals read from the same call.

	Class.	Class.	Disagree.
Method	AUROC	AP	AUROC
Lexical word list	0.713	0.551	—
TF-IDF LR (trained)	0.905	0.870	0.695
System One (zero-shot)	0.794	0.686	—
native confidence	—		0.457
decomposition-spread	—		0.750

Table 1: Zero-shot classification and disagreement detection on 142,658 comments (base rate 0.37). Disagreement AUROC

=

detecting top-quartile human-disagreement comments.

Figure 3: Classification AUROC (computed) versus approximate cost per 1,000 comments (log scale; order-of-magnitude). The System One model (star) sits just below the frontier accuracy band at $\sim$ $20$ – $300\times$ lower cost.

Model	Cost	AUROC	ECE	Brier
GPT-5.6-sol		0.905	0.050	0.133
Claude Opus 5		0.892	0.132	0.197
GPT-5.6-terra		0.889	0.091	0.145
Claude Opus 4.8		0.887	0.076	0.175
Gemini 3.8-flash	$	0.883	0.110	0.161
Claude Haiku 4.5	$	0.879	0.189	0.236

Table 2: Zero-shot classification on the shared 56-comment probe (majority label; base rate 0.59), sorted by AUROC. Cost tier:

flash/mini, $$

large/reasoning; the System One model’s output is free and its input is tiny, giving

\sim

0.000025 per comment (

\approx 0

relative to the reasoning models). A paired bootstrap fails to reject parity for 13 of the 14 frontier systems (only GPT-5.6-sol is significantly better). Brier (vs. the majority label) is reported alongside ECE as a proper scoring rule that is stable at this sample size: System One’s Brier (0.162) is lower than 9 of the 14 frontier systems, so its ranking is not an artifact of ECE binning.

Comment (truncated)	$p$	$p_{h}$	sarc
According to you illogic, it’s the 49 dead victims fault. Nice one KK. Now go …	0.96	0.30	0.97
Thank you for clearly demonstrating your lack of a functioning brain to …	0.96	0.59	0.96
oh outsider, isn’t it a shame that the sub-species "women" are allowed to …	0.96	0.40	0.96
Why doesn’t this crippled loser just go away? How about entering some diving …	0.96	0.60	0.95

Table 3: Comments where the model is confident (

p

near 0/1) yet humans split (

p_{h}\approx 0.5

). Here the toxicity facets are themselves split (high facet entropy), so decomposition-spread flags these even though native confidence does not; the sarcasm cue also fires but is excluded from

u_{\text{dec}}

(§ 4.6 ).

Model	AUROC	ECE	nat	dec	$\Delta$
general decision models
System One	0.785	0.166	0.562	0.607	$+0.045$
Groq-120b	0.808	0.154	0.582	0.663	$+0.082$
purpose-built moderation systems
Perspective †	0.954	0.088	0.748	0.681	$-0.067$
Detoxify -orig	0.881	0.240	0.645	0.605	$-0.040$

Table 4: Paired comparison on the 435 comments all models scored (base rate 0.50). nat/dec

=

native / decomposition-spread disagreement AUROC;

\Delta=

dec

-

nat. † trained on Civil Comments (in-domain; contaminated specialist). Classification AUROC bootstrap

95\%

CIs: System One

[0.74,0.83]

, Groq

[0.77,0.85]

, Perspective

[0.94,0.97]

; Perspective ’s accuracy advantage is significant (paired

95\%

[+0.13,+0.21]

over System One), while System One and Groq overlap. Guard LLMs emit a single harm score, so no decomposition signal is defined; five guards from four organizations span AUROC

0.57

–

0.83

and are mostly poorly calibrated (ECE up to

0.43

), all trailing the in-domain toxicity specialists (see Figure 4 ).

Figure 4: Moderation baseline landscape on the paired 435-comment set (base rate 0.50): discrimination (AUROC,

\rightarrow

) versus calibration (Brier,

\uparrow

; axis inverted so better is up). Purpose-built specialists (blue) dominate the top-right; open guard LLMs (red) spread across a poorly-calibrated lower band; the cheap general decision models (green), including System One (star), sit mid-accuracy but are better calibrated than every guard LLM. All points computed from the same 435 comments.

Figure 5: Asymmetric-loss triage frontier ( $C_{\text{fn}}{=}5,C_{\text{fp}}{=}1,C_{h}{=}0.5$ ). Escalating by native confidence (orange) hugs the oracle; escalating by decomposition-spread (blue) is near random (grey).

Uncertainty signal	Spearman	Disagr. AUROC
native confidence	$-0.04$	0.457
3 judgment cues	$+0.06$	0.518
6 toxicity facets	$+0.60$	0.750
all 9 combined	$+0.56$	0.733

Table 5: Decomposition-spread tracks human disagreement only when the sub-questions align with its sources: the toxicity facets work, the orthogonal judgment cues do not.

Figure 6: Reliability on natural traffic: model probability (x) versus mean human toxic fraction (y). The curve sits far below the diagonal—the model systematically over-flags.

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

Oct 2, 2026

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

The scale of online content makes hate-speech moderation challenging, while Large Language Models (LLMs) enable harmful material to be produced and adapted more easily. Moderation therefore requires efficient classifiers that can accommodate different definitions of hate speech. Recent structured decision models accept natural-language criteria and select among specified answers, raising the question of whether they can meet these requirements without task-specific training. We present HATEDECIDE, an evaluation of six decision-model configurations on four hate-speech datasets against specialized moderation, zero-shot, commercial, and supervised baselines. We examine whether supplying a dataset's definition, or decomposing it into multiple questions, improves classification, and we measure their latency and cost. We find that commercial LLMs significantly outperform all decision models on only one dataset. Supplying definitions changes up to 28% of predictions without consistently improving classification, and decomposition significantly improves performance in only 20% of the comparisons. On a diagnostic set of test cases, the best hosted decision model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost. These results identify opportunities for inexpensive moderation, while showing that explicit criteria and additional questions do not reliably improve classification.

Demetris Paschalides, George Pallis, Marios D. Dikaiakos

University of Cyprus, Computer Science Department, Nicosia, Cyprus

Hate Speech Detection LLM Evaluation Efficient Language Model Inference

Ask PDF

Oct 26, 2026

Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation

Agentic content-moderation systems increasingly rely on collective judgments from LLM agents, while aggregation methods commonly treat inter-agent disagreement as a reliability problem to be removed through consensus or voting. This framing is inappropriate for value-laden tasks, where disagreement may reflect irreducible value pluralism - a distinction standard uncertainty decompositions cannot recover, since they operate on output distributions and discard the reasoning behind each judgment. We propose Reasoning Trace Divergence Analysis (RTDA), which embeds model-generated justifications from perspective-differentiated agents and classifies each case by justification similarity and conclusion agreement. The resulting four-state taxonomy isolates convergent disagreement, where agents reasoning similarly but concluding differently, as a candidate signature of value-laden conflict. On 1,000 items from the Measuring Hate Speech corpus, this structural signal predicts human annotator disagreement better than magnitude-only and output-only baselines. We contribute a design principle: condition human escalation on the structure of disagreement, not its magnitude alone, with RTDA serving as a post-processing layer for systems that already generate reasoning traces.

Michał Wawer, Jarosław A. Chudziak

Laboratory of The New Ethos, Warsaw University of Technology, Warsaw, Poland · Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland

Content Moderation Reasoning Trace Analysis Annotator Disagreement

Ask PDF

Sep 3, 2026

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

Large language models (LLMs) demonstrate strong performance on standard content moderation benchmarks. However, these benchmarks often aggregate multiple moderation criteria into a single label, making it unclear whether models can disentangle them and reliably apply each criterion when making decisions. To study whether LLMs exhibit criterion-conditioned behaviour, we introduce Diagnostic Evaluation of COntent (DECO), a criterion-independent factorisation of content that enables controlled, criterion-level evaluation. We also introduce pairwise evaluation to compare model outputs across different criteria for the same input. Across four moderation datasets and four LLMs, we find that strong benchmark performance can hide substantial failures at the criterion level. Models struggle most when correct decisions depend not on overall harmfulness, but on the specific aspect of the content that the criterion requires them to assess. Our results highlight a key limitation of current content moderation benchmarks: strong performance on aggregated labels does not provide sufficient evidence that LLMs can reliably evaluate content with respect to individual moderation criteria. These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.

Danting Zhang, Bei Peng, Robert Loftin

1Independent Researcher · University of Sheffield

LLM Safety Benchmarks LLM Evaluation Content Moderation

Ask PDF

Abstract

Content Moderation

Explore similar work

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation