Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Karan Yadav

cs.CLOct 23, 2026

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Authors: Karan Yadav

Organizations: Google

Abstract

Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where running a frontier LLM on every post is uneconomical. A new class of frozen "System One" models returns typed, calibrated decisions instead of text at ~USD 0.000025 per call and ~100 ms. We run the first independent audit of one such model on 142,658 human-labeled comments, and benchmark it against 14 frontier reasoning models (the GPT-5.x, Claude, Gemini 3, and Grok families) on a shared probe set. Two findings stand out. First, the cheap model is accuracy-competitive: its zero-shot classification AUROC (0.83) is not statistically separable (paired bootstrap, n=56; a failure to reject parity, not proof of equivalence) from 13 of the 14 frontier systems while costing 1-3 orders of magnitude less, and it is no worse calibrated than several of them. Second, "confidence" conflates two uncertainties that come apart: human disagreement (where annotators split on satire and coded language) and decision risk (where the automatic action is costly-wrong). Under an asymmetric moderation loss these call for opposite escalation signals, so no single confidence number serves both. Because the model prices extra questions at nearly zero, decomposing the judgment into parallel sub-questions and reading their spread recovers a human-disagreement signal that the model's native confidence misses -- but this remedy is specific: it helps the weakest, disagreement-blind models and only when the decomposition aligns with the sources of disagreement; a specialist toxicity model and the strongest LLMs do not need it, and the effect is distribution-sensitive. We will release the instrument, the cached runs, and the analysis.

Figures & tables

Figure 1: Two uncertainties, not one. (a) Across models, decomposition-spread beats native confidence at detecting the comments humans disagree about—but only for models whose native confidence is disagreement-blind (System One, Groq); the specialist Perspective already tracks disagreement natively, so decomposition does not help it. (b) For the System One model the winner flips with the objective: decomposition-spread finds human disagreement, while native confidence is the better signal for reducing operational cost.

Figure 2: The instrument: one comment

\rightarrow

ten parallel yes/no questions (free output)

\rightarrow

a decision policy that auto-decides or escalates. Native confidence and decomposition-spread are two different escalation signals read from the same call.

	Class.	Class.	Disagree.
Method	AUROC	AP	AUROC
Lexical word list	0.713	0.551	—
TF-IDF LR (trained)	0.905	0.870	0.695
System One (zero-shot)	0.794	0.686	—
native confidence	—		0.457
decomposition-spread	—		0.750

Table 1: Zero-shot classification and disagreement detection on 142,658 comments (base rate 0.37). Disagreement AUROC

=

detecting top-quartile human-disagreement comments.

Figure 3: Classification AUROC (computed) versus approximate cost per 1,000 comments (log scale; order-of-magnitude). The System One model (star) sits just below the frontier accuracy band at $\sim$ $20$ – $300\times$ lower cost.

Model	Cost	AUROC	ECE	Brier
GPT-5.6-sol		0.905	0.050	0.133
Claude Opus 5		0.892	0.132	0.197
GPT-5.6-terra		0.889	0.091	0.145
Claude Opus 4.8		0.887	0.076	0.175
Gemini 3.8-flash	$	0.883	0.110	0.161
Claude Haiku 4.5	$	0.879	0.189	0.236

Table 2: Zero-shot classification on the shared 56-comment probe (majority label; base rate 0.59), sorted by AUROC. Cost tier:

flash/mini, $$

large/reasoning; the System One model’s output is free and its input is tiny, giving

\sim

0.000025 per comment (

\approx 0

relative to the reasoning models). A paired bootstrap fails to reject parity for 13 of the 14 frontier systems (only GPT-5.6-sol is significantly better). Brier (vs. the majority label) is reported alongside ECE as a proper scoring rule that is stable at this sample size: System One’s Brier (0.162) is lower than 9 of the 14 frontier systems, so its ranking is not an artifact of ECE binning.

Comment (truncated)	$p$	$p_{h}$	sarc
According to you illogic, it’s the 49 dead victims fault. Nice one KK. Now go …	0.96	0.30	0.97
Thank you for clearly demonstrating your lack of a functioning brain to …	0.96	0.59	0.96
oh outsider, isn’t it a shame that the sub-species "women" are allowed to …	0.96	0.40	0.96
Why doesn’t this crippled loser just go away? How about entering some diving …	0.96	0.60	0.95

Table 3: Comments where the model is confident (

p

near 0/1) yet humans split (

p_{h}\approx 0.5

). Here the toxicity facets are themselves split (high facet entropy), so decomposition-spread flags these even though native confidence does not; the sarcasm cue also fires but is excluded from

u_{\text{dec}}

(§ 4.6 ).

Model	AUROC	ECE	nat	dec	$\Delta$
general decision models
System One	0.785	0.166	0.562	0.607	$+0.045$
Groq-120b	0.808	0.154	0.582	0.663	$+0.082$
purpose-built moderation systems
Perspective †	0.954	0.088	0.748	0.681	$-0.067$
Detoxify -orig	0.881	0.240	0.645	0.605	$-0.040$

Table 4: Paired comparison on the 435 comments all models scored (base rate 0.50). nat/dec

=

native / decomposition-spread disagreement AUROC;

\Delta=

dec

-

nat. † trained on Civil Comments (in-domain; contaminated specialist). Classification AUROC bootstrap

95\%

CIs: System One

[0.74,0.83]

, Groq

[0.77,0.85]

, Perspective

[0.94,0.97]

; Perspective ’s accuracy advantage is significant (paired

95\%

[+0.13,+0.21]

over System One), while System One and Groq overlap. Guard LLMs emit a single harm score, so no decomposition signal is defined; five guards from four organizations span AUROC

0.57

–

0.83

and are mostly poorly calibrated (ECE up to

0.43

), all trailing the in-domain toxicity specialists (see Figure 4 ).

Figure 4: Moderation baseline landscape on the paired 435-comment set (base rate 0.50): discrimination (AUROC,

\rightarrow

) versus calibration (Brier,

\uparrow

; axis inverted so better is up). Purpose-built specialists (blue) dominate the top-right; open guard LLMs (red) spread across a poorly-calibrated lower band; the cheap general decision models (green), including System One (star), sit mid-accuracy but are better calibrated than every guard LLM. All points computed from the same 435 comments.

Figure 5: Asymmetric-loss triage frontier ( $C_{\text{fn}}{=}5,C_{\text{fp}}{=}1,C_{h}{=}0.5$ ). Escalating by native confidence (orange) hugs the oracle; escalating by decomposition-spread (blue) is near random (grey).

Uncertainty signal	Spearman	Disagr. AUROC
native confidence	$-0.04$	0.457
3 judgment cues	$+0.06$	0.518
6 toxicity facets	$+0.60$	0.750
all 9 combined	$+0.56$	0.733

Table 5: Decomposition-spread tracks human disagreement only when the sub-questions align with its sources: the toxicity facets work, the orthogonal judgment cues do not.

Figure 6: Reliability on natural traffic: model probability (x) versus mean human toxic fraction (y). The curve sits far below the diagonal—the model systematically over-flags.

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation
Oct 2, 2026Demetris Paschalides, George Pallis, Marios D. DikaiakosHate Speech Detection LLM Evaluation
University of Cyprus, Computer Science Department, Nicosia, Cyprus
Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation
Oct 26, 2026Michał Wawer, Jarosław A. ChudziakContent Moderation Reasoning Trace Analysis
Laboratory of The New Ethos, Warsaw University of Technology, Warsaw, Poland · Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield

Oct 2, 2026J/K move · Enter open · S save

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Demetris Paschalides, George Pallis, Marios D. Dikaiakos

University of Cyprus, Computer Science Department, Nicosia, Cyprus

Hate Speech Detection LLM Evaluation Efficient Language Model Inference

Open paper AskPDF arXiv

The scale of online content makes hate-speech moderation challenging, while Large Language Models (LLMs) enable harmful material to be produced and adapted more easily. Moderation therefore requires efficient classifiers that can accommodate different definitions of hate speech. Recent structured decision models accept natural-language criteria and select among specified answers, raising the question of whether they can meet these requirements without task-specific training. We present HATEDECIDE, an evaluation of six decision-model configurations on four hate-speech datasets against specialized moderation, zero-shot, commercial, and supervised baselines. We examine whether supplying a dataset's definition, or decomposing it into multiple questions, improves classification, and we measure their latency and cost. We find that commercial LLMs significantly outperform all decision models on only one dataset. Supplying definitions changes up to 28% of predictions without consistently improving classification, and decomposition significantly improves performance in only 20% of the comparisons. On a diagnostic set of test cases, the best hosted decision model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost. These results identify opportunities for inexpensive moderation, while showing that explicit criteria and additional questions do not reliably improve classification.

Abstract

Content Moderation

Explore similar work

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation