Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation
Authors: Michał Wawer, Jarosław A. Chudziak
Organizations: Laboratory of The New Ethos, Warsaw University of Technology, Warsaw, Poland · Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland
Agentic content-moderation systems increasingly rely on collective judgments from LLM agents, while aggregation methods commonly treat inter-agent disagreement as a reliability problem to be removed through consensus or voting. This framing is inappropriate for value-laden tasks, where disagreement may reflect irreducible value pluralism - a distinction standard uncertainty decompositions cannot recover, since they operate on output distributions and discard the reasoning behind each judgment. We propose Reasoning Trace Divergence Analysis (RTDA), which embeds model-generated justifications from perspective-differentiated agents and classifies each case by justification similarity and conclusion agreement. The resulting four-state taxonomy isolates convergent disagreement, where agents reasoning similarly but concluding differently, as a candidate signature of value-laden conflict. On 1,000 items from the Measuring Hate Speech corpus, this structural signal predicts human annotator disagreement better than magnitude-only and output-only baselines. We contribute a design principle: condition human escalation on the structure of disagreement, not its magnitude alone, with RTDA serving as a post-processing layer for systems that already generate reasoning traces.
Figures & tables
Figure 1. Convergent disagreement: when agents reason similarly but conclude differently. We study this pattern as a candidate signal for human review.
Method
r
ρ
AUROC
F1
B1 Divergence-only
0.38
0.41
0.68
0.51
B2 Vote entropy
0.34
0.36
0.66
0.50
B3 Verbalized conf.
0.26
0.29
0.61
0.47
B4 Self-consistency
0.31
0.33
0.64
0.49
B5 Semantic uncertainty
0.35
0.37
0.67
0.51
B6 Supervised †
0.43
0.45
0.73
0.56
Table 1. Predictive validity and routing performance. r , ρ : Pearson and Spearman correlation with human annotator disagreement, † denotes supervised baseline.
State
n
Mean dˉh
r [95% CI]
DA(c)
197
0.34
0.19 [0.05,0.32]
CA(c)
41
0.59
0.27 [−0.04,0.53]
DD(c)
635
0.73
0.38 [0.31,0.44]
CD(c)
127
0.81
0.52 [0.38,0.64]
Table 2. Per-state distribution and Pearson correlation with human annotator disagreement on Measuring Hate Speech ( n=1,000 ); brackets give Fisher- z 95% confidence intervals.
Figure 2. F1 over the similarity threshold τs at fixed τa=0.80 . The structural advantage persists across an interval rather than at a single operating point.
Base model
Φ
B1 div.
Δ F1
r on CD
GPT-5.4-Nano
0.59
0.51
+0.08
0.52
Claude 4.5 Son.
0.57
0.50
+0.07
0.49
MiniMax M3.
0.58
0.49
+0.09
0.51
Table 3. Base-model ablation: routing F1 on Measuring Hate Speech, fixed agent prompts.
Configuration
F1 ( Φ )
Δ F1 vs. B1
Full 5 perspectives
0.59
+0.08
k=3 (drop legal, community)
0.55
+0.06
k=7 (add procedural, victim)
0.60
+0.08
Random perspective swap
0.56
+0.05
Drop harm-focused
0.53
+0.04
Drop free-expression
0.54
+0.05
Table 4. Agent-composition ablation (GPT-5.4-Nano). Δ F1 reported against B1 divergence-only at matching k .
Laboratory of The New Ethos Warsaw University of Technology, Warsaw, Poland · Institute of Computer Science Faculty of Electronics and Information Technology Warsaw University of Technology, Warsaw, Poland