Web communities rely on moderators to catch harmful messages such as insults, hate speech, and scams, but no one can read every message. Existing tools give each message a risk score, which tells moderators where to look but not what to read or how to check it. Yet meaning depends on context: the same reply, "I agree.", can support or oppose removing someone from a group. Moderators therefore need source-grounded evidence, not just scores: a few key messages kept with their original wording and replies. We present CST, a lightweight triage pipeline that first decides where to look, then what to show. It scores risk by pairing character patterns, which capture exact wording, with sentence meaning from a fixed multilingual model, so it handles several languages and runs on an ordinary CPU. It then groups related messages and picks from each group in turn, so a small reading budget covers different parts of a discussion, and links every pick back to the original thread. Across six tasks from five datasets in several languages, CST ranks harmful messages more accurately than a fully fine-tuned multilingual model and, under the same budget, finds more of them than reading by score alone. In a user study, reviewers needed at least 27% less time with CST's grouped evidence while keeping decision accuracy above 94%. CST turns risk scores into evidence that moderators can read quickly and check in context. Code and dataset reconstruction resources will be released at https://github.com/Kirito14IT/CST-MIL.
Figures & tables
Figure 1: Reply context for moderation. “I agree.” supports removal in (a) and opposes it in (b).
Figure 2: Two controlled comparisons. (a) Selection can change message IDs under the same discussions, risk scores, and budget. (b) A separate comparison fixes three messages and their text, scores, parents, and source links.
Figure 3: cst framework for risk screening and evidence organization. Shared character and semantic features support message risk scoring; message scores, semantic summaries, and reply relations inform discussion ranking. The organization branch groups messages and extracts phrases with source spans, enabling evidence inspection within the original reply thread.
Split sizes
Observed test labels
Source
Train
Dev.
Test
Task
Count
Pos. (%)
SCCD
21,762
7,849
8,028
A
8,028
24.4
CONDA
14,547
4,315
4,746
A
4,746
23.8
ToxiCN
7,720
1,866
2,409
A/H
2,212
48.6/41.2
ScamSpot
1,847
623
582
S
582
32.0
HateBR
4,129
1,154
1,717
A
1,717
59.6
Table 1: Message counts and observed test labels. A/H/S denote attack/offense, hate, and spam/scam. Positive rates use observed labels; ToxiCN counts apply to each task.
Selector
Evidence precision (%)
Global recall (%)
Original order
47.778
1.463
Risk order
91.667
2.806
MMR
90.000
2.755
Group round-robin
96.667
2.959
Table 2: Primary screening at L=20,B=3 , selecting at most 60 messages from the same 20 discussions. Recall covers all 1,960 dangerous test messages.
Figure 4: Evidence-only budget sensitivity. Cells show recall (%) across message budgets B , using 1,960 dangerous messages in the full test set (a) and 413 in the isolated set (b) as denominators. Every discussion receives a budget; this differs from the top- L screening endpoint.
Method
SCCD
CONDA
HateBR
ToxiCN-A
ToxiCN-H
ScamSpot
LDA
0.307
0.363
0.642
0.621
0.623
0.829
BTM
0.356
0.300
0.842
0.589
0.534
0.731
GSDMM
0.402
0.385
0.643
0.597
0.567
0.630
NMF
0.329
0.609
0.729
0.541
0.487
0.606
BERTopic
0.354
0.413
0.698
0.592
0.542
0.699
MiniLM–LR
0.597
0.694
0.951
0.758
0.735
0.954
Table 3: Message test AP. Topic representations are evaluated with supervised LR adapters.
Cohort
n
Clusters
BTM
MiniLM–LR
cst
Full
136
47
0.890
0.957
0.976
Isolated
40
40
0.819
0.938
0.963
Largest removed
47
46
0.769
0.912
0.948
Table 4: Discussion AP. n : discussions; clusters: duplicate-connected units. Development AP selects BTM as the topic baseline for comparison.
Figure 5: Paired score differences on a 0–2 rubric: complete minus control (a,b) and grouped minus flat with fixed evidence (c, n=36 ).
Figure 6: CPU service cost on SCCD: four threads and three passes per method.
Figure 7: Cross-source attack AP and joint-minus-MiniLM gains.
Removed
Message AP ( Δ )
Discussion AP ( Δ )
Character features
0.597 -2pt
0.957 -2pt
Message semantics
0.524 -2pt
0.953 -2pt
Discussion semantics
0.712 -2pt
0.962 -2pt
Reply statistics
0.712 -2pt
0.959 -2pt
Risk statistics
0.712 -2pt
0.943 -2pt
Source balance
0.686 -2pt
0.965 -2pt
Table 5: SCCD component removals. Each cell gives AP after removal above the complete-minus-removal difference. Complete-model message/discussion AP is 0.712/0.976.
Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community may be perfectly acceptable in another. Can AI models help moderate such pluralistic communities? We formalize the task as a multiple-choice problem, mirroring how human moderators operate in the real world: given a comment and its surrounding context, identify which specific rule, if any, is violated. We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages. Using this benchmark, we show that state-of-the-art vision-language models struggle significantly: even GPT-5.2 with high reasoning performs only slightly better than a trivial baseline. We also find that bigger models and increased context provide marginal gains, and universal rules like civility and self-promotion are easier to detect. Our results show that moderation of pluralistic communities on social media is a fundamental challenge for language models. Our code and benchmark are publicly available.
Zoher Kachwala, Bao Tran Truong, Rasika Muralidharan +3
Observatory on Social Media, Indiana University, USA · Center Synergy of Systems, TUD Dresden University of Technology, Germany
What the Web community believes about toxic conversations, such as that toxicity rises in longer and deeper threads, was measured mostly with Google's Perspective API, which is shutting down; large language models (LLMs) will make the next measurements. We find that LLM labelers mistake second-person address for toxicity, and that this one bias manufactures laws that human labels do not show. On 1.25 million news comments with crowd labels, Claude Haiku flags 12% of the non-toxic comments that contain no "you" and 51% of those with four or more; a single crowd rater flags 6% and 10%. Deeper replies address someone more often, so under Haiku toxicity rises with reply depth (+0.93 percentage points per level), while under the crowd's labels it slightly declines (-0.12). Rewriting "you" in the third person removes two thirds of the false rise; flagging only as many comments as the crowd does not remove it, and showing Haiku the parent comment enlarges it. All seven LLMs we test over-flag "you", and at the crowd's flag rate, without the parent, any false rise of the open ones vanishes at a fixed number of second-person words; Haiku also over-flags "you" on Reddit, Gab and Wikipedia talk pages. Agreement with human labels does not reveal the bias: we prove that a fixed Cohen's kappa leaves a law nearly unconstrained. We therefore certify the law instead of the labeler. Threadsafe uses the labeler's scores to choose which comments human raters label and to predict the others, and corrects the predictions with the human labels; linear laws are estimated without bias whatever the labeler does. In simulated audits its intervals for the laws keep nominal coverage from 250 labels on, where uniform-sample methods such as PPI++ fall short, and at 1,000 to 8,000 labels are 0.75 times as wide as those of design-based supervised learning (DSL); a thousand human labels rule out Haiku's rising law.
Liang Yue, Eric A. Vance
University of Colorado Boulder Boulder, Colorado, USA
Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: https://cosmosyi.github.io/ReasoningShield/, Code: https://github.com/CosmosYi/ReasoningShield, Models: https://huggingface.co/ReasoningShield/models, and Dataset: https://huggingface.co/datasets/ReasoningShield/ReasoningShield-Dataset.