CST: Lightweight Risk Triage with Source-Grounded Evidence for Web Discussions
Organizations: Fudan University · University of Shanghai for Science and Technology · Alibaba Group
Abstract
Web communities rely on moderators to catch harmful messages such as insults, hate speech, and scams, but no one can read every message. Existing tools give each message a risk score, which tells moderators where to look but not what to read or how to check it. Yet meaning depends on context: the same reply, "I agree.", can support or oppose removing someone from a group. Moderators therefore need source-grounded evidence, not just scores: a few key messages kept with their original wording and replies. We present CST, a lightweight triage pipeline that first decides where to look, then what to show. It scores risk by pairing character patterns, which capture exact wording, with sentence meaning from a fixed multilingual model, so it handles several languages and runs on an ordinary CPU. It then groups related messages and picks from each group in turn, so a small reading budget covers different parts of a discussion, and links every pick back to the original thread. Across six tasks from five datasets in several languages, CST ranks harmful messages more accurately than a fully fine-tuned multilingual model and, under the same budget, finds more of them than reading by score alone. In a user study, reviewers needed at least 27% less time with CST's grouped evidence while keeping decision accuracy above 94%. CST turns risk scores into evidence that moderators can read quickly and check in context. Code and dataset reconstruction resources will be released at https://github.com/Kirito14IT/CST-MIL.
Figures & tables
| Split sizes | Observed test labels | |||||
| Source | Train | Dev. | Test | Task | Count | Pos. (%) |
| SCCD | 21,762 | 7,849 | 8,028 | A | 8,028 | 24.4 |
| CONDA | 14,547 | 4,315 | 4,746 | A | 4,746 | 23.8 |
| ToxiCN | 7,720 | 1,866 | 2,409 | A/H | 2,212 | 48.6/41.2 |
| ScamSpot | 1,847 | 623 | 582 | S | 582 | 32.0 |
| HateBR | 4,129 | 1,154 | 1,717 | A | 1,717 | 59.6 |
| Selector | Evidence precision (%) | Global recall (%) |
|---|---|---|
| Original order | 47.778 | 1.463 |
| Risk order | 91.667 | 2.806 |
| MMR | 90.000 | 2.755 |
| Group round-robin | 96.667 | 2.959 |
| Method | SCCD | CONDA | HateBR | ToxiCN-A | ToxiCN-H | ScamSpot |
|---|---|---|---|---|---|---|
| LDA | 0.307 | 0.363 | 0.642 | 0.621 | 0.623 | 0.829 |
| BTM | 0.356 | 0.300 | 0.842 | 0.589 | 0.534 | 0.731 |
| GSDMM | 0.402 | 0.385 | 0.643 | 0.597 | 0.567 | 0.630 |
| NMF | 0.329 | 0.609 | 0.729 | 0.541 | 0.487 | 0.606 |
| BERTopic | 0.354 | 0.413 | 0.698 | 0.592 | 0.542 | 0.699 |
| MiniLM–LR | 0.597 | 0.694 | 0.951 | 0.758 | 0.735 | 0.954 |
| Cohort | Clusters | BTM | MiniLM–LR | cst | |
|---|---|---|---|---|---|
| Full | 136 | 47 | 0.890 | 0.957 | 0.976 |
| Isolated | 40 | 40 | 0.819 | 0.938 | 0.963 |
| Largest removed | 47 | 46 | 0.769 | 0.912 | 0.948 |
| Removed | Message AP ( ) | Discussion AP ( ) |
|---|---|---|
| Character features | 0.597 -2pt | 0.957 -2pt |
| Message semantics | 0.524 -2pt | 0.953 -2pt |
| Discussion semantics | 0.712 -2pt | 0.962 -2pt |
| Reply statistics | 0.712 -2pt | 0.959 -2pt |
| Risk statistics | 0.712 -2pt | 0.943 -2pt |
| Source balance | 0.686 -2pt | 0.965 -2pt |
| Measure | Gemini | GPT-4o-mini |
|---|---|---|
| Raw-label discussions | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} |
| Configuration discussions | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} |
| Fixed-evidence discussions | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} | {\color[rgb]{0.1367,0.3906,0.6484}40}{\color[rgb]{0.4375,0.2578,0.6797}+7} |
| First-attempt validity | {\color[rgb]{0.0313,0.4844,0.4688}\geq 183/186} | {\color[rgb]{0.0313,0.4844,0.4688}\geq 184/186} |
| Percentage (rounded) | 98.387 | 98.925 |
| Final response validity (%) | 100 | 100 |
| Dimension | Observed result and scope |
|---|---|
| Development repartitioning | 10 cluster repartitions; Taskwise AP SD ; paired wins over MiniLM. Mean AP meets 0.712, 0.887, 0.986, 0.893, 0.879, 0.981. |
| SCCD threshold | Precision 0.719; recall 0.713; macro-F1 0.812. Counts: 8,028 messages, 1,960 positives. |
| External discussions | sources; independent discussions each. Per-source AP ; paired gain over MiniLM . |
| Startup and assets | Loading s; first request s; matched hardware/cache. Encoder and learned assets: 477.844 MiB; other assets additional. |
| Human review | Median paired time ratio ( reduction); accuracy ; missed risk ; location success . |
| Case audit | 3 platforms; 12 independent discussions each. All tested identifier/span/parent mappings correct; semantic passes (0.917). |
Content Moderation
Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.
See all 91 papers