Content moderation in online platforms faces persistent challenges due to the evolving complexity of user-generated content and the limitations of traditional rule-based and machine learning approaches. While recent advances in large language models (LLMs) have enabled more sophisticated moderation via direct prompting or fine-tuning, these approaches often exhibit limited generalization, interpretability, and adaptability to unseen or ambiguous cases. In this work, we propose a novel moderation framework that leverages analogical examples to enhance rule induction and decision reliability. Our approach integrates end-to-end optimization of analogical retrieval, rule generation, and moderation classification, enabling the dynamic adaptation of moderation rules to diverse content scenarios. Through comprehensive experiments, we demonstrate that our method significantly outperforms both rule-injected fine-tuning baselines and multi-stage static RAG pipelines in terms of moderation accuracy and rule quality. Further evaluations, including human assessments and external model generalization tests, confirm that our framework produces rules with better clarity, interpretability, and applicability. These findings show that analogical example-driven methods can advance robust, explainable, and generalizable content moderation in real-world applications.
Figures & tables
Figure 1: Comparison of the Chain of Thought (CoT) method and the ChAIRO framework for content moderation. CoT fails to identify the implicit bias and classifies it as ’Safe,’ while ChAIRO , leveraging analogical reasoning and explicit rule induction, correctly identifies it as ’Unsafe’ by recognizing the underlying discriminatory logic.
Figure 2: The workflow of the ChAIRO framework.
Category
Model/Method
Average
Politics
Pornography
Violence
Gambling
Bias
Harmless
General LLMs
GPT-4
72.3
58.6
88.7
79.8
92.7
64.3
56.8
DeepSeek R1
77.1
72.7
91.4
86.1
94.3
64.6
59.7
DeepSeek V3
80.3
79.0
90.3
89.8
95.0
70.5
62.5
Qwen2.5-32B-Instruct
74.3
59.1
91.1
84.4
95.4
67.9
54.2
QwQ-32B
69.1
75.4
69.6
72.0
84.9
60.7
54.6
LLaMA3-8B
67.5
58.5
55.9
81.0
90.6
65.2
44.2
Table 1: Moderation F1 Scores for General LLMs, Specific LLMs and ChAIRO
Setting
Overall F1
Politics
Pornography
Violence
Gambling
Bias
Harmless
w/o k-NN Retrieval
86.9 (-2.3)
85.4 (-3.9)
69.0 (-2.5)
96.6 (-1.2)
78.5 (-3.5)
93.6 (-2.5)
98.2 (-0.4)
w/o Second-stage SFT
88.0 (-1.2)
86.1 (-3.2)
69.3 (-2.2)
97.2 (-0.6)
81.2 (-0.8)
95.8 (-0.3)
98.4 (-0.2)
ChAIRO (Ours)
89.2
89.3
71.5
97.8
82.0
96.1
98.6
Table 2: Ablation Study on Model Components with Performance Change
Categories
Qwen3-8B
RAG
SFT
ChAIRO (Ours)
Hate
67.2
62.3
74.7
81.5
Sexual
65.2
72.3
71.4
81.6
Confessions
66.4
68.3
80.0
82.5
Harassment
25.8
27.6
26.7
40.7
Profanity
40.0
42.9
55.0
53.9
Average F1-Score
52.9
54.7
61.6
68.0
Table 3: F1-Score Comparison on Aegis Dataset
Model
F1
Human (%)
Qwen2.5-32B-Instruct
74.3
-
Simple Rule
75.1
15
RQ4
88.7
85
Table 4: Rule Quality Evaluation Results. The F1-score reflects the generalization ability of rules when applied to an external moderation model. The "Human (%)" denotes the preference rate derived from a double-blind comparison of 100 test cases by three annotators with content moderation experience, where rules are assessed for contextual relevance, completeness, and alignment with human judgment criteria.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Prompt for synthesizing the chain of analogical inductive reasoning.
Current large language models (LLMs), even those explicitly trained for reasoning, often struggle with ambiguous content moderation cases due to misleading "decision shortcuts" embedded in context. Inspired by cognitive psychology insights into expert moderation, we introduce \caro (Chain-of-Analogy Reasoning Optimization), a novel two-stage training framework to induce robust analogical reasoning in LLMs. First, \caro bootstraps analogical reasoning chains via retrieval-augmented generation (RAG) on moderation data and performs supervised fine-tuning (SFT). Second, we propose a customized direct preference optimization (DPO) approach to reinforce analogical reasoning behaviors explicitly. Unlike static retrieval methods, \caro dynamically generates tailored analogical references during inference, effectively mitigating harmful decision shortcuts. Extensive experiments demonstrate that \caro substantially outperforms state-of-the-art reasoning models (DeepSeek R1, QwQ), specialized moderation models (LLaMA Guard), and advanced fine-tuning and retrieval-augmented methods, achieving an average F1 score improvement of 24.9% on challenging ambiguous moderation benchmarks.
Bingzhe Wu, Haotian Lu, Yuchen Mou
Shenzhen University · Tsinghua University · National University of Singapore
The growth of online platforms and user content requires strong content moderation systems that can handle complex inputs from various media types. While large language models (LLMs) are effective, their high computational cost and latency present significant challenges for scalable deployment. To address this, we introduce Tool-MCoT, a small language model (SLM) fine-tuned for content safety moderation leveraging external framework. By training our model on tool-augmented chain-of-thought data generated by LLM, we demonstrate that the SLM can learn to effectively utilize these tools to improve its reasoning and decision-making. Our experiments show that the fine-tuned SLM achieves significant performance gains. Furthermore, we show that the model can learn to use these tools selectively, achieving a balance between moderation accuracy and inference efficiency by calling tools only when necessary.
Large language models (LLMs) demonstrate strong performance on standard content moderation benchmarks. However, these benchmarks often aggregate multiple moderation criteria into a single label, making it unclear whether models can disentangle them and reliably apply each criterion when making decisions. To study whether LLMs exhibit criterion-conditioned behaviour, we introduce Diagnostic Evaluation of COntent (DECO), a criterion-independent factorisation of content that enables controlled, criterion-level evaluation. We also introduce pairwise evaluation to compare model outputs across different criteria for the same input. Across four moderation datasets and four LLMs, we find that strong benchmark performance can hide substantial failures at the criterion level. Models struggle most when correct decisions depend not on overall harmfulness, but on the specific aspect of the content that the criterion requires them to assess. Our results highlight a key limitation of current content moderation benchmarks: strong performance on aggregated labels does not provide sufficient evidence that LLMs can reliably evaluate content with respect to individual moderation criteria. These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.