Overview

Content moderation aims to identify material such as hate speech, harassment, self-harm, and unsafe images or videos, then guide decisions about removal, review, or rewriting. Approaches range from text classifiers and multimodal vision-language models to policy-guided language models using retrieval, examples, or step-by-step reasoning; some systems also screen content as it is generated or streamed. Researchers evaluate these methods on labeled datasets and policy-specific benchmarks, including tests of multilingual content, memes, evolving rules, and adversarial changes to text or images. A central tension is that models must apply context-dependent policies consistently while avoiding unfair errors, resisting evasion, and meeting the cost and latency limits of real moderation workflows.

Latest papers

91 papers9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

May 27, 2025

Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate language in online comments. With the increasing challenge of moderating vast volumes of user-generated content on social network sites, the role of AI in content moderation has gained prominence. We compared ChatGPT's performance against crowd-sourced annotations and expert evaluations to assess its accuracy, scope of detection, and consistency. Our findings show that ChatGPT performs well in detecting inappropriate content, and that almost the entire gain from iterative prompt refinement came from a single change --- simplifying the label scheme and moving to GPT-4 --- after which further adjustments to prompt structure and sampling temperature had little effect. Its performance on targeting language was weaker and less stable, with higher false positive rates than expert judgments before refinement and higher false negative rates after a change of model. This study contributes to the field by demonstrating the potential of AI models like ChatGPT to enhance automated content moderation systems while also identifying areas for further improvement. The results underscore the importance of careful prompt design and model selection, and of contextual understanding, in supporting automated moderation and mitigating harmful online behavior.