cs.AINov 7, 2026

TrustMod-SM: A Multi-Axis Benchmark for Evaluating Trustworthiness of LLMs in Social Media Content Moderation

Authors: Aizan Zafar, Sreekantam Sai Venkata, Gathik Jindal, Ritu Raj Sharma, Zishan Ahmad, Tulika Saha, Srinath Srinivasa

Organizations: International Institute of Information Technology Bangalore, India · Quantiphi Analytics, India · University of Liverpool Bengaluru, India

Abstract

Large Language Models (LLMs) show strong potential for social media content moderation, but their trustworthiness beyond overall accuracy remains underexplored. We introduce TrustMod-SM, a benchmark of roughly 29,000 instances across eight datasets, to evaluate LLM moderators on five dimensions: trustfulness, fairness, safety, robustness, and context integrity. Evaluating 16 open-weight models (0.5B-284B parameters, spanning text-only and vision-language architectures) and one closed frontier model, we find persistent trustworthiness gaps. Higher detection accuracy often coincides with greater demographic disparity, and models frequently comply with concealment or exaggeration prompts, assign high confidence to wrong predictions, rarely escalate ambiguous cases, and struggle to separate hate speech from counter-speech or reclaimed language (especially below 8B parameters). Scaling does not uniformly resolve these issues, and no single model excels across all five dimensions.

Figures & tables

Appendix figures & tables70 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

    Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety BenchmarksLLM Evaluation

  2. LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation

    Sep 27, 2026Jie Wang, Yanbo Sun, Zheng Yan +2LLM EvaluationTrust in AI

  3. TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

    Apr 15, 2026Yichuan Mo, Yukun Jiang, Yanbo Shi +4Diffusion Language Model BenchmarkingLLM Safety Benchmarks