Tool-MCoT: Tool Augmented Multimodal Chain-of-Thought for Content Safety Moderation

Shutong Zhang; Dylan Zhou; Yinxiao Liu; Yang Yang; Huiwen Luo; Wenfei Zou

cs.CLMar 15, 2026

Tool-MCoT: Tool Augmented Multimodal Chain-of-Thought for Content Safety Moderation

Authors: Shutong Zhang, Dylan Zhou, Yinxiao Liu, Yang Yang, Huiwen Luo, Wenfei Zou

Organizations: Stanford University, USA · Google LLC, USA

Abstract

The growth of online platforms and user content requires strong content moderation systems that can handle complex inputs from various media types. While large language models (LLMs) are effective, their high computational cost and latency present significant challenges for scalable deployment. To address this, we introduce Tool-MCoT, a small language model (SLM) fine-tuned for content safety moderation leveraging external framework. By training our model on tool-augmented chain-of-thought data generated by LLM, we demonstrate that the SLM can learn to effectively utilize these tools to improve its reasoning and decision-making. Our experiments show that the fine-tuned SLM achieves significant performance gains. Furthermore, we show that the model can learn to use these tools selectively, achieving a balance between moderation accuracy and inference efficiency by calling tools only when necessary.

Figures & tables

Figure 1. An overview of our two-stage pipeline. We first generate tool reasoning data using a Gemini 2.0 flash as a teacher model, then use the generated reasoning data to fine-tune the SLM. During inference, the fine-tuned SLM utilizes the tool framework to conduct the content safety moderation task. Method figure.

	HatefulMemes		MMHS150K		UnsafeBench
Model	Accuracy	F1-Score	Accuracy	F1-Score	Accuracy	F1-Score
Gemma3-4B-it	0.646	0.613	0.395	0.458	0.486	0.523
w/ LoRA	0.738	0.734	0.626	0.625	0.709	0.741
w/ Reasoning LoRA	0.777	0.763	0.655	0.677	0.743	0.757
w/ GRPO	0.808	0.808	0.662	0.674	0.767	0.773

Table 1. Fine-tune small language model using tool reasoning dataset.

Model	Accuracy	F1-Score	OCR Ratio	Captioner Ratio	Detector Ratio	Run Time (s)
Gemma3-4B-it	0.395	0.458	-	-	-	0.13
w/ Force Tool	0.655	0.677	1	1	1	1.67
w/ Select Tool	0.634	0.651	0.203	0.484	0.263	0.86

Table 2. Finetune small language model to actively call tools when necessary.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2. Qualitative comparison between model outputs with and without tool reasoning. The tool reasoning model (bottom) correctly identifies the nuanced context in both cases, where the standard model fails. Qualitative comparison showing two examples of model outputs with and without tool reasoning. The tool reasoning model identifies the intended context more accurately than the standard model.

Figure 3. Multi-turn tool calling conversation. Model select necessary tools for harder samples. Tool calling example.

Multimodal Content Moderation

See all 21 papers

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
May 22, 2025Changyi Li, Jiayi Wang, Xudong Pan +2Language Model Safety Evaluation LLM Safety Benchmarks
Fudan University · Shanghai Innovation Institute
Nemotron 3.5 Content Safety Moderator: A Compact Multimodal, Multilingual, and Reasoning Enabled Content Safety Moderator
Aug 27, 2026Varun Singh, Anuj Doshi, Makesh Narsimhan Sreedhar +2Multimodal Content Moderation LLM Guardrails
NVIDIA Santa Clara, CA
No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
Aug 22, 2026Afshin Orojlooyjadid, Hitesh PatelLLM Safety Evaluation LLM Safety Benchmarks
Oracle AI

May 22, 2025J/K move · Enter open · S save

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

Changyi Li, Jiayi Wang, Xudong Pan, Geng Hong+1

Fudan University · Shanghai Innovation Institute

Language Model Safety Evaluation LLM Safety Benchmarks Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: https://cosmosyi.github.io/ReasoningShield/, Code: https://github.com/CosmosYi/ReasoningShield, Models: https://huggingface.co/ReasoningShield/models, and Dataset: https://huggingface.co/datasets/ReasoningShield/ReasoningShield-Dataset.

Abstract

Multimodal Content Moderation

Explore similar work

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

Nemotron 3.5 Content Safety Moderator: A Compact Multimodal, Multilingual, and Reasoning Enabled Content Safety Moderator

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios