ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
Organizations: Fudan University · Shanghai Innovation Institute
Abstract
Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated (Query, CoT) pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B/3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. Resources are publicly available: Project Page: https://cosmosyi.github.io/ReasoningShield/, Code: https://github.com/CosmosYi/ReasoningShield, Models: https://huggingface.co/ReasoningShield/models, and Dataset: https://huggingface.co/datasets/ReasoningShield/ReasoningShield-Dataset.
Figures & tables
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Model Consistency Rate | Accuracy | Precision | Recall | F1 |
| 97.3 | 92.3 | 93.7 | 92.4 | 93.0 |
| Model | Size | GPU Usage (GB) | Throughput (Samples/s) | Latency(s) | Load Time(s) | Average Tokens Per Sample | Tokens Gen Per Second |
| LlamaGuard-1 | 7B | 12.55 | 5.02 | 0.20 | 3.67 | 1.42 | 7.13 |
| LlamaGuard-2 | 8B | 14.96 | 6.18 | 0.16 | 4.19 | 2.14 | 13.22 |
| LlamaGuard-3 | 8B | 14.96 | 5.41 | 0.18 | 3.95 | 3.14 | 17.00 |
| WildGuard | 7B | 27.00 | 0.67 | 1.49 | 9.17 | 18.00 | 12.08 |
| Qwen2.5-72B | 72B | 135.43 | 0.02 | 55.08 | 33.79 | 429.80 | 7.80 |
| ReasoningShield (Ours) | 1B | 2.30 | 0.22 | 4.63 | 1.61 | 362.23 | 78.19 |
| Source | Count | Quantity in Each Safety Level | ||
| Harmful | Potentially Harmful | Safe | ||
| AIR-Bench | 2932 | 1110 | 651 | 1171 |
| SALAD-Bench | 3004 | 1353 | 734 | 917 |
| Alpaca | 1064 | 1 | 2 | 1061 |
| Source | Count | Quantity in Each Safety Level | ||
| Harmful | Potentially Harmful | Safe | ||
| AIR-Bench | 452 | 164 | 89 | 199 |
| SALAD-Bench | 544 | 210 | 77 | 257 |
| BeaverTails | 560 | 119 | 91 | 350 |
| Jailbreak-Bench | 444 | 135 | 63 | 246 |
| TruthfulQA | 200 | 0 | 14 | 186 |
| Source | Count | Quantity in Each Safety Level | ||
| Harmful | Potentially Harmful | Safe | ||
| BeaverTails-30k-Test | 300 | 179 | - | 121 |
| WildGuard-Test | 300 | 150 | - | 150 |
| Parameter | Stage 1 SFT | Stage 2 DPO |
| Dataset | 4,358 agreed-upon samples ( ) | 2,642 hard negative samples ( ) |
| Batch Size | 2 | 2 |
| Gradient Accumulation Steps | 8 | 8 |
| Learning Rate | ||
| Learning Rate Scheduler | Cosine | Cosine |
| Epochs | 3 | 2 |
| Inference | Model name | Temperature | Top-p |
| vLLM | QWQ-32B | 0.6 | 0.9 |
| DeepSeek-R1-32B | 0.6 | 0.9 | |
| DeepSeek-R1-8B | 0.6 | 0.9 | |
| OpenThinker-7B | 0.6 | 0.9 | |
| Qwen2.5-72B-Instruct | 0.6 | 0.95 | |
| Gemma-3-27b-it | 0.6 | 0.95 |
| Model | AIR | SALAD | BeaverTails | JailbreakBench | Average |
| GuardReasoner-1B | 73.6 | 82.9 | 77.9 | 80.2 | 78.7 |
| GuardReasoner-3B | 73.8 | 82.8 | 80.1 | 83.2 | 80.0 |
| GuardReasoner-8B | 75.4 | 85.0 | 78.2 | 83.7 | 80.6 |
| ReasoningShield-1B | 91.2 | 87.4 | 84.3 | 86.9 | 87.5 |
| ReasoningShield-3B | 92.2 | 90.6 | 86.3 | 89.1 | 89.6 |
| Model | Safe | Potentially Harmful | Harmful | Macro-F1 |
| ReasoningShield-3B | 93.0 | 82.4 | 91.9 | 89.1 |
| Model | Risky-Trace Evidence Hit Rate | Safe False-Localization Rate |
| Qwen2.5-32B-Instruct | 62.5 | 3.75 |
| Llama-3.2-3B-Instruct | 54.2 | 43.75 |
| ReasoningShield-3B | 89.2 | 0.0 |
| Base Model (Qwen2.5) | Overall F1 | Level-0.5 Accuracy (%) | Latency (s/sample) | GPU Mem (GB) | Load Time (s) |
| 0.5B | 0.801 | 65.4 | 9.29 | 0.92 | 4.57 |
| 1.5B | 0.848 | – | 10.31 | 2.88 | 4.84 |
| 3B | 0.873 | 76.8 | 12.96 | 5.75 | 7.22 |
| 7B | 0.885 | 81.2 | 14.00 | 14.19 | 11.05 |
| 32B | 0.895 | 83.7 | 22.56 | 61.03 | 125.66 |
| 72B | 0.901 | – | 30.27 | 135.43 | 277.69 |
| Prompt component | Accuracy | F1 |
| Baseline (our prompt) | 89.4 | 88.0 |
| w/o evaluation guidance | 83.0 (-6.4) | 83.0 (-5.0) |
| w/o analysis process | 67.5 (-21.9) | 72.3 (-15.7) |
| LlamaGuard-3’s prompt | 82.4 (-7.0) | 79.3 (-8.7) |
| Training strategy | CoT moderation | Answer moderation | ||
| Accuracy | F1 | Accuracy | F1 | |
| SFT+DPO (ours) | 87.0 | 86.3 | 79.8 | 81.3 |
| SFT only | 77.1 (-9.9) | 76.8 (-9.5) | 70.0 (-9.8) | 72.5 (-8.8) |
| Llama-3.2-1B | 6.1 (-80.9) | 4.2 (-82.1) | 6.7 (-73.1) | 5.7 (-75.6) |
| Model | Accuracy | F1 Score |
| LlamaGuard-4-12B | 0.333 | 0.500 |
| ReasoningShield-3B | 0.873 | 0.932 |
| Model | Accuracy | F1 Score |
| LlamaGuard-4-12B | 0.495 | 0.281 |
| ReasoningShield-3B | 0.868 | 0.897 |
| Model & Strategy | Overall F1 |
| LlamaGuard-4-12B (Holistic / Standard) | 50.7% |
| LlamaGuard-4-12B (Stepwise – New) | 76.7% |
| WildGuard-7B (Holistic / Standard) | 63.9% |
| WildGuard-7B (Stepwise – New) | 85.3% |
| ReasoningShield-3B (Ours) | 92.5% |
| Configuration | Metric | Dataset | |
| BeaverTails | WildGuard-Test | ||
| RS-3B (Permissive) | Precision | 0.908 | 0.964 |
| Recall | 0.659 | 0.720 | |
| F1 | 0.764 | 0.824 | |
| RS-3B (Strict) | Precision | 0.821 | 0.830 |
| Recall | 0.844 | 0.880 | |
| Risk category | Precision | Recall | F1 |
| Violence | 0.814 | 0.960 | 0.881 |
| Sex | 0.787 | 0.937 | 0.855 |
| Child | 0.901 | 0.877 | 0.889 |
| Cyber | 0.906 | 0.928 | 0.917 |
| Hate | 0.815 | 0.907 | 0.858 |
| Deception | 0.863 | 0.910 | 0.886 |
| Model | Safety Level (Label) | ||
| Safe (0) | Potentially Harmful (0.5) | Harmful (1) | |
| ReasoningShield-3B | 0.896 | 0.817 | 0.989 |
| Model | CoT Moderation | Answer Moderation |
| Moderation API | ||
| Perspective | 0.00 | 0.00 |
| OpenAI Moderation | 0.00 | 0.00 |
| Prompted LLM | ||
| GPT-4o (LG-3) | 0.51 | 0.40 |
| Qwen-2.5 (LG-3) | 0.75 | 0.43 |
Language Model Safety Evaluation
Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.
See all 521 papers