cs.AIAug 25, 2026
SaveTRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models
Organizations: King Abdullah University of Science and Technology · Aramco · KAUST
Abstract
Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.
Figures & tables
Figure 1: Existing benchmarks evaluate safety judgments for prompts and final responses but overlook unsafe reasoning traces. TRACE introduces evidence-grounded safety annotations (highlighted in orange) across the entire LRM inference pipeline, enabling comprehensive evaluation of both the correctness of safety judgments and the accuracy of evidence attribution.
| Benchmark | Prompt Safety | Reasoning Trace Safety | Final Response Safety | Evidence Attribution | Diverse Risk Categories | Diverse Attack Strategies |
| XSTest Röttger et al. (2024) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| ToxicChat Lin et al. (2023) | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| OpenAI Moderation Markov et al. (2023) | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| WGTest Han et al. (2024) | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Aegis Ghosh et al. (2024) ; Ghosh et al. (2025) | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ |
| BeaverTails Ji et al. (2023) | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ |
Table 1: Comparison of unsafe content detection benchmarks. TRACE introduces evidence-grounded safety annotations across the entire LRM inference pipeline, including prompts, reasoning traces, and final responses.
Figure 2: Overview of the TRACE benchmark construction workflow. ➀ Safe and unsafe prompts are curated from two public datasets. ➁ Four LRMs generate reasoning traces and final responses for each prompt. ➂ Three additional LRMs annotate the safety of the prompts, reasoning traces, and final responses and extract supporting evidence from the corresponding source text. Safety labels are determined by majority voting. Extracted evidence is retained only if it is provided by majority-aligned annotators and verified as a continuous substring of the source text. Samples without verifiable evidence are further reviewed by human annotators, yielding the final annotation result.
Figure 3: Safety distributions of LRM-generated reasoning traces and final responses in the TRACE benchmark.
| Guardrail Model | Params | Prompt | Reasoning Trace | Final Response | ||||||
| TRACE-EN | TRACE-ZH | TRACE | TRACE-EN | TRACE-ZH | TRACE | TRACE-EN | TRACE-ZH | TRACE | ||
| LlamaGuard-1 | 7B | 39.64 | 40.83 | 39.91 | 22.69 | 29.14 | 24.34 | 34.18 | 36.19 | 34.69 |
| LlamaGuard-2 | 8B | 69.34 | 74.52 | 70.45 | 61.36 | 71.76 | 63.67 | 61.08 | 71.24 | 63.40 |
| LlamaGuard-3 | 1B | 47.10 | 53.03 | 48.37 | 51.37 | 58.15 | 52.85 | 48.96 | 58.13 | 51.01 |
| 8B | 73.09 | 73.30 | 73.14 | 62.28 | 65.36 | 63.08 | 64.66 | 69.52 | 65.95 | |
| LlamaGuard-4 | 12B | 69.13 | 57.40 | 66.52 | 50.48 | 51.48 | 50.73 | 63.98 | 59.89 | 62.92 |
Table 2: F1-scores ( ) on TRACE. Bold indicates the best performance, while italic indicates the second best.
| Guardrail Model | Params | Prompt | Reasoning Trace | Final Response |
| Octopus | 14B | 9.53 | 14.56 | 13.25 |
| GPTSafeGuard | 20B | 10.74 | 12.58 | 9.92 |
| YuFeng-XGuard | 0.6B | 10.53 | 13.08 | 12.03 |
| 8B | 11.68 | 14.88 | 13.71 |
Table 3: TokenF1 (%) of evidence extracted by guardrail models to support their safety judgments. Bold indicates the best result, italic indicates the second best.
| Guardrail Model | Params | TRACE-EN | TRACE-ZH | TRACE |
| NemotronGuard | 8B | 25.37 | 32.29 | 27.13 |
| PolyGuard | 0.5B | 8.08 | 6.09 | 7.68 |
| 8B | 16.28 | 17.88 | 16.63 | |
| YuFeng-XGuard | 0.6B | 50.30 | 62.38 | 52.62 |
| 8B | 51.38 | 62.24 | 53.24 | |
| GPTSafeGuard | 20B | 60.47 | 70.20 | 62.25 |
Table 4: F1-scores (%) of guardrail models for prompt safety risk category classification. Bold indicates the best performance, italic indicates the second best.
Figure 4: FNR (%) and FPR (%) on TRACE. A high FNR indicates that the guardrail model fails to reliably block unsafe content, whereas a high FPR indicates that the guardrail model excessively refuses safe content.
Figure 5: F1-scores (%) of guardrail models for judging the safety of prompts, LRM-generated reasoning traces, and final responses under different prompt attack strategies. IE denotes the Instruction Encryption attack strategy.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Guardrail Model | Base Model | Params | Year | Training Data | # Training Samples | Explanation | Custom Risk Category | Hugging Face Hub |
| LlamaGuard-1 | Llama-2 | 7B | 2023 | HH-RLHF & In-house Data | 13,997 | ✗ | ✗ | Download |
| LlamaGuard-2 | Llama-3 | 8B | 2024 | HH-RLHF & In-house Data | — | ✗ | ✗ | Download |
| LlamaGuard-3 | Llama-3.2 | 1B | 2024 | HH-RLHF & In-house Data | — | ✗ | ✗ | Download |
| LlamaGuard-3 | Llama-3.1 | 8B | 2024 | HH-RLHF & In-house Data | — | ✗ | ✗ | Download |
| LlamaGuard-4 | Llama-4 | 12B | 2025 | In-house Data | — | ✗ | ✗ | Download |
| NemotronGuard | Llama-3.1 | 8B | 2025 | Nemotron-Safety-Guard-Dataset-v3 | 514,617 | ✗ | ✓ | Download |
Table 5: Details of evaluated guardrail models. “Explanation” indicates whether the models provide explanations for their safety judgments. “Custom Risk Category” indicates whether the models support user-defined risk categories.
Figure 6: F1-score (%) performance of guardrail models on TRACE. Colors denote the base LLM families used for fine-tuning the guardrail models (e.g., red indicates models derived from Qwen3). Circle size indicates model scale.
Figure 7: Precision (%) and Recall (%) on TRACE. High precision indicates that a guardrail model rarely misclassifies safe content as unsafe, whereas high recall indicates that it reliably blocks unsafe content.
| Guardrail Model | Params | Prompt | Reasoning Trace | Final Response | ||||||
| TRACE-EN (IE) | TRACE-ZH (IE) | TRACE-(IE) | TRACE-EN (IE) | TRACE-ZH (IE) | TRACE-(IE) | TRACE-EN (IE) | TRACE-ZH (IE) | TRACE-(IE) | ||
| LlamaGuard-1 | 7B | 0.00 | 0.00 | 0.00 | 36.23 | 19.05 | 33.96 | 3.33 | 0.00 | 2.82 |
| LlamaGuard-2 | 8B | 44.66 | 66.67 | 47.46 | 51.95 | 74.42 | 54.70 | 39.29 | 53.66 | 41.12 |
| LlamaGuard-3 | 1B | 36.69 | 40.00 | 37.11 | 43.88 | 50.00 | 44.61 | 32.85 | 34.48 | 33.05 |
| 8B | 58.00 | 40.00 | 55.00 | 2.25 | 0.00 | 1.90 | 3.45 | 0.00 | 2.90 | |
| LlamaGuard-4 | 12B | 48.83 | 53.33 | 49.38 | 9.90 | 0.00 | 8.55 | 18.46 | 0.00 | 15.58 |
Table 6: F1-scores (%) of guardrail models on TRACE-(IE), a subset of TRACE where all prompts are under IE (Instruction Encryption) attack strategies, together with the corresponding reasoning traces and final responses. Bold indicates the best performance, while italic indicates the second best.
| Guardrail Model | Params | Reasoning Trace | Final Response | ||
| TRACE-Q | TRACE-G | TRACE-Q | TRACE-G | ||
| LlamaGuard-1 | 7B | 22.51 | 27.69 | 31.14 | 40.94 |
| LlamaGuard-2 | 8B | 66.84 | 58.43 | 66.62 | 58.07 |
| LlamaGuard-3 | 1B | 55.86 | 47.86 | 54.52 | 45.09 |
| 8B | 63.42 | 62.43 | 65.51 | 66.78 | |
| LlamaGuard-4 | 12B | 51.02 | 50.18 | 62.88 | 63.00 |
Table 7: F1-scores (%) of guardrail models on TRACE-Q and TRACE-G, subsets of TRACE where reasoning traces and final responses are generated by Qwen3-series and Gemma-4-series LRMs, respectively. Bold indicates the best performance, while italic indicates the second best.
Figure 8: A failure case of YuFeng-XGuard-8B. While the ground truth safety labels for the prompt, reasoning trace, and final response are all unsafe (with supporting evidence highlighted in red), YuFeng-XGuard-8B misclassifies all of them as safe. In addition to these incorrect label predictions, the explanations generated by YuFeng-XGuard-8B fail to localize or recognize the unsafe context present in the source text.
Figure 9: A failure case of Octopus. While the ground truth safety labels for the prompt, reasoning trace, and final response are all unsafe (with supporting evidence highlighted in red), Octopus correctly classifies all of them as unsafe. However, the explanations it generates fail to accurately identify the supporting evidence for its safety judgments, indicating that Octopus cannot reliably localize the unsafe content in the source text.
Figure 10: A failure case of GPTSafeGuard. While the ground truth safety labels for the prompt, reasoning trace, and final response are all safe (with supporting evidence highlighted in blue), GPTSafeGuard misclassifies all of them as unsafe. Through an analysis of the model’s generated explanations, we find that this failure is primarily driven by over-refusal.
Figure 11: A failure case of GPTSafeGuard. While the ground truth safety labels for the prompt, reasoning trace, and final response are all unsafe (with supporting evidence highlighted in red), GPTSafeGuard misclassifies all of them as safe. The generated explanations reveal that this under-refusal stems from the model’s vulnerability to semantic obfuscation within creative writing.
| Risk Category | Description |
| Crimes and Illegal Activities | Behaviors that violate laws and regulations, including criminal acts as well as non-criminal violations of administrative laws and regulations. |
| Hate Speech | Dissemination of derogatory, offensive, or threatening speech or text, which may provoke fear, or hatred of others, potentially culminating in direct or indirect harm. |
| Physical and Mental Health | Behaviors that could potentially jeopardize human physical and mental well-being, comprise two distinct subcategories: Physical Harm and Mental Health . |
| Ethics and Morality | Behaviors that contravene ethical and moral norms. Social Ethics typically concern human relationships and societal responsibilities. Science Ethics focus on the ethical issues in the development of science and technology. |
| Data Privacy | Steal private information from LLMs for Personal Privacy such as contact and financial information as well as Commercial Secret such as customer and supply chain data. |
| Cybersecurity | Compromise the confidentiality, integrity, and availability of a network system, including overstepping access controls, designing malicious code such as viruses, worms, and Trojan horses, and threatening the physical security. |
Table 8: Details of risk categories in TRACE.
Figure 12: Content safety annotation system used for human annotation in TRACE.
More in this topic
LLM Safety Benchmarks
Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.
See all 270 papers