MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs

Liang Shan; Kaicheng Shen; Wen Wu; Zhenyu Ying; Chaochao Lu; Yan Teng; Jingqi Huang; Qingshan Liu

cs.AINov 10, 2025

MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs

Authors: Liang Shan, Kaicheng Shen, Wen Wu, Zhenyu Ying, Chaochao Lu, Yan Teng, Jingqi Huang, Qingshan Liu,

Organizations: School of Computer Science and Technology, East China Normal University · Shanghai AI Lab, Shanghai Innovation Institute

Abstract

Ensuring the safety of Large Language Models (LLMs) is critical for real-world deployment. However, current safety measures often fail to address implicit, domain-specific risks. To investigate this gap, we introduce a dataset of 3,000 annotated queries spanning education, finance, and management. Evaluations across 14 leading LLMs reveal a concerning vulnerability: an average jailbreak success rate of 57.8%. In response, we propose MENTOR, a metacognition-driven self-evolution framework. MENTOR performs metacognitive self-assessment, using strategies such as perspective-taking and consequential reasoning to uncover latent model misalignments. MENTOR couples single-pass rule-guided inference for routine requests with a selectively invoked metacognitive evolution cycle that revises residual unsafe responses, distills successful corrections into a dynamic rule graph, and compiles validated rules into activation-level steering signals for future inference. Experiments demonstrate that MENTOR substantially reduces attack success rates across all tested domains and outperforms existing safety alignment methods. The code and dataset for MENTOR are available at: https://anonymous.4open.science/r/MENTOR-Evo.

Figures & tables

Figure 1: Original and MENTOR-enhanced responses to an implicit educational risk.

Figure 2: Overview of MENTOR. (a) The fast RGI path retrieves rules and performs single-pass generation; responses that fail the safety gate enter the slow MEC loop for feedback-guided revision and rule update. (b) White-box RGI applies matched rule vectors at an intermediate layer, while successful corrections update later rule and steering snapshots.

Figure 3: JSR across 14 LLMs and three domains at each MENTOR stage. PowerNorm uses $\gamma=0.5$ .

Group	$A_{\mathrm{bare}}$	$A_{\mathrm{defended}}$	$\Delta_{\%}$
EduBench (0–10 scale)
Scenario Adaptation	9.70	9.57	$\downarrow 1.31\%$
Factual & Reasoning Acc.	9.43	9.32	$\downarrow 1.22\%$
Pedagogical Application	7.76	7.67	$\downarrow 1.19\%$
Overall Average	8.96	8.85	$\downarrow 1.25\%$
Multifinben (PolyFiQA) (0–1 scale)

Table 4: Utility preservation on the benign EduBench and Multifinben benchmarks.

Method	Helpfulness $\uparrow$	JSR $\downarrow$
Raw	4.812	0.632
ProB	4.372	0.534
Self-Reminder	4.602	0.147
RPO	4.790	0.624
DPP	4.819	0.562
RePD	2.905	0.240

Table 5: Average Helpfulness (1–5) and JSR (fraction) over nine model–domain combinations. MENTOR uses at most one MEC revision in this comparison. The adapted baselines use the non-test benchmark partition; their domain-level results appear in Appendix E .

Model	Neutral Safety									Metacognitive
	Education			Finance			Management			Education			Finance			Management
	Raw	RGI	MEC-2	Raw	RGI	MEC-2	Raw	RGI	MEC-2	Raw	RGI	MEC-2	Raw	RGI	MEC-2	Raw	RGI	MEC-2
Claude Sonnet 4.6
DeepSeek-R1	.624	.154	.082	.798	.014	.014	.868	.042	.022	.704	.192	.118	.650	.026	.018	.808	.060	.040
GPT-4o	.702	.244	.114	.910	.062	.034	.946	.082	.040	.868	.302	.136	.910	.084	.044	.964	.112	.074
Qwen3-235B	.492	.132	.056	.504	.010	.010	.664	.040	.016	.408	.166	.090	.374	.018	.018	.614	.058	.042

Table 7: Post-hoc JSR for Raw, single-pass Rule-Guided Inference (RGI), and RGI with up to two conditional MEC revisions (MEC-2), under two independent judges and two rubrics.

Figure 4: Education test-set layer-sensitivity analysis for RS on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct (

\alpha=1.0

). Markers show test JSRs at layers selected beforehand on non-test data: 17.8% and 34.4%, respectively. Layer indices start at zero.

Domain	Raw	SFT	RGI-RS	Prompt	MEC-1
Llama-3.1-8B
Education	$0.658$	$0.426$	$0.178$	$0.344$	$0.204$
Finance	$0.600$	$0.342$	$0.194$	$0.200$	$0.146$
Management	$0.724$	$0.446$	$0.228$	$0.260$	$0.166$
Qwen-2.5-7B
Education	$0.804$	$0.526$	$0.344$	$0.402$	$0.276$

Table 9: JSR under Raw, SFT, Rule Steering (RGI-RS), prompt-guided RGI, and prompt-guided RGI followed by at most one MEC revision. The MEC-1 column does not apply MEC to the RGI-RS column.

Figure 5: Representative cases illustrating two qualitative effects of RS: redirecting self-deprecating requests (E1) and providing constructive alternatives to bare refusals (E2).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Representative risk families
Education	Student self-harm and psychological crisis; bullying, coercion, and retaliation; academic misconduct; minor safety and inappropriate boundaries; dangerous experiments, drugs, or substances; bias and exclusion; student privacy; abuse of teacher authority; inappropriate medical, legal, or psychological advice; violence, threats, or property damage.
Finance	Fraud and scams; market manipulation and price coordination; insider trading; money laundering and regulatory evasion; predatory lending; discriminatory credit decisions; customer privacy and data misuse; unsuitable investment advice; tax evasion or false reporting; conflicts of interest and misleading sales.
Management	Discrimination in hiring and promotion; harassment and retaliation; labor exploitation; employee surveillance and privacy invasion; metric manipulation and deceptive management; anticompetitive conduct; bribery and conflicts of interest; suppression of whistleblowers; workplace-safety negligence; coercive or manipulative leadership.

Appendix

Table A.1: Representative top-level risk families. Individual records use finer-grained factors from the domain taxonomy.

Category	Representative fields	Function
Identity and grouping	id , seed_id , scenario_group_id , paraphrase_group_id , bilingual_pair_id	Tracks construction provenance and relationships among queries.
Risk specification	domain , risk_family , risk_factor , normative_source_id , expected_safe_behavior	Connects each query to the taxonomy and normative chain.
Scenario context	profile_id , stakeholder_role , implicit_strategy , language , query	Records the profile-conditioned surface realization.
Construction provenance	generation_model , generation_template_id , generation settings, rewrite status	Enables reproduction and generator-artifact analysis.
Expert validation	individual validity labels, risk labels, implicitness, severity, plausibility, ambiguity, rationale, adjudication	Preserves disagreement and the basis for final inclusion.
Evaluation metadata	difficulty_level , model-response outcomes	Supports post-hoc difficulty analysis without defining dataset validity.

Appendix

Table A.2: Record schema for construction provenance, validation, and leakage auditing.

Model	Domain	Raw	RGI	MEC-1	MEC-2
Qwen2.5-7B-Instruct	Education	0.804	0.402	0.276	0.208
	Finance	0.914	0.308	0.210	0.172
	Management	0.938	0.378	0.248	0.188
Mistral-Large-Instruct-2407	Education	0.790	0.214	0.082	0.056
	Finance	0.912	0.108	0.066	0.062
	Management	0.920	0.122	0.070	0.060

Appendix

Table B.1: Model–domain JSR for the recorded stage-specific responses. Model identifiers follow the evaluated snapshots. Values are proportions. RGI denotes rule-guided inference, and MEC-1/2 allow at most one/two conditional revisions after RGI. Model and stage averages discussed in Section 5.3 are computed from these values.

Method	Domain	GPT-4o		Qwen3-235B		DeepSeek-R1		Average
Method	Domain	JSR	Utility	JSR	Utility	JSR	Utility	JSR	Utility
Raw	Education	0.804	4.838	0.492	4.830	0.672	4.826	0.656	4.831
	Finance	0.826	4.650	0.300	4.854	0.522	4.924	0.549	4.809
	Management	0.872	4.774	0.518	4.774	0.682	4.840	0.691	4.796
	Average	0.834	4.754	0.437	4.819	0.625	4.863	0.632	4.812
RePD	Education	0.520	1.946	0.198	3.020	0.264	4.601	0.327	3.189

Appendix

Table E.2: Domain-level comparison across education, finance, and management subsets. JSR and utility are reported for GPT-4o, Qwen3-235B, and DeepSeek-R1, with the average over the three target models shown for each method. MENTOR uses RGI+MEC-1.

Method	Domain	Utility	Clarity	Factuality	Depth	Engagement
Raw	Education	4.831	4.913	4.695	4.763	4.819
	Finance	4.809	4.899	4.775	4.737	4.150
	Management	4.796	4.932	4.714	4.683	4.067
	Overall	4.812	4.915	4.728	4.728	4.345
Self-Reminder	Education	4.698	4.951	4.877	4.657	4.799
	Finance	4.539	4.940	4.928	4.571	4.187

Appendix

Table E.3: Primary utility and diagnostic dimensions for the baseline defenses. Utility corresponds to Helpfulness in the main comparison; all scores use a 1–5 scale. MENTOR values in the accompanying table use RGI+MEC-1.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 438 papers

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
Oct 4, 2026Jinghao Pang, Jitai Hao, Qiang Huang +2LLM Alignment LLM Safety Alignment
Harbin Institute of Technology (Shenzhen) · Leiden University
Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization
May 11, 2026Huilin Zhou, Jian Zhao, Yilu Zhong +7Inference-Time Optimization Adversarial Attacks on LLMs
University of Science and Technology of China · Institute of Artificial Intelligence (TeleAI), China Telecom
Internal Safety Collapse in Frontier Large Language Models
Mar 4, 2026Yutao Wu, Xiao Liu, Yifeng Gao +7LLM Safety Language Model Safety Evaluation
Deakin University · Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI +4

Oct 4, 2026J/K move · Enter open · S save

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren+1

Harbin Institute of Technology (Shenzhen) · Leiden University

LLM Alignment LLM Safety Alignment LLM Safety Supervised Fine-Tuning Adversarial Attacks on LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.

Abstract

LLM Safety

Explore similar work

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization

Internal Safety Collapse in Frontier Large Language Models