MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Models

Yan Zhang; Chuming Wei; Ruien Li; Yaoyao Peng; Wusheng Zhang; Guangwen Yang

cs.AISep 30, 2026

MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Models

Authors: Yan Zhang, Chuming Wei, Ruien Li, Yaoyao Peng, Wusheng Zhang, Guangwen Yang

Organizations: Department of Computer Science and Technology, Tsinghua University · Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University · Department of Computer Sciences, University of Wisconsin–Madison · Law School, University of Chinese Academy of Social Sciences

Abstract

Large Language Models (LLMs) have been applied in various fields. However, ensuring compliance and safety of LLMs, such as avoiding discrimination and bias, still remains a challenge. Current efforts mainly focus on detecting and filtering inputs and outputs of the trained models, rather than studying the intrinsic architecture of the models in real-time. To tackle this challenge, we analyze the LLMs training process and discover two critical issues: 1) Most of the existing methods are predominantly static in their approach to detection and filtering, achieving only localized optimizations without systematically enhancing the compliance of LLMs. 2) Another issue with existing approaches is the lack of real-time risk detection and mitigation across the full training process, which leads to limited flexibility. Motivated by these, we propose MASCRDM (Multi-Agent System for Compliance Risk Detection and Mitigation) during the LLM training process. Firstly, we develop a set of compliance rules based on existing Artificial Intelligence (AI) laws and a compliance-specific LLM with the instruction of compliance law experts. Then, we deconstruct LLMs into several components and identify key nodes based on the compliance knowledge graph. During LLMs training, we implement our multiple agents in the whole process, giving compliance risk alerts and suggestions for LLM developers. Experiments on discrimination and bias benchmark demonstrate that our multi-agent system can effectively improve the compliance while maintaining reasonable semantic performance. The results indicate that our method provides an executable path for mitigating compliance risk from within the LLMs systematically.

Figures & tables

Figure 1: Multi-Agent System for Compliance Risk Detection and Mitigation. A systematic framework of real-time risk monitoring and mitigation during the LLM training process.

Method	BBQ					CrowS-Pairs
	Acc.	A.Amb	A.Dis	B.Amb	B.Dis	SS
	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\approx 0$ )	( $\approx 0$ )	( $\approx 50$ )
Qwen3-8B-Base	61.42	46.17	76.67	3.99	1.02	64.46
KLAAD	62.09	45.65	78.52	5.94	0.83	69.56
Fairness mediator	59.46	45.01	73.90	3.41	1.43	62.40
Bias Unlearning	63.01	52.13	73.89	3.37	1.11	61.74

Table 1: Evaluation of MASCRDM on BBQ and CrowS-Pairs datasets. “A.”=Accuracy, “B.”=BiasScore. ”Amb” = Ambiguous context, ”Dis” = Disambiguated context. We highlight the best-performing score in bold and the second-best with an underline for each metric.

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Profession (Engineering Branches)	Qwen3-8B-Base	+0.12	+0.32	-0.13	+0.27	0.22	0.14	0.14	0.16	0.14
	KLAAD	+0.21	+0.33	-0.13	+0.23	0.24	0.15	0.15	0.17	0.15
	Fairness mediator	+0.13	+0.33	-0.14	+0.28	0.22	0.14	0.14	0.16	0.14
	Bias Unlearning	+0.13	+0.33	-0.14	+0.27	0.22	0.14	0.14	0.15	0.14
	MASCRDM	+0.12	+0.32	-0.13	+0.27	0.22	0.14	0.14	0.15	0.14

Table 2: Selected evaluation of MASCRDM on the BOLD dataset. “V” = Valence, “A” = Arousal, and “D” = Dominance. Smaller absolute values are better for Sentiment, VAD, and BE5. We highlight the best-performing score in bold and the second-best with an underline for each metric.

Figure 2: Training loss trajectories of standard SFT and MASCRDM on Llama3.1-8B, recorded every 10 global steps (left). Changes in pairwise $L_{2}$ distances between sensitive-word embeddings for Llama3.1-8B, computed as MASCRDM minus standard SFT. Red and blue indicate increases and decreases, respectively. The panels use different color scales (right).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Multi-Agent System for Compliance Risk Detection and Mitigation. A systematic framework of real-time risk monitoring and mitigation during the LLM training process.

Parameter	Value
Training data	Alpaca + Stereo
Visible GPUs	4 A800
Maximum sequence length	1024
Micro-batch size	4
Gradient accumulation steps	8
Effective batch size per update	32

Appendix

Table 3: Shared fine-tuning parameters used across the compared methods. The model path and output directory are method-specific and are therefore not listed as shared parameters.

Parameter	Value
Audit frequency	Every global step
Probe set	36 S/C/N triplets
Embedding trigger	$\Delta S>10^{-6}$ / $\Delta C>10^{-6}$
Embedding correction	$\alpha=0.25$
Attention threshold	P90
Attention strength	$\alpha\in[0.10,0.50]$

Appendix

Table 4: Real-time auditing parameters used in the MASCRDM instance.

Figure 4: Training loss trajectories of standard SFT and MASCRDM on Qwen3-8B, recorded every 10 global steps (left). Changes in pairwise $L_{2}$ distances between sensitive-word embeddings for Qwen3-8B, computed as MASCRDM minus standard SFT. Red and blue indicate increases and decreases, respectively. The panels use different color scales (right).

Method	BBQ
	Acc.	A.Amb	A.Dis	B.Amb	B.Dis
	( $\uparrow$ )	( $\uparrow$ )	( $\uparrow$ )	( $\approx 0$ )	( $\approx 0$ )
Qwen3-8B-Base	60.94	47.54	74.34	3.59	1.13
Qwen3-8B-Base+Data Audit	63.41	50.36	76.45	3.76	0.92
+Embedding	63.43	50.57	76.29	3.73	1.10
+Attention	62.88	52.53	73.22	3.37	1.00

Appendix

Table 5: Ablation Study of MASCRDM on BBQ and CrowS-Pairs datasets. “A.”=Accuracy, “B.”=BiasScore. ”Amb” = Ambiguous context, ”Dis” = Disambiguated context. We highlight the best-performing score in bold and the second-best with an underline for each metric.

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Gender (Male)	Qwen3-8B-Base	+0.30	+0.43	-0.06	+0.30	0.24	0.14	0.14	0.16	0.13
	KLAAD	+0.72	+0.60	+0.03	+0.44	0.29	0.14	0.14	0.16	0.13
	Fairness mediator	+0.21	+0.42	-0.05	+0.28	0.24	0.14	0.14	0.16	0.14
	Bias Unlearning	+0.28	+0.43	-0.06	+0.28	0.24	0.14	0.14	0.15	0.13
	MASCRDM	+0.29	+0.44	-0.07	+0.29	0.24	0.14	0.14	0.15	0.13

Appendix

Table 6: Evaluation results of Qwen3-8B-Base, KLAAD, Fairness mediator, Bias Unlearning and MASCRDM on the BOLD dataset across multiple domains and emotional dimensions (Part 1/5).

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Political Ideology (Nationalism)	Qwen3-8B-Base	+0.15	+0.28	+0.01	+0.41	0.21	0.15	0.15	0.17	0.15
	KLAAD	+0.30	+0.32	-0.03	+0.37	0.24	0.16	0.16	0.17	0.15
	Fairness mediator	+0.13	+0.31	-0.01	+0.41	0.22	0.15	0.15	0.17	0.15
	Bias Unlearning	+0.12	+0.30	-0.02	+0.40	0.21	0.15	0.15	0.16	0.15
	MASCRDM	+0.13	+0.29	-0.02	+0.41	0.21	0.15	0.15	0.16	0.15

Appendix

Table 7: Evaluation results of Qwen3-8B-Base, KLAAD, Fairness mediator, Bias Unlearning and MASCRDM on the BOLD dataset across multiple domains and emotional dimensions (Part 2/5).

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Profession (Film And Television Occupations)	Qwen3-8B-Base	+0.23	+0.42	-0.11	+0.28	0.24	0.14	0.14	0.15	0.14
	KLAAD	+0.49	+0.47	-0.16	+0.31	0.28	0.14	0.14	0.16	0.14
	Fairness mediator	+0.20	+0.41	-0.11	+0.27	0.23	0.14	0.14	0.15	0.14
	Bias Unlearning	+0.19	+0.42	-0.11	+0.27	0.24	0.14	0.14	0.15	0.14
	MASCRDM	+0.19	+0.42	-0.12	+0.26	0.23	0.14	0.14	0.16	0.14

Appendix

Table 8: Evaluation results of Qwen3-8B-Base, KLAAD, Fairness mediator, Bias Unlearning and MASCRDM on the BOLD dataset across multiple domains and emotional dimensions (Part 3/5).

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Profession (Theatre Personnel)	Qwen3-8B-Base	+0.24	+0.42	-0.10	+0.30	0.23	0.14	0.14	0.15	0.14
	KLAAD	+0.40	+0.47	-0.12	+0.29	0.27	0.14	0.14	0.16	0.14
	Fairness mediator	+0.23	+0.43	-0.09	+0.30	0.24	0.14	0.14	0.15	0.14
	Bias Unlearning	+0.23	+0.42	-0.10	+0.28	0.23	0.14	0.14	0.15	0.14
	MASCRDM	+0.22	+0.42	-0.09	+0.30	0.23	0.14	0.14	0.15	0.14

Appendix

Table 9: Evaluation results of Qwen3-8B-Base, KLAAD, Fairness mediator, Bias Unlearning and MASCRDM on the BOLD dataset across multiple domains and emotional dimensions (Part 4/5).

Type	Method	Sentiment	VAD			BE5
			V	A	D	Joy	Anger	Sadness	Fear	Disgust
Religious Ideology (Islam)	Qwen3-8B-Base	+0.11	+0.31	-0.15	+0.31	0.22	0.16	0.16	0.17	0.15
	KLAAD	+0.06	+0.19	-0.02	+0.30	0.26	0.19	0.19	0.21	0.18
	Fairness mediator	+0.11	+0.36	-0.11	+0.38	0.23	0.16	0.16	0.18	0.16
	Bias Unlearning	+0.12	+0.32	-0.12	+0.36	0.22	0.16	0.16	0.17	0.15
	MASCRDM	+0.08	+0.33	-0.17	+0.31	0.21	0.16	0.16	0.17	0.16

Appendix

Table 10: Evaluation results of Qwen3-8B-Base, KLAAD, Fairness mediator, Bias Unlearning and MASCRDM on the BOLD dataset across multiple domains and emotional dimensions (Part 5/5).

Explore similar work

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4Compliance Legal Domain
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiLarge Language Model Safety Guardrail
Wrynx Inc. · Independent Researcher
Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems
Jun 5, 2026Yiyang Zhao, Zhuo Zhang, Qingxuan Le +2Agentic Benchmarks Compliance
Fudan University Shanghai, China · Shanghai Academy of AI for Science Shanghai, China · Shanghai Artificial Intelligence Laboratory Shanghai, China +1

Sep 22, 2026, cs.AIJ/K move · Enter open · S save

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

Yan Zhang, Ruien Li, Yaoyao Peng, Wanxin Ren+3

Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences

Compliance Legal Domain

Open paper AskPDF arXiv

Large Language Models (LLMs) have been used in various industries. However, ensuring their compliance with complex laws and regulatory frameworks remains a great challenge. Existing evaluation paradigms mainly rely on static benchmarks that suffer from three severe limitations: First, the compliance rules being used do not comply with the requirements of Artificial Intelligence (AI) laws and regulations; Second, they only handle apparent, explicit compliance risks, leaving implicit and covert compliance risks undetected; Third, they fail to track the systematic propagation of risks along logical dependency chains or evaluate compliance within nuanced, context-based real-world scenarios. To bridge this critical gap, we introduce EADC, a novel advanced evaluation benchmark of LLMs based on an AI compliance knowledge graph and AI compliance legal experts. By mapping abstract legal rules into structured logical multi-relational graphs, our framework enables automated, evolving agents to distill and synthesize highly sophisticated adversarial scenarios. This compliance benchmark is reviewed and corrected by human AI legal experts throughout the whole process. The resulting dataset (4,435+ QA pairs) provides an extensive, multi-dimensional taxonomy covering critical regulatory frontiers, including bias and discrimination, fairness, personal privacy protection, and values. Crucially, our compliance dataset moves beyond shallow string-matching by incorporating contextual long-horizon interactions and logic-driven hazard chains, capturing deeply embedded compliance anomalies that bypass traditional filters. Experiment evaluations demonstrate that our framework exposes critical regulatory blind spots in state-of-the-art LLMs, offering a rigorous, AI laws and regulations-aligned benchmark to safeguard high-level and deep compliance in the application of LLMs.

Abstract

Explore similar work

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems