ARCCS: An Automated Regulatory Compliance Checking System

Giorgos Filandrianos; José Menezes; Chrysoula Zerva; Alessandro Gianola

cs.CLOct 1, 2026

ARCCS: An Automated Regulatory Compliance Checking System

Authors: Giorgos Filandrianos, José Menezes, Chrysoula Zerva, Alessandro Gianola

Organizations: Instituto de Telecomunicações · AILS Lab; ECE, National Technical University of Athens · Instituto de Telecomunicações/IST, Universidade de Lisboa · INESC-ID/IST, Universidade de Lisboa

Abstract

Regulatory compliance checking - deciding whether a target document satisfies the obligations of a regulation - requires interpreting dense legal text, identifying which provisions apply, and grounding each decision in explicit evidence. We present ARCCS, an end-to-end, automated, agentic, and regulation-agnostic Legal NLP system for compliance checking. ARCCS decomposes raw regulatory text into atomic, traceable requirements and evaluates a target document against them using retrieved evidence, confidence scores, and human-interpretable justifications. This design decouples compliance assessment from any fixed regulatory template or predefined rule set, enabling the pipeline to operate over regulations of varying size and structure. We evaluate ARCCS in two complementary settings. First, in a GDPR policy-document evaluation, LLM-based judges find its decisions and justifications legally and evidentially consistent in up to 96.67% of the assessed cases. Second, on an EU public-procurement benchmark comprising more than 1,200 individual rule checks, the system attains 98.8% accuracy in violation detection. ARCCS is, to our knowledge, the first fully open-source system for end-to-end regulatory compliance checking and auditable report generation.

Figures & tables

Figure 1: Overview of the proposed regulatory compliance assessment pipeline.

Evaluator (LLM-as-a-Judge)	Accuracy (%)
gpt-5.1	96.67
gpt-5.2	90.00
gpt-5.2-pro	90.00

Table 1: Accuracy of model-based evaluators on the sampled set (30 checks per terms-of-use doc.; 90 total).

Agreement	$\kappa$	$p$ -value
Cohen’s $\kappa$ (5.1 vs 5.2)	0.47	0.19
Cohen’s $\kappa$ (5.1 vs 5.2-pro)	0.47	0.19
Cohen’s $\kappa$ (5.2 vs 5.2-pro)	1.00	$<0.001$
Fleiss’ $\kappa$ (3 raters)	0.69	$<0.001$

Table 2: Inter-evaluator agreement on the binary verdict ( yes / no ) for the sampled set.

Model	Acc.	Prec.	Rec.	$F_{1}$
gpt-5.4	98.5	83.2	98.8	90.3
gpt-5.2	98.8	85.9	100.0	92.4
gpt-5-mini	98.8	85.9	100.0	92.4

Table 3: Rule-level performance on the procurement benchmark (100 traces, 1,200 rule decisions per model).

Model	Acc.	Prec.	Rec.	$F_{1}$
gpt-5.4	98.0	96.8	100.0	98.4
gpt-5.2	100.0	100.0	100.0	100.0
gpt-5-mini	100.0	100.0	100.0	100.0

Table 4: Case-level (per-trace) performance on the procurement benchmark.

Model	Exact Match (%)
gpt-5.4	83.0
gpt-5.2	86.0
gpt-5-mini	87.0

Table 5: Document-level exact-match performance: the percentage of traces for which all 12 rule decisions are simultaneously correct.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: The ARCCS web demonstration interface, designed for users without programming experience. The workflow takes a regulation and a target document as input and returns an auditable, evidence-linked compliance report.

Rule	FP	Prec.	Rec.
R06 – award within 70 days	7	0.42	1.00
R09 – end-after-pub. lifecycle	5	0.50	1.00
R05 – pub./part. require award	2	0.83	1.00
Other 9 rules (each)	0	1.00	1.00

Appendix

Table 6: Per-rule error breakdown on the procurement benchmark ( gpt-5-mini ; 100 cases/rule). All residual errors are false positives on three temporal rules; no rule produces a false negative.

Model	Snapshot
gpt-5.2	gpt-5.2-2025-12-11
gpt-5.1	gpt-5.1-2025-11-13
gpt-5.2-pro	gpt-5.2-pro-2025-12-11
gpt-5.4	gpt-5.4-2026-03-05
gpt-5-mini	gpt-5-mini-2025-08-07

Appendix

Table 7: Exact model snapshots used in the experiments.

Document	Chars	Words	Sent.	Paras	Lines
WhatsApp	323,682	51,698	1,862	1,382	3,723
Netflix	15,777	2,538	128	39	158
ChatGPT	20,404	3,324	171	100	197

Appendix

Table 8: Terms-of-use document sizes used in our experiments.

Document	Comp.	Non-C.	Insuf.	Human
WhatsApp	2	2	89	7
Netflix	0	0	99	1
ChatGPT	0	4	94	2

Appendix

Table 9: Compliance-label distribution over the 100 GDPR-derived requirements for each evaluated terms-of-use document.

Explore similar work

ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detection
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuCompliance Regulation
The University of Hong Kong · Stellaris AI Limited
CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs
Aug 3, 2026Muhammad Roman, Karen Rafferty, Barry DevereuxCompliance AI Trustworthiness
Bristol Research and Innovation Laboratory (BRIL), Toshiba Europe Ltd., Bristol, United Kingdom · Queen’s University Belfast, United Kingdom
Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code
Sep 16, 2026Jisoo Kim, Taeyoon Kwack, Jinwoo Jang +2Compliance Legal Reasoning Tasks
Sungkyunkwan University Suwon, Republic of Korea

Apr 26, 2026, cs.CLJ/K move · Enter open · S save

ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detection

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong · Stellaris AI Limited

Compliance Regulation Banking

Open paper AskPDF arXiv

Financial institutions must track over 60,000 regulatory events annually, overwhelming manual compliance teams; the industry has paid over USD 300 billion in fines and settlements since the 2008 financial crisis. We present ComplianceNLP, an end-to-end system that automatically monitors regulatory changes, extracts structured obligations, and identifies compliance gaps against institutional policies. The system integrates three components: (1) a knowledge-graph-augmented RAG pipeline grounding generations in a regulatory knowledge graph of 12,847 provisions across SEC, MiFID II, and Basel III; (2) multi-task obligation extraction combining NER, deontic classification, and cross-reference resolution over a shared LEGAL-BERT encoder; and (3) compliance gap analysis that maps obligations to internal policies with severity-aware scoring. On our benchmark, ComplianceNLP achieves 87.7 F1 on gap detection, outperforming GPT-4o+RAG by +3.5 F1, with 94.2% grounding accuracy (

r=0.83

vs. human judgments) and 83.4 F1 under realistic end-to-end error propagation. Ablations show that knowledge-graph re-ranking contributes the largest marginal gain (+4.6 F1), confirming that structural regulatory knowledge is critical for cross-reference-heavy tasks. Domain-specific knowledge distillation (70B

\to

8B) combined with Medusa speculative decoding yields

2.8\times

inference speedup; regulatory text's low entropy (

H=2.31

bits vs.

3.87

general text) produces 91.3% draft-token acceptance rates. In four months of parallel-run deployment processing 9,847 updates at a financial institution, the system achieved 96.0% estimated recall and 90.7% precision, with a

3.1\times

sustained analyst efficiency gain. We report deployment lessons on trust calibration, GRC integration, and distributional shift monitoring for regulated-domain NLP.

Abstract

Explore similar work

ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detection

CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs

Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code