ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance

Haoran Li; Yulin Chen; Huihao Jing; Wenbin Hu; Tsz Ho Li; Chanhou Lou; Hong Ting Tsang; Sirui Han

cs.CLApr 14, 2026

ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance

Authors: Haoran Li, Yulin Chen, Huihao Jing, Wenbin Hu, Tsz Ho Li, Chanhou Lou, Hong Ting Tsang, Sirui Han, +1 more

Organizations: Beihang University · HKUST · National University of Singapore · Faculty of Law, University of Macau

Abstract

Individuals' concerns about data privacy and AI safety are highly contextualized and extend beyond sensitive patterns. Addressing these issues requires reasoning about the context to identify and mitigate potential risks. Though researchers have widely explored using large language models (LLMs) as evaluators for contextualized safety and privacy assessments, these efforts typically assume the availability of complete and clear context, whereas real-world contexts tend to be ambiguous and incomplete. In this paper, we propose ContextLens, a semi-rule-based framework that leverages LLMs to ground the input context in the legal domain and explicitly identify both known and unknown factors for legal compliance. Instead of directly assessing safety outcomes, our ContextLens instructs LLMs to answer a set of crafted questions that span over applicability, general principles and detailed provisions to assess compliance with pre-defined priorities and rules. We conduct extensive experiments on existing compliance benchmarks that cover the General Data Protection Regulation (GDPR) and the EU AI Act. The results suggest that our ContextLens can significantly improve LLMs' compliance assessment and surpass existing baselines without any training. Additionally, our ContextLens can further identify the ambiguous and missing factors.

Figures & tables

Figure 1: An example of ContextLens ’ regulation chunking on the GDPR. We decompose the regulation following the chapter

\rightarrow

article

\rightarrow

item

\rightarrow

sub-item structure. We leverage LLM-as-a-judge to perform legal document chunking at the item and sub-item level with human annotator verification.

Domain	Permit	Prohibit	Not Applicable	Total
GDPR	150	478	0	628
AI Act	202	187	211	600

Table 1: Testing data statistics for legal compliance.

Evaluation Types	Evaluated Models	EU AI Act		GDPR		Average
Evaluation Types	Evaluated Models	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
Direct	Qwen2.5-7B-Instruct	46.33	41.89	88.37	85.36	67.35	63.63
	Llama-3.1-8B-Instruct	58.67	59.36	86.88	84.06	72.78	71.71
	OpenThinker-7B	70.50	69.72	87.26	81.82	78.88	75.77
	GPT-4o-mini	71.16	70.92	92.19	89.29	81.68	80.11
	GPT-4o	75.33	73.39	92.19	90.02	83.76	81.71

Table 2: Overall compliance evaluation results reported under %. Averaged differences between ContextLens and the best-performing baselines are reported in parentheses.

	GDPR		AI Act
	Ratio	Avg #	Ratio	Avg #
Llama-3.1-8B-Instruct	98.57	4.93	98.33	5.22
GPT-4o-mini	99.04	5.22	100.00	5.50
Gemini-2.5-Flash	100.00	7.48	100.00	5.73

Table 3: LLMs identified cases with imperfect context. “Ratio” measures the % of imperfect cases. “Avg #” denotes the average unknown factors per case.

	Article 5: Principles			Article 6: Lawfulness			Article 40,42: Obligations
	Correct	Wrong	$\Delta$	Correct	Wrong	$\Delta$	Correct	Wrong	$\Delta$
Llama-3.1-8B-Instruct	80.82	95.71	+14.89	75.44	74.28	-1.16	91.03	95.71	+4.68
GPT-4o-mini	90.46	100.00	+9.54	39.86	68.62	+28.76	90.12	100.00	+9.88
Gemini-2.5-Flash	99.23	100.00	+0.77	58.23	63.20	+4.97	95.21	97.16	+1.95

Table 4: Proportion of imperfect cases with unknown contextual factors on specific GDPR articles. “Correct” denotes the imperfect ratio among all correct predictions, while “Wrong” refers to the ratio of wrong predictions.

\Delta

= “Wrong” - “Correct”.

Case Context: Microsoft updated its Privacy Policy and communicated the changes via email and on its website. All users were informed, and the updates did not reduce user rights but clarified existing practices.

Ground Truth: Permitted

Gemini-2.5-Flash’s Structured Output :

{

Table 5: A case study examining a real scenario permitted under the GDPR.

Method	Dataset	Model	Input #	Response #
Direct	AI Act	GPT-4o	238.31	8.36
	AI Act	GPT-4o-mini	238.31	92.92
	GDPR	GPT-4o	149.17	7.45
	GDPR	GPT-4o-mini	149.17	68.62
ContextLens	AI Act	GPT-4o	522.29	195.30
	AI Act	GPT-4o-mini	522.29	194.56

Table 6: Token cost analysis.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	GDPR		AI Act
	Cohen	Fleiss	Cohen	Fleiss
Llama-3.1-8B-Instruct	100.00		95.87
GPT-4o-mini	100.00	21.25	96.63	79.93
Gemini-2.5-Flash	100.00		91.02

Appendix

Table 7: Evaluation on human-LLM and inter-LLM agreement. “Cohen” measures Cohen’s kappa coefficient between human evaluators and LLMs. “Fleiss” denotes the Fleiss’ kappa for inter-LLM agreement. All coefficients are reported with a fixed scaling factor of 100.

Multiple Choice Template Used for Direct Prompting

Determine the relationship between the provided event and the <domain> regulations. You have three options for your conclusion:

(A) The event is prohibited by <domain> .

(B) The event is permitted by <domain> .

Event: <event> Output Format: Choice: [A. Prohibited | B. Permitted | C. Not related ]

Appendix

Table 8: Evaluated multiple-choice prompt template for direct prompting. Light blue texts inside each “ <> ” block denote a string variable.

ContextLens’ Context Analysis Prompt Template for the EU AI Act

As an expert context analyzer, your task is to analyze the given case and identify if there is any AI system involved. If there is AI system involved, please identify the name, type, usage and actions of the AI system.

Definition of AI system:

An AI system is defined as: A machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.

Please follow these steps:

1. Analyze the case and identify if there is any AI system involved. If there is an AI system involved, please identify the name of the AI system, the type of the AI system, and the usage of the AI system.

2. If there is an AI system involved, please identify atomic actions that are performed by the AI system and the target of the action. If there is more than one action, please identify all of them.

Appendix

Table 9: Evaluated prompt templates for ContextLens. Texts inside each “ <> ” block denote a string variable.

Model	Default #	Original Acc	Updated Acc
Qwen2.5-7b-Instruct	22	85.19	85.64
Llama-3.1-8B-Instruct	8	88.85	89.19
GPT-4o	19	89.31	90.13
DeepSeek-R1 (671B)	26	80.89	83.22
o3-mini	24	85.35	86.42
Gemini-2.5-Flash	31	83.12	85.09

Appendix

Table 10: Model performance with and without permissive basis. Default # refers to the indeterminate cases with no permissive and prohibitive basis.

RAG Types	Models	EU AI Act	GDPR
LegalBERT	GPT-4o-mini	69.67	92.20
	GPT-4o	78.83	90.76
	o3-mini	34.33	69.11
BM25	GPT-4o-mini	74.33	91.24
	GPT-4o	77.75	90.92
	o3-mini	80.33	91.24

Appendix

Table 11: Accuracy evaluation with different RAG implementations.

Dataset	Model	Accuracy ( $\pm$ SD)	95% CI
AI_ACT	gpt-4o	$87.38\%\pm 0.25\%$	$[86.75,88.01]$
AI_ACT	gpt-4o-mini	$89.28\%\pm 0.25\%$	$[88.65,89.91]$
GDPR	gpt-4o	$86.25\%\pm 2.56\%$	$[79.89,92.61]$
GDPR	gpt-4o-mini	$91.88\%\pm 0.16\%$	$[91.48,92.27]$

Appendix

Table 12: Model Performance Comparison (Values represent Accuracy

\pm

Sample SD)

Case Context: A doctor operates a website to provide information and services to patients and potential patients. However, the website lacks a privacy policy, which is necessary to inform users about how their personal data is collected, used, and protected. This omission means that users are not informed about the handling of their personal data when they visit the website.

Ground Truth: prohibited

DeepSeek-R1’s Structured Output :

{

Appendix

Table 13: A case study examining a real scenario prohibited under the GDPR.

Explore similar work

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
Jan 24, 2025Guangzhi Sun, Xiao Zhan, Shutong Feng +2LLM Safety Benchmarks Language Model Safety Evaluation
Trinity College, University of Cambridge, Cambridge, United Kingdom · Department of Informatics, King’s College London, United Kingdom · Institut für Informatik, Heinrich-Heine-Universität Düsseldorf, Germany +1
Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code
Sep 16, 2026Jisoo Kim, Taeyoon Kwack, Jinwoo Jang +2LLM Grounding Legal Reasoning
Sungkyunkwan University Suwon, Republic of Korea

Sep 22, 2026J/K move · Enter open · S save

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

Yan Zhang, Ruien Li, Yaoyao Peng, Wanxin Ren+3

Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences

LLM Safety Benchmarks LLM Evaluation LLM Auditing

Open paper AskPDF arXiv

Large Language Models (LLMs) have been used in various industries. However, ensuring their compliance with complex laws and regulatory frameworks remains a great challenge. Existing evaluation paradigms mainly rely on static benchmarks that suffer from three severe limitations: First, the compliance rules being used do not comply with the requirements of Artificial Intelligence (AI) laws and regulations; Second, they only handle apparent, explicit compliance risks, leaving implicit and covert compliance risks undetected; Third, they fail to track the systematic propagation of risks along logical dependency chains or evaluate compliance within nuanced, context-based real-world scenarios. To bridge this critical gap, we introduce EADC, a novel advanced evaluation benchmark of LLMs based on an AI compliance knowledge graph and AI compliance legal experts. By mapping abstract legal rules into structured logical multi-relational graphs, our framework enables automated, evolving agents to distill and synthesize highly sophisticated adversarial scenarios. This compliance benchmark is reviewed and corrected by human AI legal experts throughout the whole process. The resulting dataset (4,435+ QA pairs) provides an extensive, multi-dimensional taxonomy covering critical regulatory frontiers, including bias and discrimination, fairness, personal privacy protection, and values. Crucially, our compliance dataset moves beyond shallow string-matching by incorporating contextual long-horizon interactions and logic-driven hazard chains, capturing deeply embedded compliance anomalies that bypass traditional filters. Experiment evaluations demonstrate that our framework exposes critical regulatory blind spots in state-of-the-art LLMs, offering a rigorous, AI laws and regulations-aligned benchmark to safeguard high-level and deep compliance in the application of LLMs.

Abstract

Explore similar work

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code