ASIRF: An Agentic Framework for Context-Dependent Sensitive Information Redaction

Sudha Priyadarshini; Mohamed Chahine Ghanem

cs.AISep 24, 2026

ASIRF: An Agentic Framework for Context-Dependent Sensitive Information Redaction

Authors: Sudha Priyadarshini, Mohamed Chahine Ghanem

Organizations: University of Liverpool, UK · Keele University, UK

Abstract

Sensitive information is defined by domain and intent, not a universal category, yet redaction systems such as privacy filters and named-entity recognizers fix a taxonomy at training time, requiring retraining for each new domain. We introduce ASIRF (Agentic Sensitive Information Redaction Framework), which retrieves domain-specific definitions based on the input's domain from a flexible knowledge base at inference time, needing no retraining to adapt. Two architectures, a three-call multi-agent pipeline and a single-agent variant, are evaluated across ten small open-weight models and eight datasets, including out-of-distribution fictional domains, against the OpenAI Privacy Filter (OPF) as a trained-classifier baseline. With only a few dozen expert-authored definitions per domain and no training data, ASIRF's recall exceeds OPF's in 68 of 80 model-domain combinations (85 percent), by at least one of the two architectures, with shortfalls confined mostly to OPF's training-distribution domains.

Figures & tables

Figure 1: Two architectures of ASIRF - ASIRF-Multi as three-step pipeline (classify, retrieve, extract) and ASIRF-Single as a single agent call

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.801	0.761	0.936	0.821	0.922	0.972	0.908	0.911
qwen3-14b	0.610	0.629	0.732	0.626	0.726	0.900	0.783	0.807
gemma-3-12b	0.676	0.581	0.923	0.826	0.947	0.960	0.902	0.878
ministral-8b	0.664	0.626	0.806	0.777	0.869	0.951	0.832	0.846
qwen3-8b	0.621	0.694	0.787	0.705	0.698	0.891	0.754	0.791

Table 1: ASIRF-Multi recall by model and domain. The top-performing model exceeds OPF on every dataset.

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.699	0.671	0.930	0.791	0.930	0.972	0.891	0.905
qwen3-14b	0.595	0.452	0.864	0.727	0.849	0.962	0.793	0.765
gemma-3-12b	0.645	0.528	0.805	0.674	0.853	0.983	0.802	0.760
ministral-8b	0.623	0.494	0.827	0.713	0.779	0.972	0.776	0.781
qwen3-8b	0.684	0.660	0.943	0.805	0.916	0.979	0.840	0.872

Table 2: ASIRF-Single recall by model and domain. The top-performing model exceeds OPF on every dataset.

Figure 2: Recall, every model plus OPF, across all 8 datasets, Single-Agent and Multi-Agent side by side (same values as Tables 1–2). Datasets: Fin=Finance, Health=Healthcare, Cyber=Cybersecurity, PIIm=pii-masking, SPY=SPY, Cred=CredData, Arch=Archive of Forgotten Futures, Inter=Interstellar Immigration Bureau.

Model	ASIRF-Multi	No-RAG
ministral-14b	0.908	0.698
qwen3-14b	0.783	0.750
gemma-3-12b	0.902	0.877

Table 3: Recall on Archive of Forgotten Futures custom dataset, by model and architecture.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.508	0.602	0.690	0.882	0.494	0.710	0.627	0.605
qwen3-14b	0.612	0.750	0.844	0.914	0.501	0.828	0.785	0.827
gemma-3-12b	0.552	0.608	0.659	0.917	0.498	0.701	0.657	0.654
ministral-8b	0.539	0.592	0.812	0.900	0.495	0.795	0.696	0.671
qwen3-8b	0.609	0.707	0.801	0.911	0.478	0.815	0.728	0.740

Appendix

Table 4: ASIRF-Multi precision, by model and domain.

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.535	0.648	0.658	0.920	0.508	0.793	0.628	0.596
qwen3-14b	0.704	0.803	0.791	0.943	0.537	0.795	0.807	0.814
gemma-3-12b	0.628	0.746	0.831	0.903	0.501	0.731	0.791	0.741
ministral-8b	0.677	0.797	0.865	0.931	0.529	0.751	0.808	0.690
qwen3-8b	0.694	0.743	0.807	0.931	0.520	0.738	0.752	0.726

Appendix

Table 5: ASIRF-Single precision, by model and domain.

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.622	0.672	0.794	0.850	0.643	0.821	0.742	0.727
qwen3-14b	0.611	0.684	0.784	0.743	0.593	0.862	0.784	0.817
gemma-3-12b	0.608	0.594	0.769	0.869	0.653	0.810	0.760	0.750
ministral-8b	0.595	0.609	0.809	0.834	0.630	0.866	0.758	0.749
qwen3-8b	0.615	0.700	0.794	0.795	0.568	0.851	0.741	0.765

Appendix

Table 6: ASIRF-Multi F1, by model and domain.

	Real-world domains						Fictional domains
Model	Finance	Healthcare	Cybersecurity	pii-masking	SPY	CredData	Archive of Forgotten Futures	Interstellar Immigration Bureau
ministral-14b	0.606	0.659	0.771	0.851	0.657	0.874	0.737	0.719
qwen3-14b	0.645	0.578	0.826	0.821	0.658	0.871	0.800	0.789
gemma-3-12b	0.636	0.618	0.818	0.772	0.631	0.838	0.796	0.750
ministral-8b	0.649	0.610	0.846	0.808	0.630	0.848	0.792	0.733
qwen3-8b	0.689	0.699	0.870	0.863	0.663	0.841	0.794	0.792

Appendix

Table 7: ASIRF-Single F1, by model and domain.

Explore similar work

RedactionBench
Jun 17, 2026Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali +2Named-Entity Recognition
A10 Networks, Inc.
Strong Multilingual Privacy Tagging at Encoder Speed
Sep 29, 2026Jonathan GraehlNamed-Entity Recognition
RWS Language Weaver
PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents
Jun 16, 2026Aaditya PaiDocument Parsing Personalized Large Language Model Agents
Data Science Institute Columbia University

Jun 17, 2026, cs.CLJ/K move · Enter open · S save

RedactionBench

Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali, Diptanshu Purwar+1

A10 Networks, Inc.

Named-Entity Recognition

Open paper AskPDF arXiv

Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing benchmarks conflate extraction mechanics with privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small Language Models, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.

Abstract

Explore similar work

RedactionBench

Strong Multilingual Privacy Tagging at Encoder Speed

PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents