CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

Suparno Roy Chowdhury; Manan Roy Choudhury; Dhruv Madhwal; Vivek Gupta

cs.CRSep 7, 2026

CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

Authors: Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta

Organizations: Arizona State University

Abstract

Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.

Figures & tables

Figure 1: Language models must jointly reason over structured attributes and free-text narratives in privacy-constrained hybrid records.

Figure 2: CIPHER construction pipeline. Source records combine structured attributes with narrative text. Paired question and executable SQL templates produce benchmark instances, which are then reviewed, revised when necessary, or removed.

Questions
	FIR	CFPB	MIMIC
# Questions	1,571	1,356	500
Avg. question length	13.7	24.4	20.0
Operation category (%)
Aggregation	25.5	25.0	20.2
Trend	15.8	16.3	18.2

Table 1: Table statistics for the benchmark in regards to question lengths, operation categories, records, and answer types.

Figure 3: Template-validation outcomes by domain

Reviewer pair	Cohen’s $\boldsymbol{\kappa}$	Jaccard
$\alpha_{1,2}$	0.93	0.93
$\alpha_{2,3}$	0.96	0.96
$\alpha_{3,1}$	0.94	0.94

Table 2: Cohen’s

\kappa

calculated among three NLP experts.

Figure 4: $M_{3}$ accuracy across models, methods, and domains under the support-controlled setting. Every input contains the records required to answer the question together with same-domain distractors.

Figure 5: Effect of surrogate restoration on instance-level

M_{3}

for Llama 3.3 70B. Each arrow runs from the score under native redacted evidence (open circle) to the score under surrogate-restored evidence (arrowhead). Green arrows point right, where restoration improves performance; red arrows point left, where the native redacted condition is stronger.

	CFPB			FIR
Method	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$
BlendSQL	30.8	42.6	35.8	33.2	33.4	33.3
CoT	42.8	41.2	42.0	23.9	29.0	26.2
LtM	54.5	32.5	40.7	35.8	43.0	39.1

Table 3: Long-table results with Gemini 2.0 Flash.

Figure 6: Question-level distribution of primary failure modes across operation categories. Percentages are computed over majority-failed questions in the 30% seed-42 sample; each question contributes one dominant failure label.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Gemini 2.5 Pro			Qwen 3-80B
		$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$
CFPB	BlendSQL	52.4%	77.0%	62.4%	41.3%	20.7%	27.6%
CFPB	CoT	51.0%	73.0%	60.0%	55.3%	43.0%	48.4%
FIR	BlendSQL	59.0%	87.0%	70.3%	29.3%	38.4%	33.2%
FIR	CoT	57.0%	86.0%	68.6%	23.8%	39.6%	29.7%

Appendix

Table 4: Gemini 2.5 Pro vs. Qwen 3. Best in-slot results are highlighted in bold. Results are on a 30% sample.

Dataset	Cat.	Method	Gemini 2.0 Flash			Qwen3-next			Llama 3.3			GPT-OSS-120B			GPT-OSS-20B
Dataset	Cat.	Method	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$
FIR	Prompting	CoT	48.5	63.6	43.5	44.4	46.8	34.3	40.2	48.9	31.0	39.9	48.4	34.0	46.5	58.0	34.7
		PoT	42.8	21.1	13.4	56.4	40.8	33.9	39.8	19.3	10.1	21.0	17.0	5.9	43.9	49.5	29.2
		LtM	24.0	16.8	9.6	61.6	60.6	48.6	43.2	47.2	32.8	39.6	45.6	33.3	44.2	48.1	28.5
		Zero-shot	15.3	30.3	11.1	9.4	18.4	5.6	17.5	23.9	9.8	16.6	24.4	12.0	44.4	49.9	30.2
		Few-shot	47.4	61.7	42.4	43.4	54.4	37.3	41.1	48.9	32.8	39.5	48.7	34.7	42.0	37.4	22.8

Appendix

Table 5: Results on CIPHER .

M_{1}

is token-overlap score,

M_{2}

is judge acceptance rate, and

M_{3}

is the mean instance-level harmonic score.

Family	Method	CFPB			FIR			MIMIC
Family	Method	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$	$M_{1}$	$M_{2}$	$M_{3}$
Prompting	CoT	50.4 ( +1.4 )	29.7 ( -17.7 )	21.3 ( -15.8 )	42.9 ( +2.7 )	43.6 ( -5.3 )	31.0 ( +0.0 )	32.0 ( +7.8 )	51.6 ( +3.0 )	34.2 ( +7.7 )
	PoT	47.6 ( +16.7 )	47.6 ( +11.2 )	31.7 ( +6.7 )	38.0 ( -1.8 )	35.8 ( +16.5 )	24.9 ( +14.8 )	31.3 ( +9.6 )	55.9 ( +11.5 )	29.6 ( +6.2 )
	LtM	57.6 ( +3.7 )	38.9 ( -2.2 )	31.1 ( -2.5 )	46.4 ( +3.2 )	40.9 ( -6.3 )	31.3 ( -1.5 )	49.8 ( +11.1 )	63.2 ( +13.5 )	46.6 ( +8.3 )
	Zero-shot	51.0 ( -1.3 )	33.2 ( -11.1 )	24.8 ( -10.1 )	43.5 ( +26.0 )	47.3 ( +23.4 )	33.0 ( +23.2 )	31.8 ( +8.0 )	51.4 ( +2.2 )	34.4 ( +7.7 )
	Few-shot	49.0 ( +1.7 )	30.3 ( -19.8 )	21.2 ( -16.2 )	42.4 ( +1.3 )	43.8 ( -5.1 )	31.0 ( -1.8 )	30.7 ( +7.0 )	52.4 ( +3.4 )	32.8 ( +7.0 )

Appendix

Table 6: Performance under surrogate-restored evidence on CIPHER , with redacted-to-restored deltas for Llama 3.3 70B.

Domain	# Pairs	Cohen’s $\kappa$
CFPB	250	0.91
FIR	250	0.88
MIMIC	250	0.93
Overall	750	0.91

Appendix

Table 7: Agreement between the human annotators and the LLM-as-a-Judge decisions, reported by domain.

Criterion	Agreement (%)
Fluency and naturalness	84.4
Semantic consistency	87.1
Matching intent	94.6

Appendix

Table 8: Agreement between the two reviewers on the quality of the 895 surrogate replacements, reported per criterion.

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 260 papers

RedactionBench
Jun 17, 2026Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali +2LLM Evaluation Contextual Integrity
A10 Networks, Inc.
SHIP: Structured and Hierarchical Insight with Privacy
Oct 15, 2026Kai Chen, Guisi Ni, Albert Cheu +4Differentially Private Synthetic Data Generation Privacy-Utility Tradeoff
University of Virginia · Google Research
DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
Jul 19, 2026Jun Nie, Zhiqin Yang, Zhenheng Tang +4AI Agent Reliability Deep Research Agents
Hong Kong Baptist University · University of Science and Technology of China · The Hong Kong University of Science and Technology

Jun 17, 2026J/K move · Enter open · S save

RedactionBench

Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali, Diptanshu Purwar+1

A10 Networks, Inc.

LLM Evaluation Contextual Integrity PII Detection

Open paper AskPDF arXiv

Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing benchmarks conflate extraction mechanics with privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small Language Models, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.

Abstract

Question Answering

Explore similar work

RedactionBench

SHIP: Structured and Hierarchical Insight with Privacy

DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments