Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

Holly Baker; Michael Kuo; Kyana Pollas

cs.CLOct 21, 2026

Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

Authors: Holly Baker, Michael Kuo, Kyana Pollas

Organizations: PolicingLab, Miami, Florida, United States

Abstract

Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.

Figures & tables

Figure 1: Policy Central, traced on one real benchmark pair (hp-0026, Appendix A ). One ingestion pipeline (A) writes three artifacts. Retrieve-Judge (B) reads only the vector index and judges each section’s top-

k

neighbours; for this pair, the partner section is the nearest neighbour. Screen-Confirm (C) reads the structural index and provisions store, never the vector index, and sweeps every section through a cheap-screen / expensive-confirm cascade. Every model output shown is the scored pipeline’s actual output, abbreviated with “…”.

Label	Definition	Example (from the corpus)
Conflict	Requirements that cannot both be satisfied.	“Must attend a two hour refresher training annually.” vs. “Must attend a four (4) hour refresher training annually.”
Redundancy	Same requirement; one section is deletable with no loss.	“In sensitive criminal matters, the Unit Commander shall be notified prior to the reopening of an investigation” , in two units’ SOPs.
Similarity	Related; a reader needs both, but neither subsumes the other.	“…kept secret unless …duty requires otherwise” vs. “At no time …release any information considered to be confidential.”
Different	No operationally relevant relation.	“Being armed in any manner.” vs. “Must be a minimum of 14 years of age.”

Table 1: The four relation labels, each with a verbatim excerpt pair from the pilot department’s corpus that the deployed system labelled correctly (provenance in Appendix A ).

	Screen-Confirm	Retrieve-Judge
Given the pair	Screen: the gold provision and 15 others from its document, against the gold section. Confirm: each hit.	Judge: the two gold sections.
Must find the pair	Screen: every anchor provision (mean 614, max 13,352) against every section of the compared document. Confirm: each hit.	Retrieve: nearest of 19,737 chunks by cosine ( $k{=}20$ in deployment; recall scored at 10). Judge: each retrieved pair.

Table 2: What each system is shown in the two settings of Table 3 .

Must find the pair
System	Confl.	Redun.	Simil.	Agg. †
	(26)	(57)	(44)	(127)
Retrieve-Judge, retrieval only	0.571	0.846	0.480	0.663
Retrieve-Judge, Sonnet 4.5 judge	0.418	0.668	0.207	0.457
Retrieve-Judge, Llama judge	0.462	0.757	0.262	0.525
Screen-Confirm, end-to-end ∗	0.615	0.561	0.273	0.472

Table 3: Recall by relation class. † Support-weighted recall over Conflict, Redundancy, Similarity (

n=127

); Different is excluded, since predicting Different for everything would score well on it while catching nothing. Upper block: the system must find the partner section. Retrieve-Judge rows are retrieval recall (R@10, over the 137 pairs whose quotes map to indexed leaf chunks, Section 4.2 ) alone, then retrieval times judge recall for two of eight judges (Appendix B ); ∗ measured on the production data path. Lower block: the system is handed the pair. Confl., Redun., Simil.: Conflict, Redundancy, Similarity.

Screen-Confirm, measured targeted scans
Scan	Calls	Cost	Time
SOP (54 prov.) vs. 2 SOPs, 885 sec.	885+216	$1.60	150 s
SOP (56 prov.) vs. 1 SOP, 100 sec.	100+66	$0.41	42 s
Screen-Confirm, projected
One SOP vs. library, 19.6k sec.	25–33k	$40–80	${\sim}$ 1 h
Retrieve-Judge, measured audit

Table 4: Cost and wall-clock of production scans on the deployment (ten scan workers). Calls are screen + confirm for Screen-Confirm and judge calls for Retrieve-Judge; times are from the run records; costs are Bedrock’s logged token counts for each run at list rate, cache reads at the cache rate. The projected row scales the measured per-section cost and time; it is an upper bound, not the intended use.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Benchmark construction. Claims are extracted per document and paired by shared terms with no model judgment; every pair is labelled by a non-Claude judge, each label is attacked by a separate model, and only disputed rows are hand-checked; the hand-checked label is final and replaces the judge’s.

Judge model	Aggregate recall
Sonnet 4.5	0.457
Sonnet 4.6	0.539 ∗
Sonnet 5	0.497
Llama 3.3 70B	0.525
GPT-5.6 Sol	0.574 ∗
GPT-5.6 Terra	0.548 ∗

Appendix

Table 5: Full eight-model Retrieve-Judge leaderboard (composed retrieval+judge aggregate recall, 302-pair benchmark). Sonnet 4.5, Sonnet 5, and Llama 3.3 70B are measured directly; cells marked ∗ are composed from each model’s measured per-class judge recall and the shared leaf-corrected per-class retrieval recall (Conflict 0.571, Redundancy 0.846, Similarity 0.480), support-weighted over Conflict/Redundancy/Similarity. The retrieval factor is common to every row, so it sets the band’s absolute level but not the model ordering.

Confirm model	Confl.	Redun.	Simil.	Diff.	Acc.
Sonnet 4.5 (production)	0.885	0.842	0.682	0.789	0.791
Sonnet 5	0.692	0.930	0.773	0.709	0.758
Opus 4.8	0.423	0.895	0.568	0.823	0.765
GPT-5.6 Terra	0.808	0.895	0.705	0.709	0.752
GPT-5.6 Luna	0.808	0.877	0.705	0.623	0.699
GPT-OSS 120B	0.731	0.912	0.727	0.543	0.656

Appendix

Table 6: Screen-Confirm with its confirm model swapped (Haiku screen fixed): per-class recall (including Different) and four-class accuracy on the 302 pairs. Conflict recall is not monotonic in general model strength; the Different column tracks each model’s ability to walk a false escalation back to Different, where Llama is weakest (0.474).

Class ( $n$ )	Absent	No prov.	Screen	Confirm	Found	Recall
Confl. (26)	0	0	6	4	16	16/26
Redun. (57)	3	3	8	11	32	32/51
Simil. (44)	1	0	17	14	12	12/43
All (127)	4	3	31	29	60	60/120

Appendix

Table 7: Where each of the 127 gold positives ended on the production data path, final run. Absent: its text is not in the index. No prov. (no provision): indexed, but distillation produced no provision for it. Screen, Confirm: lost at that stage. Found: labelled correctly. Recall: over pairs that reached the screen.

System	Cost	$/1k	Latency
Screen-Confirm	$1.95	$6.47	${\sim}$ 19 min §
Retrieve-Judge	$2.13 ‡	$7.04	${\sim}$ 28 min §
Llama (zero-shot)	$0.01	$0.03	${\sim}$ 2 min
DeBERTa NLI	$0.00	$0.00	${\sim}$ 2 min

Appendix

Table 8: Benchmark-scale cost and latency, sequential single worker, not production cost. Costs are summed from logged token counts at list rate. ‡ Retrieve-Judge output tokens estimated from a 40-pair sample; with prompt caching its cost falls to

{\sim}

$0.91. § Wall-clock from an earlier 376-pair run of the same pipelines; the 302-pair run did not log it.

Class ( $n$ )	Prec.	Recall	95% CI
Conflict (26)	0.404	0.885	[0.710, 0.960]
Redundancy (57)	1.000	0.842	[0.726, 0.915]
Similarity (44)	0.612	0.682	[0.534, 0.800]
Different (175)	0.932	0.789	[0.722, 0.843]

Appendix

Table 9: Screen-Confirm precision/recall per class, with 95% Wilson intervals on recall.

Policy Evaluation

Policy evaluation estimates the outcomes, value, risks, and fairness of decision rules before or after deployment.

See all 55 papers

PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines
Jun 30, 2026Sameer Malik, Ayush Singh, Amar Prakash AzadLegal Document Analysis Neuro-Symbolic Reasoning
1Fujitsu Research India Private Limited, Bengaluru, India
Which Rules Matter Now? Policy-Centroid Routing Before an Intelligent System Acts
Aug 31, 2026Thomson D. NguyAI Agent Governance
Radiant Institute for Manifold Studies
Didact: A Cross-Domain Capability Discovery System for Defence
Jun 5, 2026Aarya Bodhankar, Aditya Joshi, Bao Gia Doan +3Evidence Retrieval
University of New South Wales · Sydney, Australia · Australia +1

Jun 30, 2026J/K move · Enter open · S save

PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines

Sameer Malik, Ayush Singh, Amar Prakash Azad

1Fujitsu Research India Private Limited, Bengaluru, India

Legal Document Analysis Neuro-Symbolic Reasoning

Open paper AskPDF arXiv

Policy-grounded document review requires determining whether a target document complies with organization-specific policies, guidelines, or playbooks. While large language models can assist with policy interpretation and document analysis, end-to-end prompting leaves the applied policy logic implicit, making compliance decisions difficult to inspect, update, and test. We present PolicyGuard, a neuro-symbolic framework for policy-grounded document compliance review. PolicyGuard converts organizational policy guidance into an executable review engine consisting of typed relational logic rules and atom-level extraction questions. During review, LLMs answer these local questions using retrieved document evidence, and a symbolic evaluator applies the formal rules to detect non-compliance. We instantiate and evaluate PolicyGuard on company-specific NDA compliance review, where contract clauses must be checked against organization-specific negotiation policies. By separating policy formalization, local document interpretation, and symbolic compliance evaluation, PolicyGuard makes document review more explicit, maintainable, and systematically testable.

Abstract

Policy Evaluation

Explore similar work

PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines

Which Rules Matter Now? Policy-Centroid Routing Before an Intelligent System Acts

Didact: A Cross-Domain Capability Discovery System for Defence