Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.
Figures & tables
Figure 1: Policy Central, traced on one real benchmark pair (hp-0026, Appendix A ). One ingestion pipeline (A) writes three artifacts. Retrieve-Judge (B) reads only the vector index and judges each section’s top- k neighbours; for this pair, the partner section is the nearest neighbour. Screen-Confirm (C) reads the structural index and provisions store, never the vector index, and sweeps every section through a cheap-screen / expensive-confirm cascade. Every model output shown is the scored pipeline’s actual output, abbreviated with “…”.
Label
Definition
Example (from the corpus)
Conflict
Requirements that cannot both be satisfied.
“Must attend a two hour refresher training annually.” vs. “Must attend a four (4) hour refresher training annually.”
Redundancy
Same requirement; one section is deletable with no loss.
“In sensitive criminal matters, the Unit Commander shall be notified prior to the reopening of an investigation” , in two units’ SOPs.
Similarity
Related; a reader needs both, but neither subsumes the other.
“…kept secret unless …duty requires otherwise” vs. “At no time …release any information considered to be confidential.”
Different
No operationally relevant relation.
“Being armed in any manner.” vs. “Must be a minimum of 14 years of age.”
Table 1: The four relation labels, each with a verbatim excerpt pair from the pilot department’s corpus that the deployed system labelled correctly (provenance in Appendix A ).
Screen-Confirm
Retrieve-Judge
Given the pair
Screen: the gold provision and 15 others from its document, against the gold section. Confirm: each hit.
Judge: the two gold sections.
Must find the pair
Screen: every anchor provision (mean 614, max 13,352) against every section of the compared document. Confirm: each hit.
Retrieve: nearest of 19,737 chunks by cosine ( k=20 in deployment; recall scored at 10). Judge: each retrieved pair.
Table 2: What each system is shown in the two settings of Table 3 .
System
Confl.
Redun.
Simil.
Agg. †
(26)
(57)
(44)
(127)
Must find the pair
Retrieve-Judge, retrieval only
0.571
0.846
0.480
0.663
Retrieve-Judge, Sonnet 4.5 judge
0.418
0.668
0.207
0.457
Retrieve-Judge, Llama judge
0.462
0.757
0.262
0.525
Screen-Confirm, end-to-end ∗
0.615
0.561
0.273
0.472
Table 3: Recall by relation class. † Support-weighted recall over Conflict, Redundancy, Similarity ( n=127 ); Different is excluded, since predicting Different for everything would score well on it while catching nothing. Upper block: the system must find the partner section. Retrieve-Judge rows are retrieval recall (R@10, over the 137 pairs whose quotes map to indexed leaf chunks, Section 4.2 ) alone, then retrieval times judge recall for two of eight judges (Appendix B ); ∗ measured on the production data path. Lower block: the system is handed the pair. Confl., Redun., Simil.: Conflict, Redundancy, Similarity.
Scan
Calls
Cost
Time
Screen-Confirm, measured targeted scans
SOP (54 prov.) vs. 2 SOPs, 885 sec.
885+216
$1.60
150 s
SOP (56 prov.) vs. 1 SOP, 100 sec.
100+66
$0.41
42 s
Screen-Confirm, projected
One SOP vs. library, 19.6k sec.
25–33k
$40–80
∼ 1 h
Retrieve-Judge, measured audit
Table 4: Cost and wall-clock of production scans on the deployment (ten scan workers). Calls are screen + confirm for Screen-Confirm and judge calls for Retrieve-Judge; times are from the run records; costs are Bedrock’s logged token counts for each run at list rate, cache reads at the cache rate. The projected row scales the measured per-section cost and time; it is an upper bound, not the intended use.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Benchmark construction. Claims are extracted per document and paired by shared terms with no model judgment; every pair is labelled by a non-Claude judge, each label is attacked by a separate model, and only disputed rows are hand-checked; the hand-checked label is final and replaces the judge’s.
Judge model
Aggregate recall
Sonnet 4.5
0.457
Sonnet 4.6
0.539 ∗
Sonnet 5
0.497
Llama 3.3 70B
0.525
GPT-5.6 Sol
0.574 ∗
GPT-5.6 Terra
0.548 ∗
Appendix
Table 5: Full eight-model Retrieve-Judge leaderboard (composed retrieval+judge aggregate recall, 302-pair benchmark). Sonnet 4.5, Sonnet 5, and Llama 3.3 70B are measured directly; cells marked ∗ are composed from each model’s measured per-class judge recall and the shared leaf-corrected per-class retrieval recall (Conflict 0.571, Redundancy 0.846, Similarity 0.480), support-weighted over Conflict/Redundancy/Similarity. The retrieval factor is common to every row, so it sets the band’s absolute level but not the model ordering.
Confirm model
Confl.
Redun.
Simil.
Diff.
Acc.
Sonnet 4.5 (production)
0.885
0.842
0.682
0.789
0.791
Sonnet 5
0.692
0.930
0.773
0.709
0.758
Opus 4.8
0.423
0.895
0.568
0.823
0.765
GPT-5.6 Terra
0.808
0.895
0.705
0.709
0.752
GPT-5.6 Luna
0.808
0.877
0.705
0.623
0.699
GPT-OSS 120B
0.731
0.912
0.727
0.543
0.656
Appendix
Table 6: Screen-Confirm with its confirm model swapped (Haiku screen fixed): per-class recall (including Different) and four-class accuracy on the 302 pairs. Conflict recall is not monotonic in general model strength; the Different column tracks each model’s ability to walk a false escalation back to Different, where Llama is weakest (0.474).
Class ( n )
Absent
No prov.
Screen
Confirm
Found
Recall
Confl. (26)
0
0
6
4
16
16/26
Redun. (57)
3
3
8
11
32
32/51
Simil. (44)
1
0
17
14
12
12/43
All (127)
4
3
31
29
60
60/120
Appendix
Table 7: Where each of the 127 gold positives ended on the production data path, final run. Absent: its text is not in the index. No prov. (no provision): indexed, but distillation produced no provision for it. Screen, Confirm: lost at that stage. Found: labelled correctly. Recall: over pairs that reached the screen.
System
Cost
$/1k
Latency
Screen-Confirm
$1.95
$6.47
∼ 19 min §
Retrieve-Judge
$2.13 ‡
$7.04
∼ 28 min §
Llama (zero-shot)
$0.01
$0.03
∼ 2 min
DeBERTa NLI
$0.00
$0.00
∼ 2 min
Appendix
Table 8: Benchmark-scale cost and latency, sequential single worker, not production cost. Costs are summed from logged token counts at list rate. ‡ Retrieve-Judge output tokens estimated from a 40-pair sample; with prompt caching its cost falls to ∼ $0.91. § Wall-clock from an earlier 376-pair run of the same pipelines; the 302-pair run did not log it.
Class ( n )
Prec.
Recall
95% CI
Conflict (26)
0.404
0.885
[0.710, 0.960]
Redundancy (57)
1.000
0.842
[0.726, 0.915]
Similarity (44)
0.612
0.682
[0.534, 0.800]
Different (175)
0.932
0.789
[0.722, 0.843]
Appendix
Table 9: Screen-Confirm precision/recall per class, with 95% Wilson intervals on recall.
Policy-grounded document review requires determining whether a target document complies with organization-specific policies, guidelines, or playbooks. While large language models can assist with policy interpretation and document analysis, end-to-end prompting leaves the applied policy logic implicit, making compliance decisions difficult to inspect, update, and test. We present PolicyGuard, a neuro-symbolic framework for policy-grounded document compliance review. PolicyGuard converts organizational policy guidance into an executable review engine consisting of typed relational logic rules and atom-level extraction questions. During review, LLMs answer these local questions using retrieved document evidence, and a symbolic evaluator applies the formal rules to detect non-compliance. We instantiate and evaluate PolicyGuard on company-specific NDA compliance review, where contract clauses must be checked against organization-specific negotiation policies. By separating policy formalization, local document interpretation, and symbolic compliance evaluation, PolicyGuard makes document review more explicit, maintainable, and systematically testable.
Sameer Malik, Ayush Singh, Amar Prakash Azad
1Fujitsu Research India Private Limited, Bengaluru, India
Before an intelligent system can decide whether an action is allowed, it must first know which rules the action has approached. A single proposed action can implicate several policy regimes at once. Their requirements may stack, overlap, or qualify one another, yet many remain written in natural language while the action itself arrives as an incomplete description of intent. The first problem is not judgment. It is attention. Policy-centroid routing creates a layer before adjudication. It compresses expressions within each policy regime into one or more representative centroids, places the proposed action in the same semantic space, applies a declared measure, and routes every regime crossing a declared threshold to authoritative review. Several regimes may trigger at once. The output is a review agenda, not permission, prohibition, legality, breach, compliance, certification, or enforcement. The paper develops six falsifiable propositions and seven follow-on studies comparing the hypothesis with structured workflows, lexical and semantic retrieval, hierarchical and direct classification, and selective prediction under matched review burden. The studies are designed to identify where policy geometry recovers applicable regimes, where compression loses rare or overlapping obligations, and where the mechanism should abstain. The paper includes a synthetic worked example and reports no empirical efficacy result.
Policymakers in defence and defence-aligned sectors must monitor rapidly evolving research alongside sector priorities relevant to operational and strategic needs. In practice, these sources are fragmented across heterogeneous formats, disjoint repositories, and siloed update streams, making capability discovery slow and difficult to audit. We present Didact, a prototype that integrates publicly available defence reports and policy documents from Australia with a purpose-built knowledge graph derived from Australian research publications. Didact provides natural language conversations for policy-oriented workflows, and leverages a composite retrieval-augmented generation (RAG) pipeline. A key feature of Didact is an interactive Evidence Rail that visualises retrieved evidence and source relationships. Our evaluation of the output quality and runtime of Didact highlights its utility. While Didact has been co-developed as an academia-industry project for the Australian context, it is adaptable to other domains where knowledge is similarly fragmented. A demonstration video is available here:
Aarya Bodhankar, Aditya Joshi, Bao Gia Doan +3
University of New South Wales · Sydney, Australia · Australia +1