cs.AIOct 22, 2026

SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs

Authors: Arnon Mazza, Elad Levi

Organizations: Plurai

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories. Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. Programs optimized with the weakest executor transfer to other models without further optimization. This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. Additionally, we validate the key components of our method through an ablation study. SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.

Figures & tables

Figure 1: (1) Given a natural-language policy and a small pool of seed samples, (2) the optimizer compiles the policy into (3) a decomposition program: predicates answered by weak models, deterministic extractors, and Python code that composes their outputs into a verdict. (4) Adversarial generators extend the training pool with counterexamples where the program departs from the policy, at the level of a single predicate (local) or the composition as a whole (systemic). The loop repeats, optimizing the program and its training data jointly.
TaskInputAvg. lengthLabel spaceCore challengeSize
CoachingMessage, critic output260Pass / Fail / BugNested audit174
EscalationMulti-turn dialogue1.6kTrue / FalseCross-turn aggregation371
PlanFull agent trace45kPass / FailSparse evidence163
Tile-matchConversation, tile list630Dynamic, multi-labelDistractor candidates163
Table 1: Policy-enforcement benchmark of four tasks from real-world agentic systems. The tasks span input types, lengths, and label spaces, and each poses a distinct core challenge.
Figure 2: Test accuracy by executor model and method, one panel per task. Values are from Table 6 . Executors are ordered from weakest to strongest. CheckEval and RLIE are evaluated on Escalation and Plan only.
MethodCoachingEscalationPlanTile-match
Direct prompting0.60 ±\pm 0.030.56 ±\pm 0.030.59 ±\pm 0.030.38 ±\pm 0.03
Prompt optimization
GEPA0.58 ±\pm 0.010.61 ±\pm 0.010.66 ±\pm 0.020.55 ±\pm 0.06
MIPROv20.69 ±\pm 0.010.57 ±\pm 0.020.58 ±\pm 0.030.56 ±\pm 0.05
Workflow generation
AFlow0.83 ±\pm 0.020.86 ±\pm 0.030.75 ±\pm 0.080.68 ±\pm 0.09
Table 2: Test accuracy with gpt-5.4-nano as the executor (mean Jaccard for Tile-match), averaged over three optimization runs, each evaluated three times, with standard deviations. Scope outperforms all prompt-optimization and decomposition baselines on every task, and is the only method that remains strong across all four. Best in bold , second underlined .
Figure 3: Effect of executor capability on the decomposition. (a) Accuracy gained when moving from the weakest to the strongest executor is substantially less for Scope than for prompt optimization (average gain over tasks: 0.09 vs. 0.26). (b) Number of predicates in the final decomposition (median over repetitions). Executors are ordered from weakest to strongest by prompt-optimized accuracy, an ordering that holds on all four tasks.
ExecutorCoachingEscalationPlanTile-match
DeepSeek-V4.1-Flash (reasoning disabled)0.78 ↑\uparrow 0.020.91 ↑\uparrow 0.190.86 ↑\uparrow 0.330.89 ↑\uparrow 0.34
Qwen3-235B-A22B-Instruct-25070.90 ↑\uparrow 0.270.78 ↑\uparrow 0.040.86 ↓\downarrow 0.040.86 ↑\uparrow 0.37
Qwen3-30B-A3B-Instruct-25070.87 ↑\uparrow 0.040.78 ↑\uparrow 0.150.83 ↑\uparrow 0.120.81 ↑\uparrow 0.37
Gemma-3-27B-it0.86 ↑\uparrow 0.080.72 ↑\uparrow 0.010.75 ↑\uparrow 0.120.77 ↑\uparrow 0.29
Nemotron-3.5-Lightning0.79 ↑\uparrow 0.180.72 ↑\uparrow 0.140.69 ↑\uparrow 0.070.67 ↑\uparrow 0.30
gpt-5.5 (high, undecomposed)0.940.890.890.82
Table 3: Accuracy of the decompositions found with gpt-5.4-nano, executed by five open-weight models, without further optimization, and with reasoning disabled. Arrows show the gain over the same model on the undecomposed task. As a baseline, two high-end reasoning models run on the undecomposed task: gpt-5.5 with high reasoning effort, and gemini-3.1-pro with dynamic thinking.
VariantCoachingEscalationPlanTile-match
No adversary0.7830.8160.8130.744
Full0.8390.8150.8650.782
Table 4: Ablation of adversarial generation with gpt-5.4-nano as the executor, averaged over five optimization runs. Bold : significant improvement ( p<0.05p<0.05 , Welch’s tt -test).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Decomposition example for the Coaching task.
 Cost per 1000 samples ($) 
TaskDecomposedgemini-3.1-proRatio ( ×\times )
Coaching0.806.889
Escalation0.2310.7146
Plan1.4921.9915
Tile-match1.0850.7947
Appendix
Table 5: Comparison of inference costs of decompositions run by an inexpensive executor and of gemini-3.1-pro on the undecomposed tasks. Ratios computed before rounding.
ModelMethodCoachingEscalationPlanTile-match
gpt-5.4-nanoDirect prompting0.600.560.590.38
Prompt optimization
GEPA0.580.610.660.55
MIPROv20.690.570.580.56
Task decomposition
CheckEvalN/A0.580.54N/A
Appendix
Table 6: Test accuracy on four classification tasks under three executor models, configured to run with no reasoning. For the Coaching, Escalation and Plan tasks the accuracy is computed by the percentage of correct predictions. For the Tile-match task, the accuracy is computed as the mean-jaccard index between the correct labels and the predicted labels, over all samples. All reported numbers are based on 9 measurements, averaged over three independent runs of the optimization process and three repetitions of the test evaluation. The final numbers are rounded to two decimal places.
More in this topic

LLM Safety Monitoring

See all 54 papers

Explore similar work

CardsList
Aug 16, 2026

PL-Guard: Probabilistic Logic Reasoning for LLM Guardrails

Large language model guardrails can be viewed as policy-consistency problems: a system must determine which policy-relevant facts hold in a prompt-response pair and what those facts imply under a given policy. Common approaches, including policy prompting and LLM-as-a-judge pipelines, often overlap the tasks of semantic grounding and policy reasoning: the model both interprets the prompt-response pair and reasons about whether a policy has been violated. This can lead to unsafe compliance with harmful prompts, or refusals to assist benign ones. To separate grounding and reasoning roles, we propose PL-Guard, a neurosymbolic guardrail architecture. Using a symbolic policy interface consisting of predicates and ProbLog rules, a local LLM grounds prompt-response pairs into predicate probabilities using renormalized True/False token scores, while ProbLog performs explicit probabilistic rule inference over the symbolic policy. On the XSTest benchmark, an offline Qwen-based evaluator finds that PL-Guard with a hand-curated policy reduces unsafe compliance from 22.0% for the base model to 0.5%, and below the 6.0% rate of an LLM-as-a-judge baseline. This comes at the cost of higher over-refusal than the LLM-as-a-judge baseline, 14.4% versus 5.2%. These results suggest that separating neural grounding from probabilistic symbolic reasoning can expose the safety-helpfulness tradeoff while making the guardrail's intermediate reasoning steps explicit and auditable.

Satchit Chatterji, Shihan Wang, Giovanni Sileno +1

University of Amsterdam · Utrecht University

LLM GuardrailsLanguage Model Safety EvaluationNeuro-Symbolic Reasoning
AskPDF
Oct 8, 2026

NOMOS: Compiling Written Policies into Statically Verified Tool-Call Gates for LLM Agents

Tool-using LLM agents violate the policies they are deployed to enforce, often silently. Prior defenses hand-write rules, query an LLM verifier per action, or compile policies through heavyweight formal machinery. Naive compilation fails: extracted rules block the tool satisfying their own precondition, or read arguments their tool lacks. NOMOS, a four-pass compiler, turns a natural-language policy into a deterministic tool-call gate; static verification with tool-schema-level checks alone (no prover, solver, or LLM) repairs or rejects 37% (airline) and 13% (retail) of candidates, without which most shipped rules are inoperable. Replaying compiled rules over undefended transcripts flags bindings that refuse legitimate work (a development binding refused 95.9% of task-passing calls); no evaluation binding is flagged. On τ2τ^2-bench the gate cuts violations of reference-encoded clauses among state-changing calls from 66.3% to 2.6% (airline) and 30.8% to 6.9% (retail), raising airline task success significantly for 2≤k≤42 \le k \le 4; a 26B on-premise compilation is not significantly worse than hand-written or frontier-compiled rules. Unlike AgentDojo's shipped defenses, it reaches a zero attack success rate (ASR) on banking, where nine attack families collapse onto three structural rules. On the other three suites its ASR is at most 3.6%, from goals with no tool call to govern and one write admitted by a binding weaker than its clause; a second agent model, Llama-3.3-70B, reproduces the effect on both benchmarks. Decisions take microseconds without an LLM call, at a domain-dependent benign-utility cost; compilation runs on-premise on open-weight gemma-4-26B.

Min-Young Yu, Tony Kim, Jang Won Choi

Corners Co., Ltd.

Tool Access Control for LLM AgentsLLM Agent SafetyRuntime Enforcement for AI AgentsFormal Verification
AskPDF
Jun 30, 2026

PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines

Policy-grounded document review requires determining whether a target document complies with organization-specific policies, guidelines, or playbooks. While large language models can assist with policy interpretation and document analysis, end-to-end prompting leaves the applied policy logic implicit, making compliance decisions difficult to inspect, update, and test. We present PolicyGuard, a neuro-symbolic framework for policy-grounded document compliance review. PolicyGuard converts organizational policy guidance into an executable review engine consisting of typed relational logic rules and atom-level extraction questions. During review, LLMs answer these local questions using retrieved document evidence, and a symbolic evaluator applies the formal rules to detect non-compliance. We instantiate and evaluate PolicyGuard on company-specific NDA compliance review, where contract clauses must be checked against organization-specific negotiation policies. By separating policy formalization, local document interpretation, and symbolic compliance evaluation, PolicyGuard makes document review more explicit, maintainable, and systematically testable.

Sameer Malik, Ayush Singh, Amar Prakash Azad

1Fujitsu Research India Private Limited, Bengaluru, India

Legal Document AnalysisNeuro-Symbolic Reasoning
AskPDF