POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents

Yunju Kang; Seonghyeon Cho; Irene Li; Yeo-Chan Yoon; Chanjun Park

cs.AIOct 6, 2026

POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents

Authors: Yunju Kang, Seonghyeon Cho, Irene Li, Yeo-Chan Yoon, Chanjun Park

Organizations: Soongsil University · Tokyo University · Jeju National University

Abstract

LLM tool-use agents operate in dynamic environments where many actions carry operational risk. However, most safety mechanisms react only after errors manifest. Existing pre-emptive approaches either fine-tune the agent on chain-of-thought deliberation or compile natural-language guardrails into runtime checks, but they do so without exposing a structural, auditable verdict. We propose POLAR, a guardrail framework for small tool-calling agents that assesses reversibility through a structured two-layer ontology. POLAR assigns each action a graded reversibility score by deriving a candidate inverse sequence; calls failing a threshold are pruned before execution. Evaluated on $τ^2$ -bench across six agent models, POLAR improves mean task reward by 0.11 to 0.18 points on airline for four of six agents, but only eight of eighteen model--domain cells improve overall; retail and stronger agents often regress. POLAR provides an auditable structural check and characterizes its task-utility trade-offs. Reward is not a direct measure of prevented harm.

Figures & tables

Figure 1: Four safety patterns for tool-calling LLM agents. (a) Unguarded: the agent simply calls the tool. (b) Post-Hoc Risk Detection: the tool executes and a registered compensator may undo it after observation. (c) Proactive Agent with LLM-as-a-Judge: an LLM gates the call with an allow-or-block label and no structural basis. (d) POLAR (Ours): a structural reverse search over the typed action ontology (Layer A) and the actual state (Layer B) produces an inverse sequence $\pi$ and an auditable score, which a pluggable DecisionPolicy converts into the verdict.

Figure 2: Structural overview of the POLAR framework. In the evaluated wrapper, the candidate tool sequence is checked against the available current-state projection; a complete post-action state is not simulated, and the classifier and intent parser shown as modular components are not invoked at decision time.

tier	relation	per-step $\phi$	band
$T_{1}$	inverse_of $(T_{a},T_{b},\mathrm{Slot},w)$	$w$	reversible
$T_{2}$	bounded_reversibility $(T,\mathrm{Bound},w)$	$w\cdot\mathbf{1}[\mathrm{Bound}\ \text{at}\ s]$	reversible
$T_{3}$	compensates $(T_{a},T_{b},\mathrm{LossClass},w)$	$w\cdot(1-w_{\mathrm{loss}})$	compensable
$T_{4}$	irreversibly_consumes $(T,\mathrm{EntitySlot})$	$0$	irreversible
$T_{5}$	external_side_effect $(T,\mathrm{SinkClass})$	$0$	irreversible

Table 1: Reversibility tiers and per-step

\phi

score. Each Layer-A annotation carries a curator weight

w\in(0,1]

; the per-step

\phi

inherits

w

and, for

T_{3}

, the LossClass weight

w_{\mathrm{loss}}

. Tiers

T_{1}

–

T_{2}

form the reversible band;

T_{3}

is the compensable band (

0<\phi<1

);

T_{4}

–

T_{5}

are irreversible (

\phi=0

, fail-closed). The score measures annotated recoverability, not overall action safety or user authorization.

model	domain	baseline	InferAct	saga_lite	POLAR (Ours)
qwen3-8B	airline	0.351 $\pm$ 0.039	0.393 $\pm$ 0.040	0.440 $\pm$ 0.041	0.462 $\pm$ 0.041
	retail	0.213 $\pm$ 0.033	0.147 $\pm$ 0.029	0.111 $\pm$ 0.026	0.113 $\pm$ 0.026
	telecom	0.030 $\pm$ 0.015	0.076 $\pm$ 0.023	0.157 $\pm$ 0.033	0.068 $\pm$ 0.022
llama-3.1-8b	airline	0.291 $\pm$ 0.040	0.402 $\pm$ 0.054	0.403 $\pm$ 0.044	0.473 $\pm$ 0.047
	retail	0.047 $\pm$ 0.021	0.063 $\pm$ 0.023	0.067 $\pm$ 0.024	0.076 $\pm$ 0.026
	telecom	0.009 $\pm$ 0.009	0.085 $\pm$ 0.033	0.026 $\pm$ 0.018	0.062 $\pm$ 0.025

Table 2: Mean task reward

\pm

SE on

\tau^{2}

-bench across six agent models (150 attempted runs per cell; error-terminated runs excluded). Bold = best arm in row.

Figure 3: Mean task reward by domain, model, and safety arm (150 attempted runs each). Within each model group, the four bars are baseline , InferAct , saga_lite , POLAR (left to right).

model	airline	retail	telecom
qwen3-8B	$\mathbf{+0.111}$	$-0.100$	$\mathbf{+0.038}$
llama-3.1-8b	$\mathbf{+0.182}$	$\mathbf{+0.029}$	$\mathbf{+0.053}$
nemotron-9b	$\mathbf{+0.110}$	$-0.033$	$-0.053$
gemini-flash-lite	$\mathbf{+0.109}$	$-0.026$	$\phantom{+}0.000$
gpt-oss-120B	$-0.073$	$-0.242$	$\mathbf{+0.020}$
qwen3.5-27B	$-0.119$	$-0.209$	$-0.060$

Table 3: Difference of separately error-filtered arm means

\Delta_{\mathrm{POLAR}}=r_{\mathrm{POLAR}}-r_{\mathrm{baseline}}

across the

18

(model, domain) cells (150 attempted runs per arm; valid counts differ). Bold sign marks

\Delta>0

. A displayed

0.000

is a rounded difference, not evidence that both arm means are zero. Matched differences and effective pair counts are in Appendix D .

Figure 4: Descriptive per-domain fits to differences of arm means across eighteen model–domain cells. Dashed lines are OLS fits, with airline $\Delta=-0.76\,b+0.37$ ( $R^{2}=0.92$ ), retail $\Delta=-0.67\,b+0.06$ ( $R^{2}=0.98$ ), and telecom $\Delta=-0.31\,b+0.03$ ( $R^{2}=0.65$ ). These six-point fits are not predictive laws or deployment thresholds; matched estimates with uncertainty are in Appendix D .

Figure 5: Local anchor sensitivity for qwen3-8B : reported reward difference across the 58 sweep cells, with the main anchor marked. Maximum cell-to-anchor delta is $0.053$ , within the anchor’s standard-error band.

scope	$0.50$	$0.70$	$0.80$	$0.85$	$0.90$	$0.95$
all	100%	71.1%	70.7%	62.0%	50.6%	50.6%
airline	100%	36.6%	36.6%	17.6%	17.6%	17.6%
retail	100%	100%	100%	100%	48.9%	48.9%
telecom	100%	100%	98.7%	98.7%	98.7%	98.7%

Table 4: Fraction of 996 logged decisions with nonempty candidate pools for which at least one candidate meets each reversibility threshold in the instrumented xmodel_phi_logged run. The 979 already-approved paths in Table 8 come from different runs.

variant	$\Delta$	bootstrap 95% CI	perm. $p$
full (T1, T2, T3)	$0.080$	$[0.013,0.147]$	$0.036$
no T3 (T1, T2)	$0.073$	$[0.020,0.127]$	$0.013$
T1 + T3 only	$0.087$	$[0.027,0.147]$	$0.010$
T1 only	$0.087$	$[0.020,0.153]$	$0.017$
no irreversible	$0.060$	$[-0.007,0.127]$	$0.121$

Table 5: Tier ablation on qwen3-8B airline (150 attempted runs per arm). Each row uses its own baseline rerun; full in this ablation differs from the main experiment. no irreversible admits T4, T5 with

\phi=1

(no fail-closed). Archived stdout preserves rewards and summaries but not complete trial JSONL or exact configurations.

model	domain	tool	count
GPT-OSS-120B	retail	return_delivered_order_items	150
GPT-OSS-120B	retail	exchange_delivered_order_items	103
Nemotron-9B	airline	update_reservation_flights	77
Nemotron-9B	retail	exchange_delivered_order_items	69
Nemotron-9B	retail	return_delivered_order_items	69
GPT-OSS-120B	airline	update_reservation_flights	62

Table 6: Top blocked (model, domain, tool) triples across the six-agent run, ranked by block count. The corresponding bottleneck reasons ( no_candidate for airline, external_side_effect for retail) are discussed in the text and decomposed fully in Appendix B.3 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	Role	Identifier	Provider
Varying	Agent 1	qwen/qwen3-8b 1 1 1 https://huggingface.co/Qwen/Qwen3-8B	OpenRouter
	Agent 2	meta-llama/llama-3.1-8b-instruct 2 2 2 https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct	OpenRouter
	Agent 3	nvidia/nemotron-nano-9b-v2 3 3 3 https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2	OpenRouter
	Agent 4	google/gemini-2.5-flash-lite 4 4 4 https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-lite	OpenRouter
	Agent 5	openai/gpt-oss-120b 5 5 5 https://huggingface.co/openai/gpt-oss-120b	OpenRouter
	Agent 6	qwen/qwen3.5-27b 6 6 6 https://huggingface.co/Qwen/Qwen3.5-27B	OpenRouter

Appendix

Table 7: Models used in the

6\times 3\times 4

multi-model multi-trial evaluation. The six agents are the only varying axis; every other listed evaluated role is intended to be fixed across (agent, domain, arm) cells. Table 3 reports differences of arm means, not matched-pair estimates.

$\phi_{\mathrm{POLAR}}$	Count	Share	Annotation origin
$1.000$	$372$	$38.0\%$	$T_{1}$ inverse_of , $w=1$
$0.990$	$291$	$29.7\%$	$T_{2}$ bounded , $w=0.99$
$0.855$	$211$	$21.6\%$	$T_{3}$ compensates , low loss class
$0.810$	$105$	$10.7\%$	$T_{3}$ compensates , high loss class

Appendix

Table 8: Plan-level

\phi_{\mathrm{POLAR}}

over the

979

admitted plans. Because

\phi_{\mathrm{POLAR}}(\pi)=\min_{u\in\pi}\phi(u)

, the distribution collapses onto four discrete buckets, each with a structural origin in a Layer A tier and curator weight. The bucket boundaries match the tier boundaries of Table 1 .

Policy	Behavior	Comparative reward
Veto	blocks unreachable calls	main table only
Escalate	requests consent	not verified
Advisory	logs without blocking	not verified

Appendix

Table 9: Implemented policy behavior. Archived Escalate and Advisory reward means conflict with the submitted table; the trial JSONL and exact comparison baseline are unavailable, so cross-policy deltas and intervals are withheld.

Bottleneck reason	Count	Share
no_candidate	$910$	$52.1\%$
external_side_effect	$833$	$47.7\%$
depth_exceeded	$2$	$0.1\%$

Appendix

Table 10: Bottleneck-reason decomposition over all

1{,}745

blocked decisions. The

T_{5}

fail-closed pathway ( external_side_effect ) accounts for almost half of the block weight despite covering a small fraction of the action surface.

Domain	Blocked	Share
retail	$662$	$37.9\%$
airline	$569$	$32.6\%$
telecom	$514$	$29.5\%$

Appendix

Table 11: Per-domain blocking share. Retail leads, driven by VetoPolicy over-blocking on user-intent-dominated return and exchange tools (Section 4.6 ); these counts are not independently labeled as harmful or benign.

Model	Domain	baseline	InferAct	saga	POLAR
qwen3-8B	airline	2	0	0	5
	retail	0	0	6	0
	telecom	16	19	29	18
llama-3.1-8b	airline	23	68	26	38
	retail	44	39	45	45
	telecom	39	79	74	54

Appendix

Table 12: Per-cell error counts (out of

n=150

runs per cell). Two checkpoints served via lighter-weight OpenRouter routes ( llama-3.1-8b , gemini-flash-lite ) dominate the high-error cells; the two highest-reward agents ( gpt-oss-120B , qwen3.5-27B ) are near-zero.

Construct	Form	Meaning
Slot definition	slot_definition(entity, slot, type)	declare a typed slot on an entity
Loss class	loss_class_definition(name, weight= $w$ )	named cost dimension with anchor weight $w\in[0,1]$
Sink class	sink_definition(name, external=true\|false)	external sink for irreversible effects
$T_{1}$	inverse_of(a, b, slot, $w$ )	$a$ is the deterministic inverse of $b$ on slot
$T_{2}$	bounded_reversibility(a, bound, loss_class, $w$ )	$a$ self-undoes when bound holds at $s^{\prime}$
$T_{3}$	compensates(a, b, slot, loss_class, $w$ )	$a$ partially compensates $b$ , paying loss_class

Appendix

Table 13: Layer A annotation grammar. Tier

T_{1}

T_{5}

corresponds to the gradient of Table 1 ;

w

is the annotation’s anchor weight; loss_class is a name declared by loss_class_definition ; slot is a qualified entity.slot ; bound is a Layer B predicate expression ( TRUE for unconditional cases).

Relation	Per-edge $\phi$
inverse_of ( $T_{1}$ )	$w_{a}$
compensates ( $T_{3}$ )	$w_{a}\cdot(1-w_{\mathrm{loss}})$
bounded_reversibility ( $T_{2}$ )	$w_{a}$ if bound holds at $s^{\prime}$ , else $0$
irreversibly_consumes ( $T_{4}$ )	$0$
external_side_effect ( $T_{5}$ )	$0$

Appendix

Table 14: Per-edge

\phi

formulas.

w_{a}

is the annotation’s anchor weight;

w_{\mathrm{loss}}

is the weight of the matched loss_class_definition . Results are clamped to

[0,1]

defensively.

Class	Tokens
English affirmative	yes , y , yeah , yep , yup , sure , ok , okay , please , confirm(ed) , proceed , go ahead , do it , sounds good , agreed , approve(d)
English negation veto	not , never , cannot , can’t , don’t , won’t , n’t , no way , hold on , stop
Korean affirmative	ne (네), ye (예), joa-yo (좋아요), joh-seubnida (좋습니다), jin-haeng (진행), hwag-in (확인), seung-in (승인), dong-ui (동의), heo-rak (허락)

Appendix

Table 15: Default consent-detector token sets used by EscalatePolicy . Korean tokens are shown in Revised Romanization alongside their original Hangul glyphs. The Korean column has

9

affirmative tokens covering yes/proceed/confirm/agree/permit variants; the initial implementation does not include a Korean negation list.

Tool category	Diagnostic question	Annotation tier	Example
State-change, exact inverse exists	“Does another tool $T_{b}$ exactly undo $T_{a}$ on a slot?”	$T_{1}$ inverse_of	an exact-inverse tool pair on one typed slot
State-change, self-undoes under a bound	“Does $T_{a}$ revert its own effect when a Layer B condition holds at $s^{\prime}$ ?”	$T_{2}$ bounded_reversibility	update_reservation_baggages on the same reservation
State-change, compensable with named residue	“Does $T_{a}$ have an inverse that leaves a named cost (fee, audit trail, ordering shift)?”	$T_{3}$ compensates with loss_class_definition	update_reservation_flights paying rebooking_price_diff
State-change, consumes a non-replenishable resource	“Does $T_{a}$ strictly consume a domain resource without an external surface?”	$T_{4}$ irreversibly_consumes	single-use certificate_token
User-intent-dominated, external surface	“Is $T_{a}$ a delivery, outbound message, or hand-off to a human?”	$T_{5}$ external_side_effect with sink_definition	send_certificate (sink: email ), transfer_to_human_agents (sink: human_handoff )
Read-only	“Does $T_{a}$ mutate any typed slot or external sink?” (no)	no annotation	calculate_* , get_* , list_* , search_* , find_*

Appendix

Table 16: Per-tool authoring decision table. Apply rows top-to-bottom; the first matching row determines the annotation. Read-only tools match no row and carry no Layer A entry. The recommended domain default policy is Veto when state-change tools dominate the action surface, Escalate when user-explicit-intent tools dominate.

Model	Domain	Pairs	Matched $\Delta$	Task-cluster 95% CI
qwen3-8B	airline	143	$+0.119$	$[+0.057,+0.184]$
qwen3-8B	retail	150	$-0.100$	$[-0.187,-0.020]$
qwen3-8B	telecom	118	$+0.034$	$[+0.000,+0.077]$
llama-3.1-8b	airline	102	$+0.137$	$[+0.049,+0.230]$
llama-3.1-8b	retail	74	$+0.000$	$[+0.000,+0.000]$
llama-3.1-8b	telecom	70	$+0.057$	$[+0.014,+0.115]$

Appendix

Table 17: Matched task–trial reward differences. Each arm attempted 150 runs per cell; the intersection can be substantially smaller.

Explore similar work

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4Refusals Evaluation Benchmarks
Peking University · Beijing Academy of Artificial Intelligence
Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents
Jul 31, 2026Minghui Pan, Jiayuxuan Yang, Yuanyuan Yuan +2Risk Schema
Beijing University of Posts and Telecommunications · Beihang University · Tsinghua University
DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
Aug 6, 2026Wenhao Lin, Chenyu Yu, Xingwei Lin +6Runtime Safety Filtering Large Language Model Agents
Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University

Sep 27, 2026, cs.AIJ/K move · Enter open · S save

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents

Tianzhuo Yang, Zirui Mi, Yantao Huang, Guoxi Zhang+3

Peking University · Beijing Academy of Artificial Intelligence

Refusals Evaluation Benchmarks

Open paper AskPDF arXiv

Safety alignment for large language models (LLMs) in conversational settings is largely framed around whether to answer or refuse a request. In agentic settings, however, the same models must decide whether to act as permission-critical evidence emerges during execution. This creates a distinct challenge: apparent risk, action permissibility, and task competence are easily confounded, making agentic over-refusal difficult to distinguish from ordinary task failure. To address this, we introduce AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety. AgentBound transforms the same executable workflow by independently varying apparent risk and action permissibility, enabling controlled comparisons of risky-looking but authorized tasks and routine-looking but unauthorized tasks. These comparisons jointly diagnose over-refusal and unsafe compliance while controlling for task competence. We instantiate AgentBound as a human-validated 4,000-task evaluation suite with trajectory-based and post-state-based judgments. Across 17 model and harness configurations, high safety frequently coexists with poor authorized-task completion: GPT-5.5 blocks 99.5% of routine-looking unauthorized actions yet completes only 28.7% of risky-looking authorized tasks. We further train a lightweight runtime calibration module that improves authorized-task completion by 18.2% on average across 10 evaluated configurations, while improving unsafe-action blocking by 5.4% on average. These show that effective agentic alignment requires action decisions to track permission-relevant execution evidence, rather than refusal strength alone.

Abstract

Explore similar work

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model