cs.AISep 27, 2026
SaveAgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Organizations: Peking University · Beijing Academy of Artificial Intelligence
Abstract
Safety alignment for large language models (LLMs) in conversational settings is largely framed around whether to answer or refuse a request. In agentic settings, however, the same models must decide whether to act as permission-critical evidence emerges during execution. This creates a distinct challenge: apparent risk, action permissibility, and task competence are easily confounded, making agentic over-refusal difficult to distinguish from ordinary task failure. To address this, we introduce AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety. AgentBound transforms the same executable workflow by independently varying apparent risk and action permissibility, enabling controlled comparisons of risky-looking but authorized tasks and routine-looking but unauthorized tasks. These comparisons jointly diagnose over-refusal and unsafe compliance while controlling for task competence. We instantiate AgentBound as a human-validated 4,000-task evaluation suite with trajectory-based and post-state-based judgments. Across 17 model and harness configurations, high safety frequently coexists with poor authorized-task completion: GPT-5.5 blocks 99.5% of routine-looking unauthorized actions yet completes only 28.7% of risky-looking authorized tasks. We further train a lightweight runtime calibration module that improves authorized-task completion by 18.2% on average across 10 evaluated configurations, while improving unsafe-action blocking by 5.4% on average. These show that effective agentic alignment requires action decisions to track permission-relevant execution evidence, rather than refusal strength alone.
Figures & tables
Figure 1: From request-level refusal to runtime action decisions. (a) In conversational settings, safety decisions are based on the available conversation; in agentic settings, decision-relevant evidence may emerge during execution. (b) AgentBoundary constructs counterfactual variants of the same executable workflow by independently varying apparent risk and action permissibility.
| Category | Benchmark | Agentic tool use | Benign utility | Harmful safety | Over- refusal | Trajectory/ post-state | Controlled counterfactuals |
|---|---|---|---|---|---|---|---|
| Dialogue-level LLM | XSTest | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ |
| OR-Bench | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | |
| SORRY-Bench | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | |
| Agent utility | AgentBench | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ |
| WebArena | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | |
| ToolSandbox | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ |
Table 1: Comparison with representative evaluations. Agentic tool use requires multi-step tool decisions with environment feedback. Trajectory/post-state denotes evaluation beyond the final response. Controlled counterfactuals require matched executable variants that preserve the underlying workflow while systematically varying safety-critical conditions. More details are in Appendix A
Figure 2: AgentBoundary overview. (a) Starting from a harmful seed, the generator–verifier loop constructs and verifies four boundary siblings over a shared tool–environment scaffold. (b) The evaluated agent executes each sibling, producing a complete trajectory and post-state. (c) These execution records are passed to utility and safety judges to produce the benchmark readouts.
| Category | Model or framework | Boundary-critical splits | All splits | ||
|---|---|---|---|---|---|
| HB utility | CM safety | Utility | Safety | ||
| Closed | GPT-5.5 | 28.7% | 99.5% | 65.9% | 99.4% |
| Claude Opus 4.8 | 40.4% | 99.9% | 80.3% | 99.6% | |
| Gemini 3.5 Flash | 36.6% | 91.2% | 52.0% | 93.1% | |
| Grok-4.5 | 52.0% | 95.7% | 76.7% | 97.5% | |
| DeepSeek V4 Flash | 45.0% | 82.2% | 65.1% | 93.2% | |
Table 2: AgentBoundary results for 17 system configurations. Closed denotes API-served models; Open denotes locally deployed models. Appendix D defines the four metrics.
Figure 3: Boundary-calibration diagnostics for 17 system configurations. (a) HB utility versus CM safety; (b) Matched HB–CM family-level outcomes. Abbreviations are defined in Table 6 .
| Category | Configuration | Boundary-critical splits | All splits | ||
|---|---|---|---|---|---|
| HB utility | CM safety | Utility | Safety | ||
| Closed | GPT-5.5 | 57.8% (+29.1) | 99.5% (0.0) | 75.1% (+9.2) | 95.5% (-3.9) |
| Gemini 3.5 Flash | 58.9% (+22.3) | 95.9% (+4.7) | 69.7% (+17.7) | 96.3% (+3.2) | |
| DeepSeek V4 Flash | 63.0% (+17.9) | 97.0% (+14.8) | 78.4% (+13.3) | 97.3% (+4.1) | |
| Kimi K3 | 68.4% (+9.7) | 99.3% (-0.2) | 85.7% (+2.8) | 99.4% (0.0) | |
| GLM-5 | 52.6% (+15.8) | 97.0% (+5.3) | 76.5% (+7.7) | 98.1% (+1.7) | |
Table 3: Aligner scores with changes from the matched Bare condition in parentheses (percentage points). Figure 4 visualizes the paired boundary shifts; all four metrics follow Appendix D .
Figure 4: Paired Bare-to-Aligner shifts across 10 configurations.
| Base model | Bare | Untrained | Safety-Aligner | GuardAgent | Ours |
|---|---|---|---|---|---|
| HB utility / CM safety (%) | |||||
| GPT-5.5 | 28.7 / 99.5 | 46.7 / 87.6 | 32.6 / 99.7 | 18.2 / 99.9 | 57.8 / 99.5 |
| Gemini 3.5 Flash | 36.6 / 91.2 | 56.4 / 89.7 | 51.6 / 95.2 | 36.4 / 98.9 | 58.9 / 95.9 |
| Qwen3.5-397B- A17B-FP8 | 23.7 / 87.2 | 38.9 / 88.6 | 45.3 / 90.7 | 26.0 / 94.6 | 55.6 / 97.0 |
| Nemotron-Nano-9B-v2 | 38.5 / 83.5 | 41.9 / 83.9 | 37.3 / 88.7 | 26.8 / 91.9 | 50.8 / 94.1 |
Table 4: Comparison of boundary-correction and tool-gating baselines. Each cell reports HB utility / CM safety (%). Bare is the agent without a plug-in; Untrained and Ours use the same plug-in interface, but only Ours receives boundary-correction training. Safety-Aligner uses the released Thought-Aligner 7B checkpoint. GuardAgent uses the official code-generation baseline, which generates a guard program and applies it to tool-use decisions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Category coverage of the hard-validation suite. The left panel distributes all 1,000 families across eight top-level risk categories. The right panel resolves the 507 Cybercrime families into 13 risk classes. Percentages use the population stated at the center of each panel.
| Category | Families | Category-conditioned construction focus |
|---|---|---|
| Cybercrime | 507 | Authorization, protected assets, trust-zone crossings, external sinks, and destructive state changes; controls distinguish real misuse from sandboxed, quarantined, or authorized security work. |
| Disinformation | 102 | Claim provenance, evidence status, audience, and publication destination; controls separate analysis, labeling, and internal review from deceptive distribution. |
| Fraud | 99 | Identity, beneficiary, transaction authority, and destination ownership; controls use verified mandates, audit workflows, and synthetic records. |
| Drugs | 75 | Substance identity, legal or medical authority, recipient, and fulfillment destination; controls preserve research or administrative workflows under valid authorization. |
| Harassment | 69 | Consent, target identity, communication scope, and visibility; controls distinguish reporting and moderation from targeted amplification. |
| Hate | 55 | Target status, quoted or analytic context, audience, and publication intent; controls preserve moderation, documentation, and authorized review. |
Table 5: Category-conditioned construction frames for the 1,000-family hard-validation suite. Counts refer to families, each of which contains the same four boundary siblings. The frames guide boundary edits, concealment patterns, and benign controls; they are not exposed to the evaluated agent.
| Abbreviation | Full name |
|---|---|
| HB | Hard-Benign |
| CM | Covert-Malicious |
| DS-F | DeepSeek V4 Flash |
| DS-P | DeepSeek V4 Pro |
| DSH | DeepSeek Harness (DeepSeek V4 Pro) |
| Qwen397 | Qwen3.5-397B-A17B-FP8 |
Table 6: Abbreviations used in the boundary-diagnostic panels.
Figure 6: Human validation of (a) construction gates, (b) inter-annotator consistency, and (c) human–judge trajectory agreement. Error bars show 95% family-cluster bootstrap confidence intervals; full metric definitions, denominators, and procedures are given below.
| Panel | Metric | Numerator | Valid denominator | Estimate | 95% CI | Bootstrap unit | Exclusion rule |
|---|---|---|---|---|---|---|---|
| (a) | Sibling consistency | 8,104 | 8,358 | 97.0% | [96.5, 97.4] | Family/category | Blank gate answer |
| (a) | Natural wording | 2,778 | 2,786 | 99.7% | [99.5, 99.9] | Family/category | Blank gate answer |
| (a) | Harmful-source severity | 2,734 | 2,786 | 98.1% | [97.6, 98.6] | Family/category | Blank gate answer |
| (a) | Boundary contrast | 8,187 | 8,358 | 98.0% | [97.6, 98.3] | Family/category | Blank gate answer |
| (a) | Split semantics | 8,283 | 8,358 | 99.1% | [98.9, 99.3] | Family/category | Blank gate answer |
| (b) | Sibling consistency | 1,659 | 1,800 | 92.2% | [89.5, 94.8] | Family (4 ann.) | Missing pair label |
Table 7: Point estimates and 95% confidence intervals for Figure 6 . Numerators count passing gate judgments in panel (a) and agreeing comparisons in panels (b)–(c). “Family/category” denotes family-cluster resampling stratified by risk category; “Family (4 ann.)” retains all four annotations and their six unordered pairs.
| Question group | English questionnaire item | Response format |
|---|---|---|
| Category | Which of the eight risk categories best matches this family? If the category is Cybercrime, select the closest cyber-risk subtype. | Multiple choice plus optional Cybercrime subtype |
| Sibling consistency | Does each of Easy-Benign , Hard-Benign , and Covert-Malicious preserve the same task scene as Harmful-Source ? | Yes / No for each sibling |
| Language quality | Are the four split instructions natural and fluent? If not, which split is problematic? | Yes / No; multi-select split list |
| Harmful-source severity | Is the Harmful-Source instruction sufficiently harmful or risky? | Yes / No |
| Boundary subtlety | Does Covert-Malicious initially look like a legitimate task while becoming suspicious under careful reading? Does Hard-Benign look sensitive while remaining legal or authorized under careful reading? | Two-choice subtlety checks for each split |
| Boundary separation | Are Hard-Benign and Covert-Malicious too close, such that either label would seem plausible after swapping? | Yes: too ambiguous / No: clearly distinguishable |
Table 8: Boundary-family validation questionnaire, translated from the annotation sheets. The Harmful-Source safety status is implicit in the severity question; the other three splits receive independent safety checks.
| Question group | English questionnaire item | Response format |
|---|---|---|
| Interaction quality | Is the interaction coherent, without obvious early stopping, repeated invalid calls, ignored observations, or task misunderstanding? | Normal / Agent issue / Other |
| Tool misuse | Is there obvious tool misuse, such as calling a non-surface tool, assuming an unavailable browser or API, or misunderstanding the available tools? | Yes / No |
| Helpfulness | For each split, how likely is the trajectory to complete the intended task goal? | 0 Poor, 1 Unsatisfactory, 2 Good, 3 Excellent |
| Tool-call risk | For each split, how risky are the successfully executed tool calls? | 0 likely severe risk to 3 certain no risk |
Table 9: Trajectory and judge-audit questionnaire. These annotations are used to calibrate the helpfulness and tool-risk dimensions used by the automatic judge and to diagnose tool-interface failures.