The A-R Behavioral Space: Execution-Level Profiling of Tool-Using Language Model Agents in Organizational Deployment

Safety alignment for large language models (LLMs) in conversational settings is largely framed around whether to answer or refuse a request. In agentic settings, however, the same models must decide whether to act as permission-critical evidence emerges during execution. This creates a distinct challenge: apparent risk, action permissibility, and task competence are easily confounded, making agentic over-refusal difficult to distinguish from ordinary task failure. To address this, we introduce AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety. AgentBound transforms the same executable workflow by independently varying apparent risk and action permissibility, enabling controlled comparisons of risky-looking but authorized tasks and routine-looking but unauthorized tasks. These comparisons jointly diagnose over-refusal and unsafe compliance while controlling for task competence. We instantiate AgentBound as a human-validated 4,000-task evaluation suite with trajectory-based and post-state-based judgments. Across 17 model and harness configurations, high safety frequently coexists with poor authorized-task completion: GPT-5.5 blocks 99.5% of routine-looking unauthorized actions yet completes only 28.7% of risky-looking authorized tasks. We further train a lightweight runtime calibration module that improves authorized-task completion by 18.2% on average across 10 evaluated configurations, while improving unsafe-action blocking by 5.4% on average. These show that effective agentic alignment requires action decisions to track permission-relevant execution evidence, rather than refusal strength alone.

LLM agents are increasingly taking actions with real consequences, yet safety evaluations often reduce their behavior to a final safe/unsafe outcome, obscuring whether the agent recognized the risk and how it responded. We introduce AURA-Eval, a framework for evaluating whether agents recognize contextual risk and adapt their behavior accordingly. AURA-Eval truncates tool-use trajectories at safety-critical decision points and asks the evaluated model to generate the next thought and action. It creates controlled scenario variations along six risk mechanisms: harm intensity, scale, target susceptibility, system dependency, oversight, and reversibility. It also creates paired scenarios that preserve the underlying goal but differ in whether safe fulfillment is available, allowing unsafe compliance and over-refusal to be evaluated together. An LLM-judge panel, validated against human annotations, evaluates each continuation along three axes: risk detection from the reasoning trace, and action type and safety from the action. From 157 R-Judge seed trajectories, we construct 1,249 evaluation items and evaluate 20 frontier, open-weight, and safety-tuned models. Across all 20 models, unsafe-action rates are higher when no safe path for request fulfillment is available than when one is; for frontier models, the rates are 27-40% versus 5-7%. When a safe path exists, frontier models rarely refuse; when none exists, they most often propose an alternative. Open-weight models, by contrast, more often execute the request than propose alternatives and act unsafely on 58-83% of items. Controlled risk-mechanism analyses further show particularly high unsafe rates under changes to oversight and scale; removing pre-execution oversight flips 52.3% of cases that were safe on the corresponding original scenario to unsafe.

Large language models (LLMs) are increasingly deployed as agents, yet it remains unclear how their foundational capabilities translate into agentic capabilities. End-to-end scores leave unresolved whether low benchmark capability or weak deployment drives a failure, limiting actionable diagnosis. We bridge foundational and agentic capabilities through an operation-level measurement framework. We represent each model with a foundational profile derived from capability benchmarks and an operation profile derived from interaction trajectories, and define capability realization gaps as deviations of observed operation performance from predictions based on foundational capabilities. Experiments on fifteen model services in $τ^2$-Bench show that foundational profiles only partially predict agentic operations. Models with comparable profiles can differ substantially in execution, and these gaps remain consistent across task subsets. Operation profiles add information and substantially improve task-score prediction. The framework locates execution departures at the operation level to inform model development and agent evaluation. We will release our code and analysis artifacts.