ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

Hainiu Xu; Vítor N. Lourenço; Mohnish Dubey; Yunfei Bai; Yulan He; Caroline Catmur; Aline Paes; Marco Caserta

cs.AIOct 2, 2026

ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

Authors: Hainiu Xu, Vítor N. Lourenço, Mohnish Dubey, Yunfei Bai, Yulan He, Caroline Catmur, Aline Paes, Marco Caserta,

Organizations: Amazon · King’s College London · The Alan Turing Institute · Universidade Federal Fluminense

Abstract

Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.

Figures & tables

Figure 1: Different roles asking the same query pursue different goals . A perspective-aware LLM agent ought to infer the goal of the given role and take actions admissible inside the role’s capability and knowledge boundary. In cases where the role does not have the remit to accomplish the goal, the agent must identify capable roles and route the work accordingly.

Panel of Table 1 — Table 1: Comparison of the features of against representative LLM agent benchmarks. ● Feature Supported ◐ Feature Partially Supported ○ Feature Not Supported.

Component	Role	Contents	Example
Declared once per domain
$\mathsf{Pred}$	Vocabulary	Predicates from which facts are built	status(device, faulty)
$\mathsf{Cap}$	Perspective axis	Capability gates: authority to act	repair , approve
$\mathsf{Know}$	Perspective axis	Knowledge domains: competence to interpret	hardware , scheduling
$\mathsf{Src}$	Retrieval targets	Knowledge bases a retrieval tool may read	manuals , ticket_log
$\mathsf{Fn}$	Tool taxonomy	The four kinds of thing a tool can do	retrieval , action

Table 2: Components of the hand-curated domain file, with illustrative example.

O : Operator T : Technician

C : Control Engineer P : Planner M : Manager

Table 3: Role gates: capability (authority to act) and knowledge (competence to interpret) assigned independently.

\bullet

granted,

\bullet

withheld.

Figure 2: Benchmark generation, from curated transcripts to executable per-role environments. Every stage pairs a model proposal with a deterministic check holding veto power. An effect the Domain File $\mathcal{D}$ cannot produce is dropped even under unanimous agreement. See Appendix E for elaboration on the dropping criteria.

	Query-Specific Tool Set (QTS)					Full Tool Set (FTS)
Model	P.R. $\uparrow$	P.C. $\uparrow$	O.R. $\downarrow$	U.R. $\downarrow$	Eff. $\uparrow$	P.R. $\uparrow$	P.C. $\uparrow$	O.R. $\downarrow$	U.R. $\downarrow$	Eff. $\uparrow$
with Function Calling
Open-Sourced Models
Qwen3
32B	$0.536_{\scriptscriptstyle\pm.024}$	$0.546_{\scriptscriptstyle\pm.017}$	$0.348_{\scriptscriptstyle\pm.038}$	$0.472_{\scriptscriptstyle\pm.055}$	$0.860$	$0.397_{\scriptscriptstyle\pm.046}$	$0.414_{\scriptscriptstyle\pm.044}$	$0.476_{\scriptscriptstyle\pm.031}$	$0.651_{\scriptscriptstyle\pm.039}$	$0.740$
80B-A3B	$0.388_{\scriptscriptstyle\pm.039}$	$0.472_{\scriptscriptstyle\pm.042}$	$0.588_{\scriptscriptstyle\pm.036}$	$0.567_{\scriptscriptstyle\pm.042}$	$0.581$	$0.152_{\scriptscriptstyle\pm.026}$	$0.305_{\scriptscriptstyle\pm.046}$	$0.414_{\scriptscriptstyle\pm.034}$	$0.439_{\scriptscriptstyle\pm.054}$	$0.320$

Table 4: Main results (Mean

{}_{\pm\text{Variance}}

) across two tool-set regimes (QTS, FTS) and two scaffolds (Function Calling, ReAct).

\uparrow

\downarrow

mark whether higher/lower is better; bold and underline give the best and runner-up within each scaffold block.

Figure 4: Consult-mode escalation collapses under distractor tools. Each knowledge-gate ( consult ) trajectory is scored as a correct consult (teal), an escalation abandoned before answering (amber), or a failure to escalate (red), across both scaffolds split by tool environment.

Figure 5: Perspective-awareness analysis. (a) Capability- vs. knowledge-gate violations. (b) Trajectory divergence vs. Pass against the reference divergence. (c) Per-persona Pass/Role/Over/Under for a proprietary and an open-weight model.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic		Count
Pairs (role $\times$ task)		150
Unique sessions		40
Role	Technician	38
	Control Engineer	34
	Operator	28
	Manager	28

Appendix

Table A1: Detailed statistics of the dataset.

Figure A1 : An illustration of the goal-annotation interface used by domain experts.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Sep 1, 2026

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios. To address these concerns, we present WorldBench: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions. WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators with language- and culture-specific expertise. For evaluation, we extend metrics from previous works and introduce Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and other complementary metrics through deterministic and LLM-as-a-Judge evaluations. Our experiments show that frontier models reach only 49.2% CTS, with all models demonstrating large gaps between correctness and environment preservation. We thereby show that current agents remain brittle in multilingual, agentic scenarios, especially for long-horizon tasks and under state-preservation constraints

Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1

(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

Cross-Cultural Language Model Evaluation LLM Agent Evaluation AI Agent Benchmarks Cultural Knowledge in Language Models Long-Horizon LLM Agents

Ask PDF

Apr 17, 2026

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.

Bhaskar Gurram

Zasti Inc., Ashburn, VA, USA

AI Agent Reliability Tool-Augmented Language Model Agents LLM-as-a-Judge LLM Agent Evaluation AI Agent Benchmarks LLM Reliability+2

Ask PDF

Jun 4, 2026

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a

2 \times 2

taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale

3.66\times

slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.

Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5

Shanghai AI Laboratory · East China Normal University · Soochow University +2

AI Agent Reliability AI Agent Benchmarks LLM Reliability LLM Tool Use LLM Agent Planning LLM Agent Reliability

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents