ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

Hainiu Xu; Vítor N. Lourenço; Mohnish Dubey; Yunfei Bai; Yulan He; Caroline Catmur; Aline Paes; Marco Caserta

cs.AIOct 2, 2026

ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

Authors: Hainiu Xu, Vítor N. Lourenço, Mohnish Dubey, Yunfei Bai, Yulan He, Caroline Catmur, Aline Paes, Marco Caserta,

Organizations: Amazon · King’s College London · The Alan Turing Institute · Universidade Federal Fluminense

Abstract

Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.

Figures & tables

Figure 1: Different roles asking the same query pursue different goals . A perspective-aware LLM agent ought to infer the goal of the given role and take actions admissible inside the role’s capability and knowledge boundary. In cases where the role does not have the remit to accomplish the goal, the agent must identify capable roles and route the work accordingly.

Panel of Table 1 — Table 1: Comparison of the features of against representative LLM agent benchmarks. ● Feature Supported ◐ Feature Partially Supported ○ Feature Not Supported.

Component	Role	Contents	Example
Declared once per domain
$\mathsf{Pred}$	Vocabulary	Predicates from which facts are built	status(device, faulty)
$\mathsf{Cap}$	Perspective axis	Capability gates: authority to act	repair , approve
$\mathsf{Know}$	Perspective axis	Knowledge domains: competence to interpret	hardware , scheduling
$\mathsf{Src}$	Retrieval targets	Knowledge bases a retrieval tool may read	manuals , ticket_log
$\mathsf{Fn}$	Tool taxonomy	The four kinds of thing a tool can do	retrieval , action

Table 2: Components of the hand-curated domain file, with illustrative example.

O : Operator T : Technician

C : Control Engineer P : Planner M : Manager

Table 3: Role gates: capability (authority to act) and knowledge (competence to interpret) assigned independently.

\bullet

granted,

\bullet

withheld.

Figure 2: Benchmark generation, from curated transcripts to executable per-role environments. Every stage pairs a model proposal with a deterministic check holding veto power. An effect the Domain File $\mathcal{D}$ cannot produce is dropped even under unanimous agreement. See Appendix E for elaboration on the dropping criteria.

	Query-Specific Tool Set (QTS)					Full Tool Set (FTS)
Model	P.R. $\uparrow$	P.C. $\uparrow$	O.R. $\downarrow$	U.R. $\downarrow$	Eff. $\uparrow$	P.R. $\uparrow$	P.C. $\uparrow$	O.R. $\downarrow$	U.R. $\downarrow$	Eff. $\uparrow$
with Function Calling
Open-Sourced Models
Qwen3
32B	$0.536_{\scriptscriptstyle\pm.024}$	$0.546_{\scriptscriptstyle\pm.017}$	$0.348_{\scriptscriptstyle\pm.038}$	$0.472_{\scriptscriptstyle\pm.055}$	$0.860$	$0.397_{\scriptscriptstyle\pm.046}$	$0.414_{\scriptscriptstyle\pm.044}$	$0.476_{\scriptscriptstyle\pm.031}$	$0.651_{\scriptscriptstyle\pm.039}$	$0.740$
80B-A3B	$0.388_{\scriptscriptstyle\pm.039}$	$0.472_{\scriptscriptstyle\pm.042}$	$0.588_{\scriptscriptstyle\pm.036}$	$0.567_{\scriptscriptstyle\pm.042}$	$0.581$	$0.152_{\scriptscriptstyle\pm.026}$	$0.305_{\scriptscriptstyle\pm.046}$	$0.414_{\scriptscriptstyle\pm.034}$	$0.439_{\scriptscriptstyle\pm.054}$	$0.320$

Table 4: Main results (Mean

{}_{\pm\text{Variance}}

) across two tool-set regimes (QTS, FTS) and two scaffolds (Function Calling, ReAct).

\uparrow

\downarrow

mark whether higher/lower is better; bold and underline give the best and runner-up within each scaffold block.

Figure 4: Consult-mode escalation collapses under distractor tools. Each knowledge-gate ( consult ) trajectory is scored as a correct consult (teal), an escalation abandoned before answering (amber), or a failure to escalate (red), across both scaffolds split by tool environment.

Figure 5: Perspective-awareness analysis. (a) Capability- vs. knowledge-gate violations. (b) Trajectory divergence vs. Pass against the reference divergence. (c) Per-persona Pass/Role/Over/Under for a proprietary and an open-weight model.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic		Count
Pairs (role $\times$ task)		150
Unique sessions		40
Role	Technician	38
	Control Engineer	34
	Operator	28
	Manager	28

Appendix

Table A1: Detailed statistics of the dataset.

Figure A1 : An illustration of the goal-annotation interface used by domain experts.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
Jun 4, 2026Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5AI Agent Reliability AI Agent Benchmarks
Shanghai AI Laboratory · East China Normal University · Soochow University +2

Sep 1, 2026J/K move · Enter open · S save

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Leonardo Ranaldi, Sherrie Shen, Jushi Kai, Alexandra Birch

(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

Cross-Cultural Language Model Evaluation LLM Agent Evaluation AI Agent Benchmarks Cultural Knowledge in Language Models Long-Horizon LLM Agents

Open paper AskPDF arXiv

Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios. To address these concerns, we present WorldBench: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions. WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators with language- and culture-specific expertise. For evaluation, we extend metrics from previous works and introduce Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and other complementary metrics through deterministic and LLM-as-a-Judge evaluations. Our experiments show that frontier models reach only 49.2% CTS, with all models demonstrating large gaps between correctness and environment preservation. We thereby show that current agents remain brittle in multilingual, agentic scenarios, especially for long-horizon tasks and under state-preservation constraints

Abstract

LLM Agent Evaluation

Explore similar work

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents