OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants

Ruoyu Zhang; Zhaocen Liu; Weiyi Tian; Melissa Feng; Shilong Liu

cs.CLOct 20, 2026

OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants

Authors: Ruoyu Zhang, Zhaocen Liu, Weiyi Tian, Melissa Feng, Shilong Liu

Organizations: University of British Columbia · University of Chicago · Independent Researcher · Princeton University

Abstract

Long-term assistants must follow unfinished tasks across days as information and circumstances change. Finding information or starting an action, however, does not ensure that the task gets completed. We introduce OptionWorld, a benchmark that traces information acquisition, judgments, actions, and outcomes in simulated daily-life tasks to identify where current LLMs fall short as long-term personal assistants. In runs with archived decision inputs, the key facts remain in the model's last input before the deadline in 35.5% of failed instances. Failure analysis identifies the three most frequent problems: needed information is not obtained or combined, available opportunities to act go unused, and agents impose restrictions that the task does not require. Three follow-up experiments show that supplying missing facts restores task success in some cases; action reminders can restart work while subsequent steps still miss the deadline; and correcting a supplied note that wrongly forbids contact restores permitted inquiry. These findings locate shortcomings in maintaining task state: connecting facts to unfinished work, following through before deadlines, and separating actual restrictions from the agent's assumptions.

Figures & tables

Figure 1: Following each target from evidence to action and outcome. Agents discover needs through events and tools, carrying memory and watchpoints across days. In the insurance example, a paid premium is later listed as outstanding. Updated permission allows consultation, but final commitments still require confirmation; the agent must complete the protective action by Day 37. Hidden-target evaluation links six judgments per target to decision inputs and repeated outcomes. Six systems in eight 60-day worlds yield 144 trajectories over three runs.

	Agent capabilities				Failure analysis
Benchmark	Need discovery	State tracking	Active search	Constraints & deadlines	Access vs. judgment	Judgment– outcome link	Decision-input facts	Repeated success
$\tau$ -bench ( 2024 )	$\times$	$\triangle$	$\checkmark$	$\triangle$	$\times$	$\triangle$	$\times$	$\checkmark$
LongMemEvalV2 ( 2026 )	$\times$	$\checkmark$	$\triangle$	$\times$	$\triangle$	$\times$	$\triangle$	$\times$
PM-Bench ( 2026 )	$\times$	$\checkmark$	$\checkmark$	$\triangle$	$\triangle$	$\triangle$	$\triangle$	$\times$
ChronosBench ( 2026 )	$\times$	$\checkmark$	$\checkmark$	$\triangle$	$\times$	$\triangle$	$\times$	$\times$
VitaBench 2.0 ( 2026 )	$\triangle$	$\checkmark$	$\checkmark$	$\triangle$	$\triangle$	$\triangle$	$\times$	$\checkmark$

Table 1: Comparison of benchmark capabilities and failure analysis.

\checkmark

denotes explicit coverage,

\triangle

partial or related coverage, and

\times

no reported coverage. Discovery includes unstated needs and hidden requirements. Constraints require both deadlines and permissions (one alone:

\triangle

). Failure analysis is target-level. Criteria and evidence: Appendix A .

System	Source access ( $O$ )	Risk hypothesis ( $H$ )	Grounded discovery ( $G$ )	Timely grounded ( $T$ )	Intervention delivered ( $I$ )	Target success ( $Y$ / OPR)
DeepSeek	96.5%	86.3%	80.6%	78.1%	57.1%	$48.3\%\pm 13.0\%$
Fable	84.9%	60.2%	55.0%	54.9%	40.6%	$36.1\%\pm 5.2\%$
Gemini	87.5%	56.2%	49.7%	48.1%	37.5%	$30.7\%\pm 6.6\%$
Grok	86.8%	58.9%	50.7%	50.2%	28.0%	$23.6\%\pm 4.5\%$
Qwen	81.4%	63.0%	54.7%	54.3%	40.8%	$35.8\%\pm 4.9\%$
Astra (Codex)	85.1%	80.7%	78.5%	77.8%	68.6%	$61.6\%\pm 2.6\%$

Table 2: Main results: mean percentages across three runs (192 targets each);

\pm

denotes sample standard deviation across runs. OPR: option-preservation rate (target success). Per-run counts: Table 15 .

Figure 2: Repeated outcomes and fact acquisition. Six systems, three runs. (a) Outcomes across runs for 192 targets per system; mixed success means succeeding once or twice. (b) Failed run–target instances grouped by whether all designated primary facts were acquired before the deadline;

n

denotes each system’s failure count.

System	$N$	Information gap	Action selection / adjustment	Self- restriction	State / update error	Execution error	Insufficient evidence
DeepSeek	296	18.6%	16.2%	20.6%	13.9%	1.7%	29.1%
Fable	367	46.3%	10.1%	23.4%	4.9%	0.3%	15.0%
Gemini	364	35.2%	23.4%	9.3%	6.3%	2.7%	23.1%
Grok	373	13.7%	47.7%	22.3%	3.8%	0.8%	11.8%
Qwen	325	24.9%	12.9%	29.8%	7.4%	6.2%	18.8%
Astra (Codex)	211	46.9%	10.4%	2.4%	1.4%	0.0%	38.9%

Table 3: Primary-problem profiles by system (

N=1{,}936

); one label per instance. Rows total 100% before rounding, including the insufficient-evidence status. Information gap: missing, unassembled, or unretained facts.

Figure 3: Targeted support experiments. (a–b) Three continuations per condition from natural failures. (a) Target outcomes; reformatting transposes the fact table. Both training checkpoints concern one target. (b) Qualifying intervention (

I

) and target success (

Y

). (c) Accepted inquiries under externally supplied permission notes: six checkpoints

\times

ten repeats. The updated presentation jointly reorders records and releases waiting.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Pattern	Targets
Single-record constraint	29
Cross-record conflict	33
Cross-day accumulation	32
Missing expected event	31
Clarification before action	32
Preference or multi-party decision	35

Appendix

Table 4: Construction patterns across 192 distinct targets. All six patterns appear in every world.

System	Recorded model identifier	Execution route
DeepSeek	deepseek/deepseek-v4-pro-0813	DeepSeek official API
Fable	anthropic/claude-fable-5.1	OpenRouter / Anthropic
Gemini	google/gemini-3.1-pro-preview-customtools	OpenRouter / Google AI Studio
Grok	x-ai/grok-4.6	OpenRouter / xAI
Qwen	qwen/qwen3.8-max-0902	OpenRouter / Alibaba
Astra (Codex)	gpt-6-astra-pro-local (local slot)	Local subagent bridge

Appendix

Table 5: Recorded identifiers and routes for the main study. Each system contributes 24 trajectories.

Operation	Per day	Per horizon
Read source	4	54
Inspect object	2	46
Compare objects	1	24
Contact stakeholder	1	—
Request confirmation	1	—
External commitment	1	—

Appendix

Table 6: Environment operation allowances in the eight base worlds. Source and object rereads use a turn but do not consume another read or inspection allowance. A dash means no additional horizon cap.

Tool	Inputs	Return or effect
read_source	Source ID	Opens the source and its linked object IDs.
inspect_object	Object ID	Returns object fields after its source has been opened.
compare_objects	Two object IDs	Returns recorded similarities and differences.
lookup_process	Matter, evidence, lookup objective	Returns a registered procedure, operation ID, and applicable deadline or request type.
save_memory	Summary, provenance	Saves a cross-day note.
register_watchpoint	Condition, source scope	Saves a condition for later monitoring.

Appendix

Table 7: Model-facing tools in the API main study. The supplement supplies the complete schemas, including required fields, enumerated values, and text limits. Evidence citations are checked against information actually acquired before the action.

Judgment	Agreement	Cohen’s $\kappa$
Recognition ( $H$ )	785/792 (99.1%)	0.971
Grounding ( $G$ )	786/792 (99.2%)	0.978
Intervention eligibility	729/792 (92.0%)	0.829

Appendix

Table 8: Three-way LLM decision agreement in the 33-trajectory supplemental review batch, with 792 paired targets per row. The last row concerns semantic intervention eligibility, before delivery and timing checks.

Day	Index	Recorded event	Scoring implication
7	45	Inspection of obj-0008 : “receipt includes commercial plan”; follow-up expected by Day 8.	Successful acquisition supports $O$ .
8	57	Inspection of obj-0025 : “console shows personal plan”; the ledger is complete through Day 8 with no matching follow-up. Neither record replaces the other.	Both substantive records are now acquired.
8	62	A correction request cites both objects, identifies the conflict, and asks the software licensing desk to reconcile the records. The environment returns contact-0001 .	Both reviewers support $H,G,T,I$ ; delivery alone does not give $Y$ .
9	69	Feedback to contact-0001 returns the materials and asks for a corrected case-reference field derived from an inspected object.	The initial request has not resolved the issue.
10	97	A revised request supplies the case reference and its source field, cites the returned feedback and both objects, and receives contact-0002 .	A second grounded, timely, authorized delivery supports $I$ .
11	107	Feedback replies to contact-0002 , names the matching target effect, includes both evidence IDs, and reports authoritative issue resolution.	The same-receipt outcome link establishes $Y$ .

Appendix

Table 9: A recorded scoring sequence: DeepSeek, Run 2, independent-contractor world. Indices are zero-based positions in the original trace. Quoted record contents are verbatim; action descriptions are condensed.

Target / actions	Evidence and decision	Confirmed score
Plan mismatch / 62, 97	Both reviewers accept the grounded requests. Feedback 107 resolves the issue and links to request 97.	$O,H,G,T,I,Y=1$ .
Business name / 112, 134	Both reviewers accept the clarification requests. Neither has qualifying outcome feedback. A later operation (166) chooses a governing name without authority established by the reply.	$O,H,G,T,I=1$ , $Y=0$ ; no uncertainty.
Withholding / 317	The tool accepts a change, but both reviewers find that selecting a governing tax status is unsupported by the inspected records.	$O,H,G,T=1$ , $I=Y=0$ ; no uncertainty.
Residency record / 84	Reviewer A marks the correction’s appropriateness uncertain; reviewer B does not accept it. The receipt alone cannot settle whether one record may supersede the other.	$O,H,G,T=1$ ; $I,Y$ have no confirmed credit and remain uncertain .
Work limit / 519	Reviewer A is uncertain whether changing the governing entry implements a work limit; reviewer B accepts the user-confirmed protective operation. Feedback 528 reports option preservation.	$O,H,G,T=1$ ; the disagreement leaves $I,Y$ uncertain, with no confirmed credit .

Appendix

Table 10: Archived scoring decisions from DeepSeek Run 2 in the independent-contractor world. “Uncertain” remains distinct from a jointly unsupported judgment, even when both receive zero confirmed credit. Reviewer A/B identify the two sealed LLM review contexts.

Day	Index	Event	Excerpt
19	286	Tool rejection	“public process identity does not match the complete visible evidence or complete public evidence state”
19	287	Proposed watchpoint	“so the record set is likely incomplete or the step must be a direct reversible act, not a request to the office.” The watchpoint also proposes waiting for a companion record, but exceeds the length limit and is not saved.
19	288	Saved watchpoint	“Watch for a companion seq0/b or seq2/b entry; read it, re-run lookup, then retry once by day 24 as a direct reversible step, not a request to the office.”
19	292	Saved memory	“execute refused (process identity/evidence mismatch), retry after a companion entry.”
24	367	Later assessment	“No companion entry for obj-0025, so operation-3cc1d15b93ddb3 (by 26) not retried.”

Appendix

Table 11: Selected verbatim excerpts from the Qwen rent trace. Text outside quotation marks describes the event. The supplement preserves the selected structured entries and their source hash.

System	$O$	$H$	$G$	$T$	$I$	$Y$	OPR
DeepSeek	556	497	464	450	329	278	48.3%
Fable	489	347	317	316	234	208	36.1%
Gemini	504	324	286	277	216	177	30.7%
Grok	500	339	292	289	161	136	23.6%
Qwen	469	363	315	313	235	206	35.8%

Appendix

Table 12: Five API systems under the common rubric: eight worlds and three runs, or 576 target instances per system. OPR is shown as a percentage,

100\times Y/576

System	$n$	Never	Mixed	Always	Ever
DeepSeek	192	43	112	37	149
Fable	192	86	71	35	106
Gemini	192	90	85	17	102
Grok	192	118	54	20	74
Qwen	192	88	70	34	104

Appendix

Table 13: Same-target consistency in eight worlds (192 targets per system, three runs). Mixed: one or two successes; always: three; ever: at least one.

System	$Y=0$	Facts missing	All acquired
DeepSeek	298	57	241
Fable	368	221	147
Gemini	399	196	203
Grok	440	236	204
Qwen	370	237	133
Total	1875	947	928

Appendix

Table 14: Failure counts by timely primary-fact acquisition across five API systems, eight worlds, and three runs. Facts must be acquired before the target deadline.

System	Run	Source access ( $O$ )	Risk hypothesis ( $H$ )	Grounded discovery ( $G$ )	Timely grounded ( $T$ )	Intervention delivered ( $I$ )	Target success ( $Y$ )	OPR (%)
DeepSeek	1	187	163	153	148	112	105	54.7%
	2	180	164	152	147	93	64	33.3%
	3	189	170	159	155	124	109	56.8%
Fable	1	164	125	116	116	86	77	40.1%
	2	163	107	98	97	64	58	30.2%
	3	162	115	103	103	84	73	38.0%

Appendix

Table 15: Complete per-run results under one semantic scoring rubric. Each row covers eight worlds and 192 targets;

O

–

Y

count targets meeting each criterion.

World	Run	DeepSeek	Fable	Gemini	Grok	Qwen	Astra (Codex)
Caregiver returning to work	1	17	12	9	12	12	17
	2	16	3	2	13	6	14
	3	7	9	7	8	10	19
Independent creative contractor	1	18	11	9	6	11	16
	2	4	1	3	10	12	10
	3	10	11	6	8	13	18

Appendix

Table 16: Outcomes for all 144 trajectories, scored with the same rubric. Every entry is out of 24.

System	Pooled OPR (%)	Three-run range (%)	World-resampling interval (%)
DeepSeek	48.3	33.3–56.8	40.6–54.9
Fable	36.1	30.2–40.1	33.5–38.9
Gemini	30.7	25.0–38.0	25.0–37.0
Grok	23.6	18.8–27.6	15.8–31.9
Qwen	35.8	30.2–39.6	29.2–41.7
Comparison	Difference	Paired interval	W/T/L

Appendix

Table 17: Sensitivity to worlds and repeated runs. Intervals are exact 95% percentile world-resampling intervals, retaining all three runs within each resampled world. Paired comparisons use identical world draws. W/T/L counts worlds where DeepSeek wins, ties, or loses. All differences are percentage points.

System	Recorded–permissive $Y$
DeepSeek	278–384
Fable	208–316
Gemini	177–270
Grok	136–232
Qwen	206–369

Appendix

Table 18: Review sensitivity in the full API comparison, out of 576 target instances per system. The permissive bound treats every unsuccessful instance with any uncertain stage judgment as successful.

Scoring assumption	Triplets	Persistent / failed
Recorded	960	1275/1875 (68.0%)
Unambiguous triplets	538	582/883 (65.9%)
Extreme permissive	960	582/1309 (44.5%)

Appendix

Table 19: Persistent-failure sensitivity to uncertain scores. Each triplet contains three outcomes for the same system, world, and target, one from each run.

Cohort	$Y=0$	Visible	Share
All API	1295	460	35.5%
Unambiguous subset	920	310	33.7%

Appendix

Table 20: Failed target instances with all designated primary facts in the last input within the valid action window (API Runs 2–3). The unambiguous subset excludes instances with any uncertain stage judgment.

System	Targets	$Y=0$	Visible
DeepSeek	384	211	158
Fable	384	253	71
Gemini	384	280	62
Grok	384	301	107
Qwen	384	250	62

Appendix

Table 21: Input visibility across all eight worlds in Runs 2–3: 384 target instances per API system. Visible counts failures with all designated primary facts in the last input within the valid window.

System	$N$	Information gap	Action selection / adjustment	Self- restriction	State / update error	Execution error	Insufficient evidence
DeepSeek	296	55	48	61	41	5	86
Fable	367	170	37	86	18	1	55
Gemini	364	128	85	34	23	10	84
Grok	373	51	178	83	14	3	44
Qwen	325	81	42	97	24	20	61
Astra (Codex)	211	99	22	5	3	0	82

Appendix

Table 22: Counts for the mutually exclusive primary-problem classification. Each row sums to

N

; insufficient evidence is retained as a separate status.

System	All $Y=0$	Normal constraints	$N$	Specific problem	Insufficient evidence
DeepSeek	298	2	296	210	86
Fable	368	1	367	312	55
Gemini	399	35	364	280	84
Grok	440	67	373	329	44
Qwen	370	45	325	264	61
Astra (Codex)	221	10	211	129	82

Appendix

Table 23: Classification coverage. Normal constraints are excluded only from the error-classification denominator. Within that denominator, specific problems and insufficient evidence sum to

N

Target type	$n$	$O$	$H$	$G$	$T$	$I$	$Y$
Cross-day accumulation	480	446	316	303	302	153	153
Cross-source conflict	495	465	308	277	268	172	172
Missing expected event	465	434	324	302	299	276	264
Preference / multiple parties	525	510	418	297	289	197	94
Single-source constraint	435	314	233	230	226	143	143
Stakeholder clarification	480	349	271	265	261	234	179

Appendix

Table 24: Stage counts by target type across five API systems and three runs. Each row pools the same target instances across all six criteria.

System	All (311)	Unambiguous (139)
DeepSeek	161	110
Fable	179	103
Gemini	142	87
Grok	121	70
Qwen	184	98

Appendix

Table 25: Target successes on shared timely-fact subsets. Both columns require all five systems to obtain the primary facts; the second also excludes any system’s uncertain criterion judgments.

System	Cases	Days left	Comparisons left
Fable	6	6–10	11–16
Gemini	3	7–10	9–20
Grok	2	9–9	17–20
Qwen	4	5–9	11–22

Appendix

Table 26: Original-checkpoint fact-acquisition replays: successful cases, days until deadline, and remaining horizon comparisons. Every checkpoint has exhausted new-object inspections but permits a daily comparison.

Tool	Fable	Qwen
Read source	1,252	1,687
Inspect object	1,166	1,920
Compare objects	259	241
Look up procedure	351	1,195
Report risk	286	1,337
Register watchpoint	713	1,629

Appendix

Table 27: Tool-call allocation across 24 trajectories per system. Attempts are counted whether accepted or rejected; counts are distinct from qualifying interventions and successful target outcomes.

(a) Information support: original target outcome, success / failure / unknown
Checkpoint (Gemini)	Original	Added facts	Reformatted facts
Buyout (R2)	0 / 3 / 0	3 / 0 / 0	2 / 1 / 0
Training (R2)	0 / 2 / 1	0 / 1 / 2	0 / 0 / 3
Training (R3)	0 / 3 / 0	3 / 0 / 0	2 / 0 / 1

Appendix

Table 28: Information support can restore task success; action reminders can restore action without timely completion. Three continuations per checkpoint and condition, all through Day 60. (a) Each cell totals three; unknown scores remain in the denominator. R2/R3 identify source runs, not new repeats. (b) New action uses the original timely, grounded intervention criterion (

I

); outcome uses the original target criterion (

Y

). The two training checkpoints share one world–target pair. In (a), new-

I

and

Y

counts coincide; the criteria remain distinct.

Experiment	Checkpoint	World / focal matter	Model (run)	Start day	Deadline
Information	4c9bfe779e6bb394	Creative contractor / buyout	Gemini (2)	46	55
	b71e084171730d98	Laid-off worker / training hours	Gemini (2)	48	57
	7f10db11d4df4f8e	Laid-off worker / training hours	Gemini (3)	48	57
Action	ca3a58a5edf2b82c	Food business / inspection and pickup	Gemini (2)	15	22
	05cdc677c2bc1e7d	Multigenerational household / shared car	DeepSeek (2)	16	23

Appendix

Table 29: Frozen natural-failure checkpoints. The deadline is the latest qualifying action day, not the end of the continuation; later feedback is checked against the action it confirms. The two training checkpoints test the same target in the same world.

Checkpoint	Condition	Authority attempts	Actions	Other outcomes	Requests
Information experiment
Buyout (R2)	Original	0–13	319	24–35	371
Buyout (R2)	Added facts	0–0	75	24–33	118
Buyout (R2)	Reformatted facts	0–1	119	22–32	166
Training (R2)	Original	0–1	41	25–36	84
Training (R2)	Added facts	0–4	73	23–33	116

Appendix

Table 30: Secondary checks summed over three continuations per row.

		Current state			Update
Original world	Matter	Retain	Rewrite	Correct	Retain	Rewrite	Correct
Returning caregiver	Childcare funding	3/10	7/10	10/10	10/10	10/10	10/10
Newcomer coordinator	Credential account contact	0/10	0/10	10/10	6/10	5/10	10/10
Multigenerational household	Co-owner signatures	0/10	0/10	10/10	6/10	6/10	10/10
Creative contractor	Insurance timing	1/10	0/10	10/10	9/10	10/10	10/10
Retired tenant	Account contact authority	1/10	1/10	10/10	10/10	10/10	10/10

Appendix

Table 31: Accepted clarification inquiries when contact is permitted but commitments are not. Each row is one checkpoint from a different original world, with ten repeats per cell. The total row supplies the six bars in Figure 3 c.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
Sep 7, 2026Yongchang Peng, Qingshui Gu, Liya Zhu +31Pragmatic Reasoning in Language Models LLM Agent Evaluation
ByteDance Seed · M-A-P
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
Aug 11, 2026Xiaohongshu Dots Studio, Evolvent AIProactive Agents Long-Horizon Agent Evaluation
LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation
Mar 12, 2026Feiyu Duan, Xuanjing Huang, Zhongyu WeiUser Simulation Long-Horizon Agent Evaluation
School of Data Science, Fudan University · College of Computer Science and Artificial Intelligence, Fudan University · Shanghai Innovation Institute +2

Sep 7, 2026J/K move · Enter open · S save

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

Yongchang Peng, Qingshui Gu, Liya Zhu, Ge Zhang+30

ByteDance Seed · M-A-P

Pragmatic Reasoning in Language Models LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.

Abstract

LLM Agent Evaluation

Explore similar work

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation