cs.HCOct 25, 2026

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Authors: Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, 

Organizations: Meta · Duke University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.

Figures & tables

Figure 1: (Top) Interaction Loop in InteractiveWorld. The Assistant agent provides guidance based on current observations and history, which the Human agent grounds into executable actions to update the environment.(Bottom) Example task from InteractiveBench. The Assistant helps the human choose dinner (using a daily plan and food labels) and guides its placement on the table. Interaction relies on natural language and is evaluated by state-verifiable metrics.
Figure 2: InteractiveWorld Overview. Three layers enable realistic human–assistant collaboration: (1) Physical : realistic humans, assets, and physics simulation; (2) Human Action : autonomous, stable high-level daily action execution; (3) Semantic : grounding free-form guidance into action sequences.
CategoryEvaluation focusExample task
GuidanceGuide user step by step based on the user’s progress, helping them complete task.Lab cleanup. A new lab worker must put away 14 chemical bottles and clear broken glass. The assistant must use the labels, chemical knowledge and the safety rules to guide the cleanup.
PlanningOutline the steps for tasks within time and resource limits, flexibly adjust plan.Cooking for guests. A chef need cook for guests with different food allergies, using one oven and pan within 30 minutes. The assistant must choose and schedule as many safe dishes as possible.
AccessibilityDescribe the scene for blind or low-vision users so they can complete everyday tasks.Kitchen cleanup. A blind user needs to clean a kitchen with broken glass, food, and dishes. The assistant must read the cleanup guide, describe the scene, and help the user finish safely.
DecisionHelp with daily decision using current observations and assistant’s knowledge.Supplier comparison. A buyer sees an order for 40 filters, then sees two quotes with different box sizes in another room. The assistant must choose the cheaper offer for the full order.
PreventionWarn before mistakes happen , while avoiding unnecessary interruptions.Leaving home. A user is about to leave, but the fridge and living-room window were left open earlier. The assistant must remember these problems and remind the user before they leave.
FindingHelp find missing objects by remembering where they moved and guiding search.Finding stored tools. A user cannot find the black and blue squeegees after storing them in a lidded box inside a cabinet in another room. The assistant must remember where they went.
Table 1: Overview of the InteractiveBench Classification. To simulate real-world scenarios, visual inputs are intentionally withheld from human agents during accessibility tasks.
Table 2: Performance Across Six Task Categories. Pass is the percentage of fully completed instances (all objectives met). MCR (Mean Completion Rate) is the average fraction of objectives completed per instance. We verified that, under our evaluation protocol, the same Human agent achieves a 100% pass rate with an Oracle Assistant and 0% without assistance.
Table 3: Performance Across Nine Capabilities. Each of the 120 tasks is annotated with required capabilities and evaluated using separate state-verifiable checks. For each capability, we report the percentage of passed evaluation points across all applicable tasks. Models are ordered by overall task pass rate in descending order. The nine capabilities are Object Recognition, Spatial Grounding, OCR, Long-horizon Memory, Planning, Knowledge, Error Detection, Proactivity, and Communication.
Figure 4: Examples and Distribution of Error Types. We analyze initial failure of failed tasks and grouped into six types. (Top) Examples of each error type. (Bottom) Error type distributions.
Figure 5: (Left) Human Agent Impact. We report average MCR across Guidance, Planning, and Accessibility (tasks requiring maximal human involvement) when cross-evaluating four diverse models interchangeably as assistant and human agents. (Right) Memory Augmentation Impact. We report model pass rates with vs. without inference-time memory augmentation.
Figure 6: Performance Across Task Horizons. We report model success rates against task lengths (derived from oracle actions). Task horizons are normalized per category on the x-axis (0: shortest, 1: longest), while the y-axis shows the average pass rate at each normalized length step.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4: Comparison with existing egocentric AI assistant benchmarks and embodied simulators. InteractiveWorld is the only environment that satisfies all six properties. Ego. View : first-person observations of a human. Human Actor : the task is physically performed by a (real or simulated) human, not a robot. Guided Human : the human autonomously grounds the agent’s free-form language guidance into actions. NL Dialog : multi-turn natural-language interaction. Closed Loop : agent outputs change the environment and the subsequent observations. Verif. Success : success is checked against ground-truth environment states rather than by answer matching or LLM judges. : partially supported.
PrimitiveSurface verbs and argumentsOperational semantics
Locomotion and posture
navigate_toWalk , Run , WalkTowards targetPlans and executes a collision-aware route to a live interaction stance near the target; Run is an alias and selects no separate speed controller.
forwardForward , WalkForward optional distanceCollision-checks and executes a straight body-frame displacement.
turnTurnLeft , TurnRight , TurnAround optional angleRotates in place while preserving any wrist-carried object; the optional angle applies to TurnLeft and TurnRight only.
sitSit seatLowers and anchors the body on the live seat top; records sitting .
standupStandUpReverses the seated pose, steps away from the seat, and clears sitting .
Appendix
Table 5: The 36 canonical action primitives, grouped by function. The second column lists the accepted surface verbs (written as [Verb] in a command, e.g., [Walk] ) followed, in italics, by the arguments in order. Distance, angle, duration, and angle error are numeric; all other arguments refer to scene entities. “Held” means that the named object must match the persistent held-object state.
Figure 7: InteractiveBench task generation. For each task family and target capability, we define a scenario and fix its assessment target. We then build an executable task, check its semantics and actions, and validate it through Isaac Sim replay, visual inspection, and causal controls. The dashed arrow shows how repairable errors return for revision and revalidation without changing the assessment target. The final benchmark contains 120 tasks, with 20 per family.
IDCapabilityBoundary
C1Object/state recognitionIdentify the current object and its current state; temporal updating belongs to C4.
C2Spatial groundingLocate objects, destinations, and relations in the egocentric scene.
C3Scene-text readingTranscribe and bind visible text; conclusions inferred from text belong to C6.
C4Episodic memory/state trackingRetain and update identities, events, locations, and states across time.
C5Planning/replanningForm, sequence, and revise a plan; Human execution cannot substitute for Assistant planning.
C6Knowledge/reasoningApply rules, arithmetic, domain knowledge, and constraints to choose or infer.
Appendix
Table 6: Capability ontology.
TaskEpisode and capability testedEvaluator validation
1Read seven beverage labels and solve a globally coupled three-box allocation; tests OCR, arithmetic, and global constraint satisfaction.Exact composites: A = milk-500 + water-100; B = milk-500 + juice-200; C = milk-500 + juice-200 + water-100. No wrong categories; all seven bottles form one exact partition.
2Recall carton sizes seen before a phone interruption and fill three later-arriving orders; tests episodic visual memory and numerical allocation.Exact size composites A = 500+100, B = 500+250, C = 500+250+100; all seven cartons partitioned once, no wrong sizes, and the phone preserved.
3Archive obsolete briefing material before placing a new packet and pen; tests document-grounded ordered guidance.Old material archived; new packet and pen on the reset table; new-item credit depends on removal of the obsolete material.
4Use a stocking rule and price cards to arrange bread, candy, chocolate, and their labels in ascending-price order.Three exact two-object composites occupy the low/mid/high promotional slots; none of the six objects remains on the cart.
5Follow a tomato recipe through cutting, cooking, plating, and serving.Tomato cut and safely cooked; cooktop off; the cooked form is the sole plated food; completed plate held; knife and spatula returned.
6Sort four pantry goods by storage requirements; tests rule reading and storage knowledge.Candy/chocolate exactly in the light display; coriander/coffee exactly in the closed dark cabinet.
Appendix
Table 7: Guidance tasks.
TaskEpisode and capability testedEvaluator validation
1Pack an exact shipment from nine capacity-labeled packages; tests numerical planning and subset selection.Exact water set {3 L, 3 L, 2 L}, juice set {4 L, 3 L, 2 L}, and 3-L lemonade; seven candidates exactly, excluding water-4 L and juice-5 L.
2Execute a complete lunch order with schnitzel, broccolini, sealed sauce, tools, cooking, plating, and delivery.Both foods cooked/browned and plated; cooktop off and spatula returned; exact two-food plate plus sealed sauce delivered to service pass.
3Execute an ordered, multi-room housekeeping closeout.Gift inside closed locker; water refrigerated; towel inside washer; washer closed and running with the towel.
4Prepare a snack and replan when cutting reveals mold.Moldy bread cut then discarded; fresh bread cut and plated with cheese/apple; safe plate at service pass; negative/exact constraints and knife return hold.
5Balance three classroom desks with one water and one juice each using a short one-hand route.Exact water counts across all three desks form one bundle and exact juice counts another; route length is reported separately.
6Repair a three-way grocery misplacement with a continuous short route.Rice to grains, barbecue sauce to sauces, and iced tea to beverages, one complete active-result bundle each.
Appendix
Table 8: Planning tasks.
TaskEpisode and capability testedEvaluator validation
1Visually identify orange juice among a can, carton, and bottle and give body-anchored guidance.Juice on breakfast table; soda and water remain at baseline (and never move under the transition gate).
2Repeatedly reassess a growing gap between a pot and bowl, moving a scale only when it fits.Scale on clearing counter; three bottles on holding counter; pot/bowl at baseline. STG keeps the scale at source until all bottles are cleared.
3Compare remembered and current table settings and restore only a moved blue bowl.Bowl restored to guest table; added coffee cup, unchanged wineglass/water, and all setting objects remain at handoff baseline.
4Interrupt selection of a hazardous cleanser and redirect the wearer to baby oil.Baby oil on side table; cleanser stably returned to cabinet, not held/on floor; bath salt at baseline. STG requires cleanser return before oil acquisition.
5Admit that spray is absent from the current view, avoid a cactus, scan, and relocate the spray.Spray on front cabinet; cactus, skin cream, and sponge at baseline; protected objects never move.
6Prioritize putting down a carving knife safely before searching for a ringing phone.Knife stably on prep counter and phone held. STG requires knife safety before leaving or acquiring the phone.
Appendix
Table 9: Accessibility tasks.
TaskEpisode and capability testedEvaluator validation
1Choose the highest-calorie option without exceeding the corrected daily allowance; tests report reading, spoken correction, and constrained argmax ( 2000−1780−110=1102000-1780-110=110 ).The 45-kcal chocolate syrup is on the breakfast table.
2Fuse a children table and gifts-per-child table across rooms; compute 1×2+2×1.5+0×1=51\times 2+2\times 1.5+0\times 1=5 .Five gifts distributed exactly 2/2/1 among three cubbies; spare gift remains at supply.
3Intersect an earlier remembered promo-brand set with twelve later router cartons.AXO, BYT, and CUE cartons on display; all nine regular cartons at source/unmoved.
4Reconstruct which items remain unfinished in a seven-point closeout after a long action chain.Cabinet closed, oven off, mug/plate in cabinet, juice in fridge, can in trash, fridge closed.
5Count stocked cartons, then consult a price sheet: 5\times\3+4\times$4+2\times$5=$41$ .41invoiceonapprovaldesk;41 invoice on approval desk; 39 and $45 alternatives remain at source/unmoved.
6Aggregate two cabinet inventories, subtract removed items, and compute deficits against 4 towels, 5 mugs, and 7 waters.Two replacement towels and one mug inserted; removed towel/water remain staged; both cabinets closed.
Appendix
Table 10: Decision tasks.
TaskError and capability testedEvaluator validation
1Cold cuts are moved before sanitizer is stored and the chemical cabinet closed; tests procedural order and food/chemical safety.Mug in dish rack; towel in used sink; sanitizer in closed chemical cabinet; cold cuts on prep table and clean.
2Adding another two-serving box would make seven, not exactly six; tests quantity tracking after a delayed update.One- and three-serving boxes plus exactly one interchangeable two-serving box on service table; the other two-serving box at supply.
3A used sponge remains on the table after the wearer reaches the break area; tests delayed omitted-step detection.Sponge, mug, and towel in sink; phone on shelf; table physically clean.
4A third wine and, later, chips violate “two wines, burger, no sides”; requires two interventions.Exactly two interchangeable wines and burger on packing table; remaining wine and chips at stock.
5Dough is about to enter the oven without an X score; tests prerequisite detection and recovery.Dough cut and in oven; oven closed/running; current bake cycle began only after valid preparation.
6A used sponge is confused with an identical clean sponge; tests event-bound identity.Used sponge in used bin; clean sponge in closed clean cabinet; persistent used/clean attributes agree with history.
Appendix
Table 11: Prevention tasks.
TaskEpisode and capability testedEvaluator validation
1Inspect labels to distinguish freezer from chiller, then classify five cold-delivery items.Ice cream/lolly exactly in closed freezer; milk, yogurt, and lettuce exactly in closed chiller (GUIDE StateScore-v2).
2Inspect contents of two otherwise ambiguous refrigerators to infer the dairy unit.Cheese in secured dairy fridge; existing milk remains there, chicken in the other unit, and the other fridge remains closed (GUIDE StateScore-v2).
3Track which of two identical glasses received vodka across a phone delay.Alcohol-free glass on guest table; actual vodka-containing glass on adult counter, using persistent contents rather than hidden identity (WATCH binary).
4Search three rooms for a sponge hidden in one of several similar low cabinets when its initial location was never shown.Search exposes sponge before grab; sponge reaches result counter without hidden-ID navigation.
5Retrieve the one of two identical erasers that touched a loose sheet rather than a folder.Paper-contact eraser, not distractor, visibly acquired and placed in storage cabinet.
6Recall the independent locations of two identical coconut-milk bottles.Garden and fridge bottles directly retrieved to final tray; fridge opened before the hidden bottle is acquired.
Appendix
Table 12: Finding tasks.
ModelGuidancePlanningAccessibilityDecisionPreventionFinding
GPT-5.545%50%50%75%70%55%
GPT-5.6 Sol45%50%60%70%75%50%
GPT-5.6 Luna35%50%55%60%65%40%
Claude Opus 540%45%50%70%70%50%
Claude Fable 5.145%65%65%70%75%50%
Claude Opus 4.835%30%25%60%45%40%
Appendix
Table 13: Static accuracy across the six task families. Every entry uses a fixed denominator of 20 tasks. Static items without enough evidence are counted as failures.
Figure 8: Performance on Static Adaptations of InteractiveBench inflate 8 models’ scores ; large Static Only (red) and Dynamic Only (blue) regions show a severe evaluation gap.
Figure 9: Complete-task success probability versus normalized reference task horizon across six task families. Each colored curve is a model-specific Firth logistic fit. The red curve is the pooled fit across models, and the gray band is its pointwise 95%95\% task-bootstrap interval. Horizon is normalized separately within each task family. The figure compares different tasks and does not represent progress within a single episode. No monotonicity constraint was applied.
 Model-based Human agentHuman participant
AssistantFable 5.1Muse 1.1GPT-5.6Gemini 3.1Human 1Human 2Human 3
Fable 5.113/3615/3613/3613/3615/3616/3615/36
Muse 1.19/369/368/369/3610/3611/369/36
GPT-5.6 Sol8/369/3611/3610/368/3611/369/36
Gemini 3.18/368/365/367/368/368/367/36
Appendix
Table 14: Assistant performance with model-based and human executors. Rows identify the evaluated Assistant; columns identify the executor. Each entry is the number of successful tasks out of the same 36-task subset. Human 1–3 are three individual humans, not three independent task sets.
Figure 10: Task-level outcomes for Fable 5.1 with different executors. Rows are executors and columns are the 36 tasks, grouped into six families. Filled squares indicate success and open squares indicate failure. Task numbers restart within each family. The dashed line separates model-based Human agents from human participants.
More in this topic

Multimodal Agent Evaluation

See all 22 papers

Explore similar work

CardsList
Oct 10, 2026

Words to World: Benchmarking LLMs in Interactive Physical Environments

Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments. We present a benchmark for measuring LLMs in closed-loop physical interaction, where models control a harness system connected to real-world components including robotic arms, cameras, and sensors. In each task, the model must issue instructions, receive physical feedback from the environment, interpret the resulting state, and adapt its next action accordingly. This setting evaluates capabilities beyond language understanding, including grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty. Through experiments with representative LLMs, we characterize their ability to translate linguistic reasoning into physical actions and identify common failure modes such as hallucinated state changes, poor recovery from unexpected outcomes, and unstable long-horizon plans. Our benchmark aims to provide a systematic foundation for evaluating LLMs as interactive agents that connect words, tools, and the physical world.

Xinnan Dai, Kaiqi Yang, Yaxin Li +9

Michigan State University · Independent Researcher · Zhejiang University +3

LLM Agent EvaluationAI Agent BenchmarksLarge Language Model-Based Robot Planning
AskPDF
Feb 28, 2026

LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks

The rapid progress of Multimodal Large Language Models (MLLMs) marks a significant step toward artificial general intelligence, offering great potential for augmenting human capabilities. However, their ability to provide effective assistance in dynamic, real-world environments remains largely underexplored. Existing video benchmarks predominantly assess passive understanding through retrospective analysis or isolated perception tasks, failing to capture the interactive and adaptive nature of real-time user assistance. To bridge this gap, we introduce LifeEval, a multimodal benchmark designed to evaluate real-time, task-oriented human-AI collaboration in daily life from an egocentric perspective. LifeEval emphasizes three key aspects: task-oriented holistic evaluation, egocentric real-time perception from continuous first-person streams, and human-assistant collaborative interaction through natural dialogues. Constructed via a rigorous annotation pipeline, the benchmark comprises 4,075 high-quality question-answer pairs across 6 core capability dimensions. Extensive evaluations of 26 state-of-the-art MLLMs on LifeEval reveal substantial challenges in achieving timely, effective and adaptive interaction, highlighting essential directions for advancing human-centered interactive intelligence.

Hengjian Gao, Kaiwei Zhang, Shibo Wang +8

Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Shanghai University of Electric Power +1

AI Agent BenchmarksMultimodal Model EvaluationHuman-AI CollaborationEgocentric Video Understanding
AskPDF
May 5, 2026

iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework

Achieving Artificial General Intelligence (AGI) requires agents that learn and interact adaptively, with interactive world models providing scalable environments for perception, reasoning, and action. Yet current research still lacks large-scale datasets and unified benchmarks to evaluate their physical interaction capabilities. To address this, we propose iWorld-Bench, a comprehensive benchmark for training and testing world models on interaction-related abilities such as distance perception and memory. We construct a diverse dataset with 330k video clips and select 2.1k high-quality samples covering varied perspectives, weather, and scenes. As existing world models differ in interaction modalities, we introduce an Action Generation Framework to unify evaluation and design six task types, generating 4.9k test samples. These tasks jointly assess model performance across visual generation, trajectory following, and memory. Evaluating 14 representative world models, we identify key limitations and provide insights for future research. The iWorld-Bench model leaderboard is publicly available at iWorld-Bench.com.

Jianjie Fang, Yingshan Lei, Qin Wan +8

Tsinghua University, China · Northeastern University, China · South China University of Technology, China

World Model EvaluationInteractive World ModelsVideo World Models
AskPDF