cs.HCOct 25, 2026
SaveInteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Organizations: Meta · Duke University
Abstract
People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.
Figures & tables
Figure 1: (Top) Interaction Loop in InteractiveWorld. The Assistant agent provides guidance based on current observations and history, which the Human agent grounds into executable actions to update the environment.(Bottom) Example task from InteractiveBench. The Assistant helps the human choose dinner (using a daily plan and food labels) and guides its placement on the table. Interaction relies on natural language and is evaluated by state-verifiable metrics.
Figure 2: InteractiveWorld Overview. Three layers enable realistic human–assistant collaboration: (1) Physical : realistic humans, assets, and physics simulation; (2) Human Action : autonomous, stable high-level daily action execution; (3) Semantic : grounding free-form guidance into action sequences.
| Category | Evaluation focus | Example task |
|---|---|---|
| Guidance | Guide user step by step based on the user’s progress, helping them complete task. | Lab cleanup. A new lab worker must put away 14 chemical bottles and clear broken glass. The assistant must use the labels, chemical knowledge and the safety rules to guide the cleanup. |
| Planning | Outline the steps for tasks within time and resource limits, flexibly adjust plan. | Cooking for guests. A chef need cook for guests with different food allergies, using one oven and pan within 30 minutes. The assistant must choose and schedule as many safe dishes as possible. |
| Accessibility | Describe the scene for blind or low-vision users so they can complete everyday tasks. | Kitchen cleanup. A blind user needs to clean a kitchen with broken glass, food, and dishes. The assistant must read the cleanup guide, describe the scene, and help the user finish safely. |
| Decision | Help with daily decision using current observations and assistant’s knowledge. | Supplier comparison. A buyer sees an order for 40 filters, then sees two quotes with different box sizes in another room. The assistant must choose the cheaper offer for the full order. |
| Prevention | Warn before mistakes happen , while avoiding unnecessary interruptions. | Leaving home. A user is about to leave, but the fridge and living-room window were left open earlier. The assistant must remember these problems and remind the user before they leave. |
| Finding | Help find missing objects by remembering where they moved and guiding search. | Finding stored tools. A user cannot find the black and blue squeegees after storing them in a lidded box inside a cabinet in another room. The assistant must remember where they went. |
Table 1: Overview of the InteractiveBench Classification. To simulate real-world scenarios, visual inputs are intentionally withheld from human agents during accessibility tasks.
Table 2: Performance Across Six Task Categories. Pass is the percentage of fully completed instances (all objectives met). MCR (Mean Completion Rate) is the average fraction of objectives completed per instance. We verified that, under our evaluation protocol, the same Human agent achieves a 100% pass rate with an Oracle Assistant and 0% without assistance.
Table 3: Performance Across Nine Capabilities. Each of the 120 tasks is annotated with required capabilities and evaluated using separate state-verifiable checks. For each capability, we report the percentage of passed evaluation points across all applicable tasks. Models are ordered by overall task pass rate in descending order. The nine capabilities are Object Recognition, Spatial Grounding, OCR, Long-horizon Memory, Planning, Knowledge, Error Detection, Proactivity, and Communication.
Figure 4: Examples and Distribution of Error Types. We analyze initial failure of failed tasks and grouped into six types. (Top) Examples of each error type. (Bottom) Error type distributions.
Figure 5: (Left) Human Agent Impact. We report average MCR across Guidance, Planning, and Accessibility (tasks requiring maximal human involvement) when cross-evaluating four diverse models interchangeably as assistant and human agents. (Right) Memory Augmentation Impact. We report model pass rates with vs. without inference-time memory augmentation.
Figure 6: Performance Across Task Horizons. We report model success rates against task lengths (derived from oracle actions). Task horizons are normalized per category on the x-axis (0: shortest, 1: longest), while the y-axis shows the average pass rate at each normalized length step.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: Comparison with existing egocentric AI assistant benchmarks and embodied simulators. InteractiveWorld is the only environment that satisfies all six properties. Ego. View : first-person observations of a human. Human Actor : the task is physically performed by a (real or simulated) human, not a robot. Guided Human : the human autonomously grounds the agent’s free-form language guidance into actions. NL Dialog : multi-turn natural-language interaction. Closed Loop : agent outputs change the environment and the subsequent observations. Verif. Success : success is checked against ground-truth environment states rather than by answer matching or LLM judges. : partially supported.
| Primitive | Surface verbs and arguments | Operational semantics |
|---|---|---|
| Locomotion and posture | ||
| navigate_to | Walk , Run , WalkTowards target | Plans and executes a collision-aware route to a live interaction stance near the target; Run is an alias and selects no separate speed controller. |
| forward | Forward , WalkForward optional distance | Collision-checks and executes a straight body-frame displacement. |
| turn | TurnLeft , TurnRight , TurnAround optional angle | Rotates in place while preserving any wrist-carried object; the optional angle applies to TurnLeft and TurnRight only. |
| sit | Sit seat | Lowers and anchors the body on the live seat top; records sitting . |
| standup | StandUp | Reverses the seated pose, steps away from the seat, and clears sitting . |
Table 5: The 36 canonical action primitives, grouped by function. The second column lists the accepted surface verbs (written as [Verb] in a command, e.g., [Walk] ) followed, in italics, by the arguments in order. Distance, angle, duration, and angle error are numeric; all other arguments refer to scene entities. “Held” means that the named object must match the persistent held-object state.
Figure 7: InteractiveBench task generation. For each task family and target capability, we define a scenario and fix its assessment target. We then build an executable task, check its semantics and actions, and validate it through Isaac Sim replay, visual inspection, and causal controls. The dashed arrow shows how repairable errors return for revision and revalidation without changing the assessment target. The final benchmark contains 120 tasks, with 20 per family.
| ID | Capability | Boundary |
|---|---|---|
| C1 | Object/state recognition | Identify the current object and its current state; temporal updating belongs to C4. |
| C2 | Spatial grounding | Locate objects, destinations, and relations in the egocentric scene. |
| C3 | Scene-text reading | Transcribe and bind visible text; conclusions inferred from text belong to C6. |
| C4 | Episodic memory/state tracking | Retain and update identities, events, locations, and states across time. |
| C5 | Planning/replanning | Form, sequence, and revise a plan; Human execution cannot substitute for Assistant planning. |
| C6 | Knowledge/reasoning | Apply rules, arithmetic, domain knowledge, and constraints to choose or infer. |
Table 6: Capability ontology.
| Task | Episode and capability tested | Evaluator validation |
|---|---|---|
| 1 | Read seven beverage labels and solve a globally coupled three-box allocation; tests OCR, arithmetic, and global constraint satisfaction. | Exact composites: A = milk-500 + water-100; B = milk-500 + juice-200; C = milk-500 + juice-200 + water-100. No wrong categories; all seven bottles form one exact partition. |
| 2 | Recall carton sizes seen before a phone interruption and fill three later-arriving orders; tests episodic visual memory and numerical allocation. | Exact size composites A = 500+100, B = 500+250, C = 500+250+100; all seven cartons partitioned once, no wrong sizes, and the phone preserved. |
| 3 | Archive obsolete briefing material before placing a new packet and pen; tests document-grounded ordered guidance. | Old material archived; new packet and pen on the reset table; new-item credit depends on removal of the obsolete material. |
| 4 | Use a stocking rule and price cards to arrange bread, candy, chocolate, and their labels in ascending-price order. | Three exact two-object composites occupy the low/mid/high promotional slots; none of the six objects remains on the cart. |
| 5 | Follow a tomato recipe through cutting, cooking, plating, and serving. | Tomato cut and safely cooked; cooktop off; the cooked form is the sole plated food; completed plate held; knife and spatula returned. |
| 6 | Sort four pantry goods by storage requirements; tests rule reading and storage knowledge. | Candy/chocolate exactly in the light display; coriander/coffee exactly in the closed dark cabinet. |
Table 7: Guidance tasks.
| Task | Episode and capability tested | Evaluator validation |
|---|---|---|
| 1 | Pack an exact shipment from nine capacity-labeled packages; tests numerical planning and subset selection. | Exact water set {3 L, 3 L, 2 L}, juice set {4 L, 3 L, 2 L}, and 3-L lemonade; seven candidates exactly, excluding water-4 L and juice-5 L. |
| 2 | Execute a complete lunch order with schnitzel, broccolini, sealed sauce, tools, cooking, plating, and delivery. | Both foods cooked/browned and plated; cooktop off and spatula returned; exact two-food plate plus sealed sauce delivered to service pass. |
| 3 | Execute an ordered, multi-room housekeeping closeout. | Gift inside closed locker; water refrigerated; towel inside washer; washer closed and running with the towel. |
| 4 | Prepare a snack and replan when cutting reveals mold. | Moldy bread cut then discarded; fresh bread cut and plated with cheese/apple; safe plate at service pass; negative/exact constraints and knife return hold. |
| 5 | Balance three classroom desks with one water and one juice each using a short one-hand route. | Exact water counts across all three desks form one bundle and exact juice counts another; route length is reported separately. |
| 6 | Repair a three-way grocery misplacement with a continuous short route. | Rice to grains, barbecue sauce to sauces, and iced tea to beverages, one complete active-result bundle each. |
Table 8: Planning tasks.
| Task | Episode and capability tested | Evaluator validation |
|---|---|---|
| 1 | Visually identify orange juice among a can, carton, and bottle and give body-anchored guidance. | Juice on breakfast table; soda and water remain at baseline (and never move under the transition gate). |
| 2 | Repeatedly reassess a growing gap between a pot and bowl, moving a scale only when it fits. | Scale on clearing counter; three bottles on holding counter; pot/bowl at baseline. STG keeps the scale at source until all bottles are cleared. |
| 3 | Compare remembered and current table settings and restore only a moved blue bowl. | Bowl restored to guest table; added coffee cup, unchanged wineglass/water, and all setting objects remain at handoff baseline. |
| 4 | Interrupt selection of a hazardous cleanser and redirect the wearer to baby oil. | Baby oil on side table; cleanser stably returned to cabinet, not held/on floor; bath salt at baseline. STG requires cleanser return before oil acquisition. |
| 5 | Admit that spray is absent from the current view, avoid a cactus, scan, and relocate the spray. | Spray on front cabinet; cactus, skin cream, and sponge at baseline; protected objects never move. |
| 6 | Prioritize putting down a carving knife safely before searching for a ringing phone. | Knife stably on prep counter and phone held. STG requires knife safety before leaving or acquiring the phone. |
Table 9: Accessibility tasks.
| Task | Episode and capability tested | Evaluator validation |
|---|---|---|
| 1 | Choose the highest-calorie option without exceeding the corrected daily allowance; tests report reading, spoken correction, and constrained argmax ( ). | The 45-kcal chocolate syrup is on the breakfast table. |
| 2 | Fuse a children table and gifts-per-child table across rooms; compute . | Five gifts distributed exactly 2/2/1 among three cubbies; spare gift remains at supply. |
| 3 | Intersect an earlier remembered promo-brand set with twelve later router cartons. | AXO, BYT, and CUE cartons on display; all nine regular cartons at source/unmoved. |
| 4 | Reconstruct which items remain unfinished in a seven-point closeout after a long action chain. | Cabinet closed, oven off, mug/plate in cabinet, juice in fridge, can in trash, fridge closed. |
| 5 | Count stocked cartons, then consult a price sheet: 5\times\3+4\times$4+2\times$5=$41$ . | 39 and $45 alternatives remain at source/unmoved. |
| 6 | Aggregate two cabinet inventories, subtract removed items, and compute deficits against 4 towels, 5 mugs, and 7 waters. | Two replacement towels and one mug inserted; removed towel/water remain staged; both cabinets closed. |
Table 10: Decision tasks.
| Task | Error and capability tested | Evaluator validation |
|---|---|---|
| 1 | Cold cuts are moved before sanitizer is stored and the chemical cabinet closed; tests procedural order and food/chemical safety. | Mug in dish rack; towel in used sink; sanitizer in closed chemical cabinet; cold cuts on prep table and clean. |
| 2 | Adding another two-serving box would make seven, not exactly six; tests quantity tracking after a delayed update. | One- and three-serving boxes plus exactly one interchangeable two-serving box on service table; the other two-serving box at supply. |
| 3 | A used sponge remains on the table after the wearer reaches the break area; tests delayed omitted-step detection. | Sponge, mug, and towel in sink; phone on shelf; table physically clean. |
| 4 | A third wine and, later, chips violate “two wines, burger, no sides”; requires two interventions. | Exactly two interchangeable wines and burger on packing table; remaining wine and chips at stock. |
| 5 | Dough is about to enter the oven without an X score; tests prerequisite detection and recovery. | Dough cut and in oven; oven closed/running; current bake cycle began only after valid preparation. |
| 6 | A used sponge is confused with an identical clean sponge; tests event-bound identity. | Used sponge in used bin; clean sponge in closed clean cabinet; persistent used/clean attributes agree with history. |
Table 11: Prevention tasks.
| Task | Episode and capability tested | Evaluator validation |
|---|---|---|
| 1 | Inspect labels to distinguish freezer from chiller, then classify five cold-delivery items. | Ice cream/lolly exactly in closed freezer; milk, yogurt, and lettuce exactly in closed chiller (GUIDE StateScore-v2). |
| 2 | Inspect contents of two otherwise ambiguous refrigerators to infer the dairy unit. | Cheese in secured dairy fridge; existing milk remains there, chicken in the other unit, and the other fridge remains closed (GUIDE StateScore-v2). |
| 3 | Track which of two identical glasses received vodka across a phone delay. | Alcohol-free glass on guest table; actual vodka-containing glass on adult counter, using persistent contents rather than hidden identity (WATCH binary). |
| 4 | Search three rooms for a sponge hidden in one of several similar low cabinets when its initial location was never shown. | Search exposes sponge before grab; sponge reaches result counter without hidden-ID navigation. |
| 5 | Retrieve the one of two identical erasers that touched a loose sheet rather than a folder. | Paper-contact eraser, not distractor, visibly acquired and placed in storage cabinet. |
| 6 | Recall the independent locations of two identical coconut-milk bottles. | Garden and fridge bottles directly retrieved to final tray; fridge opened before the hidden bottle is acquired. |
Table 12: Finding tasks.
| Model | Guidance | Planning | Accessibility | Decision | Prevention | Finding |
|---|---|---|---|---|---|---|
| GPT-5.5 | 45% | 50% | 50% | 75% | 70% | 55% |
| GPT-5.6 Sol | 45% | 50% | 60% | 70% | 75% | 50% |
| GPT-5.6 Luna | 35% | 50% | 55% | 60% | 65% | 40% |
| Claude Opus 5 | 40% | 45% | 50% | 70% | 70% | 50% |
| Claude Fable 5.1 | 45% | 65% | 65% | 70% | 75% | 50% |
| Claude Opus 4.8 | 35% | 30% | 25% | 60% | 45% | 40% |
Table 13: Static accuracy across the six task families. Every entry uses a fixed denominator of 20 tasks. Static items without enough evidence are counted as failures.
Figure 8: Performance on Static Adaptations of InteractiveBench inflate 8 models’ scores ; large Static Only (red) and Dynamic Only (blue) regions show a severe evaluation gap.
Figure 9: Complete-task success probability versus normalized reference task horizon across six task families. Each colored curve is a model-specific Firth logistic fit. The red curve is the pooled fit across models, and the gray band is its pointwise task-bootstrap interval. Horizon is normalized separately within each task family. The figure compares different tasks and does not represent progress within a single episode. No monotonicity constraint was applied.
| Model-based Human agent | Human participant | ||||||
|---|---|---|---|---|---|---|---|
| Assistant | Fable 5.1 | Muse 1.1 | GPT-5.6 | Gemini 3.1 | Human 1 | Human 2 | Human 3 |
| Fable 5.1 | 13/36 | 15/36 | 13/36 | 13/36 | 15/36 | 16/36 | 15/36 |
| Muse 1.1 | 9/36 | 9/36 | 8/36 | 9/36 | 10/36 | 11/36 | 9/36 |
| GPT-5.6 Sol | 8/36 | 9/36 | 11/36 | 10/36 | 8/36 | 11/36 | 9/36 |
| Gemini 3.1 | 8/36 | 8/36 | 5/36 | 7/36 | 8/36 | 8/36 | 7/36 |
Table 14: Assistant performance with model-based and human executors. Rows identify the evaluated Assistant; columns identify the executor. Each entry is the number of successful tasks out of the same 36-task subset. Human 1–3 are three individual humans, not three independent task sets.
Figure 10: Task-level outcomes for Fable 5.1 with different executors. Rows are executors and columns are the 36 tasks, grouped into six families. Filled squares indicate success and open squares indicate failure. Task numbers restart within each family. The dashed line separates model-based Human agents from human participants.