cs.AIMar 16, 2026
SaveAsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback
Organizations: Microsoft Research
Abstract
With AsgardBench we aim to evaluate visually grounded, high-level action sequence generation and interactive planning, focusing specifically on plan adaptation during execution based on visual observations rather than navigation or low-level manipulation. In the landscape of embodied AI benchmarks, AsgardBench targets the capability category of interactive planning, which is more sophisticated than offline high-level planning as it requires agents to revise plans in response to environmental feedback, yet remains distinct from low-level execution. Unlike prior embodied AI benchmarks that conflate reasoning with navigation or provide rich corrective feedback that substitutes for perception, AsgardBench restricts agent input to images, action history, and lightweight success/failure signals, isolating interactive planning in a controlled simulator without low-level control noise. The benchmark contains 108 task instances spanning 12 task types, each systematically varied through object state, placement, and scene configuration. These controlled variations create conditional branches in which a single instruction can require different action sequences depending on what the agent observes, emphasizing conditional branching and plan repair during execution. Our evaluations of leading vision language models show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking that ultimately undermine interactive planning. Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
Figures & tables
Figure 1: Agent observations and corresponding action plans in AsgardBench. Below each image is the action plan generated from that observation. This illustrates how AsgardBench requires agents to update or change their plans based on new visual evidence rather than following a fixed action sequence.
| Task Type | Scene Type | # Tasks | Task Variations |
|---|---|---|---|
| “Consume coffee from a mug, then wash and store the mug” | Kitchen | 9 | • Vary whether the mug is clean or dirty • Vary the initial fill state of the mug (empty, has coffee) |
| “Microwave a potato and serve it in a bowl” | Kitchen | 6 | • Requires a bowl to serve • Vary whether the bowl is dirty |
| “Cook an egg in a pan and serve it on a plate” | Kitchen | 12 | • Requires a pan to cook and a plate to serve • Vary whether the pan and plate are dirty/clean |
| “Make a slice of toast and serve it on a plate” | Kitchen | 6 | • Vary whether the plate is clean or dirty |
| “Fry a potato slice and serve it on a plate” | Kitchen | 12 | • Vary whether some or all items are clean or dirty |
| “Put away the {Food}” | Kitchen | 3 | • Vary which items are clean/dirty |
Table 1: Twelve Task types are included in the AsgardBench benchmark. The # Tasks column indicates the number of task instances in the 108-task test set for each type. In addition to the task-specific variations, object locations were randomized for all Task types, including the contents of the SinkBasin for Kitchen scenes.
Figure 2: Success rates for each model under image based and Text-Only conditions. Visual input substantially improves performance for all but the weakest models, confirming that AsgardBench requires perception-conditioned reasoning. Agents can’t rely on memorized action templates or detailed feedback. Text-Only performance remains low across models, in contrast to prior embodied benchmarks where Text-Only agents can perform competitively.
| No Feedback | Success / Failure (Baseline) | Detailed Feedback |
|---|---|---|
| 1. FIND Plate 2. FIND Pan 3. PICKUP Egg 4. PICKUP Mug 5. CLEAN Mug | 1. FIND Plate Success 2. FIND Pan Success 3. PICKUP Egg Failure 4. PICKUP Mug Success 5. CLEAN Mug Failure | 1. FIND Plate Success 2. FIND Pan Success 3. PICKUP Egg Failure: Cannot pick up Egg as it is not visible. 4. PICKUP Mug Success 5. CLEAN Mug Failure: Mug must be in the SinkBasin to clean. |
Table 2: Past-actions prompt under different feedback conditions. The left column shows the prompt when the agent receives No Feedback , the center column shows the prompt with Simple Feedback , and the right column shows the prompt with Detailed Feedback .
Figure 3: Relationship between task difficulty and the range of steps required for successful completion. We found that tasks that require longer or more variable action sequences tend to have lower success rates, indicating that models struggle with long-horizon dependencies and conditional branching. Each point represents one of the 108 tasks in AsgardBench.
Figure 4: Effect of feedback type on model performance. Removing success/failure signals ( No Feedback ) reduces accuracy, while providing detailed error messages ( Detailed Feedback ) sharply increases performance, including for Text-Only agents.
Figure 5: Model performance (blue) and percentage of undoable actions (orange). Higher performing models produce fewer undoable actions, while weaker models generate a larger fraction of actions that cannot be executed. This alignment between success rate and undoable action frequency suggests how difficulties in state tracking and plan adjustment contribute directly to overall task failure.
Figure 6: Termination reasons. Less capable models frequently terminate due to repeated actions/action sequences or consecutive failures, indicating that they become stuck in narrow behavior loops. More capable models terminate primarily due to step limits, reflecting longer but still unsuccessful attempts at plan execution.
| Observation | Model Output |
|---|---|
| From the image, I can see that the Pan is already on the StoveBurner, and the StoveBurner is turned on because the Pan is visibly on fire. This suggests that the Pan may already be hot, and I need to proceed carefully. | |
| Looking at the scene, I see the Mug in the SinkBasin, which appears clean. |
Figure 7: Examples of visual misinterpretations made by GPT-4o. Models occasionally confuse reflections, shadows, or clutter for task-relevant objects, leading to incorrect state inferences and subsequent planning errors.
Figure 8: Ambiguity in interpreting held objects without the hand overlay. In both images, the saltshaker is being held by the agent, but without the superimposed hand overlay it can easily be misinterpreted as resting on the table. This illustrates how subtle spatial cues in AI2-THOR can lead to incorrect state inferences, motivating the inclusion of the hand overlay in AsgardBench.
Figure 9: Inventory check prompt instructions with and without hand overlay.
Figure 10: Effect of removing the hand overlay on model performance. The blue bars show baseline performance with the hand overlay included, while the orange bars show performance when the overlay is removed. The drop in accuracy highlights how models rely on the hand cue to correctly infer when an object is being held.
| Observation | Model Output |
|---|---|
| <think> First, I check the bottom center of the image for any floating object — I see nothing floating, so I am NOT holding anything. I see a CoffeeMachine, Toaster, Mug (with coffee) and a green object (likely a Lettuce or another vegetable) on the DiningTable… |
Figure 11: Example of a held object being misinterpreted as resting on the table. Model incorrectly infers that the coffee mug is on the table, even though it is being held by the agent. This illustrates how the absence of clear hand cues can lead to incorrect state estimation in cluttered scenes.
| Observation | Things to Remember |
|---|---|
| (Claude) • Completed: Found Knife ✓, Picked up Knife ✓, Sliced Tomato ✓ • Current: Put down Knife to free hand • Remaining: Find Plate, Put TomatoSliced on Plate • TomatoSliced is on CounterTop next to SinkBasin • Plate location unknown — likely in Cabinet | |
| (Qwen) • Completed: Sliced Tomato ✓ • Current: Put down Knife to pick up TomatoSliced • Remaining: Pick up TomatoSliced, Put TomatoSliced in Plate • Constraint: Must put down Knife before picking up TomatoSliced (one item at a time) | |
| (Llama) • Completed: Sliced Tomato ✓ • Current: Find TomatoSliced • Remaining: Put TomatoSliced on Plate |
Figure 12: Things to Remember prompts for different models. The top, middle, and bottom rows show the memory-scaffold prompts used for Claude, Qwen, and Llama, respectively. These examples illustrate how each model is provided with a structured summary of past actions and observations, enabling it to maintain context across steps.
Figure 13: Baseline performance (blue) compared to performance without the Things to Remember scaffold (orange). Removing the memory scaffold produced mixed results: more capable models appeared to benefit from the additional information, while less capable models performed worse without it.
Figure 14: Image guidance prompt with dual-image baseline vs. single current-state image.
Figure 15: Effect of providing previous-state images on model performance. The blue bars show performance when the agent is given both the current visual observation and the previous visual observation. The orange bars show performance when only the current observation image is provided. This comparison highlights how access to a short visual history can influence model behavior.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Unsliced | Sliced | With Nudge |
Figure 16: Adding small positional nudges to sliced items to improve visibility. Sliced objects in AI2-THOR often overlap or collapse into nearly indistinguishable configurations. Adding slight positional nudges after slicing makes the resulting pieces more visually separable, reducing ambiguity.
| Scene | Goals |
| Kitchen | 1. CoffeeMachine must be off 2. Toaster must be off 3. StoveBurner must be off 4. Microwave must be closed 5. Fridge must be closed 6. Faucet must be off |
Table 3: Kitchen-specific goals shared across all tasks. Because every kitchen task requires leaving the environment in a reasonable state, all tasks in kitchen scenes share a common set of cleanup-oriented goals. This table lists those goals, which ensure that agents not only complete the primary task but also restore the scene to an acceptable post-task configuration.
| Task | Scene | Goals |
|---|---|---|
| Microwave a potato and serve it in a bowl | Kitchen | • Potato must be cooked • Potato must be in a Bowl • Bowl must be clean if it starts dirty |
| Cook an egg in a pan and serve it on a plate | Kitchen | • Egg must be cooked • EggCracked must be on a Plate • Pan must be clean if it starts dirty • Plate must be clean if it starts dirty |
| Make a slice of toast and serve it on a plate | Kitchen | • Bread must be cooked/toasted • BreadSliced must be on a Plate • Plate must be clean if it starts dirty |
| Fry a potato slice and serve it on a plate | Kitchen | • PotatoSliced must be cooked • PotatoSliced must be on a Plate • Pan must be clean if it starts dirty • Plate must be clean if it starts dirty |
| Put away the {Food} (e.g., Apple, Bread, Egg, Lettuce, Potato, Tomato) | Kitchen | • Each specified food item must be put away (stored in Fridge) • Items must be clean if they start dirty |
| Put away the {Dishes} (e.g., Bowl, Cup, Mug, Plate, Pot) | Kitchen | • Each specified dish must be put away (stored in Cabinet) • Items must be clean if they start dirty |
Table 4: Task-specific goals for each benchmark scenario. In addition to the shared kitchen goals listed in Table 3 , each task includes its own task-specific goals that define the required object states or interactions needed to complete that scenario. This table lists those per-task goals, which capture the unique requirements and success conditions for each individual task.
Figure 17: Distribution of undoable-action error types across models in AsgardBench. Each bar shows the proportion of undoable actions attributed to different failure categories (e.g., holding state errors, visibility errors, incompatibility errors). Higher performing models exhibit fewer holding-state errors, while visibility related errors remain relatively consistent across models.
Figure 18: Conditional branching from a single instruction in AsgardBench. Because objects are initialized in different configurations (e.g., clean vs. dirty) and placed in different positions, a single Task instruction requires distinct action plans for each initialization, even within the same Scene. Shown here is the Task "Consume coffee from a mug, then wash and store the mug" in "FloorPlan3", where each initial state leads to a different valid execution sequence.
| Model | Provider | Max tokens |
|---|---|---|
| GPT-4o | Azure Foundry | 4096 |
| GPT-5.2 | Azure Foundry | 16384 |
| Kimi-K2.5 | Azure Foundry | 16384 |
| Claude Opus 4.5 | OpenRouter | 16384 |
| Gemini 3 Pro Preview | OpenRouter | 16384 |
| Qwen3-VL-235B-A22B (Thinking) | OpenRouter | 16384 |
Table 5: Inference settings used in evaluation. All models were evaluated with temperature set to 0. For models that support a reasoning-effort setting, we always use the high level.
| Model | Baseline | Detailed | No Feedback | No Hand | No Memory | No Prev. Image | Text Only ( ) | Text Only ( ) | Text Only ( ) |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.5 | 76.5 | – | – | – | – | 77.8 | 36.1 | – | – |
| Gemini 3 Pro | 72.5 | – | – | – | – | 78.7 | 35.2 | – | – |
| GPT-5.2 | 71.3 | – | – | – | – | – | 25.0 | – | – |
| Kimi-K2.5 | 68.8 | 66.4 | 55.2 | 61.4 | 61.1 | 64.8 | 21.9 | 41.4 | 14.2 |
| Qwen3-VL-235B | 32.4 | 35.2 | 26.9 | 24.1 | 29.6 | 31.5 | 13.9 | 38.3 | 8.3 |
| GLM-4.6V | 30.6 | 36.1 | 23.1 | 15.7 | 27.8 | 23.1 | 8.3 | 13.9 | 2.8 |
Table 6: Median success rates (%) across all experimental conditions. All values are medians over N independent runs (see Table 5 for per-model run counts). “–” indicates conditions not evaluated for that model due to budget constraints. The three Text Only conditions use text-only observations (no images) with different feedback modes: = simple feedback, = detailed feedback, = no feedback.
| Observation | Model Output |
|---|---|
| I am holding something green at the bottom center of the view; it looks like a Stool seat/back. Based on the visible object in-hand, I am holding the Stool. | |
| A Mug is visible in my hand at the bottom center; it contains a light-brown liquid (coffee). | |
| I do not see a Toaster or a Plate in this view. | |
| I do not see a Plate, Pan, or StoveBurner in this view. |
Figure 19: Examples of GPT-5.2 visual interpretation errors. Each row shows a visual observation (left) paired with the model’s description of what it sees (right). These examples highlight common failure modes where GPT-5.2 misidentifies object states or scene configurations despite clear visual evidence.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,494 papers