With AsgardBench we aim to evaluate visually grounded, high-level action sequence generation and interactive planning, focusing specifically on plan adaptation during execution based on visual observations rather than navigation or low-level manipulation. In the landscape of embodied AI benchmarks, AsgardBench targets the capability category of interactive planning, which is more sophisticated than offline high-level planning as it requires agents to revise plans in response to environmental feedback, yet remains distinct from low-level execution. Unlike prior embodied AI benchmarks that conflate reasoning with navigation or provide rich corrective feedback that substitutes for perception, AsgardBench restricts agent input to images, action history, and lightweight success/failure signals, isolating interactive planning in a controlled simulator without low-level control noise. The benchmark contains 108 task instances spanning 12 task types, each systematically varied through object state, placement, and scene configuration. These controlled variations create conditional branches in which a single instruction can require different action sequences depending on what the agent observes, emphasizing conditional branching and plan repair during execution. Our evaluations of leading vision language models show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking that ultimately undermine interactive planning. Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
Figures & tables
Figure 1: Agent observations and corresponding action plans in AsgardBench. Below each image is the action plan generated from that observation. This illustrates how AsgardBench requires agents to update or change their plans based on new visual evidence rather than following a fixed action sequence.
Task Type
Scene Type
# Tasks
Task Variations
“Consume coffee from a mug, then wash and store the mug”
Kitchen
9
• Vary whether the mug is clean or dirty • Vary the initial fill state of the mug (empty, has coffee)
“Microwave a potato and serve it in a bowl”
Kitchen
6
• Requires a bowl to serve • Vary whether the bowl is dirty
“Cook an egg in a pan and serve it on a plate”
Kitchen
12
• Requires a pan to cook and a plate to serve • Vary whether the pan and plate are dirty/clean
“Make a slice of toast and serve it on a plate”
Kitchen
6
• Vary whether the plate is clean or dirty
“Fry a potato slice and serve it on a plate”
Kitchen
12
• Vary whether some or all items are clean or dirty
“Put away the {Food}”
Kitchen
3
• Vary which items are clean/dirty
Table 1: Twelve Task types are included in the AsgardBench benchmark. The # Tasks column indicates the number of task instances in the 108-task test set for each type. In addition to the task-specific variations, object locations were randomized for all Task types, including the contents of the SinkBasin for Kitchen scenes.
Figure 2: Success rates for each model under image based and Text-Only conditions. Visual input substantially improves performance for all but the weakest models, confirming that AsgardBench requires perception-conditioned reasoning. Agents can’t rely on memorized action templates or detailed feedback. Text-Only performance remains low across models, in contrast to prior embodied benchmarks where Text-Only agents can perform competitively.
1. FIND Plate Success 2. FIND Pan Success 3. PICKUP Egg Failure: Cannot pick up Egg as it is not visible. 4. PICKUP Mug Success 5. CLEAN Mug Failure: Mug must be in the SinkBasin to clean.
Table 2: Past-actions prompt under different feedback conditions. The left column shows the prompt when the agent receives No Feedback , the center column shows the prompt with Simple Feedback , and the right column shows the prompt with Detailed Feedback .
Figure 3: Relationship between task difficulty and the range of steps required for successful completion. We found that tasks that require longer or more variable action sequences tend to have lower success rates, indicating that models struggle with long-horizon dependencies and conditional branching. Each point represents one of the 108 tasks in AsgardBench.
Figure 4: Effect of feedback type on model performance. Removing success/failure signals ( No Feedback ) reduces accuracy, while providing detailed error messages ( Detailed Feedback ) sharply increases performance, including for Text-Only agents.
Figure 5: Model performance (blue) and percentage of undoable actions (orange). Higher performing models produce fewer undoable actions, while weaker models generate a larger fraction of actions that cannot be executed. This alignment between success rate and undoable action frequency suggests how difficulties in state tracking and plan adjustment contribute directly to overall task failure.
Figure 6: Termination reasons. Less capable models frequently terminate due to repeated actions/action sequences or consecutive failures, indicating that they become stuck in narrow behavior loops. More capable models terminate primarily due to step limits, reflecting longer but still unsuccessful attempts at plan execution.
Observation
Model Output
From the image, I can see that the Pan is already on the StoveBurner, and the StoveBurner is turned on because the Pan is visibly on fire. This suggests that the Pan may already be hot, and I need to proceed carefully.
Looking at the scene, I see the Mug in the SinkBasin, which appears clean.
Figure 7: Examples of visual misinterpretations made by GPT-4o. Models occasionally confuse reflections, shadows, or clutter for task-relevant objects, leading to incorrect state inferences and subsequent planning errors.
Figure 8: Ambiguity in interpreting held objects without the hand overlay. In both images, the saltshaker is being held by the agent, but without the superimposed hand overlay it can easily be misinterpreted as resting on the table. This illustrates how subtle spatial cues in AI2-THOR can lead to incorrect state inferences, motivating the inclusion of the hand overlay in AsgardBench.
Figure 9: Inventory check prompt instructions with and without hand overlay.
Figure 10: Effect of removing the hand overlay on model performance. The blue bars show baseline performance with the hand overlay included, while the orange bars show performance when the overlay is removed. The drop in accuracy highlights how models rely on the hand cue to correctly infer when an object is being held.
Observation
Model Output
<think> First, I check the bottom center of the image for any floating object — I see nothing floating, so I am NOT holding anything. I see a CoffeeMachine, Toaster, Mug (with coffee) and a green object (likely a Lettuce or another vegetable) on the DiningTable…
Figure 11: Example of a held object being misinterpreted as resting on the table. Model incorrectly infers that the coffee mug is on the table, even though it is being held by the agent. This illustrates how the absence of clear hand cues can lead to incorrect state estimation in cluttered scenes.
Observation
Things to Remember
(Claude) • Completed: Found Knife ✓, Picked up Knife ✓, Sliced Tomato ✓ • Current: Put down Knife to free hand • Remaining: Find Plate, Put TomatoSliced on Plate • TomatoSliced is on CounterTop next to SinkBasin • Plate location unknown — likely in Cabinet
(Qwen) • Completed: Sliced Tomato ✓ • Current: Put down Knife to pick up TomatoSliced • Remaining: Pick up TomatoSliced, Put TomatoSliced in Plate • Constraint: Must put down Knife before picking up TomatoSliced (one item at a time)
(Llama) • Completed: Sliced Tomato ✓ • Current: Find TomatoSliced • Remaining: Put TomatoSliced on Plate
Figure 12: Things to Remember prompts for different models. The top, middle, and bottom rows show the memory-scaffold prompts used for Claude, Qwen, and Llama, respectively. These examples illustrate how each model is provided with a structured summary of past actions and observations, enabling it to maintain context across steps.
Figure 13: Baseline performance (blue) compared to performance without the Things to Remember scaffold (orange). Removing the memory scaffold produced mixed results: more capable models appeared to benefit from the additional information, while less capable models performed worse without it.
Figure 14: Image guidance prompt with dual-image baseline vs. single current-state image.
Figure 15: Effect of providing previous-state images on model performance. The blue bars show performance when the agent is given both the current visual observation and the previous visual observation. The orange bars show performance when only the current observation image is provided. This comparison highlights how access to a short visual history can influence model behavior.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Unsliced
Sliced
With Nudge
Appendix
Figure 16: Adding small positional nudges to sliced items to improve visibility. Sliced objects in AI2-THOR often overlap or collapse into nearly indistinguishable configurations. Adding slight positional nudges after slicing makes the resulting pieces more visually separable, reducing ambiguity.
Scene
Goals
Kitchen
1. CoffeeMachine must be off 2. Toaster must be off 3. StoveBurner must be off 4. Microwave must be closed 5. Fridge must be closed 6. Faucet must be off
Appendix
Table 3: Kitchen-specific goals shared across all tasks. Because every kitchen task requires leaving the environment in a reasonable state, all tasks in kitchen scenes share a common set of cleanup-oriented goals. This table lists those goals, which ensure that agents not only complete the primary task but also restore the scene to an acceptable post-task configuration.
Task
Scene
Goals
Microwave a potato and serve it in a bowl
Kitchen
• Potato must be cooked • Potato must be in a Bowl • Bowl must be clean if it starts dirty
Cook an egg in a pan and serve it on a plate
Kitchen
• Egg must be cooked • EggCracked must be on a Plate • Pan must be clean if it starts dirty • Plate must be clean if it starts dirty
Make a slice of toast and serve it on a plate
Kitchen
• Bread must be cooked/toasted • BreadSliced must be on a Plate • Plate must be clean if it starts dirty
Fry a potato slice and serve it on a plate
Kitchen
• PotatoSliced must be cooked • PotatoSliced must be on a Plate • Pan must be clean if it starts dirty • Plate must be clean if it starts dirty
Put away the {Food} (e.g., Apple, Bread, Egg, Lettuce, Potato, Tomato)
Kitchen
• Each specified food item must be put away (stored in Fridge) • Items must be clean if they start dirty
Put away the {Dishes} (e.g., Bowl, Cup, Mug, Plate, Pot)
Kitchen
• Each specified dish must be put away (stored in Cabinet) • Items must be clean if they start dirty
Appendix
Table 4: Task-specific goals for each benchmark scenario. In addition to the shared kitchen goals listed in Table 3 , each task includes its own task-specific goals that define the required object states or interactions needed to complete that scenario. This table lists those per-task goals, which capture the unique requirements and success conditions for each individual task.
Figure 17: Distribution of undoable-action error types across models in AsgardBench. Each bar shows the proportion of undoable actions attributed to different failure categories (e.g., holding state errors, visibility errors, incompatibility errors). Higher performing models exhibit fewer holding-state errors, while visibility related errors remain relatively consistent across models.
Figure 18: Conditional branching from a single instruction in AsgardBench. Because objects are initialized in different configurations (e.g., clean vs. dirty) and placed in different positions, a single Task instruction requires distinct action plans for each initialization, even within the same Scene. Shown here is the Task "Consume coffee from a mug, then wash and store the mug" in "FloorPlan3", where each initial state leads to a different valid execution sequence.
Model
Provider
Max tokens
GPT-4o
Azure Foundry
4096
GPT-5.2
Azure Foundry
16384
Kimi-K2.5
Azure Foundry
16384
Claude Opus 4.5
OpenRouter
16384
Gemini 3 Pro Preview
OpenRouter
16384
Qwen3-VL-235B-A22B (Thinking)
OpenRouter
16384
Appendix
Table 5: Inference settings used in evaluation. All models were evaluated with temperature set to 0. For models that support a reasoning-effort setting, we always use the high level.
Model
Baseline
Detailed
No Feedback
No Hand
No Memory
No Prev. Image
Text Only ( Fs )
Text Only ( Fd )
Text Only ( Fn )
Claude Opus 4.5
76.5
–
–
–
–
77.8
36.1
–
–
Gemini 3 Pro
72.5
–
–
–
–
78.7
35.2
–
–
GPT-5.2
71.3
–
–
–
–
–
25.0
–
–
Kimi-K2.5
68.8
66.4
55.2
61.4
61.1
64.8
21.9
41.4
14.2
Qwen3-VL-235B
32.4
35.2
26.9
24.1
29.6
31.5
13.9
38.3
8.3
GLM-4.6V
30.6
36.1
23.1
15.7
27.8
23.1
8.3
13.9
2.8
Appendix
Table 6: Median success rates (%) across all experimental conditions. All values are medians over N independent runs (see Table 5 for per-model run counts). “–” indicates conditions not evaluated for that model due to budget constraints. The three Text Only conditions use text-only observations (no images) with different feedback modes: Fs = simple feedback, Fd = detailed feedback, Fn = no feedback.
Observation
Model Output
I am holding something green at the bottom center of the view; it looks like a Stool seat/back. Based on the visible object in-hand, I am holding the Stool.
A Mug is visible in my hand at the bottom center; it contains a light-brown liquid (coffee).
I do not see a Toaster or a Plate in this view.
I do not see a Plate, Pan, or StoveBurner in this view.
Appendix
Figure 19: Examples of GPT-5.2 visual interpretation errors. Each row shows a visual observation (left) paired with the model’s description of what it sees (right). These examples highlight common failure modes where GPT-5.2 misidentifies object states or scene configurations despite clear visual evidence.
Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution. We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and assessing their potential as decision makers for physical tasks. Our benchmark and code are available at https://github.com/MinneFeng/Plan-Bench.
Juntong Feng, Qingan Liu, Jingru Dong +8
Dalian University of Technology · Nanyang Technological University
Intelligent embodied agents should not simply follow instructions, as real-world environments often involve unexpected conditions and exceptions. However, existing methods usually focus on directly executing instructions, without considering whether the target objects can actually be manipulated, meaning they fail to assess available affordances. To address this limitation, we introduce DynAfford, a benchmark that evaluates embodied agents in dynamic environments where object affordances may change over time and are not specified in the instruction. DynAfford requires agents to perceive object states, infer implicit preconditions, and adapt their actions accordingly. To enable this capability, we introduce ADAPT, a plug-and-play module that augments existing planners with explicit affordance reasoning. Experiments demonstrate that incorporating ADAPT significantly improves robustness and task success across both seen and unseen environments. We also show that a domain-adapted, LoRA-finetuned vision-language model used as the affordance inference backend outperforms a commercial LLM (GPT-4o), highlighting the importance of task-aligned affordance grounding.
Pei-An Chen, Yong-Ching Liang, Jia-Fong Yeh +4
National Taiwan University · National Yang Ming Chiao Tung University · National Tsing Hua University
Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing benchmarks still underexplore adaptive planning under such progressively revealed dual constraints. To address this gap, we introduce AdaPlanBench, a dynamic interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints. At runtime, agents interact with the environment in a multi-turn protocol where hidden constraints are revealed only when the agent proposes a plan that violates them, requiring iterative plan revision under accumulating feedback. This makes planning challenging, as agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM agents.