A computational framework for understanding embodied plans
Authors: Aalap D. Shah, Basil Saeed, Joshua B. Tenenbaum, Tobias Gerstenberg, Hyowon Gweon, Ilker Yildirim
Organizations: Department of Psychology, Yale University, New Haven, CT, USA. · Department of Statistics, Stanford University, Palo Alto, CA, USA. · Department of Brain & Cognitive Sciences, MIT, Cambridge, MA, USA. · Department of Psychology, Stanford University, Palo Alto, CA, USA. · Wu Tsai Institute, Yale University, New Haven, CT, USA.
Reasoning about people's plans and goals based on their actions is fundamental to human social interaction and essential for cooperative, embodied artificial intelligence. Formalizing this capacity in naturalistic, embodied settings has been a longstanding challenge. Here, we present GenEA, a novel agent representation capable of reasoning over multi-step, long-horizon embodied plans. At the heart of GenEA lies a multilevel state-space that couples the coarse biomechanics of an articulated body and its interactions with nearby objects by extending the task-and-motion planning framework from robotics with simulation-based intuitive physics. By embedding this state-space in a generative model, GenEA yields a joint probability distribution over goals and embodied plans. Across several experiments spanning the qualitatively distinct domains of object manipulation and navigation, featuring diverse tasks and multiple behavioral measures, we show that GenEA, through simple transformations of this joint distribution, recapitulates how humans reason about embodied plans. Crucially, GenEA consistently outperforms strong alternative models, including plausible heuristics, standard task-and-motion planning, and a state-of-the-art vision-language model. GenEA thereby offers a new perspective on the architecture of human reasoning and social cognition, suggesting that embodied artificial intelligence systems seeking to seamlessly interact with humans should adopt GenEA-like generative models of action planning.
Figures & tables
Figure 1: Understanding embodied plans. (a) We can tell that the child’s goal location for slotting the block held in her left hand is the space beside the pink block, and can reason about why she might be holding onto several other pieces between her chin and right hand. (b) We can appreciate that the man’s jump helps him simultaneously achieve several goals, one being clearing the puddle. Reasoning about others’ embodied plans is critical for our everyday physical and social interactions, including knowing when to intervene to safeguard a child or a friend (c) , collaborating with others to form and accomplish shared embodied goals (d) , and making informed decisions about which tasks to undertake by considering task difficulty relative to our own skill level (e) . Images (a-d) were taken either from stock.adobe.com or Freepik. Images in (e) were borrowed from Gweon et al. Gweon et al. (2017) .
Figure 2: Overview of GenEA. (a) At the core of GenEA is a multilevel state-space that couples the coarse biomechanics of an agent β and its interactions with the nearby objects ϕ . Its Symbolic Level f(S) defines an automaton of action primitives for the body (such as picking up an object or jumping across platforms), and how these actions impact the objects in the scene. The Kinematics Level f(K) captures the noisy motion trajectories of the agent’s end-effectors and, correspondingly, of objects being manipulated. The Dynamics Level f(D) updates the objects’ states by simulating gravity and the forces acting between them. (b) GenEA defines a conditional distribution over plans under this multilevel state space. Given a reward, a specified final configuration, GenEA performs classical planning using a task planner, a motion planner, and a forward engine (each corresponding to the three functions of the multilevel state space).
Figure 3: Reasoning queries in GenEA (a) An example action inference trial where the GenEA model is provided with the reward query ϕ(final)∈ξ(final) and two plan constraints: ξ1(constr) representing a one-handed Pick action on the red block P (right hand is idle), and ξ2(constr) representing a two-handed Pick action where the left hand picks the blue block P and a right hand picks the yellow block P . The model is also given access to the initial configuration ϕ(init)∈ξ(init) . (b) A bar plot showing the distribution of plans based on the reward query. Each bar corresponds to a different plan, with textured bars highlighting plans that further satisfy the corresponding plan-constraint query. (c) An illustration of a particularly efficient plan, generated by GenEA, that takes advantage of gravity in achieving the reward configuration. This plan consists of two one-handed actions. The first left-handed action, to Pick the red block, is represented as a red box P paired with a white box marked with a dash indicating that the right hand is idle. Similarly, the second action is depicted as a red box F (denoting a left-handed Fix action to position the red block relative to the blue block), alongside a white box with a dash (denoting the idle right hand)
Figure 4: Explaining social inferences. Experiment 1 comprised two kinds of trials: goal inference and action inference trials. (a) Example stimuli used in the goal inference trials. In these trials, participants viewed an action performed on the initial block configuration and were asked to infer the agent’s intended goal from two possible alternatives. (b) Example stimuli used in the action inference trials. In these trials, participants were shown the initial and final block configurations and were asked to judge which of two candidate actions produced the given final configuration. (c) GenEA strongly correlates with human ratings, reported as the average preference for the option favored by the majority (thus, the axes span half of the scale). The colored data points in the plot correspond to the trials shown in panels (a) and (b), matching their respective frame colors. Note that the y-axis of this scatter plot begins at 50 because any preference for one option logically entails an equivalent lack thereof for the other. (d) GenEA achieves a significantly stronger correlation with participant judgments compared to the alternatives, including TAMP and heuristics models. Error bars indicate bootstrapped 95% confidence intervals; “” for p<.001 . (e) Example stimuli used in Experiment 2 (initial configurations shown in insets), where participants were presented with initial and final block configuration pairs, and asked to report whether the agent who reconfigured the blocks used one hand or two hands. (f) GenEA closely captures human intuitions in the embodiment strategies task. The colored data points in the plot correspond to the trials shown in panel (e). (g) GenEA achieves a significantly stronger correlation with participant judgments compared to alternatives, including TAMP and a heuristic model. Error bars indicate bootstrapped 95% confidence intervals; “”for p<.001 .
Figure 5: Explaining human response times. (a) A scatter plot of GenEA’s correlation with human response times in the action and goal inference trials from Experiment 1 . The greater the rate of evidence accumulation predicted by GenEA, the shorter people took to respond. (b) GenEA explains significantly more variance than what TAMP and humans’ preference ratings can explain in response times. (c) A scatter plot of GenEA’s correlation with human response times in the embodiment strategy inference trials from Experiment 2 . The greater the rate of evidence accumulation predicted by GenEA, the shorter people take to respond. (d) GenEA explains a similar amount of variance as TAMP and more than what can be explained using participants’ preference ratings.
Figure 6: Testing the generality of GenEA in the domain of navigation. (a) Example stimuli from the navigation task of Experiment 3 , showing an agent standing on the left-most “start” platform, one or more intermediate platforms and the “final” platform to the extreme right. On each trial of this experiment, participants reported their judgment of how difficult it would be for the agent to reach the right-most platform. (b) GenEA closely captures people’s difficulty judgments ( r=.91,p<.001 ). Colored dots correspond to the data points of the trials visualized in panel (a). (c) GenEA explains more variance than alternatives, including TAMP and heuristic (stability) models. (d) Both GenEA as well as humans, but not TAMP, show sensitivity to the material composition of the intermediate platforms, with both judging trials with wooden platforms to be on average more difficult to traverse than those with stone platforms.
Supplementary Fig. 1 : GenEA is robust to the choice of the cost function that parametrizes the distribution over plans, Pr (π∣R,ξ) . This figure shows bootstrapped correlations between GenEA and human performance across all experiments we reported in the main text, and compares to standard TAMP. Irrespective of the function we used to compute the cost, the GenEA model outperformed TAMP-based agent representation. Error bars in this figure indicate 95% bootstrapped confidence intervals; ”***” for p<.001 .
Supplementary Fig. 2 : Our results are not sensitive to the choice of the temperature parameter κ . This plot shows that for each cost function we considered, and across all temperature values (x-axis), GenEA consistently outperforms the corresponding TAMP-based model. The dotted lines in the plot indicate the temperature values we used for each cost function to generate the results shown in Supplementary Fig. 1 : in Experiment 1 , κ=3.1 for Kinetic energy (also the value we used in main text), κ=4.4 for plan length, and κ=2.5 for sum of squared errors; and in Experiment 2 , , κ=3.5 for Kinetic energy (also the value we used in main text), κ=2.4 for plan length, and κ=1.1 for sum of squared errors
Supplementary Fig. 3 : All goal inference stimuli used in Experiment 1 .
Supplementary Fig. 4 : All action inference stimuli used in Experiment 1 .
Supplementary Fig. 5 : All stimuli used in Experiment 2 .
Supplementary Fig. 6 : The stimuli set from Experiment 3 .
Supplementary Fig. 7 : Platforms used in the Experiment 3 stimuli set, along with their dimensions. The platforms are depicted with a wooden material composition, but in the actual stimuli set they could also appear as though made of stone.
Supplementary Fig. 8 : (a) Each column shows example goal inference trials from Experiment 1 . In each column, the top row displays the initial block configuration ϕ(init) , the second and third rows show the reward query QR={ϕ1(final),ϕ2(final)} . Note that Qπ={ξ(init)} . (b) We use this caption to provide qualitative insight as to how GenEA aligns with human judgments. The scatter plot is reproduced from the main text (Fig. 4) for reader’s reference, with the colored dots corresponding to the trials shown on panel (a). As a reminder, for both the participants and the model, small values indicate preference toward the more optimal goal, whereas larger values indicate ambivalence between the two options. Here, the only difference between the “golden” (left-most column) and “brown” (middle column) trials is in their respective final configurations ϕ1(final) — the positions of the top two blocks in the stack are swapped between the two reward configurations. As a result, it is only in the “golden” trial that the final location of the green block, picked up during the first action, is immediately available. The results suggest that the judgments of both the human participants as well as GenEA are affected by this disparity. While they both show a strong preference for ϕ2(final) , the presence of a relatively less plausible alternative in the “golden” trial (i.e., ϕ1(final) ) introduces greater ambiguity for this trial as compared to the “brown” trial. The “pink” trial is identical to the “golden” trial in every way except the initial action taken by the agent. With a two-handed action available in the “pink” trial, both options become equally plausible, and we see that the preferences of the human participants as well as GenEA strongly reflect this ambivalence.
Supplementary Fig. 9 : (a) Each column shows an example action inference trials from Experiment 1 . In each column, the top row displays an initial configuration ϕ(init) , the middle row shows a final configuration QR={ϕ(final)} , and the third row shows the two possible first actions, from the two query constraints Qπ={ξ1(constr),ξ2(constr)} , the agent may take on the initial block configuration. Notice that for all three example trials shown in this figure, the action options are identical: (i) a two-handed action involving picking the black block with the left hand and picking the yellow block with the right hand; (ii) a one-handed action involving picking the yellow block with the right hand (i.e., no action is scheduled on the left hand). (b) The scatter plot, reproduced from the main text (Fig. 4) for reference, with the colored dots corresponding to the trials shown on panel (a). As a reminder, for both the participants and the model, larger values on either axis indicate preference toward the more optimal action, whereas smaller values indicate increasing ambivalence between the two action options. The “purple” and the “black” trials have the same initial conditions but differ markedly in their final configurations. In the “purple” trial, merely moving the yellow block allows the black block to stand upright (due to gravity), and achieve the final position in the process. Consequently, for the “purple” trial, both humans and GenEA showed some ambivalence with only a slight preference for the one-handed action which leverages gravity. In contrast, for the “black” trial, both humans and GenEA strongly preferred the more efficient two-handed action. The “orange” trial introduces a more nuanced variation to the “purple” trial with the only difference being the tilt angle of the black block in the initial configuration. Unlike the “purple” trial, where gravity can assist the black block in reaching its final position, the “orange” trial features a larger tilt angle, necessitating a two-handed action to accomplish the task efficiently. We see in the plot that, in this trial as well, both humans and GenEA exhibited a strong preference for the two-handed action.
Supplementary Fig. 10 : (a) Each image shows an example trial in Experiment 2 , with an initial configuration (inset) ϕ(init) and final configuration QR={ϕ(final)} . The two query constraints are Qπ={ξ1(constr),ξ2(constr)} , plans using one-hand and plans using two-hands, respectively. (b) The scatter plot, reproduced from the main text (Fig. 5) for reference, where the colored dots correspond to the trials shown on panel (a). As a reminder, for both the participants and the model, smaller values imply preference toward one-hand, whereas larger values imply preference toward two-hand. The cluster of points in the lower left corner (e.g., the “blue” trial) represent trials where only one block was moved in the initial-final configuration pair, and both humans and GenEA show preference for one-hand. The trials in the top-right corner also demonstrate a high degree of alignment between humans and GenEA. These trials involve configurations where the blocks’ support structure necessitates using both hands during their construction. Take, for instance, the “green” trial where the black block cannot be held in its place without the yellow block’s support. To prevent the black block from toppling, the yellow block must be “Fixed” simultaneously. In some ways, it is the trials situated toward the center of the plot that most clearly reveal GenEA’s strengths. Typically, a two-hand plans are more efficient than their one-handed counterparts: they involve fewer moves and incur lower execution costs. Such trials are displayed in the right half (for the model) and upper half (for humans) of the scatter plot. In the “golden” trial, for instance, which involves swapping the positions of two vertically stacked blocks, a two-handed solution is clearly more efficient and therefore also more preferable. This advantage is less pronounced in the “purple” trial, which, like the “golden” trial, also involves swapping but presents fewer positional constraints since the blocks to be swapped are positioned on the table. In the “pink” trial, whose final configuration can be conveniently achieved using either one-handed or two-handed solutions, both humans as well as GenEA show the most ambivalence, with only a slight preference for the more efficient two-handed solution. Our stimuli set also includes trials where leveraging external forces, such as gravity, can make one-handed plans more efficient. For instance, in the “orange” trial, moving the yellow block causes the black block to fall into its final position. Such trials lie in the lower-left quadrant of the plot.
Supplementary Fig. 11 : (a) Each image shows the initial agent and object configuration for a representative trial from Experiment 3 . (b) We use this caption to provide qualitative insight into how GenEA’s judgments aligns with those of human participants in Experiment 3 . The scatter plot, reproduced from the main text ( Fig. 6 ), shows colored dots corresponding to the trials depicted in panel (a). As a reminder, smaller values indicate easier trials for the participants (and lower cost under GenEA), while higher values indicate harder trials. Both humans and GenEA consistently judge configurations that can be traversed by walking as easier than those requiring jumps. Similarly, they are both sensitive to the material composition of the platforms, judging configurations with wooden platforms as more difficult than otherwise identical configurations with stone platforms, as is evident in the orange versus green and purple versus blue trial pairs. The gold trial illustrates that platform count and material alone do not determine task difficulty. Although this trial contains the most intermediate platforms, all wooden and tapered at the base in a manner that renders them precarious, their close positioning affords a walkable path, and the trial is accordingly judged among the least difficult to traverse. Both humans and GenEA further show that, in configurations admitting multiple candidate paths, difficulty judgments implicitly depend on first inferring which path the agent would take. In the pink trial, for example, a single long, stable platform affords a walkable route, and both humans and GenEA ground their difficulty judgments in this route rather than in the two comparatively unstable platforms in the foreground. Accordingly, both humans and GenEA judge the pink trial among the easiest. The blue and purple trials offer only a small landing area and as a result are regarded as the most difficult configurations by both humans and GenEA. Notably, this agreement between humans and GenEA is not confined to the extremes of the difficulty scale; but extends to configurations of intermediate difficulty as well.
Supplementary Fig. 12 : VLM prompts used for the action trials of Experiment 1 .
Supplementary Fig. 13 : VLM prompts used for the goal trials of Experiment 1 .
Supplementary Fig. 14 : VLM prompts used for Experiment 2 .
Supplementary Fig. 15 : VLM prompts used for trials in Experiment 3 .