Robots Influencing Humans to Reveal their Goals during Collaboration and Competition
Authors: Debasmita Ghose, Oz Gitelson, Michal Lewkowicz, Jake Brawer, Marynel Vazquez, Brian Scassellati
Organizations: Department of Computer Science, Yale University, 51 Prospect Street, New Haven, 06511, CT, USA. · Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, 125 Massachusetts Ave, Cambridge, 02139, MA, USA. · Department of Computer Science, University of Colorado, Boulder, 430 UCB, 1111 Engineering Dr, Boulder, 80309, CO, USA.
We propose a unified strategy for fast goal inference in human-robot interaction. The core idea is to drive the human toward Critical Decision Points (CDPs)-states where competing human strategies prescribe different next actions and thus maximally reveal the goal. We formalise CDPs using a goal-conditioned policy divergence measure and incorporate them into a Receding-Horizon Planner that explores future action sequences while optimizing a cost function balancing task progress and information gain. We evaluate this approach in both a collaborative, fully observable cooking task and a competitive, partially observable hide-and-seek game, each in simulation and on real robots. In both scenarios, our method infers human goals more accurately and earlier than baseline strategies.
Figures & tables
Figure 1: Critical Decision Point Computation . For each state, the robot computes how much the action distributions induced by policies for different candidate goals disagree. States with high disagreement are Critical Decision Points because observing the human’s next action helps distinguish between goals. In the example, s1 is a CDP because the policies associated with different goals prescribe different actions, while s2 and s3 are less informative compared to s1 because the predicted actions largely overlap.
Figure 2: Robot reducing uncertainty about the Human’s Goal at a CDP: As shown by the initial state, ingredients and utensils common to salad and stew are collaboratively brought by the human and the robot in the workspace. The robot reasons about possible next actions to influence the human to reveal their goal. The robot takes an action to complete gathering all the ingredients, rather than adding them to the pan or bowl, to observe the human in a CDP. This allows the human’s next action (either adding the kale to the pan or the bowl) to reduce uncertainty about the goal. The human clarifies their goal through their action, and the robot takes action to support the human.
Figure 3: Selecting actions to infer the human’s goal at a CDP for a human-robot collaborative cooking task: A Receding Horizon Planning (RHP) tree of all possible actions the human and the robot can take in a turn-based interaction is expanded, where the human takes the first turn. The circles correspond to states, and the arrows correspond to the possible action sequences taken at a given time step. The color of the circles indicates the possible goals that can be reached from a given state. The belief distribution is shown at steps t0 and t2 , as the robot updates its belief over the human’s goals when a human action is observed. The green arrows show the actions that would optimize the cost function in Eq. 10 . See the text for more details.
Food Type
Basic Ingredients
Variants
Additional Ingredients
Oatmeal
water, oats, salt
Plain, Fruit, Chocolate, Fruit Chocolate
strawberries, blueberries, chocolate chips
Parfait
yogurt, oats, sugar
Plain, Fruit, Fruit Nut, Fruit Chocolate, Nut Chocolate, Fruit Nut Chocolate
strawberries, blueberries, nuts, chocolate chips
Pasta
tomato, oil, pasta
Tomato Cheese, Veggie, Veggie Cheese
cheese, carrot, onion, broccoli
Salad
cucumber, lettuce, tomato, onion, kale, salt, oil
Veggie, Pasta, Avocado, Nut Avocado, Fruit Avocado, Fruit Nut, Pasta Fruit Nut
pasta, avocado, nuts, bananas, apple
Smoothie
water, yogurt, sugar
Berry, Fruit, Chocolate, Chocolate Fruit, Kale Tomato, Kale Berry
Table 1: List of Recipes that the Human and the Robot could collaboratively cook in our experimental setup & the required ingredients.
Figure 4: Setup for our Physical Robot Experiments for the Human-Robot Collaborative Cooking Task
Figure 5: Results of the Collaborative Cooking Evaluation: Comparison of performance metrics between the proposed approach, vanilla CDP Ghose et al. (2024) , Information Gain Maximization Sadigh et al. (2016) , and Bayesian Delegation Wu et al. (2021) in the collaborative cooking simulation. Metrics include a) the first correct guess, b) the last incorrect guess, c) the number of extra steps, and d) the percentage of correct guesses. The means and standard deviations of the metrics are calculated over 30 recipes and evaluated across three trials.
Figure 6: Ablation Study over the components of the Cost Function for the Collaborative Cooking Task
Figure 7: Ablation Study over using temporal modeling for computing CDPs for the Collaborative Cooking Task
Figure 8: Collaborative Cooking Case Study: Results of the Case Study with the Physical Robot running Bayesian Delegation, Vanilla CDP (history agnostic), Information Gain Maximization, and Our Proposed Algorithm: For Case Studies, a) and b), a group of columns represents the method. For each group, the first column depicts the agent taking action, the second column depicts the action sequences taken by the respective method, and the third column denotes the presence and position of the goal recipe in the top-3 most probable recipes from the goal prediction model (left-most being the most probable recipe).
Figure 9: Example RHP tree rollout for influencing the hider in a competitive hide-and-seek game: The green autonomous seeker robot and blue human-operated hider robot play hide-and-seek on a grid with brick-wall obstacles. The seeker uses RHP to steer the hider toward a Critical Decision Point (the red cell), where the hider’s next move can reveal its hidden strategy and speed capture. The green action labels (a1,a2,a3,…,an) denote candidate seeker actions in the RHP tree. The black arrows inside each grid show the hider’s movement under a possible policy from the policy bank. At each timestep, the seeker rolls out possible future interactions to horizon (t4) , considering both its own candidate actions and the hider’s possible policy-conditioned responses. At (t1) , both agents follow their underlying plans. When the hider spots the seeker at (t2) , it detours evasively at (t3) , then resumes its original route at (t4) , landing on the CDP unobserved and allowing the seeker to infer its strategy and catch it. The rollout identifies the optimal seeker action sequence (a2→a1→a1) shown by the green arrows; by RHP, the seeker executes only the first move (a2) before replanning. See the text for more details.
Figure 10: Test Environments and Computed CDPs for the Competitive Hide and Seek Task: (Top) Photo-realistic Simulated and real-world environments for the hide-and-seek task. (Bottom) Heatmaps of Critical Decision Points for corresponding environments. States closer to red denote the most Critical Decision Points, and states closer to blue denote the least critical points. Note: Although the figure displays Critical Decision Points for every state, typically, it is sufficient to compute them in a small region around the robot.
Figure 11: The robot seeker (S) influences the human hider (H) towards a Critical Decision Point (CDP) in a hide-and-seek game: a), b) The human-controlled hider sees the robot seeker on its way to its goal while following its high-level strategy. c) The robot seeker actively influences the human-controlled hider to escape towards a CDP (gray area). d) When the hider is at the CDP, the robot seeker optimizes for not being seen by the hider. So, the hider continues to navigate toward its goal. The robot seeker sees the hider take action toward its goal and identifies its strategy.
Figure 12: Results of Competitive Evaluation: Mean and Standard error of timesteps taken by the seeker to catch the hider for each experiment aggregated across all hider policies for a given environment and number of trials (lower is better)
We present a structured large-language-model (LLM) architecture for zero-shot human--robot coordination in a cooperative construction task with private goal views. Guided by a Dec-POMDP formulation, the architecture decomposes decision-making into (i) action-conditioned Theory-of-Mind (ToM) inference, (ii) hierarchical planning, (iii) conversation interpretation, (iv) action verification, and (v) feedback-based replanning. We compare the proposed method with an ablation without ToM inference and a multi-agent reinforcement-learning policy trained offline over many goal pairs. In human-participant experiments, the proposed method required fewer interaction steps and yielded higher post-interaction trust ratings than both baselines. These results suggest that systematically decomposing the team decision problem, using LLMs as tractable surrogates for otherwise intractable inference and planning computations, and retaining conventional verification for physical feasibility can improve both task coordination and the human experience.
Dong Hae Mangalindan, Anand Gokhale, Francesco Bullo +1
Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824 · Center for Control, Dynamical Systems, and Computation, UC Santa Barbara, Santa Barbara, CA 93106 USA
Approaches to incorporating human awareness into mobile robot decision-making mainly focus on collision avoidance in low-level motion planning, often overlooking the challenges posed by human presence and high-level behavior. To address this vacancy, we present HINT-Plan, a novel approach to integrate human intention prediction into robot task planning. HINT-Plan employs Vision Language Models (VLMs) to anticipate high-level human intentions from third-person image observations, convert them into goal states, and solve joint task-planning problems. To effectively enable scene awareness in context-rich environments, we use hierarchical Scene Graphs (SGs) as high-level representations of the environment, and translate environmental topology and actionable knowledge into formal planning language to ensure executable plans. Evaluated in a photorealistic simulation, HINT-Plan achieves an overall success rate of 69.71% in joint human-robot task planning, substantially outperforming the baselines by up to 35.29%, while also reducing functional conflicts. The results show the effectiveness of explicitly incorporating inferred human intentions into formal multi-agent task planning for proactive human-aware robot decision-making.
Yuchen Liu, Luigi Palmieri, Lujun Li +3
University of Stuttgart, Germany · University of Luxembourg, Luxembourg · Bosch Center for Artificial Intelligence, Germany
Effective human-robot collaboration in open-world environments requires joint planning under uncertainty about the task, the environment, and the human teammate. Communication is the most direct means of resolving such uncertainty, yet most existing systems support only one-way communication: robots listen and act, treating humans as passive supervisors rather than conversational teammates capable of two-way dialogue. We propose a unified human-robot joint planning system in which the robot actively resolves uncertainty through two complementary communication channels. When uncertainty is decision-critical, an uncertainty-mitigation joint planning module engages the human in clarification dialogue: it grounds ambiguous instructions via an LLM-assisted active elicitation mechanism, enumerates traversability hypotheses through a hypothesis-augmented A* search, and computes a cost-optimal querying policy via dynamic programming, so that the robot asks only the questions whose answers actually matter for the plan. When explicit dialogue is unnecessary or impractical, a real-time intent-aware collaboration module instead reads implicit, nonverbal cues, maintaining a probabilistic belief over the human's latent task intent from spatial and directional signals to enable coordination-aware task selection without any communication overhead. We validate the proposed system in both Gazebo simulations and real-world UAV deployments, integrated with a voice dialogue interface and a Vision-Language Model (VLM)-based 3D semantic perception pipeline. Experimental results show that cost-optimal clarification dialogue cuts the interaction cost by 51.9% while maintaining a 100% task success rate, and implicit intent reading reduces the cooperative task execution time by 25.4% compared to the baselines.
Zeyu Fang, Yuxin Lin, Cheng Liu +6
Department of Electrical and Computer Engineering, George Washington University · the Department of Electrical and Computer Engineering, Northeastern University · the Department of Mechanical and Aerospace Engineering, George Washington University