Organizations: Department of Electrical and Computer Engineering, George Washington University · the Department of Electrical and Computer Engineering, Northeastern University · the Department of Mechanical and Aerospace Engineering, George Washington University
Effective human-robot collaboration in open-world environments requires joint planning under uncertainty about the task, the environment, and the human teammate. Communication is the most direct means of resolving such uncertainty, yet most existing systems support only one-way communication: robots listen and act, treating humans as passive supervisors rather than conversational teammates capable of two-way dialogue. We propose a unified human-robot joint planning system in which the robot actively resolves uncertainty through two complementary communication channels. When uncertainty is decision-critical, an uncertainty-mitigation joint planning module engages the human in clarification dialogue: it grounds ambiguous instructions via an LLM-assisted active elicitation mechanism, enumerates traversability hypotheses through a hypothesis-augmented A* search, and computes a cost-optimal querying policy via dynamic programming, so that the robot asks only the questions whose answers actually matter for the plan. When explicit dialogue is unnecessary or impractical, a real-time intent-aware collaboration module instead reads implicit, nonverbal cues, maintaining a probabilistic belief over the human's latent task intent from spatial and directional signals to enable coordination-aware task selection without any communication overhead. We validate the proposed system in both Gazebo simulations and real-world UAV deployments, integrated with a voice dialogue interface and a Vision-Language Model (VLM)-based 3D semantic perception pipeline. Experimental results show that cost-optimal clarification dialogue cuts the interaction cost by 51.9% while maintaining a 100% task success rate, and implicit intent reading reduces the cooperative task execution time by 25.4% compared to the baselines.
Figures & tables
Fig. 1 : Overview of the proposed system and the two planning modes inside the core planning engine. The system first analyzes the surrounding environment via perception models; the central interface then activates one planning mode based on the task type. The planning results, either way-point paths or high-level goals, are sent to the low-level controller to drive the drone. Details of the two modes are elaborated in Section II .
Fig. 2 : Semantic Fusion: The top row displays the original RGB inputs. The bottom row illustrates the semantic segmentation masks after the recursive fusion strategy.
Fig. 3 : Voice interface and Gazebo simulation environment.
Fig. 4 : Evaluation setups: (a) shows the joint planning configuration, while (b) details the intent-aware setup.
Method
Time (s)
Total Dist (m)
Human Dist (m)
Proposed
14.83 ± 0.79
24.46 ± 1.41
12.26 ± 0.76
Non-Cooperative
19.26 ± 0.90
27.40 ± 1.33
14.90 ± 0.90
TABLE I : Simulation performance of the intent-aware cooperation module compared to the non-cooperative baseline.
Metric
Rational
Ambiguous
Overall
Avg. True-Target Prob.
75.2% ± 2.2%
65.3% ± 3.0%
71.9% ± 2.0%
Top-1 Acc.
90.6% ± 2.5%
70.7% ± 3.2%
84.0% ± 2.8%
TABLE II : Intent recognition performance measured by average true-target probability and top-1 accuracy.
Fig. 5 : Task completion progress (left) and expected remaining travel distance (right) over time for the proposed method and non-cooperative baseline.
Attribute
Blue box
Black box
People
Yellow Box
Net
Confidence
0.8147
0.6342
0.5336
0.4644
0.3371
Range x
[7.39,7.64]
[−0.02,0.47]
[4.89,6.53]
[5.10,5.70]
[3.02,3.66]
Range y
[−1.04,−0.68]
[−0.09,0.30]
[0.58,1.21]
[−1.30,−0.62]
[0.49,1.17]
TABLE III : 2D Object Localizations and Confidence Scores.
Fig. 6 : Visualized results of uncertainty-mitigation in Gazebo simulations. Note that we only compare paths successfully generated by all three methods for length comparison.
Method
Time (s)
Total Dist (m)
Human Dist (m)
Proposed
20.59 ± 1.25
16.31 ± 1.91
9.22 ± 2.66
Non-Cooperative
27.61 ± 2.99
19.86 ± 2.88
11.29 ± 2.38
TABLE IV : Real-world performance of the intent-aware cooperation module compared to the non-cooperative baseline.
Method
Success (%)
Queries
Tokens
Time (s)
Passive
33.3
0.0
1079.3±50.2
8.80±1.07
Exhaustive
100
2.7
4109.0±81.1
13.35±1.51
Ours
100
1.3
3434.8±74.2
11.84±1.47
TABLE V : Real-world experimental results for uncertainty-mitigation joint planning, averaged on three commands.
Anticipating and adapting to failures is a key capability robots need to collaborate effectively with humans in complex domains. This continues to be a challenge despite the impressive performance of state of the art AI planning systems and Large Language Models (LLMs) because of the uncertainty associated with the tasks and their outcomes. Toward addressing this challenge, we present a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language. For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it; predicts potential failures due to lack of ability (in the human) or lack of relevant domain objects; and executes actions to prevent such failures or recover from them. Experimental evaluation in the VirtualHome 3D simulation environment demonstrates substantial improvement in performance compared with state of the art baselines.
Nabanita Dash, Ayush Kaura, Shivam Singh +4
Robotics Research Center, IIIT Hyderabad, India · TCS Research, Tata Consultancy Services, India · School of Informatics, University of Edinburgh, UK
Interactive robot planning requires robots to infer and execute human intentions from natural language instructions that are often ambiguous, incomplete, or underspecified. Although large language models (LLMs) provide a powerful interface for clarification, relying on the generative model to drive an multi-turn conversation can introduce systematic failures. We propose a Bayesian framework that treats clarification as an active learning problem over grounded Signal Temporal Logic (STL) task specifications. Our method uses LLMs to initialize candidate formal specifications and translate informative contrasts into natural-language clarification questions, while Bayesian optimization maintains uncertainty estimation over user intent and selects queries that maximize information gain. After convergence, the inferred STL specification is passed to a formal planner to synthesize a verifiable robot trajectory. Across four simulated and real-world task domains, our approach generally achieves higher task satisfaction and requires fewer clarification rounds than LLM baselines, while helping smaller models close the performance gap against larger reasoning models.
Approaches to incorporating human awareness into mobile robot decision-making mainly focus on collision avoidance in low-level motion planning, often overlooking the challenges posed by human presence and high-level behavior. To address this vacancy, we present HINT-Plan, a novel approach to integrate human intention prediction into robot task planning. HINT-Plan employs Vision Language Models (VLMs) to anticipate high-level human intentions from third-person image observations, convert them into goal states, and solve joint task-planning problems. To effectively enable scene awareness in context-rich environments, we use hierarchical Scene Graphs (SGs) as high-level representations of the environment, and translate environmental topology and actionable knowledge into formal planning language to ensure executable plans. Evaluated in a photorealistic simulation, HINT-Plan achieves an overall success rate of 69.71% in joint human-robot task planning, substantially outperforming the baselines by up to 35.29%, while also reducing functional conflicts. The results show the effectiveness of explicitly incorporating inferred human intentions into formal multi-agent task planning for proactive human-aware robot decision-making.
Yuchen Liu, Luigi Palmieri, Lujun Li +3
University of Stuttgart, Germany · University of Luxembourg, Luxembourg · Bosch Center for Artificial Intelligence, Germany