Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language

Zachary Ravichandran; Jonathan Diller; Fernando Cladera; Varun Murali; George J. Pappas; Vijay Kumar

cs.ROSep 24, 2026

Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language

Authors: Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas, Vijay Kumar

Organizations: GRASP Laboratory, University of Pennsylvania · Electrical & Computer Engineering, Texas A&M University

Abstract

Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.

Figures & tables

Figure 1: Example Task. The robot must identify a hazardous barrel, but it does not know barrel locations or what constitutes a hazard a priori . It uses a language-embedded map constructed from RGB-D data to identify possible barrel locations. It rules out one as a false positive upon inspection. The other two candidates are barrels but are occluded by tarps. The robot removes the tarps, reads the signs hidden underneath, and finds the third barrel contains dangerous chemicals.

Figure 2: CLUE architecture. CLUE uses an LLM as a policy to realize underspecified language tasks in a closed-loop manner via two action types—hypothesis grounding and active information gathering. The LLM then constructs actions comprising manipulation, inspection, and navigation to resolve contextual uncertainties and make task progress. Information from the physical world is provided to the LLM via a symbolic graph-based observation space.

Figure 3: Hypothesis grounding. Sensory data is used to construct a language-embedded map. Given a concept from the policy’s hypothesis, candidate entities are identified by finding high-relevancy voxels and clustering. Viewpoints from which the candidates may be observed are identified via the map’s occupancy grid. These candidates and viewpoints form a graph which is provided to the policy via a textual observation along with a minimum distance visitation order.

Figure 4: Experimental snapshots showing the evaluation environments: two indoor settings and one outdoor setting.

Method	Success			Actions	Time (s)	LLM Calls	LLM Tok. (k)	VLM Calls	VLM Tok. (k)
Method	Dis.	Func.	Occl.	Actions	Time (s)	LLM Calls	LLM Tok. (k)	VLM Calls	VLM Tok. (k)
Oracle	83.3%	100.0%	100.0%	4.3 $\pm$ 1.0	175.4 $\pm$ 62.0	4.4 $\pm$ 1.2	2.3 $\pm$ 8.0	1.1 $\pm$ 0.5	0.9 $\pm$ 0.4
CLUE	83.3%	100.0%	66.7%	8.1 $\pm$ 4.9	222.9 $\pm$ 120.6	6.3 $\pm$ 3.6	5.7 $\pm$ 5.2	2.9 $\pm$ 2.1	2.1 $\pm$ 1.7
NLMaps [ 5 ]	16.7%	33.3%	0.0%	4.3 $\pm$ 1.2	115.2 $\pm$ 58.4	3.4 $\pm$ 0.8	1.2 $\pm$ 4.6	0.9 $\pm$ 0.6	0.5 $\pm$ 0.3

Table 1: Baseline comparison mean and std. across fifteen real-world experiments, with success broken down by task type: object disambiguation (Dis.), functional inference (Func.), and occlusion reasoning (Occl.).

Figure 5: Qualitative Results. Our method sequentially inspects candidates derived from the map, reasoning about functional use until it arrives at a suitable object. NLMaps only visits one candidate, which is a false positive.

Method	Success			LLM Calls	LLM Tok. (k)	VLM Calls	VLM Tok. (k)
Method	Dis.	Func.	Occl.	LLM Calls	LLM Tok. (k)	VLM Calls	VLM Tok. (k)
CLUE	66.7%	100.0%	66.7%	5.2 $\pm$ 3.1	4.2 $\pm$ 4.4	1.9 $\pm$ 1.3	1.4 $\pm$ 1.2
DAAAM [ 11 ]	33.3%	33.3%	0.0%	2.1 $\pm$ 0.6	3.4 $\pm$ 0.7	5.4 $\pm$ 0.5	17.1 $\pm$ 2.4

Table 2: Map retrieval mean and std. across nine real-world experiments, with success broken down by task type: object disambiguation (Dis.), functional inference (Func.), and occlusion reasoning (Occl.).

Figure 6: LLM ability to plan distance-efficient paths (mean and std.)

Language-Conditioned Robot Planning

See all 16 papers

Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning
Sep 28, 2026Huao Li, Carson Sobolewski, Augustinos Saravanos +3Large Language Model-Based Robot Planning Active Learning
Massachusetts Institute of Technology, Cambridge, MA, USA · GE Vernova
When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
Feb 25, 2026Jessie Yuan, Yilin Wu, Andrea BajcsyRobot Policy Adaptation VLMs for Robotics
Carnegie Mellon University
VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning
Mar 23, 2026Dmitrii Plotnikov, Iaroslav Kolomiets, Dmitrii Maliukov +9Robot Policy Learning Large Language Model-Based Robot Planning
The authors are with the Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology.

Sep 28, 2026J/K move · Enter open · S save

Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning

Huao Li, Carson Sobolewski, Augustinos Saravanos, William Tan+2

Massachusetts Institute of Technology, Cambridge, MA, USA · GE Vernova

Large Language Model-Based Robot Planning Active Learning Robot Task Planning Human-Robot Interaction

Open paper AskPDF arXiv

Interactive robot planning requires robots to infer and execute human intentions from natural language instructions that are often ambiguous, incomplete, or underspecified. Although large language models (LLMs) provide a powerful interface for clarification, relying on the generative model to drive an multi-turn conversation can introduce systematic failures. We propose a Bayesian framework that treats clarification as an active learning problem over grounded Signal Temporal Logic (STL) task specifications. Our method uses LLMs to initialize candidate formal specifications and translate informative contrasts into natural-language clarification questions, while Bayesian optimization maintains uncertainty estimation over user intent and selects queries that maximize information gain. After convergence, the inferred STL specification is passed to a formal planner to synthesize a verifiable robot trajectory. Across four simulated and real-world task domains, our approach generally achieves higher task satisfaction and requires fewer clarification rounds than LLM baselines, while helping smaller models close the performance gap against larger reasoning models.

Abstract

Language-Conditioned Robot Planning

Explore similar work

Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning

When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering

VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning