Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.
Figures & tables
Figure 1: Example Task. The robot must identify a hazardous barrel, but it does not know barrel locations or what constitutes a hazard a priori . It uses a language-embedded map constructed from RGB-D data to identify possible barrel locations. It rules out one as a false positive upon inspection. The other two candidates are barrels but are occluded by tarps. The robot removes the tarps, reads the signs hidden underneath, and finds the third barrel contains dangerous chemicals.
Figure 2: CLUE architecture. CLUE uses an LLM as a policy to realize underspecified language tasks in a closed-loop manner via two action types—hypothesis grounding and active information gathering. The LLM then constructs actions comprising manipulation, inspection, and navigation to resolve contextual uncertainties and make task progress. Information from the physical world is provided to the LLM via a symbolic graph-based observation space.
Figure 3: Hypothesis grounding. Sensory data is used to construct a language-embedded map. Given a concept from the policy’s hypothesis, candidate entities are identified by finding high-relevancy voxels and clustering. Viewpoints from which the candidates may be observed are identified via the map’s occupancy grid. These candidates and viewpoints form a graph which is provided to the policy via a textual observation along with a minimum distance visitation order.
Figure 4: Experimental snapshots showing the evaluation environments: two indoor settings and one outdoor setting.
Method
Success
Actions
Time (s)
LLM Calls
LLM Tok. (k)
VLM Calls
VLM Tok. (k)
Dis.
Func.
Occl.
Oracle
83.3%
100.0%
100.0%
4.3 ± 1.0
175.4 ± 62.0
4.4 ± 1.2
2.3 ± 8.0
1.1 ± 0.5
0.9 ± 0.4
CLUE
83.3%
100.0%
66.7%
8.1 ± 4.9
222.9 ± 120.6
6.3 ± 3.6
5.7 ± 5.2
2.9 ± 2.1
2.1 ± 1.7
NLMaps [ 5 ]
16.7%
33.3%
0.0%
4.3 ± 1.2
115.2 ± 58.4
3.4 ± 0.8
1.2 ± 4.6
0.9 ± 0.6
0.5 ± 0.3
Table 1: Baseline comparison mean and std. across fifteen real-world experiments, with success broken down by task type: object disambiguation (Dis.), functional inference (Func.), and occlusion reasoning (Occl.).
Figure 5: Qualitative Results. Our method sequentially inspects candidates derived from the map, reasoning about functional use until it arrives at a suitable object. NLMaps only visits one candidate, which is a false positive.
Method
Success
LLM Calls
LLM Tok. (k)
VLM Calls
VLM Tok. (k)
Dis.
Func.
Occl.
CLUE
66.7%
100.0%
66.7%
5.2 ± 3.1
4.2 ± 4.4
1.9 ± 1.3
1.4 ± 1.2
DAAAM [ 11 ]
33.3%
33.3%
0.0%
2.1 ± 0.6
3.4 ± 0.7
5.4 ± 0.5
17.1 ± 2.4
Table 2: Map retrieval mean and std. across nine real-world experiments, with success broken down by task type: object disambiguation (Dis.), functional inference (Func.), and occlusion reasoning (Occl.).
Figure 6: LLM ability to plan distance-efficient paths (mean and std.)
Interactive robot planning requires robots to infer and execute human intentions from natural language instructions that are often ambiguous, incomplete, or underspecified. Although large language models (LLMs) provide a powerful interface for clarification, relying on the generative model to drive an multi-turn conversation can introduce systematic failures. We propose a Bayesian framework that treats clarification as an active learning problem over grounded Signal Temporal Logic (STL) task specifications. Our method uses LLMs to initialize candidate formal specifications and translate informative contrasts into natural-language clarification questions, while Bayesian optimization maintains uncertainty estimation over user intent and selects queries that maximize information gain. After convergence, the inferred STL specification is passed to a formal planner to synthesize a verifiable robot trajectory. Across four simulated and real-world task domains, our approach generally achieves higher task satisfaction and requires fewer clarification rounds than LLM baselines, while helping smaller models close the performance gap against larger reasoning models.
Policy steering is an emerging way to adapt robot behaviors at deployment-time: a learned verifier analyzes low-level action samples proposed by a pre-trained policy (e.g., diffusion policy) and selects only those aligned with the task. While Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities, existing frameworks often assume these models are well-calibrated. In practice, the overconfident judgment from VLM can degrade the steering performance under both high-level semantic uncertainty in task specifications and low-level action uncertainty or incapability of the pre-trained policy. We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility, and selects an uncertainty resolution strategy: execute a high-confidence action, clarify task ambiguity via natural language queries, or ask for action interventions to correct the low-level policy when it is deemed incapable at the task. We leverage conformal prediction to calibrate the composition of the VLM and the pre-trained base policy, providing statistical assurances that the verifier selects the correct strategy. After collecting interventions during deployment, we employ residual learning to improve the capability of the pre-trained policy, enabling the system to learn continually but with minimal expensive human feedback. We demonstrate our framework through experiments in simulation and on hardware, showing that UPS can disentangle confident, ambiguous, and incapable scenarios and minimizes expensive user interventions compared to uncalibrated baselines and prior human- or robot-gated continual learning approaches. Videos can be found at https://jessie-yuan.github.io/ups/
We introduce VersualRL, a closed-loop framework for task-level robot planning that uses visual execution feedback to iteratively refine executable Behavior Trees through structured natural-language critique. VersualRL combines a vision-language model critic, which analyzes visual observations and Behavior Tree execution traces, with a large language model actor that performs discrete and interpretable policy updates. During physical deployment, both foundation models remain frozen, while the executed Behavior Tree is updated at the symbolic level without online gradient-based policy optimization. This enables transparent reasoning, explicit causal feedback, and human-interpretable policy evolution. We validate VersualRL on a real mobile robot performing a multi-stage navigation and manipulation task under execution uncertainty. Experimental results show that the framework supports explainable policy improvements, closed-loop adaptation to execution failures, and successful deployment on physical robotic systems.