Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning

Yoonwoo Kim; Raghav Arora; Roberto Martín-Martín; Peter Stone; Ben Abbatematteo; Yoonchang Sung

cs.ROMar 4, 2026

Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning

Authors: Yoonwoo Kim, Raghav Arora, Roberto Martín-Martín, Peter Stone, Ben Abbatematteo, Yoonchang Sung

Organizations: The University of Texas at Austin · Nanyang Technological University

Abstract

Robot planning in partially observable environments, where not all objects are known or visible, is a challenging problem, as it requires reasoning under uncertainty through partially observable Markov decision processes. During the execution of a computed plan, a robot may unexpectedly observe task-irrelevant objects, which are typically ignored by naive planners. In this work, we propose incorporating two types of common-sense knowledge: (1) certain objects are more likely to be found in specific locations; and (2) similar objects are likely to be co-located, while dissimilar objects are less likely to be found together. Manually engineering such knowledge is complex, so we explore leveraging the powerful common-sense reasoning capabilities of large language models (LLMs). Our planning and execution framework, CoCo-TAMP, introduces a hierarchical state estimation that uses LLM-guided information to shape the belief over task-relevant objects, enabling efficient solutions to long-horizon task and motion planning problems. In experiments, CoCo-TAMP achieves an average reduction of 62.7% in planning and execution time in simulation, and 72.6% in real-world demonstrations, compared to a baseline that does not incorporate either type of common-sense knowledge.

Figures & tables

Fig. 1 : The initial beliefs about the semantic locations of objects, $\text{bel}(x_{r,0}^{k})$ and $\text{bel}(x_{s,0}^{k})$ , are derived from LLMs, while the initial beliefs about their poses, $\text{bel}(x_{p,0}^{k})$ , are uniformly distributed across all surfaces. The TAMP problem specification $(\mathcal{O,P,I,G,A})$ , where the beliefs are incorporated into the cost of the observation action, is provided to a TAMP planner PDDLStream. The planner outputs a plan, which the robot executes. Upon executing the observation action, the co-location toggler determines whether to use the co-location model or not, based on the observed object. Then, the beliefs are updated using the proposed observation model, and the planner replans with the updated beliefs. Planning and execution are repeated until the goal state $\mathcal{G}$ is reached. Further implementation details, derivations, additional experimental results, and link to the code are available at our project page: https://coco-tamp.github.io .

Fig. 2 : Example of a simulated household environment.

	Baseline	Co-Model	LGBU	MCQA	MCQA with Co-Model	MCQA with LGBU
$bel(x_{s,0}^{k})$ , $bel(x_{s,0}^{k})$	Uniform	Uniform	Uniform	MCQA	MCQA	MCQA
co-location model	No	Yes	No	No	Yes	No
LLM generated update	No	No	Yes	No	No	Yes

TABLE I : Comparison method details.The terms

bel(x_{s,0}^{k})

and

bel(x_{r,0}^{k})

denote the initial belief over surfaces and rooms, respectively, for object

k

	4 rooms 8 surfs	4 rooms 16 surfs	6 rooms 12 surfs	6 rooms 24 surfs	8 rooms 16 surfs	8 rooms 32 surfs
Baseline	$632.5\pm 124.2$	$715.1\pm 240.5$	$1205.3\pm 272$	$1994\pm 602.7$	$1564.9\pm 362$	$3017.5\pm 1066.1$
Co-Model	$121.2\pm 58.4$	$70.2\pm 198.5$	$185.1\pm 145.6$	$355.3\pm 398.7$	$398.8\pm 272$	$0$
MCQA	$99.4\pm 95.3$	$169.9\pm 306.4$	$248.2\pm 262.2$	$1212.6\pm 735.1$	$134\pm 231$	$1925.3\pm 1320.8$
MCQA with Co-Model	$0$	$0$	$0$	$0$	$0$	$50.5\pm 834.8$
LGBU	$164.4\pm 92.9$	$605.8\pm 288.8$	$894.8\pm 551.9$	$509.2\pm 405$	$787.2\pm 321.3$	$1066.9\pm 1054.3$
MCQA with LGBU	$0.2\pm 48.8$	$339.2\pm 286.5$	$101.0\pm 210.7$	$351.1\pm 361.2$	$288.8\pm 308.3$	$408.5\pm 871.7$

TABLE II : Pairwise comparison of cumulative planning and execution time (in seconds) relative to the best-performing method. Numbers represent the mean

\pm

95% confidence interval calculated over 50 problems, and 0 indicates the best-performing method.

	4 rooms 8 surfs	4 rooms 16 surfs	6 rooms 12 surfs	6 rooms 24 surfs	8 rooms 16 surfs	8 rooms 32 surfs
Baseline	$9.5\pm 3.0$	$7.1\pm 6.8$	$17.2\pm 5.4$	$25.9\pm 8.8$	$19.8\pm 6.4$	$35.3\pm 13.7$
Co-Model	$3.0\pm 1.9$	$3.4\pm 5.8$	$5.5\pm 3.1$	$7.9\pm 5.5$	$10.6\pm 6.1$	$10.5\pm 7.4$
MCQA	$2.4\pm 2.7$	$0$	$2.9\pm 4.9$	$12.8\pm 8.7$	$0$	$13.9\pm 14.1$
MCQA with Co-Model	$0.6\pm 1.4$	$0.3\pm 6.6$	$0$	$0$	$1.4\pm 4.1$	$0$
LGBU	$4.1\pm 2.9$	$18.5\pm 10.7$	$23.9\pm 14.6$	$17.9\pm 10.1$	$21.3\pm 8.3$	$15.5\pm 8.8$
MCQA with LGBU	$11.2\pm 9.4$	$0$	$3.1\pm 5.5$	$10.2\pm 7.8$	$6.6\pm 7.0$	$6.1\pm 8.2$

TABLE III : Pairwise comparison of number of replans relative to the best-performing method. Numbers represent the mean

\pm

95% confidence interval calculated over 50 problems, and 0 indicates the best-performing method.

Fig. 3 : Average cumulative planning and execution time over 50 environments in household layout with 6 rooms and 12 surfaces with 95% confidence interval.

Fig. 4 : Pairwise comparison of number of replans required to complete the task compared to MCQA with co-location Model with 95% confidence interval.

Fig. 5 : Performance metrics over 50 environments in a household layout (6 rooms, 12 surfaces), comparing different LLMs for the MCQA setting. The plots show the number of replans with 95% confidence intervals. We see GPT-4o outperforms the other smaller models, and hence we have used GPT-4o in all other experiments in this paper.

Fig. 6 : Average number of replans over 5 environments in household layout with 6 rooms and 12 surfaces with 95% confidence interval. Note: The LGBU and MCQA with LGBU experiments had a total limit of 100 replans, which was exceeded 3 out of 5 times this experiment was run.

Fig. 7 : Environment for real-world experiments.

Motion Planning

Motion planning finds safe, feasible paths and trajectories for robots and vehicles amid geometric, dynamic, and task constraints.

See all 392 papers

CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning
Jun 5, 2024Xinrui Lin, Yangfan Wu, Huanyu Yang +3Large Language Model-Based Robot Planning Robot Task Planning
University of Science and Technology of China, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China
Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability
Sep 17, 2026Antareep Singha, Shivaram Kumar, Yoonwoo Kim +1Belief-Space Planning Active Perception
Nanyang Technological University · The University of Texas at Austin
Deciding If, When, and Where to Replan for Robust Manipulation
Aug 28, 2026Narendhiran Vijayakumar, Puru Ojha, Nav Singhal +2Large Language Model-Based Robot Planning Robotic Manipulation
Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad

Jun 5, 2024J/K move · Enter open · S save

CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning

Xinrui Lin, Yangfan Wu, Huanyu Yang, Yu Zhang+2

University of Science and Technology of China, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China

Large Language Model-Based Robot Planning Robot Task Planning

Open paper AskPDF arXiv

Large Language Models (LLMs) possess extensive foundational knowledge and moderate reasoning abilities, making them suitable for general task planning in open-world scenarios. However, it is challenging to ground a LLM-generated plan to be executable for the specified robot with certain restrictions. This paper introduces CLMASP, an approach that couples LLMs with Answer Set Programming (ASP) to overcome the limitations, where ASP is a non-monotonic logic programming formalism renowned for its capacity to represent and reason about a robot's action knowledge. CLMASP initiates with a LLM generating a basic skeleton plan, which is subsequently tailored to the specific scenario using a vector database. This plan is then refined by an ASP program with a robot's action knowledge, which integrates implementation details into the skeleton, grounding the LLM's abstract outputs in practical robot contexts. Our experiments conducted on the VirtualHome platform demonstrate CLMASP's efficacy. Compared to the baseline executable rate of under 2% with LLM approaches, CLMASP significantly improves this to over 90%.

Abstract

Motion Planning

Explore similar work

CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning

Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability

Deciding If, When, and Where to Replan for Robust Manipulation