LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments

Ivy Xiao He; Stefanie Tellex; Jason Xinyu Liu

cs.ROMar 5, 2026

LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments

Authors: Ivy Xiao He, Stefanie Tellex, Jason Xinyu Liu

Organizations: Brown University Providence, USA

Abstract

To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.

Figures & tables

Figure 1. Multimodal fusion with belief updates disambiguates human instructions and identifies the intended object among multiple candidates. Given a spoken instruction and a pointing gesture, the robot disambiguates the target object among multiple candidates and retrieves the correct one.

Figure 2. system diagram. system diagram

Figure 3. Example frame showing different vector- and cone-based models of the pointing direction, with the target marked in green. Example frame showing different vector- and cone-based models of the pointing direction, with the target marked in green.

Pointing Representation	Cov. @25%	$\theta_{\text{diff}}$ ( ∘ )
Eye-to-Wrist	0.718 $\pm$ 0.014	24.4 $\pm$ 0.8
Nose-to-Wrist	0.746 $\pm$ 0.014	23.2 $\pm$ 0.8
Shoulder-to-Wrist	0.865 $\pm$ 0.011	17.0 $\pm$ 0.7
Elbow-to-Wrist	0.772 $\pm$ 0.013	20.2 $\pm$ 0.8
Gesture Cone	0.890 $\pm$ 0.010	14.4 $\pm$ 0.4

Table 1. Comparison of gesture representation for pointing estimation for pointing estimation. Metrics are reported as mean

\pm

95% confidence interval (CI).

Figure 4. Visual grounding comparison between SoM prompting and a detector baseline (GroundingDINO). Panels comparing detections from a VLM against SoM; SoM isolates intended objects more accurately when ambiguity is present.

Grounding Acc	LLM (SoM)	Detector (DINO)
None	0.793	0.603
Spatial	0.957	0.577
Attribute	0.944	0.665
Spatial + Attribute	0.818	0.956

Table 2. Grounding Success rate under different language conditions. LLM grounding (SoM: SAM2 + GPT-4o) vs. detector grounding (GroundingDINO).

Belief	Solver	Success	Steps	Time [s]
Histogram	Heuristic	0.68 $\pm$ 0.11	111.3 $\pm$ 24.0	12.0 $\pm$ 2.6
	Greedy	0.63 $\pm$ 0.11	227.7 $\pm$ 20.4	24.6 $\pm$ 2.2
	PO-UCT	0.96 $\pm$ 0.06	124.9 $\pm$ 14.7	32.2 $\pm$ 8.3
Particles	Heuristic	0.21 $\pm$ 0.10	42.3 $\pm$ 7.9	4.7 $\pm$ 0.9
	Greedy	0.27 $\pm$ 0.10	183.6 $\pm$ 18.4	20.5 $\pm$ 2.1
	POMCP	0.24 $\pm$ 0.09	183.4 $\pm$ 20.2	36.8 $\pm$ 4.5

Table 3. Solver performance under histogram vs. particle belief. Metrics are mean

\pm

95% CI over all trials.

Modality	Success Rate	Steps	Time [s]
multimodal conflicted	$0.024\pm 0.007$	$59.0\pm 17.3$	$21.1\pm 12.8$
wrong language	$0.093\pm 0.064$	$95.9\pm 35.2$	$24.5\pm 10.8$
wrong gesture	$0.170\pm 0.049$	$91.2\pm 33.1$	$22.8\pm 11.1$
No Input	$0.482\pm 0.130$	$162.3\pm 35.2$	$37.0\pm 10.4$
Gesture	$0.618\pm 0.045$	$122.5\pm 25.5$	$23.2\pm 4.7$
Language	$0.710\pm 0.057$	$95.8\pm 34.4$	$20.4\pm 5.4$

Table 4. Modality comparison. Success rate, steps, and time are reported with 95% confidence intervals.

Figure 5. Belief convergence in the large environment . (Top) Max-belief traces show how certainty in the most likely state evolves over time. (Bottom) Target-belief traces show probability mass assigned to the true target. Belief convergence comparison

Figure 6. Success rates by instruction modality across environments of increasing complexity. Success rates by instruction modality across environments of increasing complexity.

Figure 7. LEGS-POMDP robot testing and dashboard interface. The figure illustrates both the real-robot experimental setup and the integrated UI, which visualizes multimodal grounding, POMDP belief states, and robot control in real time. LEGS-POMDP robot testing and dashboard interface. The figure illustrates both the real-robot experimental setup and the integrated UI, which visualizes multimodal grounding, POMDP belief states, and robot control in real time.

Figure 8. Entropy loss curve. LEGS-POMDP robot testing and dashboard interface. The figure illustrates both the real-robot experimental setup and the integrated UI, which visualizes multimodal grounding, POMDP belief states, and robot control in real time.

Explore similar work

POMDP-based Object Search with Growing State Space and Hybrid Action Domain
Apr 16, 2026Yongbo Chen, Hesheng Wang, Shoudong Huang +1Mobile Robotics Monte Carlo Tree Search
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, 200240, People’s Republic of China · School of Computing, Australian National University (ANU), Canberra, ACT, 2601, Australia (The main part of this work is completed at ANU) · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, 200240, People’s Republic of China +1
PO-PDDL: Learning Symbolic POMDPs from Visual Demonstrations for Robot Planning Under Uncertainty
Jun 14, 2026Wenjing Tang, Xuanjin Jin, Yuan Liu +3Belief-Space Planning Symbolic Planning
Shanghai Jiao Tong University · Shanghai Innovation Institute
Selective Commitment for Language-Guided Object Retrieval under Partial Observability
Sep 19, 2026Wonhee Koh, Sushil Samuel Dinesh, Hansol Ko +2Vision-Language Retrieval Active Vision for Robot Manipulation
School of Electrical, Computing, and Software Engineering, University of Arizona, Tucson, AZ 85721, USA · Department of Electrical and Computer Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia

Apr 16, 2026J/K move · Enter open · S save

POMDP-based Object Search with Growing State Space and Hybrid Action Domain

Yongbo Chen, Hesheng Wang, Shoudong Huang, Hanna Kurniawati

School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, 200240, People’s Republic of China · School of Computing, Australian National University (ANU), Canberra, ACT, 2601, Australia (The main part of this work is completed at ANU) · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, 200240, People’s Republic of China+1

Mobile Robotics Monte Carlo Tree Search Robotics Partially Observable Markov Decision Processes

Open paper AskPDF arXiv

Efficiently locating target objects in complex indoor environments with diverse furniture, such as shelves, tables, and beds, is a significant challenge for mobile robots. This difficulty arises from factors like localization errors, limited fields of view, and visual occlusion. We address this by framing the object-search task as a highdimensional Partially Observable Markov Decision Process (POMDP) with a growing state space and hybrid (continuous and discrete) action spaces in 3D environments. Based on a meticulously designed perception module, a novel online POMDP solver named the growing neural process filtered k-center clustering tree (GNPF-kCT) is proposed to tackle this problem. Optimal actions are selected using Monte Carlo Tree Search (MCTS) with belief tree reuse for growing state space, a neural process network to filter useless primitive actions, and k-center clustering hypersphere discretization for efficient refinement of high-dimensional action spaces. A modified upper-confidence bound (UCB), informed by belief differences and action value functions within cells of estimated diameters, guides MCTS expansion. Theoretical analysis validates the convergence and performance potential of our method. To address scenarios with limited information or rewards, we also introduce a guessed target object with a grid-world model as a key strategy to enhance search efficiency. Extensive Gazebo simulations with Fetch and Stretch robots demonstrate faster and more reliable target localization than POMDP-based baselines and state-of-the-art (SOTA) non-POMDP-based solvers, especially large language model (LLM) based methods, in object search under the same computational constraints and perception systems. Real-world tests in office environments confirm the practical applicability of our approach. Project page: https://sites.google.com/view/gnpfkct.

Abstract

Explore similar work

POMDP-based Object Search with Growing State Space and Hybrid Action Domain

PO-PDDL: Learning Symbolic POMDPs from Visual Demonstrations for Robot Planning Under Uncertainty

Selective Commitment for Language-Guided Object Retrieval under Partial Observability