Selective Commitment for Language-Guided Object Retrieval under Partial Observability
Authors: Wonhee Koh, Sushil Samuel Dinesh, Hansol Ko, Shinkyu Park, Eungjoo Lee
Organizations: School of Electrical, Computing, and Software Engineering, University of Arizona, Tucson, AZ 85721, USA · Department of Electrical and Computer Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia
Language-guided object retrieval under partial observability requires deciding whether to gather more evidence, interact with the scene, grasp a candidate, or abstain. We present a closed-loop framework that coordinates these decisions for retrieving a target specified in relation to a reference container. The framework maintains a persistent joint belief over target identity, container relation, and presence through tracked-object, unobserved-target, and target-absent hypotheses. View-conditioned categorical VLM observations update this belief; conformal grasp eligibility and robot feasibility govern commitment, while finite-horizon belief-space planning selects information-gathering actions. Across five different scenarios, our proposed method succeeds in 19/25 simulation episodes versus 12/25 for the best-performing task-adapted baseline and is the only evaluated policy to achieve at least one success in each scenario. Ablations show that cross-view memory improves success under partial occlusion, while the full system does not consistently outperform simplified variants. Real-robot trials demonstrate closed-loop re-observation and autonomous recovery from injected grasp failures, while injected viewpoint failures end in false defer. Experimental results demonstrate the feasibility of coordinating evidence gathering and selective grasp commitment within a unified framework for retrieval under partial observability.
Figures & tables
Fig. 2: Closed-loop architecture. Categorical VLM observations update the joint belief using a view-conditioned observation model. The scene graph represents persistent object tracks ci and their container relations ri . A finite-horizon planner selects actions, with grasping subject to both conformal eligibility and robot feasibility. Execution outcomes update the physical state, and new RGB-D observations provide feedback.
Fig. 3: Representative simulated scenes (top) and evaluation protocol (bottom). Scenario labels and the displayed occlusion percentage are used only for analysis and are not provided to the policy.
Scenario
Decision challenge
Visible/Open
Identify and retrieve the visible target without unnecessary sensing.
Partially Occluded
Acquire another view to resolve incomplete identity or visibility before commitment.
Covered Container
Remove the cover and re-observe before deciding to grasp or defer.
Ambiguous Inside/Outside
Resolve conflicting identity and container-relation evidence among similar candidates.
Target Absent
Preserve uncertainty when target evidence remains unresolved. Defer rather than commit to a distractor.
TABLE I: Scenario families and decision challenges.
Method
Analyzed N
Visible
Partial
Covered
Ambig.
Absent
Task success (Wilson 95% CI)
RoboEXP [ 4 ]
25
2/5
0/5
5/5
0/5
0/5
28.0% [14.3, 47.6]
DovSG [ 5 ]
24
2/5
0/5
0/5
0/4
5/5
29.2% [14.9, 49.2]
IntroPlan [ 13 ]
25
5/5
0/5
0/5
2/5
5/5
48.0% [30.0, 66.5]
ActPerMoMa [ 3 ]
24
0/5
0/5
0/5
1/4
5/5
25.0% [12.0, 44.9]
Ours
25
5/5
3/5
3/5
3/5
5/5
76.0% [56.6, 88.5]
TABLE II: Baseline task success on 25 scheduled evaluation episodes per method. Cells show successful/analyzed episodes; DovSG and ActPerMoMa each exclude one pre-observation timeout in the ambiguous scenario, leaving N=24 .
Method variant
Visible
Partial
Covered
Ambig.
Absent
Task success (Wilson 95% CI)
Ours (full)
5/5
3/5
3/5
3/5
5/5
76.0% [56.6, 88.5]
Observations pooled across views
5/5
2/5
3/5
3/5
5/5
72.0% [52.4, 85.7]
No cross-view memory
4/5
0/5
5/5
4/5
5/5
72.0% [52.4, 85.7]
Factorized belief over identity and relation
5/5
3/5
4/5
3/5
5/5
80.0% [60.9, 91.1]
Hypothesis-independent future observations
5/5
4/5
4/5
3/5
5/5
84.0% [65.3, 93.6]
No terminal execution-risk term
5/5
2/5
4/5
3/5
5/5
76.0% [56.6, 88.5]
TABLE III: Ablation study on 25 episodes per variant. Cells show successful episodes out of five for each scenario family; the final three variants separately remove the conformal grasp rule, the defer action, and both.
Fig. 4: Example simulated retrieval episode. (a)–(c) Observations from the center, right, and close-high views. (d) The “Other track hypotheses” category also includes the unobserved-target hypothesis zunobs , while zabs is shown separately. (e) The corresponding action sequence, ending with grasping T2 inside the container. “Grasp eligible” indicates that the conformal eligibility and robot feasibility checks are satisfied.
Condition
N
Task success
Mean duration (s)
Visible/Open
5
5/5
134
Viewpoint change (right)
5
5/5
160
Covered Container
5
5/5
209
Target Absent
5
5/5
175
Injected grasp-failure report
5
5/5
180
Injected viewpoint-failure report
5
0/5
166
TABLE IV: Real-robot evaluation with five trials per condition.
Department of Mechanical Engineering, Federal University of Uberlândia, Uberlândia, Brazil · Department of Mechanical Engineering, University of São Paulo, São Carlos, Brazil · Graduate School of Information Sciences, Tohoku University, Sendai, Japan +1