Active Perception for Embodied Disambiguation

Yiwei Liu; Luwei Yang

cs.AIAug 11, 2026

Active Perception for Embodied Disambiguation

Authors: Yiwei Liu, Luwei Yang

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data, Shenzhen 518172, China

Abstract

Natural language provides robots with a flexible task interface, but target ambiguity in embodied environments arises not only from user intent; it can also result from missing taskrelevant physical evidence in the current observation. Existing interactive disambiguation methods primarily obtain additional information by asking the user, whereas occlusion, restricted viewpoints, unreadable text, and unobserved targets require the robot to actively change its observation. We propose an active-perception framework for embodied target disambiguation that uses active observation as the backbone for information acquisition and uses a vision-language model to decide, on the basis of accumulated visual evidence and interaction information, whether to continue observing, request clarification, or complete target selection. Active observation can both directly recover missing discriminative evidence and reveal object names, labels, and semantic attributes, thereby improving user clarification when it remains necessary. Real-robot experiments show that the framework combines physical information acquisition and userintent clarification within a unified embodied disambiguation process.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the active-perception-based embodied disambiguation framework. Active observation provides the visual evidence used by subsequent clarification and target selection.

Fig. 2: Experimental setup of the PiPER platform with the eye-in-hand DABAI DC1 RGB-D camera.

Scene	Regime	Scene setup	Task / requested evidence	Method	Resolution outcome	Grounding source	Obs.	Q
A1	Observation incompleteness	Top surface partly occluded	Letter “A” on box top	Proposed	Correct	Fresh RGB-D	3	0
				Passive	Correct	Unsupported initial-view selection	0	0
				Greedy	Correct	User answer; fixed view	0	1
				Static	Correct	User answer; fixed view	0	1
A2	Observation incompleteness	Small text at initial distance	Book text “美国” (“USA”)	Proposed	Correct	Fresh RGB-D	3	0
				Passive	Incorrect	Initial-view evidence	0	0

TABLE I: Matched real-robot scenarios and outcomes. “Obs.” denotes the number of pre-selection active observations, and “Q” denotes the number of clarification questions.

Scene	Method	Original Chinese Interaction	English Translation
A6	Proposed	Robot: 桌上有三盒药：左边是粉色的‘盐酸左西替利嗪片’，中间是白色的‘复方嗜酸乳杆菌片’，右边是白色的‘健胃消食片’。请问您需要拿哪一盒？ User: 我想要一个可以治疗肠胃的	Robot: There are three medicine boxes on the table: the pink one on the left is Levocetirizine Hydrochloride Tablets, the white one in the middle is Compound Eosinophil-Lactobacillus Tablets, and the white one on the right is Jianwei Xiaoshi Digestive Tablets. Which one do you need? User: I want one that can treat stomach and intestinal problems.
A6	Greedy	Robot: 你指的是右边带彩色图案的物体吗？ User: 是	Robot: Do you mean the object on the right with a colorful pattern? User: Yes.
A6	Static	Robot: 你要左边白色的物体，还是右边带条纹侧面的物体或右边带彩色图案的物体？ User: 右边带条纹侧面的物体	Robot: Do you mean the white object on the left, the object on the right with a striped side, or the object on the right with a colorful pattern? User: The object on the right with a striped side.
A7	Proposed	Robot: 请问您需要拿哪一盒药？是左边粉色包装的，中间白色带绿色条纹的，还是右边印有‘健胃消食片’字样的？ User: 我想要健胃消食片	Robot: Which box of medicine do you need: the one with pink packaging on the left, the one with white packaging and green stripes in the middle, or the one on the right printed with the words Jianwei Xiaoshi Digestive Tablets? User: I want Jianwei Xiaoshi Digestive Tablets.
A7	Greedy	Robot 1: 你指的是右边带彩色图案的物体吗？ User 1: 我要健胃消食片 Robot 2: 你指的是右边带彩色图案的物体吗？ User 2: 是	Robot 1: Do you mean the object on the right with a colorful pattern? User 1: I want Jianwei Xiaoshi Digestive Tablets. Robot 2: Do you mean the object on the right with a colorful pattern? User 2: Yes.
A7	Static	Robot: 你要左边白色的物体，还是右边黑色的物体或右边带彩色图案的物体？ User: 我要健胃消食片	Robot: Do you want the white object on the left, the black object on the right, or the object on the right with a colorful pattern? User: I want Jianwei Xiaoshi Digestive Tablets.

TABLE II: Representative Human–Robot Clarification Transcripts. The Original Chinese Interaction column preserves the recorded Chinese exchange; the English Translation column gives a faithful English rendering. Passive trials are omitted because they contain no clarification dialogue.

Active Visual Perception

Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.

See all 47 papers

ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Oct 3, 2026Ruoxuan Feng, Yutong Chen, Ruihua Song +4Robotic Perception Multimodal Sensor Fusion
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beijing Academy of Artificial Intelligence +3
AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation
Oct 16, 2026Jiawei Geng, Zhenyang Liu, Yuzheng Wu +1Robotic Manipulation Active Vision for Robot Manipulation
Fudan University · Shanghai Innovation Institute
Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning
Sep 28, 2026Huao Li, Carson Sobolewski, Augustinos Saravanos +3Large Language Model-Based Robot Planning Active Learning
Massachusetts Institute of Technology, Cambridge, MA, USA · GE Vernova

Oct 3, 2026J/K move · Enter open · S save

ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception

Ruoxuan Feng, Yutong Chen, Ruihua Song, Huan Yang+3

Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beijing Academy of Artificial Intelligence+3

Robotic Perception Multimodal Sensor Fusion

Open paper AskPDF arXiv

Humans inherently understand the physical world through an active process. When sensory evidence is insufficient to infer physical properties, we naturally interact with the environment by deciding what information is missing, how to acquire it, and when sufficient evidence has been obtained. In stark contrast, existing multi-sensory robot systems mainly integrate sensory inputs rather than actively acquiring missing evidence through interactions. In this work, we introduce ROMA, an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. ROMA integrates vision, audio, tactile, and force sensing into a reasoning-interaction-feedback loop. The model identifies missing evidence and determines the target objects, interactions, and modalities, while a physical interface executes the selected interactions and collects the multi-sensory feedback. To support this capability, we construct ROMI-2K, a large-scale real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized sensory feedback. Building on these data, we develop a two-stage training framework that aligns sensory modalities and equips the LLM to assess evidence sufficiency, select informative interactions, and reason over the multi-sensory feedback. We further characterize active perception as perception chains, where acquired evidence guides subsequent interactions and reasoning, and establish ROMA Bench to evaluate single-attribute, long-horizon multi-attribute, and intent-driven active perception. Experiments show that ROMA can actively acquire missing evidence and solve complex, long-chain multi-sensory perception tasks that existing methods struggle to handle, laying a strong perceptual foundation for active multi-sensory embodied agents.

Abstract

Active Visual Perception

Explore similar work

ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception

AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation

Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning