RoboQuest: Generalist Physical Agents that Search, Inspect and Test

Liu Renhang; Navonil Majumder; Tej Deep Pala; Soujanya Poria

cs.ROOct 7, 2026

RoboQuest: Generalist Physical Agents that Search, Inspect and Test

Authors: Liu Renhang, Navonil Majumder, Tej Deep Pala, Soujanya Poria

Organizations: Nanyang Technological University

Abstract

Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool. We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing. We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a $π_{0.5}$ policy fine-tuned on the full-episode demonstrations we release. The best agent succeeds in only 23% of the episodes, and the fine-tuned policy almost never succeeds. Isolated tests of the execution skills the tasks are built from, with the hidden information supplied, show that the agents can carry out most of the required actions, and our failure analysis attributes only a minority of the failures to execution. Our failure analysis further finds that the agents often stop exploring too early as they make decisions before observing the required evidence for task completion. We also find that agents rarely prevent or repair the disturbances caused by their exploration. Moreover, learning by trial and error remains difficult for most models.

Figures & tables

Figure 1: A visual depiction of the salient features of RoboQuest that includes tasks demanding information-seeking physical interactions with active perception.

		Benchmark features				Evidence seeking
Benchmark	Focus	Long horizon	Hidden	Memory	Mobile	Search	Inspect	Test
RLBench ( James et al., 2020 )	Visuomotor skills	$\sim$	$\sim$	–	–	–	–	–
ManiSkill3 ( Tao et al., 2024 )	Scalable manipulation	$\sim$	–	–	–	–	–	–
RoboTwin 2.0 ( Chen et al., 2025 )	Bimanual manipulation	$\sim$	–	–	–	–	$\sim$	–
LIBERO ( Liu et al., 2023 )	Lifelong skill transfer	$\checkmark$	–	–	–	–	–	–
CALVIN ( Mees et al., 2022 )	Long-horizon manipulation	$\checkmark$	$\sim$	–	–	–	–	–

Table 1: Comparison with representative manipulation benchmarks.

\checkmark

: explicit target;

\sim

: partial; –: not targeted. Hidden : task information absent from the current observation.

Figure 2: RoboQuest tasks. Left: four tasks, from the initial scene through revealing the hidden information to completion. Right: the other six tasks. Colours mark families.

Family	Task	Hidden information	How it is revealed
Search	Locked Storage	target behind locks	find the tokens that unlock each compartment
	Search Room	where the targets are	open storage places until the set is complete
	Blackout Search	targets in the dark	carry a light and search what it illuminates
Inspect	Painted Cubes	marks on unseen faces	turn each cube to see every face
	Marked Mugs	labels under vessels	lift or tilt each vessel to read its underside
	Unfamiliar Containers	how containers open	try each opening mechanism

Table 2: Tasks at a glance. What each task hides and how the agent can reveal it. Table 9 ( Appendix E ) lists the capabilities each task targets.

		GPT-6 Astra		Claude Opus 5.5		GPT-6.1 Sol		Claude Fable 5.1		Gemini 3.8 Flash
Family	Task	SR	Prog	SR	Prog	SR	Prog	SR	Prog	SR	Prog
Search	Locked Storage	28.0	52.9	6.0	35.2	12.0	38.5	8.0	21.7	0.0	9.2
	Search Room	0.0	31.0	0.0	26.7	0.0	19.4	0.0	19.2	2.0	17.5
	Blackout Search	0.0	23.7	0.0	17.4	0.0	18.5	0.0	17.1	0.0	4.4
Inspect	Painted Cubes	30.0	74.8	14.0	59.2	18.0	69.2	14.0	61.5	2.0	11.3
	Marked Mugs	38.0	69.7	16.0	51.4	6.0	37.2	12.0	31.8	0.0	0.8

Table 3: Main results. SR: success rate. Prog: mean progress. Both in %, 50 episodes per cell. Best SR per row in bold; shading darkens from 0 to 100%. Models are ordered by overall SR. Overall w/o Puzzle Box excludes the easiest task, an outlier on which every model scores far above its other tasks.

Figure 3: Results per model at a glance. Top: success, progress, how the episodes ended, and submission precision, the share of submissions that were correct. Bottom: cost, tokens, and wall-clock time per episode. All values are over the 500 episodes of each model. Bars are ordered from best to worst, except in How episodes end, where “Stopped” counts episodes the model ended on its own, by a stop call or a text answer, and “Timed out” episodes that ran out of the 200 decisions. Exact values: Tables 3 , 7 and 10 .

	Skill	Used in	GPT-6 Astra	Claude Opus 5.5	GPT-6.1 Sol
General	Pick and place (on counter)	All tasks	95 (19)	90 (21)	95 (18)
General	Drive and pick	Most tasks	100 (23)	100 (23)	90 (23)
Task- specific	Open drawer/cabinet	Search tasks	50 (22)	60 (31)	30 (36)
	Pick from drawer/cabinet	Search tasks	70 (15)	60 (17)	60 (20)
	Open named container	Containers	100 (20)	90 (16)	95 (21)
	Pick from open container	Containers	90 (20)	90 (24)	70 (19)

Table 4: Execution skills in isolation. Success rate (%), 20 scenes per skill. Best per row in bold. Grey: mean decisions of the successful episodes. Details in Appendix G .

Figure 4: Fine-tuned

\pi_{0.5}

evaluation results.

Figure 5: Failure attribution by family, in % of failures, with each failed episode weighted equally, per task and over all tasks. Containers: Unfamiliar Containers. Stamps: Stamp Composition. Counts in Table 16 .

	GPT-6 Astra	Opus 5.5	GPT-6.1 Sol
Open drawer/cabinet
Isolated	50	60	30
Full task	90	78	75
$\Delta$	$+$ 40	$+$ 18	$+$ 45
Take target out
Isolated	70	60	60

Table 5: Left: skills in isolation and in the full tasks. Success (%) in the isolated tests ( Section 4.3 ) and in the full tasks ( Table 12 ). In the full tasks, taking the target out counts only targets the model tried to grasp. Right: Puzzle Box with the locks hidden. Success rate, progress, and mean decisions per episode, on chain lengths 4 and 5.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Arguments	Default ticks
ARM	gripper-site position (required), orientation as a quaternion, and gripper command in $[-1,1]$ , where $-1$ opens and $+1$ closes	120
BASE	body-frame forward, left, and yaw velocity, each in $[-0.5,0.5]$	20
WAIT	optional gripper command, holds the arm and base	20
STOP	none, abandons the episode, which fails	–

Appendix

Table 6: Action tools. Positions are in world metres, and a tick is 0.05 s of simulated time. Every tool except STOP accepts a tick count from 1 to 200.

	Effort per episode			Tokens per episode			Cost (USD)
Model	Decisions	Sim min.	Wall min.	In (M)	Out (M)	Cached (%)	$/episode	$/success
GPT-6 Astra	143.7	10.4	53.8	5.82	0.02	91.8	11.29	48.7
Claude Opus 5.5	143.0	5.4	32.7	10.70	0.07	93.5	6.83	49.5
GPT-6.1 Sol	157.4	11.2	62.7	6.94	0.02	92.4	1.99	16.3
Claude Fable 5.1	148.8	6.1	54.7	14.55	0.11	94.2	19.22	168.6
Gemini 3.8 Flash	160.8	5.1	41.0	19.66	0.13	89.3	3.37	168.3

Appendix

Table 7: Cost and effort per episode. Sim / Wall: simulated / wall-clock minutes. In / Out: tokens in millions. Costs in USD at list prices. Best per column in bold (the lowest value, or the highest cache rate).

Category	Hyperparameter	Value
Training	Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.999$ , $\epsilon=10^{-8}$ )
	Weight Decay	$1\times 10^{-4}$
	Peak Learning Rate	$2.5\times 10^{-5}$
	Learning Rate Schedule	Cosine decay with 1000 warmup steps
	Global Batch Size	32
	Gradient Clipping Norm	$1.0$

Appendix

Table 8:

\pi_{0.5}

Training and Inference Hyperparameters.

Figure 6: All ten RoboQuest tasks. Five frames from one scripted demonstration of each task, from a scene camera, running from the initial scene through the actions that reveal the hidden information to the completed task. Family labels are given in colour.

			Evidence acquisition		Evidence use		Interactive inference			Action organization
Family	Task	Hidden information	DS	AP	EI	M	AD	XI	CI	CA	LP
Search	Locked Storage	target behind locks	$\checkmark$	–	$\checkmark$	$\sim$	–	–	$\sim$	$\checkmark$	$\checkmark$
	Search Room	where the targets are	$\checkmark$	$\sim$	$\checkmark$	$\checkmark$	–	–	–	–	$\sim$
	Blackout Search	targets in the dark	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–	–	–	$\sim$
Inspect	Painted Cubes	marks on unseen faces	–	$\checkmark$	$\checkmark$	$\sim$	–	–	–	$\sim$	$\sim$
	Marked Mugs	labels under vessels	–	$\checkmark$	$\sim$	$\sim$	–	–	–	$\checkmark$	$\sim$

Appendix

Table 9: Capabilities each task targets, with what each task hides (

\checkmark

: directly targeted,

\sim

: supporting, –: none). DS: directed search. AP: active perception. EI: evidence integration. M: memory. AD: affordance discovery. XI: experimental identification. CI: physical causal inference. CA: consequential action, marked where an information-gathering step can irreversibly damage the goal. LP: long-horizon planning. The table lists what each task demands, not where agents fail.

Model	Correct submission	Wrong submission	Stopped	Reply with no action	Out of decisions	Precision
GPT-6 Astra	23.2	61.8	5.4	0.0	9.6	27.3
Claude Opus 5.5	13.8	31.6	48.6	4.8	1.2	30.4
GPT-6.1 Sol	12.2	62.8	3.4	0.0	21.6	16.3
Claude Fable 5.1	11.4	53.6	28.6	5.4	1.0	17.5
Gemini 3.8 Flash	2.0	55.6	0.8	0.0	41.6	3.5

Appendix

Table 10: How episodes end, in % of 500 episodes per model. The first five columns sum to 100. Only a correct submission succeeds. Precision is the share of submissions that were correct.

Skill	Reached when	Reached	Completed
Open drawer/cabinet	the gripper touched the handle	85 / 85 / 75	50 / 60 / 30
Pick from drawer/cabinet	the gripper touched the object	90 / 80 / 100	70 / 60 / 60
Shim the short leg	the shim came within 3 cm of the leg	100 / 90 / 95	35 / 60 / 40
Drive and pick	the base came within 0.8 m of the object	100 / 100 / 95	100 / 100 / 90

Appendix

Table 11: Reaching the target and completing the skill, in % of the 20 episodes per skill, GPT-6 Astra / Opus 5.5 / GPT-6.1 Sol.

Stage	GPT-6 Astra	Opus 5.5	GPT-6.1 Sol	Isolated test
Search Room and Locked Storage: target in a compartment
Opened $\|$ tried	52/58 (90%)	28/36 (78%)	24/32 (75%)	Open drawer/cabinet: 50 / 60 / 30
Decisions per opening (mean)	10.4	25.7	15.2	Open drawer/cabinet: 22.1 / 30.8 / 35.8
Taken out $\|$ grasp tried	30/38 (79%)	17/18 (94%)	13/16 (81%)	Pick from drawer/cabinet: 70 / 60 / 60
Unfamiliar Containers: item in a box
Opened $\|$ tried	57/81 (70%)	35/77 (45%)	49/63 (78%)	Open named container: 100 / 90 / 95

Appendix

Table 12: Skills in the full tasks. Opening the compartment or box of a hidden target once it was tried, and taking the target out once a grasp was tried. The last column gives the isolated test of the same skill (success %, GPT-6 Astra / Opus 5.5 / GPT-6.1 Sol).

Task	Scripted information gathering	Minutes	Hours	Subtasks
Locked Storage	follows the token chain	3.7	31.1	16
Search Room	opens compartments nearest-first until the targets are seen	3.0	25.1	11
Blackout Search	carries the lamp along a nearest-first search	7.5	62.1	33
Painted Cubes	removes covers and turns each cube face by face	5.1	42.4	14
Marked Mugs	lifts and tilts each vessel to read its label	3.4	28.0	15
Unfamiliar Containers	visits boxes nearest-first and tries each knob’s actions until one opens the box	6.4	53.0	29

Appendix

Table 13: Demonstration dataset. Scripted information gathering in each oracle’s route, which the oracle performs although it knows the hidden information. Every task has 500 successful, verified episodes. Minutes: mean episode length. Hours: interaction time at 20 Hz. Subtasks: mean number of subtask annotations per episode.

	SR				Prog
	GPT-6 Astra	Opus 5.5	GPT-6.1 Sol	Pooled	GPT-6 Astra	Opus 5.5	GPT-6.1 Sol	Pooled
Occlusion (4 tasks)
Visible	28.8	16.3	14.0	19.7	57.6	45.8	39.5	47.7
Look	16.3	7.8	5.0	9.7	53.5	38.6	32.6	41.6
Uncover	22.3	7.5	6.1	12.0	53.1	35.9	34.1	41.1
Look $-$ Visible	$-$ 12.5	$-$ 8.6	$-$ 9.0	$-$ 10.0	$-$ 4.1	$-$ 7.2	$-$ 7.0	$-$ 6.1

Appendix

Table 14: Occlusion and workload. Success and progress (%) by level, pooled over the three models and per model. Occlusion on the four tasks that vary it, workload on the six.

	Missing	Wrong	Execution + side effect	All
Occlusion (4 tasks)
Visible	0.62	1.24	0.72	2.58
Look	0.89	1.32	0.53	2.74
Uncover	0.94	1.28	0.72	2.94
Look $-$ Visible	0.27	0.08	$-$ 0.18	0.16
Uncover $-$ Visible	0.32	0.04	0.00	0.36

Appendix

Table 15: Where the extra failures come from. Unmet requirements per episode by failure family (execution failures and side effects summed), pooled over GPT-6 Astra, Opus 5.5 and GPT-6.1 Sol. Occlusion on the four tasks that vary it, workload on the six.

Figure 7: Painted Cubes: error rates by rule. False-negative rate (FNR) and false-positive rate (FPR) over committed cubes, in %.

Task	Policy	Failed episodes	Units	Missing	Wrong	Side effect	Execution	Ran out
Locked Storage	GPT-6 Astra	36	87	48	23	0	16	7
	Opus 5.5	47	118	76	29	0	13	1
	GPT-6.1 Sol	44	108	69	30	0	9	42
Search Room	GPT-6 Astra	50	121	67	25	0	29	32
	Opus 5.5	50	118	81	19	0	18	0
	GPT-6.1 Sol	50	132	87	18	0	27	59

Appendix

Table 16: Failure attribution per task, in units (unmet requirements of failed episodes). Ran out: units in the Missing or Wrong columns whose episode ended by the decision budget.

Family	Object	GPT-6 Astra	Opus 5.5	GPT-6.1 Sol
Missing evidence	Never placed	31.9	33.5	34.2
	Placed before the evidence decided	11.2	9.4	11.4
Wrong decision	Never placed	12.8	19.1	15.9
	Placed against the evidence	10.2	11.9	9.3
Both	Never placed	44.6	52.6	50.1
	Placed wrongly	21.4	21.3	20.7

Appendix

Table 17: Missing evidence and wrong decisions by what the agent did, in % of failures, each failed episode weighted equally.

Explore similar work

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
Oct 20, 2025Yulin Luo, Chun-Kai Fan, Menghang Dong +19Long-Horizon Robotic Manipulation Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Oct 7, 2026Zhiqin Yang, Chenxin Li, Xiaomeng Hu +30Robot Manipulation Benchmarks Multimodal Agents
RescueBench: Can Embodied Agents Save Lives in the Wild ?
Jun 1, 2026Kui Wu, Beiyu Guo, Hao Chen +6Agent Memory Embodied Navigation
1Beihang University · 5ATEC2025 Challenge Committee · 4City University of Macau +2

Oct 20, 2025, cs.ROJ/K move · Enter open · S save

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

Yulin Luo, Chun-Kai Fan, Menghang Dong, Jiayu Shi+18

State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing+2

Long-Horizon Robotic Manipulation Multimodal Large Language Models Robot Manipulation Benchmarks Embodied AI Multimodal Model Evaluation Embodied Reasoning

Open paper AskPDF arXiv

Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems often follow a dual-system paradigm, where System 2 performs high-level reasoning and System 1 handles low-level control. We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this embodied brain is crucial, existing benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a benchmark for evaluating multimodal large language models (MLLMs) as embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.

Abstract

Explore similar work

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

RescueBench: Can Embodied Agents Save Lives in the Wild ?