Towards a Systematic Study of Embodied System 2 over Full Task Execution Cycles

Yuxi Wei; Ye Fang; Shuai Yang; Tong Wu; Tai Wang; Xiaojuan Qi

cs.ROOct 13, 2026

Towards a Systematic Study of Embodied System 2 over Full Task Execution Cycles

Authors: Yuxi Wei, Ye Fang, Shuai Yang, Tong Wu, Tai Wang, Xiaojuan Qi

Organizations: The University of Hong Kong · Morphi Robot · Fudan University · Shanghai Jiao Tong University

Abstract

An embodied brain, or System 2, must turn an abstract goal into actions while discovering relevant scene conditions and checking whether executed skills achieved their intended effects. Open-loop evaluations cannot test observations shaped by an agent's decisions, while some closed-loop evaluations provide limited evidence during execution or assume idealized skill outcomes. We introduce SCOPER, a suite for studying observation-grounded planning, execution, and recovery. SCOPER-bench combines high-level goals, ongoing visual observations, a shared rule-based System 1 for low-level actions, and controlled execution failures. Its construction and validation pipeline yields 870 tasks across BEHAVIOR and ProcTHOR, complemented by about 3,500 situated questions tied to the evaluated agent's execution. Benchmark results reveal difficulties in retaining scene evidence, monitoring progress, and recovering from unexpected outcomes. To address these difficulties, SCOPER-harness combines Evidence Memory and Progress Verification with a dynamic task agenda and bounded exploration, without additional model training. Across two evaluated open-source backbones and two simulators, the harness improves task success by 11.1-28.6%; a harnessed 9B model outperforms a directly executed 27B model on both simulators. In an offline next-action evaluation using observations from 200 real-robot demonstrations, the harness also improves full-call accuracy by 14.7% and 15.3%. SCOPER provides a setting for diagnosing and improving embodied reasoning throughout a complete task execution cycle.

Figures & tables

Figure 1: Embodied System 2 throughout closed-loop task execution. Open-loop reference trajectories cannot test decision-dependent interaction; closed-loop settings omitting execution-time observations or failures leave key capabilities untested. The SCOPER suite combines high-level goals, ongoing visual observations, and controlled failures in SCOPER-bench, with Evidence Memory and Progress Verification in SCOPER-harness.

Figure 2: SCOPER-bench construction. Scene-grounded templates undergo static checks and privileged Oracle System 1 execution, including failures and recovery. Traces support situated QA construction; evaluation answers follow the evaluated episode’s state. Average steps serve as a reference metric.

Figure 3: SCOPER-harness. Evidence Memory: agenda revision, action selection; Progress Verification: execution outcomes for planning, recovery; anti-deadlock control: balancing exploration and task advancement.

Figure 4: Example task traces: (1) recovery from controlled grasp and navigation failures; (2) observation-conditioned household actions identifying targets undisclosed in instructions; and (3) observation activating and fulfilling an instruction-specified cleaning requirement during placement.

BEHAVIOR
Model	Routine Execution				Observation-dependent				Failure Recovery				Composite				Overall
	SR	Prog.	Cov.	QA (T / A)	SR	Prog.	Cov.	QA (T / A)	SR	Prog.	Cov.	QA (T / A)	SR	Prog.	Cov.	QA (T / A)	SR	Prog.	Cov.	QA (T / A)
API model
GPT-6 Astra	42.0	74.6	72.6	81.8 / 54.3	40.0	56.9	69.1	85.9 / 82.5	76.2	85.1	88.2	84.4 / 74.7	35.0	75.4	59.4	43.9 / 58.1	50.5	74.2	73.5	74.4 / 68.9
Gemini-3.8-flash	25.0	44.6	42.3	48.5 / 30.6	25.6	35.4	43.5	54.4 / 53.2	50.8	57.3	59.2	57.0 / 51.1	20.0	41.2	32.4	24.2 / 31.9	31.9	45.7	45.4	46.5 / 44.1
Embodied models

Table 1: BEHAVIOR and ProcTHOR results by task group (%). QA reports trigger rate / triggered answer accuracy;

\Delta

denotes the change from the corresponding direct-execution baseline in percentage points. Bold indicates the best result among non-API models within each backend.

ProcTHOR (450 episodes per variant)
Variant	Routine Execution			Observation-dependent			Failure Recovery			Composite			Overall
	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.
Full harness	33.1	71.1	64.6	37.8	61.5	70.0	39.2	64.7	74.4	17.0	65.3	50.3	32.2	66.1	65.4
w/o Anti-deadlock	21.5	45.5	40.5	24.4	41.7	47.5	23.9	42.5	48.0	11.0	40.7	30.4	20.4	42.8	41.8
$\Delta$ vs. full (pp)	-11.5	-25.7	-24.2	-13.3	-19.8	-22.5	-15.4	-22.2	-26.4	-6.0	-24.7	-19.9	-11.8	-23.3	-23.5
w/o Recovery	34.6	74.0	67.1	42.2	61.1	70.8	27.7	50.7	62.4	18.0	60.3	49.3	30.4	61.7	62.5

Table 2: Qwen3.8-27B harness component ablation results on ProcTHOR and BEHAVIOR

API model
Model	Routine Execution		Observation-dependent		Failure Recovery		Critical Decisions		Overall
	Tool	Call	Tool	Call	Tool	Call	Obs. Call	Repair Call	Tool	Call
GPT-6 Astra	86.8	79.6	78.2	69.4	73.8	63.5	64.4	56.7	79.5	70.8
Embodied models
Embodied-R1.5	55.4	45.4	44.3	31.7	40.4	28.3	22.2	20.0	46.7	35.1
RoboBrain2.5-8B	51.9	40.8	38.1	26.0	35.4	22.7	16.7	13.3	41.8	29.8

Table 3: Evaluation of normal execution, observation, and failure recovery on real execution data.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	ProcTHOR	BEHAVIOR
GPT-6 Astra	23.07	22.45
Embodied-R1.5	12.88	14.46
RoboBrain2.5-8B	10.08	12.54
RynnBrain11-9B	21.88	21.21
Molmo2-ER-4B	18.08	18.31
Qwen3.5-9B	20.06	19.45

Appendix

Table 4: Average high-level calls over all scored episodes, including agent failures. Lower values do not necessarily indicate better efficiency.

API model
Model	Routine Execution			Observation-dependent			Failure Recovery			Composite			Overall
	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.
GPT-6 Astra	0.0	22.2	37.5	1.1	18.5	34.8	0.8	28.5	48.1	2.0	29.5	30.1	1.0	25.1	38.5
Gemini-3.8-flash	0.0	14.5	24.7	1.1	10.9	20.4	0.8	17.8	30.0	1.0	15.5	15.7	0.7	15.0	23.3
Embodied models
Embodied-R1.5	2.0	24.8	16.0	7.8	20.6	25.6	8.5	18.9	23.5	3.0	21.8	13.0	5.5	21.3	19.6

Appendix

Table 5: BEHAVIOR blind testing by task group (%).

API model
Model	Routine Execution			Observation-dependent			Failure Recovery			Composite			Overall
	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.	SR	Prog.	Cov.
GPT-6 Astra	4.6	38.6	43.4	3.3	35.4	46.6	3.8	35.1	51.4	2.0	32.2	36.2	3.6	35.5	44.7
Gemini-3.8-flash	2.3	24.4	27.4	3.3	24.0	31.1	2.3	21.6	32.0	1.0	20.4	23.6	2.2	22.6	28.6
Embodied models
Embodied-R1.5	1.5	23.0	6.2	5.6	12.6	15.0	3.1	14.5	14.8	1.0	22.2	4.9	2.7	18.3	10.1

Appendix

Table 6: ProcTHOR blind testing by task group (%).

Skill	Preconditions and input	Controlled execution and observations
Navigation	Resolved object or room target; reachable interaction pose.	Follow an A* path, update the agent pose, and provide observations along the route.
Scan	Available viewpoints around the current pose.	Rotate the view and acquire multiple observations of the surroundings.
Target search	Public target reference.	Direct observation toward relevant views to obtain visual evidence about the target.
Pick	Pickupable, accessible, interaction-ready object.	Establish the held relation and make the object pose follow the executor.
Place / release	Held object; an admissible destination for placement.	Place at a support or containment pose, or release at the current position; remove the held relation and update spatial relations.
Open / close	Openable, accessible container.	Update openness and the corresponding joint or visual configuration.

Appendix

Table 7: Unified System 1 skills. Rows describe semantic capabilities rather than identical API names across backends.

Tool	Arguments	Meaning and returned observation
navigate_to	object_ref	Object/receptacle, or a public room label in ProcTHOR; route and endpoint images.
navigate_to_room	room	Public room label in BEHAVIOR; process images.
scan	None.	Multiple in-place views obtained by rotating the camera.
look_for	object_ref	Target-directed camera views; does not replace approaching the object.
pick_up	object_ref	Object to pick up; before-and-after images.
put_down	object_ref	Destination receptacle or support, not the held object; before-and-after images.

Appendix

Table 8: Public tool parameters and observation semantics.

Mechanism	Trigger and realized consequence	Recovery requirement
Grasp slip	A pickup releases the object at a recoverable nearby location.	Relocalize, approach, and pick again.
Dropped placement	Placement releases the object beside the destination, violating the required relation.	Reacquire the object and place it again.
Partial opening	An open request leaves insufficient openness.	Inspect and complete opening.
Partial closing	A close request leaves the container partly open.	Inspect and complete closing.
Wrong-target navigation	Navigation executes toward an alternative object.	Detect the mismatch and approach the intended target.
Incomplete navigation	The final agent pose falls short of the required interaction position.	Reassess the current pose and navigate onward.

Appendix

Table 9: Execution failures: activation, state consequences, and recovery requirements.

Component	Specification
Roles	Object categories, affordances, optionality, and inter-role relations.
Initialization	Object properties, initial containment and placement, and task-relevant initial visibility.
Goals	Object-state and spatial-relation predicates, including conditional or set-valued goals.
Instruction	Public descriptions grounded in bound roles and observation-conditioned handling rules.
Events and QA	Activation conditions, state interventions, questions, and scoring rules.
Composition	Child identities, scopes, and handoff relations.

Appendix

Table 10: Semantic structure of a task template.

Template	Roles and initialization / events	Goal predicates
Simple placement	Object $x$ , receptacle $r$ in the same room. $x$ visible; $\neg P(x,r)$ .	$P(x,r)$ .
Device activation	Toggleable device $d$ . $\neg\mathrm{On}(d)$ .	$\mathrm{On}(d)$ .
Clean and place	Dirtyable $x$ , sink $w$ , faucet $f$ , destination $r$ . $\mathrm{Dirty}(x)$ ; compatible sink–faucet pair.	$\mathrm{Clean}(x)\land\allowbreak P(x,r)\land\allowbreak C^{*}(r)$ .
Heat and place	Object $x$ , microwave $h$ , destination $r$ . $\neg\mathrm{On}(h)$ .	$\mathrm{Hot}(x)\land\allowbreak P(x,r)\land\allowbreak\mathrm{Closed}(h)\land\allowbreak\neg\mathrm{On}(h)$ .
Cool and place	Object $x$ , refrigerator $h$ , destination $r\neq h$ . Compatible cooling and placement roles.	$\mathrm{Cool}(x)\land\allowbreak P(x,r)\land\allowbreak\mathrm{Closed}(h)$ .
Slice and place	Sliceable $x$ , knife $k$ , destination $r$ . Compatible slicing and placement roles.	$\mathrm{Sliced}(x)\land\allowbreak P(x,r)$ .

Appendix

Table 11: Benchmark task templates: roles, initialization, and goal predicates.

Category	Template	Representative instruction	Key execution mechanism
R	Simple placement	Put the mug on the table.	Transfer the object to its destination.
O	Device activation	Turn on the desk lamp.	Device initially off; activate it.
R	Clean and place	Wash the bowl and put it in the cabinet. Close the cabinet when finished.	Object initially dirty; establish cleanliness and storage.
R	Heat and place	Heat the potato in the microwave, then put it on the plate. Leave the microwave closed and switched off.	Heating must precede final placement; leave the microwave off and closed.
R	Cool and place	Cool the apple in the refrigerator, then move it to the table. Leave the refrigerator closed.	Cooling must precede transfer to a different destination.
R	Slice and place	Slice the apple and put it on the plate.	Slicing requires a compatible cutting tool.

Appendix

Table 12: Representative task instructions, template categories, and key execution mechanisms. R: routine; O: observation-dependent; F: failure recovery. Category and mechanism annotations are not shown to the model.

Question type	Target of assessment
Target state	Whether an object is currently clean, open, or activated.
Target relation	Whether an object is inside a container or on a designated surface.
Post-action outcome	Whether the preceding action achieved its intended effect.
Active issue	Which issue or execution discrepancy remains unresolved.
Inserted obligation	Which additional requirement was introduced by a new instruction.
Remaining requirement	Which task requirement has not yet been satisfied.

Appendix

Table 13: The eight situated QA templates and the understanding they probe.

Figure 5: Sample 1: Retrieving a fallen plant pot and completing placement.

Figure 6: Sample 2: Retrieving an apple after a slipped grasp.

Figure 7: Sample 3: Correcting navigation to the wrong target.

Figure 8: Sample 4: Inspecting and retrying a partial drawer opening.

Figure 9: Sample 5, part 1: Storing the plant pot and approaching the coffee cup.

Figure 10: Sample 5, part 2: Reopening the cabinet to store the coffee cup.

Figure 11: Sample 6, part 1: Finding a dirty mug and bringing it to the faucet.

Figure 12: Sample 6, part 2: Washing the mug and putting it away.

Figure 13: Sample 7: Turning off an observed running faucet and closing an open fridge.

Figure 14: Sample 8, part 1: Placing the tomato and retrieving an observed dirty bowl.

Figure 15: Sample 8, part 2: Washing and storing the bowl as required by the instruction.

Figure 16: Sample 9: Setting down the knife before picking up and placing the apple slices.

Figure 17: Real-data examples: Observing action-relevant evidence through movement and reinspection.

Figure 18: Real-data examples: Synthetic early-stop nodes with incomplete actions.

Embodied Reasoning

Embodied reasoning studies how agents use perception, memory, and physical knowledge to plan and act in interactive environments.

See all 77 papers

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Sep 25, 2026Maokai Qin, Chuan Qin, Qi Zhang +5Embodied AI AI Agent Benchmarks
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences · Beihang University
Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
Sep 11, 2026Baoyang Jiang, Fengchun Zhang, Leyuan Wang +9Benchmark Construction Embodied QA
QiYuan Lab · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications +3
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
Oct 20, 2025Yulin Luo, Chun-Kai Fan, Menghang Dong +19Long-Horizon Robotic Manipulation Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2

Sep 25, 2026J/K move · Enter open · S save

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu+4

Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences · Beihang University

Embodied AI AI Agent Benchmarks Synthetic Benchmark Generation Scientific Workflow Automation Laboratory Automation

Open paper AskPDF arXiv

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.

Abstract

Embodied Reasoning

Explore similar work

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain