Embodied agents are expected to accomplish real-world goals by deciding for themselves how and when to interact with their surroundings. For many such goals, changing the physical state of the world is the means rather than the end: the agent changes the state of a system through interaction, observes the feedback, and explores until it either achieves the goal itself (e.g., discovering physical laws) or learns what the goal requires; it acts to know. Manipulation benchmarks stop at acting to change, with the target state given; scientific discovery benchmarks evaluate knowing but replace physical experimentation with symbolic actions or numerical queries. We introduce EmbodiedLab, a benchmark in which a robot must act to know. Quantitative science experiments make this setting controllable: the parameters of the system are resampled for every instance, so the result cannot be known in advance, yet it can be scored exactly. Across 30 experiments in five disciplines on 14 apparatus platforms, the agent receives an objective and the apparatus, chooses its own procedure, and operates a physics-based simulated laboratory, reading the instruments through its cameras. Evaluation checks what the agent came to know, grounded in readings recorded during the episode, as well as the physical state the task requires; every experiment is verified to be solvable. We evaluate six frontier models in their developers' coding agents with native observation and control, with added perception and motion planning, and with manipulation skills distilled from the strongest agents. With native control, even the strongest agent succeeds less than half the time; for every agent, most failures arise in interacting with the apparatus, and manipulation skills help weaker agents more than perception and planning. The benchmark and its evaluation code are available at https://github.com/HexaFuture/EmbodiedLab.
Figures & tables
Figure 1: Overview of EmbodiedLab. Thirty quantitative experiments in five disciplines ask the agent to measure, predict or configure a system drawn anew per instance, by changing its state and reading the feedback on the instruments. Agents are ranked by their success rate at the lowest interface level, T0, and also evaluated at T1; the diagnostic level T2 and four capability dimensions serve the analysis.
Figure 2: Acting to know. (a) Manipulation benchmarks stop at acting to change the scene into a named target state; science benchmarks find things out without physical action. (b) Given a goal, a EmbodiedLab agent changes the state of the apparatus through interaction, observes the feedback and acts again on what it learned, in the same loop across disciplines; at submission, what it found out, the evidence it recorded and the bench state are scored.
Benchmark
Task
Environment
Metric
Goal
Left to the agent
Interaction
Information
Success
RoboCasa [ 13 ]
target state
motion
robot
not needed
final state
RoboTwin 2.0 [ 14 ]
target state
motion
robot
not needed
final state
RoboDojo [ 15 ]
target state
motion
robot
not needed
final state
BEHAVIOR-1K [ 31 ]
goal predicates
action order
robot
scene search
goal predicates
LabUtopia [ 21 ]
protocol
motion
robot
not needed
steps completed
Table 1: What a task hands to the agent , along task, environment and metric.
Figure 3: One episode on the lens experiment. (a) A hidden θ fixes the system; the task card gives the objective, public parameters and rules, but no method. (b) The agent acts and observes through three cameras while the environment keeps a record. (c) Success requires every gate and an error within tolerance of the hidden target.
Figure 4: Interface levels. Robot, bench and camera views are the same at every level; each level supplies more between the agent and the robot. (a) T0 supplies only the camera images, the robot state, joint position targets and gripper commands. (b) T1 supplies ground-truth depth on segmentation masks and cuRobo [ 41 ] planning against the true obstacles, shown with real outputs of T1 runs. (c) T2 adds a skill library, distilled from the successful runs of the strongest agents, in the agent’s workspace.
Figure 5: Construction pipeline , illustrated with the cantilever experiment. Every experiment passes five stages before release; a defect found at any stage is fixed in the bench and the affected stages run again. The validation data are in Fig. 8 .
Capability dimension
Tier
Agent
Overall
Manip. (7)
Long (7)
Read (7)
Data (9)
Basic (18)
Adv. (12)
GPT-6 Astra [ 43 ]
40.0 62.8
28.6 57.9
21.4 47.9
42.9 62.9
61.1 78.3
50.0 73.9
25.0 46.2
GPT-6.1 Sol [ 44 ]
28.3 55.7
50.0 69.3
14.3 45.0
35.7 52.1
16.7 56.1
36.1 58.9
16.7 50.8
Claude Opus 5.5 [ 46 ]
43.3 64.0
42.9 65.0
42.9 63.6
42.9 65.0
44.4 62.8
38.9 61.9
50.0 67.1
DeepSeek-V4.1-Flash-1017 [ 48 ]
10.0 36.0
14.3 31.4
0.0 21.4
14.3 47.1
11.1 42.2
16.7 38.9
0.0 31.7
Kimi K3 [ 50 ]
10.0 35.7
28.6 48.6
0.0 27.1
14.3 28.6
0.0 37.8
16.7 38.3
0.0 31.7
Table 2: Main results at T0. Success rate (%) with the process score (0–100) in grey, overall and per capability dimension and tier; numbers of experiments in parentheses; best success rate in bold.
Figure 6: Where runs stop at T0. Share of runs that pass each milestone, per agent, averaged as in Eq. 4 ; the last point is the success rate, given in brackets in the legend below, and the steepest segment of a curve is the stage where the agent loses most runs. L1–L2 change the state of the apparatus, L3–L4 observe the feedback until the readings suffice, L5 knows the result within tolerance and L6 restores the bench.
Agent
T0
T1
T2
T0 → T1
T1 → T2
GPT-6 Astra [ 43 ]
40.0 62.8
46.7 61.0
–
+ 6.7
–
GPT-6.1 Sol [ 44 ]
28.3 55.7
30.0 51.7
–
+ 1.7
–
Claude Opus 5.5 [ 46 ]
43.3 64.0
53.3 73.7
–
+ 10.0
–
DeepSeek-V4.1-Flash-1017 [ 48 ]
10.0 36.0
13.3 36.7
40.0 57.7
+ 3.3
+ 26.7
Kimi K3 [ 50 ]
10.0 35.7
20.0 39.7
33.3 58.3
+ 10.0
+ 13.3
GLM-5.3-Flash [ 52 ]
0.0 16.3
3.3 20.7
13.3 38.3
+ 3.3
+ 10.0
Table 3: Success across levels : success rate (%) with the process score (0–100) in grey, and the change in success rate in percentage points. T2 skills are distilled from the successful runs of the GPT and Claude agents and are not given back to them.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of the 30 experiments , grouped by primary capability dimension, as seen by one of the robot’s fixed cameras during a reference run. The tag gives the tier.
Crystal-violet kinetics (#22), specific heat (#24)
Appendix
Table 4: Coverage by capability dimension and tier. Superscript numbers mark same-apparatus pairs ( Appendix B ); targets and tolerances are in Table 6 , benches in Fig. 7 .
Question
Determine the bending stiffness EI of a metal strip clamped at one end and the rotational stiffness kθ of its clamp, which turns in proportion to the moment at the strip’s root.
Apparatus
The strip sticks out horizontally from its clamp; four knife-edge saddles ride on it at printed distances from the clamp; three steel weights of 50, 100 and 100 g stand in a rack. A digital indicator on a carriage reads the strip’s height on an LCD in mm to three decimals; the carriage clicks into holes along a rail, each printed with its distance from the clamp, and a ZERO cap sets the reading to zero where it stands.
Hidden quantities
EI∼U(3,8) N m 2 and kθ∼U(80,300) N m/rad, scored; the indicator offset U(−0.5,0.5) mm, the rail tilt U(−1.5,1.5) mm/m and the bow of the unloaded strip U(−0.3,0.3) mm, which make the unloaded reading differ from hole to hole.
Tier
Advanced: (a) the agent chooses which loads at which distances to read, and (c) EI and kθ add to every deflection and are identifiable only jointly, from readings whose load moments differ.
Dimension
Fine reading.
Acceptance
Every cited reading comes from a different observation of the episode, with the carriage at rest in the named hole and the value equal to what the LCD showed within 0.001 mm. At submission all weights stand in the rack, nothing rests on the strip or its saddles, the carriage stands in a hole, the robot holds nothing, and no loose item is tipped over or off the bench. One submission within the budget.
Process checks
L1: a weight lifted, ZERO pressed or the carriage pushed. L2: a weight pressing on the strip, untouched by the robot, with the carriage in a hole, held for 1.2 s. L3: a first observation of such a loaded state with the LCD readable. L4: valid readings whose differences at a common hole span both unknowns. L5 and L6 as for every experiment.
Appendix
Table 5: One experiment in full : the cantilever (#15), condensed from its task card and specification.
#
Experiment
Target
Evidence the run must contain
Tol.
Disc.
Dim.
Tier
1
Hooke’s law
spring stiffness k
ruler readings at ≥ 4 loads, tray at rest and untouched ∗
2%
Mech.
Read
basic
2
Pendulum g
acceleration g
a completed timing of each of P1–P3, read from its timer ∗
0.3%
Mech.
Manip.
basic
3
Coupled oscillators
normal-mode frequencies ω1<ω2
recordings with an untouched stretch of ≥ 40 s showing both modes
0.2%
Mech.
Manip.
basic
4
Spring oscillation
spring stiffness k from oscillation
encoder records of ≥ 5 free periods at ≥ 3 loads ∗
0.2%
Mech.
Data
basic
5
Torsion pendulum
moment of inertia of an unknown body
photogate records of the empty turntable, the disc and the body
1%
Mech.
Data
basic
6
Air-track collision
mass of the unknown cart
a recorded collision, B at rest and no outside force on either cart ∗
0.5%
Mech.
Data
basic
Appendix
Table 6: The 30 experiments. Tol.: tolerance, relative unless a unit is given; Dim.: primary capability dimension. Letters mark the decisions an advanced experiment requires ( Appendix B ): (a) choosing the conditions, (b) sequential design, (c) model identification, (d) acting on the result. Evidence: what a run must contain beyond readings cited from its own episode that match the instrument; ∗ : the result must also agree with the evaluator’s refit of the cited readings.
Figure 8: Validation. (a) Solvable: largest error of the reference solution over its calibration runs, relative to the tolerance, one bar per experiment colored by discipline; every experiment is solved within its tolerance. (b) Tight: relative tolerance of the experiments, the tightest step their instruments support.
Level
What the agent receives
T0
Three RGB views with fixed camera calibration, robot state with the hand pose, seven-joint position targets (one per physics step) and gripper commands, an inverse-kinematics solver that turns a hand pose into joint angles without moving the robot, checking collisions or planning a path, the instruments of the apparatus, and submit . Trajectories, collision avoidance, perception and every manipulation skill are the agent’s own.
T1
T0 plus perception and planning: point-prompted segmentation [ 42 ] with a zoomed view; ideal depth, rendered without noise and quantized to 1 mm, with which a chosen pixel is back-projected to the 3D point of the visible surface and a mask to its visible-surface point cloud; Cartesian hand motion and return-to-start planned collision-free [ 41 ] against the bench’s own collision model (as-built geometry, current part poses and the held object); and a slow, short contact move without collision checking. Object poses, centres and occluded geometry are not given; the agent infers them from the visible surface.
T2
T1 plus a library of manipulation skills in the agent’s workspace, distilled from the successful runs of the GPT and Claude agents: locate, grasp, carry, place, press, push and slide. Each skill is a short sequence of public T1 calls, keeps the hand pointing down and reads only the robot state and the public calibration; the agent may read, copy and change it.
Appendix
Table 7: Interface levels in full. Each level adds one layer to the level below, and tools outside a level are neither listed nor callable.
Level
Group
Tools
T0
Task and observation
get_task_spec (task card, submission schema, public calibration); get_observation (the three RGB views, with an evidence id where readings are cited); get_robot_state (joint positions, gripper width, whether the gripper holds a part, and the hand pose)
Motion
execute_joint_trajectory (seven-joint position targets, one per physics step); set_gripper ; wait (at most 30 s of simulated time); solve_ik (kinematics only, without collision checking)
Instrument records
one export per experiment that records a dynamic curve, 19 in all: export_logger (six circuit experiments), export_recording (#3, #6, #7), export_photogate (#5, #13), export_titration_log (#19, #21), export_spectro_scans (#20, #30), export_encoder (#4), export_flight (#14), export_spectro_kinetics (#22), export_temperature_log (#24); a static reading is read from the cameras
Submission
submit_result (once; the first submission closes the episode)
T1
Perception
segment_sam3_point_prompt (mask candidates at a chosen pixel); inspect_mask_image (close-up of a candidate); localize_rgbd_point (3D point of a chosen pixel); mask_geometry (bounds and median of a mask’s visible points)
Planning
move_hand (collision-free path to a hand pose); plan_return_to_start ; translate_contact (slow straight move of at most 5 cm, to touch or back out of a contact); get_robot_pose
Appendix
Table 8: Public tools by group and level. T2 adds no tools: its skills are code in the agent’s workspace that calls the T1 tools.
#
Experiment
Astra
Sol
Opus 5.5
DeepSeek
Kimi K3
GLM
1
Hooke’s law
100 100
50 55
0 35
0 70
0 0
0 10
2
Pendulum g
100 100
100 100
100 100
0 30
100 100
0 20
3
Coupled oscillators
0 70
100 100
0 45
100 100
100 100
0 10
4
Spring oscillation
100 100
50 65
50 70
0 30
0 30
0 30
5
Torsion pendulum
50 85
50 70
0 20
0 30
0 30
0 30
6
Air-track collision
100 100
50 60
0 20
100 100
0 70
0 10
Appendix
Table 9: Per-experiment success at T0 (%), with the process score in grey, each averaged over the runs of the experiment.
Figure 9: Lowest level that solves each experiment , per agent; a dot marks an experiment the agent solves at one level but not at the next level it is run at. The GPT and Claude agents are not run at T2.
Figure 10: Why runs fail at T0. (a) Failed runs by the cause of their first decisive error, in percent of each agent’s failed runs with an attributed cause, grouped by the stage of acting to know. (b) Typical failures, five per stage, each shown in a frame from the agent’s own cameras with the decisive spot ringed; the chip on each frame names the cause.
Realizing Embodied AI for Science requires robots to translate experimental goals into reliable physical procedures, reducing hazardous and repetitive human work while generating evidence for scientific discovery. Although language-based planning and learned manipulation provide complementary capabilities, their integration requires robots to adapt subsequent operations to actual execution outcomes. For example, an unsuccessful placement changes what must be done next, even when the experimental goal remains unchanged. Existing benchmarks assess planning, manipulation, and laboratory workflows through task completion and reasoning or execution metrics; however, aggregate performance leaves the connection between outcome-dependent choices and physical execution unresolved. To address this evaluation gap, we introduce Labference, a benchmark for evaluating how observed physical outcomes guide subsequent choices and how those choices translate into robotic laboratory actions. Its evaluation protocol is designed to test outcome-dependent behavioral adaptation through controlled evidence variations, while independently measuring operation selection, physical execution, and task completion. Matched Oracle-assisted continuations from shared checkpoints then assess which forms of support improve subsequent choices and task completion. Initial weighing evaluations reveal target-selection errors in the tested VLA policies despite successful object transfers, whereas GPT-driven control achieves correct selection but encounters execution failures and substantial computation costs. Together, these findings highlight the need to connect task-appropriate reasoning with reliable, timely action. By making this connection measurable and diagnosable, Labference supports targeted improvements toward autonomous experimentation for Embodied AI for Science.
Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.
Maokai Qin, Chuan Qin, Qi Zhang +5
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences · Beihang University
Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool. We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing. We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a π0.5 policy fine-tuned on the full-episode demonstrations we release. The best agent succeeds in only 23% of the episodes, and the fine-tuned policy almost never succeeds. Isolated tests of the execution skills the tasks are built from, with the hidden information supplied, show that the agents can carry out most of the required actions, and our failure analysis attributes only a minority of the failures to execution. Our failure analysis further finds that the agents often stop exploring too early as they make decisions before observing the required evidence for task completion. We also find that agents rarely prevent or repair the disturbances caused by their exploration. Moreover, learning by trial and error remains difficult for most models.