cs.ROOct 26, 2026
SaveEmbodiedLab: Benchmarking Embodied Agents That Act to Know
Organizations: HexaFuture Inc. · Tsinghua University · Peking University · National University of Singapore
Abstract
Embodied agents are expected to accomplish real-world goals by deciding for themselves how and when to interact with their surroundings. For many such goals, changing the physical state of the world is the means rather than the end: the agent changes the state of a system through interaction, observes the feedback, and explores until it either achieves the goal itself (e.g., discovering physical laws) or learns what the goal requires; it acts to know. Manipulation benchmarks stop at acting to change, with the target state given; scientific discovery benchmarks evaluate knowing but replace physical experimentation with symbolic actions or numerical queries. We introduce EmbodiedLab, a benchmark in which a robot must act to know. Quantitative science experiments make this setting controllable: the parameters of the system are resampled for every instance, so the result cannot be known in advance, yet it can be scored exactly. Across 30 experiments in five disciplines on 14 apparatus platforms, the agent receives an objective and the apparatus, chooses its own procedure, and operates a physics-based simulated laboratory, reading the instruments through its cameras. Evaluation checks what the agent came to know, grounded in readings recorded during the episode, as well as the physical state the task requires; every experiment is verified to be solvable. We evaluate six frontier models in their developers' coding agents with native observation and control, with added perception and motion planning, and with manipulation skills distilled from the strongest agents. With native control, even the strongest agent succeeds less than half the time; for every agent, most failures arise in interacting with the apparatus, and manipulation skills help weaker agents more than perception and planning. The benchmark and its evaluation code are available at https://github.com/HexaFuture/EmbodiedLab.
Figures & tables
Figure 1: Overview of EmbodiedLab. Thirty quantitative experiments in five disciplines ask the agent to measure, predict or configure a system drawn anew per instance, by changing its state and reading the feedback on the instruments. Agents are ranked by their success rate at the lowest interface level, T0, and also evaluated at T1; the diagnostic level T2 and four capability dimensions serve the analysis.
Figure 2: Acting to know. (a) Manipulation benchmarks stop at acting to change the scene into a named target state; science benchmarks find things out without physical action. (b) Given a goal, a EmbodiedLab agent changes the state of the apparatus through interaction, observes the feedback and acts again on what it learned, in the same loop across disciplines; at submission, what it found out, the evidence it recorded and the bench state are scored.
| Benchmark | Task | Environment | Metric | ||
| Goal | Left to the agent | Interaction | Information | Success | |
| RoboCasa [ 13 ] | target state | motion | robot | not needed | final state |
| RoboTwin 2.0 [ 14 ] | target state | motion | robot | not needed | final state |
| RoboDojo [ 15 ] | target state | motion | robot | not needed | final state |
| BEHAVIOR-1K [ 31 ] | goal predicates | action order | robot | scene search | goal predicates |
| LabUtopia [ 21 ] | protocol | motion | robot | not needed | steps completed |
Table 1: What a task hands to the agent , along task, environment and metric.
Figure 3: One episode on the lens experiment. (a) A hidden fixes the system; the task card gives the objective, public parameters and rules, but no method. (b) The agent acts and observes through three cameras while the environment keeps a record. (c) Success requires every gate and an error within tolerance of the hidden target.
Figure 4: Interface levels. Robot, bench and camera views are the same at every level; each level supplies more between the agent and the robot. (a) T0 supplies only the camera images, the robot state, joint position targets and gripper commands. (b) T1 supplies ground-truth depth on segmentation masks and cuRobo [ 41 ] planning against the true obstacles, shown with real outputs of T1 runs. (c) T2 adds a skill library, distilled from the successful runs of the strongest agents, in the agent’s workspace.
Figure 5: Construction pipeline , illustrated with the cantilever experiment. Every experiment passes five stages before release; a defect found at any stage is fixed in the bench and the affected stages run again. The validation data are in Fig. 8 .
| Capability dimension | Tier | ||||||
| Agent | Overall | Manip. (7) | Long (7) | Read (7) | Data (9) | Basic (18) | Adv. (12) |
| GPT-6 Astra [ 43 ] | 40.0 62.8 | 28.6 57.9 | 21.4 47.9 | 42.9 62.9 | 61.1 78.3 | 50.0 73.9 | 25.0 46.2 |
| GPT-6.1 Sol [ 44 ] | 28.3 55.7 | 50.0 69.3 | 14.3 45.0 | 35.7 52.1 | 16.7 56.1 | 36.1 58.9 | 16.7 50.8 |
| Claude Opus 5.5 [ 46 ] | 43.3 64.0 | 42.9 65.0 | 42.9 63.6 | 42.9 65.0 | 44.4 62.8 | 38.9 61.9 | 50.0 67.1 |
| DeepSeek-V4.1-Flash-1017 [ 48 ] | 10.0 36.0 | 14.3 31.4 | 0.0 21.4 | 14.3 47.1 | 11.1 42.2 | 16.7 38.9 | 0.0 31.7 |
| Kimi K3 [ 50 ] | 10.0 35.7 | 28.6 48.6 | 0.0 27.1 | 14.3 28.6 | 0.0 37.8 | 16.7 38.3 | 0.0 31.7 |
Table 2: Main results at T0. Success rate (%) with the process score (0–100) in grey, overall and per capability dimension and tier; numbers of experiments in parentheses; best success rate in bold.
Figure 6: Where runs stop at T0. Share of runs that pass each milestone, per agent, averaged as in Eq. 4 ; the last point is the success rate, given in brackets in the legend below, and the steepest segment of a curve is the stage where the agent loses most runs. L1–L2 change the state of the apparatus, L3–L4 observe the feedback until the readings suffice, L5 knows the result within tolerance and L6 restores the bench.
| Agent | T0 | T1 | T2 | T0 T1 | T1 T2 |
| GPT-6 Astra [ 43 ] | 40.0 62.8 | 46.7 61.0 | – | 6.7 | – |
| GPT-6.1 Sol [ 44 ] | 28.3 55.7 | 30.0 51.7 | – | 1.7 | – |
| Claude Opus 5.5 [ 46 ] | 43.3 64.0 | 53.3 73.7 | – | 10.0 | – |
| DeepSeek-V4.1-Flash-1017 [ 48 ] | 10.0 36.0 | 13.3 36.7 | 40.0 57.7 | 3.3 | 26.7 |
| Kimi K3 [ 50 ] | 10.0 35.7 | 20.0 39.7 | 33.3 58.3 | 10.0 | 13.3 |
| GLM-5.3-Flash [ 52 ] | 0.0 16.3 | 3.3 20.7 | 13.3 38.3 | 3.3 | 10.0 |
Table 3: Success across levels : success rate (%) with the process score (0–100) in grey, and the change in success rate in percentage points. T2 skills are distilled from the successful runs of the GPT and Claude agents and are not given back to them.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of the 30 experiments , grouped by primary capability dimension, as seen by one of the robot’s fixed cameras during a reference run. The tag gives the tier.
| Capability dimension | Basic tier (18) | Advanced tier (12) |
| Fine manipulation and contact (7) | Pendulum (#2), coupled oscillators (#3), incline friction (#7), dye absorptivity 4 (#30) | Spring launcher (#14), strong-acid titration (#19), weak-acid titration (#21) |
| Long-horizon multi-stage work (7) | LED wavelength (#10) | Centroid and inertia 1 (#13), three-terminal black box 2 (#16), rectifier supply (#17), spectrophotometric calibration 4 (#18), two-dye mixture 4 (#20), two-lens relay 3 (#23) |
| Fine reading (7) | Hooke’s law (#1), lens focal length 3 (#11), grating wavelength (#12), single-slit width (#25), plate refractive index (#26), thermal expansion (#29) | Cantilever compliance (#15) |
| Multi-point acquisition and analysis (9) | Spring oscillation (#4), torsion pendulum 1 (#5), air-track collision (#6), RC capacitance (#8), black-box resistor 2 (#9), battery EMF (#27), thermistor (#28) | Crystal-violet kinetics (#22), specific heat (#24) |
Table 4: Coverage by capability dimension and tier. Superscript numbers mark same-apparatus pairs ( Appendix B ); targets and tolerances are in Table 6 , benches in Fig. 7 .
| Question | Determine the bending stiffness of a metal strip clamped at one end and the rotational stiffness of its clamp, which turns in proportion to the moment at the strip’s root. |
| Apparatus | The strip sticks out horizontally from its clamp; four knife-edge saddles ride on it at printed distances from the clamp; three steel weights of 50, 100 and 100 g stand in a rack. A digital indicator on a carriage reads the strip’s height on an LCD in mm to three decimals; the carriage clicks into holes along a rail, each printed with its distance from the clamp, and a ZERO cap sets the reading to zero where it stands. |
| Hidden quantities | N m 2 and N m/rad, scored; the indicator offset mm, the rail tilt mm/m and the bow of the unloaded strip mm, which make the unloaded reading differ from hole to hole. |
| Tier | Advanced: (a) the agent chooses which loads at which distances to read, and (c) and add to every deflection and are identifiable only jointly, from readings whose load moments differ. |
| Dimension | Fine reading. |
| Acceptance | Every cited reading comes from a different observation of the episode, with the carriage at rest in the named hole and the value equal to what the LCD showed within 0.001 mm. At submission all weights stand in the rack, nothing rests on the strip or its saddles, the carriage stands in a hole, the robot holds nothing, and no loose item is tipped over or off the bench. One submission within the budget. |
| Process checks | L1: a weight lifted, ZERO pressed or the carriage pushed. L2: a weight pressing on the strip, untouched by the robot, with the carriage in a hole, held for 1.2 s. L3: a first observation of such a loaded state with the LCD readable. L4: valid readings whose differences at a common hole span both unknowns. L5 and L6 as for every experiment. |
Table 5: One experiment in full : the cantilever (#15), condensed from its task card and specification.
| # | Experiment | Target | Evidence the run must contain | Tol. | Disc. | Dim. | Tier |
| 1 | Hooke’s law | spring stiffness | ruler readings at 4 loads, tray at rest and untouched ∗ | 2% | Mech. | Read | basic |
| 2 | Pendulum | acceleration | a completed timing of each of P1–P3, read from its timer ∗ | 0.3% | Mech. | Manip. | basic |
| 3 | Coupled oscillators | normal-mode frequencies | recordings with an untouched stretch of 40 s showing both modes | 0.2% | Mech. | Manip. | basic |
| 4 | Spring oscillation | spring stiffness from oscillation | encoder records of 5 free periods at 3 loads ∗ | 0.2% | Mech. | Data | basic |
| 5 | Torsion pendulum | moment of inertia of an unknown body | photogate records of the empty turntable, the disc and the body | 1% | Mech. | Data | basic |
| 6 | Air-track collision | mass of the unknown cart | a recorded collision, B at rest and no outside force on either cart ∗ | 0.5% | Mech. | Data | basic |
Table 6: The 30 experiments. Tol.: tolerance, relative unless a unit is given; Dim.: primary capability dimension. Letters mark the decisions an advanced experiment requires ( Appendix B ): (a) choosing the conditions, (b) sequential design, (c) model identification, (d) acting on the result. Evidence: what a run must contain beyond readings cited from its own episode that match the instrument; ∗ : the result must also agree with the evaluator’s refit of the cited readings.
Figure 8: Validation. (a) Solvable: largest error of the reference solution over its calibration runs, relative to the tolerance, one bar per experiment colored by discipline; every experiment is solved within its tolerance. (b) Tight: relative tolerance of the experiments, the tightest step their instruments support.
| Level | What the agent receives |
| T0 | Three RGB views with fixed camera calibration, robot state with the hand pose, seven-joint position targets (one per physics step) and gripper commands, an inverse-kinematics solver that turns a hand pose into joint angles without moving the robot, checking collisions or planning a path, the instruments of the apparatus, and submit . Trajectories, collision avoidance, perception and every manipulation skill are the agent’s own. |
| T1 | T0 plus perception and planning: point-prompted segmentation [ 42 ] with a zoomed view; ideal depth, rendered without noise and quantized to 1 mm, with which a chosen pixel is back-projected to the 3D point of the visible surface and a mask to its visible-surface point cloud; Cartesian hand motion and return-to-start planned collision-free [ 41 ] against the bench’s own collision model (as-built geometry, current part poses and the held object); and a slow, short contact move without collision checking. Object poses, centres and occluded geometry are not given; the agent infers them from the visible surface. |
| T2 | T1 plus a library of manipulation skills in the agent’s workspace, distilled from the successful runs of the GPT and Claude agents: locate, grasp, carry, place, press, push and slide. Each skill is a short sequence of public T1 calls, keeps the hand pointing down and reads only the robot state and the public calibration; the agent may read, copy and change it. |
Table 7: Interface levels in full. Each level adds one layer to the level below, and tools outside a level are neither listed nor callable.
| Level | Group | Tools |
| T0 | Task and observation | get_task_spec (task card, submission schema, public calibration); get_observation (the three RGB views, with an evidence id where readings are cited); get_robot_state (joint positions, gripper width, whether the gripper holds a part, and the hand pose) |
| Motion | execute_joint_trajectory (seven-joint position targets, one per physics step); set_gripper ; wait (at most 30 s of simulated time); solve_ik (kinematics only, without collision checking) | |
| Instrument records | one export per experiment that records a dynamic curve, 19 in all: export_logger (six circuit experiments), export_recording (#3, #6, #7), export_photogate (#5, #13), export_titration_log (#19, #21), export_spectro_scans (#20, #30), export_encoder (#4), export_flight (#14), export_spectro_kinetics (#22), export_temperature_log (#24); a static reading is read from the cameras | |
| Submission | submit_result (once; the first submission closes the episode) | |
| T1 | Perception | segment_sam3_point_prompt (mask candidates at a chosen pixel); inspect_mask_image (close-up of a candidate); localize_rgbd_point (3D point of a chosen pixel); mask_geometry (bounds and median of a mask’s visible points) |
| Planning | move_hand (collision-free path to a hand pose); plan_return_to_start ; translate_contact (slow straight move of at most 5 cm, to touch or back out of a contact); get_robot_pose |
Table 8: Public tools by group and level. T2 adds no tools: its skills are code in the agent’s workspace that calls the T1 tools.
| # | Experiment | Astra | Sol | Opus 5.5 | DeepSeek | Kimi K3 | GLM |
| 1 | Hooke’s law | 100 100 | 50 55 | 0 35 | 0 70 | 0 0 | 0 10 |
| 2 | Pendulum | 100 100 | 100 100 | 100 100 | 0 30 | 100 100 | 0 20 |
| 3 | Coupled oscillators | 0 70 | 100 100 | 0 45 | 100 100 | 100 100 | 0 10 |
| 4 | Spring oscillation | 100 100 | 50 65 | 50 70 | 0 30 | 0 30 | 0 30 |
| 5 | Torsion pendulum | 50 85 | 50 70 | 0 20 | 0 30 | 0 30 | 0 30 |
| 6 | Air-track collision | 100 100 | 50 60 | 0 20 | 100 100 | 0 70 | 0 10 |
Table 9: Per-experiment success at T0 (%), with the process score in grey, each averaged over the runs of the experiment.
Figure 9: Lowest level that solves each experiment , per agent; a dot marks an experiment the agent solves at one level but not at the next level it is run at. The GPT and Claude agents are not run at T2.
Figure 10: Why runs fail at T0. (a) Failed runs by the cause of their first decisive error, in percent of each agent’s failed runs with an attributed cause, grouped by the stage of acting to know. (b) Typical failures, five per stage, each shown in a frame from the agent’s own cameras with the decisive spot ringed; the chip on each frame names the cause.