cs.ROOct 25, 2026
SaveHow High Can GPT-6 Astra Climb in Contact-Rich Manipulation? PhysLadder: A Hierarchical, Matched Sim-Real Benchmark for Physical Interaction
Organizations: Tuojing Intelligence · The University of Hong Kong · Chinese Academy of Sciences · Nanyang Technological University ·
Abstract
We introduce PhysLadder, a hierarchical Sim-Real benchmark for evaluating physical interaction capabilities in contact-rich manipulation. Its 16 household tasks span three cumulative levels: contact selection and geometric alignment (L1), force regulation within sustained contact (L2), and coordinated contact reconfiguration and force regulation (L3). Each task is instantiated in physically and visually matched simulated and real-world environments, with shared task definitions and success criteria. We evaluate GPT-6 Astra zero-shot with three trials per task in each environment. Real-world success rates decline from 80.0% at L1 to 47.6% at L2 and 0.0% at L3, with corresponding simulation rates of 66.7%, 47.6%, and 0.0%. Ten of the 16 tasks have identical success counts across environments; the remaining six differ by one success out of three trials. Across both environments, recurring difficulties in spatial reasoning, contact selection, force regulation, and coordination often prevent local physical progress from accumulating into task completion. These findings characterize Astra's observed capability boundary across increasing interaction demands and, for this model, support matched simulation as a repeatable, scalable testbed for studying the sustained, coordinated physical interactions that limit reliable manipulation.
Figures & tables
Figure 2: Contact-rich manipulation tasks and representative initial-state variations. (a) The 16 tasks span three interaction levels, identified by task tags colored by level: L1 (5 tasks) emphasizes contact selection and geometric alignment; L2 (7 tasks) adds force regulation within sustained contact; L3 (4 tasks) requires coordinated contact reconfiguration and force regulation. Each task unit pairs a simulated view (left, violet bar) with a real-world view (right, orange bar). (b) Four representative simulation tasks are each shown in three initial configurations, illustrating variations in object positions, relative placement, and orientation. Marker Cap Removal is shown with horizontal, vertical, and diagonal marker placements. Task tags follow the same L1–L3 grouping and task IDs in both panels. Task specifications are detailed in Appendix A .
| Level | Force regulation | Contact configuration | Examples of observable failure |
| L1 | No additional regulation beyond ordinary support, transport, and geometric placement | Suitable contacts must be selected; sequential grasps for positioning are allowed | Misalignment; a grasp location that blocks placement or alignment |
| L2 | Required, in the task-level sense defined above | Reconfiguration is not required to continue the main operation | Slip, stalled motion, crushing or damage, displacement, spilling |
| L3 | Required and coordinated with contact reconfiguration | Contacts must be re-established to continue the main operation | Lost torque transmission or support after a contact change; progress that does not accumulate |
Table 1: Interaction-level criteria. Requirements accumulate across levels. The last column gives examples of observable failure, which do not uniquely identify the failed requirement. Table S1 applies the criteria to all 16 tasks.
Figure 3: Real-world experimental setup. (a) Two ARX R5 arms with two-finger grippers share a tabletop workspace observed by one external Orbbec Gemini Pro camera and two wrist-mounted Orbbec Gemini 305 cameras. The external-camera mount is visible at left; the inset identifies the camera used. (b) Left and (c) right wrist cameras with their two-finger grippers. Photographs show the hardware before task objects are placed.
Figure 4: Simulation task assets and interaction-relevant features. (a) Illustrative views of 21 representative task asset types. (b) Representative examples illustrate relative motion and separation between components, deformation under contact, inter-object sliding, and changes in visible surface state during wiping. Jelly deformation and wiping are shown using recorded simulation frames; component and book arrangements illustrate representative configurations. Asset views are not to scale. Simulation settings and contact and material models are provided in Appendix B.2 .
Figure 5: Physical and visual matching of real and simulated environments. Violet and orange bars denote simulated and real views, respectively; paired views place simulation to the left of or above the corresponding real view. (a) Workspace appearance and initial object placement for Block Sweeping (left) and the initial screw–nut engagement for Nut Unscrewing (right). (b1) Capped marker geometry with simulated and real lengths, and printed surface texture; the real length is documented by a ruler photograph. (b2) The configured jelly mass in simulation (26.4 g) and the documented scale reading (26.4 g). (b3) Recorded jelly states before and during gripper contact: simulation at 00:30/00:40 and real world at 01:16/01:22 of their source clips. The same global color-balance adjustment is applied to both simulated frames; object geometry and deformation are unmodified. These qualitative views are not synchronized or a quantitative validation of matched physical response. (c) Arm-base placement and empty two-finger gripper geometry. (d) External, left-wrist, and right-wrist views for Block Sweeping. The external views and simulated left-wrist view show the initial state. The real left-wrist frame at 16:40.5 and right-wrist frame at 28:18 are selected during execution to match the gripper–box and gripper–dustpan geometry, respectively. The views are unsynchronized.
Figure 6: Model inputs and closed-loop execution. (a) One decision cycle. 1 The task instruction and the interface description are fixed for the episode; the interface description is given as model instructions in simulation and by the basic Skills in the real world, and no further hints are provided. 2 The model observes the external and wrist camera views, proprioceptive states, execution feedback, and the interaction history. 3 GPT-6 Astra selects the next action. 4 In simulation, the cuRobo planner and joint drives execute a structured action; in the real world, tool calls drive the ARX R5 arms through a control service. 5 New observations verify the outcome before the model continues, corrects, or requests termination. (b) The real-world basic Skills describe the embodiment and interface and give general guidance on observation and verification, dual-arm coordination, and local recovery; their SKILL.md files remain fixed across trials (Appendix C.1 ). (c) In matched simulated and real-world Marker Cap Removal scenes, both grasps must remain stable and the cap must separate before the next action.
| Task ID | Task | Simulation zero-shot | Real-world zero-shot |
| SR ( ) | SR ( ) | ||
| L1 mean | 66.7% (10/15) | 80.0% (12/15) | |
| L1-01 | Marker Placement | 100% (3/3) | 100% (3/3) |
| L1-02 | Charger Alignment | 66.7% (2/3) | 100% (3/3) |
| L1-03 | Hat Hanging | 0% (0/3) | 0% (0/3) |
| L1-04 | Cup Nesting | 66.7% (2/3) | 100% (3/3) |
Table 2: Zero-shot performance across tasks and interaction levels. Real and simulated results show broadly similar performance profiles, with success rates declining from L1 to L3 and no observed L3 successes. Each task has three trials per environment; denotes successes/trials. All means weight tasks equally.
Figure 7: Failure-pattern comparison across real and simulated executions. (a) Hat–rack alignment. (b) Cap contact during uncapping. (c) Unsuccessful jelly lifting under coupled contact and force demands. (d) Cap slippage. (e) Coordination of spatial reasoning, contact selection, and force regulation during sweeping. Headers give the task ID, interaction bottleneck (Section 6.4 ), and success counts in simulation and the real world (Table 2 ). Multiview images place the wrist views above the external view. Labels describe observable states.
Figure 8: Comparing local interaction progress and task outcomes across real and simulated environments. Panel (a) concerns assessment granularity: the same alignment criterion was judged visually in the real world and checked with a fixed geometric tolerance in simulation. The unsuccessful simulated trial exhausted its model-call budget during repeated corrections. Panels (b–d) concern final-state stability after release, grasp selection with bimanual force regulation, and local drawer motion and force regulation. Headers follow Figure 7 , except that panel (b) reports an observed outcome without assigning a specific interaction bottleneck.
Figure 9: Initial configurations change the interaction demands of Block Sweeping. (a) Two blocks near the dustpan entrance are collected through local sweeping under a relatively stable contact mode, consistent with L2. (b) Three dispersed blocks require successive contact reconfiguration and force regulation, consistent with L3; the recorded execution remains incomplete. Each row shows initial, intermediate, and final states. These variants differ in block count, layout, and dustpan orientation; they are not a controlled layout ablation or extra trials in Table 2 .
Figure 10: Additional failure cases. (a) Grasping attempts leave an upright orange segment on its support. (b) Wiping leaves visible writing on the board. (c) The nut remains on the threaded rod despite bimanual contact; the execution record reports infeasible subsequent turning poses. (d) Supported pushing produces partial translation and rotation without completing the full-turn transfer. These cases complement Figures 7 and 8 . SIM/REAL labels report successes out of three trials per task (Table 2 ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Contact configuration | Force-regulation requirements | Contact reconfiguration |
| L1: contact selection and geometric alignment | |||
| L1-01 Marker Placement | Grasp leaves one end free for placement | No additional requirement | Not required |
| L1-02 Charger Alignment | Gripper–charger; prongs aligned, not inserted | No additional requirement | Not required |
| L1-03 Hat Hanging | Grasp leaves the head opening accessible | No additional requirement | Not required |
| L1-04 Cup Nesting | Sequential grasps of the two cups | No additional requirement | Placement-only sequential grasps |
| L1-05 Flower Transfer | Grasp leaves sufficient stem length free | No additional requirement | Not required |
Table S1: Interaction-level assignment of the 16 tasks. Requirements follow the definitions in Section 3.1 (Table 1 ). No additional requirement means no force-regulation requirement beyond ordinary support, transport, and geometric placement. Natural rolling, sliding, and content release do not by themselves constitute contact reconfiguration.
| Component | Setting |
| Physics stepping | 240 Hz by default; 1920 Hz for Marker Cap Removal, Bottle Cap Unscrewing, and Nut Unscrewing; 360 Hz for Jelly Handling. |
| Constraint solver | TGS; 32 articulation position iterations; 4 velocity iterations for L1 tasks and 0 for all other tasks. |
| Joint drives | Stiffness/damping in Isaac Sim joint-drive units: arm joints 1500/80; finger joints 2000/40. |
| Gripper opening | Range 0–70 mm; each finger joint moves by half of the commanded opening. |
| Camera images | Rendered at pixels (external) and pixels (each wrist); downscaled to a 768-pixel long edge before being sent to the model. |
| Orange contact margins | Contact offset 3 mm; rest offset 1.5 mm. |
Table S2: Simulation settings used in the evaluation. Material and wiping values are simulation parameters rather than measured physical properties.
Figure S1: Basic Skills for real-world execution. The guidance covers embodiment checks, active observation, gripper control, contact verification, dual-arm coordination, recovery, and completion assessment.
| Task ID | Task | Simulation | Real world |
| L1 mean | 46 | 203 | |
| L1-01 | Marker Placement | 18 | 124 |
| L1-02 | Charger Alignment | 74 | 188 |
| L1-03 | Hat Hanging | 84 | 311 |
| L1-04 | Cup Nesting | 35 | 236 |
| L1-05 | Flower Transfer | 16 | 156 |
Table S3: Mean model calls per task, reported for reference. Successful and unsuccessful trials are included. Values are rounded to the nearest integer; level and overall rows average unrounded task means. Counts are not directly comparable between environments: each simulated call returns one structured action executed by the planner, whereas a real-world action takes several tool calls, since the model requests fresh observations and moves the arms in bounded segments to refine their poses.
More in this topic
Contact-Rich Robotic Manipulation
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
See all 413 papers