JustRUA: A Controlled Study of GPT-6 Robot-Use Agents
Organizations: Tsinghua University · ModelBest · Huazhong University of Science and Technology · Peking University ·
Abstract
GPT-6 Astra can drive a robot arm directly through a fixed harness, and scores of such robot-use agents on RoboDojo have risen quickly since. Reported systems differ in harness and test subset and often change several designs at once, so it is hard to tell which choices produce the gains. We take these designs apart and test them in one framework, along the harness around the model, the action interface, the reuse of experience and the scene geometry given to the model. A lean harness that keeps the full episode history and stops each motion early goes a long way. Reusing experience changes how the model acts but is not a free lunch, and scene geometry helps most, even when depth is predicted from RGB alone. JustRUA starts from the official L3 harness and keeps only what matters most. Even with no learned policy, self-improvement loop or task-specific tuning, this simple harness raises the score on the RoboDojo simulation benchmark substantially, from 29.7 to 50.3.
Figures & tables
| Configuration | Gen. | Prec. | Long. | Mem. | Open | Overall | Success (%) | Cost ($) |
| Harness | ||||||||
| Official L3 harness † | 33.4 | 12.7 | 21.5 | 43.0 | 34.4 | 29.7 | 22.5 | – |
| Codex, full history | 36.0 | 9.4 | 24.5 | 52.5 | 50.9 | 34.7 | 27.1 | 3.76 |
| Action, on full history | ||||||||
| 20-step cap (default testbed) | 42.5 | 25.0 | 24.7 | 49.6 | 48.1 | 38.0 | 31.0 | 4.94 |
| Model-chosen step count | 40.4 | 14.4 | 32.5 | 60.4 | 41.6 | 37.9 | 30.0 | 4.95 |
| Absolute targets | Relative targets | Semantic operations | Move sequences | Model-chosen step count | |
| Empty grasps | 47 | 54 | 47 | 38 | 45 |
| Dropped objects | 26 | 30 | 35 | 27 | 29 |
| Fine placement failures | 20 | 25 | 19 | 20 | 22 |
| View adjustments | 59 | 54 | 211 | 45 | 61 |
| Handovers | 15 | 15 | 18 | 14 | 14 |
| Arm hits an object | 7 | 20 | 12 | 1 | 11 |
| Default | Visual demonstration | Text trajectory | Self-built library | |
| Handovers through the table | 9 | 15 | 13 | 9 |
| Dropped objects | 26 | 20 | 17 | 22 |
| Arm-to-arm collisions | 6 | 0 | 0 | 0 |
| Empty grasps | 47 | 50 | 53 | 33 |
| Decisions per episode | 67.7 | 65.8 | 71.1 | 60.9 |
| Cost per episode ($) | 4.94 | 5.11 | 5.84 | 5.43 |
| Reference | Final library | Reference is | |
| Evaluation set | 38.0 | 37.0 | the default testbed |
| Iterate set | 35.4 | 36.4 | the runs inside the loop |
| Empty grasps, first round | 47 | 33 | the default testbed |
| Arm-to-arm collisions, first round | 6 | 0 | the default testbed |
| Configuration | Empty grasps | Fine placement failures | Hand- overs | Free-arm views | Added context |
| Robot readings | |||||
| None (default testbed) | 47 | 20 | 15 | 34 | 0 |
| Full JSON | 47 | 23 | 11 | 32 | 3,100 |
| Compact text | 41 | 18 | 13 | 32 | 1,800 |
| Depth | |||||
| Simulator depth (reference) | 17 | 15 | 9 | 25 | 3,500 |
| Cause | Definition and example | Tasks | Potential fix |
| Strategy | The plan does not suit the arm, e.g. one arm cannot flip a tile and place it at once ( make_kong ). | 8 | Planning with high-level knowledge of the arm and its limits. |
| Collision | Unplanned knocks, as the model misjudges the size of the arm or of what it holds ( pack_objects_into_box ). | 6 | Collision checking with complete assets of the arm and held objects. |
| Unstable grasp | Irregular or round objects slip out of the gripper, such as a nesting doll. | 6 | Grasp planning on the same assets. |
| Prompt or scoring | The task looks easy, but the prompt or the scoring seems to be at fault ( fasten_screws ). | 5 | Revising the task prompts, which raises the first-round score from 54.0 to 55.6. |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Axis | Variant | Description |
| Harness the loop around the model | Official L3 | Reference loop, keeps the images of the last two observations |
| Full history | Codex session that keeps every image of the episode | |
| Low effort | Reasoning effort below the default, medium | |
| High effort | Reasoning effort above the default | |
| Action how a decision becomes motion | 20-step cap | Each motion stops after at most 20 control steps |
| Model-chosen steps | 3 to 32 steps per motion, set by the model |
| Interface | Tools | Semantics |
| Absolute targets (default) | move_eef(targets, note) give_up() | World-frame targets for the grasp point of either arm, left_x/y/z in metres, left_pitch/roll/yaw_deg , and left_gripper from 0 (closed) to 1 (open), with the same fields for the right arm. Both arms can move in one call. Unnamed dimensions keep their observed value. Orientation is measured from a fixed straight-down grasp rather than added to the current one, and a gripper command runs after the arm motion. give_up ends an episode the model judges unrecoverable. |
| Relative targets | move_eef_relative( targets, note) give_up() | Increments left_dx/dy/dz in metres and left_drotation_x/y/z_deg from the full-precision pose measured at this decision, along and about fixed world axes. Missing increments are zero. Rotations apply about , then , then , and keep the grasp point in place. The gripper stays absolute. Observations and recipes still report absolute positions. |
| Semantic operations | approach , transport , retreat , grasp , pick , clamp , place , deposit , release , handover , pour_step , adjust_view , move_eef , give_up | Each call names the object it acts on, a label that records intent and does not steer motion, and expands into stages. Every stage runs through the original planner for at most 20 control steps, and the next stage starts only after the arm arrives within 1 cm and 5 ∘ . approach , transport and retreat move the named pose dimensions while the rest of the pose and the grip hold. grasp and place move and then close or open if asked. pick moves, closes and lifts, and deposit moves, opens and retreats, in one call. clamp and release set the jaw only. In handover the giver moves, the receiver opens, approaches, grasps and closes, and the giver keeps its grip until an explicit release. pour_step is one pouring motion and adjust_view one move of a wrist camera to see better. |
| Model-chosen steps | move_eef(..., num_actions) | As the default, with 3 to 32 control steps per motion chosen by the model in place of the fixed cap of 20. |
| Move sequences | move_eef_sequence( num_moves, moves) | Up to four move_eef moves per decision, executed in order. |
| Default | Sequences | |
| Evaluation set | 38.0 | 42.0 |
| capped six keep old scores | – | 39.4 |
| Decisions per episode (mean) | 63.2 | 52.7 |
| Decisions, typical episode | 65 | 51 |
| Decisions, 31 both finish | 50.2 | 40.0 |
| Successes, capped six | 0 | 3 |
| Configuration | First round (42 episodes) | Evaluation set (111 episodes) |
| Official L3 harness | 32.3 | 29.7 |
| Codex, full history | 43.5 | 34.7 |
| 20-step cap (default testbed) | 42.4 | 38.0 |
| Compact robot readings | 38.3 | 39.8 |
| Predicted depth | 47.7 | 42.4 |
| Coordinate solving (JustRUA) | 54.0 | 50.3 |
| Behaviour | Counts | Does not count |
| Empty grasp | The gripper closes on an object but does not hold it. Each separate attempt counts once, whether or not it is retried. | Closing on purpose to press or push, or closing an empty hand. |
| Dropped object | An object that has been lifted and held is lost before the model releases it. | An object that tips over or lands off target after a deliberate release. |
| Fine placement failure | One attempt at insertion, hanging, assembly, stacking or precise alignment that does not meet the task’s requirement. Backing off and trying again starts a new attempt. | Small corrections within one continuous attempt, and ordinary drops during transport. |
| View adjustment | An executed motion made to see something, by the acting arm or by the free arm, which may move to look at an occluded area or out of the way. Counted once per motion phase. | Normal approach motions, and moves made only to avoid a collision. |
| Handover | An executed transfer of one object between the arms, directly from gripper to gripper or through the table or another support. | A transfer planned only in text, or picking up an object the other arm dropped by accident. |
| Unwanted contact | Real, unintended contact between the arms, or between an arm and an object. One continuous contact counts once. | Intended contact in grasping, pushing, pressing or assembly, and motions the planner rejected. |
| View adjustments | Handovers | Unwanted contact | |||||||
| Configuration | Empty grasps | Drops | Fine placement | All | Free arm | Direct | Via a support | Arms | Arm, object |
| Harness and action | |||||||||
| Default testbed | 47 | 26 | 20 | 59 | 34 | 5 | 10 | 6 | 7 |
| Low effort | 34 | 21 | 26 | 48 | 31 | 3 | 11 | 0 | 2 |
| High effort | 42 | 27 | 22 | 64 | 50 | 4 | 11 | 1 | 4 |
| Relative targets | 54 | 30 | 25 | 54 | 34 | 5 | 10 | 6 | 20 |
| Round | Group and per-task scores | Seen in the runs | Changed for the next round |
| 1 | stack_bowls 1, plug_in_charger 0, general_pickup 1, fill_pen_holder 0, sweep_blocks 0, deposit_coin 0.2, cover_blocks 0, classify_objects 0.4, push_T 1 | In plug_in_charger and fill_pen_holder the model let go before the plug or the pen was in place. | place and grasp can stop before letting go or closing, so the model can look first. A new insert lines a held tip up with a hole and pushes it in, only once the hand is within 2 mm and 1 ∘ of the entry. |
| 2 | hang_mugs 0.15, insert_tubes 0.2, align_blocks 0, put_bottles_into_dustbin 0.25, stack_blocks 1, insert_key 0.15, pour_balls_into_vase 0, build_tower 0.1, pack_objects_into_box 0.1 | Rising straight up after letting go pulled the tube into a lean in insert_tubes and caught the handle in hang_mugs. In pour_balls_into_vase repeated turns moved the lip off the vase and spilled the balls. | place can back away in a chosen direction before rising. A new pivot turns a held object while one point on it, such as a lip or a tip, stays where it is. |
| 3 | sort_nesting_dolls_by_size 0, fill_egg_holder 0.1, fold_clothes 1, play_Xylophone 0, swap_blocks 0 | In play_Xylophone and swap_blocks the model counted presses that never registered. | The guide tells the model to record an attempted press separately from a confirmed one. A new tap makes one press and returns, only once the hand is within 2 mm and 1 ∘ of the target. |
| 4 | make_toast 0, press_by_number 1, make_kong 0, arrange_largest_number 1, stack_blocks_by_language 1, classify_objects_by_language 1, store_tools_in_toolbox 0 | In arrange_largest_number and stack_blocks_by_language long carries failed, and the model succeeded only after breaking them into shorter moves itself. In store_tools_in_toolbox the tool slipped as it was lifted. | transport can split a carry into short segments, and can stop right after the lift so the model can check the grip. |
| 5 | store_laptop_and_headphones 0, match_and_pick_from_conveyor 1, pour_liquid_into_cup 0, play_tic_tac_toe 0.1, pick_from_conveyor_by_image 1, pour_by_language 0 | In play_tic_tac_toe the model waited for the opponent one short step at a time and used up its decisions. | A new hold keeps both arms and jaws still for a set number of steps, so a long wait takes one decision. |
| Cause | Task | What goes wrong |
| Strategy | make_kong | One arm cannot flip the tile and place it at once. The model never hands it over. |
| deposit_coin | Tries a sideways handover and the coin drops. Keeping it in one hand works. | |
| pour_balls_into_vase | Pours forward, which the arm does poorly. The task needs a side pour. | |
| pour_by_language | Pours forward and the grip slips. | |
| pour_liquid_into_cup | Pours forward and blocks its own view. | |
| pick_from_conveyor_by_image | Grasps the basket at a poor point and cannot time the belt. |
Robotic Control
Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.
See all 620 papers