Reins: Frontier Multimodal Models Driving Real Robot Arms on Long-Horizon Tasks, Zero-Shot
Organizations: MIT CSAIL · Harvard University
Abstract
Frontier models such as GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5 can now plan and call tools, yet they still struggle to drive real robots through long-horizon, contact-rich manipulation, blind while the arm moves and while they think. We present Reins, a harness in which an off-the-shelf model operates a dual-arm robot with 3D perception tools and atomic actions that report contact and grip. The model chains actions in one call, each with a condition that must hold, stopping at the first that fails; a second instance, the narrator, watches the cameras continuously and reports what changed with every result. From images, Reins also builds a simulation-ready twin of the scene for continual policy learning. With natural-language prompts and no task-specific code, GPT-6.1 Sol succeeds in 88% of real episodes on five tasks; without the harness, it fails every shell game, taking 8 times as long. Three frontier models perform alike but differ in tracking a scene that changes while the actor thinks. Code will be released upon publication.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| input tokens from cache | compactions per episode | largest prompt | ||
|---|---|---|---|---|
| model | actor | narrator | median (max) | median (max) |
| GPT-6.1 Sol | 90% | 73% | 1 (5) | 61k (171k) |
| GPT-6 Astra | 91% | 68% | 1 (2) | 45k (65k) |
| Claude Opus 5.5 | 88% | 56% | 2 (3) | 71k (114k) |
| group | tool | what it does |
| perception | look | returns fresh images from the head and wrist cameras, with known object ids drawn on them |
| find | detects every instance of a named object in one camera (SAM 3 on RGB-D) and registers each as an id with a 3D position in the base frame | |
| measure | turns one pixel, or an ordered list of pixels, into base-frame 3D points | |
| motion | move_above | moves the gripper to an object or point plus offsets, with given tool angles |
| move_relative | moves the gripper by , at most 0.15 m per axis | |
| descend | moves straight down to an object’s grasp height, to a given height or by a given distance |
| check | measure | pass |
|---|---|---|
| closed mesh | non-manifold edges; signed volume | ; |
| collision fidelity | primitives’ surface outside the mesh (p95) | mm |
| stays put | drift during 1–1.5 s of simulation | mm, |
| matches the frame | outline distance to the SAM mask (p90) | px |
| no initial contact | deepest penetration while settling | mm |
| cloth, at rest | drift over the last 0.25 s; edge strain (p95); outline distance | mm; ; px |
| task | certified | IoU | outline (px) | light | scripted task in MuJoCo |
|---|---|---|---|---|---|
| shell game | 5/5 | 0.92–0.96 | 0.95–1.64 | 11.4 | cover a cap, push it 70 mm |
| sweep | 3/5 | 0.95–0.97 | 0.68–1.37 | 17.6 | sweep 2 caps and a sponge into the pan |
| drawer | 1/2 | 0.90 | 1.37 | 19.0 | open 120 mm, place the brick, close |
| towel | 1/1 | 0.99 | 1.16 | 16.2 | fold in half (296 151 mm) |
| shirt | 1/1 | 0.99 | 2.39 | 17.8 | fold over (678 441 mm) |
Long-Horizon Robotic Manipulation
Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.
See all 462 papers