Learning from Physical Experience:Context-Space Adaptation for Robotic Manipulation
Organizations: The Chinese University of Hong Kong · University of Electronic Science and Technology of China
Abstract
Recent multimodal foundation models exhibit strong perception, reasoning, and in-context learning capabilities, but translating these capabilities into physical action remains challenging. Through real-world experiments across diverse manipulation tasks, we find that ChatGPT-6-Astra, a state-of-the-art general-purpose reasoner, can serve as a physically actionable, context-responsive manipulation policy, demonstrating effective fine manipulation and spatial reasoning. Building on these findings, we introduce LEXP, a harness for learning from physical experience through context-space adaptation. LEXP combines task-level planning with an experience module that interprets execution evidence, extracts and retains lessons with applicability conditions, and reuses relevant lessons to generate contextual feedback for subsequent actions. Model weights and the action interface remain fixed throughout. Across 12 real-world manipulation tasks, LEXP achieves successful trials (63.9%), compared with (52.8%) with memory alone and (5.6%) for vanilla Astra. On three tasks, bread transfer, sort trash and toiletries, and retrieve drawer, removing planning reduces success from to , while removing retained experience reduces it to . Execution traces illustrate feedback-guided corrections and cross-episode reuse of an inspection strategy. Execution analyses illustrate how task structure and experience-derived feedback guide task progression, action correction, and experience reuse. These results support context-space adaptation as a practical mechanism for turning physical interaction into reusable guidance for robotic manipulation.
Figures & tables
| Task | Instruction | Initial configuration | Success criterion |
|---|---|---|---|
| Pick up a ping-pong ball | Pick up the ball. | A ping-pong ball rests inside a shallow transparent dish on the tabletop; both arms start parked clear of the dish. | The ball is enclosed between the fingertips and held clear of the dish, which remains on the tabletop. |
| Pick up a cable tie | Pick up the cable tie. | A white cable tie lies on an orange tray on the tabletop. | The cable tie is retained between the fingertips, with its free length extending from the raised gripper clear of the tray. |
| Pick up a key | Pick up the key. | A key on a split ring with an attached tag lies on the table. | The key, ring, and tag are suspended from the raised gripper clear of the surface; the grasp may be on the key, ring, or tag. |
| Pick up an L-shaped hex key | Pick up the hex key. | An L-shaped hex key lies on the table with its long leg and bent end accessible. | The hex-key shaft is pinched between the opposing gripper pads, with its bent end projecting clear of the work surface. |
| Stand up a bottle | Stand the bottle upright. | The bottle lies on its side on the tabletop. | The bottle stands cap-up on its base on the tabletop, with visible clearance from both grippers and no robot support. |
| Grasp out of reach plush | Pick up the plush toy. | A plush toy rests on a mat positioned beyond the arm’s direct reach. | Plush fabric is retained between the fingertips, and the toy hangs from the raised gripper clear of the mat, which remains flat below. |
| GPT-6 Astra | Opus 5.5 | Sol 5.6 | |||
| Task | Vanilla | Memory | LEXP | Memory | Memory |
| Pick up a ping-pong ball | 2/3 | 2/3 | 3/3 | 1/3 | 0/3 |
| Pick up a cable tie | 0/3 | 3/3 | 2/3 | 1/3 | 0/3 |
| Pick up a key | 0/3 | 3/3 | 3/3 | 1/3 | 0/3 |
| Pick up an L-shaped hex key | 0/3 | 2/3 | 2/3 | 2/3 | 0/3 |
| Stand up a bottle | 0/3 | 2/3 | 2/3 | 0/3 | 0/3 |
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,805 papers