Organizations: All authors are with the University of Illinois Urbana-Champaign, Urbana, IL, USA. · Zhexi Wang is also with Tsinghua University, Beijing, China.
Coding agents can translate language instructions into robot programs. For contact-rich manipulation, a key challenge is to generate and refine control logic that responds to physical interaction. We present FORCA, a coding agent that takes visual observations and wrist force/torque (F/T) measurements as input to write Python programs for contact probing and force-responsive motion. Between executions, the agent inspects force records alongside visual outcomes to diagnose failures and revise both motion and force-feedback control logic. FORCA requires no demonstrations, task-specific training, or simulation. Across five real-robot tasks, we demonstrate its effectiveness in three forms of contact-rich manipulation: improving manipulation accuracy, adapting to changes in object or surface properties, and regulating contact under surface variation and object motion. Compared with a vision-only variant of the same agent, FORCA improves grasp accuracy and task success and reduces contact loss in wiping and peeling.
Figures & tables
Fig. 1: Overview of Forca . The agent generates a task program that uses force feedback during execution. Recorded force, robot-state, and visual observations inform revisions between rollouts. Generated routines and notes are retained for later reuse.
Fig. 2: Predefined skill modules. The agent is initialized with these modules, assembled from open-source components under human supervision. (a) SAM 3 [ 31 ] localizes the tool named in the instruction. (b) GraspGen [ 32 ] proposes candidate grasps, and the VLM selects one. (c) PyRoKi [ 30 ] plans the trajectory. (d) The robot executes the resulting routine to acquire the tool.
Fig. 3: Illustrative program revision for cucumber peeling. The agent compares force records with visible peeling gaps and adds a condition that pauses forward motion until contact is restored. Photos are from real runs; force traces and skin patches are idealized, and the code is simplified.
Fig. 4: Example rollouts and outcome comparisons. (a) Drawer opening and (b) door-handle operation. Panels 1–3 show Forca estimating geometry visually, refining it through contact probing, and grasping to pull the drawer or turn the handle. In (a1), dashed and solid outlines mark the estimated pre-grasp and grasp poses; the axes indicate closing and approach directions. Numbered markers in (a2) identify three probe locations on the drawer knob. In (b2), orange markers indicate fingertip contacts, and blue shading marks the door-panel position confirmed by contact. Panels (a4) and (b4) compare example outcomes: the vision-only agent misses the drawer grasp and loses its grip on the door handle, while Forca opens both mechanisms. The dashed circle and red arrow in (b4) mark the fingertip-only grasp and the handle’s rebound after slipping.
Task
Method
Trans. err. (mm) ↓
Rot. err. ( ∘ ) ↓
Success (%) ↑
Drawer
Vision-only
8.3±5.4
9.6±3.2
40
Forca
5.3±1.6
1.0±0.3
100
Door
Vision-only
65.3±16.9
55.5±14.9
10
Forca
20.0±0.9
0.6±0.5
100
TABLE I: Grasp accuracy and task success for drawer opening and door-handle operation. Grasp errors: mean ± std. over 10 trials.
Fig. 5: Cube-to-target adaptation. (a) Object positions across successive attempts. The dashed outline marks the initial pose, and shading indicates the target region. (b) Object and surface configurations.
Fig. 6: Reuse across physical conditions. Distance to the target band per attempt (0: inside), starting from the program developed for the cube on the whiteboard.
Fig. 7: Vase wiping: (a) static vase and (b) zero-shot transfer to a hand-held vase. Each panel shows representative outcomes (top), execution frames (middle), and force traces (bottom). Both programs are refined in the static setting and transferred without further revision. As the vase moves, Forca maintains more consistent contact and high marker removal, whereas the vision-only program exhibits contact loss and large force variations, leaving substantial residual marks.
Vase
Method
Removal (%) ↑
Contact loss (%) ↓
Max. force (N) ↓
Static
Vision-only
73
42.6
11.8
Forca
99
0.7
11.3
Hand-held
Vision-only
21
55.9
17.5
Forca
93
1.6
12.3
TABLE II: Vase wiping results averaged over 10 trials.
Method
Contact loss (%) ↓
Excessive force (%) ↓
Peeling ratio (%) ↑
Vision-only
40.6
13.4
37.4
Forca
2.2
2.7
81.1
TABLE III: Peeling a hand-held cucumber (averaged over 10 trials)
Fig. 8: Cucumber peeling rollouts. (a) Hand-held setup and a magnified view of the boxed blade–surface contact. (b) Peeled strips after one representative pass, covering 90.6% of the intended length for Forca and 28.9% for the vision-only agent. (c) Execution frames at the marked instants. (d) Normal contact force. Shaded intervals indicate contact loss; the blue band marks the 4.5–7.5 N target range, and the dashed line marks the 30 N force limit.
Task
Iterations
Agent reasoning (min / iteration)
Robot motion (min / iteration)
Door handle
17
10
3
Drawer
8
5
1.5
Cube-to-target, from scratch
13
4
1
Cube-to-target, new condition
2
4
1
Vase wiping
20
4
1
Cucumber peeling
20
5
1.5
TABLE IV: Development iterations and mean time per iteration. For new cube configurations, the count is the mean number of revisions after the first execution, averaged over three configurations.
It.
Failed at
What happened
Revision
Forca
1–2
Approach
Camera put the door 3–4 cm too close; the probe missed it, then rejected a real touch.
Search farther; trust the touch and update the door depth.
3
Press down
First firm grasp, but at the 10 N force cap the handle turned only 18 ∘ , too little to unlatch.
Press along the handle’s arc about its axis.
4–5
Press down
Its own motion checks read slightly short moves as blocked; the handle never passed 5 ∘ .
Detect blocking by force; use the measured handle pose.
6–8
Press down
Still latched at 72.5 ∘ : grasp 21 mm and axis 15 mm off; handle twisted in the fingers.
Model the handle; press about a measured rotation axis.
9–10
Approach, press down
Setup faults: the cable plug, missing from the collision model, hit the door frame, and overly strict safety checks stopped motions; the press stopped at 49.7 ∘ .
Fix the setup; in the recorded press force, find latch release (61–65 ∘ ) and end stop (jump past 74.5 ∘ ); press to the stop.
TABLE V: Door-handle iterations of Forca and the vision-only agent.
Fig. 9: Door-handle development histories. Each marker denotes an execution at its failure stage or at task completion. Executions 18–20 repeat Forca ’s final program. Detailed failures and revisions are listed in Table V .
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Yuyao Liu, Jiayuan Mao, David Hsu +2
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
Force signals provide critical interaction cues for contact-rich robotic manipulation. However, existing methods mostly use force as an additional observation modality, without fully exploiting its role in modeling future interaction dynamics or guiding execution-time feedback correction. In this paper, we propose FAWAM, a force-aware world action model that incorporates force information at three levels: perception, prediction, and closed-loop execution. FAWAM first encodes historical 6-axis force/torque signals to modulate action generation, then jointly predicts future actions and end-effector wrenches to explicitly model contact evolution. It further introduces a residual correction module that uses the predicted wrench trajectory as an execution-time reference to refine actions online based on real-time force feedback. Real-world experiments across multiple contact-rich tasks show that FAWAM improves the average success rate by 36.25% over vision-only baselines and 21.25% over existing force-aware baselines, demonstrating the effectiveness of our force-aware framework for robust contact-rich manipulation.
Haotian He, Zeyu Yan, Qipeng Liu +2
School of Mathematical Sciences, Peking University · School of Artificial Intelligence, Shanghai Jiao Tong University.
Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and agents built in this way now operate robots without robot-specific training. Whether this paradigm is also safe, however, has not been asked. We evaluate coding agents under a safety constraint, where each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal but collides with the obstacle in most cases, treating task completion as its sole objective. The agent reasons about the obstacle in its traces, and the prompt already forbids touching it, so neither perception nor instruction is at fault; the fault lies in the planning, where the stated constraint never becomes a priority. By decomposing manipulation into a route phase and a contact-rich moment, we locate the source of the failure. Along the route, the model cannot prioritize the safety constraint, having no notion of a clearing route and none of replanning once a chosen route becomes infeasible. At the contact, it is unaware that contact execution is bounded by the same constraint. To close this gap, we present SafeHarness, which equips the model with two obstacle-aware harnesses. Obstacle-aware route planning grounds the objects as bounding boxes and draws candidate routes over them as sequences of waypoints. The agent then plans a route in advance, verifies it, replans when necessary, and only then executes it. Obstacle-aware contact execution instead selects the contact position so that the contact itself avoids the obstacle. SafeHarness attains 81.2% task success and 91.9% collision avoidance with GPT-6-Astra, surpassing the previous SOTA by 13.7 and 23.0 points, and the same agent without harnesses by 31.2 and 57.5 points, respectively.