ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models
Organizations: Show Lab, National University of Singapore
Abstract
Frontier multimodal models can now control robots zero-shot from images, yet many manipulation tasks depend on contact forces that images cannot show. Bringing force to an API-only frozen model raises two challenges. First, contact evolves much faster than an API model can decide, so the system must close the loop on force beneath the model. Second, a frozen model cannot interpret raw force streams, so force must be returned in a form the model can reason about. Therefore, we present ForceHarness, a harness that lets frozen models use force without demonstrations or finetuning. After calibrating hand force from a wrist sensor or joint torques alone, ForceHarness addresses the first challenge with self-terminating force primitives that regulate force on their own and return at a contact event. To address the second challenge, a physics translation layer turns each force trace into physical quantities with uncertainty as the tool result. To evaluate ForceHarness against expert force controllers and learned policies under identical conditions, we also build HiddenPhys, a seven-task MuJoCo benchmark with hidden physical variables. On a real arm without a wrist force sensor, ForceHarness raises the zero-shot success rate of GPT-6 Astra from 3.3% to 90.0%. In simulation, it adds 17.5 percentage points on the public ForceBench suite and 34.3 percentage points on HiddenPhys, surpassing force-aware policies trained on demonstrations on both benchmarks and matching hand-written expert controllers on HiddenPhys. Ablations further show that smaller models make significantly better decisions when given physical quantities rather than raw force values.
Figures & tables
| Tool | What it does | Quantities returned |
| move_until_contact , probe | Approach until contact | Contact height, impact, stiffness |
| push | Hold a normal force | Mean normal force |
| slide | Slide while pressing | Normal force, friction |
| search | Spiral while pressing until the part drops | Advance, lateral force |
| insert | Push in, slowing as force rises | Advance, lateral force |
| wiggle | Press while rocking | Outcome |
| Task | Hidden variable | What force must do |
|---|---|---|
| Weight | Cube mass, 0.15 or 0.60 kg | Weigh the held cube |
| CoM | Bar’s center of mass, centered or 4 cm off | Measure the held moment |
| Wipe | Table height, up to about 6 cm low | Press 5 to 15 N on 12 spots |
| Stack | Lower cube’s top, known to 1 cm | Feel contact, place gently |
| Peg | Hole position, 15 mm error per axis | Search until the peg drops |
| Latch | Unlock angle, 15 to 35 ∘ | Press to release, then pull |
| GPT-6 Astra | GPT-6.1 Sol | ||
|---|---|---|---|
| Task | No force | ForceHarness | ForceHarness |
| Wipe a whiteboard | 1/10 | 8/10 | 6/10 |
| Drive a screw | 0/10 | 9/10 | 7/10 |
| Peel an eggplant | 1/10 | 9/10 | 9/10 |
| Insert a USB plug | 0/10 | 8/10 | 7/10 |
| Sort three bottles by weight | 0/10 | 10/10 | 8/10 |
| Method | Weight | CoM | Faucet | Faucet dir. | Stove | Stove press | Drawer | Nut | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|
| Trained | , no force | 54.0 | 45.5 | 9.5 | 31.5 | 8.5 | 1.0 | 4.5 | 31.5 | 23.25 |
| + Prompt Fusion | 88.0 | 56.5 | 21.0 | 41.5 | 16.0 | 2.5 | 23.0 | 34.0 | 35.31 | |
| + Dual-Path Fusion | 99.5 | 72.0 | 22.0 | 30.0 | 17.5 | 19.0 | 4.5 | 16.5 | 35.13 | |
| [ 3 ] | 61.5 | 39.0 | 3.5 | 27.0 | 7.0 | 5.0 | 7.5 | 36.0 | 23.31 | |
| ForceVLA | 35.5 | 45.5 | 12.5 | 14.0 | 10.0 | 12.0 | 20.0 | 26.0 | 21.94 | |
| ForceFlow [ 42 ] | 51.5 | 3.0 | 0.0 | 0.5 | 0.0 | 0.0 | 0.0 | 0.0 | 6.88 |
| Method | Weight | CoM | Wipe | Stack | Peg | Latch | Lock | Total | Damage | Demos | Code | Shift | |
| Classical | Expert force controller | 20 | 19 | 20 | 20 | 18 | 19 | 20 | 136 | 1 | 0 | ✓ | 77 68 |
| Position only | 10 | 10 | 0 | 17 | 5 | 13 | 16 | 71 | 6 | 0 | ✓ | 25 22 | |
| Learned | DP | 16 | 17 | 4 | 11 | 15 | 9 | 11 | 83 | 20 | 200 | – | 52 40 |
| DP + force | 18 | 17 | 7 | 6 | 15 | 1 | 10 | 74 | 13 | 200 | – | 57 43 | |
| DP, images only | 9 | 17 | 4 | 12 | 14 | 0 | 9 | 65 | 25 | 200 | – | 44 42 | |
| DP, images only + force | 16 | 17 | 5 | 10 | 14 | 3 | 9 | 74 | 16 | 200 | – | 52 43 |
| GPT-6.1 Sol | Smaller models | |||
|---|---|---|---|---|
| Form | Success | Decision | Success | Decision |
| No force | 0.63 | 0.73 | 0.18 | 0.57 |
| Event words | 0.77 | 0.70 | 0.54 | 0.64 |
| Force plot | 0.90 | 1.00 | 0.56 | 0.71 |
| Raw numbers | 0.87 | 1.00 | 0.63 | 0.69 |
| Derived signals | 0.87 | 0.97 | 0.69 | 0.85 |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| Event | Rule | Stops |
|---|---|---|
| contact_start | on ticks | if the tool asks |
| contact_end | on ticks | reported |
| force_cap | always | |
| torque_cap | always | |
| stall | in contact, the target moved 2 mm in six ticks and the hand under 0.35 of that | reported, tools test stalls their own way |
| slip | gripper closed, fell below half its six-tick peak, which exceeded | reported |
| Constant | MuJoCo | ForceBench | Real arm |
|---|---|---|---|
| Tick | 0.05 s | 1/15 s | 0.05 s nominal, about 66 ms measured |
| , | 20, 100 mm/s | 20, 100 mm/s | 4, 8 mm/s |
| : push , pull , press_turn / slide , search | 0.0045 / 0.0015 (m/s)/N | same | same, no press_turn |
| Stiffness command | 150, 40 on a soft axis | joint gains 80 and 12, fixed | 750 N/m and 15 N m/rad, fixed |
| Effective when blocked | about 3 N/mm at 150, 0.4 to 0.6 at 40 | 1.23 N/mm near, about 0.5 far | 0.75 to 1 N/mm |
| Lead limit : free, guarded | 30, 12 mm | 30, 12 mm | 8, 8 mm |
| Tool | What it does | What it returns |
|---|---|---|
| grasp , release | close or open the gripper, clear the tare | opening, holding or not |
| locate | map a pixel to a 3D point, no motion | point, valid pixels |
| locate_object | fit a known part’s axis to a clicked mask | axis, overlap score |
| scene | describe the scene, no motion | poses, or photos |
| done | end the episode | logged claim only |
| Constant | MuJoCo | ForceBench | FR3 |
|---|---|---|---|
| Detector lag | 4 mm | 5.3 mm | 0.2 mm |
| Contact | 3 mm | 4 mm | 0.3 mm |
| Noise per tick | 0.5 N | 0.5 N | 0.0688 N |
| Bias per episode | 0.3 N | 0.3 N | 0.0747 N |
| Item | Value |
|---|---|
| Simulator | robosuite 1.5.1 on MuJoCo 3.2.7 |
| Robot | Panda, BASIC composite controller, fixed impedance, damping ratio 1 |
| Action | operational-space pose delta and gripper at 20 Hz, full scale 5 cm and 0.5 rad per tick, no gripper on Wipe |
| Stiffness kp | 150, 40 along a pressed axis, 10 on wrist yaw in pull , 1500 for the stiff arm of the Lock checks |
| Arrival tolerance | 3 mm, 0.1 mm on Lock |
| Force sensor | MuJoCo wrist sensor, rotated to the world frame |
| One program | Step by step | ||||
|---|---|---|---|---|---|
| Task | None | Force | None | Force | Expert |
| Weight | 0 | 5 | 3 | 5 | 5 |
| CoM | 0 | 4 | 3 | 5 | 4 |
| Wipe | 3 | 5 | 4 | 5 | 5 |
| Stack | 0 | 3 | 5 | 5 | 5 |
| Peg | 0 | 0 | 0 | 2 | 4 |
| Sol | Smaller | Smaller | |
|---|---|---|---|
| Comparison | success | success | decisions |
| L2, raw | 4, 2, 0.69 | 23, 14, 0.19 | 18, 6, 0.023 |
| L2, events | 11, 2, 0.023 | 34, 13, 0.003 | 24, 8, 0.007 |
| L2, plot | 3, 3, 1 | 28, 10, 0.005 | 19, 8, 0.052 |
| L2, signals | 3, 1, 0.63 | 18, 16, 0.86 | 9, 9, 1 |
| L2, L1 | 1, 2, 1 | 20, 12, 0.22 | 8, 7, 1 |
| Form | Sentence |
|---|---|
| No force | You receive NO force or torque information; only motion results. |
| Event words | … a list of detected contact EVENTS (contact start/end, safety cap, stall, slip, resistance drop) and whether the tool is in contact at the end. |
| Force plot | … a PLOT (image) of the wrist force and torque versus time. |
| Raw numbers | … the raw wrist force/torque numbers (current value and recent samples). |
| Derived signals | … detected contact EVENTS plus derived signals (peak force, contact direction, payload estimate when lifting, mean normal force when pushing/sliding, torque peak when rotating). |
| Quantities, L1 | … detected contact EVENTS plus physical MEASUREMENTS computed from the wrist force/torque (payload mass, centre-of-mass offset of the held load, contact height/point, …), each with units. |
| Tool | Parameters | Description |
| Both interfaces, pose and force | ||
| scene | none | Get the current scene description … and the robot state. |
| move_to | x, y, z, speed? | Move the tool centre point to an absolute world position … G |
| move_by | dx, dy, dz, speed? | Move the tool centre point by a relative offset … G |
| rotate_z | angle_deg, torque_cap?, stop_on_resistance_drop? | Rotate the tool about the vertical axis by angle_deg … H |
| grasp | none | Close the gripper. … |
| Diffusion Policy | ForceVLA | ||
|---|---|---|---|
| Model | ResNet18 per view from scratch, GroupNorm, spatial softmax, 32 keypoints, 1D U-Net 256, 512, 1024 with FiLM, 100 DDPM and 16 DDIM steps | pi05_base , all parameters trained | from pi0_base , force token, LIMoE of 4 experts, ForceBench code |
| State | proprioception 11, scene 5 per object | proprioception 11, scene 15, as prompt tokens after quantile normalization | continuous, with the noisy scene |
| Force | 6 dims in the world frame, zeroed at reset, noisy | 6 dims as prompt tokens | force token |
| Action | 6 pose dims on Wipe, 2 frames observed, 16 predicted, 8 executed | 10 predicted, 5 executed, min-max for stiffness, pose deltas widened to | 13 dims, not the 7-dim absolute pose of ForceBench |
| Data | one model per task, first 200 saved episodes by seed, 185 on Peg, 197 on Latch | one model for 7 tasks, all 1423 episodes, task selected by prompt | as |
| Training | 40 000 steps, batch 64, AdamW at 1e-4, cosine, 500 warmup steps, EMA | 30 000 steps, batch 32, seed 42, AdamW, peak 5e-5, 10 000 warmup steps, EMA 0.999 | 30 000 steps, global batch 64, optimizer and seed of ForceBench |
| Task | Attempts | Saved | Frames | Longest | Horizon |
|---|---|---|---|---|---|
| Weight | 210 | 210 | 144 312 | 723 | 1084 |
| CoM | 210 | 208 | 140 607 | 709 | 1063 |
| Wipe | 210 | 207 | 68 960 | 453 | 679 |
| Stack | 210 | 206 | 121 311 | 793 | 1189 |
| Peg | 240 | 185 | 122 194 | 2310 | 3465 |
| Latch | 210 | 197 | 111 329 | 673 | 1009 |
| Task | Hidden variable | Success in the evaluator we ran | Also required in the paper |
|---|---|---|---|
| Weight Sort | mass class | correct container, release, settling | – |
| CoM Sort | CoM centered or 2 cm off | correct destination, release, settling | – |
| Faucet Turn | from 20 ∘ to 30 ∘ | angle past , water visible | final angle in , release, clearance, rest |
| Faucet Direction | and | final , water visible, release, clearance, rest | – |
| Stove Turn | from 60 ∘ to 90 ∘ | crossing at , final angle within 3 ∘ , flame visible | release, clearance |
| Stove Press | and | while pressed | – |
| Method | Weight | CoM | Faucet | Faucet dir. | Stove | Stove press | Drawer | Nut | Mean |
|---|---|---|---|---|---|---|---|---|---|
| , no force | 54.0 | 45.5 | 9.5 | 31.5 | 8.5 | 1.0 | 4.5 | 31.5 | 23.25 |
| + Token Fusion | 50.5 | 50.0 | 12.0 | 28.5 | 6.0 | 2.5 | 13.0 | 26.0 | 23.56 |
| + Prompt Fusion | 88.0 | 56.5 | 21.0 | 41.5 | 16.0 | 2.5 | 23.0 | 34.0 | 35.31 |
| + Cross-Attention | 54.5 | 51.0 | 6.5 | 33.0 | 5.5 | 2.5 | 6.5 | 17.5 | 22.13 |
| + Token Fusion + Pred | 78.0 | 48.5 | 13.5 | 32.5 | 6.0 | 1.0 | 4.0 | 26.5 | 26.25 |
| + Temporal Prompt | 90.0 | 69.5 | 17.5 | 31.0 | 7.0 | 21.5 | 2.5 | 22.0 | 32.63 |
| Task | Scene | What the hand must do | Actions |
|---|---|---|---|
| Wipe a whiteboard | marker writing, an eraser | find the board by touch and wipe until the images show no writing | 40 |
| Drive a screw | a screw and a gear with a threaded center hole, on a wooden board | find the hole by touch, press the tip in, and turn with the thread, grasping again at the wrist limit | 60 |
| Peel an eggplant | an eggplant on foam, a clamped peeler | keep the blade in the curved skin while moving along | 60 |
| Insert a USB plug | a USB-A plug on a cable, a power strip | reach a port, find it by a light sliding search, and seat the plug | 40 |
| Sort three bottles by weight | look-alike bottles of 294, 204, and 160 g, a white and a green plate | lift each by its cap, take the heaviest to the white plate and the lightest to the green one | 80 |
| Tell soft from hard | two wrapped objects of one shape | say which is soft without grasping | 15 |
| GPT-6 Astra | GPT-6.1 Sol | |||||
| Task | Force | None | Pairs | Force | None | Pairs |
| Weight Sort | 10 | 8 | 2 / 0 | 10 | 9 | 1 / 0 |
| CoM Sort | 7 | 5 | 3 / 1 | 6 | 5 | 4 / 3 |
| Faucet Turn | 10 | 10 | 0 / 0 | 10 | 9 | 1 / 0 |
| Faucet Direction | 5 | 5 | 4 / 4 | 4 | 3 | 4 / 3 |
| Stove Turn | 10 | 10 | 0 / 0 | 10 | 10 | 0 / 0 |
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,831 papers