Many essential manipulation tasks - such as food preparation, surgery, and craftsmanship - remain intractable for autonomous robots. These tasks are characterized not only by contact-rich, force-sensitive dynamics, but also by their "implicit" success criteria: unlike pick-and-place, task quality in these domains is continuous and subjective (e.g. how well a potato is peeled), making quantitative evaluation and reward engineering difficult. We present a learning framework for such tasks, using peeling with a knife as a representative example. Our approach follows a two-stage pipeline: first, we learn a robust initial policy via force-aware data collection and imitation learning, enabling generalization across object variations; second, we refine the policy through preference-based finetuning using a learned reward model that combines quantitative task metrics with qualitative human feedback, aligning policy behavior with human notions of task quality. Using only 50-200 peeling trajectories, our system achieves over 90% average success rates on challenging produce including cucumbers, apples, and potatoes, with performance improving by up to 40% through preference-based finetuning. Remarkably, policies trained on a single produce category exhibit strong zero-shot generalization to unseen in-category instances and to out-of-distribution produce from different categories while maintaining over 90% success rates.
Figures & tables
Fig. 2: A overview of our two-stage learning framework. This includes details on data and model architecture for compliant data collection, force-aware imitation learning, and preference-based finetuning from a learned reward model.
Fig. 3: End-effector mount. A CAD visualization of our custom end-effector mount design, including an arm connector, a force-torque sensor plate, two camera mounts, and a knife mount.
Fig. 4: Front-view visualization of qualitative score metric. We use integer scores from 0 to 9 (the higher the better) to capture subjective human preferences based on the overall visual appearance of the peel.
Fig. 5: Wrist-view visualization of quantitative score metric. We use six discrete thickness categories, where nominal denotes the most desired thickness. Details of how these categories are mapped to normalized scalar rewards can be found in Appendix -B .
Fig. 6: Model-based planner. A visualization of the planning procedure of our heuristic planner. We use a calibrated top-view LiDAR camera to capture the scene, extract a segmented point cloud of the object, and plan a dense trajectory on its surface that represents the peeling path. We then calculate the desired knife pose trajectory based on the surface normal of each waypoint on the path, and execute the planned trajectory with human-in-the-loop correction of knife pose via keyboard control (e.g. +20 degrees in yaw).
Data Collection
Success %
Avg. Score
Avg. Time (s)
Heuristic Planner
20
1.8
50
VR
20
2.1
69.5
SpaceMouse (ours)
100
8.5
46
Kinesthetic Teaching
100
7.2
13.8
Replay
0
1.6
N.A.
TABLE I: Quality and efficiency of data collection across different methods. For each experiment, qualitative scores of 10 trajectories collected from cucumbers are averaged. A trajectory with score above 3 is counted as a success. Average time taken is only calculated from successful trajectories. We treat Kinesthetic Teaching as a special category because the collected data has ambiguous compliant parameters. We therefore show results from both the initial data collection and the best trajectory replay where compliance parameters are tuned using heuristics similar to [ 31 ] .
Fig. 7: Qualitative comparison with baselines. We show qualitative comparison between our policy and the baselines. The peel quality is substantially higher even when both are counted as success, as indicated by the qualitative score (top right corner of each image).
Method
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
Base Only
60
3.8
80
5.9
Quan. R.
40
3.8
60
4.0
Qual. R.
0
1.4
60
4.4
IQL-w
0
0.2
0
2.6
TABLE V: Comparison with baselines for finetuning. Task success rates (%) and average scores of four reward alignment methods from experiments on apples (A) and potatoes (P).
Reward
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
PS+B
0
0.0
0
0.0
P60+FG
0
1.0
20
3.8
PT+FG
40
2.3
0
1.2
TABLE VI: Ablation studies on reward design. We study how reward density across time horizon – per-step (PS) vs per-60-steps (P60) vs per-trajectory (PT) – and in value – binary (B) vs fine-grained (FG) – affects effectiveness of finetuning. Empirically, we find the right balance lies at per-step, fine-grained reward values (ours).
Method
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
One-Step
0
1.2
60
4.6
Filter
0
1.0
80
5.2
TABLE VII: How to finetune policies with learned reward. We compare our finetuning method to two other ways to finetune policies given per-step reward: using one-step advantage instead of raw reward (OneStep), and applying reward only on high-scoring steps filtered with a hard threshold (Filter). Neither outperforms our method.
Training
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
Scratch
0
0.4
0
0.0
No Res.
0
1.2
40
3.0
TABLE VIII: Comparison with different training schemes. Our final policy is first learned from scratch without reward and then finetune with reward on a residual network. We compare with two alternatives: training with reward from scratch (Scratch), and finetuning without residual network (No Res.). We find that our training scheme is the only that ensures stable learning.
α
0.01
Kr
[0.3, 0.3, 0.3, 0.3, 0.18, 0.18, 0.18]
Kl
[106.2, 100.8, 106.2, 106.2, 131.4, 106.2, 106.2]
Klp
[11.89, 25.52, 22.0, 22.0, 22.0, 22.0, 22.0]
Kp
[382.2, 296.4, 347.1, 400.0, 200.0, 200.0, 200.0]
Kd
[21.0, 17.5, 10.0, 10.0, 5.0, 5.0, 5.0]
TABLE IX: Impedance controller parameters.
Quantitative label
Apple Rquant
Potato Rquant
below nominal
0.3
0.5
nominal
1.0
1.0
slightly above nominal
0.8
0.5
above nominal
0.3
0.1
excessive
0.0
0.0
N/A
0.0
0.0
TABLE X: Quantitative reward mapping for apple and potato.
Qualitative score
Descriptor
Apple Rqual
Potato Rqual
0
discard
0.0
0.0
1
too low
0.1
0.1
2
too high
0.2
0.2
3
too short
0.3
0.3
4
short, thick
0.4
0.4
5
short, thin
0.5
0.5
TABLE XI: Qualitative reward mapping for apple and potato.