Many essential manipulation tasks - such as food preparation, surgery, and craftsmanship - remain intractable for autonomous robots. These tasks are characterized not only by contact-rich, force-sensitive dynamics, but also by their "implicit" success criteria: unlike pick-and-place, task quality in these domains is continuous and subjective (e.g. how well a potato is peeled), making quantitative evaluation and reward engineering difficult. We present a learning framework for such tasks, using peeling with a knife as a representative example. Our approach follows a two-stage pipeline: first, we learn a robust initial policy via force-aware data collection and imitation learning, enabling generalization across object variations; second, we refine the policy through preference-based finetuning using a learned reward model that combines quantitative task metrics with qualitative human feedback, aligning policy behavior with human notions of task quality. Using only 50-200 peeling trajectories, our system achieves over 90% average success rates on challenging produce including cucumbers, apples, and potatoes, with performance improving by up to 40% through preference-based finetuning. Remarkably, policies trained on a single produce category exhibit strong zero-shot generalization to unseen in-category instances and to out-of-distribution produce from different categories while maintaining over 90% success rates.
Figures & tables
Fig. 2: A overview of our two-stage learning framework. This includes details on data and model architecture for compliant data collection, force-aware imitation learning, and preference-based finetuning from a learned reward model.
Fig. 3: End-effector mount. A CAD visualization of our custom end-effector mount design, including an arm connector, a force-torque sensor plate, two camera mounts, and a knife mount.
Fig. 4: Front-view visualization of qualitative score metric. We use integer scores from 0 to 9 (the higher the better) to capture subjective human preferences based on the overall visual appearance of the peel.
Fig. 5: Wrist-view visualization of quantitative score metric. We use six discrete thickness categories, where nominal denotes the most desired thickness. Details of how these categories are mapped to normalized scalar rewards can be found in Appendix -B .
Fig. 6: Model-based planner. A visualization of the planning procedure of our heuristic planner. We use a calibrated top-view LiDAR camera to capture the scene, extract a segmented point cloud of the object, and plan a dense trajectory on its surface that represents the peeling path. We then calculate the desired knife pose trajectory based on the surface normal of each waypoint on the path, and execute the planned trajectory with human-in-the-loop correction of knife pose via keyboard control (e.g. +20 degrees in yaw).
Data Collection
Success %
Avg. Score
Avg. Time (s)
Heuristic Planner
20
1.8
50
VR
20
2.1
69.5
SpaceMouse (ours)
100
8.5
46
Kinesthetic Teaching
100
7.2
13.8
Replay
0
1.6
N.A.
TABLE I: Quality and efficiency of data collection across different methods. For each experiment, qualitative scores of 10 trajectories collected from cucumbers are averaged. A trajectory with score above 3 is counted as a success. Average time taken is only calculated from successful trajectories. We treat Kinesthetic Teaching as a special category because the collected data has ambiguous compliant parameters. We therefore show results from both the initial data collection and the best trajectory replay where compliance parameters are tuned using heuristics similar to [ 31 ] .
Fig. 7: Qualitative comparison with baselines. We show qualitative comparison between our policy and the baselines. The peel quality is substantially higher even when both are counted as success, as indicated by the qualitative score (top right corner of each image).
Method
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
Base Only
60
3.8
80
5.9
Quan. R.
40
3.8
60
4.0
Qual. R.
0
1.4
60
4.4
IQL-w
0
0.2
0
2.6
TABLE V: Comparison with baselines for finetuning. Task success rates (%) and average scores of four reward alignment methods from experiments on apples (A) and potatoes (P).
Reward
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
PS+B
0
0.0
0
0.0
P60+FG
0
1.0
20
3.8
PT+FG
40
2.3
0
1.2
TABLE VI: Ablation studies on reward design. We study how reward density across time horizon – per-step (PS) vs per-60-steps (P60) vs per-trajectory (PT) – and in value – binary (B) vs fine-grained (FG) – affects effectiveness of finetuning. Empirically, we find the right balance lies at per-step, fine-grained reward values (ours).
Method
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
One-Step
0
1.2
60
4.6
Filter
0
1.0
80
5.2
TABLE VII: How to finetune policies with learned reward. We compare our finetuning method to two other ways to finetune policies given per-step reward: using one-step advantage instead of raw reward (OneStep), and applying reward only on high-scoring steps filtered with a hard threshold (Filter). Neither outperforms our method.
Training
Success % (A)
Score (A)
Success % (P)
Score (P)
Ours
100
7.1
100
7.3
Scratch
0
0.4
0
0.0
No Res.
0
1.2
40
3.0
TABLE VIII: Comparison with different training schemes. Our final policy is first learned from scratch without reward and then finetune with reward on a residual network. We compare with two alternatives: training with reward from scratch (Scratch), and finetuning without residual network (No Res.). We find that our training scheme is the only that ensures stable learning.
α
0.01
Kr
[0.3, 0.3, 0.3, 0.3, 0.18, 0.18, 0.18]
Kl
[106.2, 100.8, 106.2, 106.2, 131.4, 106.2, 106.2]
Klp
[11.89, 25.52, 22.0, 22.0, 22.0, 22.0, 22.0]
Kp
[382.2, 296.4, 347.1, 400.0, 200.0, 200.0, 200.0]
Kd
[21.0, 17.5, 10.0, 10.0, 5.0, 5.0, 5.0]
TABLE IX: Impedance controller parameters.
Quantitative label
Apple Rquant
Potato Rquant
below nominal
0.3
0.5
nominal
1.0
1.0
slightly above nominal
0.8
0.5
above nominal
0.3
0.1
excessive
0.0
0.0
N/A
0.0
0.0
TABLE X: Quantitative reward mapping for apple and potato.
Qualitative score
Descriptor
Apple Rqual
Potato Rqual
0
discard
0.0
0.0
1
too low
0.1
0.1
2
too high
0.2
0.2
3
too short
0.3
0.3
4
short, thick
0.4
0.4
5
short, thin
0.5
0.5
TABLE XI: Qualitative reward mapping for apple and potato.
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
In the development of cooking robots, mastering the task of cutting is crucial. A significant challenge lies in the diverse properties of food, which necessitate distinct cutting policies and even different knives for optimal processing. This paper presents a perception-manipulation framework for food-cutting tasks. Our system features a knife selection module that utilizes force data from a preliminary fixed trial cut to select the appropriate knife for the given food. This is followed by an adaptive cutting phase using reinforcement learning (RL) to balance cutting speed and energy efficiency. In our experiments, the knife selection module achieved 100% successful rate on unseen food, and we compared the performances of fixed policy, RL policy, with human operators. Our method not only achieves high performance but also demonstrates comparable results to those of human participants.
Pretrained robotic policies can suffer substantial performance degradation under out-of-distribution (OOD) conditions encountered during deployment, motivating post-training through real-world interaction. However, reinforcement-learning (RL)-based post-training typically requires substantial environment interactions, a burden that is especially significant in manipulation, where each trial can be slow, costly, or destructive. Therefore, we present PEARS, a physics-prior-guided hybrid RL framework for sample-efficient online adaptation of pretrained policies with tactile feedback. After each episode, its physics-guided force reasoning (PFR) module uses physical priors encoded in a vision-language model (VLM) to diagnose failures from the visual outcome and tactile interaction history and update task-appropriate contact-force bounds. A high-frequency hybrid force-position controller then enforces these bounds during contact. Complementarily, tactile-conditioned diffusion steering reinforcement learning adjusts the latent noise of the frozen flow-matching policy to correct errors in free-space motion and contact timing without updating the base model. In simulation, PEARS improves success rates by 12.4-37.4 percentage points over the strongest per-task baselines. PEARS also reduces the number of interaction episodes required for a certain success threshold by up to 53.2% relative to the fastest baseline. In real-world experiments, PEARS achieves success rates of 95% on Whiteboard Erasing and 90% on Pipette Liquid Aspiration. These results show that combining the PFR module with policy steering can accelerate adaptation while reducing costly interactions. The project website is available at https://song-kun.github.io/pears.
Kun Song, Yiming Wang, Yilin Chen +5
The University of Hong Kong · Shanghai Jiao Tong University · Xense Robotics +1