Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge
Abstract
The CoRL 2026 RoboEval challenge has eight bimanual tasks, and it ranks a policy not only on whether it succeeds but on how: every stage of the task, done safely, efficiently and with both arms in coordination, scored against human demonstrations. My solution has two parts. The generic part, carried over from my LeHome challenge solution, is off-policy RL on a -based policy that is also its own value function. It is trained with AWR and RECAP on advantages of the exact official score, in a loop where every rollout is at the same time a Thompson-sampling trial of about 20 inference parameters. Out of the box, it took the policy to 0.85 RDS on held-out seeds in two days (and it was still very far from convergence). The competition-specific part adds data that local exploration cannot find: generated demonstrations that reach the full reward, and recovery episodes that start from the policy's own failures. The submitted policy reaches 0.990 RDS on held-out seeds and 0.987 on the official seeds.
Figures & tables
| Task | Success | Stages | Budget | Demos | Full | Demo RDS |
|---|---|---|---|---|---|---|
| LiftPot | pot lifted 10 cm, level | left grasp right grasp lifted lifted level, both holding | 10 | 72/77 | 68 | 0.914 |
| LiftTray | heavy tray held by both hands, off the table | left grasp right grasp held clear of the table lifted 2.3 cm | 10 | 100/154 | 100 | 0.698 |
| StackTwoBlocks | one cube stacked on the other | a cube grasped the other one grasped too stacked, off the table | 20 | 135/141 | 1 | 0.830 |
| StackSingleBookShelf | book on the shelf, released | grasp (after a push) lift touch the shelf place and release | 40 | 66/73 | 66 | 0.909 |
| CubeHandover | a rod passed from one hand to the other in the air | grasp transfer transfer with the rod off the table | 20 | 98/101 | 98 | 0.938 |
| PackBox | both box flaps closed | left flap held right flap held right closed left closed box closed | 20 | 98/99 | 2 | 0.850 |
| Question | How it was tested | Answer |
|---|---|---|
| Which inputs are needed? | input dropout + hide switches | state yes; velocities and front cameras no |
| How much sampling noise? | temperature 0.1–1.2 | 0.1–0.55 |
| How much guidance? | CFG scale 1–10 | 2.5–9.5 |
| Best of how many chunks? | up to 6, ranked by the Q head | 2–6 |
| Execute faster than trained? | speed 0.75–5 | first 1.5–3.9 , now 0.8–1.1 |
| Task | rank | flow steps | temp. | CFG | replan | keep | speed | gripper | hidden | |
|---|---|---|---|---|---|---|---|---|---|---|
| LiftPot | 5 | succ. | 19 | 0.10 | 9.4 | 8 | 5 | 1.00 | 0.90 | front, external |
| LiftTray | 3 | succ. | 11 | 0.10 | 4.5 | 24 | 6 | 1.05 | 0.89 | front, external |
| StackTwoBlocks | 6 | score | 15 | 0.21 | 4.7 | 9 | 0 | 1.05 | 0.60 | external, velocity |
| StackSingleBookShelf | 6 | succ. | 5 | 0.15 | 4.7 | 7 | 6 | 0.86 | 0.57 | front, velocity |
| CubeHandover | 5 | succ. | 17 | 0.35 | 4.7 | 7 | 0 | 0.91 | 0.69 | – |
| PackBox | 5 | score | 7 | 0.36 | 6.3 | 25 | 5 | 0.95 | 0.66 | external |
| Task | Success | Full | Outcome | Safety | Efficiency | Coord. | Score |
|---|---|---|---|---|---|---|---|
| LiftPot | 1.000 | 0.986 | 0.999 | 1.000 | 1.000 | 1.000 | 0.999 |
| LiftTray | 0.998 | 0.998 | 0.998 | 1.000 | 1.000 | 1.000 | 0.998 |
| StackTwoBlocks | 0.996 | 0.994 | 0.997 | 0.993 | 0.920 | 1.000 | 0.979 |
| StackSingleBookShelf | 0.998 | 0.998 | 0.998 | 0.999 | 1.000 | 1.000 | 0.998 |
| CubeHandover | 1.000 | 1.000 | 1.000 | 0.993 | 1.000 | 1.000 | 0.998 |
| PackBox | 1.000 | 0.990 | 0.999 | 0.918 | 1.000 | 1.000 | 0.974 |
| Task | Success | Full | Outcome | Safety | Efficiency | Coord. | Score |
|---|---|---|---|---|---|---|---|
| LiftPot | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| LiftTray | 0.980 | 0.980 | 0.984 | 1.000 | 1.000 | 1.000 | 0.984 |
| StackTwoBlocks | 1.000 | 1.000 | 1.000 | 0.978 | 0.919 | 1.000 | 0.977 |
| StackSingleBookShelf | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| CubeHandover | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| PackBox | 1.000 | 1.000 | 1.000 | 0.898 | 1.000 | 1.000 | 0.969 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Robot Policy Learning
Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.
See all 403 papers