Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce Latent RL Prior, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of π0.5 and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using 10× and 5× fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge. Project website: https://yixiaowang7.github.io/latent-rl-prior-page/.
Figures & tables
Figure 1: Latent RL Prior. (a) Privileged pretraining: a state tokenizer and Q-Former supply task context to an action decoder conditioned on flow time and single-arm proprioception. (b) Visual adaptation: an image encoder and Q-Former learn context tokens while the decoder remains frozen. For bimanual assembly, left- and right-arm queries interact within the same Q-Former; each output group conditions the shared decoder with its corresponding arm’s proprioception.
Figure 2: Pretraining and downstream tasks. Left: the pretraining task of target pose alignment with varied procedural objects; the inset shows example object templates. Right: the downstream tasks of target pose alignment with a held-out brush, door lifting, and bimanual assembly (left to right). The same pretrained action decoder is reused for visual adaptation across all three downstream tasks.
Figure 3
Figure 4: Comparison with full-parameter fine-tuning of GR00T N1.7 and π0.5 across demonstration budgets. (a) Target pose alignment at strict ( ϵ=0.015 m) and relaxed ( ϵ=0.05 m) thresholds; (b) door lifting; (c) bimanual assembly. We report average success rate and standard deviation across three random seeds.
Figure 5: Target pose-alignment ablations with the action decoder frozen for both latent-transfer variants. Results show mean success rates and standard deviations over three random seeds, except for explicit transfer, which uses five seeds.
Figure 6: Training stability on target pose alignment at a relaxed ( ϵ=0.05 m) threshold. Dotted curves show individual runs for latent transfer, explicit transfer, and scratch training; shaded regions show the mean ± one standard deviation. Explicit transfer uses five random seeds, while latent transfer and scratch training use three seeds each.
Figure 7: Learning-rate ablations for (a) the image encoder with the action decoder frozen and (b) the action decoder with the image encoder frozen. Panels report strict ( ϵ=0.015 m) and relaxed ( ϵ=0.05 m) pose-alignment success. We report average success rate and standard deviation across three random seeds. Each scale multiplies the tested module’s original learning rate; zero freezes it.
Figure 8: Generalization to increased object mass. (a) Target pose alignment at a relaxed ( ϵ=0.05 m) thresholds. (b) Door lift. The leftmost point in each plot corresponds to the reference mass used for downstream visual training, e.g., for Dvis .
Figure 9: Linear contact-force probes of the visual policies. Panels predict (a) total grasp normal force, (b) per-finger normal forces, (c) per-finger tangential forces, and (d) per-finger Cartesian force vectors from each layer of the action decoder. Colors denote visual-demonstration budgets; solid and dashed curves denote the frozen pretrained and trainable scratch action decoders. Both policies fine-tune the image encoder at 0.2× its base learning rate for 100k steps. Curves average three policy seeds on the same IID probe test episodes.