Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics
Authors: Yibo Qiu, Shu'ang Sun, Haoliang Ye, Ronald X Xu, Mingzhai Sun
Organizations: Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China
Long-horizon precision manipulation in laboratory automation, such as pipette tip attachment and liquid transfer, requires policies that respect strict procedural logic while operating in continuous, high-dimensional state spaces. However, existing approaches struggle with reward sparsity, multi-stage structural constraints, and noisy or imperfect demonstrations, leading to inefficient exploration and unstable convergence. We propose a Keyframe-Guided Reward Generation Framework that automatically extracts kinematics-aware keyframes from demonstrations, generates stage-wise targets via a diffusion-based predictor in latent space, and constructs a geometric progress-based reward to guide online reinforcement learning. The framework integrates multi-view visual encoding, latent similarity-based progress tracking, and human-in-the-loop reinforcement fine-tuning on a Vision-Language-Action backbone to align policy optimization with the intrinsic stepwise logic of biological protocols. Across four real-world laboratory tasks, including high-precision pipette attachment and dynamic liquid transfer, our method achieves an average success rate of 82% after 40--60 minutes of online fine-tuning. Compared with HG-DAgger (42%) and Hil-ConRFT (47%), our approach demonstrates the effectiveness of structured keyframe-guided rewards in overcoming exploration bottlenecks and providing a scalable solution for high-precision, long-horizon robotic laboratory automation.
Figures & tables
Fig. 1: Illustration of our Keyframe-guided Online RL framework. Given an initial observation, the framework generates a sequence of multi-view keyframes (e.g., Position → Aspirate → Lift) to represent the target task. These keyframes act as intermediate goals to provide stage-wise rewards via state-similarity calculation. Compared to state-of-the-art baselines, our approach significantly accelerates convergence and achieves a superior average success rate of 82%, outperforming the baseline by 47%.
Fig. 2: Overview of our Keyframe-guided RL framework. First, the system extracts keyframes by filtering feature dynamics from demonstrations. A diffusion-based generator is then trained to predict these keyframe sequences from initial observations. During online RL, the reward module calculates latent similarity to provide stage-wise guidance. These rewards drive the Hil-ConRFT update for policy learning.
Fig. 3: Biological Laboratory task illustrations and common failure modes. (A) Petri Dish De-lidding. This task involves using a gripper to lift and relocate the lid of a petri dish to the side. Common failures include premature release, or colliding with the dish base due to insufficient height. (B) Centrifuge Tube Loading. This task involves picking a tube from a rack and placing it into a centrifuge slot. Common failures include missing the tube, dropping it early, or colliding with the centrifuge rim. (C) Precision Liquid Transfer. This task involves absorbing liquid from a tube and spraying it into a petri dish. Common failures include misalignment with the tube opening, colliding with the tube wall, or spraying liquid before reaching the target. (D) Pipette Tip Attachment. This task involves inserting the pipette into a tip box to secure a new tip. Common failures include misalignment, insertion errors, or the tip slipping off during raising.
Fig. 4: Impact of keyframe introduction. Precision liquid transfer requires a strict sequence from insertion to dispensing. Unlike our kinematics-heuristic extraction, the baseline uses uniform sampling and misses critical bottleneck states like insertion and aspiration. This leads to erroneous spatial guidance and execution failure. In contrast, our method anchors these intermediate states, ensuring correct step-wise logic.
Task
Ours
HG- DAgger
HIL- ConRFT
HIL- SERL
Petri Dish De-lidding
1.00
0.70
0.50
0.00
Centrifuge Tube Loading
0.50
0.50
0.40
0.00
Precision Liquid Transfer
1.00
0.00
0.10
0.00
Pipette Tip Attachment
0.80
0.50
0.90
0.00
Mean
0.82
0.42
0.47
0.00
TABLE I: Performance Comparison of Different Methods on Robotic Manipulation Tasks
Fig. 5: Learning curves during online training. This figure presents the success rates, intervention rates, and episode lengths for HIL-ConRFT, HG-DAgger, HIL-SERL, and our method across four representative laboratory tasks. The metrics are displayed as a running average over 10 episodes against training time in minutes.
Task
Step-wise Coverage (All) ↑
Core Milestone Recall (Core) ↑
w/o KE
w/ KE
w/o KE
w/ KE
Petri Dish De-lidding
0.817
0.917
0.550
0.900
Centrifuge Tube Loading
0.667
0.933
0.350
0.850
Precision Liquid Transfer
0.550
0.900
0.400
1.000
Pipette Tip Attachment
0.933
0.983
0.800
1.000
Mean
0.742
0.933
0.525
0.938
TABLE II: Quantitative Analysis of Multi-Level Keyframe Identification Accuracy
Task
wwrist
wfront
θ
Keyframe Sequence
Petri Dish De-lidding
0.5
0.5
0.7
Reach → Grasp → Lift → Relocate
Centrifuge Tube Loading
0.6
0.4
0.6
Approach → Pick → Align → Insert
Precision Liquid Transfer
0.7
0.3
0.6
Insert → Aspirate → Lift → Spray
Pipette Tip Attachment
0.7
0.3
0.8
Approach → Insert → Lift
TABLE III: Task-Specific View Weights and Keyframe Semantic Sequences
Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.
Yiming Mao, Zixi Yu, Weixin Mao +5
LimX Dynamics · Beijing University of Posts and Telecommunications · Zhejiang University
Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying reward functions that are both semantically meaningful and reusable across tasks. In this paper, we propose Large Reward Models (LRMs), a framework that adapts foundation VLMs into frame-level reward generators for robot policy refinement. We specialize a state-of-the-art VLM on a multi-source dataset spanning real-world robot trajectories, human-object interactions, and simulated manipulation environments. Unlike prior approaches that mainly evaluate trajectories post-hoc, LRMs expose multiple reward interfaces from visual observations: progress estimation, task completion, and temporal contrastive comparison. Starting from an imitation-learned policy, we use these VLM-derived rewards to guide PPO refinement on held-out long-horizon manipulation tasks. Our experiments show that LRM progress rewards provide the strongest non-privileged online refinement signal, improving the IL baseline and narrowing the gap to privileged environment rewards. We further deploy progress rewards for progress-weighted behavioral cloning on four real-world manipulation tasks spanning two robot platforms, improving over SFT on all four tasks. These results suggest that modality-specific specialization of foundation VLMs can provide practical visual reward signals for both simulated policy refinement and physical robot self-improvement without hand-coded task rewards.
Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.