We present ProgAgent, a continual reinforcement learning (CRL) agent that unifies progress-aware reward learning with a high-throughput, JAX-native system architecture. Lifelong robotic learning grapples with catastrophic forgetting and the high cost of reward specification. ProgAgent tackles these by deriving dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations. We theoretically interpret this as a learned state-potential function, delivering robust guidance in line with expert behaviors. To maintain stability amid online exploration - where novel, out-of-distribution states arise - we incorporate an adversarial push-back refinement that regularizes the reward model, curbing overconfident predictions on non-expert trajectories and countering distribution shift. By embedding this reward mechanism into a JIT-compiled loop, ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible: it merges PPO with coreset replay and synaptic intelligence for an enhanced stability-plasticity balance. Evaluations on ContinualBench and Meta-World benchmarks highlight ProgAgent's advantages: it markedly reduces forgetting, boosts learning speed, and outperforms key baselines in visual reward learning (e.g., Rank2Reward, TCN) and continual learning (e.g., Coreset, SI) - surpassing even an idealized perfect memory agent. Real-robot trials further validate its ability to acquire complex manipulation skills from noisy, few-shot human demonstrations.
Figures & tables
Figure 1. Overview of ProgAgent’s training pipeline, illustrating the unified loop of reward learning, policy optimization, and continual mechanisms under JAX acceleration.
Figure 2. The complete training pipeline of ProgAgent. Our approach utilizes two parallel data streams. Top path An offline stream uses unlabeled expert videos to train a Progress Prediction Network. The network’s output is optimized via a contrastive expert loss ( Lexpert ) to accurately estimate task progress. Bottom path An online stream collects agent experience through large-scale parallel rollouts. To ensure robustness against out-of-distribution states, an adversarial push-back loss ( Lpush ) regularizes the reward function by pushing predictions on these non-expert trajectories toward a low-confidence (zero-mean, high-variance) prior. These two objectives are combined into a final loss, Ltotal , to produce a dense and stable reward signal for policy training.
Figure 3. Qualitative comparison of the learned policies for the Button-Press task in the ContinualBench environment. The sequences display the agent’s behavior at the end of training. From top to bottom: a standard Finetuning agent, the Online Agent , and our ProgAgent . ProgAgent successfully learns a direct and stable policy to complete the task, whereas the baseline methods exhibit less optimal or failed behaviors.
Method
button-press
door-open
window-close
Success Rate ↑
AP ↑
Regret ↓
Success Rate ↑
AP ↑
Regret ↓
Success Rate ↑
AP ↑
Regret ↓
Fine-tuning
15.2 ± 2.1
24.9 ± 3.5
37.7 ± 2.8
12.5 ± 1.9
24.9 ± 3.5
37.7 ± 2.8
35.8 ± 4.1
24.9 ± 3.5
37.7 ± 2.8
SI
22.5 ± 3.0
40.0 ± 4.1
33.6 ± 2.5
41.3 ± 3.8
40.0 ± 4.1
33.6 ± 2.5
50.1 ± 4.5
40.0 ± 4.1
33.6 ± 2.5
TCN
38.6 ± 3.5
45.3 ± 3.9
33.0 ± 2.1
49.2 ± 4.2
45.3 ± 3.9
33.0 ± 2.1
56.4 ± 3.9
45.3 ± 3.9
33.0 ± 2.1
Rank2Reward
43.1 ± 4.2
50.2 ± 4.0
32.1 ± 1.9
54.8 ± 3.7
50.2 ± 4.0
32.1 ± 1.9
61.7 ± 4.1
50.2 ± 4.0
32.1 ± 1.9
GAIL
48.9 ± 4.5
55.8 ± 3.8
31.6 ± 1.5
60.3 ± 4.1
55.8 ± 3.8
31.6 ± 1.5
65.2 ± 3.8
55.8 ± 3.8
31.6 ± 1.5
Table 1. Detailed performance comparison on three tasks from ContinualBench. For Success Rate and AP, higher is better ( ↑ ). For Regret, lower is better ( ↓ ). ProgAgent demonstrates the best performance across all tasks and metrics.
Figure 4. Comparative performance of ProgAgent against baseline methods on the final evaluation reward after sequential training on three tasks from ContinualBench: button-press , door-open , and window-close . Bars indicate the mean reward. ProgAgent consistently outperforms all other methods across all tasks.
Figure 5. Learning curves of reward versus training steps on ContinualBench tasks. Shaded regions indicate standard deviation. ProgAgent (yellow) shows consistently faster learning and achieves higher final rewards across all tasks, demonstrating its superior sample efficiency.
Figure 6. Qualitative analysis of the learned potential function Φϕ(ot) . The plot compares the potential value over time for an expert trajectory (blue), a successful agent trajectory (green, dashed), and a failure trajectory (red). Successful paths exhibit a smooth, monotonically increasing potential, demonstrating that the learned function provides effective, dense guidance toward the goal. In contrast, the potential for the failure path remains low and stagnant, confirming the model correctly penalizes non-progressive behavior and avoids spurious reward hacking.
Figure 7. Ablation study on ProgAgent components, evaluated after training on the full task sequence. Performance is reported on the final task ( window-close ). The results highlight the critical role of adversarial push-back in reward stability and CL regularizers in mitigating catastrophic forgetting.
Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to 11.57% success rate gains.
Xinbei Ma, Congmin Zheng, Jiyang Qiu +11
Shanghai Jiao Tong University · OPPO Research Institute
We present ProgVLA, a compact vision-language-action (VLA) model designed for reliable robot manipulation under tight compute and memory budgets. The model specifically focuses on efficiently processing long multi-modal sequences by maintaining an explicit representation of task progress over extended horizons. To this end, ProgVLA integrates two key components. First, a multi-modal encoder with a two-stage Perceiver resampling scheme compresses variable-length visual, language, and proprioceptive streams into a fixed set of control-ready context tokens, substantially reducing sequence length while preserving cross-modal grounding. Second, an auxiliary set of progress heads is trained with offline reinforcement learning (RL) objectives to jointly learn critics over normalized remaining-horizon targets. This provides the policy with an internal estimate of task progress and enables advantage- and success-weighted flow-matching imitation learning. On two well-established multi-task robot manipulation benchmarks, a 0.1B-parameter ProgVLA model reaches success rates that are competitive with, and on long-horizon and harder task tiers exceed, substantially larger pretrained baselines. Ablations indicate that the learned context resampler and task-adaptive visual fine-tuning are the largest single contributors, while progress-aware training provides a consistent additional gain that is concentrated on long-horizon and multi-object tasks. We further validate the approach in real-world toy-kitchen environments.
Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.
Zixuan Chen, Lin Cao, Fenggang Zhang +5
Nanjing University · Southeast University · Fudan University