ProgAgent:A Continual RL Agent with Progress-Aware Rewards

Jinzhou Tan; Gabriel Adineera; Jinoh Kim

cs.LGMar 8, 2026

ProgAgent:A Continual RL Agent with Progress-Aware Rewards

Authors: Jinzhou Tan, Gabriel Adineera, Jinoh Kim

Organizations: University of California, San Diego, USA · Texas A&M University-Commerce, USA

Abstract

We present ProgAgent, a continual reinforcement learning (CRL) agent that unifies progress-aware reward learning with a high-throughput, JAX-native system architecture. Lifelong robotic learning grapples with catastrophic forgetting and the high cost of reward specification. ProgAgent tackles these by deriving dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations. We theoretically interpret this as a learned state-potential function, delivering robust guidance in line with expert behaviors. To maintain stability amid online exploration - where novel, out-of-distribution states arise - we incorporate an adversarial push-back refinement that regularizes the reward model, curbing overconfident predictions on non-expert trajectories and countering distribution shift. By embedding this reward mechanism into a JIT-compiled loop, ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible: it merges PPO with coreset replay and synaptic intelligence for an enhanced stability-plasticity balance. Evaluations on ContinualBench and Meta-World benchmarks highlight ProgAgent's advantages: it markedly reduces forgetting, boosts learning speed, and outperforms key baselines in visual reward learning (e.g., Rank2Reward, TCN) and continual learning (e.g., Coreset, SI) - surpassing even an idealized perfect memory agent. Real-robot trials further validate its ability to acquire complex manipulation skills from noisy, few-shot human demonstrations.

Figures & tables

Panel of Figure 1 — Figure 1. Overview of ProgAgent’s training pipeline, illustrating the unified loop of reward learning, policy optimization, and continual mechanisms under JAX acceleration.

Figure 2. The complete training pipeline of ProgAgent. Our approach utilizes two parallel data streams. Top path An offline stream uses unlabeled expert videos to train a Progress Prediction Network. The network’s output is optimized via a contrastive expert loss ( $L_{\text{expert}}$ ) to accurately estimate task progress. Bottom path An online stream collects agent experience through large-scale parallel rollouts. To ensure robustness against out-of-distribution states, an adversarial push-back loss ( $L_{\text{push}}$ ) regularizes the reward function by pushing predictions on these non-expert trajectories toward a low-confidence (zero-mean, high-variance) prior. These two objectives are combined into a final loss, $L_{\text{total}}$ , to produce a dense and stable reward signal for policy training.

Figure 3. Qualitative comparison of the learned policies for the Button-Press task in the ContinualBench environment. The sequences display the agent’s behavior at the end of training. From top to bottom: a standard Finetuning agent, the Online Agent , and our ProgAgent . ProgAgent successfully learns a direct and stable policy to complete the task, whereas the baseline methods exhibit less optimal or failed behaviors.

Method	button-press			door-open			window-close
	Success Rate $\uparrow$	AP $\uparrow$	Regret $\downarrow$	Success Rate $\uparrow$	AP $\uparrow$	Regret $\downarrow$	Success Rate $\uparrow$	AP $\uparrow$	Regret $\downarrow$
Fine-tuning	15.2 $\pm$ 2.1	24.9 $\pm$ 3.5	37.7 $\pm$ 2.8	12.5 $\pm$ 1.9	24.9 $\pm$ 3.5	37.7 $\pm$ 2.8	35.8 $\pm$ 4.1	24.9 $\pm$ 3.5	37.7 $\pm$ 2.8
SI	22.5 $\pm$ 3.0	40.0 $\pm$ 4.1	33.6 $\pm$ 2.5	41.3 $\pm$ 3.8	40.0 $\pm$ 4.1	33.6 $\pm$ 2.5	50.1 $\pm$ 4.5	40.0 $\pm$ 4.1	33.6 $\pm$ 2.5
TCN	38.6 $\pm$ 3.5	45.3 $\pm$ 3.9	33.0 $\pm$ 2.1	49.2 $\pm$ 4.2	45.3 $\pm$ 3.9	33.0 $\pm$ 2.1	56.4 $\pm$ 3.9	45.3 $\pm$ 3.9	33.0 $\pm$ 2.1
Rank2Reward	43.1 $\pm$ 4.2	50.2 $\pm$ 4.0	32.1 $\pm$ 1.9	54.8 $\pm$ 3.7	50.2 $\pm$ 4.0	32.1 $\pm$ 1.9	61.7 $\pm$ 4.1	50.2 $\pm$ 4.0	32.1 $\pm$ 1.9
GAIL	48.9 $\pm$ 4.5	55.8 $\pm$ 3.8	31.6 $\pm$ 1.5	60.3 $\pm$ 4.1	55.8 $\pm$ 3.8	31.6 $\pm$ 1.5	65.2 $\pm$ 3.8	55.8 $\pm$ 3.8	31.6 $\pm$ 1.5

Table 1. Detailed performance comparison on three tasks from ContinualBench. For Success Rate and AP, higher is better (

\uparrow

). For Regret, lower is better (

\downarrow

). ProgAgent demonstrates the best performance across all tasks and metrics.

Figure 4. Comparative performance of ProgAgent against baseline methods on the final evaluation reward after sequential training on three tasks from ContinualBench: button-press , door-open , and window-close . Bars indicate the mean reward. ProgAgent consistently outperforms all other methods across all tasks.

Figure 5. Learning curves of reward versus training steps on ContinualBench tasks. Shaded regions indicate standard deviation. ProgAgent (yellow) shows consistently faster learning and achieves higher final rewards across all tasks, demonstrating its superior sample efficiency.

Figure 6. Qualitative analysis of the learned potential function

\Phi_{\phi}(o_{t})

. The plot compares the potential value over time for an expert trajectory (blue), a successful agent trajectory (green, dashed), and a failure trajectory (red). Successful paths exhibit a smooth, monotonically increasing potential, demonstrating that the learned function provides effective, dense guidance toward the goal. In contrast, the potential for the failure path remains low and stagnant, confirming the model correctly penalizes non-progressive behavior and avoids spurious reward hacking.

Figure 7. Ablation study on ProgAgent components, evaluated after training on the full task sequence. Performance is reported on the final task ( window-close ). The results highlight the critical role of adversarial push-back in reward stability and CL regularizers in mitigating catastrophic forgetting.

Continual RL

Continual reinforcement learning studies how agents adapt to changing tasks while retaining skills learned earlier.

See all 50 papers

Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Jun 12, 2026Xinbei Ma, Congmin Zheng, Jiyang Qiu +11Reward Shaping LLM Agent Training
Shanghai Jiao Tong University · OPPO Research Institute
ProgVLA: Progress-Aware Robot Manipulation Skill Learning
May 27, 2026Seungsu Kim, Jinyoung Choi, Seungmin Baek +1Cross-Modal Learning Efficient Multimodal Inference
NAVER LABS Europe · NAVER LABS
RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation
Oct 24, 2026Zixuan Chen, Lin Cao, Fenggang Zhang +5Long-Horizon Robotic Manipulation Robot Policy Adaptation
Nanjing University · Southeast University · Fudan University

Jun 12, 2026J/K move · Enter open · S save

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

Xinbei Ma, Congmin Zheng, Jiyang Qiu, Jiale Hong+10

Shanghai Jiao Tong University · OPPO Research Institute

Reward Shaping LLM Agent Training Long-Horizon LLM Agents Policy Learning RL for LLM Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to 11.57% success rate gains.

Abstract

Continual RL

Explore similar work

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

ProgVLA: Progress-Aware Robot Manipulation Skill Learning

RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation