Gradient Boosting Reinforcement Learning

Benjamin Fuhrer; Chen Tessler; Gal Dalal

cs.LGJul 11, 2024

Gradient Boosting Reinforcement Learning

Authors: Benjamin Fuhrer, Chen Tessler, Gal Dalal

Organizations: NVIDIA, Tel-Aviv, Israel · NVIDIA Research, Tel-Aviv, Israel.

Abstract

We present Gradient Boosting Reinforcement Learning (GBRL), a framework that adapts the strengths of gradient boosting trees (GBT) to reinforcement learning (RL) tasks. While neural networks (NNs) have become the de facto choice for RL, they face significant challenges with structured and categorical features and tend to generalize poorly to out-of-distribution samples. These are challenges for which GBTs have traditionally excelled in supervised learning. However, GBT's application in RL has been limited. The design of traditional GBT libraries is optimized for static datasets with fixed labels, making them incompatible with RL's dynamic nature, where both state distributions and reward signals evolve during training. GBRL overcomes this limitation by continuously interleaving tree construction with environment interaction. Through extensive experiments, we demonstrate that GBRL outperforms NNs in domains with structured observations and categorical features while maintaining competitive performance on standard continuous control benchmarks. Like its supervised learning counterpart, GBRL demonstrates superior robustness to out-of-distribution samples and better handles irregular state-action relationships.

Figures & tables

Figure 1: The GBRL framework. The actor’s policy and critic’s value function are parameterized by the tree ensemble

F_{k}

. For example,

F_{k}(\state)=[\mu(\state),\sigma(\state),V(\state)]

for a Gaussian policy. Starting from

F_{0}

, at each training iteration,

k

, GBRL collects a rollout and computes the gradient

\nabla_{F_{k-1}}J(\pi_{F_{k-1}})

with respect to the current ensemble. This gradient is then used to fit the next tree. Adding the tree to the ensemble updates it to

F_{k}(\state)=F_{k-1}(\state)+\epsilon h_{k}(\state)

, where

\epsilon

is the learning rate.

Library	Incremental Learning (sec)	Standard Learning (sec)
CatBoost	$36.17\pm 2.77$	$1.15\pm 0.03$
XGBoost	$67.11\pm 0.70$	$2.26\pm 3.70$
GBRL	$\mathbf{5.47}\pm\mathbf{1.19}$	N/A

Table 1: Incremental Learning Speed in GBT Libraries. Training times (seconds) for 1000 boosting iterations on random batches of 128 samples with 20 input features and 3 targets. While CatBoost and XGBoost train efficiently in standard mode using the same batch for all iterations, their training time increases significantly in incremental learning, where each batch receives only one boosting iteration. GBRL, designed explicitly for incremental updates, remains much faster.

Figure 2: GBT library comparison, Cartpole. CatBoost and XGBoost are intractable in RL. CatBoost’s lack of GPU support for custom losses leads to low FPS and early termination.

Figure 3: Shared Actor-Critic, MiniGrid. The shared tree structure significantly increases efficiency, without impacting the score. Aggregated results over three tasks are shown here, full per-task curves are available in Appendix ( Figure 20 )

Figure 4: GBRL vs NN in standard environments (PPO). Aggregated mean and standard deviation of the normalized average reward for the final 100 episodes.

Figure 5: Signal dilution, variable isolation task. Mean and standard deviation of the average episodic reward during training. GBRL was trained for 15M training steps and NN for 30M. Episodes are terminated after 50 steps if not solved.

Figure 6: Spurious correlations, MiniGrid. Mean and standard deviation of the episodic reward. Plots outlined in green are in-distribution, while plots outlined in red are out-of-distribution.

Panel of Figure 7 — Figure 7: Spurious correlations revealed by feature importance in MiniGrid. The visualization shows normalized SHAP values for the concluding action in a successful evaluation run, revealing the magnitude and direction of each feature’s impact on model predictions. In this scenario, a red box was positioned next to the target (red ball), and both models converged to a successful policy. GBRL prioritizes the mission, ignores the red box, and contrasts the blue ball. In contrast, the NN focuses on the cells surrounding the target, while ignoring the blue ball and contrasting the box and the mission.

Figure 8: Robustness to irrelevant objects, MiniGrid. Mean and standard deviation of the average episodic reward during training. GBRL was trained for 5M training steps, and NN for 20M.

Figure 9: Robustness to input perturbations, classic-control. Models were trained without perturbations and then evaluated on noisy observations. At each timestep, the observation was perturbed following

\state_{t}=\state_{t}+\epsilon|\state_{t}|

, where

\epsilon\sim\mathcal{N}(0,\text{var})

. We report the mean and standard deviation of episodic reward across 100 evaluation episodes for each

\text{Variance}\in[0,0.2]

Figure 10: Out-of-distribution evaluation, football. Mean and standard deviation of episodic reward across 100 evaluation episodes. Plots outlined in green are in-distribution, while plots outlined in red are out-of-distribution.

Figure 11: Ablation: Learning rate variation across policy (left) and value function (right) components . Mean and standard deviation of the average episodic reward during training. Each subplot shows the effect of varying the learning rate for one component (policy or value function) while keeping the other fixed.

Figure 12: Ablation: Effect of tree depth on convergence . We varied the maximum depth of the decision trees used in the gradient-boosted models and evaluated their impact on learning dynamics. The plot shows the mean and standard deviation of the average episodic reward over time and the total runtime.

Figure 13: Ablation: Batch size impact on convergence . Mean and standard deviation of the average episodic reward during training. We fix the rollout length to 2048 and vary the batch size. Smaller batches improve early learning but suffer from instability, while larger batches stabilize training at the cost of slower convergence.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14: Multi-objective gradient split strategies in PPO-GBRL on CartPole-v1 . We compare gradient normalization and cosine similarity as methods for handling actor-critic gradient imbalance when building trees. Both strategies yield good performance, with cosine similarity achieving slightly faster convergence. Curves show average episodic reward over 1M timesteps.

Figure 15: Effect of number of gradient updates per rollout on PPO-GBRL’s performance in CartPole-v1 . We fix the batch size and the number of samples collected per rollout, while varying the number of PPO epochs, thereby changing the number of gradient updates per rollout. This improves sample efficiency, with more epochs leading to faster convergence. Results are shown over 1M timesteps. The number of samples collected per rollout was fixed to 2048 for all experiments.

Table 2: GBRL hyperparameters - NN represented by an MLP with two hidden layers.

	Acrobot	CartPole	LunarLander	MountainCar	MountainCar Continuous	Pendulum-v1
NN: A2C	$-82.27\pm 3.29$	$500.00\pm 0.0$	$-43.01\pm 106.26$	$-148.90\pm 24.10$	$92.66\pm 0.32$	$-183.64\pm 22.32$
GBRL: A2C	$-90.73\pm 2.98$	$500.00\pm 0.0$	$47.93\pm 41.00$	$-124.42\pm 5.74$	$93.15\pm 1.19$	$-538.83\pm 66.25$
NN: AWR	$-102.53\pm 57.25$	$500.00\pm 0.0$	$282.48\pm 1.96$	$-160.65\pm 53.97$	$18.93\pm 42.34$	$-159.64\pm 9.42$
GBRL: AWR	$-118.12\pm 33.54$	$497.54\pm 3.11$	$76.03\pm 56.62$	$-146.68\pm 24.53$	$44.38\pm 45.94$	$-1257.61\pm 98.10$
NN: PPO	$-74.83\pm 1.22$	$500.00\pm 0.0$	$261.73\pm 6.93$	$-115.53\pm 1.39$	$85.81\pm 7.51$	$-249.31\pm 60.00$
GBRL: PPO	$-87.82\pm 2.16$	$500.00\pm 0.0$	$248.72\pm 59.10$	$-110.55\pm 15.60$	$89.42\pm 5.73$	$-246.89\pm 20.61$

Appendix

Table 3: Continuous-Control and Box2D environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 4: Football Academy environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 5: Atari-ramNoFrameskip-v4 environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 6: MiniGrid environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 16: Classic Control and Box2D environments: Training reward as a function of environment steps. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 17: Football Academy environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 18: Atari-ramNoFrameskip-v4 environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 19: MiniGrid environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 20: Sharing actor critic tree structure significantly increases efficiency while retraining similar performance . Training reward, GPU memory usage, and FPS, are compared across 10M environment (5 seeds, 3 MiniGrid environments)

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Jul 2, 2026

Learning the Supports for Categorical Critic in Reinforcement Learning

Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to bootstrapped target values. Meanwhile, in distributional RL, a distribution of returns is modelled based on the distributional Bellman operator. This work investigates the Gaussian Histogram Loss (HL-Gauss), a recent approach that reframes value estimation as classification by encoding each scalar Bellman target as a Gaussian-smoothed categorical target. Despite its potential, applying histogram-based losses to RL presents inherent challenges, most notably the requirement to pre-define a fixed support interval, which is often complicated by the non-stationary and stochastic nature of target values typically found in RL tasks. In this work, we propose an approach that dynamically learns the lower and upper bounds of the support instead of assigning them beforehand. We derive an objective that jointly learns these bounds whilst learning the categorical representation of the scalar values, and we show that this objective forms an upper bound on the mean-squared Bellman error. Our theoretical analysis further shows that this bound is tighter than that of non-learned supports of HL-Gauss. Empirically, the proposed objective enables stable adaptation of the support interval and matches HL-Gauss-based actor-critic algorithms on most continuous-control tasks whilst improving on a subset, without requiring a pre-specified support interval.

Jen-Yen Chang, Takayuki Osa, Tatsuya Harada

Graduate School of Information Science and Technology, The University of Tokyo · RIKEN AIP

Distributional RL

Ask PDF

May 18, 2026

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models. At its core, the training loop of GRPO and its variants alternates between rollout sampling and policy update: the policy first samples rollouts from its action space, and then updates its parameters according to the advantages computed over them. Unlike supervised learning, where each gradient step is anchored to an explicit ground-truth target, the optimal gradient direction for updating model parameters in this setting is not known a priori; the high-quality rollouts drawn during the sampling stage therefore act as the implicit "teacher" that guides every parameter update. However, mainstream RL algorithms such as GRPO adopt a simple sampling scheme that conditions all rollouts on the same original prompt. When a task lies beyond the policy model's current capability, this sampling scheme rarely yields a high-quality rollout, leaving the policy model without a meaningful gradient direction when updating its parameters, which causes training to stall. To address this issue, we propose FBOS-RL. Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment, and on top of this we design two mutually reinforcing training objectives: EPA and ECC. Extensive experiments demonstrate that EPA and ECC can mutually reinforce each other, forming a positive flywheel effect that significantly improves both the training efficiency and the final performance ceiling of reinforcement learning. Specifically, under both an identical number of rollouts and the same number of training steps, FBOS-RL learns substantially faster than GRPO and feedback-based baselines and ultimately attains a higher performance ceiling, while exhibiting higher policy entropy and lower gradient norms throughout training.

Xikai Zhang, Yongzhi Li, Likang Xiao +6

Hangzhou International Innovation Institute, Beihang University · Kuaishou Technology · School of Artificial Intelligence, Beihang University

RL for Language Model Reasoning RL Exploration RL Post-Training RL Fine-Tuning

Ask PDF

Dec 24, 2025

Meta-RL with Bayesian Linear Task Models

Deep Bayesian reinforcement learning adapts to unseen tasks by inferring latent transition and reward models, but existing methods typically rely on variational posteriors and evidence lower bounds, introducing approximation error and unstable task representations. We introduce GLiBRL, a deep Bayesian RL framework that combines generalised linear task models with learnable non-linear basis functions. GLiBRL features conjugate Bayesian inference, yielding exact, sequential posterior updates over task parameters and model noise, together with a closed-form marginal likelihood that eliminates variational inference. The update is naturally permutation-invariant, allowing GLiBRL to integrate with both off- and on-policy algorithms. GLiBRL also learns task representation admitting an exact kernel identity, relating distances between task representations to kernel discrepancies over the task contexts. Compared against eight representative or recent meta reinforcement learning methods, GLiBRL achieves the highest aggregate zero-shot test performance on both the MuJoCo locomotion and MetaWorld manipulation benchmarks.

Jingyang You, Hanna Kurniawati

School of Computing Australian National University

Representation Learning Reinforcement Learning Bayesian RL Meta-Reinforcement Learning Generalization in RL Model-Based RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Learning the Supports for Categorical Critic in Reinforcement Learning

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

Meta-RL with Bayesian Linear Task Models