Gradient Boosting Reinforcement Learning

Benjamin Fuhrer; Chen Tessler; Gal Dalal

cs.LGJul 11, 2024

Gradient Boosting Reinforcement Learning

Authors: Benjamin Fuhrer, Chen Tessler, Gal Dalal

Organizations: NVIDIA, Tel-Aviv, Israel · NVIDIA Research, Tel-Aviv, Israel.

Abstract

We present Gradient Boosting Reinforcement Learning (GBRL), a framework that adapts the strengths of gradient boosting trees (GBT) to reinforcement learning (RL) tasks. While neural networks (NNs) have become the de facto choice for RL, they face significant challenges with structured and categorical features and tend to generalize poorly to out-of-distribution samples. These are challenges for which GBTs have traditionally excelled in supervised learning. However, GBT's application in RL has been limited. The design of traditional GBT libraries is optimized for static datasets with fixed labels, making them incompatible with RL's dynamic nature, where both state distributions and reward signals evolve during training. GBRL overcomes this limitation by continuously interleaving tree construction with environment interaction. Through extensive experiments, we demonstrate that GBRL outperforms NNs in domains with structured observations and categorical features while maintaining competitive performance on standard continuous control benchmarks. Like its supervised learning counterpart, GBRL demonstrates superior robustness to out-of-distribution samples and better handles irregular state-action relationships.

Figures & tables

Figure 1: The GBRL framework. The actor’s policy and critic’s value function are parameterized by the tree ensemble

F_{k}

. For example,

F_{k}(\state)=[\mu(\state),\sigma(\state),V(\state)]

for a Gaussian policy. Starting from

F_{0}

, at each training iteration,

k

, GBRL collects a rollout and computes the gradient

\nabla_{F_{k-1}}J(\pi_{F_{k-1}})

with respect to the current ensemble. This gradient is then used to fit the next tree. Adding the tree to the ensemble updates it to

F_{k}(\state)=F_{k-1}(\state)+\epsilon h_{k}(\state)

, where

\epsilon

is the learning rate.

Library	Incremental Learning (sec)	Standard Learning (sec)
CatBoost	$36.17\pm 2.77$	$1.15\pm 0.03$
XGBoost	$67.11\pm 0.70$	$2.26\pm 3.70$
GBRL	$\mathbf{5.47}\pm\mathbf{1.19}$	N/A

Table 1: Incremental Learning Speed in GBT Libraries. Training times (seconds) for 1000 boosting iterations on random batches of 128 samples with 20 input features and 3 targets. While CatBoost and XGBoost train efficiently in standard mode using the same batch for all iterations, their training time increases significantly in incremental learning, where each batch receives only one boosting iteration. GBRL, designed explicitly for incremental updates, remains much faster.

Figure 2: GBT library comparison, Cartpole. CatBoost and XGBoost are intractable in RL. CatBoost’s lack of GPU support for custom losses leads to low FPS and early termination.

Figure 3: Shared Actor-Critic, MiniGrid. The shared tree structure significantly increases efficiency, without impacting the score. Aggregated results over three tasks are shown here, full per-task curves are available in Appendix ( Figure 20 )

Figure 4: GBRL vs NN in standard environments (PPO). Aggregated mean and standard deviation of the normalized average reward for the final 100 episodes.

Figure 5: Signal dilution, variable isolation task. Mean and standard deviation of the average episodic reward during training. GBRL was trained for 15M training steps and NN for 30M. Episodes are terminated after 50 steps if not solved.

Figure 6: Spurious correlations, MiniGrid. Mean and standard deviation of the episodic reward. Plots outlined in green are in-distribution, while plots outlined in red are out-of-distribution.

Panel of Figure 7 — Figure 7: Spurious correlations revealed by feature importance in MiniGrid. The visualization shows normalized SHAP values for the concluding action in a successful evaluation run, revealing the magnitude and direction of each feature’s impact on model predictions. In this scenario, a red box was positioned next to the target (red ball), and both models converged to a successful policy. GBRL prioritizes the mission, ignores the red box, and contrasts the blue ball. In contrast, the NN focuses on the cells surrounding the target, while ignoring the blue ball and contrasting the box and the mission.

Figure 8: Robustness to irrelevant objects, MiniGrid. Mean and standard deviation of the average episodic reward during training. GBRL was trained for 5M training steps, and NN for 20M.

Figure 9: Robustness to input perturbations, classic-control. Models were trained without perturbations and then evaluated on noisy observations. At each timestep, the observation was perturbed following

\state_{t}=\state_{t}+\epsilon|\state_{t}|

, where

\epsilon\sim\mathcal{N}(0,\text{var})

. We report the mean and standard deviation of episodic reward across 100 evaluation episodes for each

\text{Variance}\in[0,0.2]

Figure 10: Out-of-distribution evaluation, football. Mean and standard deviation of episodic reward across 100 evaluation episodes. Plots outlined in green are in-distribution, while plots outlined in red are out-of-distribution.

Figure 11: Ablation: Learning rate variation across policy (left) and value function (right) components . Mean and standard deviation of the average episodic reward during training. Each subplot shows the effect of varying the learning rate for one component (policy or value function) while keeping the other fixed.

Figure 12: Ablation: Effect of tree depth on convergence . We varied the maximum depth of the decision trees used in the gradient-boosted models and evaluated their impact on learning dynamics. The plot shows the mean and standard deviation of the average episodic reward over time and the total runtime.

Figure 13: Ablation: Batch size impact on convergence . Mean and standard deviation of the average episodic reward during training. We fix the rollout length to 2048 and vary the batch size. Smaller batches improve early learning but suffer from instability, while larger batches stabilize training at the cost of slower convergence.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14: Multi-objective gradient split strategies in PPO-GBRL on CartPole-v1 . We compare gradient normalization and cosine similarity as methods for handling actor-critic gradient imbalance when building trees. Both strategies yield good performance, with cosine similarity achieving slightly faster convergence. Curves show average episodic reward over 1M timesteps.

Figure 15: Effect of number of gradient updates per rollout on PPO-GBRL’s performance in CartPole-v1 . We fix the batch size and the number of samples collected per rollout, while varying the number of PPO epochs, thereby changing the number of gradient updates per rollout. This improves sample efficiency, with more epochs leading to faster convergence. Results are shown over 1M timesteps. The number of samples collected per rollout was fixed to 2048 for all experiments.

Table 2: GBRL hyperparameters - NN represented by an MLP with two hidden layers.

	Acrobot	CartPole	LunarLander	MountainCar	MountainCar Continuous	Pendulum-v1
NN: A2C	$-82.27\pm 3.29$	$500.00\pm 0.0$	$-43.01\pm 106.26$	$-148.90\pm 24.10$	$92.66\pm 0.32$	$-183.64\pm 22.32$
GBRL: A2C	$-90.73\pm 2.98$	$500.00\pm 0.0$	$47.93\pm 41.00$	$-124.42\pm 5.74$	$93.15\pm 1.19$	$-538.83\pm 66.25$
NN: AWR	$-102.53\pm 57.25$	$500.00\pm 0.0$	$282.48\pm 1.96$	$-160.65\pm 53.97$	$18.93\pm 42.34$	$-159.64\pm 9.42$
GBRL: AWR	$-118.12\pm 33.54$	$497.54\pm 3.11$	$76.03\pm 56.62$	$-146.68\pm 24.53$	$44.38\pm 45.94$	$-1257.61\pm 98.10$
NN: PPO	$-74.83\pm 1.22$	$500.00\pm 0.0$	$261.73\pm 6.93$	$-115.53\pm 1.39$	$85.81\pm 7.51$	$-249.31\pm 60.00$
GBRL: PPO	$-87.82\pm 2.16$	$500.00\pm 0.0$	$248.72\pm 59.10$	$-110.55\pm 15.60$	$89.42\pm 5.73$	$-246.89\pm 20.61$

Appendix

Table 3: Continuous-Control and Box2D environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 4: Football Academy environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 5: Atari-ramNoFrameskip-v4 environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Table 6: MiniGrid environments: Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 16: Classic Control and Box2D environments: Training reward as a function of environment steps. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 17: Football Academy environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 18: Atari-ramNoFrameskip-v4 environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 19: MiniGrid environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.

Figure 20: Sharing actor critic tree structure significantly increases efficiency while retraining similar performance . Training reward, GPU memory usage, and FPS, are compared across 10M environment (5 seeds, 3 MiniGrid environments)

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Learning the Supports for Categorical Critic in Reinforcement Learning
Jul 2, 2026Jen-Yen Chang, Takayuki Osa, Tatsuya HaradaDistributional RL
Graduate School of Information Science and Technology, The University of Tokyo · RIKEN AIP
FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning
May 18, 2026Xikai Zhang, Yongzhi Li, Likang Xiao +6RL for Language Model Reasoning RL Exploration
Hangzhou International Innovation Institute, Beihang University · Kuaishou Technology · School of Artificial Intelligence, Beihang University
Meta-RL with Bayesian Linear Task Models
Dec 24, 2025Jingyang You, Hanna KurniawatiRepresentation Learning Reinforcement Learning
School of Computing Australian National University

Jul 2, 2026J/K move · Enter open · S save

Learning the Supports for Categorical Critic in Reinforcement Learning

Jen-Yen Chang, Takayuki Osa, Tatsuya Harada

Graduate School of Information Science and Technology, The University of Tokyo · RIKEN AIP

Distributional RL

Open paper AskPDF arXiv

Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to bootstrapped target values. Meanwhile, in distributional RL, a distribution of returns is modelled based on the distributional Bellman operator. This work investigates the Gaussian Histogram Loss (HL-Gauss), a recent approach that reframes value estimation as classification by encoding each scalar Bellman target as a Gaussian-smoothed categorical target. Despite its potential, applying histogram-based losses to RL presents inherent challenges, most notably the requirement to pre-define a fixed support interval, which is often complicated by the non-stationary and stochastic nature of target values typically found in RL tasks. In this work, we propose an approach that dynamically learns the lower and upper bounds of the support instead of assigning them beforehand. We derive an objective that jointly learns these bounds whilst learning the categorical representation of the scalar values, and we show that this objective forms an upper bound on the mean-squared Bellman error. Our theoretical analysis further shows that this bound is tighter than that of non-learned supports of HL-Gauss. Empirically, the proposed objective enables stable adaptation of the support interval and matches HL-Gauss-based actor-critic algorithms on most continuous-control tasks whilst improving on a subset, without requiring a pre-specified support interval.

Abstract

Reinforcement Learning

Explore similar work

Learning the Supports for Categorical Critic in Reinforcement Learning

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

Meta-RL with Bayesian Linear Task Models