Deep reinforcement learning systems often suffer from unstable training dynamics due to non-stationarity, where learning objectives and data distributions evolve over time. We show that under non-stationary targets, isotropic Gaussian embeddings are provably advantageous. In particular, they induce stable tracking of time-varying targets for linear readouts, achieve maximal entropy under a fixed variance budget, and encourage a balanced use of all representational dimensions--all of which enable agents to be more adaptive and stable. Building on this insight, we propose the use of Sketched Isotropic Gaussian Regularization for shaping representations toward an isotropic Gaussian distribution during training. We demonstrate empirically, over a variety of domains, that this simple and computationally inexpensive method improves performance under non-stationarity while reducing representation collapse, neuron dormancy, and training instability.
Figures & tables
Figure 1: Illustration of two tracking regimes. Left: the norm of the tracking error exhibits non-monotonic behavior and fails to converge, indicating unstable tracking. Right: the tracking error decreases monotonically and converges to zero, corresponding to stable tracking dynamics.
Figure 2: Directly shaping a multivariate distribution. SIGReg first projects the embeddings onto a small set of random directions ( pi : sketching), producing multiple univariate distributions. Each projection is then matched to the corresponding univariate target distribution.
Figure 3: Non-stationary CIFAR-10. Training under repeated label shuffling. The baseline shows poor recovery after each shift, with SIGReg loss spikes, rank collapse, and increased dormancy. Enforcing isotropic Gaussian representations stabilizes training, accelerates recovery, preserves rank, and reduces dormancy.
Figure 4: Two Atari-10 games (PQN). Without isotropy regularization, representations exhibit rank collapse, increased neuron dormancy, and early performance saturation. Encouraging isotropic geometry leads to improved representation quality and higher, more stable performance.
Figure 5: 2D PCA of embedding covariance over training. Without constraints, representations collapse onto a few dominant principal components. Encouraging isotropic Gaussian structure yields more evenly distributed variance and higher effective dimensionality, reflected in reduced concentration on the leading components (PQN: [0.4,0.2]→[0.9,0.8] vs. PQN+SIGReg: [0.3,0.2]→[0.1,0.1] ).
Figure 6: Full Atari suite. Effect of isotropic Gaussian regularization. Left: IQM human-normalized learning curves as a function of environment steps for PQN and PPO, with and without isotropic regularization. Right: Per-game improvement in AUC obtained by encouraging isotropic representation geometry. Across both algorithms, isotropic regularization improves final performance and sample efficiency.
Figure 7: Isaac Gym continuous control. Learning curves on two representative locomotion tasks. Isotropic Gaussian representations improve stability and reduce variance. We report returns over 5 runs for each experiment. See Section D for additional results on four Isaac Gym control tasks.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Train Accuracy( ↑ )
SIGReg Loss( ↓ )
Feature Rank ( ↑ )
Dormant Neurons%
Adam
33.0
49.1
149.0
18.5
Adam+SIGReg
50.0
7.6
262.9
0.4
RAdam
29.8
42.3
148.8
12.6
RAdam+SIGReg
42.8
8.4
251.8
0.6
Kron
62.6
28.5
167.4
7.9
Kron+SIGReg
72.7
7.2
252.2
0.9
Appendix
Table 1: Non-stationary CIFAR-10. Area Under the Curve (AUC) for different methods across evaluation metrics. SIGReg loss minimization improve results across all metrics and optimizers.
Figure 8: Effect of non-stationarity and SIGReg loss minimization on representation stability and accelerating recovery in non-stationary CIFAR-10 experiment. Non-stationarity induced by label shuffling causes a sharp drop in accuracy, increase in SIGReg loss, collapse of feature rank, and higher neuron dormancy. It can be observed that Kronecker-factored optimizer implicitly lowers the SIGReg loss compared to first-order methods. Explicitly minimizing SIGReg further improves all optimizers by accelerating recovery, preserving feature rank, and reducing neuron dormancy.
PQN
PPO
Method
% of Games Improved ( ↑ )
Avg. Improvement ( ↑ )
% of Games ( ↑ )
Avg. Improvement ( ↑ )
Gaussian (Real and Imaginary parts)
90%
305%
70%
7.9%
Laplacian
90%
234%
60%
3.2%
Logistic
90%
209%
50%
-0.8%
Real Part Only
100%
151%
60%
2.4%
Imaginary Part Only
70%
56%
50%
1.9%
Appendix
Table 2: Ablation study on Atari-10. Percentage of games improved over baseline and average AUC improvement across all games. Top: different isotropic target distributions (Laplacian and Logistic). Middle: minimizing different SIGReg loss components. Bottom: whitening the covariance matrix without controlling tails. Gaussian with real and imaginary components performs best, while enforcing isotropy alone can improve some environments. In PQN, minimizing only the real part improves more games than the baseline Isotropic Gaussian, though with significantly smaller average improvement.
PQN
PPO
Method
% of Games Improved ( ↑ )
Avg. Improvement ( ↑ )
Avg. SPS ( ↑ )
% of Games Improved ( ↑ )
Avg. Improvement ( ↑ )
Avg. SPS ( ↑ )
Kron
98.2%
1404%
1.6k
45.6%
-2.6%
1.8k
Baseline + SIGReg
89.5%
889%
9.3k
68.4%
25.5%
2.7k
Baseline + MultiSkip + SIGReg
92.9%
1695%
11.4k
87.7%
88.6%
3.2k
Appendix
Table 3: PQN and PPO Performance Across Full Atari Suite. We report the percentage of games in which each method improves over the baseline, the average percentage of AUC improvement across all games, and the average steps per second (Avg. SPS). In terms of average AUC improvement, the baseline augmented with multi-skip residual connections and SIGReg loss minimization outperforms all other methods, while also achieving higher SPS and avoiding the memory overhead associated with computing gradient curvature. In terms of the percentage of games improved, Kronecker-factored optimization achieves the best result by a small margin in PQN, and it is outperformed by both two other cases in PPO. Overall, these results indicate that the performance gains provided by Kron do not justify the additional memory and computational cost.
Figure 9: Atari-10, PQN, Kron Optimizer. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, Kron leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy.
Figure 10: Atari-10, PQN, Multi-skip Residual Architecture. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, using Multi-skip Residual Architecture leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy.
Figure 11: Atari-10, PPO, Kron Optimizer. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, Kron leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy.
Figure 12: Atari-10, PPO, Multi-skip Residual Architecture. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, using Multi-skip Residual Architecture leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy.
Figure 13: Atari-10, PQN with Explicit SIGReg Loss Minimization. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons through time. The orange line is the baseline without SIGReg loss minimization, and the other lines are SIGReg loss minimization with different strengths (Larger λ means stronger regularization).
Figure 14: IQM Human-Normalized Results for PQN (top) and PPO (bottom) Across the Atari-10 Benchmark Games. Explicit SIGReg loss minimization encourages isotropic Gaussian embeddings, reducing the performance gap between first-order optimizers (RAdam) and second-order Kronecker-factored optimization. Adding multi-skip residual connections further improves performance.
Figure 15: PQN. Reward over time for individual games with and without SIGReg loss minimization.
Figure 16: PPO. Reward over time for individual games with and without SIGReg loss minimization.
Figure 17: Isaac Gym continuous control. Learning curves for PPO and PPO + SIGReg on four representative locomotion tasks. Encouraging isotropic Gaussian representations improves learning stability and asymptotic performance across all environments. Curves show mean episode returns over five independent runs, with shaded regions indicating variability across runs.
Table 9: Best regularization factor ( λ ) selected via AUC for PQN and PPO algorithms across Atari games (Larger means stronger regularization)
Figure 18: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for isotropic Gaussian distribution , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 19: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for anisotropic Gaussian distribution with condition number κ=10 , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 20: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for anisotropic Gaussian distribution with condition number κ=100 , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 21: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for isotropic Laplace distribution , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 22: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for anisotropic Laplace distribution with condition number κ=10 , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 23: Tracking error norm ∥et∥2=∥wt−w∗∥2 over all steps for anisotropic Laplace distribution with condition number κ=100 , across three random seeds (columns). The learner tracks a sequence of three target weights, w1∗⟶t=100w2∗⟶t=200w3∗ , each drawn uniformly at random from the unit sphere. The corresponding target angles (θ1,θ2,θ3) are reported in each column title. Green dots and dotted vertical lines mark the two task-switch points at t∈{100,200} . Dashed red vertical lines indicate steps at which ∥et+1∥2>∥et∥2 , highlighting transient divergence episodes. Each switch induces an abrupt spike in the tracking error norm due to the change in w∗ , followed by reconvergence, whose rate and steady-state level depend on the geometry of the sample distribution.
Figure 24: Tracking error vectors et=wt−w∗ in the 2D weight space for isotropic Gaussian distribution . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 25: Tracking error vectors et=wt−w∗ in the 2D weight space for anisotropic Gaussian distribution with condition number κ=10 . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 26: Tracking error vectors et=wt−w∗ in the 2D weight space for anisotropic Gaussian distribution with condition number κ=100 . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 27: Tracking error vectors et=wt−w∗ in the 2D weight space for isotropic Laplace distribution . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 28: Tracking error vectors et=wt−w∗ in the 2D weight space for anisotropic Laplace distribution with condition number κ=10 . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 29: Tracking error vectors et=wt−w∗ in the 2D weight space for anisotropic Laplace distribution with condition number κ=100 . The learner is trained sequentially on three tasks defined by target weights w1∗,w2∗,w3∗ , each drawn uniformly at random from the unit circle, with task switches at steps t=100 and t=200 . Each panel corresponds to one phase of training (rows) and one random seed (columns), and the title indicates the active target w∗ for that phase. Within each panel, black arrows originate at the origin and point to et , illustrating the instantaneous direction and magnitude of the tracking error. The green dot marks e1 , the first error vector immediately after each task switch. Red arrows connect consecutive error vectors when the error norm increases, while blue arrows indicate decreases in the error norm. Axes are shown on a symmetric logarithmic scale to simultaneously visualize transient spikes following task switches and steady-state residual errors.
Figure 30: Tracking error dynamics under isotropic Gaussian and anisotropic Laplace feature distributions. The left column shows the evolution of the tracking error vector et=wt−w∗ after the task switch, while the right column shows the corresponding error norm ∥et∥2 over time. The task switches at t=Tswitch from w1∗=(0,±1) to w2∗=(⋅,⋅) (shown in the legend), with two scenarios overlaid in each panel. Arrow plots (left). Each arrow from the origin represents et at a given step; purple and black arrows correspond to the two configurations. Colored arrows between successive tips indicate temporal evolution: red for an increase in ∥et∥2 , blue for a decrease. The first post-switch error e1 is highlighted by a color-coded dot. Axes use a symmetric log scale to capture small and large deviations. Norm plots (right). Solid curves show ∥et∥2 for each scenario; vertical dashed red lines mark norm increases and dotted lines indicate the task switch, with markers highlighting the error at switching. Initialization. w1∗=(0,±1) aligns with the principal direction of the feature distribution (largest eigenvalue of covariance), which in the anisotropic case is the dominant eigenvector of Σ . Anisotropic distribution. For anisotropic Laplace, features are sampled independently with coordinate scales to match a diagonal covariance matrix with eigenvalues (1+κ2,1+κ2κ) (trace 2, condition number κ ), ensuring both distributions have the same total variance. Observation. While isotropic Gaussian features show near-monotone error contraction, anisotropic Laplace features lead to frequent increases and non-monotone trajectories that plateau before reaching zero, highlighting the impact of anisotropy and non-Gaussianity on tracking stability.
Figure 31: Tracking error dynamics under different input distributions. Evolution of the tracking error ∥et∥2=∥wt−w∗∥2 over time for Gaussian and Laplace input distributions with varying condition numbers ( κ ), shown across three random seeds (columns). The top row corresponds to d=8 and the bottom row to d=32 . The learner tracks a sequence of target weights, w1∗t=100w2∗t=200w3∗ , each sampled uniformly from the unit sphere. Each target switch induces a sharp increase in error, followed by reconvergence, whose rate and stability depend on the geometry of the input distribution. Across both dimensions and seeds, isotropic Gaussian inputs yield smooth, monotonic, and rapid convergence to zero error, whereas isotropic Laplace inputs also converge but exhibit pronounced transient spikes. Increasing the condition number (i.e., reduced isotropy) leads to slower convergence and, in some cases, failure to fully converge.
Figure 32: Existence of divergent directions under anisotropic Gaussian distributions. We test the hypothesis that when the tracking error aligns with the eigenvector corresponding to the smallest eigenvalue of the input covariance in the reverse direction ( et=−vd ), the dynamics fail to converge to zero steady-state error. Specifically, for Gaussian inputs, we initialize the task such that the error vector is aligned with −vd , where vd is the eigenvector associated with the smallest eigenvalue, and subsequently update the target weights so that et=−βvd at each step. Results show that larger condition numbers ( κ ) lead to increased divergence, whereas the isotropic case ( κ=1 ) remains convergent. Moreover, increasing β amplifies divergence; however, for moderate conditioning (e.g., κ=5 ), smaller values of β (e.g., β=2.4 ) do not induce divergence.
Method
% of Games Improved ( ↑ )
Avg. Improvement ( ↑ )
Gaussian (Real and Imaginary parts)
90%
305%
Laplacian
90%
234%
Logistic
90%
209%
Real Part Only
100%
151%
Imaginary Part Only
70%
56%
Covariance Whitening (VICReg)
90%
144%
Appendix
Table 10: Comparison with alternative regularization methods. We report the percentage of games improved over the baseline and the average improvement across Atari-10 using the PQN algorithm. Top: different isotropic target distributions. Middle: minimizing different components. Bottom: comparison with alternative regularization methods. Among the baselines, embedding norm regularization performs the worst. Simplicial embedding improves over L2 normalization, but a substantial gap remains compared to isotropic Gaussian embeddings. Weight orthogonalization improves a similar number of games as isotropic Gaussian, but yields lower average improvement; moreover, it requires computing the Gram matrix of all weights, which is computationally expensive. VICReg, which promotes isotropy without enforcing Gaussianity, improves a similar number of games but does not match the average improvement of isotropic Gaussian embeddings.
PQN
PPO
Game
λ
Start
Mid
End
Game
λ
Start
Mid
End
0.0
5.02
1.26
1.29
0.0
20.96
4.01
3.45
Amidar
0.2
5.02
10.63
10.45
Amidar
0.2
20.96
11.93
14.08
1.0
5.02
15.66
17.93
1.0
20.96
15.19
25.75
0.0
4.32
1.46
1.39
0.0
19.60
2.85
3.24
Qbert
0.2
4.32
8.00
7.52
Qbert
0.2
19.60
12.78
15.20
Appendix
Table 11: Entropy of eigenvalues. Comparison of eigenvalue entropy at the start, middle, and end of training for PQN (left) and PPO (right) across different games and λ values. Amidar and Qbert are selected as representative cases where SIGReg improves performance in both PQN and PPO. Freeway, Pitfall, Atlantis, and Venture are cases where adding the SIGReg loss degrades performance relative to the baseline. Comparing the same games and baseline settings, PPO consistently exhibits higher final entropy, indicating less severe representation collapse. For a fixed game, increasing λ leads to higher entropy, as expected, since the SIGReg loss promotes equalization of eigenvalues.
PQN
PPO
Game
λ
Avg. Condition Number Ratio
Game
λ
Avg. Condition Number Ratio
Amidar
0.2
8.8×103
Amidar
0.2
8.3×101
1.0
2.1×104
1.0
1.8×102
Qbert
0.2
1.2×101
Qbert
0.2
1.3×102
1.0
6.7×103
1.0
4.3×102
Freeway
0.2
8.5×105
Atlantis
0.2
3.9
Appendix
Table 12: Condition number ratio averaged over training. We report the time-averaged ratio of condition numbers between the baseline and SIGReg embeddings, κSIGRegκbaseline . According to our theoretical analysis, a lower condition number is desirable, as it indicates more uniform convergence rates across directions and more controlled drift term. As shown, λ=1 yields a substantially lower condition number than the baseline, consistent with the effect of SIGReg in spreading eigenvalues and promoting isotropy.
Modern neural network optimization relies heavily on architectural priorssuch as Batch Normalization and Residual connectionsto stabilize training dynamics. Without these, or in low-data regimes with aggressive augmentation, low-bias architectures like Vision Transformers (ViTs) often suffer from optimization collapse. This work adopts Sketched Isotropic Gaussian Regularization (SIGReg), recently introduced in the LeJEPA self-supervised framework, and repurposes it as a general optimization stabilizer for supervised learning. While the original formulation targets the full characteristic function, a computationally efficient variant is derived, Weak-SIGReg, which targets the covariance matrix via random sketching. Inspired by interacting particle systems, representation collapse is viewed as stochastic drift; SIGReg constrains the representation density towards an isotropic Gaussian, mitigating this drift. Empirically, SIGReg recovers the training of a ViT on CIFAR-100 from a collapsed 20.73% to 72.02% accuracy without architectural hacks and significantly improves the convergence of deep vanilla MLPs trained with pure SGD. Code is available at github.com/kreasof-ai/sigreg.
Deep reinforcement learning policies achieve strong performance in complex continuous control environments with nonlinear contact forces. However, these policies often produce chaotic state dynamics, with trivially small changes to the initial conditions significantly impacting the long-term behaviour of the control system. This high sensitivity to initial conditions limits the application of Deep RL to real-world control systems where performance and stability guarantees are often required. To address this issue, we propose Global stabilisation via Intrinsic Fine Tuning (GIFT), a general-purpose training framework which directly optimises the global stability of existing high-performing deep RL policies using a custom reward function. We demonstrate that GIFT increase the stability of the control interaction while maintaining comparable task performance, thereby improving the suitability of deep RL policies for real-world control systems.
While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping. Instead, PPO performs persistent, directionally inefficient local updates, which indicates a lack of geometry-aware guidance for accumulating meaningful behavioral change and ultimately hindering transitions toward new behavior patterns. Although divergence-based regularization introduces partial geometric awareness, its monotonically increasing penalties implicitly discourage large policy deviations, even when such shifts are necessary for effective adaptation. To address this limitation, we propose Gaussian Trust Region Policy Optimization (GTR), which reshapes the trust region using a Gaussian kernel. The resulting constraint is bounded and non-monotonic, providing strong local stability while progressively relaxing under sustained high-advantage updates. To further improve robustness, we introduce a Mixture Gaussian Anchor that adapts to recent policy trajectories, reducing variance induced by stale references. GTR is architecture-agnostic and achieves strong performance across games, simulated robotic control, open-world exploration, and language model post-training. These results demonstrate that geometry-aware trust-region design can be a promising direction for robust reinforcement learning in complex non-stationary environments. Our code is available at https://anonymous.4open.science/r/GTR_demo/README.md.
Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5
Hong Kong University of Science and Technology · Fudan University · Mila - Québec AI Institute +2