cs.LGMar 4, 2026

When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift

Authors: Kevin Vogt-Lowell, Theodoros Tsiligkaridis, Rodney Lafuente-Mercado, Surabhi Ghatti, Shanghua Gao, Marinka Zitnik, Daniela Rus

Organizations: MIT Lincoln Laboratory · Harvard · MIT

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Real-world reinforcement learning systems must operate under distributional drift in their observation streams, yet most policy architectures implicitly assume fully observed and noise-free states. We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures that induce partial observability and representation shift. To respond to this drift, we augment PPO with temporal sequence models, including Transformers and State Space Models (SSMs), to enable policies to infer missing information from history and maintain performance. Under a stochastic sensor failure process, we prove a high-probability bound on infinite-horizon reward degradation that quantifies how robustness depends on policy smoothness and failure persistence. Empirically, on MuJoCo continuous-control benchmarks with severe sensor dropout, we show Transformer-based sequence policies substantially outperform MLP, RNN, and SSM baselines in robustness, maintaining high returns even when large fractions of sensors are unavailable. These results demonstrate that temporal sequence reasoning provides a principled and practical mechanism for reliable operation under observation drift caused by sensor unreliability.

Figures & tables

Figure 1: Sample PPO training curves on HalfCheetah-v4 under full (left) and 60% partial (right) observability. Lines represent median episodic return and shaded regions denote inter-quartile ranges across 8 random seeds. Training curves generated under partial observability rise more slowly and plateau at lower returns than those produced using fully observed states.
Figure 2: Evaluation episodic returns for PPO agents on MuJoCo environments under full (left) and 60% partial (right) observability, with task complexity roughly increasing from top to bottom. Each violin shows the distribution of pooled episodic returns from 100 episodes across 8 random seeds. Black markers denote the median with 95% bootstrapped CI. While all models suffer performance degradation under partial observability, the Transformer agent demonstrates greater robustness.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Total timesteps1×1061\times 10^{6}
Learning rate3×10−43\times 10^{-4}
Number of steps2048
Discount factor ( γ\gamma )0.99
GAE parameter ( λ\lambda )0.95
Number of minibatches32
Appendix
Table 1: Proximal Policy Optimization (PPO) Hyperparameters
ModelHiddenLayersHeadsDropoutAdditional
Transformer128220.1–
Gated TransformerXL128220Memory length 8
UniTS128220.1Patch length 1; Prompt tokens 6
GRU1284–0–
LRU1284–0rmin⁡r_{\min} 0.9; rmax⁡r_{\max} 0.999; Max phase 6.28
MLP128––––
Appendix
Table 2: Architectural Hyperparameters per Model
Figure 3: PPO training curves for Hopper-v4, Walker2d-v4, and Ant-v4 under full (left) and 60% partial (right) observability. Lines represent median episodic return and shaded regions denote inter-quartile ranges across 8 random seeds.
More in this topic

Robust RL

Robust reinforcement learning develops agents that maintain performance and safety under uncertainty, disturbances, attacks, and distribution shifts.

See all 229 papers

Explore similar work

CardsList

  1. Robustness Analysis of POMDP Policies to Observation Perturbations

    Apr 23, 2026Benjamin Kraske, Qi Heng Ho, Federico Rossi +2Robust Partially Observable Markov Decision ProcessesPartially Observable Markov Decision Processes

    University of Colorado Boulder, Boulder, Colorado, USA · Virginia Tech, Blacksburg, Virginia, USA · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, California, USA

  2. Adversarial observations in probabilistic State-Space Models for robust Reinforcement Learning

    Jun 18, 2026M. Santos-Pascual, D. Ríos InsuaAdversarial AttacksPartially Observable RL

    1Inst. Math. Sciences, Spanish Nat. Research Council, Madrid, Spain · Universidad Autónoma de Madrid, Escuela de Doctorado, Madrid, Spain

  3. Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

    Oct 26, 2026Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli +1Loss of PlasticityProximal Policy Optimization

    DAUIN, Politecnico di Torino, 10129 Turin, Italy · SAIL Research Unit, Istituto Italiano di Tecnologia, 16163 Genoa, Italy · Rehab Technologies Lab, Istituto Italiano di Tecnologia, 16163 Genoa, Italy