Linear Function Approximation

Linear function approximation aims to efficiently represent value functions in reinforcement learning and other sequential decision-making problems by using linear combinations of features. Current research emphasizes developing provably efficient algorithms for various settings, including those with off-policy learning, delayed feedback, heavy-tailed rewards, and multiple agents, often employing techniques like temporal difference learning, fitted Q-iteration, and natural policy gradients. These advancements address the computational challenges of large state spaces and improve the theoretical understanding and practical performance of reinforcement learning algorithms, impacting fields like robotics, control systems, and AI safety.

Papers

October 15, 2024

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, Yufa Zhou
Human Attention Transformer Architecture LLM Inference Linear Function Approximation Attention Matrix Softmax Attention Terminator Economy

September 18, 2024

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features
Jiuqi Wang, Shangtong Zhang
Reinforcement Learning Value Function Linear Function Approximation Temporal Difference Learning Temporal Difference Algorithm

September 16, 2024

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation
Woojin Chae, Dabeen Lee
Linear Function Approximation Bellman Equation Linear Markov Decision Process Linear Mixture Markov Decision Process

September 15, 2024

Marginalizing and Conditioning Gaussians onto Linear Approximations of Smooth Manifolds with Applications in Robotics
Zi Cong Guo, James R. Forbes, Timothy D. Barfoot
Robotics Domain RSD Difference of Gaussian Class Manifold Linear Function Approximation Conditional Distribution Dimensional Manifold Covariance Estimation

September 12, 2024

Fitted Q-Iteration via Max-Plus-Linear Approximation
Y. Liu, M. A. S. Kolarijani
Markov Decision Process Offline Reinforcement Learning Linear Function Approximation Fitted Q Iteration Max Affine

August 23, 2024

Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisation
Ben Batten, Yang Zheng, Alessandro De Palma, Panagiotis Kouvaros, Alessio Lomuscio
Neural Network Verification Task Linear Function Approximation Piecewise Linear Geometric Transformation Tight Approximation

June 26, 2024

Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs
Tian Tian, Lin F. Yang, Csaba Szepesvári
Policy Gradient Linear Function Approximation Reinforcement Learning Approach Constrained Markov Decision Process Local Planning

June 17, 2024

June 12, 2024

Finite Time Analysis of Temporal Difference Learning for Mean-Variance in a Discounted MDP
Tejaram Sangadi, L. A. Prashanth, Krishna Jagannathan
Markov Decision Process Actor Critic Algorithm Linear Function Approximation Temporal Difference Learning Finite Time Risk Sensitive Reinforcement Learning Markov Reward Process Mean Variance Optimization

June 2, 2024

Towards General Robustness Verification of MaxPool-based Convolutional Neural Networks via Tightening Linear Approximation
Yuan Xiao, Shiqing Ma, Juan Zhai, Chunrong Fang, Jinyuan Jia, Zhenyu Chen
Convolutional Neural Network Native Robustness Linear Function Approximation Robustness Verification

May 27, 2024

May 26, 2024

Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learning
Sergey Samsonov, Eric Moulines, Qi-Man Shao, Zhuo-Song Zhang, Alexey Naumov
Financial Application Parameter Estimation Stochastic Approximation Linear Function Approximation Non Asymptotic Analysis TD Learning Normal Distribution Gaussian Approximation Polyak Ruppert

May 25, 2024

Front-propagation Algorithm: Explainable AI Technique for Extracting Linear Function Approximations from Neural Networks
Javier Viaña
Neural Network Deep Neural Network Explainability Method Linear Function Approximation Front Propagation

May 3, 2024

Imitation Learning in Discounted Linear MDPs without exploration assumptions
Luca Viano, Stratis Skoulakis, Volkan Cevher
Imitation Learning Feature Imitation Efficient Exploration Linear Function Approximation Adversarial Loss Finite Horizon Linear MDPs

March 4, 2024

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation
Aritra Mitra
Stochastic Approximation Linear Function Approximation Finite Time Finite Time Convergence Multidimensional Projection TD Learning Markovian Sampling

February 24, 2024

Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation
Donghwan Lee
General Analysis Gradient Descent Model Free Reinforcement Learning Linear Function Approximation Deterministic Algorithm Multi Step Off Policy

February 23, 2024

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation
Zhishuai Liu, Pan Xu
Linear Function Approximation Distributionally Robust Provable Benefit Online POMDP Solver Distributionally Robust Markov Decision Process

February 11, 2024

Refined Sample Complexity for Markov Games with Independent Linear Function Approximation
Yan Dai, Qiwen Cui, Simon S. Du
Multi Agent Reinforcement Learning Multi Agent Markov Game Single Agent Linear Function Approximation Single Agent RL

Linear Function Approximation

Papers

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

Marginalizing and Conditioning Gaussians onto Linear Approximations of Smooth Manifolds with Applications in Robotics

Fitted Q-Iteration via Max-Plus-Linear Approximation

Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisation

Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs

The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation

Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions

Finite Time Analysis of Temporal Difference Learning for Mean-Variance in a Discounted MDP

Towards General Robustness Verification of MaxPool-based Convolutional Neural Networks via Tightening Linear Approximation

Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges

Provably Efficient Reinforcement Learning with Multinomial Logit Function Approximation

Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learning

Front-propagation Algorithm: Explainable AI Technique for Extracting Linear Function Approximations from Neural Networks

Imitation Learning in Discounted Linear MDPs without exploration assumptions

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

Refined Sample Complexity for Markov Games with Independent Linear Function Approximation