Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach

Chen Xu

cs.LGApr 27, 2026

Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach

Authors: Chen Xu

Organizations: Department of Engineering, Shenzhen MSU-BIT University. Address: 1 International University Park Road, Longgang District, Shenzhen, Guangdong, China.

Abstract

We present a framework using Relative Entropy Inverse Reinforcement Learning (RE-IRL) to recover investor reward functions from observed investment actions and market conditions. Unlike traditional IRL algorithms, RE-IRL is employed to account for environments where transition probabilities are unknown or inaccessible. To address the challenge of data sparsity, we utilize a $K$ -nearest neighbor approach to estimate the observed behavior policy. Furthermore, we propose a statistical testing framework to evaluate the validity and robustness of the estimated results.

Explore similar work

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
May 29, 2026Enoch Hyunwook KangEntropy Regularized Reinforcement Learning Offline Reinforcement Learning
Quantifying Potential Observation Missingness in Inverse Reinforcement Learning
May 12, 2026Leo Benac, Abhishek Sharma, Alihan Huyuk +1Entropy Regularized Reinforcement Learning Missing Data
Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
Dec 7, 2025Denis Belomestny, Alexey Naumov, Artemy Rubtsov +1Entropy Regularized Reinforcement Learning Optimal Policies

May 29, 2026 · cs.LGJ/K move · Enter open · S save

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Enoch Hyunwook Kang

University of Washington, Foster School of Business

In the forward reinforcement-learning problem, the reward is fixed and known; the learner is asked to find a good policy or value function. Here we turn the question around. Given offline data generated by an expert, can we recover the reward the expert was optimizing? This is the inverse reinforcement learning problem, and remarkably, two communities, structural econometricians studying dynamic discrete choice (DDC) and machine learners studying entropy-regularized IRL, have been working on exactly the same probabilistic model under different names. We begin by proving their equivalence. We then develop the classical identification result of Magnac and Thesmar and the classical computational paradigms that grew out of it: Rust's nested fixed-point algorithm, the conditional-choice-probability approach of Hotz and Miller, and the two temporal-difference approaches of Adusumilli and Eckardt: linear semi-gradient TD and approximate value iteration. Each route has its limits: dimensionality, transition-kernel estimation, the deadly triad, or projected fixed-point bias. We then walk through the modern ML/IRL strand: adversarial IRL, occupancy matching, IQ-Learn, and offline ML-IRL, deriving each method's actual objective and stating precisely what it does and does not identify. We close with the empirical-risk-minimization framework of Kang et al., which yields a gradient-based estimator for offline IRL/DDC.

Entropy Regularized Reinforcement Learning Offline Reinforcement Learning Discrete Choice Models Deep Reinforcement Learning Online Fixed-Point Iteration

Open paper Ask about it PDF

Abstract

Explore similar work

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

Statistical analysis of Inverse Entropy-regularized Reinforcement Learning