cs.LGAug 13, 2023

When the Agent Learns Too: Learning to Incentivize with Unobserved Rewards

Authors: Ilgin Dogan, Zuo-Jun Max Shen, Anil Aswani

Organizations: Department of Industrial Engineering and Operations Research, University of California, Berkeley, Berkeley, California 94720

Abstract

In practice, incentive providers (i.e., principals) often cannot observe the reward realizations of incentivized agents, which is in contrast to many principal-agent models that have been previously studied. This information asymmetry challenges the principal to consistently estimate the agent's unknown rewards by solely watching the agent's decisions, which becomes even more challenging when the agent has to learn its own rewards. This complex setting is observed in various real-life scenarios ranging from renewable energy storage contracts to personalized healthcare incentives. Hence, it offers not only interesting theoretical questions but also wide practical relevance. This paper explores a repeated adverse selection game between a self-interested learning agent and a learning principal. The agent tackles a multi-armed bandit (MAB) problem to maximize their expected reward plus incentive. On top of the agent's learning, the principal trains a parallel algorithm and faces a trade-off between consistently estimating the agent's unknown rewards and maximizing their own utility by offering adaptive incentives to lead the agent. For a non-parametric model, we introduce an estimator whose only input is the history of principal's incentives and agent's choices. We unite this estimator with a proposed data-driven incentive policy within a MAB framework. Without restricting the type of the agent's algorithm, we prove finite-sample consistency of the estimator and a rigorous regret bound for the principal by considering the sequential externality imposed by the agent. Lastly, our theoretical results are reinforced by simulations justifying applicability of our framework to green energy aggregator contracts.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Learning to Incentivize Perfect-Knowledge Agents with Unobserved Rewards

    Apr 14, 2023Ilgin Dogan, Zuo-Jun Max Shen, Anil AswaniBandit LearningMulti-Armed Bandits

  2. A No-Regret Framework for Adaptive Incentive Design

    Jun 1, 2026Georgios Vasileiou, Lantian Zhang, Silun ZhangMechanism DesignGame Theory

  3. Adaptive Incentive Design in Dynamic Principal-Agent Problem via Kernelized Bandits

    Aug 18, 2026Arghya Mallick, Anuj S. Vora, Sergio Grammatico +1Principal-Agent TheoryMechanism Design