Adaptive Incentive Design in Dynamic Principal-Agent Problem via Kernelized Bandits

Arghya Mallick; Anuj S. Vora; Sergio Grammatico; Peyman Mohajerin Esfahani

cs.MAAug 18, 2026

Adaptive Incentive Design in Dynamic Principal-Agent Problem via Kernelized Bandits

Authors: Arghya Mallick, Anuj S. Vora, Sergio Grammatico, Peyman Mohajerin Esfahani

Organizations: Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands · Jio Institute, Navi Mumbai, India · University of Toronto, Toronto, Canada

Abstract

We consider the dynamic principal-agent problem under asymmetric information, wherein a principal sequentially designs contracts to incentivize an agent with unknown preferences and hidden actions. A fundamental bottleneck in the existing literature is the assumption of deterministic agent utility, which renders the principal's expected utility discontinuous and forces computationally intractable discretizations of the contract space. In this paper, we address this limitation by introducing a stochastic counterpart into the agent's utility model, capturing the inherent physical and behavioral variations in realistic subsystems. We formally prove that this stochastic formulation restores the continuity of the principal's expected utility. Leveraging this continuous geometric structure, we formulate the interaction as a structured multi-armed bandit problem subject to heteroscedastic noise. We propose a Heteroscedastic GP-UCB algorithm that utilizes a Neural Network (Arcsin) kernel, chosen to capture the non-stationary, sigmoidal geometry of the utility landscape. For an $m$ -dimensional compact contract space, we establish a high-probability cumulative regret bound of $O\left(\sqrt{T}(\log T)^{m+1}\right)$ . Finally, we demonstrate the practical efficacy of our theoretical framework by formulating the Vehicle-to-Grid (V2G) incentive design problem, proving its equivalence to a dynamic principal-agent problem, and showing superior economic performance for grid aggregators.

Figures & tables

Figure 1. Schematic of principal-agent feedback model ( 1 ).

Panel of Figure 2 — Figure 2. (High-low example 2.1 ) (a) Visualization of ${U}^{\mathrm{PA}}$ . (b) Visualization of the variance of heteroscedastic noise $\sigma^{2}_{\varepsilon}$ . (c) Geometry of $\tilde{U}^{\mathrm{PA}}$ under the Assumption 3.1 .

Panel of Figure 3 — Figure 3. Left: The agent’s utility $U^{\mathrm{A}}(x,a)$ with optimizer marked at $a^{\star}$ . Right: The modified utility $\tilde{U}^{\mathrm{A}}(x,a):=U^{\mathrm{A}}(x,a)+\langle w,a\rangle$ with a realization $w=0.3$ . Angled pointers clearly distinguish the original optimizer $a^{\star}$ from the newly shifted optimizer $\tilde{a}^{\star}$ .

Figure 4. Representative function of $\tilde{U}^{\mathrm{PA}}$ , which is built with Neural Network kernel using kernel regression where the empirical samples are generated from an instance of ‘High-low’ example 2.1 .

Figure 5. V2G incentive design schematic showing sequential interaction between aggregators and EV users.

Agent	Low	Med.	High
Cost $c(a_{i})$	$\mathcal{N}$ $(0,\,0.05)$	$\mathcal{N}$ $(0.1,\,0.05)$	$\mathcal{N}$ $(0.2,\,0.05)$
Production $p(a_{i})$	$[0.95,$ $0.05]$	$[0.2,$ $0.8]$	$[0.05,$ $0.95]$
Principal
Outcome $i$	Low ( $i=0$ )	High ( $i=1$ )
Value $v$	$v_{0}=0.1$	$v_{1}=0.8$

Table 1. Performance comparison of different methods. Effort levels correspond to

a_{1}

(low),

a_{2}

(medium), and

a_{3}

(high).

Panel of Figure 6 — Figure 6. Performance comparison on the synthetic benchmark and V2G profit analysis: (a) Performance comparison of Heteroscedastic GP-UCB on the synthetic benchmark. The y-axis displays the Average Utility Reward ( $U_{T}$ ), not cumulative regret. Therefore, the convergence of our proposed algorithm (green line) to a high, positive horizontal asymptote ( $\approx 0.45$ ) indicates successful exploitation of the optimal contract, outperforming the baselines, which fail to converge to this optimal yield. (b) Comparison of the net V2G profit ( $J_{\text{net}}$ ) for two scenarios, highlighting the impact of V2G incentive schemes under the benchmarked algorithms.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. Assessing performance of Heteroscedastic GP-UCB (paired with Neural Network kernel) under two scenarios. In scenario $1$ , it outperforms GP-UCB [ 35 ] with a Neural Network kernel, validating the effect of considering heteroscedastic noise. In scenario $2$ , it also outperforms the radial basis function (RBF) kernel, validating our kernel choice.

Table 2. Details of uncertain and fixed parameters for numerical simulation.

Explore similar work

Jun 1, 2026

A No-Regret Framework for Adaptive Incentive Design

Incentive design studies how a central authority can influence strategic agents through payments, subsidies, or taxes, so that individual objectives align with collective welfare. This paper introduces a No-Regret Adaptive Incentive Design (RAID) framework for nonlinear games with continuous action spaces and private agent costs. In this framework, the authority (planner) designs incentives that regulate the Nash equilibrium toward a socially optimal action profile, while simultaneously learning agents' unknown preferences from repeated strategic responses. We formulate the RAID problem and construct a least-squares estimator whose strong consistency requires only diminishing excitation. Leveraging this weak excitation requirement, we propose a switching incentive policy that alternates between probing (exploration) and estimate-based (exploitation) incentives. The resulting policy achieves an

O(t^{-0.5})

parameter estimation rate and accumulates

O(t^{0.5}\log t)

squared social-cost regret, almost surely. We further extend the framework to an endogenous-noise response model, where standard least-squares estimation is biased due to an error-in-variables correlation between the noise and agent responses. We utilize a repeated-sampling estimator and corresponding switching policy that retain the same almost-sure convergence and regret rates. Numerical experiments validate the effectiveness and predicted convergence rates of the method.

Georgios Vasileiou, Lantian Zhang, Silun Zhang

Department of Mathematics, KTH Royal Institute of Technology, Stockholm, 10044, Sweden.

Mechanism Design Game Theory Regret Minimization

Ask PDF

Apr 7, 2026

Adaptive Incentive Design with Regret Minimization

Incentive design constitutes a foundational paradigm for influencing the behavior of strategic agents, wherein a system planner (principal) publicly commits to an incentive mechanism designed to align individual objectives with collective social welfare. This paper introduces the Regret-Minimizing Adaptive Incentive Design (RAID) problem, which aims to synthesize incentive laws under information asymmetry and achieve asymptotically minimal regret compared to an oracle with full information. To this end, we develop the RAID algorithm, which employs a switching policy alternating between probing (exploration) and estimate-based incentivization (exploitation). The associated type estimator relies only on a weaker excitation condition required for strong consistency in least squares estimation, substantially relaxing the persistence-of-excitation assumptions previously used in adaptive incentive design. In addition, we establish the strong consistency of the proposed type estimator and prove that the incentive obtained asymptotically minimizes the planner's average regret almost surely. Numerical experiments illustrate the convergence rate of the proposed methodology.

Georgios Vasileiou, Lantian Zhang, Silun Zhang

Dept. of Mathematics at the KTH Royal Institute of Technology, Stockholm, 10044, Sweden.

Regret Minimization Mechanism Design

Ask PDF

Apr 13, 2026

Incentive Design without Hypergradients: A Social-Gradient Method

In incentive design problems, a system planner seeks to steer self-interested agents toward a socially optimal Nash equilibrium by issuing incentives in the presence of information asymmetry, that is, uncertainty about the agents' cost functions. A common approach formulates the problem as a Mathematical Program with Equilibrium Constraints and optimizes incentives using hypergradients-the total derivatives of the planner's objective with respect to incentives. However, computing the hypergradient typically requires full or partial knowledge of equilibrium sensitivities to incentives, which is generally unavailable under information asymmetry. In this paper, we propose a hypergradient-free incentive law, called the social-gradient flow, for incentive design when the social cost depends on the agents' joint action. We prove that the social cost gradient is a descent direction for the planner's objective, irrespective of the agent cost landscape. In the idealized setting where equilibrium responses are observable, the social-gradient flow converges to the unique socially optimal incentive. When equilibria are not directly observable, the social-gradient flow emerges as the slow-timescale limit of a two-timescale interaction, in which agents' strategies evolve on a faster timescale. We establish that the joint strategy-incentive dynamics converge to the social optimum for any agent learning rule that asymptotically tracks the equilibrium. Finally, we validate theoretical results via numerical experiments.

Georgios Vasileiou, Lantian Zhang, Silun Zhang

Department of Mathematics at the KTH Royal Institute of Technology, Stockholm, 10044, Sweden.

Multi-Agent System Optimization Mechanism Design Game Theory Gradient Descent Dynamics

Ask PDF