Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

Xinyu Dai; Daniel Chen; Yian Qian

econ.THFeb 19, 2026

Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

Authors: Xinyu Dai, Daniel Chen, Yian Qian

Organizations: Department of Economics, Brown University. · Department of Applied Mathematics, Brown University.

Abstract

Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.

Figures & tables

\pi_{t+1}(\theta)=\frac{f_{\theta}(R_{t}\mid A_{t})\,\pi_{t}(\theta)}{\sum_{\vartheta\in\Theta}f_{\vartheta}(R_{t}\mid A_{t})\,\pi_{t}(\vartheta)},\qquad\forall\theta\in\Theta.

Algorithm 1 Thompson Sampling

Panel of Figure 1 — Figure 1: Crossing configuration showing misspecified models.

Panel of Figure 2 — Figure 2: Comparison of Case 1 and Case 2 scenarios. In Case 1, both models agree arm $A_{1}$ is optimal. In Case 2, models disagree: $\nu$ prefers $A_{1}$ while $\gamma$ prefers $A_{2}$ . The three regions (I, II, III) for the true expected rewards are indicated on each action line. In Region I the true expected reward is such that model $\nu$ is closer to the true DGP in KL divergence ( $\Delta_{i}>0$ ), in Region III model $\gamma$ is closer ( $\Delta_{i}<0$ ), and Region II is the neutral boundary ( $\Delta_{i}=0$ ).

Case	Condition	Behavior	Limit Distribution
Agreement	$\Delta_{1}>0$	Submartingale	$\delta_{1}$
Agreement	$\Delta_{1}=0$	Martingale	$\text{Bernoulli}(1/2)$
Agreement	$\Delta_{1}<0$	Supermartingale	$\delta_{0}$
Self-Confirming	$\Delta_{1}>0>\Delta_{2}$	Two attractors	$\text{Bernoulli}(p^{*})$
Uniform Dominance	$\Delta_{1},\Delta_{2}>0$	Global attractor	$\delta_{1}$
Self-Defeating	$\Delta_{1}<0<\Delta_{2}$	Ergodic	Invariant measure on $(0,1)$

Table 1: Complete classification of Thompson Sampling behavior under misspecification in the two-arm case

Case	Prob. of	Prob. of	Average
	Action 1	Action 2	Regret
Agreement ( $\phi(\nu)=\phi(\gamma)=1$ )	$1$	$0$	$0$
Agreement ( $\phi(\nu)=\phi(\gamma)=2$ )	$0$	$1$	$g(1)-g(2)$
Self-Confirming	Bernoulli $(p^{*})$	Bernoulli $(1-p^{*})$	$\begin{cases}0\text{ w.p. }p^{}\\ g(1)-g(2)\text{ w.p. }1-p^{}\end{cases}$
Uniform Dominance	$1$	$0$	$0$
Self-Defeating	$\alpha^{*}$	$1-\alpha^{*}$	$(1-\alpha^{*})[g(1)-g(2)]$

Table 2: Limiting action frequencies and average regret of Thompson Sampling in the two-arm model under misspecification

Panel of Figure 3 — Figure 3: Vector field visualizations. Left two columns show S-space; right two columns show simplex space. (a,b) Self-confirming equilibrium; (c,d) Self-defeating equilibrium. (e,f) Drift towards face interior; (g,h) Drift towards vertex.

Panel of Figure 4 — Figure 4: Different equilibrium types in multi-model Thompson Sampling. In each panel, the arrows $d(a_{1}),d(a_{2}),d(a_{3})$ represent the drift vectors associated with each model, which are the expected directions in which log-odds would move if that model were played with probability one. The shaded region is the convex hull $\mathrm{conv}\{d(a_{1}),d(a_{2}),d(a_{3})\}$ , representing all possible mean drift directions achievable by mixing over models. We list out 4 representative cases: (a) Interior equilibrium with self-confirming models. (b) Interior equilibrium with self-defeating models. (c) No equilibria: drift towards face interior. (d) No equilibria: drift towards vertex.

Category	Defining condition	Limiting law	2-arm case
Interior Ergodic	$0\in\mathrm{int}\,\mathrm{conv}\{d(a_{j})\}$ + First/Second Ergodic Condition	Unique invariant $\mu_{\mathrm{int}}$ on $\mathrm{int}(\Delta_{M})$	Self-defeating
Vertex Selection	$0\in\mathrm{conv}\{d(a_{j})\}$ + First/Second Transient Condition	Mixture $\sum_{v\in V_{\star}}W_{v}\,\delta_{e_{v}}$	Self-confirming
Uniform Dominance	$0\notin\mathrm{conv}\{d(a_{j})\}$ for all subfaces, recursion ends at a single vertex $e_{m^{\star}}$	Point mass $\delta_{e_{m^{\star}}}$	Uniform dominance
Face-Ergodic	Push to some face $\Delta_{I}$ (either $0\notin\mathrm{conv}$ or interior unstable); on $\Delta_{I}$ : First/Second Ergodic Condition. Models $j\notin I$ are uniformly dominated .	Unique invariant $\mu_{I}$ on $\Delta_{I}$	Self-defeating (on the face) $+$ uniform dominance
Nested/Mixed Outcomes	General recursive reduction: irrespective of interior fixed point existence, repeated tests on faces either (i) yield ergodic faces or (ii) descend to vertices.	Mixture $\displaystyle\sum_{v\in V_{\star}}W_{v}\,\delta_{e_{v}}+\sum_{I\in\mathcal{F}_{\star}}W_{I}\,\mu_{I}$	Self-confirming $+$ self-defeating $+$ uniform dominance

Table 3: Multi-model classification of TS under misspecification.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Limiting action probabilities	Limiting average expected regret
Interior Ergodic	$\alpha_{\mathrm{int}}(a)=\int p_{a}(\pi)\,\mu_{\mathrm{int}}(d\pi)$	$\displaystyle\sum_{a}\alpha_{\mathrm{int}}(a)\,[g(a^{\star})-g(a)]$
Vertex Selection	$\alpha(a)=\sum_{v\in V_{\star}}W_{v}\,\mathbf{1}\{\phi(\theta^{(v)})=a\}$	$\displaystyle\sum_{v\in V_{\star}}W_{v}\,[g(a^{\star})-g(\phi(\theta^{(v)}))]$
Uniform Dominance	$\alpha(a)=\mathbf{1}\{\phi(\theta^{(m^{\star})})=a\}$	$g(a^{\star})-g(\phi(\theta^{(m^{\star})}))$
Face-Ergodic	$\alpha_{I}(a)=\int_{\Delta_{I}}p_{a}(\pi)\,\mu_{I}(d\pi)$	$\displaystyle\sum_{a}\alpha_{I}(a)\,[g(a^{\star})-g(a)]$
Nested/Mixed Outcomes	$\alpha(a)=\sum_{v}W_{v}\mathbf{1}\{\phi(\theta^{(v)})=a\}+\sum_{I}W_{I}\int_{\Delta_{I}}p_{a}(\pi)\,\mu_{I}(d\pi)$	$\displaystyle\sum_{v}W_{v}\,r(\phi(\theta^{(v)}))+\sum_{I}W_{I}\sum_{a}\Big(\int_{\Delta_{I}}p_{a}(\pi)\,\mu_{I}(d\pi)\Big)r(a)$

Appendix

Table 4: Limiting action distributions and average expected regret across multi-model categories. Here

p_{a}(\pi)=\sum_{j}\pi_{j}\,\mathbf{1}\{\phi(\theta^{(j)})=a\}

a^{\star}=\arg\max_{a}g(a)

r(a)=g(a^{\star})-g(a)

is the expected per-period regret,

V_{\star}

are terminal vertices, and

\mathcal{F}_{\star}

are ergodic terminal faces.

Panel of Figure 5 — Figure 5: Agreement case simulation results.

Panel of Figure 6 — Figure 6: Self-confirming case simulation results.

Panel of Figure 7 — Figure 7: Uniform dominance case simulation results.

Panel of Figure 8 — Figure 8: Self-defeating case simulation results.

Explore similar work

Aug 8, 2018

Bayesian Nonparametric Thompson Sampling for Multi-Armed Bandits

In stochastic multi-armed bandits, rewards are generated by an unknown distribution; the world's reward uncertainty induces the exploration-exploitation trade-off. Thompson sampling (TS) addresses this trade-off elegantly, but requires knowing the reward model's parametric form to sample from its posterior and compute per-arm expected rewards ---which limits its applicability as it demands per-scenario model specification and hyperparameter tuning. We here adopt Bayesian nonparametric (BNP) mixture models to remove this requirement, extending TS to arbitrary reward distributions. We model each arm's unknown reward distribution with an independent BNP Gaussian mixture, which can approximate any continuous reward density, and adapts its complexity to sequentially observed data. Embedding the BNP mixture model into TS yields Nonparametric TS, an algorithm that sequentially learns the reward model that best approximates the true, yet unknown, per-arm reward distribution, avoiding case-by-case model specification and hyperparameter tuning. We empirically demonstrate that (

i

) Nonparametric TS matches the regret performance of Oracle TS policies that know the true reward model; (

ii

) it favorably compares (in cumulative regret and volatility) to a suite of state-of-the-art alternatives across multimodal, outlier-contaminated, and non-exponential-family reward distributions; and (

iii

) its Bayesian nonparametric inference remains computationally practical and accurate for regret minimization.

Iñigo Urteaga, Chris H. Wiggins

Department of Applied Physics and Applied Mathematics Data Science Institute Columbia University New York City, NY 10027

Multi-Armed Bandits Thompson Sampling Bayesian Mixture Models

Ask PDF

Jul 14, 2026

Thompson Sampling Is 2-Competitive for Mistakes

We consider Bayesian bandit models and prove that Thompson sampling makes at most twice the expected number of mistakes (selections of a suboptimal arm) as any other policy. Our analysis applies as long as the latent arm processes are independent and each arm evolves only when played. For stochastic bandits with best arm defined via mean reward, this confirms a conjecture of Guha and Munagala from 2014, where the factor

2

is already best possible. The result holds under any nonincreasing sequence of round weights, including fixed horizon and geometric discounting.

Mark Sellke, Gregory Valiant

Thompson Sampling Multi-Armed Bandits

Ask PDF

Feb 20, 2026

Bayesian Online Model Selection

Online model selection in Bayesian bandits raises a fundamental exploration challenge: When an environment instance is sampled from a prior distribution, how can we design an adaptive strategy that explores multiple bandit learners and competes with the best one in hindsight? We address this problem by introducing a new Bayesian algorithm for online model selection in stochastic bandits. We prove an oracle-style guarantee of

O\left( d^* M \sqrt{T} + \sqrt{(MT)} \right)

on the Bayesian regret, where

M

is the number of base learners,

d^*

is the regret coefficient of the optimal base learner, and

T

is the time horizon. We also validate our method empirically across a range of stochastic bandit settings, demonstrating performance that is competitive with the best base learner. Additionally, we study the effect of sharing data among base learners and its role in mitigating prior mis-specification.

Aida Afshar, Yuke Zhang, Aldo Pacchiano

Boston University · Broad Institute of MIT and Harvard

Model Selection Multi-Armed Bandits

Ask PDF