A Survey of Reinforcement Learning For Economics

Pranjal Rawat

	Training (parameters updated)	Execution (parameters frozen)
Simulator	$\cdot$ AlphaGo Zero self-play $\cdot$ RL solver in structural estimation $\cdot$ Simulation tuning for bandit algorithms	$\cdot$ Policy benchmarks on synthetic environments
Historical data	$\cdot$ Reward model from human comparisons $\cdot$ Causal off-policy evaluation $\cdot$ Policy learning from logged bandit data	$\cdot$ Off-policy evaluation of a target policy
Live market (“in-field”)	$\cdot$ Bandit pricing experiments $\cdot$ Hotel revenue-management Q-learning	$\cdot$ DiDi dispatch with fixed parameters $\cdot$ AlphaGo vs. Lee Sedol

Table 1: The reinforcement learning lifecycle grid. Most RL research operates in the top-left cell. Readers from the inference tradition typically picture the bottom-left when they hear “online.”

RL Term	Symbol	Inference Tradition	Symbol
State	$s\in\mathcal{S}$	State or sufficient history	$x_{t}$ , $\bar{H}_{t}$
Action	$a\in\mathcal{A}$	Choice, control variable	$d_{t}$ , $u_{t}$ 16 16 16 The letter $u$ appears twice in the inference-tradition column with different meanings: $u_{t}$ denotes the control variable (action), while $u(x,d)$ denotes per-period utility (reward). Context usually disambiguates, but readers should note the collision.
Reward	$r(s,a)$	Per-period utility, payoff	$u(x,d)$
Discount factor	$\gamma$ 17 17 17 RL permits $\gamma=0$ (myopic, one-step) and $\gamma=1$ (undiscounted, episodic tasks with guaranteed termination). The inference tradition requires $\beta\in(0,1)$ strictly; the contraction mapping argument for the Bellman operator relies on $\beta<1$ .	Discount factor	$\beta$
Target policy	$\pi(a\mid s)$	Decision rule, treatment regime	$d(h)$
Behavior policy	$\mu(a\mid s)$	Assignment propensity, CCP	$e(a\mid h)$

Table 2: Notation mapping between reinforcement learning and the inference tradition.

Figure 1: The Engine Replacement MDP as a labelled directed graph. Each arc carries its action and transition probability and, where the action is first named from that state, its reward. Blue arcs are the keep action and orange arcs the replace action.

reinforcement learning	object	economic model
mileage grade $s$	state	observed condition $x$
keep or replace $a$	action	discrete choice $d$
$P(s^{\prime}\mid s,a)$	transition	controlled law of motion
$r(s,a)$	reward	flow profit net of maintenance
$\pi(a\mid s)$	policy	replacement rule
$Q^{\pi}(s,a)$	action value	choice-specific value

Table 3: The Engine Replacement MDP in the vocabularies of reinforcement learning and dynamic economic models.

Figure 2: Architecture comparison of three core algorithm families: value-based, policy gradient, and actor-critic. (a) DQN maps states to Q-values for all actions, selecting the argmax. (b) REINFORCE maps states to a probability distribution over actions, then samples. (c) Actor-Critic maintains separate policy and value networks; the critic’s TD error

\delta_{t}

feeds back into both during training, yielding lower-variance policy updates than REINFORCE’s Monte-Carlo return. Each panel includes the environment feedback loop in which the agent’s action produces a reward and next state.

Figure 3: The control-as-inference graphical model. States

s_{t}

and actions

a_{t}

jointly determine the next state

s_{t+1}

(dynamics) and the optimality variable

\mathcal{O}_{t}

(reward signal). Optimality nodes

\mathcal{O}_{t}

are observed as

\mathcal{O}_{t}=1

; the posterior over actions

P(a_{t}\mid s_{t},\mathcal{O}_{t:T}=1)

yields the optimal policy.

Figure 4: The four phases of a single MCTS simulation in AlphaGo Zero. (a) Selection traverses the tree from the root, choosing at each node the action maximizing a UCB-like score balancing exploitation (

Q

) and exploration (

P/N

). (b) Expansion adds a new leaf node when the traversal reaches an unexplored position. (c) The neural network

f_{\theta}

evaluates the new position, producing move priors

\mathbf{p}

and a value estimate

v

. (d) Backup propagates

v

along the traversed path, updating mean values

Q(s,a)

and visit counts

N(s,a)

at each edge.

algorithm	family	$\hat{V}(\text{low})$	$\hat{V}(\text{high})$	$\\|\hat{V}-V^{\star}\\|_{\infty}$
MC	prediction, 2000 episodes	5.3416 (0.0059)	4.3078 (0.0039)	0.0136 (0.0040)
Q-learning	off-policy control, 50000 steps	5.3296 (0.0076)	4.2928 (0.0064)	0.0216 (0.0064)
FQI	batch VI, 20000 transitions	5.3571 (0.0105)	4.3214 (0.0095)	0.0292 (0.0058)
SARSA	on-policy control, 50000 steps	5.3122 (0.0087)	4.2754 (0.0081)	0.0389 (0.0065)
TD(0)	prediction, 20000 steps	5.3172 (0.0124)	4.2874 (0.0118)	0.0413 (0.0073)
REINFORCE	policy gradient, 5000 episodes	5.2967 (0.0038)	4.2594 (0.0045)	0.0510 (0.0045)

Table 4: Eight algorithm families on the Engine Replacement MDP, 10 seeds each, ranked by mean sup-norm distance from

V^{\star}=(5.3448,\ 4.3103)

. Standard errors across seeds in parentheses. Rows carry different data budgets and three scoring modes. Prediction rows evaluate

\pi^{\star}

; control rows learn from exploratory behavior and report

\max_{a}\hat{Q}

; policy-gradient rows report the exact value of the final learned policy.

Figure 5: The Brock–Mirman economy (

\alpha=0.36

,

\beta=0.96

, 1,000 states). (a) Value iteration on a scalar Bellman equation. The staircase iterates

V_{k+1}=TV_{k}

, converging at the linear rate

\gamma

. (b) Policy iteration as Newton’s method. Each step solves for the fixed point of the active policy operator

T^{\pi_{k}}

, jumping to the tangent line’s intersection with the diagonal. (c) Sup-norm error

\|V_{k}-V^{*}\|_{\infty}

for the discretized model; VI requires 567 iterations, PI converges in 11.

Regime	Method	Iterations	Time (s)	Final error
1. Contraction	VI	567	30.55	9.7e-11
	PI	11	0.76	—
2. LP dual	VI	—	0.007	—
	LP	—	0.017	2.3e-09
3. Rate ( $n_{k}$ =10)	VI	567	0.003	—
	PI	7	0.000	—

Table 5: Brock–Mirman Economy: VI vs PI vs LP. Wall-clock times are medians of 5 independent runs; iteration counts are deterministic.

policy	$V(\text{low})$	$V(\text{high})$
$(\text{keep},\text{replace})$	5.3448	4.3103
$(\text{keep},\text{keep})$	3.4545	2.0000
$(\text{replace},\text{keep})$	-5.0000	2.0000
$(\text{replace},\text{replace})$	-5.0000	-5.0000

Table 6: The four deterministic-policy values of the Engine Replacement MDP, exact resolvent solves listed by

V(\text{low})

. Both iteration paths start at

V^{(\text{replace},\text{replace})}

.

Figure 6: Value space of the Engine Replacement MDP,

x=V(\text{low})

,

y=V(\text{high})

. (a) The value polytope (shaded), its convex hull (dashed), the four deterministic-policy values (dots), the midpoint witness (cross) on the chord between

V^{(\text{keep},\text{replace})}

and

V^{(\text{replace},\text{keep})}

(dotted), and the paths of value iteration (circles) and policy iteration (squares) from

V^{(\text{replace},\text{replace})}

. (b) The greedy partition, three parallel cells separated by the lines

x-y=0.7778

and

x-y=3.3333

, with the polytope boundary and

V^{*}

overlaid. (c) The sup-norm error

\|V_{k}-V^{*}\|_{\infty}

against iteration on a log scale, with the contraction bound

\gamma^{k}\|V_{0}-V^{*}\|_{\infty}

(dashed); policy iteration’s last point sits at machine precision and the axis floor is clipped at

10^{-15}

.

Step	Policy Iteration	AlphaZero
Initialize	Arbitrary $\pi_{0}$	Random network $f_{\theta}$
Evaluate	Solve $V=T^{\pi_{k}}V$ exactly	Network value head $v\approx V^{\pi_{k}}$
Improve	$\pi_{k+1}=\mathop{\it argmax}_{a}\{r+\gamma PV^{\pi_{k}}\}$	MCTS simulations from $v$
Iterate	Repeat until $\pi$ is stationary	Retrain $f_{\theta}$ on self-play outcomes

Table 7: Policy iteration and AlphaZero follow the same evaluate-improve loop. The network provides approximate policy evaluation; MCTS provides approximate policy improvement via selective tree search.

Figure 7: RMSVE vs. episodes for TD(

\lambda

) on the 20-state corridor. Shaded regions show

\pm 1

SE over 20 seeds.

$\lambda$	Final RMSVE	Episodes to RMSVE $<0.05$
0.0	0.4012 $\pm$ 0.0056	$>200$
0.4	0.1902 $\pm$ 0.0028	$>200$
0.8	0.0108 $\pm$ 0.0002	141 $\pm$ 1
1.0	0.0000 $\pm$ 0.0000	52 $\pm$ 0

Table 8: TD(

\lambda

) on 20-state corridor. Mean

\pm

SE over 20 seeds, 200 episodes,

\gamma=0.99

,

\alpha=0.05

.

Method	Iterations	Error vs $V^{*}$	$P$ (recovered)	Key coefficient
Exact VI (discrete)	25	1.12e-03	—	—
FVI	9	3.23e-04	1.1294	$\hat{\theta}_{V}^{x^{2}}=-1.1294$
FQI	10	9.37e-05	1.2682	$\hat{\theta}_{Q}^{xu}=-1.0729$
DQN ( $2\times 64$ ReLU, 10 seeds)	100000	7.16e-01 $\pm$ 6.80e-02	—	—
Analytical ( $V^{*}=-Px^{2}$ )	—	0	1.1294	$c_{xu}=-1.0729$

Table 9: Fitted weights and convergence metrics for FVI, FQI, and DQN on linear-quadratic control. The FVI

x^{2}

coefficient recovers the Riccati solution

P\approx 1.129

. The FQI quadratic coefficients match the analytical

Q^{*}

to four decimal places. FVI and FQI achieve max error below

10^{-3}

against the analytical

V^{*}

; the DQN row reports mean

\pm

standard error across ten independent seeds after 100,000 gradient steps with no feature basis specified. FVI and FQI exercise the full deterministic grid with the true transition kernel, so seed-to-seed variation is exactly zero and no standard error is reported for them.

Figure 8: LQC convergence of FVI and FQI (left), DQN learning curve (middle, mean and

\pm 1

standard-error band over ten seeds), and value function recovery for all three methods (right; the DQN trace is the cross-seed mean). FVI and FQI reduce

\|V_{k}-V^{*}\|_{\infty}

to near-zero in under 10 iterations, exploiting the known polynomial structure of

Q^{*}

. DQN’s learning curve over 100,000 gradient steps is shown with no feature basis specified.

Method	$\\|V-V^{*}\\|_{\infty}$	$\\|V-V^{*}\\|_{\text{rms}}$	Policy agreement (%)	Iterations
Exact VI	0.0000	0.0000	98.0	—
FVI (linear)	0.0010	0.0008	99.0	341
FQI (linear)	1.6521	1.4805	4.0	339
Oracle-FQI	0.0000	0.0000	98.0	341
NLLS-FQI ( $\hat{\alpha}=0.3600$ )	0.0000	0.0000	98.0	341

Table 10: Convergence metrics for five methods on the Brock–Mirman economy (

N_{K}=50

,

N_{Z}=2

,

\gamma=0.96

). Policy agreement is measured against the closed-form optimal policy

k^{\prime}=\alpha\beta zk^{\alpha}

.

Figure 9: Convergence of

\|V_{k}-V^{*}\|_{\infty}

for FVI, linear FQI, Oracle-FQI, and NLLS-FQI on the Brock–Mirman economy.

Figure 10: Bus engine replacement benchmark. Left: computation time vs. fleet size (log scale). Right: discounted return vs. fleet size for DP, DQN, and heuristic baselines; DQN error bars are mean

\pm

standard error over ten seeds. At

N=6

(46,656 states), DP is omitted because plain-Python value iteration becomes wall-clock prohibitive at this scale.

$N$	$\|\mathcal{S}\|$	DP Time	DP Return	DQN Return	Thresh(3)	Never Replace
1	6	$0.71$ s	$-54.8$	$-54.8\pm 0.00$	$-55.6$	$-93.3$
2	36	$0.06$ s	$-110.0$	$-110.0\pm 0.00$	$-111.8$	$-187.5$
3	216	$0.54$ s	$-165.1$	$-165.1\pm 0.01$	$-167.7$	$-281.4$
4	1,296	$6.61$ s	$-220.4$	$-220.4\pm 0.00$	$-223.8$	$-375.6$
5	7,776	$77.82$ s	$-276.7$	$-276.9\pm 0.02$	$-281.1$	$-471.3$
6	46,656	—	—	$-334.7\pm 1.86$	$-336.7$	$-563.8$

Table 11: Bus engine replacement benchmark across fleet sizes

N

. DP time is wall-clock for value iteration; returns are discounted returns on the shared evaluation states, with DQN reported as mean

\pm

standard error over ten seeds.

Figure 11: Tabular Q-learning on the Engine Replacement MDP over ten fixed seeds, with step size

\alpha_{n}=n^{-0.6}

per state-action pair and

40{,}000

synchronous sweeps. The line is the mean sup-norm error and the band is one standard error.

quantity	value
exact target $(FQ_{5})(\text{low},\text{keep})$	2.8268
Monte Carlo mean, $N=1,000,000$	2.8268 $\pm$ 0.0001
mean absolute sampling error, $N=1,000,000$	0.0004
Q-learning final error, mean $\pm$ SE	0.0312 $\pm$ 0.0027
projected TD( $\lambda=0$ ) norm, bound 0.9000	0.5196
projected TD( $\lambda=0.5$ ) norm, bound 0.8182	0.3231

Table 12: Value-learning quantities computed on the Engine Replacement MDP. The Bellman-operator identity and Monte Carlo estimate use the intermediate table

Q_{5}

at

(s,a)=(\text{low},\text{keep})

, the engine’s one stochastic transition. The Q-learning trace uses 10 fixed seeds and 40,000 synchronous sweeps with step size

\alpha_{n}=n^{-0.6}

. Projected TD(

\lambda

) moduli use

\phi=(1,-1)

and the stationary distribution of

P^{\pi^{\star}}

. Concentrability uses the discounted occupancy from the low-mileage start.

Figure 12: Geometric intuition for the projection-norm leg of the deadly triad in

\mathbb{R}^{2}

. The gray line is the function approximation subspace span(

\Phi

); the blue arrow is

TV

, the Bellman update. (a) On-policy: the orthogonal projection

\Pi_{d^{\pi}}

(under the target-policy stationary distribution) drops

TV

perpendicularly onto the subspace, preserving the contraction. (b) Off-policy: the oblique projection

\Pi_{\mu}

(under the behavior distribution) reaches a point further from the origin than

TV

itself, so the projection can expand distances. Bootstrapping and off-policy sampling, the other two legs of the triad, are not depicted; their joint effect on divergence is exhibited numerically in Baird’s six-state star counterexample ( Baird, 1995 ) . The displayed expansion ratio is illustrative; the subspace and projection-direction angles are chosen so contraction in (a) and expansion in (b) are visible at figure scale rather than derived from a specific MDP.

Figure 13: Information geometry of the natural policy gradient. Left : schematic of the policy manifold

\mathcal{M}

with Euclidean gradient

\nabla_{\theta}J

(red) and natural gradient

F^{-1}\nabla_{\theta}J

(green) from the current iterate

\pi_{\theta_{\mathrm{old}}}

toward the optimal policy

\pi^{*}

. The two arrows are placed schematically to convey the convergence advantage of NPG; quantitative behavior depends on the loss landscape and is shown in the right panel. Right : tangent plane at

\theta_{\mathrm{old}}

showing the Euclidean unit ball

\|\Delta\theta\|_{2}\leq\varepsilon

(red) and the KL unit ball

\Delta\theta^{\top}F\Delta\theta\leq\delta

(green), with the respective steepest-ascent directions. The Euclidean step (red arrow) leaves the KL ball, violating the trust-region constraint; the natural step (green arrow) saturates the KL budget by construction.

Figure 14: Schematic of the majorize-minimize step underlying TRPO: a surrogate

L(\theta)

lower-bounds the true objective

J(\theta)

and is monotonically improved by maximizing

L

. Left : the surrogate (dashed) is tight at

\theta_{\mathrm{old}}

; the trust region (shaded) constrains the step. The gap between

L(\theta_{\mathrm{new}})

and

J(\theta_{\mathrm{new}})

is the guaranteed improvement. Right : iterative MM convergence from

\theta_{0}

through four surrogates (dashed, colored by iteration) to

\theta^{*}

. 132 132 132 The quadratic majorizer here uses a fitted constant

c

chosen so the surrogate touches the true objective at

\theta_{\mathrm{old}}

and remains below it on the visible grid. Schulman et al. (2015) ’s theoretical constant

C=4\varepsilon\gamma/(1-\gamma)^{2}

is so conservative the resulting step is unusably small in practice; modern TRPO implementations replace the penalty with a hard KL constraint. This illustrative cartoon visualizes the form of the MM bound, not its sharpness.

Figure 15: Trust region methods in the LQC monetary policy setting. A central bank learns a Taylor rule

u_{t}=-(\theta_{1}x_{1t}+\theta_{2}x_{2t})

mapping output gap

x_{1}

and inflation gap

x_{2}

to an interest rate instrument. Left : policy contour lines in state space for the current iterate

\theta_{\mathrm{old}}

, optimal weights

\theta^{*}

, and an illustrative KL-violating iterate

\theta_{\mathrm{bad}}

(constructed as a

3\times

scaling of the TRPO step, not a literal unconstrained gradient step), with phase arrows showing closed-loop dynamics under

\theta_{\mathrm{old}}

. Center : expected return

J(\theta_{1},\theta_{2})

in parameter space; hatching marks the unstable region. The KL trust region ellipse bounds the TRPO step;

\theta_{\mathrm{bad}}

lands in the unstable region, illustrating the consequence of violating the KL constraint. Right : TRPO feasible region (KL ellipse) and PPO feasible region (50% ratio-clip band over 200 sampled state-action pairs) overlaid on

J(\theta)

, with the respective constrained steps marked.

	low	high
occupancy $\rho(s)$	7.0968	2.9032
Fisher diagonal $\rho(s)\pi_{k}(s)\pi_{r}(s)$	1.7742	0.7258
gradient $\partial J/\partial\theta_{s}$	-2.2492	-0.1709
action-value gap $Q(s,\text{r})-Q(s,\text{k})$	-1.2677	-0.2355
natural gradient $F^{-1}\nabla J$	-1.2677	-0.2355
policy-gradient iterations to $J^{\star}-J<10^{-3}$	not reached in 300

Table 13: Policy-gradient objects on the Engine Replacement MDP at the uniform policy

\theta=(0,0)

, start distribution

\nu=\delta_{\text{low}}

. The Fisher matrix is diagonal under the one-logit-per-state parameterization; its inverse is taken entrywise, never a pseudo-inverse. Convergence counts iterations until

J^{\star}-J<10^{-3}

at step size 0.5.

Figure 16: The policy square of the Engine Replacement MDP. Contours show

J(\pi)=V^{\pi}(\text{low})

over

(\pi(\text{replace}\mid\text{low}),\ \pi(\text{replace}\mid\text{high}))

. Overlaid are vanilla and natural gradient ascent from the uniform policy (300 steps each, step size

0.5

), a single natural-gradient step at

\alpha=\tfrac{1}{2},1,2,\ldots,32

(dots), the two policy-iteration jumps (dashed), and the optimal vertex (star).

Model	$d$	$n$	$\|\mathcal{A}\|$	$\|\mathcal{S}\|$	Value table	One sweep
Bus engine, one bus ( Rust, 1987 )	1	175	2	175	1.4 kB	61 $\mu$ s
Wind farm, base state	3	7	11	343	2.7 kB	1 ms
Brock–Mirman growth	1	500	500	1,000	8.0 kB	500 ms
Wind farm, six state variables	6	7	11	$1.2\times 10^{5}$	941 kB	3 min
Inventory, five products	5	20	7,776	$3.2\times 10^{6}$	26 MB	922 days
Bus engine, fleet of 3	3	175	8	$5.4\times 10^{6}$	43 MB	3 days

Table 14: Cost of one exact value-iteration sweep on a uniform grid, for models used elsewhere in this survey. States are

n^{d}

over

d

state variables at

n

points each, one sweep costs

n^{2d}|\mathcal{A}|

operations, and wall-clock assumes

10^{9}

operations per second. Memory is one float64 value per state. Rows ordered by state count.

$d_{s}$	$d_{a}$	$\gamma$	$2d_{s}+d_{a}$	Mixing	General	Ratio
1	1	0.9	3	$10^{9}$	$10^{12}$	1,000
1	1	0.95	3	$8.0\times 10^{9}$	$6.4\times 10^{13}$	8,000
1	1	0.99	3	$10^{12}$	$10^{18}$	$10^{6}$
2	1	0.9	5	$10^{15}$	$10^{20}$	$10^{5}$
2	1	0.95	5	$3.2\times 10^{16}$	$10^{23}$	$3.2\times 10^{6}$
2	1	0.99	5	$10^{20}$	$10^{30}$	$10^{10}$

Table 15: Oracle queries required by Theorem 10 at

\epsilon=0.01

. The mixing column is ( 66 ) and the general column is ( 67 ); both are tight, so the gap between them is a real difference in problem difficulty and not slack in the analysis. The last column is their ratio,

(1-\gamma)^{-(2d_{s}+d_{a})}

, the price of losing the mixing condition.

Figure 17: Operation and sample counts against ambient state dimension, on a log scale, at

\epsilon=0.01

. Exact grid dynamic programming and the Chow-Tsitsiklis lower bound of Theorem 10 grow exponentially in

d

; the bilinear and factored bounds of Theorems 11 and 13 are constant in

d

at fixed structural parameters. Dashed lines mark one second, one day and one year at

10^{9}

operations per second. The Chow-Tsitsiklis curve leaves the panel because it bounds a harder class, continuous states and actions to a fixed supremum-norm accuracy.

$d$	$H$	$\epsilon$	$H^{7}$	Trajectories	Environment steps
5	10	0.1	$10^{7}$	$6.4\times 10^{14}$	$6.4\times 10^{15}$
10	24	0.1	$4.6\times 10^{9}$	$2.2\times 10^{18}$	$5.2\times 10^{19}$
10	24	0.05	$4.6\times 10^{9}$	$9.8\times 10^{18}$	$2.3\times 10^{20}$
20	50	0.1	$7.8\times 10^{11}$	$2.3\times 10^{21}$	$1.2\times 10^{23}$

Table 16: Trajectories required by ( 68 ), with the absolute constant set to one, at hypothesis-class cardinality

|\mathcal{H}|=10^{6}

, failure probability

\delta=0.05

and

B_{X}=B_{W}=1

. The horizon column isolates the factor

H^{7}

. Environment steps are trajectories times

H

. The inherent dimension

d

is the rank of the bilinear factorization, not the ambient state dimension.

		$\alpha=2$		$\alpha=8$
$d$	Required $\alpha$	Exponent	Width $m$	Exponent	Width $m$
1	0.25	0.733	44	0.686	2
2	0.50	0.778	461	0.704	7
3	0.75	0.810	2,207	0.719	19
5	1.25	0.852	$1.7\times 10^{4}$	0.746	88
10	2.50	n/a	n/a	0.795	1,079

Table 17: What Theorem 12 asks of the problem, at Besov index

p=2

, benign exploration

K=1

and

T=10^{6}

episodes. The admissibility condition

\alpha>d(1/p-1/4)_{+}

demands a minimum smoothness that grows linearly in the ambient dimension, reported in the second column. Each smoothness level then gives the regret exponent and the width ( 69 ) the theorem prescribes; n/a marks a level that fails the condition, where the theorem gives no guarantee at all. Regret is sublinear in every admissible cell, but the exponent approaches one as the smoothness-to-dimension ratio falls.

(a) Scope sizes against the full state-action count
Parent-set size $k$	One scope	Scope sum
1	20	2,000
2	200	$2.0\times 10^{4}$
3	2,000	$2.0\times 10^{5}$
4	$2.0\times 10^{4}$	$2.0\times 10^{6}$
6	$2.0\times 10^{6}$	$2.0\times 10^{8}$

Table 18: Theorem 13 in two parts, for a system of 100 components with 10 states each and

|\mathcal{A}|=2

, so the unstructured problem has

|\mathcal{S}||\mathcal{A}|=2.0\times 10^{100}

. Panel (a) is what the factorization buys: the scope sum that enters ( 75 ) in place of

|\mathcal{S}||\mathcal{A}|

, as the maximum parent-set size

k

grows. Panel (b) is what it costs: the bias floor

\mathcal{E}_{\omega}

of ( 73 ) at

\Delta_{R}=0

, which bounds accuracy from below however many samples are drawn. Cells exceeding a target

\epsilon=0.01

are the settings where the target is unreachable.

Figure 18: Computation time versus state dimension on a log scale. Circles trace completed tabular DP runs and crosses mark runs stopped at the 10 minute budget (dashed reference line); the dotted line extrapolates the exponential fit through the completed DP runs. RL curves show mean training time per seed over 10 seeds.

Method	$d=3$	$d=4$	$d=5$	$d=6$
Tabular DP	1110	1113	TIMEOUT	TIMEOUT
DQN	1108 $\pm$ 0.4	1107 $\pm$ 0.7	1107 $\pm$ 0.2	1100 $\pm$ 0.4
Factored RL	1091 $\pm$ 4.2	1083 $\pm$ 4.1	1094 $\pm$ 1.8	1084 $\pm$ 4.9
Linear AC	1002 $\pm$ 29.4	1016 $\pm$ 33.9	1023 $\pm$ 25.4	1009 $\pm$ 30.7

Table 19: Average return per episode (

/day) by method and state dimension. RL cells are mean

\pm

standard error over 10 training seeds, each evaluated over 50 episodes with common random shocks; the DP row is a single deterministic solve, with TIMEOUT marking dimensions where backward induction exceeded its 10 minute budget. Rows ordered by mean return at

d=3$ .

	low	high
network value $v=V^{(\mathrm{unif})}$	1.3387	0.8226
searched policy value $V^{\hat{\pi}}$	3.4545	2.0000
optimal value $V^{\star}$	5.3448	4.3103
	value
$\varepsilon=\\|v-V^{\star}\\|_{\infty}$	4.0061
actual loss $\\|V^{\star}-V^{\hat{\pi}}\\|_{\infty}$	2.3103

Table 20: A one-step evaluate-improve proxy on the Engine Replacement MDP. The approximate value

v

is the uniform policy’s exact value; one greedy lookahead lands on the

(\text{keep},\text{keep})

vertex of Figure 16 . The bound is Theorem 9 at

\gamma=0.9

.

state	action	in data	$Q^{\pi}$	$\widehat{Q}$	$\|\widehat{Q}-Q^{\pi}\|$
low	keep	yes	5.3448	13.5266	8.1818
low	replace	yes	4.3103	11.6740	7.3636
high	keep	yes	4.0793	13.0793	9.0000
high	replace	no	4.3103	14.3103	10.0000
maximum observed-pair residual			1.8e-15
$\sigma$	Gaussian maximum bias		bias divided by the $0.2310$ gap

Table 21: Bellman residuals and Gaussian maximum bias in the Engine Replacement MDP. The target policy keeps at low mileage and replaces at high mileage. The data omit the high-mileage replacement pair.

Method	BE on $\mathcal{D}$	VE	VE/BE	Policy agree.
BRM (mean $\pm$ SE)	$(1.39\pm 0.42)\times 10^{-4}$	$7.380\pm 0.064$	$(1.24\pm 0.68)\times 10^{6}$	$7.2\%\pm 0.3\%$
BRM (median)	$1.04\times 10^{-4}$	$7.364$	$72{,}306$	—
FQE (mean $\pm$ SE)	$0.2087\pm 0.0394$	$7.310\pm 0.107$	$44.4\pm 6.8$	$6.0\%\pm 0.0\%$
FQE (median)	$0.1911$	$7.237$	$37.2$	—
OLS	OOS MSE $=0.0959$	OOS $R^{2}=0.112$	$r(\mathrm{MSE},R^{2})=-1.000$	—

Table 22: Bellman error on dataset

\mathcal{D}

and value error on the full MDP for BRM and FQE trained on offline Brock–Mirman data, with

n=10

seeds per method. BE is mean squared Bellman error evaluated with the current network on both sides; VE is mean absolute deviation from

Q^{*}

over all 4,795 feasible state-action pairs; SE is the standard error of the mean across seeds. Policy agreement is the fraction of states where

\arg\max_{a}Q_{\theta}(s,a)=\pi^{*}(s)

. The OLS row reports a single fixed-test-set fit with

n=2{,}000

; on that test set

r(\mathrm{MSE},R^{2})=-1

is an algebraic identity (see prose).

Figure 19: Left: OLS regression of log consumption on log capital and log productivity, estimated on expanding windows from the Brock–Mirman optimal policy. Out-of-sample MSE (left axis, red) and out-of-sample

R^{2}

(right axis, blue) track each other with Pearson

r=-1.000

. Right: BRM (orange) and FQE (blue) trained on offline Brock–Mirman data

\mathcal{D}

(

T=2{,}000

transitions, 0.2% state-space coverage); note the log scale on the

y

-axis. Solid lines show Bellman error on

\mathcal{D}

(current network both sides); dashed lines show mean absolute value error against

Q^{*}

over all 4,795 feasible state-action pairs; shaded bands are

\pm 1

SE over 10 seeds.

Figure 20: NFXP versus RL-based structural estimation. Top: the nested fixed-point algorithm evaluates the likelihood by solving the Bellman equation to convergence inside each optimizer step. Bottom: single-loop stochastic approximation updates structural parameters

\theta

and value/policy weights

\omega

simultaneously from data batches. The NFXP algorithm is due to Rust (1987) .

	high-mileage keep reward		replacement cost
method	population	finite sample	population	finite sample
NFXP	0.200000	0.1993 (0.0038)	0.500000	0.4943 (0.0060)
CCP	0.200000	0.1997 (0.0038)	0.500000	0.4934 (0.0059)
TD-CCP	0.200000	0.2024 (0.0039)	0.500000	0.4908 (0.0062)
generating value	0.200000		0.500000

Table 23: Structural parameter recovery in the

+\mathrm{EV}

Engine Replacement MDP. Population columns use exact choice and transition probabilities. Finite-sample columns report means and standard errors across 20 fixed seeds, with 9,000 transitions per seed.

				RC Bias		RC RMSE		$\theta_{1}$ RMSE		$\theta_{2}$ RMSE
Method	K	$\|S\|$	Time (s)	Mean	SE	Value	SE	Value	SE	Value	SE
NFXP	1	20	0.2	+0.053	0.027	0.097	0.015	0.307	0.055	0.495	0.111
CCP	1	20	0.1	+0.055	0.027	0.099	0.015	0.316	0.055	0.510	0.112
TD-CCP Linear	1	20	0.1	-0.087	0.017	0.101	0.016	0.255	0.041	0.296	0.044
TD-CCP Neural	1	20	38.4	+0.054	0.026	0.094	0.013	0.260	0.054	0.423	0.082
NFXP	2	400	0.3	+0.027	0.028	0.087	0.018	0.270	0.061	0.344	0.077

Table 24: DDC estimation results across state-space scales. Rows show method, number of components

K

, state-space size

|\mathcal{S}|

, mean wall-clock time (seconds), RC bias, RC RMSE, and root mean squared error for

\theta_{1}

and

\theta_{2}

. Standard errors in adjacent columns; means over 10 seeds, dashes indicate method failure (sparse state coverage).

Figure 21: Wall-clock estimation time versus state-space scale for the four DDC estimators. Vertical axis is log-scaled. Each point is the mean over 10 seeds.

Method	Mean return	SE	Policy MSE vs VFI	Wall clock
KPR	45.88	0.587	0.0004	$-$
VFI	45.86	0.589	0.0000	13.8 s
PPO	45.82	0.843	0.0087	32.9 s
DDPG	45.17	1.476	0.0365	579.4 s

Table 25: Representative-agent RBC under four methods. Mean episode return averaged over 30 evaluation episodes with shared initial conditions and shock paths. The SE column reflects cross-seed standard error of per-seed mean returns for the stochastic methods (PPO and DDPG,

10

seeds) and cross-episode standard error of returns within the single deterministic solution for KPR and VFI (

30

episodes); the two are not strict comparables but are reported in the same column for compactness. Policy mean-squared error reported against VFI consumption on 1000 random states drawn uniformly from the evaluation support. Wall-clock time is per seed for PPO and DDPG, and total solver time for VFI.

Figure 22: Mean deterministic-evaluation return as a function of normalised training progress for PPO (olive) and DDPG (cyan), shaded by cross-seed standard error across ten training seeds. Horizontal lines mark the VFI benchmark mean return (dashed black) and the KPR log-linear benchmark (dotted gray).

Method	LR	Exploitability	Expected return	Train time (s)
SPG	$10^{-2}$	$86.64\pm 16.29$	$-1021.3\pm 22.3$	$24.39\pm 0.06$
RSPG	$10^{-3}$	$60.37\pm 4.11$	$-1186.9\pm 10.1$	$28.79\pm 0.58$

Table 26: Partially observable Linear-Quadratic mean field game. Approximate exploitability is computed by a best-response dynamic program on the analytic mean-field path induced by the final policy. Expected return and training time are averaged over 10 training seeds, with standard errors across seeds. Learning rates are selected from the official MFAX grid

\{10^{-4},10^{-3},10^{-2}\}

by mean final exploitability for each method.

Figure 23: Official MFAX Linear-Quadratic grid. The left panel shows approximate exploitability over policy-gradient updates for the selected SPG and RSPG learning rates, shaded by cross-seed standard error across ten training seeds. The right panel shows final exploitability across the swept learning rates.

Algorithm	Rational	Convergent	Game class	Information
Indep. Q-learning	Yes	No	Any	Own reward
Minimax-Q	No	Yes	Zero-sum	Joint actions
Nash-Q	Cond.	Cond.	General-sum	All rewards
WoLF-PHC	Yes	Yes ( $2{\times}2$ )	General-sum	Own reward

Table 27: Multi-agent Q-learning algorithms: convergence and information requirements

Game	Algorithm	Action	Profit	$\|a-a^{*}\|$
Cournot	IQL	$2.95\pm 0.05$	$9.1\pm 0.0$	$0.05$
	Nash-Q	$2.89\pm 0.33$	$8.8\pm 1.3$	$0.17$
	WoLF-PHC	$3.00\pm 0.00$	$9.0\pm 0.0$	$0.00$
Bertrand	IQL	$4.00\pm 0.00$	$18.0\pm 0.0$	$0.00$
	Nash-Q	$4.00\pm 0.00$	$18.0\pm 0.0$	$0.00$
	WoLF-PHC	$3.95\pm 0.05$	$17.8\pm 0.2$	$0.05$

Table 28: Learned actions and profits in Cournot and Bertrand duopoly

Figure 24: Expected actions over training in Cournot and Bertrand duopoly. The left panel reports Cournot and the right panel reports Bertrand. Curves use 1,000-iteration smoothing windows and averages across 20 seeds.

Method	Exploitability at Iteration				Iter to $\varepsilon<0.01$	Time (s)
	100	500	1000	2000
CFR+	0.0216	0.0098	0.0099	0.0076	340	0.87 $\pm$ 0.03
Vanilla CFR	0.0466	0.0244	0.0130	0.0090	930	0.86 $\pm$ 0.03
Fictitious Play	0.3235	0.3313	0.3323	0.3328	$>$ 2000	0.68 $\pm$ 0.03

Table 29: Exploitability in Kuhn poker at selected iterations, first iteration below

\varepsilon=0.01

, and wall-clock time (mean

\pm

standard deviation over ten runs).

Figure 25: Exploitability versus iteration (log scale) for CFR+, vanilla CFR, and fictitious play in Kuhn poker. The dashed reference line marks

\varepsilon=0.01

.

Figure 26: No-commitment price paths

p_{t}

versus normalized period

t/T

for representative horizons

T\in\{10,50,200\}

at

\delta\in\{0.5,0.75,0.9,0.95,0.99\}

. The dashed line marks the commitment price

p=1/2

. The dotted line marks marginal cost

c=0

.

Figure 27: Coase collapse. The left panel reports terminal-period price

p_{T}

against

T

on a log-

T

axis. The right panel reports opening price

p_{1}

on the same axis. Each curve fixes

\delta

.

$T$	$\delta$	$V^{\mathrm{com}}$	$V^{\mathrm{nc}}$	$V^{\mathrm{nc}}/V^{\mathrm{com}}$	$p_{1}$	$p_{T}$
2	0.50	0.2500	0.2250	0.900	0.4500	0.3000
2	0.75	0.2500	0.2232	0.893	0.4464	0.3571
2	0.90	0.2500	0.2327	0.931	0.4654	0.4231
2	0.95	0.2500	0.2397	0.959	0.4793	0.4565
2	0.99	0.2500	0.2476	0.990	0.4952	0.4903
5	0.50	0.2500	0.2084	0.834	0.4168	0.0613

Table 30: Commitment value

V^{\mathrm{com}}

, no-commitment Markov-perfect value

V^{\mathrm{nc}}

, ratio

V^{\mathrm{nc}}/V^{\mathrm{com}}

, opening price

p_{1}

, and terminal price

p_{T}

across the

(T,\delta)

grid. Uniform valuations on

[0,1]

,

c=0

.

$\pi$	$P^{*}$	P(Screen)	$\|\Delta\|$	NashConv	SE	Eq. Type
0.10	150	0.000	0.000	4.04	0.59	Pooling
0.15	150	0.000	0.000	6.04	0.88	Pooling
0.20	150	0.000	0.000	8.04	1.17	Pooling
0.25	150	0.000	0.000	10.05	1.46	Pooling
0.30	150	0.001	0.001	12.05	1.75	Pooling
0.35	150	0.001	0.001	14.06	2.04	Pooling

Table 31: CFR strategy versus the analytical equilibrium in the

\pi

-sweep at

\delta=0.5

, with

n=10

seeds per cell.

Figure 28: Revenue curves

r(p)=r^{*}-k(p-p^{*})^{2}

for four demand curvatures

k\in\{0.5,1.0,2.0,3.5\}

. All revenue curves share the maximizer

p^{*}=5

; the underlying demand curves (not shown here, see Broder and Rusmevichientong (2012) Theorem 3.1) are linearly distinct but cross at

p^{*}

. Within the illustrative exploration window

|p-p^{*}|\leq 0.7

(shaded; half-width is pedagogical, not derived), revenue separation across the four curves is at most

1.72

units, about 7% of

r^{*}

, which is small relative to typical purchase-noise variance at this scale.

Paper	Demand	Noise	Regret	Per 10K ( $d{=}5$ )
Kleinberg and Leighton (2003)	none	none	$\sqrt{T}$	$\sim$ 100 lost
Broder and Rusmevichientong (2012)	parametric	known family	$\sqrt{T}$	$\sim$ 100 lost
Broder and Rusmevichientong (2012)	param., well-sep.	known family	$\log T$	$\sim$ 9 lost
Javanmard and Nazerzadeh (2019)	linear, $s_{0}$ -sparse	known, log-conc.	$s_{0}\log d\log T$	fast if $s_{0}$ small
Xu and Wang (2021)	linear, contextual	known	$d\log T$	$\sim$ 46 lost
Xu and Wang (2021)	linear, contextual	unknown var.	$\geq\sqrt{T}$	$\sim$ 100 lost

Table 32: Regret rates in dynamic pricing under progressively stronger assumptions.

T

is the number of customers,

d

the feature dimension,

s_{0}

the sparsity level. The last column translates asymptotic rates into concrete terms for

T=10{,}000

with

d=5

, setting constants to 1.

Figure 29: Theoretical regret rate functions at constants equal to 1,

d=5

. Legend names the source paper for each rate. Two cases for the

s_{0}\log d\log T

rate are shown:

s_{0}=1

(very sparse,

\approx 15

lost at

T=10{,}000

) and

s_{0}=5

(here

s_{0}=d

,

\approx 74

lost), which at this constants-equal-to-1 calibration sits above

d\log T

because

s_{0}\log d\approx 8.0>d=5

; sparse dominates when

s_{0}\ll d

. The vertical line marks

T=10{,}000

, matching the “Per 10K” column of Table 32 .

Level	Algorithm	$T{=}10$ K	$T{=}50$ K	$T{=}100$ K	$T{=}200$ K	Rate
0	$\varepsilon$ -greedy	160	624	1,158	2,263	$\Theta(T)$
1	LTE (5%)	1,005	1,165	1,363	1,771	$O(T^{2/3})$
2	UCB1	719	2,609	4,263	6,734	$O(\sqrt{KT})$
3	Thompson	290	628	844	1,136	$O(\sqrt{KT})$
4	UCB-PI	789	2,266	3,244	4,503	$O(\log T)$
5	UCB-PI-tuned	338	541	640	780	$O(\log T)$

Table 33: Cumulative regret at four checkpoints on the Misra et al. (2019) environment, mean over 10 seeds, with each algorithm’s theoretical rate.

Figure 30: Cumulative regret on log-log axes (

K=100

,

S=1{,}000

, 10 seeds). Each algorithm’s legend entry includes its theoretical regret rate. Dashed lines show

\Theta(T)

,

O(\sqrt{T})

, and

O(\log T)

reference rates. Shaded regions are

\pm 2

standard errors.

Figure 31: Weaver-style curve-learning pricing simulation across low, middle, and high optimal-price locations (

K=10

,

T=2{,}500

, 1,000 seeds, price updates every 10 customers). The

y

-axis is cumulative profit divided by the price-set oracle. GP-UCB and GP-TS share information across nearby prices; GP-UCB-M and GP-TS-M additionally impose

D^{\prime}(p)\leq 0

through derivative-constrained GP curves. Shaded regions are

\pm 2

standard errors.

WTP	$p^{*}_{P}$	TS	GP-TS	GP-TS-M	GP-UCB-M
$B(2,9)$	0.2	83.6%	77.2%	97.5%	98.3%
$B(2,2)$	0.4	93.5%	94.8%	97.0%	97.5%
$B(9,2)$	0.7	98.2%	98.2%	95.8%	96.1%

Table 34: Final cumulative profit as a percentage of the price-set oracle by

T=2{,}500

in the Weaver-style curve-learning simulation.

Method	Mean Return	% of Optimal
DP Oracle	$192.41\pm 0.33$	$100.0\%$
RvS	$186.58\pm 0.34$	$97.0\%$
BC	$186.28\pm 0.31$	$96.8\%$
DT	$185.27\pm 0.33$	$96.3\%$
CQL	$178.08\pm 1.48$	$92.6\%$
BCQ-D	$177.05\pm 0.73$	$92.0\%$

Table 35: Policy value for each offline RL method, expressed as mean return and percentage of the DP optimal. Standard errors computed over 20 seeds.

Figure 32: Policy value (as % of DP optimal) versus behavioral policy randomness

\epsilon_{b}

for the value-based methods (FQI, CQL, IQL-argmax, BCQ-D), the supervised-conditioning family (DT, RvS), and the behavioral cloning baseline (BC). Higher

\epsilon_{b}

increases data coverage.

state	action	log share	target share	in log	exact $Q^{*}$	FQI	CQL
low	keep	9.5%	69.0%	yes	5.345	3.454	3.447
low	replace	85.8%	0.0%	yes	4.310	2.608	2.606
high	keep	0.5%	0.0%	yes	4.079	1.999	2.000
high	replace	4.2%	31.0%	no	4.310	0.000	-2.779

Table 36: Discounted state-action occupancy under the logging and target policies, finite-log coverage, and final action values. The log contains forty transitions for each marked pair and none for high mileage with replacement.

Figure 33: The learned value of replacement at high mileage, the state-action pair absent from the finite log. The reference line is the exact optimal action value.

Figure 34: RLHF versus DPO pipelines. Top row: the three-stage RLHF pipeline trains a reward model from human preferences, then uses PPO to fine-tune the policy with a KL penalty. Bottom row: DPO collapses the pipeline into a single supervised learning objective over preference pairs, eliminating the explicit reward model (ghosted box).

state	action	$r$	$\pi^{SFT}$	$e^{r/\lambda_{KL}}$	$\pi^{SFT}e^{r/\lambda_{KL}}$	$\pi^{*}$	$\lambda_{KL}\log(\pi^{*}/\pi^{SFT})$
low	keep	1.0000	0.6000	7.3891	4.4334	0.9679	0.2391
low	replace	-0.5000	0.4000	0.3679	0.1472	0.0321	-1.2609
high	keep	0.2000	0.4000	1.4918	0.5967	0.7300	0.3008
high	replace	-0.5000	0.6000	0.3679	0.2207	0.2700	-0.3992
state	$Z(s)$		$\lambda_{KL}\log Z(s)$		inverse reward minus $r(s,a)$
low	4.5806		0.7609		-0.7609

Table 37: DPO reward inversion on the Engine Replacement MDP at

\lambda_{KL}=0.5

. The four rows report the exponential tilt and the normalized policy. The final column is the reward representative selected by the DPO normalization.

Figure 35: Axiom violation rates (left) and worst-group utility of the top-ranked candidate (right) versus pairwise comparisons per pair

N

, mean and one standard error over thirty seeds. BT-MLE PO and PMC violation rates rise to one as

N

grows, in line with Theorem 20 ; LCPO violation rates remain at zero (PO by construction) or fall to zero (PMC by

N=100

).

Method	PO viol.	PMC viol.	Worst-group util.	Modal top
BT-MLE (linear)	1.00 (0.00)	1.00 (0.00)	-2.000 (0.000)	a
Leximax Copeland (LCPO)	0.00 (0.00)	0.00 (0.00)	-2.000 (0.000)	a

Table 38: Aggregator comparison at

N=2000

comparisons per pair, mean and standard error over thirty seeds. PO viol. and PMC viol. are fractions of seeds on which the output ranking violates the corresponding axiom. Worst-group util. is the type-two utility of the top-ranked candidate. Modal top names the candidate most frequently ranked first.

Figure 36: Optimal accept/reject and stay/quit boundaries in wage-amenity space. Left: the optimal accept/reject boundary for searching states. Right: the optimal stay/quit boundary for employed states. Both boundaries cut diagonally, reflecting the tradeoff between wage and amenity quality under compensating differentials.

Figure 37: Policy value

V^{\pi}(s_{0})

versus number of preference comparisons

K

for all six methods (30 seeds,

L=15

). The right axis shows the percentage of DP-optimal value.

Method	Policy agree. (%)	$V^{\pi}$ corr.	Mean amenity	Mean wage	$\hat{\alpha}$
NN RLHF	$96.4$	$1.000$	$4.67$	$73$	—
DPO	$57.1$	$0.777$	$3.38$	$63$	—
Correct	$100.0$	$1.000$	$4.84$	$71$	$0.597$
Misspecified	$50.0$	$0.889$	$3.00$	$70$	—
Optimal	$100.0$	$1.000$	$4.84$	$71$	—

Table 39: Diagnostics at

K=5{,}000

(single seed): policy agreement with

\pi^{*}

, value-function correlation, and mean accepted wage and amenity for each method.

Figure 38: Policy value

V^{\pi}(s_{0})

versus segment length

L

at

K=2{,}000

for the neural network and DPO (20 seeds). The right axis shows the percentage of DP-optimal value.

RL problem	Econometric analogue	Identifying object
Observed state blocks all confounding	Backdoor adjustment, sequential unconfoundedness, g-formula	Interventional transition and reward from conditional models given observed controls
Exogenous variation shifts actions	Instrumental variables and conditional moment restrictions	Structural transition or reward function satisfying moments conditional on instruments
Latent state is not observed but proxies are available	Proximal causal inference and negative controls	Bridge functions that recover policy value from proxy distributions
Actions operate through observed intermediate variables	Front-door, mediation, and g-methods	Mediated transition law and decomposed direct or indirect dynamic effects
No credible point-identifying source exists	Rosenbaum and Tan sensitivity, partial identification	Bounds over policy values under restricted hidden-bias models
High-dimensional nuisance functions are needed	DML, orthogonal scores, efficient OPE	Neyman-orthogonal moments for value or dynamic treatment parameters

Table 40: Econometric taxonomy for causal reinforcement learning

transition estimate	$P(\mathrm{degrade})$	$V(\mathrm{low})$	bias
interventional truth	0.5000	5.3448	0.0000
naive population formula	0.6000	4.7403	-0.6046
naive simulation	0.5992	4.7451	-0.5998 $\,(0.0044)$
backdoor simulation	0.4983	5.3560	0.0112 $\,(0.0063)$

Table 41: Target-policy evaluation on the

+U

Engine Replacement MDP. Monte Carlo entries are means and standard errors over twenty fixed seeds.

Figure 39: Causal graphs for three identification strategies in confounded MDPs. Gray dashed nodes are unobserved; dashed edges involve unobserved variables. (a) Front-door criterion with mediator

M_{t}

. (b) Instrumental variables with exogenous instrument

Z_{t}

. (c) Proximal causal inference with proxies

W_{t}^{(1)},W_{t}^{(2)}

.

Figure 40: Complete causal graph of the simulation DGP. Gray dashed node (

U_{t}

) is unobserved; dashed edges involve

U_{t}

.

Figure 41: (a) Bias of five OPE estimators as a function of confounding strength

\rho

. (b) RMSE of five estimators as a function of

\rho

. (c) IV estimator bias distribution vs. instrument strength at

\rho=1

; dashed red line is naive estimator bias.

Scenario	Estimator	Bias	Std	RMSE
Well-specified	IS	$+0.017$	$0.100$	$0.099$
	MB	$-0.001$	$0.057$	$0.056$
	CF	$+0.003$	$0.067$	$0.065$
Misspecified	IS	$+0.017$	$0.100$	$0.099$
	MB	$-0.072$	$0.061$	$0.093$
	CF	$+0.002$	$0.078$	$0.076$

Table 42: Counterfactual off-policy evaluation under a linear SCM,

n=1000

,

20

seeds.

Figure 42: Root mean squared error of three OPE estimators against sample size on a log-log scale. The left panel uses a well-specified outcome model. The right panel omits the interaction.

Figure 43: Three representations of sequential treatment. Panel (a) shows fixed potential outcomes and the noncausal consistency link that selects the realized response. Panel (b) shows a two-stage treatment graph, with the treatment-confounder feedback through

S_{2}

highlighted. Panel (c) constructs the finite-horizon decision process whose state is the full observed history.

Dynamic treatment regime	Induced finite-horizon MDP
Observed history $\bar{H}_{k}$	Stage- $k$ state $S_{k}^{\mathcal{M}}=\bar{H}_{k}$
Treatment $A_{k}\in\mathcal{A}_{k}$	Action $A_{k}\in\mathcal{A}_{k}$
Rule $\pi_{k}(\bar{h}_{k})$	Deterministic history-state policy $\pi_{k}(S_{k}^{\mathcal{M}})$
Terminal objective $Y$	Rewards $R_{k}^{\mathcal{M}}=0$ for $k<K$ and terminal return $G^{\mathcal{M}}=Y$
Assignment law $e_{k}(a\mid\bar{h}_{k})=\mathbb{P}(A_{k}=a\mid\bar{H}_{k}=\bar{h}_{k})$	Behavior policy $\mu_{k}(a\mid S_{k}^{\mathcal{M}})=e_{k}(a\mid\bar{h}_{k})$
Fixed-regime recursion $v_{k}^{\pi}(\bar{h}_{k})$	Bellman policy-evaluation recursion for $\pi$

Table 43: Exact correspondences between the identified DTR recursion and the induced full-history MDP in Theorem 23 ( Murphy, 2003 ; Schulte et al., 2014 ) .

Method	$V(\hat{\pi})/V^{*}$	MC SE
Plug-in g-computation (tabular, $N=10000$ )	1.0000	(0.0000)
Oracle $V^{*}$ (tabular)	1.0000	–
Oracle $V^{*}$ (high-dim, $p=10$ )	1.0000	–
$Q$ -learning (tabular, $N=10000$ , 100 replays)	0.9968	(0.0007)
Neural-FQI (high-dim, $N=20000$ )	0.9842	(0.0013)
DQN (high-dim, $N=20000$ )	0.9740	(0.0012)

Table 44: Recovered policy value

V(\hat{\pi})/V^{*}

at the largest cohort size in the stylized home-visiting problem, with fixed-schedule references for the tabular design. The tabular design uses

N=10{,}000

and 50 paired seeds. The continuous design uses

N=20{,}000

and 20 paired seeds.

Figure 44: Plug-in g-computation,

Q

-learning, and unadjusted treatment-count regression under sequential ignorability. Panel (Q1) shows the tabular sample-size sweep. Panel (Q2) shows the replay-budget sweep at

N=300

. Panel (Q3) shows the continuous-state sweep with a smooth contrast and shared network architecture. Error bars are 95% Monte Carlo confidence intervals for mean recovery over paired seeds. They are not confidence intervals for a regime fitted to one observed cohort.

Model class	Information and transition restriction	Required correction	Statistical consequence
Contextual bandit	One decision, $H=1$	Action ratio $\pi_{e}/\pi_{b}$	Static efficient influence function
Non-Markov process	Complete history $\mathcal{G}_{t}$ ; transitions may depend on the past	Cumulative history ratio $\rho_{1:t}$	Efficiency bound can grow exponentially in $H$
Time-varying MDP	$S_{t}$ is sufficient; transition law may vary with $t$	Stage-specific marginal state-action ratio	Polynomial horizon dependence is attainable
Stationary MDP	Markov and time-homogeneous transition law	Stationary occupancy or density ratio	Discounted or average-reward values can be studied from mixing trajectories

Table 45: Model hierarchy for off-policy evaluation ( Uehara et al., 2022 ) . The ratio column gives the correction used by an efficient score.

Estimator	Bias	SD	RMSE	Rel. RMSE at $H=64$
DM	+0.0024	0.1113	0.1112	0.026
MIS	+0.0061	0.2137	0.2136	0.134
WDR	-0.0181	0.3438	0.3439	0.209
DR	-0.0179	0.3661	0.3661	1.242
WIS	+0.0490	0.4580	0.4601	0.286
PDIS	+0.0029	0.6013	0.6007	2.055

Table 46: OPE results for the stylized home-visiting MDP. Panel A reports point-estimator performance at

n=2000

and

H=16

, with relative RMSE at

n=500

and

H=64

. Panel B reports the nuisance-misspecification design at

n=500

and

H=16

. Panel C assesses estimator-level standard errors at

n=1000

and

H=8

.

Figure 45: OPE performance over 500 replications. Panel (a) shows relative RMSE against the number of logged trajectories at

H=16

. Panel (b) shows relative RMSE against horizon at

n=500

. Panel (c) shows estimator bias under nuisance misspecification with 95% Monte Carlo confidence intervals for mean bias. These error bars are distinct from the estimator-level intervals in Equations ( 170 ) and ( 173 ).

	$\rho_{t}\,/\,\rho_{1:t}$				discounted return
trajectory	$t=1$	$t=2$	$t=3$	$t=4$	$G_{i}$
1	0 / 0	0 / 0	0 / 0	0 / 0	-1.720
2	2 / 2	0 / 0	0 / 0	0 / 0	1.488
3	0 / 0	2 / 0	2 / 0	0 / 0	1.356
4	0 / 0	2 / 0	0 / 0	0 / 0	0.708
5	2 / 2	0 / 0	2 / 0	2 / 0	1.504

Table 47: Every per-step and cumulative importance weight in the twelve-trajectory Engine Replacement MDP log. Panel B compares the five estimates with the exact finite-horizon dynamic-programming value.

Figure 46: Horizon growth in trajectory-weight variance and value-estimator error. Panel (a) compares the exact variance with the mean across forty fixed Monte Carlo seeds. Panel (b) reports root mean squared error across those seeds for IS, PDIS, and DR. WIS is conditional on logs with nonzero cumulative weight mass at every period, and each marker gives the number of defined estimates out of forty.

	$\psi_{1}^{*}=1.00$			$\psi_{2}^{*}=0.50$
Method	Bias	RMSE	Cov	Bias	RMSE	Cov
Naive OLS	-0.191	0.197	0.01	+0.933	0.934	0.00
IPTW MSM (naive SE)	+0.019	0.103	0.88	+0.146	0.192	0.73
Dynamic DML	-0.005	0.046	0.97	-0.001	0.018	0.93

Table 48: Dynamic-DML results for the stylized home-visiting SNMM at

n=4000

over 200 datasets. Panel A reports bias, RMSE, and 95% interval coverage. Panel B compares the joint sandwich covariance with its Monte Carlo target, and sets the standard error for

\psi_{1}-\psi_{2}

against the one obtained by dropping the cross-stage block.

Figure 47: Dynamic DML over 200 replications. The left panel shows absolute bias of

\hat{\psi}_{2}

against sample size. The right panel shows empirical coverage of nominal 95% confidence intervals.

Method	$V(\hat{\pi})/V^{*}$	MC SE
Oracle regime $(c_{1}^{},c_{2}^{})$	1.0000	–
Plug-in $Q$ ( $n=4000$ )	0.9996	(0.0001)
Backward AIPW ( $n=4000$ )	0.9966	(0.0006)
IPW ( $n=4000$ )	0.9911	(0.0020)
Behavior policy	0.5770	–

Table 49: Policy value relative to the threshold-class oracle at

n=4000

in the stylized home-visiting problem. Monte Carlo standard errors describe the reported mean across 40 independent replications, not one learned policy.

Figure 48: Offline threshold-policy learning over 40 replications. Panel (a) shows regret against sample size. Panel (b) shows regret under outcome-model and propensity misspecification at

n=1000

, with 95% Monte Carlo confidence intervals for mean regret. Panel (c) shows the learned AIPW thresholds at

n=250

and

n=4000

relative to the oracle pair.

Issue	What fails	Available partial remedy	What remains open
Near-zero treatment contrasts	The optimal-action map and selected thresholds are nonregular	Adaptive intervals, $m$ -out-of- $n$ bootstrap, honest value evaluation	General, powerful inference for learned multistage rules ( Laber et al., 2014 ; Wang and Tom, 2026 )
State sufficiency	A compressed state may omit history needed for exchangeability or Markov transitions	Use full recorded history; diagnostics can probe observed-law Markov fit but cannot verify exchangeability	Identification under latent state belongs to sensitivity, proxy, or IV analysis in Chapter 13
Weak overlap	Importance ratios and efficiency bounds explode even though the estimand is formally identified	Restrict the target policy, trim cautiously, collect data, or report one-sided safety bounds	Principled policy-class restriction without silently changing the target ( Uehara et al., 2022 )
Function approximation	FQE, density ratios, and orthogonal scores require coverage, completeness, and nuisance-rate conditions	Cross-fitting, doubly robust scores, and bootstrap under verified model conditions	Reliable inference with general nonlinear approximation and policy-dependent nuisances ( Hao et al., 2021 ; Kallus and Uehara, 2022 )
Policy search and inference	Optimizing and evaluating on the same observations creates selection bias	Honest splitting or an external evaluation sample	Efficient reuse of limited longitudinal data with valid post-selection inference ( Uehara et al., 2022 ; Jaman et al., 2025a )
Transport and drift	A value identified in historical trajectories need not equal value after deployment	Reweight measured population shifts and monitor support	Unmeasured environment change and feedback from deployment

Table 50: Open issues that remain material for applied work. The first five rows and the operational-objectives row are tied to the cited literature; the transport-and-drift row is practitioner synthesis.

Figure 49: Causal-bandit diagnostics. Panel (a) varies graph hardness without changing rewards. Panel (b) reports recommendation error under a shrinking-gap local alternative. Panel (c) compares marginal Thompson sampling, unseeded context-conditional

\mathrm{TS}_{Z}

, the paper’s

\mathrm{TS}_{C}

, and the separately labelled ETT warm start. Error bars and bands are 95% Monte Carlo intervals.

Method	$m=2$	$m=8$	$m=24$	$m=48$
Successive Reject	0.168 (0.003)	0.168 (0.003)	0.166 (0.003)	0.164 (0.003)
Lattimore Alg 1	0.000 (0.000)	0.017 (0.002)	0.172 (0.003)	0.263 (0.002)

Table 51: Simple regret on the parallel-bandit hard family. Entries are means with Monte Carlo standard errors in parentheses over two thousand replications.

N=50

,

T=400

, and

\epsilon=0.3

.

Algorithm	Regret at $T=100$	Regret at $T=1000$
Binary ETT warm start $+$ RDC	0.10	0.19 (0.11)
$\mathrm{TS}_{C}$ ( Bareinboim et al., 2015 )	1.17	1.21 (0.08)
Observational seed only	1.60	1.68 (0.09)
RDC only	4.13	6.23 (0.14)
$\mathrm{TS}_{Z}$ (context-conditional TS)	5.92	11.04 (0.19)
Vanilla Thompson sampling	14.99	150.00 (0.21)

Table 52: Greedy-casino cumulative pseudo-regret over five hundred paired replications. The binary ETT row is a data-fusion extension and is not Algorithm 1 of Bareinboim et al. (2015) .

Policy	Mean Return	SE	CVaR 95	CVaR 99	Avg. Order
DP Oracle	40.4	0.0	-51.5	-95.6	2.56
IQN-Neutral	34.9	0.8	-57.5	-98.0	3.21
IQN-CVaR95	36.4	0.4	-55.3	-97.8	2.99

Table 53: Risk-sensitive inventory management. Mean

\pm

SE across ten training seeds, 50,000 evaluation episodes per seed. Avg. Order is the per-state mean order quantity averaged uniformly across inventory levels.

Figure 50: Average order quantity by inventory level for the DP oracle and the two IQN evaluation policies. Mean across ten training seeds with

\pm 1

SE bands, 50,000 evaluation episodes per seed.

Method	Return	Cost	Budget	$\lambda$
LP Oracle	186.4	31.35	Y	1.20
Unconstrained Q-learning	257.8 $\pm$ 3.2	99.08 $\pm$ 2.62	N	–
Lagrangian Q-learning	178.4 $\pm$ 4.0	28.44 $\pm$ 3.60	Y	1.40 $\pm$ 0.00

Table 54: Carbon-constrained production results. Budget

d=31.35

. Q-learning rows report means and standard errors over ten seeds.

Figure 51: (a) Lagrange multiplier

\lambda

over training episodes, with the LP shadow price

\lambda^{*}

as the dashed reference. (b) Mean discounted return for unconstrained and Lagrangian Q-learning, with the constrained LP optimum as the dashed reference. Solid lines are means over ten seeds; shaded bands are

\pm 1

standard error.

Method	Nominal	Perturbed	Degradation (%)
Standard DP	$-4.941\pm 0.001$	$-8.694\pm 0.004$	$-76.0$
Q-learning	$-4.941\pm 0.001$	$-8.692\pm 0.005$	$-75.9$
Robust DP ( $\theta$ =5.0)	$-4.942\pm 0.001$	$-8.672\pm 0.004$	$-75.5$
Robust Q-learning ( $\theta$ =5)	$-4.942\pm 0.001$	$-8.672\pm 0.004$	$-75.5$
Robust DP ( $\theta$ =2.0)	$-4.948\pm 0.001$	$-8.486\pm 0.004$	$-71.5$
Robust Q-learning ( $\theta$ =2)	$-4.948\pm 0.001$	$-8.486\pm 0.004$	$-71.5$

Table 55: Discounted returns under nominal and perturbed income. DP policies are computed via value iteration; Q-learning policies are trained under the nominal model.

Figure 52: Consumption policy

c(w)

for each method. Dashed lines show Q-learning policies; solid lines show DP.

Calculation	Setting	Estimate	Check
CMDP	Best deterministic return	3.4545	feasible
CMDP	Randomized return	4.6727	$\pi(R\mid\mathrm{high})=0.4074$
CMDP	Replacement occupancy	0.2000	budget $0.2000$
CMDP	Shadow price $\lambda^{*}$	6.0909	slope $6.0909$
Geometry	Uniform-start return	3.9455	$\pi(R\mid\mathrm{high})=0.3437$
KL tilt	$\theta=10$ high-state probability	0.5233	shift $+0.0233$

Table 56: Constrained occupancy and KL-tilting calculations for the Engine Replacement MDP. The CMDP rows use a low-mileage initial engine. The geometry row uses a uniform initial distribution so every deterministic-policy occupancy is distinct.

Figure 53: The left panel shows the occupancy polytope under a full-support initial distribution, the replacement budget, and the randomized optimum. The right panel shows the worst-case high-mileage transition probability under KL multiplier preferences.

Figure 54: Sutton blocking maze. The agent starts at

S

and aims for the goal at

G

; the only opening in the wall along row two shifts from column eight in Phase 1 to column zero in Phase 2 at environment step

t=1000

. The flip is not signaled to the agent and triggers the model-staleness failure mode that the experiment is designed to expose.

Figure 55: Cumulative reward on the blocking maze for five agents under a shared three-thousand-step environment budget. The dashed vertical line marks the wall flip at

t=1000

. Lines are means and shaded bands are one standard error across thirty seeds.

Agent	End of Phase 1	Phase 2 gain	Total ( $t=3000$ )
Dyna-Q (K=50)	42.2 $\pm$ 5.3	9.8 $\pm$ 4.0	52.0 $\pm$ 4.2
Dyna-Q+ (K=50)	37.4 $\pm$ 3.7	9.6 $\pm$ 2.8	47.0 $\pm$ 4.4
Dyna-Q (K=5)	34.4 $\pm$ 4.9	4.8 $\pm$ 3.0	39.2 $\pm$ 4.7
Schmidhuber 1990 (NN)	2.5 $\pm$ 0.3	1.5 $\pm$ 0.2	4.0 $\pm$ 0.4
Q-learning (K=0)	2.9 $\pm$ 0.5	0.6 $\pm$ 0.2	3.5 $\pm$ 0.5

Table 57: Cumulative reward on the blocking maze. End of Phase 1 is at

t=1000

; Phase 2 gain is the increase over the remaining two thousand steps; Total is the value at

t=3000

. Mean

\pm

standard error across thirty seeds.

Figure 56: Cumulative regret across stability regimes for nine learning paradigms on the cobweb with adjustment cost. Lines are means and shaded bands are one standard error across twenty seeds. Panels correspond to

b/c\in\{0.5,1.0,2.0\}

. Oracle is the zero reference.

Paradigm	Stable ( $b/c=0.5$ )	Borderline ( $b/c=1.0$ )	Unstable ( $b/c=2.0$ )
Oracle	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00
RLS	5.89 $\pm$ 1.04	4.38 $\pm$ 0.43	5.87 $\pm$ 0.18
Model-Based LQ	11.65 $\pm$ 0.93	18.96 $\pm$ 1.73	42.90 $\pm$ 3.75
MB-LG-VR	38.42 $\pm$ 1.86	30.26 $\pm$ 1.86	43.59 $\pm$ 3.33
MB-LG-Pathwise	47.07 $\pm$ 0.97	35.43 $\pm$ 1.68	47.46 $\pm$ 4.04
Arifovic GA	92.89 $\pm$ 11.69	133.43 $\pm$ 8.65	308.88 $\pm$ 29.16

Table 58: Cumulative regret at

T=500

on the cobweb with adjustment cost, mean

\pm

standard error across twenty seeds. Lower is better; Oracle is the reference.

Figure 57: Parameter recovery for recursive least squares, the model-based LQ learner, and the MB-LG-REINFORCE and MB-LG-Pathwise ensemble means across stability regimes. Solid lines are means and shaded bands one standard error across twenty seeds; dashed horizontal lines mark the true values of

a

and

b

. All four methods converge to the truth within a few dozen environment steps.

Paradigm	Regime	$\|\hat{a}-a\|$	$\|\hat{b}-b\|$	$\|\hat{c}-c\|$	$\|\hat{\phi}-\phi\|$
RLS	stable	0.037 $\pm$ 0.035	0.015 $\pm$ 0.018	—	—
RLS	borderline	0.019 $\pm$ 0.026	0.011 $\pm$ 0.020	—	—
RLS	unstable	0.004 $\pm$ 0.016	0.003 $\pm$ 0.019	—	—
Model-Based LQ	stable	0.029 $\pm$ 0.019	0.013 $\pm$ 0.009	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000
Model-Based LQ	borderline	0.004 $\pm$ 0.011	0.002 $\pm$ 0.008	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000
Model-Based LQ	unstable	0.001 $\pm$ 0.007	0.003 $\pm$ 0.008	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000

Table 59: Final parameter recovery error

|\hat{\theta}-\theta|

at

t=T

, mean

\pm

standard error across twenty seeds. Cells marked — indicate parameters the paradigm does not estimate (recursive least squares assumes

c

and

\phi

known); the three model-based learners shown estimate all four.

Figure 58: Policy distance to the oracle at a regime-specific reference state, log scale. The reference state is the static optimum quantity for each regime. Lines are means and bands one standard error across twenty seeds. Oracle is identically zero by construction and is omitted.

Figure 59: Left: cumulative regret on the logistic-growth fishery for seven paradigms (twenty seeds, mean and one-standard-error bands), presented in rank order by final regret. Right: seed-0 stock trajectories (solid) and realized harvest trajectories (dashed); the carrying capacity

K=10

and the maximum sustainable yield harvest

rK/4=1.0

are shown as reference lines.

Paradigm	Final regret
Oracle	0.00 $\pm$ 0.00
RLS	13.67 $\pm$ 0.43
Model-Based DP	14.69 $\pm$ 0.73
Q-Learning	274.71 $\pm$ 24.36
Naive	447.35 $\pm$ 3.24
Arifovic GA	706.13 $\pm$ 16.65

Table 60: Final cumulative regret at

T=500

on the fishery, mean

\pm

standard error across twenty seeds. Lower is better; oracle is the zero reference. Paradigms are presented in rank order.

Paradigm	$\hat{r}$	$\hat{K}$	$\|\hat{r}-r\|$	$\|\hat{K}-K\|$
RLS	0.398 $\pm$ 0.002	10.034 $\pm$ 0.064	0.007	0.219
Model-Based DP	0.400 $\pm$ 0.002	9.965 $\pm$ 0.050	0.006	0.180

Table 61: Parameter recovery on the fishery at

t=T

for the two structured learners, mean

\pm

standard error across twenty seeds. True values are

r=0.4

and

K=10

. Recursive least squares assumes

(p,c,\sigma)

known and only estimates

(r,K)

; the model-based DP learner additionally estimates

(p,c)

(not shown).

Figure 60: Left: cumulative regret against the Clark-Scarf oracle for five paradigms, means and one-standard-error bands over twenty seeds; the world-model and decentralized traces are near zero at the scale set by the model-free and naive learners. Middle: per-period cost above the oracle over time for the three near-oracle methods, smoothed; the world model pays a large exploration cost early and then converges below the decentralized heuristic, the asymptotic advantage the terminal-cost column of Table 62 records. Right: the neural world model’s one-step forecast mean-squared error over training, seed zero, on a log scale.

Paradigm	Cumulative regret	Terminal cost/period
Oracle	0.0 $\pm$ 0.0	6.61 $\pm$ 0.17
Decentralized	1964.8 $\pm$ 50.8	9.84 $\pm$ 0.26
NN World Model	3879.4 $\pm$ 406.5	8.00 $\pm$ 0.88
Naive	29485.4 $\pm$ 6777.2	106.05 $\pm$ 29.04
Model-Free DQN	44137.1 $\pm$ 5593.1	210.79 $\pm$ 53.76

Table 62: Cumulative regret against the Clark-Scarf oracle at

T=500

and terminal per-period cost over the last eighty steps, mean

\pm

standard error across twenty seeds, in rank order by cumulative regret. Lower is better; the oracle is the zero reference and its per-period cost is 6.61.

$N$	mean kernel error	value error	s.e.	simulation bound	s.e.
25	0.1556	0.5015	0.0257	14.0040	0.7085
50	0.1028	0.3284	0.0168	9.2520	0.4764
100	0.0756	0.2435	0.0143	6.8040	0.3902
250	0.0502	0.1615	0.0087	4.5144	0.2407
500	0.0355	0.1138	0.0060	3.1950	0.1688
1,000	0.0259	0.0830	0.0045	2.3265	0.1276

Table 63: Learning the four Engine Replacement MDP transition rows. Value errors, bounds, and standard errors use two hundred fixed seeds.

Figure 61: Optimal-value error and its simulation-lemma bound against transition samples per state-action pair. Lines show means and shaded regions show one standard error over two hundred fixed seeds.

Figure 62: The Horizon production loop ( Gauci et al., 2019 ) . A serving policy

\mu

logs each decision with its propensity and metrics, the Timeline pipeline assembles transitions, a data-understanding check screens the formulation, an offline learner is trained and scored by counterfactual policy evaluation, and the policy is rolled out on a fraction of traffic with exploration and retrained daily on the data it generates.

Stage	Horizon mechanism	Notification instantiation	Source
Formulation	Data-understanding model checks the Markov and bandit reductions	Sending affects value that arrives after the decision	§4
Logging	Propensity $\mu(A_{t}\mid S_{t})$ , feasible actions, and metrics map	Serving logs send or drop and its probability	§2, §8
Transitions	Timeline join into $(S_{t},A_{t},R_{t},S_{t+1})$ with a time gap	Candidate sequence for one person	§2
Reward	$R_{t}=w^{\top}M_{t}$ formed at training time	Interactions and activity, penalty on sending	§2, §9.1
Algorithm	DQN, parametric-action DQN, DDPG, or SAC by action space	Discrete-action DQN	§5, §9.1
Evaluation	Six counterfactual estimators on logged data	Screen before any exposure	§7.2

Table 64: The Horizon production loop ( Gauci et al., 2019 ) and its instantiation for push notifications. Section numbers refer to the source paper.

Intervention	Live comparison	Reported effect
Exploratory data	Train with a 5% stochastic-policy traffic slice rather than only deterministic-policy logs	+0.07% ViewTime
Standard off-policy correction	Weight logged examples by learned behavior-policy importance ratios	+0.53% videos viewed; no significant ViewTime change
Top- $K$ off-policy correction	Train the production candidate generator with $K=16$ top- $K$ correction	+0.85% ViewTime; -0.16% videos viewed
Top-1 ablation	Set $K=1$ , reducing the correction toward the standard top-1 case, versus production $K=16$	-0.66% ViewTime
Uncapped-weight regression	Loosen the importance-weight cap from $e^{3}$ to $e^{5}$	-0.52% ViewTime

Table 65: Live YouTube experiments reported by Chen et al. (2019) . Effects are relative to the relevant control model in each sequential experiment, so rows should be read as component tests rather than additive gains.

Parameter	Value
Gradient steps during training	$600{,}000$
Batch size	$20{,}000$
Critic learning rate	$3\times 10^{-4}$
Actor learning rate	$3\times 10^{-5}$
Discount factor $\gamma$	$0.9998$
Behavior-policy standard deviation $\sigma_{\beta}$	$0.05$

Table 66: Training hyperparameters for the Meta production bidding agent ( Korenkevych et al., 2023 ) .

Panel A: offline conservative-penalty sweep in the campaign simulator
Penalty weight $\alpha$	Learning behavior	Gain vs. base policy
$0.1$ (deployed)	Stable	Above $+0.5\%$ after $400{,}000$ steps
$1.0$	Over-restrictive	Marginal
$0.01$	Unstable	Abrupt swings
$0.0$	Unstable	Abrupt swings
Panel B: production A/B tests vs. the original base policy

Table 67: Meta production bidding ( Korenkevych et al., 2023 ) . Panel A: checkpoint selection in the internal campaign simulator over the conservative-Q-learning penalty weight, with

1{,}000

evaluation episodes per checkpoint. Panel B: two one-week production A/B tests, each over roughly

50

billion impressions, reporting the change in the platform performance metric with

95\%

confidence intervals. Both tests also showed revenue gains.

Panel A: single-agent online A/B, 10 disjoint-keyword ads, vs. keyword bidding
Policy	Purchase/cost	ROI	CVR	PPC
RMDP	$+35.04\%$	$+21.38\%$	$+23.11\%$	$-5.16\%$
Panel B: multi-agent online A/B, 1000 competing ads (5 Feb 2018), vs. keyword bidding
Policy	Purchase/cost	ROI	CVR	PPC
M-RMDP	$+13.01\%$	$+39.12\%$	$+12.62\%$	$-0.74\%$
RMDP	$+6.29\%$	$+26.51\%$	$+3.12\%$	$-3.36\%$

Table 68: Alibaba sponsored-search real-time bidding ( Zhao et al., 2018 ) . Relative improvement over the keyword-bidding baseline in online A/B tests. Purchase/cost is purchase amount per unit cost, the optimized objective; ROI is return on investment, CVR conversion rate, and PPC pay per click, for which lower is better. Panel A: the single-agent robust MDP on ten advertisers with disjoint keywords. Panel B: the same policy and the cooperative massive-agent policy on a thousand competing advertisers, rank-ordered by purchase/cost, showing the single-agent gain shrinking under competition and the cooperative reward restoring it.

Panel A: two-week A/B test across five cities, call-weighted vs. control
Metric	Change
Driver income	$+1.342\%$
Completion rate	$+0.422\%$
Answer rate	$+0.296\%$
Primary metric, full deployment (synthetic diff-in-diff)	$+5.3\%$
Panel B: per-city driver income in the A/B test (matching calls)

Table 69: DiDi dispatch ( Sadeghi Eshkevari et al., 2022 ) . Panel A: call-weighted average change against control in a two-week A/B test across five cities with three-hour rotation, and the full-deployment effect on the platform’s anonymized primary metric from a synthetic difference-in-differences against a pool of control cities. Panel B: per-city driver-income change in the same A/B test, with the number of matching calls, rank-ordered by income.

Panel A: markdown pricing, 500 luxury SKUs (DQN) vs. 2000 manual, days 16–30
Conversion measure	Learned (DQN)	Manual
Revenue conversion rate	$0.22$	$0.16$
Profit conversion rate	$0.16$	$-0.04$
Panel B: daily FMCG pricing, difference-in-differences index (DRCR)
Policy	DRCR index	Control
DDPG (continuous)	$6.07$	$1.00$

Table 70: Tmall dynamic pricing ( Liu et al., 2019 ) . Panel A: rescaled revenue and profit conversion rate (revenue or profit per unique visitor) over the markdown fortnight, learned DQN policy on 500 luxury SKUs against 2000 manually priced similar products; the manual group’s profit conversion turned negative. Panel B: the difference-in-differences index (difference of revenue conversion rates at a one-year window) for daily pricing of fast-moving goods, continuous and discrete policies against a manually priced control rescaled to

1.00

, rank-ordered by the index. An earlier run of the discrete policy alone against the control scored

5.10

on the same index.

Panel A: mean treatment effect across the five pilot hotels (synthetic control)
Measure	Change	$p$ -value
Revenue per available room	$+11.80\%$	$.0090$
Average daily room rate	$+5.93\%$	$.1160$
Occupancy rate	$+5.15\%$	$.1361$
Panel B: per-hotel decomposition, rank-ordered by RevPAR (after-pilot period)
Hotel	RevPAR	ADR	Occupancy

Table 71: Hotel revenue management at China Lodging Group ( Chen et al., 2023a ) . Panel A: mean percentage change in each measure against the synthetic control across the five pilot hotels, with permutation

p

-values; only the RevPAR effect is significant at the one percent level. Panel B: the two hotels with a fully reported decomposition, rank-ordered by RevPAR, showing the same order of headline gain reached through occupancy for one hotel and through room rate for the other. Pilot: five Hanting hotels in Shanghai, March to June 2015.

Panel A: change against the incumbent policy
Measurement	Turnover time	Stockout rate
July 2024 DiD pilot, 10% intl. SKUs	$-9.53$ days	$-0.83$ points
April 2025 rollout, international SKUs	$-1$ day	no change
April 2025 rollout, domestic SKUs	$-2$ days	no change
July–Aug 2025 vs. 2024, intl. SKUs	$-20\%$	no notable change
Panel B: scale and financial impact, October 2025

Table 72: DeepStock at Alibaba Tmall ( Xie et al., 2026 ) . Panel A: change in the two operational metrics against the incumbent, from a July 2024 difference-in-differences pilot on cherry-picked international SKUs, an April 2025 counterfactual comparison at full rollout, and a July–August 2025 year-over-year comparison. Turnover time is in days (a percentage for the year-over-year row); stockout rate in percentage points of days out of stock. Panel B: deployment scale and estimated financial impact.

Panel A: coding efficiency, BD-rate vs. libvpx (more negative is better)
Agent	PSNR	SSIM	VMAF
Augmented MuZero-RC	$-6.28\%$	$-5.11\%$	$-1.88\%$
MuZero-RC	$-4.72\%$	$-3.68\%$	$-0.53\%$
Panel B: bitrate constraint satisfaction, fraction of evaluation videos
Agent	overshoot ${>}0$	overshoot ${>}5\%$	within $5\%$ of target
libvpx (baseline)	$64.00\%$	$6.13\%$	$71.34\%$

Table 73: MuZero-RC on VP9 rate control ( Mandhane et al., 2022 ) . Panel A: Bjontegaard-delta rate against the libvpx baseline for three quality measures, the average bitrate change at matched quality, so a negative value is a saving, rank-ordered by the PSNR column. Panel B: the fraction of the 3062 evaluation videos on which each policy overshoots the target at all, overshoots by more than five percent, and lands within five percent of the target. Figures are means over five seeds; the standard error is at most

0.33

points in Panel A and

2.2

points in Panel B.

Panel A: live A/B energy savings vs. the Trane heuristic controller
Facility	Energy savings
Mixed-use commercial building	$13\%$
University campus chiller plant	$9\%$
Panel B: experiment and model
A/B design	daily alternation, weather-bucketed
Duration per site	$\approx 3$ months

Table 74: BCOOLER commercial-building cooling ( Luo et al., 2022 ) . Panel A: energy savings against Trane’s heuristic controller in a live A/B test at the two facilities. The paper reports the savings as

9\%

and

13\%

respectively across the two sites, mapping to the university and mixed-use buildings in the order they are introduced. Panel B: the evaluation design and the action-value model. Both A/B tests ran about three months in the shoulder season, and the incumbent controllers had been tuned during the retrofit.

Figure 63: Left: the market reference discount visited under the three logs and under the softened true-best (no-promo) candidate, with the dashed line at the first percentile of the A/B log. Right: the fitted-Q direct-method value estimate against the true field value under the A/B log (circles) and the historical-mixture log (triangles), pooled over 10 seeds, with the 45-degree reference line.

	A/B (uniform)		Incumbent (narrow)		Historical mixture
Estimator	Regret@1	Rank corr.	Regret@1	Rank corr.	Regret@1	Rank corr.
DM	0.210 (0.048)	-0.040 (0.116)	0.246 (0.039)	-0.326 (0.136)	0.038 (0.018)	0.697 (0.053)
DR	0.175 (0.046)	0.051 (0.127)	0.181 (0.043)	-0.166 (0.116)	0.073 (0.033)	0.423 (0.113)
PDIS	0.265 (0.035)	-0.297 (0.110)	0.326 (0.010)	-0.646 (0.083)	0.268 (0.040)	0.029 (0.103)

Table 75: Off-policy selection reliability of three estimators (direct method, DM; per-decision importance sampling, PDIS; doubly robust, DR) under three logging regimes, on the targeted-promotions MDP. Regret@1 is the value gap between the selected and the best deployment policy (reward units, lower is better); rank correlation is Spearman between estimated and true policy value (higher is better). Cells are mean (standard error) over 10 seeds.

		DM error			IS ESS
Candidate	True	A/B	Inc.	Mix.	A/B	Inc.	Mix.
No promo	3.946	-0.228	-0.525	-0.026	1.5	1.6	69.0
5% discount	3.873	-0.151	-0.512	-0.076	2.1	1.7	71.6
10% discount	3.778	-0.066	-0.139	-0.013	1.9	1.7	68.9
Conservative Q-learning (CQL; offline RL)	3.639	+0.172	-0.084	-0.010	1.6	2.0	1.5
Uniform	3.612	+0.008	-0.125	-0.031	500.0	1.9	3.4

Table 76: Per-candidate mechanism behind Table 75 . True value is the on-policy field value of the softened candidate; DM error is the fitted-Q direct-method estimate minus the true value (negative means under-valued); importance-sampling effective sample size (ESS) is

(\sum w)^{2}/\sum w^{2}

over the log’s 500 trajectories. Rows are rank-ordered by true value. Cell entries are means over 10 seeds.

Domain	Representative system	Method deployed	Data or decision scale
Recommendation and ranking	YouTube top- $K$ ( Chen et al., 2019 ) ; Taobao RL-LTV ( Ji et al., 2021 )	Off-policy REINFORCE; recurrent actor-critic	Web-scale logs; item-day episodes
Notification gating	Meta Horizon ( Gauci et al., 2019 )	DQN send/drop	Production-scale logs
Bidding and auctions	Meta bidding ( Korenkevych et al., 2023 ) ; Alibaba RTB ( Zhao et al., 2018 )	Offline RL tuning a base policy; robust-MDP DQN	$\sim$ 1.2B logged steps; $\sim$ 100M auctions per day
Dispatch and matching	DiDi ( Sadeghi Eshkevari et al., 2022 )	TD value inside a matching solver	Tens of millions of orders per day
Inventory	DeepStock ( Xie et al., 2026 )	Base-stock-regularized DDPG	More than 1M SKU-warehouse pairs
Video-codec rate control	MuZero-RC ( Mandhane et al., 2022 )	Model-based planning, policy served	Frame-level control on a live traffic slice

Table 77: The shape of confirmed field deployments: the domain, a representative system, the reinforcement-learning method that ran, and the data or decision scale. Firm types and evaluation practices are discussed in the text.

chapter topic	role of the common model
Theory	value and policy geometry, Section 5.1.6
Deep RL practice	Bellman and value error, Section 6.4
Structural estimation	parameter recovery, Section 7.2
Offline RL	coverage and extrapolation, Section 11.5
Human feedback	reward equivalence, Section 12.6
Causal RL	unobserved confounding, Section 13.4

Table 78: What the Engine Replacement MDP establishes across the survey.

A Survey of Reinforcement Learning For Economics

Abstract

Figures & tables

Appendix

Reinforcement Learning

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

A Comparative Study of Dynamic Programming and Reinforcement Learning in Finite Horizon Dynamic Pricing

JaxMARL-BC: Multi-Agent Reinforcement Learning for Macroeconomic Modelling, in Seconds

	good	worn
$r(s,\text{keep})$	1.00	0.20
$r(s,\text{replace})$	-0.50	-0.50
$P(\text{worn}\mid s,\text{keep})$	0.50	1.00
$V^{\star}(s)$	5.3448	4.3103
optimal action	keep	replace
$d^{\pi^{\star}}(s)$	0.6667	0.3333

$\gamma$	$1/(1-\gamma)$	steps	amplification	$\\|V^{\pi}\\|_{\infty}$
0.500	2.0	10	2.0	1.5
0.800	5.0	35	5.0	3.0
0.900	10.0	81	10.0	5.4
0.950	20.0	180	20.0	10.3
0.990	100.0	1075	100.0	49.7
0.995	200.0	2293	200.0	98.9

$\rho(A)$	$\\|A\\|_{2}$	Transient peak	at $k$	Tail decay ratio
0.5	2.118	2.12	1	0.5027
0.8	2.281	4.14	4	0.8043
0.95	2.379	15.10	19	0.9552

$\alpha$	$d(\mathrm{good})$	modulus	error at $k=10$	error at $k=60$
0	0.6667	0.7301	4.8193	4.796
0.7048	0.8645	1.0000	6.0646	57.97
1	0.9474	1.2048	37.9147	5.558e+05

actions	$\sigma$	measured gap	$\sigma\,\mathbb{E}[\max_{a}Z_{a}]$	bias at separation $1$
2	0.25	0.1413	0.1409	0.0009
2	1.00	0.5663	0.5637	0.2024
8	0.25	0.3559	0.3558	0.0016
8	1.00	1.4232	1.4233	0.6658
32	0.25	0.5174	0.5171	0.0045
32	1.00	2.0697	2.0684	1.1568

step size	$\sum\alpha_{n}$	$\sum\alpha_{n}^{2}$	final error	conditions
constant, $\alpha=0.05$	1000.0	50.00	0.1711 (0.0220)	$\sum\alpha_{n}^{2}$ diverges
$1/n$	19.5	3.29	1.7125 (0.0235)	both hold
$1/n^{0.7}$	98.8	6.08	0.0450 (0.0057)	both hold
$1/n^{2}$	3.3	2.16	4.0291 (0.0229)	$\sum\alpha_{n}$ converges

A2C/A3C	Advantage Actor-Critic / Asynchronous Advantage Actor-Critic. Policy gradient algorithms that use an advantage function baseline to reduce variance (Section 4 ).
BCQ	Batch-Constrained Q-learning. An offline RL algorithm that restricts the learned policy to actions supported by the behavioral data (Section 11.2 ).
BwK	Bandits with Knapsacks. A bandit framework with resource constraints, where an agent must balance exploration and exploitation subject to a budget (Section 10 ).
CCP	Conditional Choice Probabilities. Choice probabilities conditional on state, used in structural estimation as an alternative to full-solution methods (Section 7 ).
CFR	Counterfactual Regret Minimization. An iterative algorithm for computing Nash equilibria in extensive-form games by minimizing regret at each information set (Section 9 ).
CQL	Conservative Q-Learning. An offline RL algorithm that adds a penalty for overestimating Q-values on out-of-distribution actions (Section 11.2 ).
DDC	Dynamic Discrete Choice. A class of structural econometric models in which agents make sequential discrete decisions under uncertainty (Section 7 ).

Abstract

Reinforcement Learning

Explore similar work

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

A Comparative Study of Dynamic Programming and Reinforcement Learning in Finite Horizon Dynamic Pricing

JaxMARL-BC: Multi-Agent Reinforcement Learning for Macroeconomic Modelling, in Seconds