When Is a Game a Markov Decision Process? Correspondences Between Game Theory and Reinforcement Learning
Organizations: DIGNUM Centre, Universidad Pontificia Comillas, Madrid, Spain
Abstract
Reinforcement learning and game theory describe the same object from two sides. Game theory says what a rational outcome of a strategic interaction is; reinforcement learning says how an agent could reach one through experience. The two literatures nevertheless keep separate vocabularies for identical structures, and the treatments that bridge them present the correspondence as analogy rather than theorem. This monograph supplies the theorems. A game is a Markov decision process in three distinct senses, and every table entry is labelled with the one that applies: a definitional identity, an exact reduction carried by a value- and solution-preserving map, or a structural analogy that fails under identifiable conditions. We take every standard class of game, from normal-form through repeated, stochastic, partially observable, extensive-form with imperfect information, Bayesian, graphical, Stackelberg, potential, coalitional, mean-field and evolutionary games, and identify for each the decision-theoretic model that reinforcement learning already studies under another name. The same treatment is applied to solution concepts: Nash equilibrium and the fixed point of the Bellman optimality operator, quantal response equilibrium and the soft Bellman operator, the Shapley value and credit assignment, Blackwell approachability and no-regret learning. We map each model class onto the training procedures sound for it and record the complexity of the associated solution problems. A final part collects the correspondences that do not hold: the non-stationarity that invalidates the independent-learner reduction, the impossibility results that forbid uncoupled convergence to Nash equilibrium, and the gap between the coarse correlated equilibria that no-regret self-play reaches and the correlated equilibria it is credited with.
Figures & tables
| Osborne and Rubinstein | This work | How to read the pair |
|---|---|---|
| Player set | Agent set | Finite; in both |
| Action set | Identical object, different font | |
| Action profile | Identical; both write | |
| Preference relation | Payoff | We assume a representation and fix it |
| Strategic game | The same tuple | |
| Mixed strategy | is common to both |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Decision node of a one-player tree | State , or history | Where you are and what you may do next |
| Move at a node | Action | The choice under your control |
| Chance node | Transition kernel , observation kernel | The choice not under your control |
| Payoff at a leaf | Discounted return | What is being maximised |
| Pure strategy | Deterministic policy | A complete contingent plan |
| Behavioural strategy | Stochastic policy | A plan that may randomise locally |
| Single-agent object | Multi-agent generalisation | Developed in |
|---|---|---|
| Action | Joint action | Section 3 |
| Policy | Policy profile | Section 3 |
| Reward | Reward profile | Section 7 |
| Greedy improvement | Best-response correspondence | Section 4 |
| Fixed point of | Nash equilibrium of the induced game | Section 4 |
| Dominated action | Strictly dominated strategy; rationalisability | Section 5 |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Agent, player | Agent, learner, policy head | One decision maker, one reward signal |
| Action set | Action space of agent | The columns of this agent’s slice of the tensor |
| Joint action | Joint action; one environment step | One cell of the payoff tensor |
| Payoff | Reward for a one-step episode | The number agent wants to be large |
| Mixed strategy | Stochastic policy over | A row of logits after a softmax |
| Strategy profile | Joint policy, product policy | What every agent does at once |
| Model | Full payoff feedback | Bandit feedback | With public state |
|---|---|---|---|
| Known objective; one-shot maximisation | Multi-armed bandit | Contextual bandit | |
| , common payoff | Team coordination problem | Cooperative bandit; distributed optimisation | Team Markov game with one state |
| , zero-sum | Matrix game; value by linear programming | Adversarial bandit; Exp3 and regret matching | Zero-sum game with public state |
| , general sum | Normal-form game; equilibrium is PPAD -complete | No-regret learning converging to | Stateless general-sum game with context |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Best response | Greedy policy improvement; | Do the best you can, others held fixed |
| Nash equilibrium | Fixed point of simultaneous improvement | Nobody gains by moving alone |
| -equilibrium | -greedy improvement; approximate PI | Nobody gains more than alone |
| Mixed strategy | Stochastic policy | Randomisation as part of the solution |
| Correlated profile | Policy conditioned on a shared random seed | A joint distribution, not a product |
| Mediator, recommendation | Shared signal, broadcast latent variable | A common random bit that both read |
| Game-theoretic concept | Reinforcement learning counterpart | Degeneracy at | |
|---|---|---|---|
| Best response | Greedy policy improvement; argmax over | Greedy policy | D |
| Nash equilibrium | Fixed point of simultaneous policy improvement | Optimal policy | D |
| -Nash | Approximate policy iteration; -greedy improvement | -optimal policy | D |
| Markov perfect equilibrium | Stationary optimal policy of each agent given the others | Optimal stationary policy | D |
| Subgame perfect equilibrium | Bellman principle of optimality; backward induction | Optimal policy | D |
| Maximin / security value | Robust objective, worst-case return | E |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Strictly dominated strategy | Action that is never greedy for any opponent policy | Safe to delete from the action space |
| Weakly dominated strategy | Action never strictly better, sometimes worse | Unsafe to delete; order of deletion matters |
| Iterated elimination (IESDS) | Iterative action masking | Prune, then prune again given the pruning |
| Rationalisable strategy | Survivor of unbounded best-response recursion | What a rational agent could justify playing |
| Common knowledge of rationality | Unbounded recursion depth in modelling | I know that you know that I know |
| Level- agent | Opponent model unrolled times | Best response to a level- model |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Stage game | One-step environment; inner reward | The game played every period |
| History | Trajectory prefix; recurrent hidden state | Everything observed so far |
| Strategy | History-dependent policy; recurrent policy | A plan contingent on the past |
| Discount factor | Discount factor | Patience; also the effective horizon |
| Perfect monitoring | Full observability of the joint action | You see what everybody did |
| Imperfect public monitoring | Shared noisy observation of the joint action | You see a common signal, not the action |
| Monitoring | Induced model | Policy class needed | Sustainable outcomes |
|---|---|---|---|
| None (one shot) | Normal-form game; bandit if | Stateless | Stage Nash equilibria |
| Perfect, finite horizon | Stochastic game on the history tree | Finite memory | Stage Nash by backward induction if unique |
| Perfect, discounted infinite | Stochastic game on the history tree | Unbounded or recurrent memory | All of ( Theorem 6.2 ) |
| Imperfect public | POSG with public state | Memory over public signals | Perfect public equilibria; self-generating sets ( Abreu et al., 1990 ) |
| Imperfect private | POSG with private histories | Belief over others’ histories | Sequential equilibria; hard |
| Fixed finite-automaton opponents | MDP over product of automaton states | Markov in the product state | Best response, computable |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Stochastic game, Markov game | Multi-agent Markov decision process | The same tuple under two names |
| State | State; observation when fully observed | What the transition kernel depends on |
| Stationary strategy | Policy of agent | A row of probabilities per state |
| Value of the game at | Discounted return of agent from | |
| Stage game | Joint-action critic cross-section at | Reward plus discounted continuation |
| Solution operator | What replaces in the backup | , , or a Nash selection |
| Structure | Sound reduction | Stage operator | Contraction | Algorithm |
|---|---|---|---|---|
| Identity; it is an MDP | Yes | Q-learning | ||
| Common payoff, centralised | Joint-action MDP | over | Yes | Joint-action Q-learning |
| Common payoff, decentralised | None exact; see Remark 7.4 | plus tie-breaking | Given a selection rule | Distributed Q, FMQ |
| Two-player zero-sum | Minimax-Bellman operator; a robust MDP when the adversary is rectangular ( Proposition 7.8 ) | (minimax) | Yes | Minimax-Q |
| Two-player constant-sum | Affine reparametrisation of zero-sum | Yes | Minimax-Q | |
| General sum | None; existence only | Nash selection | No | Nash-Q, CE-Q |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Partially observable stochastic game | General-sum partially observable MARL | Many agents, private observations, own rewards |
| Common-payoff POSG | Dec-POMDP | The same tuple; one shared reward |
| Jointly observable | Dec-MDP | Together the agents see the state |
| Private information | Agent ’s action-observation history | What only agent knows |
| Common information | What is public; broadcast channel contents | What everybody knows everybody knows |
| Interactive belief hierarchy | Nested opponent models (I-POMDP) | Beliefs about beliefs about beliefs |
| Model | Rewards | Finite-horizon complexity | |||
|---|---|---|---|---|---|
| MDP | single agent | yes | yes | MDP | P -complete |
| MMDP | common | yes | yes | MDP | P in |
| SG (Markov game) | general | yes | yes | MDP | P if zero-sum; PPAD -complete if general-sum |
| POMDP | single agent | no | no | POMDP | PSPACE -complete |
| Dec-MDP | common | no | yes | MDP | NEXP -complete (horizon ) |
| Dec-POMDP | common | no | no | POMDP | NEXP -complete (horizon ) |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Common knowledge | Public features; broadcast observations | What conditioning is legitimate on |
| Prescription | Decision rule mapping private history to action | A function, not an action |
| Coordinator | Centralised controller issuing prescriptions | A fictitious agent who sees only the public part |
| Common-information belief | Posterior over state and private histories | The coordinator’s belief state |
| Occupancy state | Distribution over state and joint history | The sufficient statistic of a Dec-POMDP |
| Public belief state | Public state plus per-agent range beliefs | The zero-sum counterpart of |
| Source model | Reduced state | Reduced action | Target | Implemented by |
|---|---|---|---|---|
| POMDP | Belief | Action | MDP | Recurrent and belief-conditioned policies |
| Partial history sharing | Common-information belief | Prescription | POMDP | Coordinator architectures |
| Dec-POMDP | Occupancy state | Joint decision rule | Deterministic MDP | Point-based oMDP solvers; BAD |
| 2p zero-sum imperfect info | Public belief state | Policy prescription | Perfect-info game | ReBeL; Student of Games |
| Dec-POMDP, group common knowledge | Common knowledge of a subset | Prescription for that subset | Hierarchical POMDP | MACKRL ( Schroeder de Witt et al., 2019 ) |
| Dec-POMDP at stage | Type distribution from | Joint type-action map | Bayesian game | Bayesian-game approximation |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Graphical game | Coordination graph; factored MARL | Payoffs depend on neighbours only |
| Neighbourhood | Scope of a local payoff factor | Who can affect agent directly |
| Local payoff | Local utility term | One summand of the joint value |
| Polymatrix game | Pairwise-decomposable joint reward | Sum of two-agent games on edges |
| Treewidth of the interaction graph | Treewidth of the coordination graph | What makes elimination cheap |
| Variable elimination | Max-plus, message passing, DCOP | Compute the argmax without enumerating |
| Class | Structure assumed | RL counterpart | What it buys |
|---|---|---|---|
| General normal form | None | Joint-action critic | Nothing; entries |
| Graphical game | Payoffs depend on | Scoped critic factors | Representation: per agent |
| Polymatrix game | Payoffs sum over edges | Pairwise value decomposition | Representation; if zero-sum, a minmax theorem and polynomial-time equilibria ( Cai and Daskalakis, 2011 ) |
| Coordination graph | Value sums over hyperedges | Max-plus, DCG | Argmax by elimination ( Proposition 10.2 ) |
| Congestion game | Cost depends on load only | Shared-resource environment | Pure equilibrium exists; potential ( Rosenthal, 1973 ) |
| Anonymous game | Payoff depends on action counts | Permutation-invariant critic | Representation linear in |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Node of the tree | Full history; state-trajectory prefix | A position including hidden information |
| Information set | Action-observation history | What the acting agent can distinguish |
| Chance node | Transition or observation kernel | Nature’s move with a known distribution |
| Behavioural strategy | Policy | Local randomisation at each decision |
| Mixed strategy | Distribution over deterministic policies | Randomise once, over whole plans |
| Perfect recall | The policy sees the full own history | You never forget what you did |
| Extensive form | Reinforcement learning | Status |
|---|---|---|
| Node of the tree | State-history pair; trajectory prefix | Definitional |
| Information set | Action-observation history | Exact under perfect recall |
| Chance node | Stochastic transition or observation | Definitional |
| Behavioural strategy | Policy | Definitional |
| Mixed strategy | Distribution over deterministic policies | Equivalent by Theorem 11.4 |
| Perfect recall | Policy conditions on full own history | Definitional |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Type | Latent task parameter; opponent identity | Private information determining preferences |
| Type space | Task family; model class | The range of what the unknown can be |
| Common prior | Task distribution; meta-training distribution | The shared belief before anything is seen |
| Harsanyi transformation | State augmentation with the latent parameter | Make the unknown part of the state |
| Interim payoff | Value conditional on the posterior | After you learn your own type |
| Ex-ante payoff | Expected return over the task distribution | Before anything is drawn |
| Game theory | Reinforcement learning | Status and caveat |
|---|---|---|
| Type | Latent task parameter; opponent model index | Definitional |
| Common prior | Task distribution; meta-training distribution | Definitional |
| Harsanyi transformation | Augmenting the state with the latent parameter | Exact |
| Bayes-Nash equilibrium | Bayes-optimal policy when | Exact at |
| Interim payoff | Value conditional on the posterior | Definitional |
| Ex-ante payoff | Expected return over the task distribution | Definitional |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Message | Communication action; emitted token | An action whose only effect is on others’ inputs |
| Cheap talk | Costless differentiable channel | Messages do not enter the payoff directly |
| Babbling equilibrium | A channel the agents ignore | Messages uncorrelated with anything |
| Informative equilibrium | A learned protocol that is used | Messages correlated with private state |
| Signalling game | Costly communication action | The message has a price that differs by type |
| Separating equilibrium | Protocol that reveals the type | Different types send different messages |
| Structure | RL counterpart | Equilibrium consequence | What decides |
|---|---|---|---|
| No channel | Independent policies | Stage equilibria only | Nothing |
| Cheap talk, common payoff | Differentiable channel, shared reward | Fully informative equilibria exist; so does babbling | Equilibrium selection |
| Cheap talk, biased | Channel with misaligned rewards | Only coarse partitions ( Theorem 13.3 ) | The bias |
| Costly signalling | Bandwidth budget; action slot | Separating equilibria can be unique | Cost differential by type |
| Bayesian persuasion | Teacher, curriculum, information design | Sender prefers noisy disclosure | Commitment power |
| Public correlation device | Shared random seed at reset | Correlated equilibrium ( Definition 4.5 ) | Availability of a seed |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Leader, first mover | Outer loop; slow timescale parameters | Commits before the other chooses |
| Follower | Inner loop; fast timescale parameters | Best responds to what it observes |
| Best-response correspondence | Inner argmax; converged inner solve | The follower’s reaction |
| Strong Stackelberg equilibrium | Bilevel optimum with favourable tie-break | Leader resolves follower indifference |
| Value of commitment | Gain from timescale separation | What acting first is worth |
| Implicit function theorem | Differentiating through the inner optimum | How the leader’s gradient is computed |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Mechanism | Reward function plus environment rules | The game the designer writes |
| Social choice function | Specification; desired behaviour | What the designer wants to happen |
| Type report | Self-reported capability or preference | What an agent tells the designer |
| Incentive compatibility | Reward hacking does not pay | Honest behaviour is optimal |
| Dominant-strategy IC | Optimal regardless of the others | Robust to any opponent behaviour |
| Individual rationality | Participation is better than opting out | Nobody is worse off for taking part |
| Mechanism | Guarantee | RL counterpart | Price paid |
|---|---|---|---|
| Second price | Dominant-strategy truthful | Reward independent of own report | Revenue below first price in some settings |
| VCG ( Vickrey, 1961 ) | Truthful and welfare maximising | Difference reward; counterfactual baseline | Not budget balanced |
| Myerson optimal | Revenue maximising in expectation | Reward shaped by a reserve threshold | Efficiency is sacrificed |
| Majority voting | Simple and anonymous | Ensemble reward by vote | Not strategy-proof ( Theorem 15.6 ) |
| Scalarised aggregation | Single scalar reward | Bradley-Terry reward model | Independence of irrelevant alternatives |
| Pigouvian toll | Shifts equilibrium to the optimum | Reward shaping that is not potential-based | Changes the equilibrium set |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Exact potential | A single scalar all agents hill-climb | Unilateral payoff differences match |
| Finite improvement property | Better-reply dynamics terminate | No cycles; a pure equilibrium exists |
| Markov potential game | Class where independent PG converges | The sequential version of the above |
| Supermodular game | Monotone best responses | More of one agent’s action invites more of the other’s |
| Strategic complements | Positive feedback between agents | Topkis’ lattice structure |
| Difference reward | Counterfactual baseline; COMA advantage | Team return minus your absence |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Characteristic function | Return with the agents outside frozen | What a subgroup achieves on its own |
| Grand coalition | Team return of the full joint policy | The number to be divided |
| Imputation | Per-agent credit vector | What each agent is told it earned |
| Efficiency | Credits sum to the realised return | No surplus invented or lost |
| Null-player axiom | Zero credit for a no-op agent | Doing nothing earns nothing |
| Symmetry | Interchangeable agents get equal credit | Credit depends on contribution, not label |
| Concept | Definition in one line | RL use | Property for learning |
|---|---|---|---|
| Difference reward | Return minus return without you | COMA-style baseline | Cheap; does not sum to the return |
| Shapley value | Order-averaged marginal contribution | Per-agent credit | Efficient, unique, continuous; terms |
| Banzhaf index | Unweighted marginal contribution | Cheaper credit proxy | Not efficient |
| Core | No coalition can improve | Stability of a federation | May be empty ( Example 17.3 ) |
| Nucleolus | Lexicographic min-max excess | Least-objectionable division | Always exists; LP sequence |
| Nash bargaining | Maximise | Welfare objective across agents | Scale invariant; ignores risk attitude |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Fictitious play | Best response to the empirical opponent | Average what you have seen, then be greedy |
| Best-response dynamics | Greedy policy iteration in self-play | Jump straight to the argmax |
| Better-reply dynamics | Any improving step | Only go uphill, not necessarily to the top |
| Multiplicative weights | Softmax over cumulative payoffs; MWU | Exponentiate the running total |
| No-regret property | Sublinear regret of the learner | Asymptotically as good as the best fixed action |
| Blackwell approachability | Vector-valued target set | Steer an average vector into a set |
| Dynamics | Converges in | Limit | RL counterpart |
|---|---|---|---|
| Best-reply dynamics | Potential, supermodular games | Pure equilibrium, iterate | Greedy policy iteration in self-play |
| Fictitious play | Zero-sum; potential ( Robinson, 1951 ; Monderer and Shapley, 1996a ) | Nash, average | Neural fictitious self-play |
| Multiplicative weights | All games, in the sense of regret | , average; value in zero-sum | Softmax policy gradient |
| Regret matching ( Hart and Mas-Colell, 2000 ) | All games | , average | CFR at an information set |
| Approachability ( Blackwell, 1956 ) | All games, vector targets | Average vector in the target set | Constrained policy optimisation |
| Log-linear learning ( Blume, 1993 ) | Potential games | Potential maximiser; risk dominant | Boltzmann exploration with annealing |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Population share | Policy probability of action | The same number in both readings |
| Replicator dynamics | Vanishing-step-size limit of the update | Grow what is above average |
| Replicator-mutator | Softmax update with entropy | Selection plus diffusion |
| Fitness | Action value | Payoff against the current population |
| Rest point | Converged training run | No component is growing |
| Evolutionarily stable strategy | Locally attracting, invasion-proof policy | Resists a small population of mutants |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Representative agent | The single policy being trained | One agent stands for all of them |
| Population distribution | Empirical state distribution of the fleet | What the others look like in aggregate |
| Exchangeability | Permutation-invariant architecture | Identities do not matter, counts do |
| Mean-field game equilibrium | Fixed point of solve-then-rollout | Policy optimal against the flow it creates |
| Mean-field control | Markov decision process on distributions | One planner for the whole population |
| Monotonicity condition | Congestion-like interaction | Crowds repel, so the fixed point is unique |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Input size | Size of the explicit model | What the exponent is measured against |
| Planning problem | Model known; compute a solution | No interaction needed |
| Learning problem | Model unknown; interact and approximate | Sample complexity, not time |
| P -complete | Solvable by linear programming | Easy, and not parallelisable in general |
| PPAD -complete | Fixed point guaranteed by a parity argument | Hard, but not NP -hard |
| PLS -complete | Local search with an improving step | Each step is cheap; the path is long |
| Problem | Target | Complexity | Source |
|---|---|---|---|
| MDP, flat, discounted | Optimal policy | P -complete | Papadimitriou and Tsitsiklis (1987) |
| POMDP, finite horizon | Optimal history-dependent policy | PSPACE -complete | Papadimitriou and Tsitsiklis (1987) |
| POMDP, infinite horizon | Policy existence; -optimality, undiscounted | Undecidable | Madani et al. (2003) |
| MMDP, flat | Optimal joint policy | P in | Boutilier (1996) ; complexity via Papadimitriou and Tsitsiklis (1987) |
| Dec-MDP, | Optimal joint policy | NEXP -hard; complete if horizon | Bernstein et al. (2002) |
| Dec-POMDP, | Optimal joint policy | NEXP -hard; complete if horizon | Bernstein et al. (2002) |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Reward structure | Shared, opposed or mixed-motive rewards | Decides which reduction is sound |
| Information structure | Who observes what, and what is public | Decides whether a sufficient statistic exists |
| Execution architecture | Centralised or decentralised at deployment | Decides whether factorisation is needed |
| Shared randomisation | A common seed available at run time | Decides whether targets are executable |
| Opponent class | Fixed, co-trained, or unknown partners | Decides whether the problem is single-agent |
| Population size | Number of interacting agents | Decides whether the mean-field limit applies |
| Model | Procedure | Target | Key reference | |
|---|---|---|---|---|
| MDP | Tabular methods under Robbins-Monro conditions | Optimal policy | T | Sutton and Barto (2018) |
| MMDP, central execution | Single-agent method on joint actions | Optimal joint policy | T | Boutilier (1996) |
| MMDP, local interaction | Coordination graph; variable elimination | Exact joint argmax | T | Guestrin et al. (2001) ; Kok and Vlassis (2006) |
| Dec-POMDP, IGM-representable | Value factorisation (VDN, QMIX, QPLEX) | Decentralised greedy play is jointly greedy ( Proposition 16.8 ) | T | Rashid et al. (2018) |
| Dec-POMDP, general | Centralised critic, decentralised actors | Local optimum; a Nash profile | H | Lowe et al. (2017) ; Yu et al. (2022) |
| Dec-POMDP, exact | Occupancy-MDP search | Optimal joint policy | T | Dibangoye et al. (2016) |
| Target | Measure | Failure detected |
|---|---|---|
| Minimax value | Exploitability; gap to best response | Cycling; overfitting to the self-play opponent |
| Nash equilibrium, | NashConv, the summed unilateral gain | Convergence to a non-equilibrium rest point |
| Coarse correlated equilibrium | Maximum constant-deviation gain under empirical play | Regret accumulating despite flat return |
| Correlated equilibrium | Maximum swap-deviation gain | Perfectly correlated play that no constant deviation beats ( Example 4.7 ) |
| Optimal joint policy | Gap to a centralised planner on the same model | Equilibrium selection failure; unrepresentable |
| Decentralised convention | Cross-play return across independent runs | Hidden handshakes and arbitrary conventions |
| Game theory | Reinforcement learning | How to read the pair |
|---|---|---|
| Non-stationary environment | Others are learning; the kernel moves | Proposition 7.11 loses its hypothesis |
| Uncoupled dynamics | Each agent sees only its own payoff | The only realistic information structure |
| Cyclic equilibrium | A cycle of policies, not a point | What value iteration reaches in general sum |
| Last-iterate divergence | Final checkpoint is not the output | Only the average converges |
| Representational failure | Hypothesis class excludes the truth | More data does not help |
| Equilibrium multiplicity | Many answers, no selection rule | “Converged” is not a performance claim |