Payoff-Aware Prediction of Population Game Dynamics
Authors: Kang Wang, Xiao Wang, Bo Li, Renzhe Xu
Organizations: School of Computing and Artificial Intelligence, Shanghai University of Finance and Economics · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics · School of Economics and Management, Tsinghua University
Population game dynamics describe how aggregate behavior evolves in response to payoff signals. In many applications, the payoff function may change, and the goal is to predict the population trajectory induced by a new payoff design before trajectory data under that design are available. We study this problem as payoff-aware prediction of population game dynamics. Standard dynamics learning methods fit the motion observed under training payoffs, but do not separate the payoff-dependent incentive from the payoff-independent response structure needed for prediction under new payoffs. Building on the existing framework of Riemannian game dynamics, we propose PA-RmD, a payoff-aware learning method that keeps payoff information explicit and learns the payoff-independent response structure from data. Theoretically, we show that the learned model preserves fixed-payoff Nash equilibria and derive finite-time one-step prediction bounds under shared-response and payoff-coverage conditions. Empirically, we evaluate PA-RmD on synthetic zero-sum and potential games, as well as a YouTube Trending multi-country dataset, showing that our method performs well under changing payoff designs.
Figures & tables
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(4.05 ± 0.00)E-2
(4.04 ± 0.00)E-1
(5.23 ± 0.00)E-1
(1.09 ± 0.00)E-1
(2.88 ± 0.00)E-1
(3.07 ± 0.00)E-1
SIAR
(5.03 ± 0.00)E-2
(3.05 ± 0.00)E-1
(3.99 ± 0.00)E-1
(5.84 ± 0.00)E-2
(2.11 ± 0.00)E-1
(2.07 ± 0.00)E-1
Neural ODE
(1.45 ± 0.05)E-2
(1.49 ± 0.16)E-1
(2.09 ± 0.28)E-1
(7.07 ± 0.20)E-3
(8.72 ± 0.62)E-2
(1.20 ± 0.07)E-1
LEADS
(1.67 ± 0.18)E-2
(1.75 ± 0.30)E-1
(2.62 ± 0.56)E-1
(7.80 ± 1.24)E-3
(9.71 ± 1.56)E-2
(1.31 ± 0.16)E-1
CoDA- ℓ1
(1.43 ± 0.02)E-2
(1.42 ± 0.13)E-1
(2.05 ± 0.29)E-1
(7.36 ± 0.98)E-3
(9.38 ± 0.94)E-2
(1.26 ± 0.15)E-1
Table 1: Synthetic held-out results under replicator dynamics. Each entry is mean ± std over five runs on held-out payoff functions. In all tables, bold and underlined entries indicate the best and second-best results for each setting and metric, respectively.
Figure 1: Error by payoff function index for the K=4 zero-sum cyclic game under replicator dynamics.
Figure 2: Held-out rollout trajectories under replicator dynamics for the four best methods in the zero-sum cyclic game, ordered from fourth-best to best by rollout RMSE. Each panel shows the model name followed by the rollout RMSE for the illustrated run in parentheses. Gray curves represent ground-truth trajectories, and colored dashed curves represent model rollouts.
Figure 3: YouTube held-out country prediction errors as the number q of training countries varies. One-step errors use observed engagement-derived payoff vectors, while rollout metrics use estimated country-specific payoff proxies.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(3.85 ± 0.00)E-1
(6.83 ± 0.00)E-1
(7.83 ± 0.00)E-1
(8.38 ± 0.00)E-2
(4.65 ± 0.00)E-1
(6.04 ± 0.00)E-1
SIAR
(7.68 ± 0.00)E-1
(6.11 ± 0.00)E-1
(7.16 ± 0.00)E-1
(1.50 ± 0.00)E-1
(2.77 ± 0.00)E-1
(2.79 ± 0.00)E-1
Neural ODE
(3.19 ± 0.41)E-2
(3.55 ± 0.71)E-1
(5.58 ± 1.12)E-1
(1.36 ± 0.20)E-2
(1.83 ± 0.29)E-1
(2.55 ± 0.43)E-1
LEADS
(3.53 ± 0.30)E-2
(4.41 ± 0.74)E-1
(6.88 ± 1.08)E-1
(1.24 ± 0.12)E-2
(1.60 ± 0.21)E-1
(2.24 ± 0.31)E-1
CoDA- ℓ1
(3.12 ± 0.38)E-2
(3.66 ± 0.41)E-1
(5.88 ± 0.56)E-1
(1.30 ± 0.09)E-2
(1.70 ± 0.14)E-1
(2.37 ± 0.20)E-1
Appendix
Table 2: Synthetic held-out results under projection dynamics. Each entry is mean ± std over five runs on held-out payoff functions.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(1.45 ± 0.00)E-1
(5.43 ± 0.00)E-1
(5.78 ± 0.00)E-1
(2.00 ± 0.00)E-1
(3.96 ± 0.00)E-1
(4.83 ± 0.00)E-1
SIAR
(1.24 ± 0.00)E-1
(3.77 ± 0.00)E-1
(4.28 ± 0.00)E-1
(1.30 ± 0.00)E-1
(2.66 ± 0.00)E-1
(2.83 ± 0.00)E-1
Neural ODE
(3.09 ± 0.35)E-2
(3.02 ± 0.77)E-1
(4.57 ± 1.54)E-1
(1.52 ± 0.05)E-2
(1.79 ± 0.10)E-1
(2.55 ± 0.17)E-1
LEADS
(3.28 ± 0.32)E-2
(3.29 ± 0.69)E-1
(5.00 ± 1.21)E-1
(1.58 ± 0.17)E-2
(1.73 ± 0.12)E-1
(2.42 ± 0.15)E-1
CoDA- ℓ1
(2.94 ± 0.12)E-2
(2.66 ± 0.58)E-1
(4.01 ± 1.15)E-1
(1.49 ± 0.18)E-2
(1.66 ± 0.23)E-1
(2.35 ± 0.36)E-1
Appendix
Table 3: Synthetic held-out results under MWU dynamics. Each entry is mean ± std over five runs on held-out payoff functions.
Figure 4: Error by payoff function index for the K=4 potential congestion game under replicator dynamics.
Figure 5: Error by payoff function index for the K=4 zero-sum cyclic game under projection dynamics.
Figure 6: Error by payoff function index for the K=4 potential congestion game under projection dynamics.
Figure 7: Error by payoff function index for the K=4 zero-sum cyclic game under MWU dynamics.
Figure 8: Error by payoff function index for the K=4 potential congestion game under MWU dynamics.
Figure 9: Full held-out rollout trajectories for the K=3 zero-sum cyclic game under replicator dynamics.
Figure 10: Full held-out rollout trajectories for the K=3 potential congestion game under replicator dynamics.
Figure 11: Full held-out rollout trajectories for the K=3 zero-sum cyclic game under projection dynamics.
Figure 12: Full held-out rollout trajectories for the K=3 potential congestion game under projection dynamics.
Figure 13: Full held-out rollout trajectories for the K=3 zero-sum cyclic game under MWU dynamics.
Figure 14: Full held-out rollout trajectories for the K=3 potential congestion game under MWU dynamics.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(2.02 ± 0.00)E-2
(2.02 ± 0.00)E-1
(2.62 ± 0.00)E-1
(7.28 ± 0.00)E-2
(1.92 ± 0.00)E-1
(2.05 ± 0.00)E-1
SIAR
(2.52 ± 0.00)E-2
(1.52 ± 0.00)E-1
(2.00 ± 0.00)E-1
(3.90 ± 0.00)E-2
(1.41 ± 0.00)E-1
(1.39 ± 0.00)E-1
Neural ODE
(7.44 ± 0.19)E-3
(7.70 ± 0.79)E-2
(1.09 ± 0.14)E-1
(4.85 ± 0.12)E-3
(6.04 ± 0.44)E-2
(8.27 ± 0.48)E-2
LEADS
(8.52 ± 0.91)E-3
(9.01 ± 1.52)E-2
(1.35 ± 0.28)E-1
(5.36 ± 0.82)E-3
(6.77 ± 1.04)E-2
(9.11 ± 1.06)E-2
CoDA- ℓ1
(7.29 ± 0.09)E-3
(7.28 ± 0.67)E-2
(1.05 ± 0.15)E-1
(5.07 ± 0.64)E-3
(6.52 ± 0.60)E-2
(8.75 ± 0.99)E-2
Appendix
Table 4: Overall results across both seen and held-out payoff environments under replicator dynamics. Each entry is mean ± std over five runs.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(1.94 ± 0.00)E-1
(3.50 ± 0.00)E-1
(4.04 ± 0.00)E-1
(5.73 ± 0.00)E-2
(3.22 ± 0.00)E-1
(4.16 ± 0.00)E-1
SIAR
(3.88 ± 0.00)E-1
(3.30 ± 0.00)E-1
(3.89 ± 0.00)E-1
(1.02 ± 0.00)E-1
(1.99 ± 0.00)E-1
(2.04 ± 0.00)E-1
Neural ODE
(1.61 ± 0.21)E-2
(1.79 ± 0.36)E-1
(2.81 ± 0.56)E-1
(9.51 ± 1.31)E-3
(1.26 ± 0.19)E-1
(1.76 ± 0.28)E-1
LEADS
(1.80 ± 0.15)E-2
(2.23 ± 0.37)E-1
(3.47 ± 0.54)E-1
(8.62 ± 0.80)E-3
(1.10 ± 0.14)E-1
(1.55 ± 0.21)E-1
CoDA- ℓ1
(1.57 ± 0.19)E-2
(1.84 ± 0.21)E-1
(2.95 ± 0.28)E-1
(9.10 ± 0.59)E-3
(1.17 ± 0.09)E-1
(1.64 ± 0.14)E-1
Appendix
Table 5: Overall results across both seen and held-out payoff environments under projection dynamics. Each entry is mean ± std over five runs.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(7.24 ± 0.00)E-2
(2.72 ± 0.00)E-1
(2.91 ± 0.00)E-1
(1.34 ± 0.00)E-1
(2.65 ± 0.00)E-1
(3.23 ± 0.00)E-1
SIAR
(6.21 ± 0.00)E-2
(1.91 ± 0.00)E-1
(2.18 ± 0.00)E-1
(8.65 ± 0.00)E-2
(1.79 ± 0.00)E-1
(1.91 ± 0.00)E-1
Neural ODE
(1.57 ± 0.17)E-2
(1.55 ± 0.37)E-1
(2.35 ± 0.76)E-1
(1.04 ± 0.04)E-2
(1.24 ± 0.07)E-1
(1.76 ± 0.12)E-1
LEADS
(1.67 ± 0.17)E-2
(1.69 ± 0.35)E-1
(2.57 ± 0.61)E-1
(1.08 ± 0.12)E-2
(1.20 ± 0.08)E-1
(1.67 ± 0.10)E-1
CoDA- ℓ1
(1.49 ± 0.06)E-2
(1.36 ± 0.29)E-1
(2.05 ± 0.57)E-1
(1.02 ± 0.12)E-2
(1.16 ± 0.15)E-1
(1.63 ± 0.24)E-1
Appendix
Table 6: Overall results across both seen and held-out payoff environments under MWU dynamics. Each entry is mean ± std over five runs.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(2.81 ± 0.00)E-6
(3.78 ± 0.00)E-5
(4.67 ± 0.00)E-5
(6.15 ± 0.00)E-5
(9.58 ± 0.00)E-4
(1.21 ± 0.00)E-3
SIAR
(4.06 ± 0.00)E-6
(5.55 ± 0.00)E-5
(7.46 ± 0.00)E-5
(1.23 ± 0.00)E-4
(2.04 ± 0.00)E-3
(2.56 ± 0.00)E-3
Neural ODE
(3.84 ± 1.11)E-4
(5.10 ± 1.58)E-3
(8.13 ± 2.93)E-3
(4.19 ± 0.46)E-4
(6.76 ± 1.45)E-3
(9.14 ± 1.43)E-3
LEADS
(3.35 ± 0.93)E-4
(5.28 ± 1.22)E-3
(7.55 ± 2.57)E-3
(4.78 ± 0.78)E-4
(8.71 ± 1.61)E-3
(1.08 ± 0.19)E-2
CoDA- ℓ1
(2.80 ± 0.68)E-4
(3.99 ± 1.03)E-3
(5.94 ± 1.49)E-3
(4.91 ± 1.18)E-4
(7.85 ± 1.57)E-3
(1.10 ± 0.28)E-2
Appendix
Table 7: Results on seen training payoff environments under replicator dynamics. Each entry is mean ± std over five runs.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(2.55 ± 0.00)E-3
(1.74 ± 0.00)E-2
(2.47 ± 0.00)E-2
(4.20 ± 0.00)E-3
(3.55 ± 0.00)E-2
(3.86 ± 0.00)E-2
SIAR
(8.69 ± 0.00)E-3
(4.88 ± 0.00)E-2
(6.09 ± 0.00)E-2
(7.68 ± 0.00)E-3
(4.43 ± 0.00)E-2
(5.54 ± 0.00)E-2
Neural ODE
(3.68 ± 1.19)E-4
(2.91 ± 0.95)E-3
(3.84 ± 1.30)E-3
(1.26 ± 0.13)E-3
(1.32 ± 0.14)E-2
(1.68 ± 0.18)E-2
LEADS
(5.95 ± 2.30)E-4
(4.75 ± 1.95)E-3
(6.32 ± 2.69)E-3
(1.07 ± 0.09)E-3
(1.13 ± 0.09)E-2
(1.48 ± 0.16)E-2
CoDA- ℓ1
(2.63 ± 1.01)E-4
(1.98 ± 0.85)E-3
(2.70 ± 1.03)E-3
(1.24 ± 0.05)E-3
(1.32 ± 0.07)E-2
(1.65 ± 0.11)E-2
Appendix
Table 8: Results on seen training payoff environments under projection dynamics. Each entry is mean ± std over five runs.
Game
Method
K=3
K=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
Zero-sum cyclic
SINDy
(1.99 ± 0.00)E-4
(1.78 ± 0.00)E-3
(2.83 ± 0.00)E-3
(1.06 ± 0.00)E-4
(2.16 ± 0.00)E-3
(2.35 ± 0.00)E-3
SIAR
(3.84 ± 0.00)E-4
(5.28 ± 0.00)E-3
(7.54 ± 0.00)E-3
(2.98 ± 0.00)E-4
(5.81 ± 0.00)E-3
(6.51 ± 0.00)E-3
Neural ODE
(5.11 ± 2.34)E-4
(7.77 ± 4.44)E-3
(1.23 ± 0.73)E-2
(6.76 ± 0.61)E-4
(1.36 ± 0.17)E-2
(1.60 ± 0.17)E-2
LEADS
(5.31 ± 1.72)E-4
(8.38 ± 3.61)E-3
(1.34 ± 0.64)E-2
(7.89 ± 0.53)E-4
(1.47 ± 0.12)E-2
(1.78 ± 0.15)E-2
CoDA- ℓ1
(4.42 ± 1.25)E-4
(6.54 ± 3.24)E-3
(9.59 ± 4.90)E-3
(8.20 ± 1.94)E-4
(1.52 ± 0.26)E-2
(1.92 ± 0.42)E-2
Appendix
Table 9: Results on seen training payoff environments under MWU dynamics. Each entry is mean ± std over five runs.
Game
Method
Replicator
Projection
MWU
K=3
K=4
K=3
K=4
K=3
K=4
Zero-sum cyclic
LEADS (few-shot)
(2.91 ± 0.37)E-1
(1.12 ± 0.12)E-1
(3.87 ± 2.07)E-2
(4.53 ± 0.45)E-2
(5.84 ± 0.31)E-1
(2.14 ± 0.15)E-1
CoDA- ℓ1 (few-shot)
(1.42 ± 0.13)E-1
(9.22 ± 1.19)E-2
(3.66 ± 0.41)E-1
(1.66 ± 0.21)E-1
(2.58 ± 0.58)E-1
(1.61 ± 0.23)E-1
CoDA- ℓ2 (few-shot)
(1.41 ± 0.12)E-1
(9.34 ± 0.99)E-2
(3.47 ± 1.09)E-1
(1.68 ± 0.21)E-1
(2.58 ± 0.58)E-1
(1.59 ± 0.22)E-1
PA-RmD (zero-shot)
(2.12 ± 0.73)E-2
(1.73 ± 0.09)E-2
(1.37 ± 0.06)E-1
(1.08 ± 0.06)E-1
(3.04 ± 0.29)E-2
(2.58 ± 0.11)E-2
Potential congestion
LEADS (few-shot)
(2.55 ± 0.48)E-2
(2.22 ± 0.15)E-2
(1.04 ± 0.24)E-2
(1.56 ± 0.10)E-2
(3.17 ± 0.60)E-2
(3.99 ± 0.17)E-2
Appendix
Table 10: Rollout RMSE on held-out payoffs across three dynamics. Each entry is mean ± std over five runs.
Method
q=3
q=4
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
One-step ( ↓ )
Rollout RMSE ( ↓ )
Rollout@ T ( ↓ )
SINDy
(9.53 ± 0.00)E-1
(9.67 ± 0.00)E-1
(9.70 ± 0.00)E-1
(9.91 ± 0.00)E-1
(1.01 ± 0.00)E+0
(1.03 ± 0.00)E+0
SIAR
(3.91 ± 0.00)E-1
(8.06 ± 0.00)E-1
(8.30 ± 0.00)E-1
(2.92 ± 0.00)E-1
(6.69 ± 0.00)E-1
(6.77 ± 0.00)E-1
Neural ODE
(1.13 ± 0.05)E-1
(3.41 ± 0.26)E-1
(4.03 ± 0.38)E-1
(9.94 ± 0.29)E-2
(3.31 ± 0.28)E-1
(3.89 ± 0.45)E-1
LEADS
(1.81 ± 0.09)E-1
(6.55 ± 0.26)E-1
(7.18 ± 0.40)E-1
(1.57 ± 0.12)E-1
(6.08 ± 0.63)E-1
(6.67 ± 0.77)E-1
CoDA- ℓ1
(1.13 ± 0.01)E-1
(3.26 ± 0.12)E-1
(3.74 ± 0.17)E-1
(1.05 ± 0.04)E-1
(3.25 ± 0.10)E-1
(3.77 ± 0.14)E-1
Appendix
Table 11: YouTube experiment results on held-out countries as q varies. Each entry is mean ± std across five independent repetitions, with results in each repetition averaged over ten randomly sampled country splits. One-step errors use observed engagement-derived payoff vectors, while rollout metrics use estimated country-specific payoff proxies.
Game
Method
α=0
α=0.4
α=0.8
α=1.6
α=3.2
SINDy
(2.88 ± 0.00)E-1
(5.76 ± 0.00)E-1
(6.49 ± 0.00)E-1
(7.47 ± 0.00)E-1
(7.37 ± 0.00)E-1
SIAR
(2.11 ± 0.00)E-1
(2.41 ± 0.00)E-1
(2.77 ± 0.00)E-1
(3.29 ± 0.00)E-1
(4.65 ± 0.00)E-1
Neural ODE
(8.72 ± 0.62)E-2
(2.15 ± 0.18)E-1
(3.67 ± 0.40)E-1
(5.82 ± 0.23)E-1
(7.27 ± 0.14)E-1
LEADS
(9.71 ± 1.56)E-2
(1.95 ± 0.10)E-1
(3.12 ± 0.47)E-1
(5.10 ± 0.72)E-1
(7.20 ± 0.35)E-1
CoDA- ℓ1
(9.38 ± 0.94)E-2
(1.89 ± 0.15)E-1
(3.21 ± 0.21)E-1
(5.45 ± 0.35)E-1
(7.35 ± 0.27)E-1
CoDA- ℓ2
(9.34 ± 0.99)E-2
(1.79 ± 0.13)E-1
(3.29 ± 0.15)E-1
(5.44 ± 0.27)E-1
(7.44 ± 0.24)E-1
Appendix
Table 12: Rollout RMSE under response perturbations. Each entry is mean ± std over five runs.
We examine the interplay between ordinal, preference-based solution concepts in games and the long-run behavior of game dynamics, asking in particular to what extent the combinatorial data of a game -- its preference graph -- determine the outcomes of no-regret learning dynamics -- such as follow-the-regularized-leader (FTRL). In one direction, we show that the skeleton of every dynamically stable set (i.e. the set of pure profiles it contains) must also be preferentially stable, that is, it must be closed under profitable deviations. We then ask the converse question: when do preferences determine the long-run behavior of the players' learning dynamics? We begin by showing that preferences characterize asymptotic stability in the case of subgames -- i.e. subsets of pure profiles obtained by restricting players' action sets. Beyond this case however, the equivalence between dynamic and preferential stability collapses: concretely, we construct a three-player game with a preferentially stable set whose span is dynamically unstable, showing in this way that preferences do not suffice as a criterion of dynamic stability. We then bridge this gap via the notion of resilience under aggregate deviations, an easy-to-check payoff-based condition that guarantees asymptotic stability of arbitrary spans of pure strategies.
Omar Abbadi, Rida Laraki, Panayotis Mertikopoulos
Univ. Mohammed VI Polytechnic, CMSIS, Moroccan Center for Game Theory, 11103 Rabat, Morocco. · Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France.
Collective responses depend on individual differences, contact opportunities, and accumulated experience. Learning their dynamics from aggregate counts requires connecting a population's response distribution to both current observations and future behavior. We introduce Differentiable Gaussian Dynamics (DGD), which learns this connection through three components: a Gaussian mixture representing heterogeneous response propensities, differentiable aggregation of contact intensity and behavioral probabilities, and feedback recurrence that updates subsequent responses. Reparameterized integration and temporal recurrence let aggregate prediction errors jointly train the distribution, observation functions, and feedback parameters. On four windows from KuaiRand-Pure and Online Retail II, DGD achieves lower joint behavioral negative log-likelihood than a DeepAR adaptation with a joint-behavior head. In Retail 2010, its one-day behavioral-count MAE is 4.71 versus 6.88 for this adaptation. Learning the distribution reduces behavioral negative log-likelihood by 10.82% relative to a fixed Gaussian in KuaiRand's standard-recommendation window; removing feedback dynamics raises joint KL from 0.0340 to 0.2577 in a controlled experiment. These results establish the value of learning population representations and their feedback process from aggregate observations. Code is available at https://github.com/OranAi-Ltd/oransim.
Jianxiang Ma, Mingfu Zhang, Xiaocui Yang +3
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · OranAI, Shenzhen 518057, China · OranAI Ltd., City of Industry, CA 91748, USA
Outcome-predictive signals are often treated as promising optimization targets in reinforcement learning, but predictive validity and optimization utility are different claims. We expose this distinction in GuanDan, a multi-round imperfect-information game in which the implication of a round outcome depends on the public match state. Held-out analysis shows that state--gain interactions provide incremental predictive value beyond a constant signed-gain effect, establishing that match progress is predictively meaningful. We then use changes in a frozen match-win predictor, ΔΦ, as a controlled round-level supervision intervention in a matched DanZero-style Monte Carlo self-play protocol. The intervention can succeed in an individual lineage: a selected progress-augmented checkpoint substantially outperforms its matched Baseline on a locked evaluation. Yet the method-level result reverses this impression: none of five preregistered replication seeds is positive, the pooled ten-seed interval includes zero, and across-iteration effects repeatedly change sign without a common late-stage advantage. These results identify a predictive--optimization gap: under the studied learner, a signal can describe outcome-relevant progress while failing to provide a reliably beneficial optimization direction.
Qicheng Zou, Biao Yuan, Siyu Song +2
the University of California San Diego, La Jolla, CA, USA. · Chabiyue (Shanghai) Information Technology Co., Ltd., Shanghai, China.