World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits
Organizations: Ideal School and College · University of the West of England, Bristol
Abstract
Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small number of real audits more informative? We derive an exact pairwise simulation identity and a relative bound separating common-mode Bellman residuals from policy-specific error. The analysis motivates RAVEL (Relative Auditing of Value Errors in Learned models), a selective policy tournament whose eliminations use held-out real outcomes. We test 45 SAC, TD3, and PPO policies per task in DMC cheetah-run and walker-walk. At rollout horizon 500, model rankings remain strong (Kendall tau = 0.962 and 0.800), and same-run differential-error ratios are 0.674 and 0.106. Yet mean paired audit-error correlations are only 0.007 and 0.090, and pairing changes standard deviation by -2.5 percent and +1.9 percent. Simultaneous intervals attain 99.98 percent and 99.75 percent coverage, but certify only 13.2 percent and 0 percent of decisions at eight audits. RAVEL improves low-budget walker-walk regret, whereas real-only successive halving is better at the largest budget, rejecting the precommitted twofold efficiency target. Frozen-model optimization lowers real return without increasing predicted return on average. Thus ranking fidelity, useful audit covariance, and upward model exploitation are distinct properties, separated in part by closed-loop trajectory divergence.
Figures & tables
| Task | H1 ratio | H2 saving | Min. coverage | H1/H2/H3/H4 | |
|---|---|---|---|---|---|
| cheetah-run | 0.962 | 0.674 | 1.000 | 0.9995 | |
| walker-walk | 0.800 | 0.106 | 1.333 | 0.9927 |
| Method / audit budget | cheetah-run | walker-walk |
|---|---|---|
| FQE / 0 | .18884 | .13064 |
| Ensemble LCB / 0 | .00261 | .06426 |
| OPCC / 0 | .00261 | .06426 |
| Raw model (also RAVEL at ) | .00044 | .06878 |
| RAVEL-P / 2,4,8 | .00313 / .00309 / .00008 | .03221 / .01890 / .01833 |
| RAVEL-U / 2,4,8 | .00313 / .00309 / .00008 | .03221 / .01890 / .01833 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers