Sampling headroom is not selection gain: a compute-value audit of test-time scaling for video world models
Organizations: Tsinghua University · University of Technology Sydney
Abstract
Test-time scaling (TTS) can improve generation only when additional compute produces better candidates and the system can reliably identify them. This distinction is especially important for video world models, where a wider sample pool may contain stronger rollouts without improving the output that is ultimately selected. We introduce the Compute-Value Audit (CVA), a sequential framework that asks whether extra sampling creates opportunity, observable signals provide a reliable state, that state supports a beneficial action, and the resulting gain exceeds the full entry fee of generation and verification. On 192 Physics-IQ scenes, expanding the pool from 4 to 16 candidates increases oracle quality by +9.23 IQ (95% CI [+7.44, +11.14]), but Flow, Cycle, and VideoReward fail to recover this headroom reliably. Across three generators, none of twelve adaptive-depth policies outperforms uniform compute; they recover only 42-69% of the measured entry fee. A matched-60-NFE Predict-and-Perturb intervention on VideoPhy2 is likewise negative across three fresh-seed replicas. These negative results are not universal: anchor-explorer passes all four stages in a sparse PRM800K setting, MMLU-Pro exposes the gap between predictive state and useful action, and a privileged paired future establishes a positive video upper bound. Together, these results show that sampling headroom has deployment value only when it can be converted into a reliable decision whose benefit survives the complete compute charge. Code is available at https://github.com/YuhuaJiang2002/sampling-headroom-is-not-selection-gain.
Figures & tables
| Method | Extra info | Wan2.2-5B | ABot-14B | Cosmos-Nano |
|---|---|---|---|---|
| Baseline (no-TTS) | – | 25.07 | 40.52 | 27.72 |
| Proprio (flow) | – | 25.30 | 41.07 | 31.31 |
| VideoReward | External RM | 25.76 | 39.62 | 29.89 |
| Native verifier (zero-memory) | Zero memory | 27.53 | 41.57 | 33.70 |
| CVA-Select † | LOFO history | 30.81 | 44.01 | 34.98 |
| Paired-future bound ‡ | Paired future | 32.19 | 45.67 | 36.83 |
| Test | Estimate [95% CI] | Interpretation |
|---|---|---|
| Baseline (no-TTS; greedy) | Acc. | Strong base |
| Sampled-anchor policy uniform | Greedy unnecessary | |
| Agreement signal | Acc./value | State routes depth |
| Policy token uniform/random | Exact-cost gain |
| State / action | Held-out result | Matched check | Gate |
|---|---|---|---|
| Target-oracle router | pp advantage | Evaluation-only routing upper bound | |
| Structured-set predictor | Negligible held-out gain | Matched random assignment | |
| Transfer / supervision | No reliable held-out gain | Refit, transfer, and action-label gates | |
| Partial-DiT preview † | Degrades quality | Candidate-specific preview and refit |
| Method | Wan2.2-5B | ABot-14B | Cosmos3-Nano |
|---|---|---|---|
| Baseline (no-TTS; ) | |||
| Proprio (flow) | |||
| VideoReward | |||
| CVA-Select (ours) | 0.7698 | 0.7725 | 0.7758 |
| Route | IQ ( ) | IQ | Opportunity | Alignment |
| Baseline (no-TTS) | (ref.) | – | – | |
| True-IQ oracle | – | – | ||
| Native cycle | ||||
| Held-out control | ||||
| Paired-future bound † |
| Signals | Wan2.2-5B | ABot-14B | Cosmos-Nano | Avg. | |
| G | 30.59 | 44.13 | 34.94 | 36.55 | |
| N | 27.53 | 41.57 | 33.70 | 34.27 | |
| M | 30.33 | 43.69 | 35.10 | 36.38 | |
| G N | 29.86 | 43.20 | 34.31 | 35.79 | |
| CVA-Select | 30.81 | 44.01 | 34.98 | 36.60 | – |
| N M | 29.14 | 42.61 | 34.58 | 35.44 |
| Model | no-TTS ( ) | Proprio | VideoReward | CVA-Select w/o I2V | CVA-Select |
|---|---|---|---|---|---|
| Wan2.2-5B | 0.7594 | 0.7630 | 0.7639 | 0.7685 | 0.7698 |
| ABot-14B | 0.7689 | 0.7704 | 0.7686 | 0.7721 | 0.7725 |
| Cosmos3-Nano | 0.7730 | 0.7756 | 0.7731 | 0.7754 | 0.7758 |
| Selector | ||
|---|---|---|
| Baseline (no-TTS; ) | ||
| Proprio (flow) | ||
| VideoReward | ||
| Native (cycle) | ||
| ABot-14B verifier | ||
| Low motion | ||
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Eval. | G1 Opportunity | G2 State | G3 Action | G4 Fee |
| Video models | ||||
| Physics-IQ ; | — | CIs cross | — | |
| VideoPhy2 ; | oracle | rank MAE | best | — |
| Trajectory P&P ; NFE | — | uncertainty mask | pp | fresh replicas |
| Adaptive depth WMs | oracle unif. ( ) | score rules | – fee | unif. |
| Reasoning | ||||
| Check | Result [95% CI] | Gate |
| Preconditions | ||
| no-TTS (greedy) | accuracy | base |
| Oracle@ | ; pp headroom | |
| Mechanism | ||
| State: agree disagree | pp | |
| Action: disagree agree | pp | |
| Policy | Depth | Acc. | uniform | random |
|---|---|---|---|---|
| Baseline (no-TTS; greedy) | – | – | ||
| Matched random | ||||
| Uniform | ||||
| Anchor–explorer |
| Test | Estimate [95% CI] | Takeaway |
|---|---|---|
| Baseline (no-TTS; greedy anchor) | accuracy | Strong base |
| Greedy sampled (base) | baseline | No anchor advantage |
| Greedy sampled (policy) | policy | Greedy unnecessary |
| Sampled policy uniform | Gain survives | |
| Agree disagree | anchor accuracy | Confidence signal |
| Disagree agree | deep-sampling value | Compute-value signal |
| vs. random | vs. uniform | ||||
|---|---|---|---|---|---|
| Generator | Pools | Fee | Best | Best | Oracle |
| VideoPhy2, 5B ( ) | ( ) | ||||
| Wan2.1-I2V-14B | ( ) | ||||
| Cosmos3-Nano (PIQ) | ( ) | ||||
| Run | Uniform Joint | P&P Joint | Joint | sign | |
|---|---|---|---|---|---|
| Primary | |||||
| Replica 1 | |||||
| Replica 2 | |||||
| Replica 3 |
| Variant | Information | Wan2.2-5B | ABot-14B | Cosmos-Nano | Avg. |
|---|---|---|---|---|---|
| Baseline (no-TTS; ) | None | ||||
| Low motion (zero memory) | None | ||||
| Held-out retrieval control | LOFO outcomes | ||||
| G: train-global score | LOFO outcomes | ||||
| N M (without G) | LOFO outcomes | ||||
| G N (without M) | LOFO outcomes |
| Setting | Effect | Control | Verdict |
|---|---|---|---|
| PIQ held-out | Across WMs | LOFO; train-global vs retrieval | Transfer killed |
| PAI five-fold | Train-only writes; frozen outer folds | Memory killed | |
| RT DEQM | ; W/ L | History shuffle | Specificity fails |
| RT Candidate-DEQM | ; W/ L | Fixed-memory shuffle | Specificity fails |
| RT fresh local | Eps. 8–11; 256 wrong-task nulls | Residual killed | |
| RT transition (e29) | Region vs RM Off.7 | No candidate/source alias; wrong-task/shuffle pass; global CI crosses | Region passes; VRM killed |
| Selector | Wan2.2-5B | ABot-14B | Cosmos-Nano |
|---|---|---|---|
| Native (no history) | |||
| PM (held-out) | |||
| PM pretrain (paired; priv.) | |||
| : pretrain held-out |
| Calm | Dynamic | ||||||
|---|---|---|---|---|---|---|---|
| Model | Low | Hist. | no-TTS | Native | Low | Hist. | [95% CI] |
| Wan2.2-5B | |||||||
| ABot-14B | |||||||
| Cosmos3-Nano | |||||||
| Method | Off.7 | zero | 95% CI | Regret | W/T/L | |
| Baseline (no-TTS) | – | – | ||||
| Native (zero memory) | – | ref. | ||||
| Pairwise RM † | ||||||
| PM (held-out residual) ‡ | ||||||
| PM pretrain | – | – | – | not measured | – | – |
| Oracle (eval. only) | – | – |
| Arm | Source object | Target labels | Role |
|---|---|---|---|
| Source-majority | Mean advantage | None | Feature-free null |
| Global ridge | 81-D parametric fit | None | Utility control |
| Global memory | Shared key/value bank | None | Nonparametric memory route |
| Value/key shuffle | Deranged bank | None | Specificity nulls |
| Target-history ridge | Private fit | Used | Supervised upper comp. |
| Tier | Data | Shift | Role |
|---|---|---|---|
| Pretrain | Physics-IQ, /64 | Natural events | Source fit |
| Near | Held-out PIQ families | New families | Public diagnostic |
| Mid | OpenS2V-Eval | New scenes/interface | Public diagnostic |
| Far-dev. | Private PAI, | Robot I2V/official8 | Retrospective |
| Far-confirm. | New robot, | Owner-held assets | Claim required |
| Benchmark | G: global evidence | M: motion evidence | Result status |
|---|---|---|---|
| Physics-IQ | Train-global LOFO utility | Negative frame difference | Best observed ; descriptive |
| PAI-Bench-robot | Three non-motion quality terms | Positive motion adequacy | Same frozen rule on 3 generators |
| OpenS2V-Eval | Disjoint-seed global utility | Negative frame difference | Fresh reanalysis; negative |
| Setting | Method | Gen. | Ver. | Ext. | Total DiT | |
|---|---|---|---|---|---|---|
| PIQ | no-TTS | |||||
| PIQ | Proprio (flow) | |||||
| PIQ | VideoReward | |||||
| PIQ | Native (cycle) | |||||
| PIQ | CVA-Select (descriptive) | |||||
| PIQ | Paired-future bound † |
| World model | Method | AQ | BG | IQ | MS | OC | SC | I2V-B | I2V-S | Mean |
|---|---|---|---|---|---|---|---|---|---|---|
| Wan2.2- 5B | Baseline (no-TTS) | 0.4642 | 0.9347 | 0.6821 | 0.9827 | 0.1934 | 0.9216 | 0.9652 | 0.9310 | 0.7594 |
| Proprio (flow) | 0.4708 | 0.9363 | 0.6860 | 0.9857 | 0.1927 | 0.9267 | 0.9692 | 0.9368 | 0.7630 | |
| VideoReward | 0.4738 | 0.9333 | 0.6972 | 0.9861 | 0.1922 | 0.9244 | 0.9691 | 0.9356 | 0.7639 | |
| CVA-Select w/o I2V | 0.4715 | 0.9391 | 0.6946 | 0.9895 | 0.1913 | 0.9368 | 0.9777 | 0.9476 | 0.7685 | |
| CVA-Select (ours) | 0.4693 | 0.9425 | 0.6947 | 0.9897 | 0.1916 | 0.9414 | 0.9791 | 0.9499 | 0.7698 | |
| Unified visual Top-3 | 0.4678 | 0.9396 | 0.6943 | 0.9880 | 0.1915 | 0.9377 | 0.9750 | 0.9440 | 0.7672 |
| Control | Wan2.2-5B | ABot-14B | Cosmos3-Nano |
|---|---|---|---|
| no-TTS mean ( ) | 0.7594 | 0.7689 | 0.7730 |
| CVA-Select mean ( ) | 0.7698 | 0.7725 | 0.7758 |
| Cycle | |||
| Motion | |||
| Consistency | |||
| I2V |
| Method | Aes. | Smooth. | Motion | FaceSim | GME | Nexus | Natural | Mean |
|---|---|---|---|---|---|---|---|---|
| Frozen base candidate | 0.1221 | 0.5709 | 0.2734 | 0.4954 | 0.3719 | 0.6546 | 0.1338 | 0.3746 |
| Qwen agent candidate | 0.1186 | 0.5750 | 0.3035 | 0.4957 | 0.3746 | 0.6668 | 0.1759 | 0.3872 |
| LingBot candidate | 0.1627 | 0.6763 | 0.1716 | 0.5429 | 0.3752 | 0.6431 | 0.3171 | 0.4127 |
| GPT-5.4 skill candidate | 0.1399 | 0.5947 | 0.1855 | 0.4844 | 0.3769 | 0.6663 | 0.2338 | 0.3831 |
| CVA-Select (ours) † | 0.1542 | 0.6477 | 0.1216 | 0.5373 | 0.3664 | 0.6874 | 0.2801 | 0.3992 |
| Model | no-TTS | Current PM | Pretrain | [95% CI] | CERT | Outer |
|---|---|---|---|---|---|---|
| Wan2.2-5B | 0.759376 | 0.769758 | 0.769828 | |||
| ABot-14B | 0.768894 | 0.772483 | 0.772483 | |||
| Cosmos3-Nano | 0.773021 | 0.776845 | 0.776704 |
| Task | RM | Slow | |
|---|---|---|---|
| move card | |||
| move stapler pad | |||
| pick bottles | |||
| place bread/skillet | |||
| place burger/fries | |||
| place cans/box |
Test-Time Scaling
Test-time scaling studies how extra inference-time computation can improve model answers, decisions, and generated outputs.
See all 225 papers