A Full-Ledger Counterfactual Audit of Adaptive Reasoning
Organizations: University of Science and Technology of China Hefei 230027, China
Abstract
Adaptive reasoning must decide whether to answer now or spend more compute. Predicting that the current answer is wrong does not establish whether further reasoning will improve it. We introduce the Full-Ledger Counterfactual Value Audit (FLCVA), which compares an immediate answer with four fixed-length continuations from the same frozen prefix under one answer procedure and accounts for the cost of every signal used by a policy. On GSM8K and MATH-500, a random 256-token continuation recovers a wrong answer in 26.6% of Qwen3-8B cases and 18.9% of DeepSeek-R1-Distill-Qwen-7B cases, while damaging a correct answer in 1.4% and 1.9%. Correctness labels from independently sampled continuations predict a separate continuation's outcome in both primary models. In offline replay, using these labels to allocate compute improves performance on DeepSeek; the Qwen3 gain remains inconclusive. Under full cost accounting, the tested lightweight, myopic controllers mainly recognize current errors and add little continuation-specific value beyond that shortcut. A Llama-8B replication and action/cost ablations examine the robustness of these findings. FLCVA separates measuring local continuation value, predicting it from observable signals, and using those predictions to allocate computation.
Figures & tables
| Work | Primary decision object | Signal or intervention | Paired same-prefix signed target | Cost role |
|---|---|---|---|---|
| APR | Serial/parallel scheduling | Learned spawn-and-join policy | No | Compute and latency |
| EAGer | Branch/sample allocation | Token-entropy branching | No | Token efficiency |
| TRACE | Early exit | Temporal consistency and confidence | No | Accuracy–token tradeoff |
| UAB | Per-question sample count | Initial-sample vote entropy | No | Fixed sampling budget |
| PoLR | Prefix-cluster expansion | Consensus among short prefixes | No | Tokens and latency |
| Prefix Consistency | Candidate weighting | Regeneration from truncated prefixes | Partial | Voting cost |
| Policy | Required observation | Extra generation | Extra scoring | Replay charge |
|---|---|---|---|---|
| Round-robin / fixed | One current spine | None | None | Prompt, reasoning, answer probe |
| TRACE / ANLL / entropy | Current/prior single-spine probes | None | None | Single-spine ledger |
| PRM | Current single-spine prefix | None | PRM prefill | Ledger plus every PRM call |
| Cross-spine agreement | Four equal-depth answers | Three synchronized spines | None | Four reasoning/probe paths |
| Ridge / two-stage | Single-spine proxies and context | None | Optional PRM | All observed inputs |
| Mixed/off-spine/DP | Continuation correctness labels | Stored branches/futures | Gold grading | Diagnostic only |
| Model | Event | Single draw | Any of four |
|---|---|---|---|
| Qwen3 | Recovery | 26.6% [22.5, 31.6] | 35.7% [30.3, 42.3] |
| Qwen3 | Damage | 1.4% [1.0, 1.8] | 2.9% [2.3, 3.6] |
| DeepSeek | Recovery | 18.9% [16.2, 22.2] | 26.0% [22.2, 30.2] |
| DeepSeek | Damage | 1.9% [1.5, 2.3] | 3.6% [3.0, 4.2] |
| Model | Target | Best proxy (available ) | AUROC (95% CI) |
|---|---|---|---|
| Qwen3 | Recovery | Persistence (2,702) | 0.658 [0.610, 0.706] |
| Qwen3 | Damage | Agreement (18,634) | 0.742 [0.699, 0.782] |
| DeepSeek | Recovery | Persistence (3,665) | 0.673 [0.632, 0.715] |
| DeepSeek | Damage | Persistence (17,139) | 0.716 [0.681, 0.747] |
| Model | Policy | Norm. AULC | Gain vs. RR | Increment vs. shortcut |
|---|---|---|---|---|
| Qwen3 | Round-robin | 0.851 | +0.00% [+0.00, +0.00] | -1.75% [-2.72, -0.72] |
| Qwen3 | Correctness shortcut | 0.866 | +1.78% [+0.73, +2.80] | +0.00% [+0.00, +0.00] |
| Qwen3 | Two-stage conditional EV | 0.860 | +1.08% [+0.28, +1.91] | -0.69% [-1.51, +0.10] |
| Qwen3 | Direct Ridge- | 0.864 | +1.57% [+0.65, +2.51] | -0.21% [-0.52, +0.12] |
| DeepSeek | Round-robin | 0.813 | +0.00% [+0.00, +0.00] | -1.53% [-2.60, -0.44] |
| DeepSeek | Correctness shortcut | 0.825 | +1.55% [+0.44, +2.67] | +0.00% [+0.00, +0.00] |
| Model | Current wrong | Recovery | Damage |
|---|---|---|---|
| Qwen3 | 0.882 | 0.774 | 0.799 |
| DeepSeek | 0.843 | 0.767 | 0.772 |
| Model | Off-spine gain vs. RR | Exact-DP gain vs. RR |
|---|---|---|
| Qwen3 | ||
| DeepSeek |
| Model | Length | Rec.% | Dmg.% | |
|---|---|---|---|---|
| Qwen3 | 128 | 19.0 | 1.6 | +0.016 |
| Qwen3 | 256 | 26.6 | 1.5 | 0.000 |
| Qwen3 | 512 | 36.0 | 1.6 | -0.031 |
| Qwen3 | Adaptive | 26.8 | 1.6 | 0.000 |
| DeepSeek | 128 | 14.3 | 1.7 | +0.021 |
| DeepSeek | 256 | 19.6 | 1.9 | 0.000 |
| Event | Draw | Any of four | Persistence AUROC |
|---|---|---|---|
| Recovery | 10.2% | 14.5% | 0.729 |
| Damage | 0.9% | 2.0% | 0.730 |
| Policy | Gain vs. RR | vs. shortcut |
|---|---|---|
| Round-robin | +0.00% | -2.85% |
| Correctness shortcut | +2.93% | +0.00% |
| Two-stage EV | +2.10% | -0.81% |
| Direct Ridge- | +2.30% | -0.61% |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Policy | Mean normalized AULC | Rel. normalized-AULC gain vs. RR (95% CI) | Final problem accuracy |
|---|---|---|---|---|
| Qwen3 | Fixed 1,024 | 0.820 | -3.66% [-4.34, -3.01] | 0.909 |
| Qwen3 | Round-robin | 0.851 | +0.00% [+0.00, +0.00] | 0.909 |
| Qwen3 | TRACE-style | 0.814 | -4.29% [-5.96, -2.54] | 0.909 |
| Qwen3 | PRM | 0.439 | -48.41% [-52.94, -43.76] | 0.476 |
| Qwen3 | ANLL | 0.834 | -2.00% [-3.52, -0.44] | 0.909 |
| Qwen3 | Entropy | 0.832 | -2.27% [-3.78, -0.70] | 0.909 |
| Model | Label-aware ordering | Mean normalized AULC | Gain vs. RR (95% CI) | Final problem accuracy |
|---|---|---|---|---|
| Qwen3 | Round-robin | 0.851 | +0.00% [+0.00, +0.00] | 0.909 |
| Qwen3 | On-spine hindsight | 0.842 | -1.10% [-3.32, +1.33] | 0.909 |
| Qwen3 | Mixed myopic | 0.869 | +2.07% [+0.19, +4.02] | 0.909 |
| Qwen3 | Off-spine myopic | 0.863 | +1.44% [-0.48, +3.37] | 0.909 |
| DeepSeek | Round-robin | 0.813 | +0.00% [+0.00, +0.00] | 0.847 |
| DeepSeek | On-spine hindsight | 0.844 | +3.88% [+1.96, +5.90] | 0.847 |
| Model | Target | Held-out branch | Event rate | AUROC (95% CI) | AUPRC (95% CI) | Brier (95% CI) | |
|---|---|---|---|---|---|---|---|
| Qwen3 | Beneficial | Pooled 1–4 | 14,744 | 26.6% | 0.943 [0.929, 0.954] | 0.865 [0.838, 0.890] | 0.061 [0.049, 0.074] |
| Qwen3 | Beneficial | 1 (on-spine) | 3,686 | 26.9% | 0.941 [0.926, 0.954] | 0.867 [0.838, 0.894] | 0.062 [0.049, 0.075] |
| Qwen3 | Beneficial | 2 (off-spine) | 3,686 | 26.2% | 0.945 [0.930, 0.958] | 0.863 [0.834, 0.891] | 0.061 [0.049, 0.075] |
| Qwen3 | Beneficial | 3 (off-spine) | 3,686 | 26.0% | 0.945 [0.930, 0.959] | 0.859 [0.826, 0.889] | 0.061 [0.049, 0.076] |
| Qwen3 | Beneficial | 4 (off-spine) | 3,686 | 27.2% | 0.940 [0.923, 0.954] | 0.872 [0.841, 0.900] | 0.061 [0.049, 0.076] |
| Qwen3 | Harmful | Pooled 1–4 | 74,536 | 1.4% | 0.849 [0.810, 0.880] | 0.490 [0.381, 0.580] | 0.009 [0.007, 0.011] |
| Model | Ordering score | Sees successor? | Mean normalized AULC | AULC gain vs. RR | Off-spine permutation ( ) |
|---|---|---|---|---|---|
| Qwen3 | Round-robin | no | 0.851 | +0.00% | – |
| Qwen3 | Mixed 4-branch | yes | 0.856 | +0.57% | – |
| Qwen3 | On-spine one-step hindsight | yes | 0.827 | -2.82% | – |
| Qwen3 | Off-spine independent | no | 0.849 | -0.19% | 0.001/0.001 |
| DeepSeek | Round-robin | no | 0.813 | +0.00% | – |
| DeepSeek | Mixed 4-branch | yes | 0.848 | +4.37% | – |
| Model | Ledger | Agreement treatment | Mean normalized AULC | Gain vs. single-spine RR (95% CI) |
|---|---|---|---|---|
| Qwen3 | Logical tokens | Single-spine round-robin | 0.851 | +0.00% [+0.00, +0.00] |
| Qwen3 | Logical tokens | Free cross-spine diagnostic | 0.830 | -2.47% [-4.71, -0.19] |
| Qwen3 | Logical tokens | Charged four-spine policy | 0.627 | -26.29% [-29.79, -22.92] |
| Qwen3 | Estimated FLOPs | Single-spine round-robin | 0.854 | +0.00% [+0.00, +0.00] |
| Qwen3 | Estimated FLOPs | Free cross-spine diagnostic | 0.831 | -2.66% [-4.78, -0.54] |
| Qwen3 | Estimated FLOPs | Charged four-spine policy | 0.504 | -40.99% [-45.31, -36.65] |
| Model | Policy | Mean normalized AULC | Gain vs. RR (95% CI) | Gap to exact DP (95% CI) |
|---|---|---|---|---|
| Qwen3 | Round-robin | 0.843 | +0.00% [+0.00, +0.00] | 6.61% [5.06, 8.31] |
| Qwen3 | Mixed myopic | 0.841 | -0.25% [-2.41, +1.88] | 6.84% [4.78, 9.00] |
| Qwen3 | On-spine one-step hindsight | 0.820 | -2.74% [-5.35, -0.15] | 9.16% [6.74, 11.67] |
| Qwen3 | Off-spine myopic | 0.832 | -1.22% [-3.42, +0.81] | 7.75% [5.65, 10.16] |
| Qwen3 | Exact non-myopic hindsight | 0.902 | +7.08% [+5.31, +9.03] | 0.00% [0.00, 0.00] |
| DeepSeek | Round-robin | 0.766 | +0.00% [+0.00, +0.00] | 10.36% [7.86, 13.46] |
| Model | Target | Proxy | AUROC (95% CI) | AUPRC (95% CI) | Brier (95% CI) | |
|---|---|---|---|---|---|---|
| Qwen3 | Beneficial | Confidence | 3,686 | 0.501 [0.428, 0.572] | 0.338 [0.281, 0.415] | 0.227 [0.206, 0.252] |
| Qwen3 | Beneficial | Agreement | 3,678 | 0.625 [0.573, 0.675] | 0.428 [0.366, 0.497] | 0.218 [0.196, 0.242] |
| Qwen3 | Beneficial | Persistence | 2,702 | 0.658 [0.610, 0.706] | 0.363 [0.295, 0.444] | 0.183 [0.155, 0.217] |
| Qwen3 | Beneficial | PRM | 3,686 | 0.604 [0.562, 0.649] | 0.494 [0.432, 0.566] | 0.221 [0.198, 0.246] |
| Qwen3 | Beneficial | Entropy | 3,686 | 0.510 [0.434, 0.584] | 0.334 [0.271, 0.427] | 0.226 [0.204, 0.252] |
| Qwen3 | Harmful | Confidence | 18,634 | 0.598 [0.549, 0.648] | 0.067 [0.042, 0.105] | 0.028 [0.022, 0.034] |
| Model | Target | Proxy | Analysis prev. | Conf. prev. | AP | AP/prev. | Brier ref. | BSS |
|---|---|---|---|---|---|---|---|---|
| Qwen3 | Beneficial | Confidence | 0.307 | 0.357 | 0.338 | 0.95 | 0.232 | +0.022 |
| Qwen3 | Beneficial | Agreement | 0.306 | 0.356 | 0.428 | 1.20 | 0.232 | +0.058 |
| Qwen3 | Beneficial | Persistence | 0.224 | 0.264 | 0.363 | 1.37 | 0.196 | +0.064 |
| Qwen3 | Beneficial | PRM | 0.307 | 0.357 | 0.494 | 1.38 | 0.232 | +0.050 |
| Qwen3 | Beneficial | Entropy | 0.307 | 0.357 | 0.334 | 0.94 | 0.232 | +0.025 |
| Qwen3 | Harmful | Confidence | 0.028 | 0.029 | 0.067 | 2.30 | 0.028 | +0.005 |
| Model | Target | Proxy | Conditional AUROC | NN disagreement | test log-loss, bits |
|---|---|---|---|---|---|
| Qwen3 | Beneficial | ANLL | 0.524 | 34.3% [30.8, 37.9] | -0.0166 [-0.0344, -0.0006] |
| Qwen3 | Beneficial | Confidence | 0.476 | 34.3% [30.9, 37.7] | -0.0166 [-0.0331, -0.0008] |
| Qwen3 | Beneficial | Agreement | 0.581 | 35.1% [31.8, 38.7] | +0.0144 [-0.0112, +0.0416] |
| Qwen3 | Beneficial | Persistence | 0.644 | 32.3% [28.7, 36.5] | +0.0468 [+0.0193, +0.0760] |
| Qwen3 | Beneficial | PRM | 0.597 | 35.6% [32.9, 38.4] | -0.0085 [-0.0233, +0.0067] |
| Qwen3 | Beneficial | Entropy | 0.534 | 36.7% [33.4, 40.1] | -0.0239 [-0.0406, -0.0088] |
| Model | Proxy | Collision rate | Beneficial | Harmful | |||
|---|---|---|---|---|---|---|---|
| Qwen3 | ANLL | 48.0% | 81.1% | +2.703 | +2.735 | -1.813 | -1.815 |
| Qwen3 | Confidence | 48.0% | 81.1% | +2.703 | +2.735 | -1.813 | -1.815 |
| Qwen3 | Agreement | 73.7% | 98.1% | +2.687 | +2.726 | -1.769 | -1.788 |
| Qwen3 | Persistence | 77.3% | 99.1% | +2.616 | +2.627 | -1.777 | -1.795 |
| Qwen3 | PRM | 45.3% | 67.3% | +2.677 | +2.699 | -1.806 | -1.814 |
| Model | Target | Proxy | Conditional AUROC | NN disagreement | test log-loss, bits | |||
|---|---|---|---|---|---|---|---|---|
| Qwen3 | Beneficial | ANLL | 0.524 | 0.543 | 34.3% | 36.4% | -0.0166 | +0.0069 |
| Qwen3 | Beneficial | Confidence | 0.476 | 0.457 | 34.3% | 36.4% | -0.0166 | +0.0068 |
| Qwen3 | Beneficial | Agreement | 0.581 | 0.577 | 35.1% | 36.6% | +0.0144 | +0.0358 |
| Qwen3 | Beneficial | Persistence | 0.644 | 0.633 | 32.3% | 33.2% | +0.0468 | +0.0520 |
| Qwen3 | Beneficial | PRM | 0.597 | 0.609 | 35.6% | 36.7% | -0.0085 | +0.0045 |
| Model | Controller | Pearson | Beneficial AUROC | Harmful AUROC ( score) |
|---|---|---|---|---|
| Qwen3 | Ridge- (single-spine) | 0.373 | 0.715 | 0.291 |
| Qwen3 | Ridge- (+ charged PRM) | 0.374 | 0.715 | 0.310 |
| DeepSeek | Ridge- (single-spine) | 0.282 | 0.726 | 0.226 |
| DeepSeek | Ridge- (+ charged PRM) | 0.284 | 0.730 | 0.235 |
| Model | Data | Wrong | Beneficial rate (95% CI) | Correct | Harmful rate (95% CI) |
|---|---|---|---|---|---|
| Qwen3 | All | 3,686 | 35.7% [30.4, 42.1] | 18,634 | 2.9% [2.3, 3.6] |
| Qwen3 | GSM8K | 1,481 | 54.0% [45.9, 64.3] | 14,479 | 1.8% [1.3, 2.5] |
| Qwen3 | MATH-500 | 2,205 | 23.4% [17.7, 30.6] | 4,155 | 6.7% [4.7, 9.2] |
| DeepSeek | All | 4,401 | 26.0% [22.2, 30.4] | 17,919 | 3.6% [3.0, 4.2] |
| DeepSeek | GSM8K | 1,985 | 32.2% [26.4, 39.4] | 13,975 | 2.9% [2.4, 3.5] |
| DeepSeek | MATH-500 | 2,416 | 20.9% [16.3, 26.4] | 3,944 | 6.0% [4.1, 8.3] |
| Model | Low group | High–low (95% CI) | ||
|---|---|---|---|---|
| Qwen3 | (tie-resolved) | 15,349 | 2,673 | +0.200 [+0.135, +0.276] |
| Qwen3 | 15,349 | 1,251 | +0.291 [+0.190, +0.401] | |
| Qwen3 | 15,349 | 398 | +0.450 [+0.267, +0.665] | |
| DeepSeek | (tie-resolved) | 15,011 | 2,128 | +0.277 [+0.201, +0.361] |
| DeepSeek | 15,011 | 937 | +0.480 [+0.341, +0.621] | |
| DeepSeek | 15,011 | 311 | +0.740 [+0.536, +0.965] |
| Model | Target | Expected event rate | Agreement with | |
|---|---|---|---|---|
| Qwen3 | Beneficial | 1 | 26.6% | 90.9% |
| Qwen3 | Beneficial | 2 | 31.2% | 95.4% |
| Qwen3 | Beneficial | 3 | 33.8% | 98.1% |
| Qwen3 | Beneficial | 4 | 35.7% | 100.0% |
| Qwen3 | Harmful | 1 | 1.4% | 98.5% |
| Qwen3 | Harmful | 2 | 2.0% | 99.1% |
| Model | Pearson | Spearman | Sign agree. | Mean |
|---|---|---|---|---|
| Qwen3 | 1.00000 | 0.99999 | 100.0% | 0.0002 |
| DeepSeek | 1.00000 | 0.99999 | 100.0% | 0.0002 |
| Length | Recovery | Damage | Mean | vs. 256 (95% CI) |
|---|---|---|---|---|
| Qwen3 | ||||
| 128 tokens | 19.0% [15.0, 24.1] | 1.6% [1.1, 2.2] | 0.137 | +0.016 [-0.001, +0.033] |
| 256 tokens | 26.6% [21.2, 33.6] | 1.5% [1.0, 2.1] | 0.121 | +0.000 [+0.000, +0.000] |
| 512 tokens | 36.0% [28.9, 45.5] | 1.6% [1.0, 2.3] | 0.090 | -0.031 [-0.040, -0.023] |
| Adaptive | 26.8% [21.2, 33.7] | 1.6% [1.0, 2.2] | 0.121 | +0.000 [-0.008, +0.007] |
| DeepSeek | ||||
| Model | Target | Condition | Rank vs. 256 | Top proxy (half-sample stability) |
|---|---|---|---|---|
| Qwen3 | Recovery | 128 tokens | 1.00 [0.82, 1.00] | persistence (89%) |
| Qwen3 | Recovery | 512 tokens | 1.00 [-0.22, 1.00] | persistence (80%) |
| Qwen3 | Recovery | Adaptive | 1.00 [0.82, 1.00] | persistence (95%) |
| Qwen3 | Damage | 128 tokens | 1.00 [0.76, 1.00] | modal_agreement (88%) |
| Qwen3 | Damage | 512 tokens | 1.00 [0.76, 1.00] | modal_agreement (80%) |
| Qwen3 | Damage | Adaptive | 1.00 [0.82, 1.00] | modal_agreement (92%) |
| Model | Token identity | Raw-text identity | Normalized-answer identity |
|---|---|---|---|
| Qwen3 | 98.9% [98.7, 99.1] | 99.6% [99.4, 99.7] | 98.5% [97.5, 99.2] |
| DeepSeek | 98.3% [98.1, 98.5] | 99.5% [99.4, 99.7] | 98.8% [98.0, 99.5] |
| Model | Cue | Parse rate | Answer agreement | Benefit agreement | Harm agreement |
|---|---|---|---|---|---|
| Qwen3 | Canonical | 99.4% [98.9, 99.7] | 100.0% [100.0, 100.0] | 100.0% [100.0, 100.0] | 100.0% [100.0, 100.0] |
| Qwen3 | Direct | 51.7% [47.9, 55.7] | 78.8% [73.0, 84.5] | 72.0% [64.0, 78.3] | 80.3% [79.1, 81.5] |
| Qwen3 | Terse | 99.9% [99.8, 99.9] | 95.2% [93.4, 96.7] | 91.0% [87.2, 94.4] | 96.5% [95.7, 97.2] |
| DeepSeek | Canonical | 99.8% [99.4, 100.0] | 100.0% [100.0, 100.0] | 100.0% [100.0, 100.0] | 100.0% [100.0, 100.0] |
| DeepSeek | Direct | 80.8% [78.1, 83.3] | 87.0% [83.0, 90.7] | 85.9% [81.5, 89.5] | 83.4% [82.0, 84.7] |
| DeepSeek | Terse | 99.8% [99.5, 100.0] | 95.3% [93.9, 96.5] | 89.8% [86.0, 93.1] | 97.0% [96.5, 97.6] |
| Pool | Gain (%) | 95% CI | |
| Qwen3 | |||
| Pooled | 200 | ||
| GSM8K | 150 | ||
| MATH-500 | 50 | ||
| DeepSeek | |||
| Pooled | 200 | ||
| Model | Raw gain (%) | Calibrated gain (%) | Calibration effect on AULC advantage (pp) |
|---|---|---|---|
| Qwen3 | |||
| DeepSeek |
| Persistence | Recovery (%) | 95% CI | |
| Qwen3 | |||
| 290 | 31.29 | ||
| 497 | 21.03 | ||
| 571 | 10.29 | ||
| Missing | 495 | 48.13 | |
| DeepSeek | |||
| Remain correct (%) | 95% CI | ||
| Qwen3 | |||
| 1 | 486 | 89.92 | |
| 4 | 458 | 83.19 | |
| DeepSeek | |||
| 1 | 415 | 85.78 | |
| 4 | 387 | 81.65 | |