Phase-Conditioned Reward Shaping for Token-Efficient LLM Reasoning
Organizations: University of Southern California · Arizona State University
Abstract
Extended reasoning can help language models solve complex problems, but long responses often contain both useful computations and redundant checks. Conventional length penalties encourage shorter responses without distinguishing the functions of intermediate steps. We propose Phase-Conditioned Reward Shaping (PCRS), a framework that combines phase identification with training-time reward allocation. We investigate two classification routes: PCRS-H with a hidden-state classifier and PCRS-T with a frozen text classifier. Our main text classifier implementation assigns overlapping reasoning, execution, and revision labels to native chunks, then uses these labels and exact repetition to adjust existing token penalties. We compare the completed configurations on six mathematical and scientific reasoning benchmarks and examine four reward-weight settings with the text classifier. Compared with a baseline using token-level length penalties, the default text classifier configuration reduces mean response length by 3.13%; sampled Pass@1 is approximately 62.07% for both methods, while oracle Pass@16 rises from 86.08% to 87.67%. These results show how phase-informed penalty allocation can reduce output volume and improve multi-attempt coverage without inference-time classification.
Figures & tables
| Full six-benchmark suite | Pass@1 (points) | Pass@16 (points) | mean tokens |
|---|---|---|---|
| Question-weighted | |||
| Six-benchmark-equal |
| Configuration | Pass@1 | Tokens | Pass@16 |
|---|---|---|---|
| DECS | 62.07 | 7,187.80 | 86.08 |
| Repetition-only | 61.77 | 7,177.99 | 86.15 |
| PCRS-T | 62.07 | 6,962.68 | 87.67 |
| Budget | Coverage (%) | PCRS-T DECS | |
|---|---|---|---|
| (tokens) | DECS | PCRS-T | pp [95% interval] |
| 8,192 | 53.30 | 55.62 | |
| 16,384 | 65.26 | 67.15 | |
| 32,768 | 74.04 | 75.27 | |
| 65,536 | 80.49 | 80.35 | |
| 131,072 | 84.55 | 85.64 | |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | PCRS-H | PCRS-T |
|---|---|---|
| Input | Adjacent hidden-state changes, normalized within the input sequence | Native chunk text and preceding context; frozen word/character TF–IDF features |
| Granularity | Response-token-aligned labels | Native-chunk labels broadcast to original token positions |
| Labels | Mutually exclusive R/E/T; thresholds 1.0 and 1.4 | Overlapping R/E/T from three logistic heads; threshold 0.5; empty sets allowed |
| Phase weight | Weight of the single predicted role | Mean weight of the active roles |
| Compute required | Hidden-state extraction, adjacent differences, and normalization | CPU text features and sparse linear heads |
| Reward insertion | Before group normalization: scale eligible gaps | Before group normalization: scale eligible gaps |
| Phase | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| R | 83.17 | 83.80 | 83.48 | 395 |
| E | 79.42 | 84.73 | 81.99 | 419 |
| T | 57.45 | 43.55 | 49.54 | 62 |
| Cohort | Macro | R | E | T | Exact | |
|---|---|---|---|---|---|---|
| Math | 100 | 80.04 | 88.57 | 89.47 | 62.07 | 58.00 |
| GPQA | 80 | 82.43 | 96.69 | 71.43 | 79.17 | 45.00 |
| Setting | Value |
| Training / validation examples | 10,112 / 492 |
| Epochs / observed updates | 2 / 52 |
| Data seed | 1 |
| Train / generation batch | 128 / 128 |
| Rollouts per prompt | 4 |
| Learning rate / weight decay | / 0.01 |
| Benchmark | Pass@1 | Tokens | Pass@16 |
|---|---|---|---|
| MATH500 | 85.11 | 4,504.46 | 97.20 |
| AIME 2024 | 30.42 | 11,891.07 | 70.00 |
| AIME 2025 | 24.58 | 11,628.34 | 46.67 |
| AMC23 | 71.09 | 7,355.80 | 95.00 |
| OlympiadBench | 54.51 | 8,662.12 | 77.45 |
| GPQA-Diamond | 32.67 | 8,150.04 | 90.40 |
| Weighting | Model | Pass@1 (%) | Avg. tokens | Pass@16 (%) |
|---|---|---|---|---|
| Question-weighted | DECS | 62.07 | 7187.80 | 86.08 |
| PCRS-H | 62.07 | 7106.23 | 86.80 | |
| PCRS-T | 62.07 | 6962.68 | 87.67 | |
| PCRS-T (E-priority) | 61.62 | 7154.58 | 86.00 | |
| PCRS-T (T-priority) | 62.33 | 7063.42 | 86.66 | |
| PCRS-T (R-milder) | 62.43 | 7048.80 | 86.87 |
| Benchmark | Model | Pass@1 (%) | Avg. tokens | Pass@16 (%) | |
|---|---|---|---|---|---|
| MATH500 | DECS | 500 | 85.10 | 4551.21 | 97.60 |
| PCRS-H | 85.46 | 4423.54 | 97.40 | ||
| PCRS-T | 85.04 | 4365.49 | 97.80 | ||
| PCRS-T (E-priority) | 85.06 | 4496.02 | 97.00 | ||
| AIME 2024 | DECS | 30 | 29.38 | 11660.99 | 63.33 |
| PCRS-H | 28.54 | 11623.01 | 66.67 |
| Scope | Pass@1 (points) | Pass@16 (points) | mean tokens |
|---|---|---|---|
| MATH500 | |||
| AIME’24 | |||
| AIME’25 | |||
| AMC23 | |||
| OlympiadBench | |||
| GPQA |
| Bucket | Step 1 | Step 52 | (pp) | Mean 1–5 | Mean 48–52 | (pp) |
|---|---|---|---|---|---|---|
| R | 4.0757 | 4.3380 | +0.2623 | 4.0422 | 4.8566 | +0.8143 |
| E | 19.6729 | 20.3644 | +0.6915 | 20.1516 | 20.1627 | +0.0111 |
| T | 0.000911 | 0.000000 | -0.000911 | 0.004118 | 0.003334 | -0.000785 |
| M | 69.1865 | 68.0526 | -1.1339 | 68.9465 | 67.3933 | -1.5532 |
| U | 7.0640 | 7.2451 | +0.1811 | 6.8556 | 7.5841 | +0.7286 |
| Case | Checkpoint | Index | Selected tokens | Selected answer | Correct / 16 | Mean tokens |
|---|---|---|---|---|---|---|
| TR005 | T-priority | 6 | 5,455 | 117 (correct) | 11/16 | 8,370.50 |
| R-milder | 6 | 5,576 | 167 (incorrect) | 12/16 | 6,738.69 | |
| TR066 | T-priority | 10 | 10,322 | 8 (correct) | 16/16 | 8,016.81 |
| R-milder | 10 | 7,225 | 2 (incorrect) | 14/16 | 8,307.38 | |
| TR071 | T-priority | 1 | 16,384 | Truncated | 2/16 | 11,259.56 |
| R-milder | 1 | 4,902 | 237 (correct) | 2/16 | 11,777.94 |
Efficient Language Model Reasoning
Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.
See all 395 papers