cs.CLMar 10, 2026
SaveThink Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
Organizations: Oracle AI Science · University of Pennsylvania
Abstract
Decoding strategies play a central role in shaping the reasoning ability of large language models (LLMs). Traditional methods such as greedy decoding and beam search often suffer from error propagation, while sampling-based approaches introduce randomness without adequate robustness. Self-consistency improves reliability by aggregating multiple rollouts, but incurs significant computational overhead. We propose an entropy-guided decoding framework that introduces token-level adaptivity into generation. At each step, the model computes the entropy of the token distribution, identifies high-uncertainty positions, and selectively branches on these vulnerable points. A dynamic pool of partial rollouts is maintained and expanded until solutions are completed, concentrating computation where uncertainty is greatest and avoiding unnecessary exploration in confident regions. To enable efficient termination, we apply a rollout-level Entropy After </Think> (EAT) stopping criterion by performing entropy evaluation after the full reasoning trace, rather than incrementally at every step. Experiments on GSM8K, AMC2023, and their perturbed variants demonstrate that our method achieves consistently strong accuracy. Notably, on smaller LLMs, performance is comparable to GPT-5 while operating at a fraction of the cost.
Figures & tables
Figure 1: Performance on GSM8K, our Entropy Decoding approach enables the 8B Llama model to outperform the base GPT models with 33x less compute.
Figure 2: Different kinds of reasoning errors made by LLMs in complex tasks are due to one or a few wrong tokens which mislead the problem-solving direction.
Figure 3: An example of parallel thinking on vulnerable tokens. Here, "equilateral" is the vulnerable token.
| Model | Data | Base Accuracy | Accuracy | Mean Jobs | Max Jobs | Mean Success Job Rate | Mean Success Depth | Mean Elapsed Time (s) |
| Llama3.1-8B | GSM8k | 84.2 | 99.6 | 1.4 | 23 | 0.993 | 0.997 | 30.2 |
| Llama3.1-8B | AMC | 45.0 | 97.5 | 1.6 | 21 | 0.786 | 1.21 | 16.9 |
| GPT4o | GSM8k | 96.3 | 100 | 1.3 | 19 | 0.987 | 0.996 | 42.2 |
| GPT4o | AMC | 75.0 | 98.25 | 1.9 | 12 | 0.759 | 1.33 | 51.0 |
Table 1: Results of entropy-guided decoding method on GSM8k and AMC2023 datasets. The table reports the base accuracy (%): pass@1 of the base model, accuracy (%): accuracy of the proposed method, job statistics, mean success job rate, mean success depth and average elapsed time in seconds.
| Perturbed questions |
| Betty is saving money for a new wallet which costs 10 for that purpose, and her grandparents two and a half times as much as her parents. How much more money does Betty need to buy the wallet? |
| Julie is reading a 110-page book. Yesterday, she was able to read 5 pages and today, she read two and a half times as many pages as yesterday. If she wants to read 45% of the remaining pages tomorrow, how many pages should she read? |
| Tina makes $17.50 an hour. If she works more than 9 hours per shift, she is eligible for overtime, which is paid by your hourly wage + 0.4 times your hourly wage. If she works 9 hours every day for 4 days, how much money does she make? |
| Randy has 55 mango trees on his farm. He also has 7 less than 0.35 times as many coconut trees as mango trees. How many trees does Randy have in all on his farm? |
Table 2: Examples of perturbed questions from the GSM8K.
| Model | Data | Base Accuracy | Proposed Accuracy | Random Accuracy | Mean Elapsed Time(s) (Proposed) | Mean Elapsed Time(s) (Random) |
| Llama3.1-8B | GSM8K_p | 75.3 | 99.2 | 85.3 | 37.8 | 95.4 |
| Llama3.1-8B | AMC_p | 42.7 | 98.3 | 92.7 | 17.4 | 21.3 |
| GPT4o | GSM8K_p | 79.5 | 99.4 | 94.5 | 45.7 | 96.9 |
| GPT4o | AMC_p | 43.2 | 91.9 | 87.6 | 57.4 | 93.8 |
Table 3: Performance of LLMs on perturbed GSM8K and AMC2023 datasets. Base Accuracy refers to the percentage accuracy of the original model. Proposed Accuracy is the accuracy (%) achieved using our proposed method. Random Accuracy is the accuracy (%) obtained when regeneration is initiated from a randomly selected token. Mean Elapsed Time (Proposed) is the average time per question when regeneration uses our proposed method. Mean Elapsed Time (Random) is the average time per question when regeneration starts from a randomly chosen token.
Figure 4: Test accuracy on the perturbed GSM8K dataset under different generation budgets. Each x-axis unit represents 128 tokens. Results are shown for the Llama3.1-8B model using entropy-based and random approaches.
| Model | Data | Accuracy | Avg. number of input tokens | Avg. number of output tokens | cost (¢) |
| GPT5 | GSM8K_p | 96.6 | 125 | 508 | 0.52 |
| Llama3.1-8B | GSM8K_p | 99.2 | 3485 | 4323 | 0.05 |
| GPT5 | AMC_p | 100 | 158 | 1779 | 1.8 |
| Llama3.1-8B | AMC_p | 98.3 | 3692 | 5285 | 0.06 |
Table 4: We compare the GPT5 model, and the LLaMA-3.1-8B model under entropy-based decoding. Performance is evaluated on the perturbed GSM8K and AMC2023 datasets. Our analysis includes accuracy, the average number of input and output tokens, and the associated API cost.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Perturbed questions | Base | Proposed |
| Issac has to buy pens and pencils for the upcoming school year. Issac buys 108 writing utensils total. He buys 12 more than 5 times the number of pencils than pens. How many pens did Issac buy? | Let’s define the variables: - Let be the number of pens Issac buys. - Let be the number of pencils Issac buys. According to the problem, he buys a total of 108 writing utensils: It is also given that he buys "12 more than 5 times the number of pencils than pens". This translates to: {\color[rgb]{1,0,0}x}=5y+12 Now we have a system of two equations: 1. 2. Substitute the expression for from equation 2 into equation 1: Combine the terms: Subtract 12 from both sides: Divide by 6: Now that we know the number of pencils, we can find the number of pens: Substitute back into equation 2: Issac buys 92 pens. | Let’s define the variables: - Let be the number of pens Issac buys. - Let be the number of pencils Issac buys. According to the problem, he buys a total of 108 writing utensils: It is also given that he buys "12 more than 5 times the number of pencils than pens". This translates to: Now we have a system of two equations: 1. 2. Substitute the expression for from equation 2 into equation 1: Combine the terms: Subtract 12 from both sides: Divide by 6: Issac buys 16 pens. |
Table 5: GSM8K example where the base model (llama3.1-8B) fails but the proposed method succeeds. x is an high entropy token.
| Perturbed questions | Base | Proposed |
| Carlos is planting a lemon tree. The tree will cost 1.5 each. It costs \3$ a year to water and feed the tree. How many years will it take before he starts earning money on the lemon tree? | 12 | 13 |
| Christina is planning a birthday party and needs .75 gift bags per invited guest, because 1/4 of attendees don’t show up. She invited 16 friends. Gift bags are $2 each. How much will she spend? | 18 | 24 |
| Lee used to be able to run the 400-meter hurdles two seconds faster than Gerald would run the 400-meter hurdles. But Gerald changed his diet, which improved his speed by 10%. If Lee runs the 400-meter hurdles in 38 seconds, how fast can Gerald, with his improved diet, run the 400-meter hurdles, in seconds? | 32.4 | 36 |
| Adrien’s total salary was 30 percent higher than Lylah’s. Four years later, his salary had increased, and he was earning 40% more than what he was making four years ago. If Adrien’s and Lylah’s salary increased simultaneously, and Adrien earned $40000 four years ago, calculate the total salary the two were receiving four years later? | 96000 | 95200 |
| A garden is filled with 105 flowers of various colors. There are twice as many red flowers as orange. There are five fewer yellow flowers than red. If there are 10 orange flowers, how many pink and purple flowers are there if they have the same amount and there are no other colors? | 60 | 30 |
| Albert wants a paintbrush that costs 4.35, and a wooden easel that costs 6.50. How much more money does Albert need? | 13 | 12 |
Table 6: GSM8K examples where the base model fails but the proposed method succeeds.
Figure 5: Consumption time (s) distribution of GPT5 model on the perturbed GSM8K dataset.
Figure 6: Distribution of the number of input tokens for the GPT5 model on the perturbed GSM8K dataset.
Figure 7: Distribution of the number of output tokens for the GPT5 model on the perturbed GSM8K dataset.
Figure 8: Distribution of the number of input tokens for the GPT5 model on the perturbed AMC dataset.
Figure 9: Distribution of the number of output tokens for the GPT5 model on the perturbed AMC dataset.
Figure 10: Consumption time (s) distribution of GPT5 model on the perturbed AMC dataset.
Figure 11: Consumption time (s) distribution of Llama3.1-8B model on the perturbed GSM8K dataset.
Figure 12: Distribution of the number of input tokens for the Llama3.1-8B model on the perturbed GSM8K dataset.
Figure 13: Distribution of the number of output tokens for the Llama3.1-8B model on the perturbed GSM8K dataset.
Figure 14: Consumption time (s) distribution of Llama3.1-8B model on the perturbed AMC dataset.
Figure 15: Distribution of the number of input tokens for the Llama3.1-8B model on the perturbed AMC dataset.
Figure 16: Distribution of the number of output tokens for the Llama3.1-8B model on the perturbed AMC dataset.
| parameter | explanation | value |
| max_degree | The maximum number of entropy-heavy tokens to branch from. | 3 |
| min_degree | The minimum number of entropy-heavy tokens to branch from. | 2 |
| degree_depth_decay | How quickly branching tapers off as moving deeper. | 0.6 |
| max_mcts_depth | How deep any branch can go. | 3 |
| max_num_create_jobs | How many total branches can be explored. | 32 |
| The mean entropy threshold for stopping | 2.3 |
Table 7: Hyperparameters used in our experiments, along with explanations and the values chosen. These values were optimized to maximize accuracy on the GSM8K train split.
| Parameter | Value | Data | Accuracy | Mean Elapsed Time(s) |
| max_degree | 2 | GSM8K_p | 98.1 | 35.2 |
| max_degree | 3 | GSM8K_p | 99.2 | 37.8 |
| max_degree | 4 | GSM8K_p | 98.7 | 36.3 |
| max_degree | 5 | GSM8K_p | 98.3 | 37.1 |
| max_degree | 2 | AMC_p | 97.5 | 18.1 |
| max_degree | 3 | AMC_p | 98.3 | 21.3 |
Table 8: Ablation studies on different parameter values. Performance on the perturbed GSM8K and AMC2023 dataset using Llama3.1-8B with entropy decoding. Mean Elapsed Time is the average time (s) per question when regeneration uses our proposed method.
More in this topic
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 168 papers