cs.CLAug 20, 2026
SaveForking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
Abstract
LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncertainty. Resampling-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM's sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs.
Figures & tables
Figure 1: Uncertainty dynamics for one question (tinyMMLU question 39; Llama-3-8B-Instruct). Colors represent the fraction of rollouts at that end with different final answers. (a) When collecting a massive number of resampled rollouts ( ) at every token ( ), the outcome distribution becomes increasingly smooth, except at forking points. (c) Uncertainty dynamics with a lower-sample analysis ( , ) using of the reference’s sampling cost. (b) Smoothing the lower-sample data closely approximates the high-sample data and recovers the same forking points. More examples in Appendix B and our interactive dashboard ( https://github.com/ericb-goodfire/forking-fast ).
Figure 2: Accuracy versus number of resampled rollouts. We find that with raw data observations (dotted lines), greater resample spacing leads to slightly lower accuracy with equal , with considerably more difference at higher . This effect is significantly more pronounced with the smoothed (solid lines), which also shows that increasing has a consistent linear effect on log-TVD. Larger having decreasing accuracy for a given has a relatively simple explanation, since this means fewer samples collected overall. Accuracy on the y-axis is measured by total variation distance (TVD) relative to the reference .
Figure 3: Accuracy versus token cost. We find that overall, with raw data, wider spacing between resampled points leads to better accuracy with lower overall token cost (dotted lines). Our statistical model significantly increases the accuracy for most and , but also negates the improvement of larger increasing accuracy with equal cost (solid lines). TVD between the reference and each low-sample estimator is shown on the y-axis, with colors representing different . The black star represents the sampling parameters used in ( Bigelow et al., 2024a ) .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Reconstructions for three questions from the full tinyMMLU evaluation set using Llama-3-8B-Instruct. From top to bottom, the rows show the every-token reference, the Full Model’s reconstruction from a nested run at 4-token spacing, and Raw Data from the same reduced run.
Figure 5: DeepSeek counterpart to Figure 2 . Pooled TVD to the full reference is plotted against the number of samples per position, , with one line per observation spacing in sentences. Dotted lines are results for raw data, solid lines are smoothed data.
Figure 6: DeepSeek counterpart to Figure 3 . Pooled TVD is plotted against the number of tokens generated per question; colors match Figure 5 . Dotted lines are results for raw data, solid lines are smoothed data. The star marks the baseline setting of at 1-sentence spacing.
Figure 7: Forking region counterpart to Figure 2 . TVD to the reference is evaluated only at forking regions, using forking thresholds of (left) and (right). Solid lines show Smoothed Data, and dotted lines show Raw Data; colors match those in the main text.
Figure 8: Fork-region counterpart to Figure 3 . TVD in forking regions is plotted against the number of tokens generated per question at forking thresholds of and . Solid lines show Smoothed Data, and dotted lines show Raw Data.
Figure 9: Paired per-question TVD differences between each ablation variant and the Full Model for DeepSeek-R1-Distill-Llama-8B at 1-sentence spacing. Error bars show 95% -intervals over tinyMMLU, and the dashed zero line denotes the Full Model. Results are shown for pooled accuracy (left) and forking region accuracy at threshold (right). Removing segmentation harms accuracy near forks but not pooled accuracy; fixing the detection cost to L2 is indistinguishable from the Full Model in both panels.
Figure 10: Ablation overlays for two example questions. Reconstructions use on the densest spacing and are evaluated against leave-replicate-out references; dashed lines mark reference forks. The left column shows Llama-3-8B-Instruct question 46 using the ablation variants, cross-validation, and operating point from the recorded store. The right column shows the recorded DeepSeek example, where the fork occurs when continuations stop resolving to an answer. Removing segmentation (Kernel Pooling) blurs the forks, whereas removing kernel pooling (Segment+Pool) reduces gradual drift to discrete steps. The L2 detection-cost variant is visually indistinguishable from the Full Model.
Figure 11: Replicate TVD versus the number of samples per position, , on log–log axes. The measured curve remains within of the exact i.i.d. multinomial null through . Triangles show the full dataset means for both models.
Figure 12: Number of detected segments as a function of the PELT penalty for dense-grid counts on the two development questions. Curves show the exact multinomial cost, a trend-aware cost, and the silently substituted L2 cost. The dashed line marks the ideal segment count.