cs.LGSep 26, 2026
SaveCounting on Thinking: Tracing Evidence Integration in Language Models
Organizations: Georgia Institute of Technology, Atlanta, GA · University of Chicago, Chicago, IL
Abstract
Finite computational resources force a tradeoff between automatic System 1 processes and costly System 2 thinking. Large language models (LLMs) can spend extra computation on hard problems, yet direct answers struggle even with counting, an elementary operation humans and animals perform automatically. We ask why this requires thinking in LLMs. Evidence integration has long been used in psychology and neuroscience to probe decision-making. Our evidence-integration task presents one letter per conversational turn and asks which of two target letters appeared more often. A running count difference solves the task optimally by weighting every letter equally; tokens at each turn could represent and update this difference. Direct responses instead weighted evidence unevenly, with strong recency effects, and assigned less probability to the correct answer as difficulty increased. Thinking improved performance and made integration weights nearly uniform, yet final-query attention remained concentrated on the sequence ends in both modes. Reasoning trajectories showed models revisiting input, recounting letters, and checking intermediate counts that informed the answer, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. Reasoning-token costs grew with the number of letters far more than with coherence. Outcome feedback did not bring this computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and recency effects strengthened, yet models grew more confident. Humans and animals amortize such computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available remains central to how future models allocate computation.
Figures & tables
Figure 1: The sequential letter-counting task. A -step trial with example target letters A and B. Letters arrive one per turn, and the model responds with WAIT , A, or B; at the final step, WAIT is disallowed. The two experiments differ in how responses are controlled. In the single-game experiment, each conversation contains one game: WAIT is teacher-forced at nonfinal steps so that the model observes the whole sequence, and at the final letter it answers either directly (non-thinking) or after thinking. In the ICRL experiment, each conversation contains ten non-thinking games: the model chooses its own responses, may answer before the final letter, and learns after each game only whether its answer was correct. The schematic depicts response control in the single-game experiment; ICRL games use the same message format. System instructions are shown schematically; Appendix A.3 gives the complete prompts.
Figure 2: Thinking lowers correct-answer NLL and makes integration weights nearly uniform, but final-query attention stays concentrated on the sequence ends. (a–c) Correct-answer NLL by sequence length, distractor density, and coherence; dashed line: uninformed guess ( ). (d) Integration kernels, fitted on the same trials in both modes. (e,f) Final-query stimulus attention from stimulus-selective heads in non-thinking (e) and thinking (f) modes, normalized over the original letters within each trial so that each curve sums to one; its scale is not comparable to the kernel in d. Panels d–f use 40 letters, coherence 0.10, and no distractors. Panels a–d average seven models and e,f the four complete-score models. Blue: non-thinking; orange: thinking. Means model SEM ( models in a–d; in e,f). Estimator details in Appendices B.2 and B.4 .
Figure 3: Selected thinking trajectories revisit the evidence and attend to written counts at the final answer. (a,c) Attention from each thinking token to the original letters for Qwen3.5-4B and Gemma-4-E2B-it, normalized within each column; dashed line: first expressed preference, the first statement in the generated text that names the eventual answer. (b,d) Trajectories are divided from the generated text into review, counting, answer preparation, and rechecking, with the final answer query summarized separately; bars show absolute shares of total attention to the original letters, letters copied into the reasoning, count text, conclusions, and other context. (e) Condensed example text for each phase. Both trials use 20 letters, coherence 0.10, and no distractors, with the stimulus-selective heads of Figure 2 e,f (Appendix B.5 ).
Figure 4: Reasoning length scales with the number of letters more than with coherence. Reasoning tokens in thinking trials, excluding answer tokens, by (a) sequence length, (b) distractor density, and (c) coherence over the tested ranges, with the other factors at the baseline. Points show equal-weight means of within-model trial means across seven models; error bars show model SEM ( models). Y-axis ranges differ across panels. Trials reaching the token budget are retained.
Figure 5: ICRL does not improve direct evidence integration. (a) Correct-answer NLL and (b) action entropy by game; (c) integration kernels for initial games (Games 1–3) and final games (Games 8–10). (d) Final-query attention share by lag from the last observed letter (0 = newest; negative values are earlier letters); (e) normalized attention entropy over the full context; (f) fraction of final-query attention on current-game letters. All panels average the two ICRL conditions, with and without distractors. Panels a–c average seven models and d–f the four complete-score models; d,e use stimulus-selective heads and f all heads in the recorded layers. Dashed light teal: initial games; solid dark teal: final games. Bands show model SEM ( models in a–c; in d–f). Statistical tests in Appendices B and B.4 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Checkpoint | Total | Active | Attention layers |
| Complete-score models (Hugging Face Transformers) | |||
| Qwen/Qwen3.5-4B | 4B | 4B | 3, 7, 15, 23, 31 |
| Qwen/Qwen3.5-9B | 9B | 9B | 3, 7, 15, 23, 31 |
| google/gemma-4-E2B-it | 5.1B | 2.3B | 4, 9, 19, 24, 34 |
| google/gemma-4-E4B-it | 8B | 4.5B | 5, 11, 23, 29, 41 |
| Returned top-20 score models (Tinker API) | |||
Table 1: Model checkpoints, parameter counts, and recorded attention layers. Complete-score models ran locally with Hugging Face Transformers; returned top-20 score models ran through the Tinker API and contributed behavior only. Total parameters include embeddings. Active parameters are those used per token: effective parameters for Gemma-4, which exclude per-layer embeddings, and active-expert parameters for the mixture-of-experts models. Inkling-Small had no public checkpoint and was accessed by name. Layer indices are zero-based transformer-block indices; recorded layers expose explicit full attention.
| Experiment | ||||||
| Single-game | .05 | 0 | 40 | 40 | 0 | 2 |
| Single-game | .10 | 0 | 40 | 40 | 0 | 4 |
| Single-game | .15 | 0 | 40 | 40 | 0 | 6 |
| Single-game | .20 | 0 | 40 | 40 | 0 | 8 |
| Single-game | .10 | .20 | 40 | 39 | 8 | 3 |
| Single-game | .10 | 0 | 20 | 20 | 0 | 2 |
Table 2: Experimental conditions. : requested horizon; : realized horizon; : distractor count; : target-count difference. Each row had 200 blocks per model. Single-game rows were evaluated in both modes; ICRL rows had ten games per block in non-thinking mode.
Figure 6: Individual-model integration kernels for the single-game mode comparison. Kernels underlying Figure 2 d, with 40 letters, coherence 0.10, and no distractors: (a) Qwen3.5-4B, (b) Qwen3.5-9B, (c) Gemma-4-E2B-it, (d) Gemma-4-E4B-it, (e) Qwen3.6-35B-A3B, (f) Inkling-Small, and (g) Nemotron-3.5-Lightning-30B-A3B-BF16. Blue: non-thinking; orange: thinking. Both modes use the same eligible trials within each model, with 200 paired trials per model in a–d and f, 195 in e, and 156 in g; eligibility is not intersected across models. Ridge fits ( ) retain raw target-action score differences without cross-model normalization, and vertical scales vary across panels. For e–g, both target-letter scores must be present among the returned top-20 scores. Curves are point estimates without within-model uncertainty bands.
Figure 7: Non-thinking kernels keep their recency effect in every single-game condition. Seven-model temporal integration kernels for the remaining single-game conditions. (a,b) Sequence lengths 20 and 60; (c–e) coherence 0.05, 0.15, and 0.20; (f) distractor density 0.20. Other factors are fixed at 40 letters, coherence 0.10, and no distractors. Panel f retains the realized 39 letter positions. Blue: non-thinking; orange: thinking. Curves show equal-weight model means with model SEM bands ( models). Both modes use the same eligible trials within each model and condition, without a cross-model trial intersection. For the three returned top-20 score models, fits use the returned target-letter scores. All panels use raw action-score differences and a shared vertical scale, which is wider than that of Figure 2 d.
Figure 8: Final-query attention stays concentrated on the sequence ends at every tested length. Final-query stimulus attention across sequence lengths in four models. (a) Non-thinking; (b) thinking. Columns show 20, 40, and 60 letters, all at coherence 0.10 without distractors. Both modes use the same frozen, thinking-trained opposite-fold mass90 heads. Within each trial, raw selected-head attention is averaged before normalization over original stimulus letters, then averaged across trials and equally across models. Each curve sums to one; bands show model SEM ( models). The 40-letter profiles reproduce Figure 2 e,f.
Figure 9: Less stimulus attention reaches current-game letters in final games. Fraction of final-query attention reaching current-game letters, comparing initial games (Games 1–3) and final games (Games 8–10) in the two ICRL conditions. Attention is averaged over all heads in the five recorded layers at the final answer query, including games answered early. Trial means are averaged within each condition and game, then conditions and games are weighted equally within each model. Bars show means model SEM ( models). This groups the same final-query measure shown by game in Figure 5 f.
Figure 10: Selected Qwen3.5-9B and Gemma-4-E4B-it trajectories also revisit the letters during thinking. Attention during thinking and across annotated reasoning phases in Qwen3.5-9B and Gemma-4-E4B-it. (a,c) Heatmaps in the same model order, normalized within each thinking-token column over original stimulus positions. Gray lines mark text-defined segment boundaries; dashed lines mark the first expressed preference, not an established internal decision time. Color limits are trial-specific. (b,d) Absolute attention shares across six token categories for queries in each reasoning phase (review, counting, answer preparation, and rechecking); the final answer query is shown separately. Segments with the same phase label are pooled by query count, so the displayed phase order need not be chronological. Missing phases are marked “Not present.” (e) Illustrative phase examples. Both trials use 20 letters, coherence 0.10, no distractors, and block 0, with the frozen opposite-fold single-game mass90 heads.