Finite computational resources force a tradeoff between automatic System 1 processes and costly System 2 thinking. Large language models (LLMs) can spend extra computation on hard problems, yet direct answers struggle even with counting, an elementary operation humans and animals perform automatically. We ask why this requires thinking in LLMs. Evidence integration has long been used in psychology and neuroscience to probe decision-making. Our evidence-integration task presents one letter per conversational turn and asks which of two target letters appeared more often. A running count difference solves the task optimally by weighting every letter equally; tokens at each turn could represent and update this difference. Direct responses instead weighted evidence unevenly, with strong recency effects, and assigned less probability to the correct answer as difficulty increased. Thinking improved performance and made integration weights nearly uniform, yet final-query attention remained concentrated on the sequence ends in both modes. Reasoning trajectories showed models revisiting input, recounting letters, and checking intermediate counts that informed the answer, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. Reasoning-token costs grew with the number of letters far more than with coherence. Outcome feedback did not bring this computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and recency effects strengthened, yet models grew more confident. Humans and animals amortize such computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available remains central to how future models allocate computation.
Figures & tables
Figure 1: The sequential letter-counting task. A T -step trial with example target letters A and B. Letters arrive one per turn, and the model responds with WAIT , A, or B; at the final step, WAIT is disallowed. The two experiments differ in how responses are controlled. In the single-game experiment, each conversation contains one game: WAIT is teacher-forced at nonfinal steps so that the model observes the whole sequence, and at the final letter it answers either directly (non-thinking) or after thinking. In the ICRL experiment, each conversation contains ten non-thinking games: the model chooses its own responses, may answer before the final letter, and learns after each game only whether its answer was correct. The schematic depicts response control in the single-game experiment; ICRL games use the same message format. System instructions are shown schematically; Appendix A.3 gives the complete prompts.
Figure 2: Thinking lowers correct-answer NLL and makes integration weights nearly uniform, but final-query attention stays concentrated on the sequence ends. (a–c) Correct-answer NLL by sequence length, distractor density, and coherence; dashed line: uninformed guess ( ln2 ). (d) Integration kernels, fitted on the same trials in both modes. (e,f) Final-query stimulus attention from stimulus-selective heads in non-thinking (e) and thinking (f) modes, normalized over the original letters within each trial so that each curve sums to one; its scale is not comparable to the kernel in d. Panels d–f use 40 letters, coherence 0.10, and no distractors. Panels a–d average seven models and e,f the four complete-score models. Blue: non-thinking; orange: thinking. Means ± model SEM ( n=7 models in a–d; n=4 in e,f). Estimator details in Appendices B.2 and B.4 .
Figure 3: Selected thinking trajectories revisit the evidence and attend to written counts at the final answer. (a,c) Attention from each thinking token to the original letters for Qwen3.5-4B and Gemma-4-E2B-it, normalized within each column; dashed line: first expressed preference, the first statement in the generated text that names the eventual answer. (b,d) Trajectories are divided from the generated text into review, counting, answer preparation, and rechecking, with the final answer query summarized separately; bars show absolute shares of total attention to the original letters, letters copied into the reasoning, count text, conclusions, and other context. (e) Condensed example text for each phase. Both trials use 20 letters, coherence 0.10, and no distractors, with the stimulus-selective heads of Figure 2 e,f (Appendix B.5 ).
Figure 4: Reasoning length scales with the number of letters more than with coherence. Reasoning tokens in thinking trials, excluding answer tokens, by (a) sequence length, (b) distractor density, and (c) coherence over the tested ranges, with the other factors at the baseline. Points show equal-weight means of within-model trial means across seven models; error bars show model SEM ( n=7 models). Y-axis ranges differ across panels. Trials reaching the token budget are retained.
Figure 5: ICRL does not improve direct evidence integration. (a) Correct-answer NLL and (b) action entropy by game; (c) integration kernels for initial games (Games 1–3) and final games (Games 8–10). (d) Final-query attention share by lag from the last observed letter (0 = newest; negative values are earlier letters); (e) normalized attention entropy over the full context; (f) fraction of final-query attention on current-game letters. All panels average the two ICRL conditions, with and without distractors. Panels a–c average seven models and d–f the four complete-score models; d,e use stimulus-selective heads and f all heads in the recorded layers. Dashed light teal: initial games; solid dark teal: final games. Bands show model SEM ( n=7 models in a–c; n=4 in d–f). Statistical tests in Appendices B and B.4 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Checkpoint
Total
Active
Attention layers
Complete-score models (Hugging Face Transformers)
Qwen/Qwen3.5-4B
4B
4B
3, 7, 15, 23, 31
Qwen/Qwen3.5-9B
9B
9B
3, 7, 15, 23, 31
google/gemma-4-E2B-it
5.1B
2.3B
4, 9, 19, 24, 34
google/gemma-4-E4B-it
8B
4.5B
5, 11, 23, 29, 41
Returned top-20 score models (Tinker API)
Appendix
Table 1: Model checkpoints, parameter counts, and recorded attention layers. Complete-score models ran locally with Hugging Face Transformers; returned top-20 score models ran through the Tinker API and contributed behavior only. Total parameters include embeddings. Active parameters are those used per token: effective parameters for Gemma-4, which exclude per-layer embeddings, and active-expert parameters for the mixture-of-experts models. Inkling-Small had no public checkpoint and was accessed by name. Layer indices are zero-based transformer-block indices; recorded layers expose explicit full attention.
Experiment
c
ρ
N
T
D
g
Single-game
.05
0
40
40
0
2
Single-game
.10
0
40
40
0
4
Single-game
.15
0
40
40
0
6
Single-game
.20
0
40
40
0
8
Single-game
.10
.20
40
39
8
3
Single-game
.10
0
20
20
0
2
Appendix
Table 2: Experimental conditions. N : requested horizon; T : realized horizon; D : distractor count; g : target-count difference. Each row had 200 blocks per model. Single-game rows were evaluated in both modes; ICRL rows had ten games per block in non-thinking mode.
Figure 6: Individual-model integration kernels for the single-game mode comparison. Kernels underlying Figure 2 d, with 40 letters, coherence 0.10, and no distractors: (a) Qwen3.5-4B, (b) Qwen3.5-9B, (c) Gemma-4-E2B-it, (d) Gemma-4-E4B-it, (e) Qwen3.6-35B-A3B, (f) Inkling-Small, and (g) Nemotron-3.5-Lightning-30B-A3B-BF16. Blue: non-thinking; orange: thinking. Both modes use the same eligible trials within each model, with 200 paired trials per model in a–d and f, 195 in e, and 156 in g; eligibility is not intersected across models. Ridge fits ( λ=1 ) retain raw target-action score differences without cross-model normalization, and vertical scales vary across panels. For e–g, both target-letter scores must be present among the returned top-20 scores. Curves are point estimates without within-model uncertainty bands.
Figure 7: Non-thinking kernels keep their recency effect in every single-game condition. Seven-model temporal integration kernels for the remaining single-game conditions. (a,b) Sequence lengths 20 and 60; (c–e) coherence 0.05, 0.15, and 0.20; (f) distractor density 0.20. Other factors are fixed at 40 letters, coherence 0.10, and no distractors. Panel f retains the realized 39 letter positions. Blue: non-thinking; orange: thinking. Curves show equal-weight model means with model SEM bands ( n=7 models). Both modes use the same eligible trials within each model and condition, without a cross-model trial intersection. For the three returned top-20 score models, fits use the returned target-letter scores. All panels use raw action-score differences and a shared vertical scale, which is wider than that of Figure 2 d.
Figure 8: Final-query attention stays concentrated on the sequence ends at every tested length. Final-query stimulus attention across sequence lengths in four models. (a) Non-thinking; (b) thinking. Columns show 20, 40, and 60 letters, all at coherence 0.10 without distractors. Both modes use the same frozen, thinking-trained opposite-fold mass90 heads. Within each trial, raw selected-head attention is averaged before normalization over original stimulus letters, then averaged across trials and equally across models. Each curve sums to one; bands show model SEM ( n=4 models). The 40-letter profiles reproduce Figure 2 e,f.
Figure 9: Less stimulus attention reaches current-game letters in final games. Fraction of final-query attention reaching current-game letters, comparing initial games (Games 1–3) and final games (Games 8–10) in the two ICRL conditions. Attention is averaged over all heads in the five recorded layers at the final answer query, including games answered early. Trial means are averaged within each condition and game, then conditions and games are weighted equally within each model. Bars show means ± model SEM ( n=4 models). This groups the same final-query measure shown by game in Figure 5 f.
Figure 10: Selected Qwen3.5-9B and Gemma-4-E4B-it trajectories also revisit the letters during thinking. Attention during thinking and across annotated reasoning phases in Qwen3.5-9B and Gemma-4-E4B-it. (a,c) Heatmaps in the same model order, normalized within each thinking-token column over original stimulus positions. Gray lines mark text-defined segment boundaries; dashed lines mark the first expressed preference, not an established internal decision time. Color limits are trial-specific. (b,d) Absolute attention shares across six token categories for queries in each reasoning phase (review, counting, answer preparation, and rechecking); the final answer query is shown separately. Segments with the same phase label are pooled by query count, so the displayed phase order need not be chronological. Missing phases are marked “Not present.” (e) Illustrative phase examples. Both trials use 20 letters, coherence 0.10, no distractors, and block 0, with the frozen opposite-fold single-game mass90 heads.
Large language models (LLMs) have been rapidly improving in long-context tasks, powered by Chain-of-Thought (CoT) reasoning. However, the internal mechanisms underlying this improvement remain unclear. We investigate these mechanisms through a needle-in-a-haystack (NIAH) counting task, where an LLM is asked to count the number of records dispersed in a long text. Across twelve model comparison groups, Thinking (or reasoning) improves counting accuracy over Non-thinking, with pronounced gains at larger counts. This motivates our mechanistic analysis, which identifies two contrasting mechanisms: (i) broad retrieval, where Non-thinking models broadly attend to multiple needles; (ii) targeted retrieval, where Thinking models use enumeration in CoT traces to successively retrieve needles. Targeted retrieval concentrates attention on individual needles and is accompanied by more compact internal representations. Moreover, causal intervention analysis suggests that Thinking models use the CoT trace to maintain and update an internal counter as needles are successively retrieved, even without explicit numbering. In small controlled experiments, both retrieval mechanisms and counter states emerge under standard autoregressive training. Together, our results connect long-context retrieval with representation geometry of counting, supporting a state-tracking account of CoT reasoning.
Liang Twist Shan, Tianyu Hu, Hao Yan +1
Department of Statistics, University of Wisconsin-Madison · Department of Computer Sciences, University of Wisconsin-Madison
Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention. We study LLMs' no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode or the absence of long traces. These proxies are unreliable: disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference. We instead normalize each response into a pre-answer trace and final answer, and evaluate it at three levels: (i) Empty-Thinking Rate for strict answer-only compliance; (ii) instruction-aware Question-Pre-answer Relevance for similarity between the question and pre-answer trace; and (iii) LLM-as-judge Explicit Inference Rate for visible explicit inference. Together, these metrics distinguish answer-only output, relevant but non-inferential text, and explicit inference. b. How does no-thinking vary across tasks and models? We evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions. We find that explicit no-think controls cannot reliably eliminate visible inference. Models instead exhibit "Thinking Inertia": explicit inference persists even under strict controls and becomes more prevalent as the answer space opens. Accuracy remains stable on Boolean and multiple-choice tasks, whereas open-ended tasks reveal a trade-off between answer-only compliance and task accuracy. Rewriting the same questions across answer spaces shows that supplying candidate answers makes answer-only responses easier to produce. These findings establish no-thinking as a non-trivial capability: stopping explicit reasoning cannot be assumed from model settings or instructions alone and deserves systematic evaluation alongside reasoning ability.
Dianqiao Lei, Kevin Qinghong Lin, Pan Lu +2
TSINGHUA UNIVERSITY · UNIVERSITY OF OXFORD · STANFORD UNIVERSITY
Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.