Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state. We introduce MEMENTO: a method that teaches models to segment reasoning into blocks, compress each block into a memento, i.e., a dense state summary, and reason forward by attending only to mementos, reducing context, KV cache, and compute. To train MEMENTO models, we release OpenMementos, a public dataset of 228K reasoning traces derived from OpenThoughts-v3, segmented and annotated with intermediate summaries. We show that a two-stage SFT recipe on OpenMementos is effective across different model families (Qwen3, Phi-4, Olmo 3) and scales (8B--32B parameters). Trained models maintain strong accuracy on math, science, and coding benchmarks while achieving ∼2.5× peak KV cache reduction. We extend vLLM to support our inference method, achieving ∼1.75× throughput improvement while also enabling us to perform RL and further improve accuracy. Finally, we identify a dual information stream: information from each reasoning block is carried both by the memento text and by the corresponding KV states, which retain implicit information from the original block. Removing this channel drops accuracy by 15,pp on AIME24.
Figures & tables
Figure 1
Figure 1: Boundary scoring assigns each inter-sentence boundary a score from 0 to 3. Sentence 179 wraps up a conclusion (score 2.0); sentence 180 pivots to a new strategy (score 3.0—the strongest possible boundary); sentences 185–186 are mid-derivation (scores 0.0—never split here). The segmentation optimizer selects cuts at high-scoring transitions.
Figure 2: OpenMementos dataset distributions by domain (228K samples). (a) Math and code have ∼ 9 blocks/sample; science has ∼ 7. (b) Block sizes range from 2.3K (science) to 3.8K (math) chars. (c) Summary sizes cluster around 509–603 chars across all domains, indicating a stable compression target. (d) Math achieves the tightest compression ratio (median 0.16) due to its larger blocks.
Figure 3: Training data scaling. Pass@1 accuracy on AIME24 and AIME25 for Qwen2.5-7B-Instruct fine-tuned on 1K–100K examples. All methods improve monotonically with data size.
AIME’26
Comp. Math
MATH-500
GPQA-D
LCB v6
Val
Δ
Val
Δ
Val
Δ
Val
Δ
Val
Δ
Qwen3-8B
Acc
66.8 1.1
54.3 0.3
90.5 0.9
61.4 2.4
73.1 1.0
Base
Peak KV
2.41
2.71
0.84
1.23
1.76
AUC KV
25.3
30.9
4.3
6.6
15.6
Acc
64.7 1.1
49.2 0.3
89.7 1.0
57.8 2.5
70.0 1.0
Control
Peak KV
2.59
2.82
0.88
1.60
1.89
Table 1: Memento achieves 2–3 × peak KV reduction on models with uniform attention layers while maintaining strong reasoning performance; RL on top of our Qwen3-8B further improves accuracy. Olmo-3-7B shows more modest savings ( ∼ 0.85–0.93 × ) due to its hybrid sliding-window architecture ( Section 4 ). The Δ columns show changes of Memento and Mem.+RL relative to Control: accuracy deltas are additive (pp); KV deltas are multiplicative (Method / Control, so 0.39 × means 61% KV reduction). Metrics. Accuracy (%): pass@1 accuracy. Peak KV (GB): peak KV cache size; determines the minimum memory required to serve a request. AUC KV (GB ⋅ ktok): area under the KV-occupancy-vs-token curve, capturing total memory-time cost that penalizes both large footprints and long generations (see Figure 4 for an illustration). Rows. For each model we report Base (unmodified), Control (Base fine-tuned on the same OpenThoughts source traces used to create OpenMementos , but without block/memento annotations, for the same number of training examples), and Memento (SFT on OpenMementos with block masking); for Qwen3-8B we additionally report Memento +RL ( Section A.2.3 ). Olmo-3 Memento competition-math evaluations use 8 generations per problem (vs. 64 for all other models) and are run with HuggingFace Transformers rather than vLLM. See Section A.2.2 for full benchmark and evaluation details.
Figure 4: KV cache traces on individual problems (Qwen3-8B, both answers correct). (a) AIME24 P2: typical sawtooth pattern with 6 compactions; peak 0.77 vs 2.17 GB ( 2.8× reduction). (b) AIME24 P26: Memento solves the problem in 12k tokens (vs 23k); frequent compactions keep peak at 0.82 vs 3.41 GB ( 4.2× ). (c) AIME24 P5: Memento generates 3× more tokens (31k vs 10k) with many compactions. Peak is still lower (1.27 vs 1.55 GB), but the total KV area-under-curve is 2.1× higher than the base—a failure mode where block masking induces excessive generation.
Figure 5: CDF of compression ratio (summary chars / block chars). OLMo3-7B and Qwen3-8B achieve the tightest compression on competition math. Phi-4 has the widest compression spread, especially on MATH-500.
Model
Bench.
Base
Memento
Ret.
Jacc.
Qwen3-8B
AIME’24
93.3
90.0
96.4
96.4
AIME’25
93.3
86.7
92.9
92.9
AIME’26
86.7
90.0
100.0
96.3
Phi-4-r (14B)
AIME’24
93.3
93.3
100.0
100.0
AIME’25
93.3
90.0
96.4
96.4
AIME’26
93.3
90.0
96.4
96.4
Table 2: Problem coverage (pass@64) and solved-set overlap for Base vs. Memento on AIME ( n=64 per problem, 30 problems per benchmark).
Figure 6: Majority-vote headroom and Qwen3-8B CISPO ( MiniMax et al., 2025 ) RL trajectory. Left: AIME 2026 maj@ k for the three Memento SFT models ( Table 1 ); horizontal lines show each Base model’s pass@1. All models match Base accuracy by k=2 – 3 . Middle: per-step RL training accuracy (faint raw trace with 25-step moving average). Right: AIME’25 validation accuracy evaluated every 25 steps, peaking at 66.2% at step 350 (used in Table 1 ). Majority voting uses uniform tie-breaking among tied majority answers.
Table 3: KV ablation on full AIME24 (30 problems, Qwen3-8B memento attention checkpoint, 32K generation). Block masking accuracy is from the 64-repetition evaluation in Table 1 ; the restart experiment uses 8 repetitions.
Figure 8: Restart ablation. Step 1 : Mi is generated with full attention to Ti (same as normal memento attention). Step 2 : KV cache is discarded and recomputed via prefill over prompt +M1..i only— Ti is masked, so Mi ’s KV states no longer encode block information. The 15 pp accuracy drop ( Table 3 ) shows the KV channel carries significant reasoning capacity.
Figure 13
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Iterative memento refinement on a reasoning block about NBA playoff probabilities. The initial memento (pass 1, score 5/10) describes the approach but omits critical formulas. After judge feedback requesting the specific recurrence and transition probabilities, the refined memento (pass 2, score 8/10) captures the full computational state: function definition, base case, target, and recurrence relation.
Benchmark
Source
# Problems
Answer format
AIME 2024
HuggingFaceH4/aime_2024
30
Integer (0–999), \boxed{}
AIME 2025
yentinglin/aime_2025
30
Integer (0–999), \boxed{}
AIME 2026
MathArena/aime_2026
30
Integer (0–999), \boxed{}
HMMT Feb 2023
MathArena/hmmt_feb_2023
30
Math expression
HMMT Feb 2024
MathArena/hmmt_feb_2024
30
Math expression
HMMT Feb 2025
MathArena/hmmt_feb_2025
30
Math expression
Appendix
Table 5: Benchmark details. Competition-math benchmarks use 64 generations per problem; MATH-500, GPQA, and LiveCodeBench use 2 generations. All use temperature 0.6 and 32K max output tokens.
Figure 11: Multi-stage SFT ablation on AIME 2024, AIME 2025, and GPQA-Diamond (Pass@1, n=8 , Qwen2.5-7B). OT = OpenThoughts only; OM/Full = OpenMementos Full Attention; OM/Mem = OpenMementos Memento Attention; 2-Stg = OT → OM/Full; 3-Stg = OT → OM/Full → OM/Mem (Ours). Training directly on OpenMementos from the base model (OM variants) substantially underperforms vanilla reasoning SFT (OT). Our three-stage pipeline enables block masking while retaining strong performance.
Figure 12: Toy transformer KV probing ( keep0 ). Left: Probe accuracy by layer depth—deeper layers carry more signal, especially for the masked condition. Center: Masked probe accuracy decays with distance from the target block but remains above chance even 7 hops away. Right: Leakage stays constant (17–19%) across training checkpoints even as task accuracy climbs from 77% to 95%, confirming the channel is architectural.
Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85% memory / context savings with an 8.67% performance drop across mathematical tasks.
While chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning tasks, the linear growth of the KV cache leads to substantial memory and inference overhead. Existing approaches such as context compression and multi-token prediction (MTP) improve efficiency from two complementary directions by compressing historical tokens and generating future tokens in parallel. However, effectively combining them remains challenging due to their different training paradigms and architectural assumptions. In this work, we propose MemoSight (Memory-Foresight-Based Reasoning), a unified framework that integrates context compression and MTP to improve inference efficiency while preserving CoT performance. MemoSight adopts a shared minimalist design based on special tokens and token-specific positional layouts for both compression and parallel prediction. Experiments on four reasoning benchmarks show that, compared to the vanilla SFT baseline, MemoSight reduces KV cache usage by up to 66% and improves inference speed by 56%, while incurring less than a 3% drop in average reasoning accuracy, yielding a better efficiency-accuracy trade-off than existing CoT compression methods.
Xinyu Liu, Xin Liu, Bo Jin +8
School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China
Large language models (LLMs) have shown great performance on complex reasoning tasks but often require generating long intermediate thoughts before reaching a final answer. During generation, LLMs rely on a key-value (KV) cache for autoregressive decoding. However, the memory footprint of the KV cache grows with output length. Prior work on KV cache optimization mostly focus on compressing the long input context, while retaining the full KV cache for decoding. For tasks requiring long output generation, this leads to increased computational and memory costs. In this paper, we introduce ZoomR, a novel approach that enables LLMs to adaptively compress verbose reasoning thoughts into summaries and uses a dynamic KV cache selection policy that leverages these summaries while also strategically "zooming in" on fine-grained details. By using summary keys as a coarse-grained index during decoding, ZoomR uses the query to retrieve details for only the most important thoughts. This hierarchical strategy significantly reduces memory usage by avoiding full-cache attention at each step. Experiments across math and reasoning tasks show that our approach achieves competitive performance compared to baselines, while reducing inference memory requirements by more than 4×. These results demonstrate that a multi-granularity KV selection enables more memory efficient decoding, especially for long output generation.
David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4