cs.LGSep 30, 2026
SaveForking: Sudden Overfitting Under Replay
Organizations: MetaCircle (元环智能) · Tsinghua University · Peking University · Shanghai Qizhi Institute
Abstract
This paper studies forking, a generalization failure discovered in NanoGPT autoresearch. Under data replay, models with an over-encoding n-gram memory branch show a sharp separation of training and validation loss at epoch boundaries, resembling the shape of forks. We study this phenomenon in a controlled vanilla NanoGPT setting and reproduce it in a DeepSeek-style model with Engram. Mechanistically, repeated updates sharpen the continuations observed in training while suppressing the probability of unseen continuations, whose loss grows with each pass. The n-gram module creates weakly interacting context-specific subspaces, amplifying this effect. Low-frequency contexts contribute most of the gap, whereas larger training budgets and heavily crowded tables suppress it. We also observe forking in short-budget, heavily repeated SFT and RL-like regimes. The contributions of this paper are twofold: (1) Forking reveals yet another curious phenomenon in deep learning, in addition to grokking and double descent. (2) Forking is an unexpected and unpleasant by-product of tricks proposed by autoresearch agents. While these agents produce an enormous number of results that seem useful, we should always be careful with their results.
Figures & tables
Figure 1 : Over-encoding and replay-aligned forking. Left: illustrative context representations. Upper right: matched models during the first pass of multi-epoch training. Lower right: the -gram model under replay; vertical lines mark pass boundaries.
Figure 2 : Forking and the -gram RMSProp update signal near the first replay boundary (boxed).
Figure 3 : Three -gram injection routes.
Figure 4 : Forking under data replay.
Figure 5 : Gap scaling with (a) context frequency, (b) table capacity, and (c) corpus size (trigram only). Panel (a) shows the mean online gap across seeds 42, 43 and 44 at step 1000, on log–log axes. Markers represent 18 frequency bins with ; dashed lines are equal-bin log–log fits, with exponents (bigram) and (trigram). Appendix E.6 compares these with measured novelty. Panels (b,c) use seed 42 and the settings described in the text.
Figure 6 : Gap growth under replay: (a) within epochs, (b) across 20 epochs.
Figure 7 : Causal interventions after epoch 1: (a) low-frequency retrieval masking, (b) table freezing.
Figure 8 : Local sharpening and cross-context recovery, illustrated schematically. Dashed marks show prior heights; darker extensions show gains. Others denotes the mean probability of the top 20 remaining tokens. Measured distributions for there is a appear in Figure 21 (e,f).
Figure 9 : Context-subspace evidence for replay-driven sharpening (seed 42). (a) Mean probability mass on train-seen continuations (top) and on validation-only continuations (bottom) over four epochs, for masked and unmasked context cohorts in the -gram and no-gram models. (b) Mean fixed-probe gap of the masked cohort, of its unmasked peers, and of normal training; dotted lines mark epoch boundaries. (c) Validation cross-entropy at the five epoch-end checkpoints for train-seen bigram contexts, split by whether the true continuation appeared in training.
Figure 10 : Continued training on replayed corpora. Solid lines show online training loss, dashed lines fixed validation loss, and the lower row their difference.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| \toprule Component | Configuration |
| \midrule Backbone | 8 layers; hidden dimension 768; 6 attention heads; head dimension 128. |
| Input and batch | Sequence length 2048; 72 sequences or 147456 tokens per update; gradient accumulation 1. |
| Tokenizer | Shared tokenizer; vocabulary 8192; built from the first ten shards. |
| N-gram tables | Two tables per order and injection layer; each has 524288 rows of dimension 384. Bigram at layers 2, 4, 6, 8; trigram at layers 2, 6, 8. |
| Optimizers | Muon for matrix parameters, base learning rate 0.04; grouped AdamW for token/value embeddings, output and scalar parameters; RMSProp for bigram/trigram tables, learning rate 0.6, second-moment decay 0.999, no weight decay. |
| Schedule | Archived baseline schedule: Muon and ordinary AdamW warmdown begins at 5% and 35% of the step budget, respectively; final multiplier 0.05. N-gram table learning rates remain constant. |
Table 1: Configuration for discovery-stage observable screening. The architecture and training settings were determined by the search agent and held fixed during screening. Layer indices start at one.
Figure 11 : Four data-reuse controls (E1–E4) in the discovery setting, each with seed 42 and 1000 updates. Blue solid and orange dashed curves show training and validation cross-entropy. Gray bands mark and, for E2, ; vertical dashed lines mark logged epoch boundaries. E4 shows steps 100–600, with replay blocks shaded orange; the other panels show steps 100–1000. Training losses are unsmoothed. Validation uses four successive held-out batches every five steps, with the latest value carried forward for display.
Figure 12 : Two representative n-gram observables. The top row shows the RMS magnitude of the RMSProp update vector; the bottom row shows its cosine similarity with the parameter vector. Columns correspond to E1–E4 from left to right. Shading and boundaries follow Figure 11 . Each panel uses its own visible vertical range. Unsmoothed curves are used to compare event timing and shape.
Figure 13 : Component removal in the discovery setting. Left: the full model. Right: bigram and trigram value pathways disabled, with unigram value embeddings retained. Both runs use shard-1 sequential replay, seed 42, and 1000 updates; the axes are shared. Blue solid curves show raw training cross-entropy. Orange dashed curves show validation cross-entropy on four successive held-out batches every 50 steps, carried forward between evaluations. Vertical dashed lines mark replay boundaries at steps 338 and 687.
| \toprule Component | Configuration |
| \midrule Attention backbone | 8 layers; hidden width 768; 6 heads of dimension 128; full causal attention in every layer. |
| Feed-forward blocks | Two linear layers with a expansion to width 3072 and a GELU activation. |
| Normalization and positions | Pre-LayerNorm in each block and a final LayerNorm, ; learned absolute position embeddings. |
| Embeddings and regularization | Vocabulary 8192; tied token/output weights; dropout 0. Attention and MLP linear layers and LayerNorm include biases; the output head does not. |
| N-gram memory | One hash table per order, bigram and trigram; full-width lookup vectors added at the input. No extra unigram or fourgram memory, gates, or multi-hash concatenation. |
| Sequence and batch | 2048 prediction positions per sequence; 72 sequences per update; 147,456 prediction tokens; one device batch per update. |
Table 2: Default architecture and data settings for the minimal experiments. A listed quantity changes only in experiments that explicitly vary it. Optimization settings and their selection evidence are given in Appendix C.2 .
Figure 14 : Selection screens for the controlled optimization setting. (a) Backbone-rate sensitivity with the absolute table rate fixed at ; solid and dashed curves show raw and matched no-gram-subtracted gaps at step 2000. (b) Table-rate multiplier scan at steps 1000 and 2000. (c) Corrected RMSProp sweep. (d) Warmup-start sweep; “none” removes warmup. Green vertical dotted lines mark the values adopted for the controlled setting: backbone learning rate , table-rate multiplier , RMSProp , and warmup-start multiplier .
Figure 15 : Training, validation, and gap trajectories for the component interventions (seed 42, logged every 10 steps). Net gap subtracts the no-gram model at matching steps; dotted lines mark the replay boundaries at steps 337 and 674. The hash-reseeding conditions change the context-to-row mapping while retaining table weights and optimizer state, either once at step 338 or at steps 338 and 675. The twice-reseeded arm ends with raw gap 0.069.
| \toprule Condition | Intervention | Reported observation |
| \midrule No-gram | Remove the n-gram module | Raw gap 0.233 at step 2000; ordinary train–validation differences remain. |
| N-gram | Input injection; table LR | Raw gap 5.676 at step 2000, with replay-related steps. |
| Low-frequency mask | Mask low-count reads in training and evaluation | removes about 72% of net gap at step 1000; higher thresholds further reduce it (raw gap 0.101 at ). |
| High-frequency mask | Mask high-count reads | leaves the step-1000 raw gap at 2.824 (control 2.724), a much smaller change than low-frequency masking. |
| Hash reseed | Change context-to-row mapping at each replay boundary; retain parameters and optimizer | Raw gap 0.069 at step 1000 when reseeded at steps 338 and 675 (1.354 with one reseed); substantial suppression of replay accumulation. |
| Freeze table | Stop table updates at step 675 | 94% of the control gap remains at step 2022 (5.386 versus 5.733) and 72% after ten passes; the gap keeps growing, more slowly than in control. Frozen at step 338, the step-1000 gap is 3.452. |
Table 3: Intervention summary across comparisons, not a list of endpoints for Figure 15 . Raw gaps and relative reductions have different denominators. All runs use input injection, table LR , and seed 42; each row states its budget.
Figure 16 : Frequency-threshold scans at step 1000 (seed 42): (a) raw gap and (b) fixed-validation loss. High masking uses and low masking . Each point is one run; lines are visual guides.
Figure 17 : Epoch-end raw gap, net gap, and absolute fixed-validation loss for trigram-only, both-table, and no-gram models (20 epochs, seed 42). Net gap subtracts the no-gram model at each boundary.
Figure 18 : Validation-loss differences relative to the no-gram baseline for unconstrained tables, masking of reads from the start, and hash reseeding at every replay boundary (both tables, 2000 updates, seed 42). These comparisons delimit the benefit of memory in the measured replay regime; they do not establish that n-gram memory should only be used as a short-term cache.
Figure 19 : Coverage and the gap. (a) Good–Turing estimate from training counts (Equation \eqref eq:gt-bin) versus the measured novel fraction; dashed line: equality. (b,c) Online gap versus measured novel fraction with affine fits at steps 670, 1000, 1340 and 2000; grey crosses show the no-gram control at step 1000. Points are frequency bins; bars show standard deviations across seeds 42, 43 and 44.
| \toprule context | class | share of positions | share of excess loss |
|---|---|---|---|
| \midrule bigram | novel continuation | 31.26% | 72.45% |
| bigram | seen continuation | 68.74% | 27.55% |
| trigram | novel continuation | 65.67% | 102.02% |
| trigram | seen continuation | 34.33% | % |
| \bottomrule |
Table 4: Share of the step-1000 excess validation loss ( nats per position, 589,248 positions) carried by each position class. Signed contributions sum to the total within either context granularity.
| \toprule Context | Gap exponent | Novelty exponent | ||
|---|---|---|---|---|
| \midrule Bigram | 0.251 | 0.997 | 0.229 | 0.981 |
| Trigram | 0.466 | 0.843 | 0.212 | 0.985 |
| \bottomrule |
Table 5: Matched power-law fits at step 1000. Gap uses the three-seed mean; novelty is measured directly on the shared fixed validation pool. Both use the same frequency bins and equal-bin log–log regression.
| \toprule Context | Step | rms | Relative | Holdout | ||
| \midrule Bigram | 670 | 4.87 | 0.137 | 8.2% | 0.169 | |
| 1000 | 9.58 | 0.233 | 6.3% | 0.269 | ||
| 1340 | 10.86 | 0.16 | 0.205 | 4.1% | 0.228 | |
| 2000 | 10.90 | 1.89 | 0.202 | 3.0% | 0.233 | |
| Trigram | 670 | 2.35 | 0.063 | 10.7% | 0.060 | |
| 1000 | 5.81 | 0.155 | 9.9% | 0.132 |
Table 6: Online gap fit to the three-seed mean over 18 bins with . Coefficients and rms errors are in nats; relative rms divides by the mean absolute bin gap. The holdout column fits alternating frequency bins and evaluates on the others.
Figure 20 : Continuation probabilities for walk their dog over five replay epochs in the bigram-plus-trigram model (seed 42). Points mark the five epoch-end checkpoints (steps 337, 674, 1011, 1348 and 1685); higher means higher probability. Lines connect successive checkpoints. Blue marks down , observed once in training; orange marks daily and for , each observed once in the counted validation window and absent from training after this context. The input is the three context tokens without BOS, using the full 8192-token softmax. The grey Other line shows the log of the arithmetic mean of the top 20 remaining token probabilities, excluding down , daily and for and reselected at each checkpoint; this is not total remaining mass.
Figure 21 : Changes in continuation probability from epoch 1 to epoch 5. Rows show petting your , introducing your , and there is a ; columns compare no-gram and bigram-plus-trigram memory. Open and filled points mark epochs 1 and 5 (steps 337 and 1685, seed 42); higher means higher probability. Connecting lines join endpoints, without implying monotonic change between them. Blue denotes train-seen continuations, orange validation-only (novel), and grey neither. The Train/Val rows give counts of each exact context–token pair in training shard 1 and held-out shards 2–4; they describe corpus membership, while points show model predictions. Each input is BOS plus three context tokens, using the full 8192-token softmax. Diamonds show the log of the arithmetic mean of the top 20 remaining token probabilities, excluding the named animals and reselected at each checkpoint; this is not total remaining mass.
| \toprule Component | Configuration |
| \midrule Decoder and embeddings | 12 layers; hidden width 768; vocabulary 8192; sequence length 2048; separate token and output embeddings; dropout 0. |
| MLA-lite attention | 12 heads; query latent rank 256 and key/value latent rank 128. Per head: 64 non-rotary query/key dimensions, 32 rotary dimensions, and 64 value dimensions; causal attention; RoPE base 10,000. |
| Feed-forward layers | Layer 1: dense SwiGLU with intermediate width 2048. Layers 2–12: 16 routed experts, top-2 selection, and two shared experts; each expert has intermediate width 512. |
| Expert balancing | Sigmoid router scores, normalized over the selected experts; auxiliary-loss-free correction bias with update rate . |
| Residual connections | Four mHC streams with learned pre-, post-, and residual mixing; 20 Sinkhorn iterations. RMSNorm is used in latent attention, memory fusion, and the output pathway. |
| Memory locations | Engram at layers 2 and 6; separate parameters at each layer; bigram and trigram retrieval. |
Table 7: Architecture of the DeepSeek-style replication. Layer indices start at one. These settings are fixed across the full-memory and two retrieval-mask conditions.
| \toprule Component | Configuration |
| \midrule Non-table parameters | AdamW; learning rate ; ; . Weight decay 0.1 on matrix and convolution weights, excluding token/output embeddings; biases and one-dimensional parameters have no decay. |
| Lookup tables | Bias-corrected, momentum-free RMSProp; learning rate ( the backbone rate); second-moment decay 0.99; ; no weight decay. |
| Learning-rate schedule | Both rates increase linearly from to over updates 1–100, then remain constant; no warmdown or later decay. |
| Numerics and execution | BF16 forward computation; FP32 model parameters, optimizer states, and cross-entropy logits; activation checkpointing; eager execution; one NVIDIA H200 per run, with the official mHC kernels. |
| Evaluation | Current-batch training loss and fixed validation loss every 10 updates; 200 recorded points per run; no smoothing. |
| \bottomrule |
Table 8: Optimization and execution settings for the DeepSeek-style runs. Learning rates are inherited from the minimal-setting protocol, whose selection evidence appears in Appendix C.2 .
Figure 22 : Forking and low-frequency retrieval masking in the DeepSeek-style model. Left: current-batch training loss before each logged update (solid) and fixed validation loss after it (dashed). Right: their difference. Dotted lines mark completed 337-update passes; masks activate before update 338. All three independent seed-42 runs complete 2000 updates; curves join the values logged every 10 updates without smoothing. Low-frequency masking attenuates the replay-associated gap growth and lowers final validation loss.
| \toprule Source | Table | Train | Validation | Gap |
|---|---|---|---|---|
| \midrule ID | Trainable | 1.913 | 4.376 | 2.463 |
| ID | Frozen | 3.030 | 3.372 | 0.341 |
| Magicoder | Trainable | 0.619 | 1.543 | 0.924 |
| Magicoder | Frozen | 1.103 | 1.301 | 0.198 |
| OpenR1 | Trainable | 0.688 | 1.366 | 0.678 |
| OpenR1 | Frozen | 1.039 | 1.197 | 0.158 |
Table 9: Endpoint measurements for continued language-model pretraining from the low-gap checkpoint. All values are nats at step 1665. ID uses seed 42; Magicoder and OpenR1 average downstream seeds 42–44. The table-trainable and table-frozen runs differ only in whether the trigram lookup table receives gradients.
| \toprule Source | Table | Train | Validation | Gap |
|---|---|---|---|---|
| \midrule RLVR-MATH | Trainable | 0.009 | 0.518 | 0.509 |
| RLVR-MATH | Frozen | 0.014 | 0.447 | 0.434 |
| \bottomrule |
Table 10: Static RLVR-like verifier-selected continued-pretraining endpoints. All values are nats at step 1665, averaged over downstream seeds 42–44.