Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models
Abstract
Among 291 surveyed open-weight language model repositories (280 distinct embedding tables), 52.6% of untied gated and hybrid checkpoints (20/38) contain reachable input embeddings with norms below , compared with 0% of weight-tied models. Such near-zero rows can arise during pretraining when unseen rare or template tokens receive zero input gradients while AdamW weight decay shrinks untied embeddings. In tested Transformer configurations, non-initial positions or a BOS token prevent this failure mode. In gated sequence models (Mamba, Gated DeltaNet, xLSTM), a near-zero token can amplify input-embedding gradients by more than 30 orders of magnitude (up to 35). At zero input, current-token readout factors (SiLU gates or linear queries) vanish, suppressing historical-state readout. Residual RMS values can then fall below the square root of the RMSNorm floor across consecutive layers, causing Pre-RMSNorm to amplify backward gradients by up to per layer. When a gradient entry exceeds , FP32 global-norm clipping in PyTorch, Hugging Face Trainer, and FSDP2 can overflow to infinity and zero the current batch's gradients. In Telugu continual pretraining of Nemotron-3-Nano-4B, this discards updates on approximately 20% of steps and leaves triggered rows frozen. We propose two remedies. For released checkpoints, GroupClip decouples input-embedding clipping using a 64-bit or rescaled norm; its rescaled implementation eliminates update loss and heals triggered rows in one step with negligible measured end-to-end overhead. For pretraining, GateFloor (, ) keeps the gate positive at zero input, reducing near-zero gradients by 28 orders of magnitude in experiments up to 370M parameters, with perplexity comparable to the baseline.
Figures & tables
| Architecture Family | At | Mid ( ) | Initial ( ) |
|---|---|---|---|
| Transformer ( Llama-3.1 ) | None | ( ) | |
| Token-shift ( RWKV-4 ) | Shift | ( ) | ( ) |
| Conv + ( Kimi-Linear ) | ( ) | ||
| Hybrid ( gated Attn) | ‡ | ||
| Pure SiLU ( Falcon-Mamba ) | |||
| Linear query ( xLSTM , ) | ( ) |
| Model (Steps) | Method | Stalled | PPL |
|---|---|---|---|
| Nemotron-4B | FP32 Global Clip | ||
| Telugu (1,000) | Oracle Mask ‡ | ||
| GroupClip (FP32) | |||
| GroupClip (FP64, Ours) | |||
| Falcon-H1-1.5B | FP32 Global Clip | ||
| Mongolian (1,000) | Oracle Mask ‡ |
| Model | Output Gate | PPL | Tok/s | Mid | Pos-0 |
|---|---|---|---|---|---|
| Mamba-2 165M | SiLU (Base, 2s) | ||||
| (1.05B) | Sigmoid † | ||||
| GateFloor ( ) † | |||||
| GateFloor ( ) † | |||||
| GateFloor ( , 2s) | |||||
| GateFloor ( ) † |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Condition (Scope) | Fwd Gate / Bwd Deriv. | (4-probe med.) | ( FP64 ) | FP32 | |
|---|---|---|---|---|---|---|
| Nemotron-4B | orig_keep (Unpatched) | Near-zero / Keep | ( ) | inf | ||
| ( Mamba | l0_orig_cut (Layer only) | Near-zero / Cut L0 | ( ) | inf | ||
| before L12 Attn) | l0_norm_keep_st (Layer only) | Normal “ the ” / Keep all (ST) | ( ) | finite | ||
| orig_cut (All Mamba) | Near-zero / Cut all | ( ) | finite | |||
| norm_keep_st (All Mamba) | Normal “ the ” / Keep all (ST) | ( ) | finite | |||
| norm_cut (All Mamba) | Normal “ the ” / Cut all | ( ) | finite |
| orig_keep | l0_orig_cut | l0_norm_keep_st | orig_cut | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Layer | Block | ||||||||
| (Input) | M | ||||||||
| (After L0) | F | ||||||||
| (After 2nd M) | F | ||||||||
| (After 3rd M) | F | ||||||||
| (After 4th M) | M | ||||||||
| Model / Corpus | Method | Reach. NZ Vocab | Corpus Hit | Hit Batches | FP32 inf Zeroed | FLORES PPL | |
|---|---|---|---|---|---|---|---|
| ( ) | (tok ppm) | ( ) | ( [ % hit ]) | ( [ % hit ]) | (Step ) | ||
| Nemotron-4B | FP32 Global Clip | [ ] | [ ] | ||||
| Telugu CPT | GroupClip (FP64) | ( ) | ( ) | ( ) | [ ] | [ ] | |
| Falcon-H1-1.5B | FP32 Global Clip | [ ] | [ ] | ||||
| Mongolian CPT | GroupClip | ( ) | ( ) | ( ) | [ ] | [ ] |
| Model (Steps) | Method | Step Regime ( ) | Fresh | Changed | End Med. | ||||
| Nemotron-4B | FP32 Global Clip | Clean ( ) | – | ||||||
| Telugu ( ) | inf overflow ( ) | ( seen, stuck) | |||||||
| Nemotron-4B | FP64 Global Norm | Clean ( ) | – | ||||||
| Telugu ( ) | ( ) | ( seen, healed) | |||||||
| Nemotron-4B | GroupClip (FP64) | Clean ( ) | – | ||||||
| Telugu ( ) | Trigger hit ( ) | ( seen, healed ) |
| Repository | Flags | Repository | Flags |
|---|---|---|---|
| 01-ai/Yi-1.5-6B | U | nvidia/NVIDIA-Nemotron-Nano-12B-v2 | UG ⋆ |
| 01-ai/Yi-1.5-9B | U | nvidia/NVIDIA-Nemotron-Nano-9B-v2 | UG ⋆ |
| ai21labs/AI21-Jamba-Reasoning-3B | TG | nvidia/OpenReasoning-Nemotron-1.5B | T |
| ai21labs/AI21-Jamba2-3B | TG | OBLITERATUS/Ornith-1.5-9B-OBLITERATED | U |
| ai21labs/AI21-Jamba2-Mini | UG | OBLITERATUS/Qwen3.8-27B-OBLITERATED | U |
| ai21labs/Jamba-v0.1 | UG | openai-community/gpt2 | T |
Sequence Modeling
Sequence modeling studies how to represent, predict, classify, and generate ordered data such as text, time series, events, and biological sequences.
See all 268 papers