Unigram Tokenizer

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 44

All topics

Forking: Sudden Overfitting Under Replay
Sep 30, 2026Shanbin Yu, Shaoyang Guo, Haoran Zhao +2Overfitting Catastrophic Forgetting
MetaCircle (元环智能) · Tsinghua University · Peking University +1
FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
Sep 28, 2026Bowen Yang, Jingbo Zhou, Qinghong Miao +1Unigram Tokenizer Large Language Model Memory
Large Model Frontier Research Department, Baidu Inc., China · Nanyang Technological University
Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Study
Sep 28, 2026Bibek Bhandari, Kshitij LingthepBiomedical Text Synthetic Data
Independent Researcher Memphis, Tennessee, USA
Structure Before Sampling: Community-Aware Core-Set Selection for Data-Efficient Text-to-Speech
Sep 21, 2026Mizbaul Haque Maruf, Muhammad Nur YanhaonaGrapheme-To-Phoneme Text Corpora
Department of Computer Science and Engineering Brac University, Dhaka, Bangladesh
Contributions to the hierarchy of probabilistic languages
Sep 20, 2026Lothar Sebastian Krapp, Remo NitschkeContext-Free Grammars Probabilistic Model
Institut für Interdisziplinäre Sprachevolutionswissenschaft, Universität Zürich, Switzerland · Fachbereich Mathematik und Statistik, Universität Konstanz, Germany
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Sep 14, 2026Franck Signe, Hippolyte Pilchen, François Yvon +1Multilingual Language Models Tokenizer
Kyutai, Paris · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble · Sorbonne Université, CNRS, Institute of Intelligent Systems and Robotics, Paris
SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data
Sep 14, 2026Praveen Kumar Myakala, Ravichandra Namburi, Sowmya Keragodu Jayaramu +1Synthetic Data Training Data
Independent AI Researcher Melissa, USA · Independent Researcher Frisco, USA · Independent Researcher Austin, USA
Semantic Fibers and Cross-Gram Interference: A Calculus of Safety Drift in Overcomplete Representations
Sep 14, 2026Mohammed Ahnouch, Lotfi ElaachackMultilingual Safety Representation-Level Misalignment
Université Paris 1 Paris, France · Faculty of Science and Technology of Tangier Abdelmalek Essaadi University Tangier, Morocco
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Sep 3, 2026Yunao Zheng, Bin Wen, Xiaojie Wang +12Unigram Tokenizer Language Model Computes
Beijing University of Posts and Telecommunications. · Kuaishou Technology.
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Autoregressive Token Prediction Unigram Tokenizer
Qwen Team
Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection
Aug 12, 2026Florian BraunTamper Localization Model Auditing
Literary Non-Style in LLM-Generated Text
Jul 19, 2026Cory MassaroHuman-Written Text Literature
AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation
Jul 17, 2026Saifur Rahman Tamim, Amir Labib KhanLarge Language Model Watermarks Watermarks
Department of Computer Science and Engineering Northern University Bangladesh
Dzongkha Next Word Prediction System
Jul 13, 2026Prerna Chhetri, Tenzin Yoezer, Phuntsho Wangmo +1Unigram Tokenizer Bangla
Information Technology Department, College of Science and Technology, Royal University of Bhutan
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Jul 6, 2026Hunter HeidenreichSubword Tokenization Unigram Tokenizer
Independent Researcher
Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG
Jul 3, 2026Haotian Zhou, Weiran Huang, Siqi Liu +3Language Pairs Unigram Tokenizer
School of Computer Science, Shanghai Jiao Tong University · Ant International, Ant Group · Shanghai Innovation Institute +1
Induction Heads Interpolate N-Grams
Jul 2, 2026Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1Transformer Architectures In-Context Learning
1TML Lab, EPFL, Switzerland. · 2IIT Madras, India. · †Work done during an internship at the TML Lab, EPFL.
Hamm-Grams: An Algorithm for Mining Regular Expressions of Bytes
Jul 1, 2026Derek Everett, Edward Raff, James HoltMalware Unigram Tokenizer
Amazon, Baltimore, USA · CrowdStrike, Austin, USA
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Jun 25, 2026Sander LandUnigram Tokenizer Token Compression
Writer, Inc.
A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic
Jun 23, 2026Jing Yang, Shuqing Zhang, Yongyi Deng +5Arabic Natural Language Processing Arabic
School of Electronic Information, Wuhan University, Wuhan Hubei, 430072, China · School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · Northwestern Polytechnical University +1
Attractive and Repulsive Pattern Control in Sequence Generation
Jun 19, 2026Francois PachetMarkov Repulsive Distribution
Augmenting Molecular Language Models with Local $n$ -gram Memory
Jun 10, 2026Xinni Zhang, Zijing Liu, He Cao +2Unigram Tokenizer Language Modeling
1The Chinese University of Hong Kong · 2International Digital Economy Academy
Operationalizing Linguistic Methods through Prompt-Engineering Skills: An Automatic Chinese Web Neologism Detection Pipeline
Jun 7, 2026Yufeng Wu, Meichun LiuLinguistics Unigram Tokenizer
City University of Hong Kong
NGram-MoSE: Efficient Remote Sensing Super-Resolution via N-Gram Context and Mixture-of-Experts
Jun 7, 2026Yun-Hsuan Huang, Trong-An Bui, Chih-Hung ChuangHyperspectral Image Super-Resolution Remote Sensing
Institute of Aerospace and Systems Engineering, National Taipei University of Technology, Taipei City, Taiwan.
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Jun 6, 2026Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides +3Unigram Tokenizer Token Embeddings
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · AIP, RIKEN, Tokyo, Japan.
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Speculative Decoding Multilingual
University of Colorado
MATCHA: Matching Text via Contrastive Semantic Alignment
May 26, 2026Siran Li, Ece Sena Etoglu, Carsten Eickhoff +1Legal Natural Language Processing Benchmarks Hierarchical Semantic Alignment
University of Tübingen
Lngram: N-gram Conditional Memory in Latent Space
May 24, 2026Yunao Zheng, Guoyang Xia, Xiaojie Wang +1Efficient Long-Context Inference Language Modeling
Beijing University of Posts and Telecommunications (BUPT), Beijing, China · Li Auto Inc., Beijing, China
Tokenization with Split Trees
May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4Subword Tokenization Unigram Tokenizer
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
May 20, 2026Runxi Cheng, Yuchen Guan, Yongxian Wei +7Large Language Model Memory Large Language Model Pretraining
Tsinghua University · Microsoft Reasearch Asia
Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels
May 18, 2026Alexander Boesgaard LorupImpact Analysis Unigram Tokenizer
Openhagen
NGM: A Plug-and-Play Training-Free Memory Module for LLMs
May 16, 2026Yuwen Qu, Wenhui Dong, Chenyang Si +1Large Language Model Memory Unigram Tokenizer
Nanjing University
A Calculus-Based Framework for Determining Vocabulary Size in End-to-End ASR
May 14, 2026Sunil Kumar KopparapuAutomatic Speech Recognition Subword Tokenization
TCS Research - Mumbai.
COVID-19 Infodemic. Understanding content features in detecting fake news using a machine learning approach
May 7, 2026Vimala Balakrishnan, Lee Zing Hii, Eric LaporteFake News Detection Fake News
Faculty of Computer Science & Information Technology, Universiti Malaya, 50603, Kuala Lumpur, Malaysia · LIGM, Univ Gustave Eiffel, CNRS, F-77454 Marne-la-Vallée, France
Cascade Token Selection for Transformer Attention Acceleration
May 4, 2026Stephen J. ThomasTransformer Attention Linear Attention
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling
Apr 23, 2026Yilong Chen, Yanxi Xie, Zitian Gao +10Unigram Tokenizer Memory-Augmented Framework
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · IQuest Research +1
Promoting Simple Agents: Ensemble Methods for Event-Log Prediction
Apr 23, 2026Benedikt Bollig, Matthias Függer, Thomas Nowak +1Ensemble Automata
Université Paris-Saclay, CNRS, ENS Paris-Saclay, LMF, Gif-sur-Yvette, France · Institut Universitaire de France, Paris, France · Direction Générale de l’Armement, Paris, France
A weighted angle distance on strings
Apr 22, 2026Grant MolnarDistance Unigram Tokenizer
ARKA
Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations
Apr 19, 2026S. A. Desimone, L. Alonso AlemanyLinguistics Large Language Model Bias
Universidad Nacional de C´ordoba · CONICET
Tokenization vs. Augmentation: A Systematic Study of Writer Variance in IMU-Based Online Handwriting Recognition
Feb 25, 2026Jindong Li, Dario Zanca, Vincent Christlein +4Handwriting Data Augmentation
Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · STABILO International GmbH, Heroldsberg, Germany · Ludwig-Maximilians-Universität München, Munich, Germany +2
Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance
Nov 30, 2025Robin Yadav, Shuo Xie, Tianhao Wang +1Stochastic Gradient Descent Signsgd
Toyota Technological Institute at Chicago · University of British Columbia · University of California, San Diego
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLarge Language Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuUnigram Tokenizer Token Embeddings
Independent Researcher Beijing, China
What Language is This? Ask Your Tokenizer
Date pendingClara Meister, Ahmetcan Yavuz, Pietro Lesci +1Tokenizer Language Pairs
EPFL · ETH Zürich · University of Cambridge

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Forking: Sudden Overfitting Under Replay

Shanbin Yu, Shaoyang Guo, Haoran Zhao, Danni Yu+1

MetaCircle (元环智能) · Tsinghua University · Peking University+1

Overfitting Catastrophic Forgetting Epoch Replay Unigram Tokenizer Openai+1

Open paper AskPDF arXiv

This paper studies forking, a generalization failure discovered in NanoGPT autoresearch. Under data replay, models with an over-encoding n-gram memory branch show a sharp separation of training and validation loss at epoch boundaries, resembling the shape of forks. We study this phenomenon in a controlled vanilla NanoGPT setting and reproduce it in a DeepSeek-style model with Engram. Mechanistically, repeated updates sharpen the continuations observed in training while suppressing the probability of unseen continuations, whose loss grows with each pass. The n-gram module creates weakly interacting context-specific subspaces, amplifying this effect. Low-frequency contexts contribute most of the gap, whereas larger training budgets and heavily crowded tables suppress it. We also observe forking in short-budget, heavily repeated SFT and RL-like regimes. The contributions of this paper are twofold: (1) Forking reveals yet another curious phenomenon in deep learning, in addition to grokking and double descent. (2) Forking is an unexpected and unpleasant by-product of tricks proposed by autoresearch agents. While these agents produce an enormous number of results that seem useful, we should always be careful with their results.

Momentum

Related topics

Latest papers 44