Tokenizer Adaptation

Ask about this topic

Momentum

1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 17

All topics
CardsList

  1. BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models

    Sep 9, 2026Shivam Singh, Aditya Yadavalli, Catherine Arnett +1Speech Foundation ModelsAutomatic Speech Recognition

    UC San Diego · EleutherAI

  2. Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

    Aug 1, 2026Bohdan DidenkoLow-Resource Language ProcessingSubword Tokenization

    Lviv Polytechnic National University

  3. Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR

    Jul 10, 2026Sanjid Hasan, Md. Abdur RahmanAutomatic Speech RecognitionLow-Resource Speech Recognition

    Khulna University of Engineering & Technology, Bangladesh · Military Institute of Science and Technology, Bangladesh

  4. LangMAP: A Language-Adaptive Approach to Tokenization

    Jun 22, 2026Clara Meister, Suchir Salhan, Andrzej Szablewski +3Multilingual Language ModelsTokenizer Adaptation

    EPFL · University of Cambridge · ETH Zürich

  5. AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens

    Jun 5, 2026Xiaocheng Lu, Yuxi Chen, Jie Zhang +5Image TokenizationVisual Tokenization

    The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University

  6. Modular Monolingual Adaptation using Pretrained Language Models

    Jun 4, 2026Nalin Kumar, Ondřej DušekLow-Resource Language ProcessingTokenizer Adaptation

    Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Prague, Czechia

  7. F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

    Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation LearningAutoencoders

    2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China

  8. Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering

    Jun 1, 2026Dongxing Mao, Jinpeng Wang, Jiahao Tang +6Image TokenizationVisual Tokenization

    Central South University · University of Oxford · Microsoft Research

  9. Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation

    May 28, 2026M. Ali Bayram, Banu Diri, Savaş YıldırımMultilingual Language ModelsSentence Embeddings

    Department of Computer Engineering Yıldız Technical University Istanbul, Turkey · Department of Computer Engineering Istanbul Bilgi University Istanbul, Turkey

  10. BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

    May 28, 2026Rohan ShravanToken EfficiencySubword Tokenization

    The School of AI India

  11. Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

    May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Domain-Adaptive PretrainingText Summarization

    Dept. of Computer Science and Engg., IIT Kharagpur, Kharagpur, India · Dept. of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA

  12. TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment

    May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Multilingual Language ModelsTokenizer Adaptation

    State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

  13. From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction

    May 12, 2026Mingcheng Zhu, Zhiyao Luo, Yu Liu +1LLM CompressionClinical Prediction

    Department of Engineering Science, University of Oxford, Oxford, United Kingdom.

  14. Adapting TrOCR for Printed Tigrinya Text Recognition: Word-Aware Loss Weighting for Cross-Script Transfer Learning

    Apr 22, 2026Yonatan Haile Medhanie, Yuanhua NiTransfer LearningOptical Character Recognition

    College of Software Engineering, Nankai University, Tianjin, China · College of Artificial Intelligence, Nankai University, Tianjin, China

  15. Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps

    Apr 20, 2026Zhichao Geng, Yang YangLearned Sparse RetrievalInformation Retrieval

    Amazon Web Service · Shanghai, China

  16. SinLlama -- A Large Language Model for Sinhala

    Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model PretrainingMultilingual Language Models

    Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1

  17. Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

    May 22, 2024Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2Language Model PretrainingMultilingual Language Models

    ITMO University · HSE University · Vikhr +2