Token Embeddings

Token embeddings, numerical representations of words or sub-word units, are fundamental to many natural language processing (NLP) models, aiming to capture semantic meaning and contextual information. Current research focuses on improving embedding efficiency and robustness, exploring techniques like decoupled embeddings, reinforced positional embeddings, and novel pooling strategies within transformer architectures to reduce computational costs and enhance performance across diverse languages and domains. These advancements are crucial for building more efficient and effective language models, impacting applications ranging from machine translation and question answering to speech recognition and information retrieval.

Papers

September 6, 2022

Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Doan Nam Long Vu, Nafise Sadat Moosavi, Steffen Eger
Multi Layer Representation Space Token Embeddings Character Embeddings Embedding Based Metric

July 6, 2022

Pure Transformers are Powerful Graph Learners
Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong
Graph Transformer Graph Learning Message Passing Graph Neural Network Token Embeddings Pure Transformer Inductive Graph Invariant Graph Network

June 23, 2022

BERT Rankers are Brittle: a Study using Adversarial Document Perturbations
Yumeng Wang, Lijun Lyu, Avishek Anand
Adversarial Attack Study Feature BERT Based Adversarial Input Token Embeddings Brittle Fracture Adversarial Text Perturbation

May 25, 2022

Improving CTC-based ASR Models with Gated Interlayer Collaboration
Yuting Yang, Yuke Li, Binbin Du
Automatic Speech Recognition Model Token Embeddings CTC Based

May 24, 2022

History Compression via Language Models in Reinforcement Learning
Fabian Paischer, Thomas Adler, Vihang Patil, Angela Bitto-Nemling, Markus Holzleitner, Sebastian Lehner, Hamid Eghbal-zadeh, Sepp Hochreiter
Language Model Reinforcement Learning Data Compression Observable Markov Decision Process Token Embeddings Language Transformer

December 13, 2021

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz
Large Language Model Cross Lingual Transfer Token Embeddings Monolingual Language Model Initialization Bias Subword Embeddings Multilingual Word Embeddings

November 18, 2021

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
Pengcheng He, Jianfeng Gao, Weizhu Chen
Pre Trained Language Model Masked Language Token Embeddings Sharing Matter Electra Style DeBERTaV3 Training