Word Prediction

Word prediction, a core task in natural language processing, aims to accurately forecast the next word in a sequence, enabling applications like autocompletion and language generation. Current research focuses on improving model performance across diverse dialects and languages, often employing transformer-based architectures like BERT and exploring techniques such as dialect adapters and knowledge distillation to enhance accuracy and robustness. This research is significant because it directly impacts the development of more effective and inclusive language models, with implications for various applications ranging from improved user interfaces to more sophisticated AI-driven communication tools. Furthermore, ongoing work investigates the interpretability and potential biases within these models, highlighting the need for responsible development and deployment.

Papers

September 19, 2024

Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction
Xin Lian, Nishant Baglodi, Christopher J. MacLellan
Language Model Concept Discovery Incremental Task Masked Prediction Word Prediction SPIDER Max

August 31, 2024

Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models
Dipankar Srirag, Aditya Joshi, Jacob Eisenstein
Decoder Model Word Prediction Target Word Dialogue Game

May 9, 2024

Evaluating Dialect Robustness of Language Models via Conversation Understanding
Dipankar Srirag, Aditya Joshi
Language Model Dialogue Datasets Word Prediction Conversation Understanding Whole Word Masking

March 27, 2024

Enhancing Metaphor Detection through Soft Labels and Target Word Prediction
Kaidi Jia, Rongsheng Li
Knowledge Distillation Prompt Learning Soft Label Topic Specific Metaphor Metaphor Detection Word Prediction

March 14, 2024

Caveat Lector: Large Language Models in Legal Practice
Eliza Mik
Large Language Model Medical LLM Complex Reasoning Task Common Pitfall Text Distribution Legal Education Word Prediction

November 1, 2023

Style Locality for Controllable Generation with kNN Language Models
Gilles Nawezi, Lucie Flek, Charles Welch
Language Model Nearest Neighbor Controllable Generation Locality Sensitive Recent Language Model Word Prediction Nearest Neighbor Language Model

October 10, 2023

Humans and language models diverge when predicting repeating text
Aditya R. Vaidya, Javier Turek, Alexander G. Huth
Language Model Text Modality Real Human Next Word Prediction Word Prediction Recency Bias

November 17, 2022

Probing for Incremental Parse States in Autoregressive Language Models
Tiwalayo Eisape, Vineet Gangireddy, Roger P. Levy, Yoon Kim
Autoregressive Language Model Linear Probing Syntactic Representation Word Prediction

October 30, 2022

Two Models are Better than One: Federated Learning Is Not Private For Google GBoard Next Word Prediction
Mohamed Suliman, Douglas Leith
Full Model New Attack Natural Language Model Next Word Prediction Virtual Keyboard Word Prediction

July 28, 2022

An Interpretability Evaluation Benchmark for Pre-trained Language Models
Yaozong Shen, Lijie Wang, Ying Chen, Xinyan Xiao, Jing Liu, Hua Wu
Pre Trained Language Model Inherent Interpretability Syntax Aware Word Prediction

June 6, 2022

A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time
Iria de-Dios-Flores, Marcos Garcia
BERT Model Multilingual Model Training Time Agreement Metric Word Prediction Monolingual BERT Model Dependency Analysis

February 7, 2022

Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals
Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda
Language Model Pre Trained Language Model Consistent Comparison Sentence Embeddings New Combination Supervision Signal Natural Language Inference Task Word Prediction