Language Model

Language models are computational systems designed to understand and generate human language, primarily aiming to improve tasks like translation, question answering, and text summarization. Current research focuses on enhancing efficiency (e.g., through novel learning rate schedules and optimized architectures), improving alignment with human preferences (via preference optimization and reward modeling), and addressing biases and limitations (including techniques for mitigating toxicity and enhancing robustness). These advancements have significant implications for various fields, impacting natural language processing research and enabling the development of more powerful and reliable AI applications.

Papers

November 12, 2024

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, Tao Yu
Language Model Text to SQL Text to SQL Benchmark SPIDER Max
EUR/USD Exchange Rate Forecasting incorporating Text Mining Based on Pre-trained Language Models and Deep Learning Methods
Xiangyu Shi, Hongcheng Ding, Salaar Faroog, Deshinta Arrova Dewi, Shamsul Nahar Abdullah, Bahiah A Malek
Language Model Deep Learning Forecasting Performance Text Mining Real Time Prediction
Model Stealing for Any Low-Rank Language Model
Allen Liu, Ankur Moitra
Large Language Model Language Model Hidden Markov Model Model Stealing
SecEncoder: Logs are All You Need in Security
Muhammed Fatih Bulut, Yingqi Liu, Naveed Ahmad, Maximilian Turner, Sami Ait Ouahmane, Cameron Andrews, Lloyd Greenwald
Large Language Model Language Model Training Data Security Related Log Message
Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models
Daria Kryvosheieva, Roger Levy
Language Model Full Model Multilingual Text Syntactic Knowledge Syntactic Generalization

November 11, 2024

Controllable Context Sensitivity and the Knob Behind It
Julian Minder, Kevin Du, Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West, Ryan Cotterell
Language Model Fine Tuned Model Contextual Knowledge
Warmstarting for Scaling Language Models
Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison, Aaron Klein, Josif Grabocka, Frank Hutter
Language Model Lightweight Model Optimal Hyperparameters Warm Start Large Scale Training Scaling Analysis
SetLexSem Challenge: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models
Bardiya Akhbari, Manish Gawali, Nicholas A. Dronen
Large Language Model Language Model Challenge Dataset Semantic Variation Lexical Aware Grouping Network Semantic Robustness Multiset Function
Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations
Kirti Bhagat, Kinshuk Vasisht, Danish Pruthi
Large Language Model Language Model Health Disparity
The Surprising Effectiveness of Test-Time Training for Abstract Reasoning
Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas
Language Model Complex Reasoning Neural Language Model Test Time Training Reasoning Corpus Surprising Effectiveness Abstract Text
Contextualized Evaluations: Taking the Guesswork Out of Language Model Evaluations
Chaitanya Malaviya, Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, Kyle Lo
Language Model Global Evaluation Implicit Bias Response Quality Subjective Question
TempCharBERT: Keystroke Dynamics for Continuous Access Control Based on Pre-trained Language Models
Matheus Simão, Fabiano Prado, Omar Abdul Wahab, Anderson Avila
Language Model Pre Trained Language Model Temporal Information Character Level Keystroke Dynamic User Identification
The Super Weight in Large Language Models
Mengxia Yu, De Wang, Qi Shan, Colorado Reed, Alvin Wan
Language Model Weight Quantization Weight Space
NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
David Robinson, Marius Miron, Masato Hagiwara, Olivier Pietquin
Language Model Computational Bioacoustics
Counterfactual Generation from Language Models
Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson, Ryan Cotterell
Language Model Counterfactual Reasoning Counterfactual Generation Meaningful Counterfactuals Generating Process
Building a Taiwanese Mandarin Spoken Language Model: A First Attempt
Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu, Tzu-Quan Lin, Hsiu-Hsuan Wang, En-Pei Hu, Chan-Jan Hsu, Liang-Hsuan Tseng, I-Hsiang Chiu, Ulin Sanga, Xuanjun Chen, Po-chun Hsu, Shu-wen Yang, Hung-yi Lee
Language Model Multi Turn Dialogue Multi Turn Conversation First Attempt Synthetic Dialogue Chinese Language Model Speech Interaction
Transformer verbatim in-context retrieval across time and scale
Kristijan Armeni, Marko Pranjić, Senja Pollak
Language Model Transformer Based Time Matter Visual Analogue Scale Context Dependent Context Retrieval
On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models
Qian Sun, Hanpeng Wu, Xi Sheryl Zhang
Language Model Supervised Fine Tuning White Box Empirical Privacy Private Fine Tuning Box Membership Inference Attack
Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition
Yoshiki Masuyama, Koichi Miyazaki, Masato Murata
Language Model Automatic Speech Recognition Speech Recognition Constructive Approach Attention Based Encoder Decoder Speech Token Decoder Only Architecture
HarmLevelBench: Evaluating Harm-Level Compliance and the Impact of Quantization on Model Alignment
Yannis Belkhiter, Giulio Zizzo, Sergio Maffeis
Language Model Fine Grained Global Impact Transformer Architecture Jailbreak Attack Quantization Operator LLM Safety Model Alignment

Language Model

Papers

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

EUR/USD Exchange Rate Forecasting incorporating Text Mining Based on Pre-trained Language Models and Deep Learning Methods

Model Stealing for Any Low-Rank Language Model

SecEncoder: Logs are All You Need in Security

Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

Controllable Context Sensitivity and the Knob Behind It

Warmstarting for Scaling Language Models

SetLexSem Challenge: Using Set Operations to Evaluate the Lexical and Semantic Robustness of Language Models

Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

Contextualized Evaluations: Taking the Guesswork Out of Language Model Evaluations

TempCharBERT: Keystroke Dynamics for Continuous Access Control Based on Pre-trained Language Models

The Super Weight in Large Language Models

NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics

Counterfactual Generation from Language Models

Building a Taiwanese Mandarin Spoken Language Model: A First Attempt

Transformer verbatim in-context retrieval across time and scale

On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models

Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition

HarmLevelBench: Evaluating Harm-Level Compliance and the Impact of Quantization on Model Alignment