Pre Trained Language Model

Pre-trained language models (PLMs) are large neural networks trained on massive text datasets, aiming to capture the statistical regularities of language for various downstream tasks. Current research focuses on improving PLM efficiency through techniques like parameter-efficient fine-tuning and exploring their application in diverse fields, including scientific text classification, mental health assessment, and financial forecasting, often leveraging architectures like BERT and its variants. The ability of PLMs to effectively process and generate human language has significant implications for numerous scientific disciplines and practical applications, ranging from improved information retrieval to more sophisticated AI assistants.

Papers

November 20, 2023

Sparse Low-rank Adaptation of Pre-trained Language Models
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, Maosong Sun
Pre Trained Language Model Low Rank Adaptation Pre Trained Large Language Model Dynamic Adaptation

November 19, 2023

Can We Utilize Pre-trained Language Models within Causal Discovery Algorithms?
Chanhui Lee, Juhyeon Kim, Yongjun Jeong, Juhyun Lyu, Junghee Kim, Sangmin Lee, Sangjun Han, Hyeokjun Choe, Soyeon Park, Woohyung Lim, Sungbin Lim, Sanghack Lee
Pre Trained Language Model Causal Discovery Causal Reasoning Causal Discovery Algorithm

November 16, 2023

Evolving Domain Adaptation of Pretrained Language Models for Text Classification
Yun-Shiuan Chuang, Yi Wu, Dhruv Gupta, Rheeya Uppaal, Ananya Kumar, Luhang Sun, Makesh Narsimhan Sreedhar, Sijia Yang, Timothy T. Rogers, Junjie Hu
Domain Adaptation Pre Trained Language Model Text Classification Pretrained Language Model Domain Adaptive Domain Adversarial

November 7, 2023

Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection
Benjamin Steenhoek, Md Mahbubur Rahman, Shaila Sharmin, Wei Le
Pre Trained Language Model Case Study Real World Code Vulnerability Detection Software Vulnerability Prediction Vulnerability Data

November 2, 2023

Continual Learning Under Language Shift
Evangelia Gogoulou, Timothée Lesort, Magnus Boman, Joakim Nivre
Language Model Continual LEArning Pre Trained Language Model Monolingual Language Model Language Change Backward Transfer

November 1, 2023

October 31, 2023

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S. Du, Huazhe Xu
Pre Trained Language Model Offline Reinforcement Learning Real Power Dense Reward

October 30, 2023

Automatic Evaluation of Generative Models with Instruction Tuning
Shuhaib Mehri, Vered Shwartz
Language Model Generative Model Pre Trained Language Model Language Generation Instruction Tuning Automatic Evaluation Instruction Tuned Model

October 29, 2023

Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning
Sapan Shah, Sreedhar Reddy, Pushpak Bhattacharyya
Language Model Pre Trained Language Model Sentiment Analysis Experienced Emotion Target Emotion Supervised Contrastive Learning Lightweight Large Language Model

October 27, 2023

October 26, 2023

October 24, 2023

Pre Trained Language Model

Papers

Sparse Low-rank Adaptation of Pre-trained Language Models

Can We Utilize Pre-trained Language Models within Causal Discovery Algorithms?

Evolving Domain Adaptation of Pretrained Language Models for Text Classification

Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection

Continual Learning Under Language Shift

Can Large Language Models Design Accurate Label Functions?

Unsupervised Lexical Simplification with Context Augmentation

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

Automatic Evaluation of Generative Models with Instruction Tuning

Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning

Fine-Tuning Language Models Using Formal Methods Feedback

ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation

Ontology Revision based on Pre-trained Language Models

Nearest Neighbor Search over Vectorized Lexico-Syntactic Patterns for Relation Extraction from Financial Documents

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

ChatGPT is a Potential Zero-Shot Dependency Parser

Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models

Speakerly: A Voice-based Writing Assistant for Text Composition

Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models