Pre Trained Language Model
Pre-trained language models (PLMs) are large neural networks trained on massive text datasets, aiming to capture the statistical regularities of language for various downstream tasks. Current research focuses on improving PLM efficiency through techniques like parameter-efficient fine-tuning and exploring their application in diverse fields, including scientific text classification, mental health assessment, and financial forecasting, often leveraging architectures like BERT and its variants. The ability of PLMs to effectively process and generate human language has significant implications for numerous scientific disciplines and practical applications, ranging from improved information retrieval to more sophisticated AI assistants.
Papers
Text-Augmented Open Knowledge Graph Completion via Pre-Trained Language Models
Pengcheng Jiang, Shivam Agarwal, Bowen Jin, Xuan Wang, Jimeng Sun, Jiawei Han
Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction?
Chenming Tang, Xiuyu Wu, Yunfang Wu
SETI: Systematicity Evaluation of Textual Inference
Xiyan Fu, Anette Frank
Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation
Tianyu Yang, Thy Thy Tran, Iryna Gurevych
Controlling Pre-trained Language Models for Grade-Specific Text Simplification
Sweta Agrawal, Marine Carpuat
Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation
Prashant Krishnan, Zilong Wang, Yangkun Wang, Jingbo Shang
Faithful Low-Resource Data-to-Text Generation through Cycle Training
Zhuoer Wang, Marcus Collins, Nikhita Vedula, Simone Filice, Shervin Malmasi, Oleg Rokhlenko
Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models
Amirhossein Kazemnejad, Mehdi Rezagholizadeh, Prasanna Parthasarathi, Sarath Chandar
Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models
Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, Zhu Zhang
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, Tengyu Ma
Reducing Sensitivity on Speaker Names for Text Generation from Dialogues
Qi Jia, Haifeng Tang, Kenny Q. Zhu
Abstractive Text Summarization Using the BRIO Training Paradigm
Khang Nhut Lam, Thieu Gia Doan, Khang Thua Pham, Jugal Kalita
Can LLMs facilitate interpretation of pre-trained language models?
Basel Mousi, Nadir Durrani, Fahim Dalvi
Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection
Rheeya Uppaal, Junjie Hu, Yixuan Li
SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations
Jesus Solano, Oana-Maria Camburu, Pasquale Minervini
DUMB: A Benchmark for Smart Evaluation of Dutch Models
Wietse de Vries, Martijn Wieling, Malvina Nissim