Pre Trained Language Model
Pre-trained language models (PLMs) are large neural networks trained on massive text datasets, aiming to capture the statistical regularities of language for various downstream tasks. Current research focuses on improving PLM efficiency through techniques like parameter-efficient fine-tuning and exploring their application in diverse fields, including scientific text classification, mental health assessment, and financial forecasting, often leveraging architectures like BERT and its variants. The ability of PLMs to effectively process and generate human language has significant implications for numerous scientific disciplines and practical applications, ranging from improved information retrieval to more sophisticated AI assistants.
Papers
Modular Visual Question Answering via Code Generation
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training
Haode Zhang, Haowen Liang, Liming Zhan, Albert Y.S. Lam, Xiao-Ming Wu
Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models
Zhiyi Wang, Shaoguang Mao, Wenshan Wu, Yan Xia, Yan Deng, Jonathan Tien
Improving Vietnamese Legal Question--Answering System based on Automatic Data Enrichment
Thi-Hai-Yen Vuong, Ha-Thanh Nguyen, Quang-Huy Nguyen, Le-Minh Nguyen, Xuan-Hieu Phan
Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks
Kanishka Misra, Cicero Nogueira dos Santos, Siamak Shakeri
CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Models
Jiazheng Li, Zhaoyue Sun, Bin Liang, Lin Gui, Yulan He
Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion
Ryo Sekizawa, Hitomi Yanaka
Text-To-KG Alignment: Comparing Current Methods on Classification Tasks
Sondre Wold, Lilja Øvrelid, Erik Velldal
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research
Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Alham Fikri Aji, Genta Indra Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Models
Fengzhu Zeng, Wei Gao
EEL: Efficiently Encoding Lattices for Reranking
Prasann Singhal, Jiacheng Xu, Xi Ye, Greg Durrett
Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers
Jiaxi Li, Wei Lu
Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Baohao Liao, Shaomu Tan, Christof Monz
Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting
Shubin Huang, Qiong Wu, Yiyi Zhou, Weijie Chen, Rongsheng Zhang, Xiaoshuai Sun, Rongrong Ji