BERT Variant

BERT variant research focuses on improving the efficiency, robustness, and applicability of the original BERT architecture for natural language processing. Current efforts concentrate on optimizing training speed through techniques like improved load balancing and novel optimizers, enhancing model resilience to parameter corruption, and developing smaller, more efficient models via methods such as dynamic embeddings and quantization. These advancements are significant for deploying BERT in resource-constrained environments and for improving the performance and scalability of various NLP applications across diverse languages and domains.

Papers

June 20, 2024

Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models
Shijie Han, Zhenyu Zhang, Andrei Arsene Simion
Language Model Ticket BERT Language Model Fine Tuning BERT Variant

February 4, 2024

Breaking MLPerf Training: A Case Study on Optimizing BERT
Yongdeok Kim, Jaehyung Ahn, Myeongwoo Kim, Changin Choi, Heejae Kim, Narankhuu Tuvshinjargal, Seungwon Lee, Yanzi Zhang, Yuan Pei, Xiongzhan Linghu, Jingkun Ma, Lin Chen, Yuehua Dai, Sungjoo Yoo
Case Study BERT Model Pre Trained BERT Machine Learning Benchmark BERT Variant Efficient BERT

December 10, 2023

FP8-BERT: Post-Training Quantization for Transformer
Jianwei Li, Tianchi Zhang, Ian En-Hsu Yen, Dongkuan Xu
Transformer Based BERT Model Ticket BERT Quantization Operator Quantization Aware Training BERT Variant 8 Bit Quantization

October 31, 2023

EELBERT: Tiny Models through Dynamic Embeddings
Gabrielle Cohn, Rishika Agarwal, Deepanshu Gupta, Siddharth Patwardhan
Transformer Based Model BERT Model Small Model Dynamic Embeddings BERT Variant

October 19, 2023

Transformer-based Entity Legal Form Classification
Alexander Arimond, Mauro Molteni, Dominik Jany, Zornitsa Manolova, Damian Borth, Andreas G. F. Hoepner
Text Classification Transformer Based Language Model BERT Based BERT Variant Name Entity

July 26, 2023

Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models
Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Development Activity Zero Shot Classification Smaller Language Model Low Resourced Language BERT Variant

June 24, 2023

Comparison of Pre-trained Language Models for Turkish Address Parsing
Muhammed Cihat Ünal, Betül Aygün, Aydın Gerek
Fine Tuning Pre Trained Language Model Large Corpus Consistent Comparison Multilingual Model BERT Variant Address Parsing

May 3, 2023

Evaluating BERT-based Scientific Relation Classifiers for Scholarly Knowledge Graph Construction on Digital Library Collections
Ming Jiang, Jennifer D'Souza, Sören Auer, J. Stephen Downie
Knowledge Graph Domain Specific BERT Based Semantic Relation BERT Variant Digital Library

November 30, 2022

HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression
Jiaqi Gu, Ben Keller, Jean Kossaifi, Anima Anandkumar, Brucek Khailany, David Z. Pan
Tensor Decomposition High Temperature BERT Variant Transformer Compression

October 21, 2022

Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
Ran Tian, Ankur P. Parikh
Learning Rate Adam Optimizer Weight Decay BERT Variant Model Scale Stochastic Gradient Based

October 15, 2022

AraLegal-BERT: A pretrained language model for Arabic Legal text
Muhammad AL-Qurishi, Sarah AlQaseemi, Riad Soussi
Language Model BERT Model BERT Variant Arabic Legal

October 12, 2022

GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods
Eldar Kurtic, Dan Alistarh
Large Language Model NCD Method Strong Baseline Magnitude Pruning BERT Variant

September 29, 2022

Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging
Jean Kaddour
Language Model Supervised ImageNet Time Matter Faster Convergence Large Datasets Model Checkpoint WEight AVERaging BERT Variant Multiple Day