BERT Architecture

BERT, a transformer-based architecture, is a foundational model in natural language processing (NLP) designed to improve understanding and processing of text. Current research focuses on enhancing BERT's performance and efficiency through techniques like multitask learning, ensemble methods, and architectural modifications such as incorporating convolutional layers or locality sensitive hashing for attention mechanisms. These advancements are driving improvements in various NLP tasks, including sentiment analysis, hate speech detection, and legal document processing, with significant implications for applications ranging from mental health assessment to malware detection. The ongoing exploration of BERT's capabilities and limitations continues to shape the landscape of NLP research.

Papers

October 21, 2024

Deep Learning and Data Augmentation for Detecting Self-Admitted Technical Debt
Edi Sutoyo, Paris Avgeriou, Andrea Capiluppi
Deep Learning Data Augmentation BERT Architecture Technical Debt Self Report

August 11, 2024

Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification
Christopher Sun, Abishek Satish
BERT Model Generative Adversarial BERT Embeddings BERT Architecture Auxiliary Classifier Multitask Finetuning

August 9, 2024

Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture
Kai Jiang, Honghao Yang, Yuexian Wang, Qianru Chen, Yiming Luo
Sentiment Analysis BERT Model Ensemble Learning Student Friendly Knowledge BERT Architecture Ensemble Neural Network Ensemble BERT

July 4, 2024

Deep Content Understanding Toward Entity and Aspect Target Sentiment Analysis on Foundation Models
Vorakit Vorakitphan, Milos Basic, Guilhaume Leroy Meline
Foundation Model Sentiment Analysis Entity Representation BERT Architecture Aspect Sentiment Triplet Extraction Aspect Category

June 2, 2024

Formality Style Transfer in Persian
Parastoo Falakaflaki, Mehrnoush Shamsfard
Style Transfer BERT Architecture Code Style Transfer

May 23, 2024

ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model
Luan Thanh Nguyen
Hate Speech Hate Speech Detection Text to Text Indonesian Language Transformer Based Pre Trained Language BERT Architecture Hate Speech Detection Model

May 6, 2024

Detecting Android Malware: From Neural Embeddings to Hands-On Validation with BERTroid
Meryam Chaieb, Mostafa Anouar Ghorab, Mohamed Aymen Saied
Malware Detection Feature Embeddings Android Malware BERT Architecture

April 15, 2024

April 12, 2024

BERT-LSH: Reducing Absolute Compute For Attention
Zezheng Li, Kingston Yip
Human Attention Self Attention Layer BERT Architecture Exact Algorithm

March 20, 2024

Efficient argument classification with compact language models and ChatGPT-4 refinements
Marcin Pietron, Rafał Olszowski, Jakub Gomułka
BERT Architecture Exploring ChatGPT Compact Language Model Argument Classification

March 13, 2024

Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking
Ming Dong, Yujing Chen, Miao Zhang, Hao Sun, Tingting He
Large Language Model BERT Model BERT Architecture Chinese Spelling Lexical Semantic

March 4, 2024

Vanilla Transformers are Transfer Capability Teachers
Xin Lu, Yanyan Zhao, Bing Qin
Mixture of Expert BERT Architecture Vanilla Transformer Model Capacity Multi Level Distillation

January 3, 2024

MLPs Compass: What is learned when MLPs are combined with PLMs?
Li Zhou, Wenyu Chen, Yong Cao, Dingyi Zeng, Wanlong Liu, Hong Qu
Multi Layer Perceptron Transformer Based Pre Trained Language BERT Architecture PLM Based

October 31, 2023

Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT
Aman Jaiswal, Evangelos Milios
BERT Model Ticket BERT Direct Convolution Pre Trained BERT Chunk Wise BERT Architecture Long Text Classification

September 12, 2023

Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO
Benjamin Icard, Vincent Claveau, Ghislain Atemezing, Paul Égré
Text Based Subjective Knowledge BERT Architecture Subjective Evaluation Expert System Neural Gas

July 28, 2023

Tutorials on Stance Detection using Pre-trained Language Models: Fine-tuning BERT and Prompting Large Language Models
Yun-Shiuan Chuang
Large Language Model Pre Trained Language Model Ticket BERT Stance Detection Tutorial Review Fine Tuned BERT BERT Architecture Video Comment Stance Analysis

June 25, 2023

Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input
Qingpei Guo, Kaisheng Yao, Wei Chu
Human Attention BERT Model Rich Input Multimodal Interaction Multimodal Machine Learning BERT Architecture

June 1, 2023

Training-free Neural Architecture Search for RNNs and Transformers
Aaron Serianni, Jugal Kalita
Transformer Megatron Decepticons Neural Architecture Search BERT Model Traditional RNNs BERT Architecture Free Neural Architecture Search

March 31, 2023

BERTino: an Italian DistilBERT model
Matteo Muffo, Enrico Bertino
Full Model BERT Architecture Many Natural Language Processing

BERT Architecture

Papers

Deep Learning and Data Augmentation for Detecting Self-Admitted Technical Debt

Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification

Ensemble BERT: A student social network text sentiment classification model based on ensemble learning and BERT architecture

Deep Content Understanding Toward Entity and Aspect Target Sentiment Analysis on Foundation Models

Formality Style Transfer in Persian

ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

Detecting Android Malware: From Neural Embeddings to Hands-On Validation with BERTroid

LegalPro-BERT: Classification of Legal Provisions by fine-tuning BERT Large Language Model

Detecting AI Generated Text Based on NLP and Machine Learning Approaches

BERT-LSH: Reducing Absolute Compute For Attention

Efficient argument classification with compact language models and ChatGPT-4 refinements

Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking

Vanilla Transformers are Transfer Capability Teachers

MLPs Compass: What is learned when MLPs are combined with PLMs?

Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT

Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO

Tutorials on Stance Detection using Pre-trained Language Models: Fine-tuning BERT and Prompting Large Language Models

Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input

Training-free Neural Architecture Search for RNNs and Transformers

BERTino: an Italian DistilBERT model