Transformer Based LLM

Transformer-based large language models (LLMs) are a class of deep learning models designed to process and generate human-like text, with applications ranging from cybersecurity to materials science. Current research focuses on improving efficiency through techniques like low-rank parameterization and binarization, as well as enhancing capabilities in long-context processing and mitigating privacy risks via improved membership inference attack defenses. These advancements aim to reduce computational costs while maintaining or improving performance across various tasks, impacting fields requiring natural language understanding and generation.

Papers

November 5, 2024

Mixtures of In-Context Learners
Giwon Hong, Emile van Krieken, Edoardo Ponti, Nikolay Malkin, Pasquale Minervini
Context Learning Mixture Component Noisy Demonstration Different Context Transformer Based LLM Context Learner

September 9, 2024

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, Arash Eshghi
Transformer Megatron Decepticons Vision Paper State Space Model Visual Grounding Multimodal Retrieval Task Agnostic Representation Transformer Based LLM

September 5, 2024

Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers
Amit Ben-Artzy, Roy Schwartz
Attention Mechanism Human Attention Meaningful Representation Importance Aware State of the Art LLM Last Layer Decoder Only LLM Transformer Based LLM

August 14, 2024

Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey
Hamza Kheddar
Transformer Megatron Decepticons Comprehensive Survey NLP Field Intrusion Detection Cybersecurity Perspective Threat Detection Transformer Based LLM

July 13, 2024

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
Xiuying Wei, Skander Moalla, Razvan Pascanu, Caglar Gulcehre
Neural Network Training Data High Efficiency Low Rank Transformer Language Model Transformer Based LLM Scaling Analysis

July 9, 2024

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation
Liqun Ma, Mingjie Sun, Zhiqiang Shen
Large Language Model Medical LLM Scratch Project Transformer Based LLM Autoregressive Distillation

July 1, 2024

Needle in the Haystack for Memory Based Large Language Models
Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, Soham Dan
Language Model Needle Insertion Video Haystack Transformer Based LLM

June 24, 2024

June 4, 2024

Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem
Reid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang, Cynthia Dwork
Generative Language Model Input Representation Transformer Based LLM Prompting Strategy

April 2, 2024

Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations
Georgy Tyukin
Model Compression Inference Cost Inference Efficiency Token Generation Novel Architecture Transformer Based LLM

February 27, 2024

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei
Large Language Model New Era P Bit Transformer Based LLM

February 14, 2024

Spectral Filters, Dark Signals, and Attention Sinks
Nicola Cancedda
Projection Bias Intermediate Representation Second Ranked Logits Transformer Based LLM Weak Signal Attention Sink

November 21, 2023

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, Penghao Zhao
Comprehensive Survey Long Context Large Language Model Transformer Based LLM Long Input Context

October 24, 2023

In-Context Learning Creates Task Vectors
Roee Hendel, Mor Geva, Amir Globerson
Context Learning Machine Learning Framework New Learning Task Vector Transformer Based LLM

October 12, 2023

Context Compression for Auto-regressive Transformers with Sentinel Tokens
Siyu Ren, Qi Jia, Kenny Q. Zhu
Attention Module Sparse Attention Sentinel 1 Domain Specific Language Model Context Compression Transformer Based LLM Auto Regressive Transformer

September 13, 2023

Pretraining on the Test Set Is All You Need
Rylan Schaeffer
Smaller Language Model Test Set Transformer Based LLM Data Mixture Dataset Mixture

April 13, 2023

Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, Earl T. Barr
Language Model Code Summarization Transformer Based LLM Semantic Augmentation

August 5, 2022

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
Large Language Model Efficient Inference Parallel Training Transformer Based LLM Expert Language Model

April 25, 2022

Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials
Lai Wei, Qinyang Li, Yuqi Song, Stanislav Stefanov, Edirisuriya M. D. Siriwardane, Fanglin Chen, Jianjun Hu
Language Model Generative Question Material Response Probabilistic Generative Model Transformer Based LLM Material Informatics Material Language

Transformer Based LLM

Papers

Mixtures of In-Context Learners

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling

Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers

Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

Needle in the Haystack for Memory Based Large Language Models

Noisy Neighbors: Efficient membership inference attacks against LLMs

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem

Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Spectral Filters, Dark Signals, and Attention Sinks

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

In-Context Learning Creates Task Vectors

Context Compression for Auto-regressive Transformers with Sentinel Tokens

Pretraining on the Test Set Is All You Need

Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials