Transformer Model

November 22, 2024

Evaluating Vision Transformer Models for Visual Quality Control in Industrial Manufacturing
Miriam Alber, Christoph Hönes, Patrick Baier
Transformer Architecture Transformer Model Detection Performance Visual Inspection Manufacturing Industry Visual Quality Inspection
FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration
Donghyeon Yi, Seoyoung Lee, Jongho Kim, Junyoung Kim, Sohmyung Ha, Ik Joon Chang, Minkyu Je
Retrieval Augmented Generation Transformer Model Post Training Quantization Analog in Memory Computing Transformer Accelerator

November 20, 2024

MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers
Ning Ding, Yehui Tang, Haochen Qin, Zhenli Zhou, Chao Xu, Lin Li, Kai Han, Heng Liao, Yunhe Wang
Large Language Model Transformer Model Fully Connected Layer

November 19, 2024

November 16, 2024

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason M. Klusowski, Jianqing Fan, Mengdi Wang
Context Learning Transformer Model Context Information Layer Transformer Softmax Attention

November 15, 2024

ULTra: Unveiling Latent Token Interpretability in Transformer Based Understanding
Hesam Hosseini, Ghazal Hosseini Mighan, Amirabbas Afzali, Sajjad Amini, Amir Houmansadr
Semantic Segmentation Transformer Based Transformer Model Segmentation Model Human Understanding Token Level Zero Shot Semantic Segmentation Latent Label

November 14, 2024

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
Zixing Zhang, Zhongren Dong, Weixiang Xu, Jing Han
Emotion Recognition Transformer Model Lightweight Transformer

November 12, 2024

Unraveling the Gradient Descent Dynamics of Transformers
Bingqing Song, Boran Han, Shuai Zhang, Jie Ding, Mingyi Hong
Transformer Megatron Decepticons Transformer Architecture Transformer Model Layer Transformer Softmax Attention Gradient Descent Dynamic

November 11, 2024

1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs
Jebish Purbey, Siddartha Pullakhandam, Kanwal Mehreen, Muhammad Arham, Drishti Sharma, Ashay Srivastava, Ram Mohan Rao Kadiyala
Large Language Model Natural Language Processing Data Detection Medical LLM Transformer Model Hate Speech NLU Model Indic Script Language Detection

November 5, 2024

Enhancing Transformer Training Efficiency with Dynamic Dropout
Hanrui Yan, Dan Shao
Transformer Model Regularization Model Transformer Training Dropout Layer

November 4, 2024

Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference
Shashank Nag, Alan T. L. Bacellar, Zachary Susskind, Anshul Jha, Logan Liberty, Aishwarya Sivakumar, Eugene B. John, Krishnan Kailas, Priscila M. V. Lima, Neeraja J. Yadwadkar, Felipe M. G. Franca, Lizy K. John
Scientific Inference Transformer Model Transformer Based Model Low Resource Energy Efficient Inference Lightweight Transformer Mini Giant

October 31, 2024

LSEAttention is All You Need for Time Series Forecasting
Dizhen Liang
Transformer Model Time Series Forecasting Multivariate Time Series Multivariate Time Series Forecasting Transformer Based Architecture

October 30, 2024

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
Zhichao Hou, Weizhi Gao, Yuchen Shen, Feiyi Wang, Xiaorui Liu
Transformer Model Plug and Play Transformer Based Architecture Robust Transformer Robust Attention

October 26, 2024

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
Avinash Maurya, Jie Ye, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae
Large Language Model Transformer Model Task Offloading GPU Memory Scalable Training Hybrid CPU GPU

October 25, 2024

Multi-Agent Reinforcement Learning with Selective State-Space Models
Jemma Daniel, Ruan de Kock, Louay Ben Nessir, Sasha Abramowitz, Omayma Mahjoub, Wiem Khlifi, Claude Formanek, Arnu Pretorius
Multi Agent Reinforcement Learning Transformer Model Mamba Based Multi Agent Environment Selective State Space Transformer Agent

October 24, 2024

October 23, 2024

Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers
Valeria Ruscio, Fabrizio Silvestri
Transformer Model Meaningful Representation Memory Trace Internal Representation Product Specific Position Information Autoregressive Transformer Rotary Position

Papers

Evaluating Vision Transformer Models for Visual Quality Control in Industrial Manufacturing

FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration

MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers

Comparing Prior and Learned Time Representations in Transformer Models of Timeseries

Ultra-Sparse Memory Network

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context

ULTra: Unveiling Latent Token Interpretability in Transformer Based Understanding

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition

Unraveling the Gradient Descent Dynamics of Transformers

1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs

Enhancing Transformer Training Efficiency with Dynamic Dropout

Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference

LSEAttention is All You Need for Time Series Forecasting

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Multi-Agent Reinforcement Learning with Selective State-Space Models

On Explaining with Attention Matrices

A Comprehensive Survey of Time Series Forecasting: Architectural Diversity and Open Challenges

The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI

Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers