Attention Condenser

Attention condensers are efficient self-attention mechanisms designed to improve the speed and resource efficiency of deep neural networks, particularly for edge computing applications. Current research focuses on developing novel attention condenser architectures, such as double-condensing and spatially transformed versions, often integrated into convolutional neural networks (CNNs) like ResNets, to enhance accuracy while minimizing computational cost. This work is significantly impacting various fields, enabling faster and more efficient deep learning solutions for tasks such as skin cancer detection, PCB component identification, and automated visual inspection in manufacturing.

Papers

November 20, 2023

Double-Condensing Attention Condenser: Leveraging Attention in Deep Learning to Detect Skin Cancer from Skin Lesion Images
Chi-en Amy Tai, Elizabeth Janes, Chris Czarnecki, Alexander Wong
Deep Learning Human Attention Skin Lesion Skin Cancer Skin Cancer Detection Melanoma Classification Attention Condenser

June 16, 2023

Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysis
Andre Hryniowski, Alexander Wong
Self Attention Self Attention Module Convolutional Architecture Local Similarity Architectural Design Attention Condenser

May 23, 2023

CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generation
Jingning Xu, Benlai Tang, Mingjie Wang, Minghao Li, Meirong Ma
High Fidelity Face Generation Facial Video Lip Sync Probability MAP Talking Face Video Attention Condenser

January 23, 2023

PCBDet: An Efficient Deep Neural Network Object Detection Architecture for Automatic PCB Component Detection on the Edge
Brian Li, Steven Palayew, Francis Li, Saad Abbasi, Saeejith Nair, Alexander Wong
Extreme Edge Efficient Deep Deep Learning Based Object Attention Condenser

August 15, 2022

Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers
Alexander Wong, Mohammad Javad Shafiee, Saad Abbasi, Saeejith Nair, Mahmoud Famouri
Self Attention Extreme Edge Efficient Attention Efficient Neural Network Architecture Attention Condenser

April 25, 2022

Attention Condenser

Papers

Double-Condensing Attention Condenser: Leveraging Attention in Deep Learning to Detect Skin Cancer from Skin Lesion Images

Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysis

CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generation

PCBDet: An Efficient Deep Neural Network Object Detection Architecture for Automatic PCB Component Detection on the Edge

Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers

CellDefectNet: A Machine-designed Attention Condenser Network for Electroluminescence-based Photovoltaic Cell Defect Inspection

LightDefectNet: A Highly Compact Deep Anti-Aliased Attention Condenser Neural Network Architecture for Light Guide Plate Surface Defect Detection