Layer Similarity

Layer similarity research focuses on understanding and quantifying the relationships between representations learned at different layers within deep neural networks (DNNs), particularly in transformer and convolutional neural networks. Current research employs various methods, including cosine similarity, centered kernel alignment (CKA), and graph-based approaches, to analyze these relationships across different models and training strategies, often with applications in model compression, efficient training, and federated learning. These analyses provide insights into model behavior, enabling improvements in training efficiency (e.g., through knowledge distillation or multi-exit architectures) and facilitating better understanding of DNN internal representations for improved interpretability and robustness.

Papers

June 20, 2024

On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier
Jiachen Jiang, Jinxin Zhou, Zhihui Zhu
Application Proficiency Transformer Model High Similarity Internal Representation Multi Exit Kernel Alignment Standard Transformer Layer Similarity

April 7, 2024

A Multi-Level Framework for Accelerating Training Transformer Models
Longwei Zou, Han Zhang, Yangdong Deng
Transformer Based Language Model Hierarchical Framework Transformer Training BERT Large Layer Similarity

February 26, 2024

SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
Luca Zampierin, Ghouthi Boukli Hacene, Bac Nguyen, Mirco Ravanelli
Knowledge Distillation Self Supervised Learning Self Supervised Low Temperature Distillation Speech Processing Task Layer Similarity

August 22, 2023

Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning
Yun-Hin Chan, Rui Zhou, Running Zhao, Zhihan Jiang, Edith C. -H. Ngai
Heterogeneous Federated Learning Homogeneity Bias System Heterogeneity Layer Similarity

April 30, 2023

Revealing Similar Semantics Inside CNNs: An Interpretable Concept-based Comparison of Feature Spaces
Georgii Mikriukov, Gesina Schwalbe, Christian Hellert, Korinna Bade
Convolutional Neural Network Comparative Study Feature Space CNN Architecture Interpretable Way CNN Performance Layer Similarity

November 22, 2021

Graph-Based Similarity of Neural Network Representations
Zuohui Chen, Yao Lu, Jinxuan Hu, Wen Yang, Qi Xuan, Zhen Wang, Xiaoniu Yang
Black Box High Similarity Neural Network Representation Layer Similarity Graph Similarity Computation

Layer Similarity

Papers

On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier

A Multi-Level Framework for Accelerating Training Transformer Models

SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning

Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning

Revealing Similar Semantics Inside CNNs: An Interpretable Concept-based Comparison of Feature Spaces

Graph-Based Similarity of Neural Network Representations