Last Layer

Research on "last layer" focuses on the final layer of neural networks, particularly in large language models (LLMs) and image classifiers, aiming to understand its role in prediction, calibration, and generalization. Current research investigates the last layer's contribution to overall model performance, exploring techniques like retraining, recalibration (e.g., using geometric adjustments), and Bayesian approaches to improve uncertainty quantification and robustness. These studies are significant because they offer avenues for enhancing model accuracy, reliability, and efficiency, impacting various applications from natural language processing to computer vision and beyond.

Papers

February 21, 2023

Improved uncertainty quantification for neural networks with Bayesian last layer
Felix Fiedler, Sergio Lucia
Neural Network Bayesian Neural Network Bayesian Perspective Last Layer Bayesian Regression

January 3, 2023

Understanding Imbalanced Semantic Segmentation Through Neural Collapse
Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, Jiaya Jia
Semantic Segmentation Neural Collapse Semantic Segmentation Benchmark Last Layer Class Imbalanced Feature Richness Enhancement

November 9, 2022

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token
Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, Christof Monz
Language Model Masked Language K TOKEN Downstream NLP Task Mask Frozen DETR Last Layer Unmasked Token

September 20, 2022

CoV-TI-Net: Transferred Initialization with Modified End Layer for COVID-19 Diagnosis
Sadia Khanam, Mohammad Reza Chalak Qazani, Subrota Kumar Mondal, H M Dipu Kabir, Abadhan S. Sabyasachi, Houshyar Asadi, Keshav Kumar, Farzin Tabarsinezhad, Shady Mohamed, Abbas Khorsavi, Saeid Nahavandi
Convolutional Neural Network Transfer Learning Pre Trained Model Covid 19 COVID 19 Diagnosis Last Layer

September 13, 2022

Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling
Dongsuk Oh, Yejin Kim, Hodong Lee, H. Howie Huang, Heuiseok Lim
Language Model Contrastive Learning Sentence Representation Pre Trained Embeddings Attention Based Pooling Last Layer

July 11, 2022

Embedding Recycling for Language Models
Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D'Arcy, Arman Cohan, Doug Downey
Language Model Jina Embeddings Neural Language Model Advanced Textile Recycling Last Layer Pretrained Model

July 4, 2022

LaTeRF: Label and Text Driven Object Radiance Fields
Ashkan Mirzaei, Yash Kant, Jonathan Kelly, Igor Gilitschenski
Radiance Field Object Representation Volumetric Representation Full Label 3D Shape Representation Last Layer

March 2, 2022

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, Zhihui Zhu
Cross Entropy Loss Neural Collapse Global Optimality Mean Squared Error Optimization Landscape Last Layer Unconstrained Feature Model

December 21, 2021

Generalized Few-Shot Semantic Segmentation: All You Need is Fine-Tuning
Josh Myers-Dean, Yinan Zhao, Brian Price, Scott Cohen, Danna Gurari
Semantic Segmentation Fine Tuning Shot Segmentation Shot Semantic Segmentation Triplet Loss Last Layer

Last Layer

Papers

Improved uncertainty quantification for neural networks with Bayesian last layer

Understanding Imbalanced Semantic Segmentation Through Neural Collapse

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token

CoV-TI-Net: Transferred Initialization with Modified End Layer for COVID-19 Diagnosis

Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling

Embedding Recycling for Language Models

LaTeRF: Label and Text Driven Object Radiance Fields

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

Generalized Few-Shot Semantic Segmentation: All You Need is Fine-Tuning