Smooth Representation

Smooth representation in machine learning aims to create continuous and differentiable representations of data, improving model generalization, robustness, and uncertainty quantification. Current research focuses on applying this principle to various model architectures, including implicit neural representations (INRs), graph neural networks (GNNs), and pre-trained language models (PLMs), often employing techniques like Jacobian and Hessian regularization or k-nearest neighbor smoothing. These advancements address challenges like over-smoothing in GNNs and discrete input limitations in NLP, leading to improved performance in tasks ranging from pose estimation and 3D reconstruction to natural language processing. The resulting smoother representations enhance model accuracy, efficiency, and interpretability across diverse applications.

Papers

March 31, 2024

From Robustness to Improved Generalization and Calibration in Pre-trained Language Models
Josip Jukić, Jan Šnajder
Native Robustness Strong Generalization Pre Trained Language Model Calibration Performance Hessian Matrix Improved Generalization Smooth Representation

January 15, 2024

Online Learning of Continuous Signed Distance Fields Using Piecewise Polynomials
Ante Marić, Yiming Li, Sylvain Calinon
Online Learning Implicit Representation Shape Prior Gradient Information Piecewise Polynomial Polynomial Basis Smooth Representation

October 28, 2023

INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddings
Amirhossein Kazerouni, Reza Azad, Alireza Hosseini, Dorit Merhof, Ulas Bagci
Neural Radiance Field Signal Representation Sinusoidal Activation Implicit Learning Smooth Representation

June 10, 2023

INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation
Wenhao Zhu, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen
Neural Machine Translation Knowledge Based Nearest Neighbor Machine Translation Digital Ink kNN MT Smooth Representation

December 20, 2022

Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis
Josip Jukić, Jan Šnajder
Pre Trained Language Model Active Learning Supervised Learning Large Pre Trained Language Model Average Smoothness Smooth Representation Label Complexity

August 18, 2022

GraTO: Graph Neural Network Framework Tackling Over-smoothing with Neural Architecture Search
Xinshun Feng, Herun Wan, Shangbin Feng, Hongrui Wang, Jun Zhou, Qinghua Zheng, Minnan Luo
Graph Neural Network Neural Architecture Search Undisciplined Over Smoothing Smoothing Problem GNN Layer Smooth Representation

June 24, 2022

Learning sparse features can lead to overfitting in neural networks
Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Eijnden, Matthieu Wyart
Neural Network Deep Network Neural Representation Feature Learning Sparse Feature Smooth Representation

March 9, 2022

Probabilistic Rotation Representation With an Efficiently Computable Bingham Loss Function and Its Application to Pose Estimation
Hiroya Sato, Takuya Ikeda, Koichi Nishiwaki
Application Proficiency Loss Function Pose Estimation Fair Representation Beta Distribution Rotation Representation Smooth Representation Probabilistic Rotation

February 28, 2022

Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Xing Wu, Chaochen Gao, Meng Lin, Liangjun Zang, Zhongyuan Wang, Songlin Hu
Data Augmentation Data Augmentation Method Text Classification Task Smooth Representation Text Contour

Smooth Representation

Papers

From Robustness to Improved Generalization and Calibration in Pre-trained Language Models

Online Learning of Continuous Signed Distance Fields Using Piecewise Polynomials

INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddings

INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation

Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis

GraTO: Graph Neural Network Framework Tackling Over-smoothing with Neural Architecture Search

Learning sparse features can lead to overfitting in neural networks

Probabilistic Rotation Representation With an Efficiently Computable Bingham Loss Function and Its Application to Pose Estimation

Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks