Hadamard Layer

The Hadamard transform, a fast and efficient mathematical operation, is finding increasing application across diverse fields. Current research focuses on leveraging its properties for parameter-efficient fine-tuning of large language models, accelerating quantum computations, and optimizing backpropagation in neural networks, often employing low-rank approximations and quantization techniques. These advancements aim to improve computational efficiency and reduce memory requirements in machine learning and quantum computing, with significant implications for resource-constrained applications and large-scale model deployment. Furthermore, the Hadamard transform is being explored for its potential in solving optimization problems in non-Euclidean spaces and enhancing the performance of convolutional neural networks.

Papers

September 20, 2024

HUT: A More Computation Efficient Fine-Tuning Method With Hadamard Updated Transformation
Geyuan Zhang, Xiaofei Zhou, Chuheng Chen
Pre Trained Language Model Parameter Efficient Fine Tuning Data Transformation Hadamard Layer Hadamard Transform

September 4, 2024

Hadamard Row-Wise Generation Algorithm
Brayan Monroy, Jorge Bacca
Efficient Algorithm Kronecker Product Hadamard Layer Single Pixel Imaging

August 10, 2024

Efficient Quantum Gradient and Higher-order Derivative Estimation via Generalized Hadamard Test
Dantong Li, Dikshant Dulal, Mykhailo Ohorodnikov, Hanrui Wang, Yongshan Ding
Variational Quantum Algorithm Parameterized Quantum Circuit Near Term Quantum Hadamard Layer Quantum Natural Gradient Circuit Optimization

June 21, 2024

HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization
Seonggon Kim, Eunhyeok Park
Low Rank Approximation Efficient Deep Hadamard Layer Training Optimization Low Rank Quantization

March 23, 2024

Horoballs and the subgradient method
Adrian S. Lewis, Genaro Lopez-Acedo, Adriana Nicolae
Convex Optimization Subgradient Method Hadamard Layer Tangent Space

February 6, 2024

QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa
Post Training Quantization Vector Quantization Codebook Learning LLM Quantization Hadamard Layer Incoherence Processing

July 7, 2023

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
Chris Kolb, Christian L. Müller, Bernd Bischl, David Rügamer
Manipulation Edge Detection Module Derivative Free Optimization Sparse Regularization Parameter Efficient Sparsity Hadamard Layer Smooth Optimization

May 31, 2023

On the Linear Convergence of Policy Gradient under Hadamard Parameterization
Jiacai Liu, Jinchi Chen, Ke Wei
Policy Gradient Linear Convergence Local Convergence Hadamard Layer Deterministic Policy Gradient

May 27, 2023

A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layer
Hongyi Pan, Xin Zhu, Salih Atici, Ahmet Enis Cetin
Quantum Computer Efficient Hybrid Convolution Layer Quantum Classical Hybrid Quantum Classical Hadamard Layer Hadamard Transform

February 20, 2023

Hadamard Layer to Improve Semantic Segmentation
Angello Hoyos, Mariano Rivera
Semantic Segmentation Hadamard Layer Training Convergence

November 26, 2022

Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penalties
David Martínez-Rubio, Sebastian Pokutta
Participation Constraint Riemannian Optimization Accelerated Gradient Descent Anderson Acceleration Hadamard Layer Non Convex Penalty

September 16, 2022

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Yongtao Wu, Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Neural Network Neural Tangent Kernel Neural Network Architecture Extrapolation Framework Spectral Bias Standard Deep Polynomial Neural Network Hadamard Layer Polynomial Network

September 6, 2022

Low-Energy Convolutional Neural Networks (CNNs) using Hadamard Method
Varun Mannam
Convolutional Neural Network Convolutional Layer Convolution Layer Energy Efficient Convolution Technique Hadamard Layer

February 23, 2022

Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms
Rajarshi Saha, Mert Pilanci, Andrea J. Goldsmith
High Dimensional Information Theoretic Efficient Algorithm Linear Model Model Quantization Information Theoretic Framework Hadamard Layer Optimal Quantization

January 31, 2022

Equivariant neural networks for recovery of Hadamard matrices
Augusto Peres, Eduardo Dias, Luís Sarmento, Hugo Penedones
Convolutional Neural Network Neural Network Architecture Geometric Deep Learning Equivariant Neural Network Prompt Recovery Hadamard Layer

Hadamard Layer

Papers

HUT: A More Computation Efficient Fine-Tuning Method With Hadamard Updated Transformation

Hadamard Row-Wise Generation Algorithm

Efficient Quantum Gradient and Higher-order Derivative Estimation via Generalized Hadamard Test

HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization

Horoballs and the subgradient method

QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

On the Linear Convergence of Policy Gradient under Hadamard Parameterization

A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layer

Hadamard Layer to Improve Semantic Segmentation

Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penalties

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Low-Energy Convolutional Neural Networks (CNNs) using Hadamard Method

Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms

Equivariant neural networks for recovery of Hadamard matrices