Effective Parameter

Effective parameter research in deep learning aims to understand and optimize the relationship between model size and performance, focusing on identifying and utilizing only the most crucial parameters for improved efficiency and generalization. Current investigations explore techniques like network pruning, self-supervised learning for symmetry embedding, and analysis of Hessian matrices to quantify "effective" parameters, often within the context of deep reinforcement learning and large language models. These efforts are significant because they promise to reduce computational costs, improve training stability, and enhance the understanding of generalization in complex neural networks, ultimately leading to more efficient and powerful AI systems.

Papers

February 19, 2024

In value-based deep reinforcement learning, a pruned network is a good network
Johan Obando-Ceron, Aaron Courville, Pablo Samuel Castro
Network Programming Deep Reinforcement Performance Improvement Magnitude Pruning Value Based Deep Reinforcement Learning Network Characteristic Effective Parameter

December 11, 2023

Optimizing Likelihood-free Inference using Self-supervised Neural Symmetry Embeddings
Deep Chatterjee, Philip C. Harris, Maanas Goel, Malina Desai, Michael W. Coughlin, Erik Katsavounidis
Parameter Inference Learned Symmetry Likelihood Free Effective Parameter

May 29, 2023

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters
Lawrence Wang, Stephen J. Roberts
Gradient Descent Generalization Performance Hessian Matrix Neural Network Weight Civil Engineering Phase Node Feature Sharpness Effective Parameter

May 19, 2023

Exploring the Complexity of Deep Neural Networks through Functional Equivalence
Guohao Shen
Deep Learning Deep Neural Network Complexity Matter Overparametrization Bound Equivalence Class Effective Parameter

February 2, 2022

Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan
Language Model Many Parameter Scaling Law Dynamic Routing Effective Parameter

Effective Parameter

Papers

In value-based deep reinforcement learning, a pruned network is a good network

Optimizing Likelihood-free Inference using Self-supervised Neural Symmetry Embeddings

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters

Exploring the Complexity of Deep Neural Networks through Functional Equivalence

Unified Scaling Laws for Routed Language Models