Multiplicative Size Scaling

Multiplicative size scaling in machine learning investigates how model performance changes with increases in model parameters, training data, and other resources. Current research focuses on optimizing this scaling across various model architectures, including transformers, diffusion models, and graph neural networks, often employing techniques like parameter-efficient fine-tuning and improved data sampling strategies to enhance efficiency and generalization. These investigations are crucial for developing more powerful and resource-efficient AI systems, impacting fields ranging from natural language processing and computer vision to scientific computing and robotics. A key theme is moving beyond simple scaling to understand and optimize the interplay between model size, data quality, and training methodologies.

Papers

October 2, 2023

MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training
Daegun Yoon, Sangyoon Oh
Multiplicative Size Scaling Random Sparsification Deep Neural Network Training Gradient Sparsification Aggregated Gradient Micro Level

September 30, 2023

Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement
Kai Xu, Rongyu Chen, Gianni Franchi, Angela Yao
Distribution Detection Multiplicative Size Scaling Training Time Modern Deep Learning

September 18, 2023

An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, Yelong Shen
Fine Tuning Empirical Study Large Multimodal Model Multimodal Data Multiplicative Size Scaling Visual Instruction Tuning

August 23, 2023

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, Quanquan Gu
Large Language Model Language Model Multiplicative Size Scaling Diffusion Language Model

August 7, 2023

Scaling may be all you need for achieving human-level object recognition capacity with human-like visual experience
A. Emin Orhan
Vision Transformer Object Recognition Multiplicative Size Scaling

August 3, 2023

Lode Enhancer: Level Co-creation Through Scaling
Debosmita Bhaumik, Julian Togelius, Georgios N. Yannakakis, Ahmed Khalifa
Neural Network Deep Neural Network Neural Network Architecture Multiplicative Size Scaling Tile Based Level Design CPA Enhancer

July 27, 2023

Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions
Timo Wilm, Philipp Normann, Sophie Baumeister, Paul-Vincent Kobow
Loss Function E Commerce Negative Sampling Multiplicative Size Scaling Recommendation Quality Recommendation Accuracy Recommender Transformer

July 26, 2023

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Huy Ha, Pete Florence, Shuran Song
Natural Language Multiplicative Size Scaling Skill Learning Language Conditioned Manipulation Trajectory

July 11, 2023

Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing
Sanae Amani, Khushbu Pahwa, Vladimir Braverman, Lin F. Yang
Optimal Policy Multiplicative Size Scaling Multi Task Reinforcement Learning Contextual Markov Decision Process Experience Sharing

June 26, 2023

Scaling and Resizing Symmetry in Feedforward Networks
Carlos Cardona
Neural Network Multiplicative Size Scaling Phase Transition Weight Initialization Symmetry Breaking Criticality Ordered Spin Sequence Initial Weight

June 21, 2023

EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
Yi-Lun Liao, Brandon Wood, Abhishek Das, Tess Smidt
Multiplicative Size Scaling Level Representation Atomistic System Equivariant Representation Equivariant Transformer

June 19, 2023

Scaling of Class-wise Training Losses for Post-hoc Calibration
Seungjin Jung, Seungmo Seo, Yonghyun Jeong, Jongwon Choi
Multiplicative Size Scaling Intra Class Variation Training Loss Scaling Factor

June 1, 2023

Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Mahoney, Amir Gholami
Fine Tuning Transfer Learning Foundation Model Pre Trained Model Multiplicative Size Scaling Scientific Machine Learning SciML Model

May 29, 2023

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees
Jihao Xin, Marco Canini, Peter Richtárik, Samuel Horváth
Multiplicative Size Scaling Distributed Learning Theoretical Guarantee Unbiased Compression

May 24, 2023

May 16, 2023

A Deep RL Approach on Task Placement and Scaling of Edge Resources for Cellular Vehicle-to-Network Service Provisioning
Cyril Shih-Huan Hsu, Jorge Martín-Pérez, Danny De Vleeschauwer, Koteswararao Kondepu, Luca Valcarenghi, Xi Li, Chrysa Papagianni
Multiplicative Size Scaling New Resource Deep RL Global Placement

May 13, 2023

Convergence and scaling of Boolean-weight optimization for hardware reservoirs
Louis Andreoli, Stéphane Chrétien, Xavier Porte, Daniel Brunner
Neural Network Early Stage Convergence Multiplicative Size Scaling Compute Node Photonic Reservoir Hardware Architecture Pseudo Boolean Optimization Physical Reservoir Computer

April 3, 2023

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, Oskar van der Wal
Large Language Model Training Data Training Dynamic Multiplicative Size Scaling Software Suite

March 27, 2023

Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture
Peiyu Liu, Ze-Feng Gao, Yushuo Chen, Wayne Xin Zhao, Ji-Rong Wen
Language Model Pre Trained Language Model Parameter Efficient Multiplicative Size Scaling Low Rank Tensor Efficient Architecture Deep Depth Deeper Model Matrix Product Operator

Multiplicative Size Scaling

Papers

MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training

Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement

An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning

Scaling may be all you need for achieving human-level object recognition capacity with human-like visual experience

Lode Enhancer: Level Co-creation Through Scaling

Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing

Scaling and Resizing Symmetry in Feedforward Networks

EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations

Scaling of Class-wise Training Losses for Post-hoc Calibration

Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Emergent inabilities? Inverse scaling over the course of pretraining

A Deep RL Approach on Task Placement and Scaling of Edge Resources for Cellular Vehicle-to-Network Service Provisioning

Convergence and scaling of Boolean-weight optimization for hardware reservoirs

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture