Model Capacity

Model capacity, the ability of a machine learning model to learn complex patterns from data, is a crucial factor determining performance and efficiency. Current research focuses on understanding how capacity impacts active learning strategies, optimizing capacity in transformer networks and Mixture-of-Experts (MoE) models, and developing efficient training methods for large models, including parameter-efficient fine-tuning and techniques like knowledge distillation. These investigations are vital for improving the performance and resource efficiency of AI systems across diverse applications, from natural language processing to resource-constrained IoT devices. Ultimately, a deeper understanding of model capacity is essential for building more powerful and practical AI systems.

Papers

August 24, 2024

Understanding Uncertainty-based Active Learning Under Model Mismatch
Amir Hossein Rahmati, Mingzhou Fan, Ruida Zhou, Nathan M. Urban, Byung-Jun Yoon, Xiaoning Qian
Prediction Uncertainty Prediction Performance Uncertainty Based Active Learning Model Mismatch Model Capacity

July 22, 2024

Empirical Capacity Model for Self-Attention Neural Networks
Aki Härmä, Marcin Pietrasik, Anna Wilbik
Large Transformer Model Model Capacity General Transformer Memorization Capability

July 7, 2024

Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients
Shaoyuan Chen, Linlin You, Rui Liu, Shuo Yu, Ahmed M. Abdelmoniem
Knowledge Transfer Joint Learning Resource Constrained Model Capacity

July 1, 2024

Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning
Haobo Song, Hao Zhao, Soumajit Majumder, Tao Lin
Foundation Model Parameter Efficient Fine Tuning Low Rank Simple Strategy Model Capacity

June 26, 2024

A Survey on Mixture of Experts
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, Jiayi Huang
Large Language Model Natural Language Processing Timely Survey Context Learning Mixture Component Expert Knowledge Model Capacity

May 23, 2024

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
Xianzhi Du, Tom Gunter, Xiang Kong, Mark Lee, Zirui Wang, Aonan Zhang, Nan Du, Ruoming Pang
Model Complexity LLM Training Dense Model Model Capacity Accuracy Tradeoff Scale LLM MoE FFD

March 4, 2024

Vanilla Transformers are Transfer Capability Teachers
Xin Lu, Yanyan Zhao, Bing Qin
Mixture of Expert BERT Architecture Vanilla Transformer Model Capacity Multi Level Distillation

February 14, 2024

Under manipulations, are some AI models harder to audit?
Augustin Godinot, Gilles Tredan, Erwan Le Merrer, Camilla Penzo, Francois Taïani
Black Box State of the Art Artificial Intelligence Model Manipulation Strategy Model Capacity Robust Method

November 27, 2023

Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence
Svetlana Pavlitska, Hannes Grolig, J. Marius Zöllner
Adversarial Robustness Narrative Review Model Compression Evidence Piece Human Relationship Deep Learning Network Model Capacity

November 14, 2023

AutoML for Large Capacity Modeling of Meta's Ranking Systems
Hang Yin, Kuang-Hung Liu, Mengying Sun, Yuxin Chen, Buyun Zhang, Jiang Liu, Vivek Sehgal, Rudresh Rajnikant Panchal, Eugen Hotaj, Xi Liu, Daifeng Guo, Jamey Zhang, Zhou Wang, Shali Jiang, Huayu Li, Zhengxing Chen, Wen-Yen Chen, Jiyan Yang, Wei Wen
Online autoML Ranking Model Meta Level AutoML System Model Capacity autoML System

November 7, 2023

Cup Curriculum: Curriculum Learning on Model Capacity
Luca Scharr, Vanessa Toborek
Natural Language Processing Curriculum Learning Magnitude Pruning Model Capacity Training Iteration

October 29, 2023

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai "Helen" Li, Yiran Chen
High Efficiency Mixture of Expert Potential Scalability Efficient Inference Efficient Model Model Capacity Sparsity Aware

June 7, 2023

MobileNMT: Enabling Translation in 15MB and 30ms
Ye Lin, Xiaohui Wang, Zhexi Zhang, Mingxuan Wang, Tong Xiao, Jingbo Zhu
Model Compression View Translation NMT Model Model Capacity Auto Regressive Decoding

May 23, 2023

One-stop Training of Multiple Capacity Models
Lan Jiang, Haoyang Huang, Dongdong Zhang, Rui Jiang, Furu Wei
Knowledge Distillation Joint Learning Training Model Two Stage Training End to End Training Model Capacity Unified Training

January 31, 2023

The Power of External Memory in Increasing Predictive Model Capacity
Cenk Baykal, Dylan J Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, Xin Wang
Deep Network Real Power Softmax Function Input Representation Model Capacity Memory Capacity Binary Token

June 17, 2022

Beyond Ridge Regression for Distribution-Free Data
Koby Bibas, Meir Feder
Empirical Risk Minimization Maximum Likelihood Ridge Regression Model Capacity

April 16, 2022

Cannikin's Law in Tensor Modeling: A Rank Study for Entanglement and Separability in Tensor Complexity and Model Capacity
Tong Yang
Quantum Entanglement Linear Separability Order Tensor Model Capacity Tensor Model Tensor Rank Tensor Optimization