Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models

Joe Dwyer

cs.LGOct 5, 2026

Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models

Authors: Joe Dwyer

Organizations: ECPI University

Abstract

Large language models (LLMs) have achieved substantial performance gains through increases in model size, training data, and computational resources. However, traditional scaling approaches produce diminishing returns, rising financial and environmental costs, and barriers to participation for researchers operating outside large industrial laboratories. This review examines the evolution of LLM scaling theory from empirical scaling laws to compute-optimal training, with particular emphasis on parameter efficiency, token utilization, data efficiency, and resource-constrained environments. Foundational work on scaling laws is synthesized alongside later research on compute-optimal training, data pruning, efficient architectures, quantization, low-rank adaptation, and edge-oriented optimization. The literature indicates a shift from scale maximization toward more deliberate allocation of parameters, tokens, compute, and hardware resources. At the same time, important empirical, theoretical, and methodological gaps remain regarding whether scaling principles established on enterprise-grade infrastructure generalize to smaller models and constrained computing environments. This review organizes these developments into a unified framework for resource-efficient LLM training and argues that future progress should evaluate efficiency not solely through model performance, but through the relationship among performance, parameter count, computational cost, token allocation, and hardware constraints.

Figures & tables

Search term	Total results	Relevant studies	Date range
“LLM Training” AND (“Compute Overhead” OR “Resource Utilization”)	371	90	2021–2025
(“Parameter Efficiency” OR “Token Optimization”) AND “Scaling Laws”	333	37	2021–2025
“Compute-Optimal Scaling” AND (“Hoffmann et al.” OR “Scaling Efficiency”)	77	1	2021–2025
“Repeated Measures Design” AND (“AI Experiments” OR “Machine Learning”)	1101	15	2021–2025
“Edge Computing” AND (“Model Training” OR “Resource Constraints”)	19872	15	2021–2025
“Large Language Models” AND “Compute Efficiency” AND “Training Costs”	44	19	2021–2025

Table 1: Documented Search Queries in the Source Literature Review

Study	Primary focus	Key insight	Efficiency relevance
Kaplan et al. (2020)	Empirical scaling laws	Loss scales predictably with model size, data, and compute.	Establishes the baseline relationship among resources and performance.
Bahri et al. (2021)	Theory of scaling laws	Statistical and geometric principles help explain observed power laws.	Adds theoretical structure to empirical scaling behavior.
Hutter (2021)	Learning-curve theory	Formal learning bounds provide an alternative perspective on performance growth.	Frames efficiency and learning limits theoretically.
Hoffmann et al. (2022)	Compute-optimal scaling	Model size and training tokens should be balanced under fixed compute.	Reorients scaling around token-to-parameter allocation.
Boopathy and Fiete (2024)	Scale-time equivalence	Smaller models trained longer can sometimes approach larger-model performance.	Challenges model-size dominance.
Sorscher et al. (2022)	Data-centric pruning	Data quality and selection can improve scaling efficiency beyond naive data growth.	Shifts attention from data quantity to data utility.

Table 2: Selected Scaling Frameworks and Their Relevance to Compute Efficiency

Panel of Figure 1 — Figure 1: A synthesis of the shift from scale maximization to compute-optimal and resource-constrained LLM research. The five efficiency dimensions are derived from the themes represented in the source literature review.

Model	Developer	Parameters	Estimated tokens
GPT-2	OpenAI	1.5B	$\sim$ 40B
GPT-3	OpenAI	175B	$\sim$ 300B
Gopher	DeepMind	280B	$\sim$ 300B
MT-NLG	Microsoft/NVIDIA	530B	$\sim$ 270–400B
Jurassic-1	AI21 Labs	178B	$\sim$ 300B
PaLM	Google	540B	$\sim$ 780B

Table 3: Examples of Models Associated with Pre-Compute-Optimal Scaling Practices

Model	Developer	Parameters	Training tokens
Chinchilla	DeepMind	70B	1.4T
Phi-1	Microsoft	1.3B	54B a
Phi-2	Microsoft	2.7B	1.4T
Gemma 2B	Google	2B	2T
Gemma 7B	Google	7B	6T
TinyLlama	Open source	1.1B	$\sim$ 1T

Table 4: Examples Used in the Source Review to Illustrate Compute-Optimized or Resource-Conscious Training

Dimension	Primary optimization question	Representative literature from the source review
Data/token	How much and what kind of data are needed to obtain useful learning?	Hoffmann et al. (2022) ; Sorscher et al. (2022) ; Pang et al. (2024)
Parameter	How much useful performance is obtained from model capacity and compute?	Maloney et al. (2022) ; Boopathy and Fiete (2024) ; Hoffmann et al. (2022)
Architecture	Can model structure reduce active compute or improve scaling behavior?	Tay et al. (2022) ; Wang et al. (2024) ; Sengupta et al. (2025)
Systems	Can memory, communication, synchronization, or numerical precision reduce realized cost?	Narayanan et al. (2021) ; Dettmers et al. (2023)
Hardware/edge	Can LLMs be trained, adapted, or served under explicit memory, power, or compute constraints?	Ahmed et al. (2025) ; Li et al. (2024) ; Yi et al. (2023) ; Yu et al. (2024) ; Moitra et al. (2025) ; Wei et al. (2025)

Table 5: Five Dimensions of Resource-Efficient LLM Research

Explore similar work

A Theory of Training Profit-Optimal LLMs
May 14, 2026Sophie Hao, William MerrillLarge Language Model Training Scaling Laws
Boston University · Allen Institute for AI
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuScaling Laws Large Language Model Training
Arena Physica
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Alignment-Aware Selection Large Language Model Training
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria

May 14, 2026, cs.LGJ/K move · Enter open · S save

A Theory of Training Profit-Optimal LLMs

Sophie Hao, William Merrill

Boston University · Allen Institute for AI

Large Language Model Training Scaling Laws Token Budget Allocation Theory

Open paper AskPDF arXiv

Scaling LLMs requires tremendous computational resources, and recent advances in AI have gone hand in hand with massive amounts of capital expenditure. While it is established that scaling up LLMs reliably increases model quality (quantified in terms of loss or downstream evaluations), it is unclear how these quality improvements translate to potential revenue, and whether revenue increases would offset costs of larger-scale training and inference. In this work, we develop an economic model for characterizing the rational behavior of an LLM training firm by combining scaling laws with microeconomic theory. Under our model of firm behavior, LLM quality can be increased with more parameters and training tokens, leading to more potential adoption by consumers, who each have a quality threshold for using the LLM. On the other hand, additional parameters and training tokens both incur additional costs. We analyze the profit maximization problem for this model under compute-bound and data-bound regimes. In the compute-bound regime, optimal model size and token budget track hardware efficiency

E

(FLOPs/$) at a near-linear rate; total training cost then scales sub-quadratically in

E

. Data efficiency improvements incentivize larger models and training expenditure. When we are limited to

D

data, profit-optimal training expenditure scales as

D^2/E

, i.e, increase with data and decreases with hardware efficiency (as well as data efficiency). Finally, we analyze practical trends in training expenditure: current trends are consistent with our most permissive model variants in the compute-bound regime, but are not profit-optimal in the data-bound regime or assuming hardware advances will stall. Overall, our results provide a theory of profit-optimal LLM training, providing a foundation for engaging critically with industry statements and supporting long-term economic decision making.

Abstract

Explore similar work

A Theory of Training Profit-Optimal LLMs

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey