Large Models

Follow this topic Ask about this topic

Recent momentum

+31%

17 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

8 new papers

A weekly snapshot of new work published in Large Models.

Period ending 2026-09-14

2 new papers

A weekly snapshot of new work published in Large Models.

Period ending 2026-09-07

8 new papers

A weekly snapshot of new work published in Large Models.

192 papers

Latest in Large Models

Open your feed →

Accelerating Sharded Data Parallelism at Scale with Federated Learning
Sep 17, 2026Gianluca Mittone, Marco AldinucciFederated Learning Parallel
The Life of a Token: from Words to Bits on the Wire
Sep 17, 2026Davide Avesani, Pengwenlong Gu, Sotiris Skaperas +1Large Models Neural Architectures
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Sep 16, 2026Mobina Kashaniyan, Ali JannesariTest-Time Scaling Large Models
Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
Sep 16, 2026Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang +4Diffusion Language Models Parallel
MiST: Mid-Training LLMs for Cybersecurity
Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Security Analysis Large Models
OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Reproducibility Large Models
Breaking the 1.58-bit Barrier for Ternary LLMs
Sep 14, 2026Evangelos Georganas, Alexander Heinecke, Pradeep DubeyLarge Language Model Quantization Large Models
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Chain-Of-Thought Distillation Large Models
EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale
Sep 12, 2026Ege C. Kaya, Abolfazl HashemiLarge Models Evolutionary Search
Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
Sep 10, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLarge Models Qwen3
Length Generalization for Transformers via Compression
Sep 8, 2026Georg Zetzsche, Hongjian Jiang, Andy Yang +4Transformer Architectures Generalization Bounds
Parallelism Strategy Chaining for Fast Training Convergence
Sep 7, 2026Minchul Kang, Changyong Shin, Younghun Go +4Large Models Parallel
BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
Sep 2, 2026Bigyan Ghimire, Jon C. CalhounParallel Large Models
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Hyperparameter Optimization Large Models
Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training
Sep 1, 2026Guangqi Li, Yongxin LiLarge Models Cognitive Science
Instella-MoE Technical Report
Sep 1, 2026Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra +10Mixture-Of-Expert Language Models Large Models
Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
Aug 31, 2026Zhiqin Yang, Jingwen Fu, Yuhan Liu +16Step-Level Supervision Large Reasoning Models
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Aug 12, 2026Arda Uzunoglu, Benjamin Van Durme, Daniel KhashabiEfficient Long-Context Inference Large Models
LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
Aug 12, 2026Xiaojun Wu, Cehao Yang, Honghao Liu +5Large Models Graphics Processing Unit Memory
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-of-Experts Models Large Models
Interpreting Language Model Hidden States at Scale
Aug 10, 2026Jordan Pettyjohn, Mansi Sakarvadia, Nathaniel Hudson +3Large Models Model Activations
Fusion Training for Mathematical Generalization in Large Language Models
Aug 10, 2026Congfeng Cao, Pengyu Zhang, Jelke BloemLarge Models Trainability
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiMixture-of-Experts Models Expert Demonstrations
Quantization Degradation in Large Language Models: A Signal-Noise Perspective
Aug 8, 2026Chenxi Zhou, Pengfei Cao, Jinyu Ye +5Large Language Model Quantization Quantization
Interpretable Adaptive Sampling for LLM Test-Time Scaling
Aug 4, 2026Mobina Kashaniyan, Ali JannesariTest-Time Scaling Test Time
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-Of-Expert Language Models Large Models
Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLarge Models Transformer Architectures
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
Aug 2, 2026Zhichao Xu, Xueguang Ma, Shengyao Zhuang +5Reranking Large Models
Towards joint scaling laws with optimal batch size schedules
Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuBatch Size Large Models
MMOE: Modernizing Diffusion Transformers with Efficient Expert Design
Jul 27, 2026Yanhao Jia, Jiepeng Wang, Haibin Huang +3Mixture-Of-Expert Language Models Large Models
Developing and Validating the Spanish Version of the Large Language Models Dependency Scale (LLM-D12-SP)
Jul 24, 2026Tran Gia Bao, Mo El-Haj, Sameha Al-Shakhsi +3Psychometrics Large Language Model Reliability
Hierarchical Grading in Large Language Models
Jul 23, 2026T. ShaskaGrading Large Models
A Defense of the Quadratic Model
Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Large Models Condition Number
Test-Time Scaling via Error Localization
Jul 23, 2026Rajiv Shailesh Chitale, Rahul Madhavan, Taneesh Gupta +2Test-Time Scaling Inference-Time
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
Jul 22, 2026Dongfang Li, Xiaodong Luo, Ruoyu Sun +62Large Models Deepseek
Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
Jul 21, 2026Nischay Dhankhar, Dos Baha, Abulhair SaparovLarge Models Hypernetwork
xHC: Expanded Hyper-Connections
Jul 16, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10Transformer Residual Streams Residual Stream
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Large Models Parallel
Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
Jul 14, 2026Xinyu Tang, Qianggang Cao, Yurou Liu +13Reinforcement Learning With Verifiable Reward Reinforcement Learning Post-Training
WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jul 12, 2026Jianhao Ma, Yuxin ChenLearning Rates Cosine Annealing
A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design
Jul 10, 2026Linhui Xiao, Guiping Cao, Mingyue Guo +6Artificial Intelligence Infrastructure Large Models
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Large Models Large Language Model Decoder
Understanding Layer Patching in Model Size Interpolation
Jul 9, 2026Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak +3Large Models
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
Jul 7, 2026Noel ThomasLow-Rank Factorization Large Models
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
Jul 6, 2026Yoshiyuki OotaniGrokking Large Models
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Large Models Memorization
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Jul 2, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3Mixture-of-Experts Models Billion-Parameter Models
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Large Models Sparsity
QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling
Jul 1, 2026Michael Y. Li, Anthony Zhan, Kanishk Gandhi +2Monte Carlo Test-Time Scaling
Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
Jul 1, 2026Xiangchen Song, Zhenhao Chen, Lingjing Kong +4Large Language Model Memory Large Models
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Jun 29, 2026Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2Asynchronous Execution Parallel
HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models
Jun 29, 2026Songxin Zhang, Zejian Xie, Zhuoyang Song +4Parallel Sequence Modeling
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Scaling Laws Large Models
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Learning Rates Large Models
On Test-Time Scaling for Vision-Language Models
Jun 27, 2026Fawaz Sammani, Tzoulio Chamiti, Nikos DeligiannisTest-Time Scaling Large Vision Language Models
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Jun 26, 2026Ruixuan Huang, Yipei Wang, Wenyi Fang +7Large Models Bregman Divergences
Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing
Jun 24, 2026Yutong Yin, Mingyu Jin, Jin Pan +12Large Language Model Reasoning Test-Time Scaling
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Jun 23, 2026Yizhou Liu, Jeff GoreScaling Laws Large Models
Can Scale Save Us From Plasticity Loss in Large Language Models?
Jun 23, 2026J. Fernando Hernandez-Garcia, Tomás Figliolia, Beren MillidgeLarge Models Plasticity
Decentralised AI Training and Inference with BlockTrain
Jun 23, 2026Peter Toth, Dan OprisaAccelerator Large Models

Sep 17, 2026 · cs.DCJ/K move · Enter open · S save

Accelerating Sharded Data Parallelism at Scale with Federated Learning

Gianluca Mittone, Marco Aldinucci

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups' size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.

Federated Learning Parallel Large Models Communication Efficiency

Open paper Ask about it PDF