Large Models

Follow this topic Ask about this topic

Recent momentum

+31%

17 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

8 new papers

A weekly snapshot of new work published in Large Models.

Period ending 2026-09-14

2 new papers

A weekly snapshot of new work published in Large Models.

Period ending 2026-09-07

8 new papers

A weekly snapshot of new work published in Large Models.

193 papers

Latest in Large Models

Browse all topics →

Decentralised AI Training and Inference with BlockTrain
Jun 23, 2026Peter Toth, Dan OprisaAccelerator Large Models
On the Smallness of the Large Language Models Scaling Exponents
Jun 23, 2026Sauro Succi, Peter V. Coveney, Alex HansenScaling Laws Large Models
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Jun 22, 2026Mansour Zoubeirou a MayakiTransformer Architectures Large Models
FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models
Jun 22, 2026Zhida Jiang, Zhaolong Xing, Yang Pei +14Large Vision Language Models Large Models
GRAIN: Group Aggregation via Min-Norm Objective
Jun 22, 2026Nghia Bui, Jiarui Yao, Lijing WangMini-Batch Stochastic Gradient Descent Large Models
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Distributed Optimization Distributional Learning
On the Expressive Power of Weight Quantization in Large Language Models
Jun 20, 2026Shao-Qun ZhangLarge Language Model Quantization Quantization
ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era
Jun 20, 2026Masao Someki, Alexander Polok, Carlos Carvalho +14Speech Large Language Models Large Models
An Empirical Study of OpenPangu Quantization on Ascend NPUs
Jun 19, 2026Tong Shi, Jiacheng Wang, Hui Xie +4Weight-Only Quantization Neural Processing Units
Token-Operations-Oriented Inference Optimization Techniques for Large Models
Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM Inference Efficiency Large Models
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Jun 17, 2026Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji +5Mixture-of-Experts Models Large Models
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
Jun 11, 2026Joe DwyerLarge Models Small Language Models
MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling
Jun 11, 2026Wenbo Chen, Puheng Li, Mengyang Liu +2Reasoning Traces Large Models
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Parallel Efficient Long-Context Inference
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-Of-Expert Language Models Large Models
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Jun 5, 2026Ernests Lavrinovics, Marco Letizia, Roy Janco +3Large Language Model Compression Context Compression
ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning
Jun 5, 2026Vladislav Smirnov, Chieu Nguyen, Sergey Senichev +14Test-Time Scaling Test Time
Generative Criticality in Large Language Model Temperature Scaling
Jun 4, 2026Huajian Ruan, Jinyang Li, Xingyu Guo +1Large Models Autoregressive Generation
MusaCoder: Native GPU Kernel Generation with Full-Stack Training on Moore Threads GPU
Jun 3, 2026Kun Cheng, Songshuo Lu, Sicong Liao +7Graphics Processing Unit Kernels Large Models
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Jun 3, 2026Jiangyang He, Shaolin Zhu, Deyi XiongMixture-Of-Expert Language Models Mixture-of-Experts Models
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarMuon Optimizer Muon
Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikParallel Large Models
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuLarge Models Learning Rates
An Asymptotic Theory of Chain-of-Thought in In-Context Learning
Jun 2, 2026Kaito Takanami, Cengiz PehlevanIn-Context Learning Chain-of-Thought Reasoning
EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
Jun 2, 2026Guhong Chen, Yingcheng Shi, Yongbin Li +6Agentic Reinforcement-Learning Framework Self-Evolving Agents
Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
Jun 2, 2026Runpeng Dai, Tong Zheng, Rui Liu +2Test-Time Scaling Adaptive Sampling
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Large Models Data Scarcity
Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
May 29, 2026Ibne Farabi Shihab, Fariya Afrin, Anuj SharmaReinforcement Fine-Tuning Prompt Tuning
Before Parc Fermé: RL-Time Pruning for Efficient Embodied LLMs in Autonomous Driving
May 29, 2026Luca Benfenati, Ali Azimi, Matteo Risso +3Large Models Autonomous Driving
Towards Efficient LLMs Annealing with Principled Sample Selection
May 29, 2026Yuanjian Xu, Jianing Hao, Wanbo Zhang +2Cosine Annealing Large Models
Bayesian Inference with Shaped Deep Non-linear MLPs
May 29, 2026Boris Hanin, Tianze JiangBayesian Neural Networks Bayesian
Demystifying Data Organization for Enhanced LLM Training
May 28, 2026Yalun Dai, Yangyu Huang, Tongshen Yang +8Large Models Data-Curation
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Transformer Layers Large Models
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuParallel Mini-Batch Stochastic Gradient Descent
On the Optimizer Dependence of Neural Scaling Laws
May 28, 2026Vansh Ramani, Shourya Vir JainScaling Laws Large Models
CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models
May 27, 2026Venkat Akhil LakkapragadaLarge Models Inference Cost
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
May 27, 2026Quanen Sun, Changxin Tian, Ke Shi +6Downstream Fine-Tuning Out-Of-Distribution
Law of Neural Interaction: Depth-Width Shape, Interaction Efficiency, and Generalization
May 27, 2026Wenjie Sun, Jinning Yang, Shuai Zhang +1Large Models Mmlu-Pro
Heterogeneous Parallelism for Multimodal Large Language Model Training
May 26, 2026Yashaswi Karnati, Kamran Jafari, Akash Mehra +10Large Models Multi-Modal Large Language Models
LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation
May 26, 2026Gabriele Cesa, Thomas Hehn, Aleix Torres-Camps +4Parallel LLM Inference Efficiency
Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling
May 26, 2026Xinglin Wang, Hao Lin, Shaoxiong Feng +9Test-Time Scaling Parallel
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
May 26, 2026Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang +4Efficient Latent Reasoning Large Models
AI-Model Network: Concept, Current State and Future
May 25, 2026Li Zhetao, Zeng Xiyu, Wang Jianhui +6Ai-Based Framework Neural Architectures
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Large Models Large Language
Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training
May 25, 2026Yingying Cheng, Jinquan Shi, Li Zhou +4Large Models Quadratic Attention
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Large Models Multi-Modal Large Language Models
A general tensor-structured compression scheme for efficient large language models
May 25, 2026Ying Lu, Peng-Fei Zhou, Qi-Xuan Fang +3Large Language Model Compression Large Models
Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling
May 24, 2026Dao Tran, Duc Anh Le, Ngoc Luu +3Test-Time Scaling Reasoning Benchmark
ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training
May 23, 2026Minghao Li, Alicia Golden, Samuel Hsia +14Large Models Cross-Architecture Generalization
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
May 22, 2026Xu Ouyang, Deyi Liu, Yuhang Cai +5Large Models Scaling Laws
Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning
May 20, 2026Benhao Huang, Zhengyang Geng, Zico KolterTest-Time Scaling Recursive Reasoner
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
May 20, 2026Dayal Singh Kalra, Maissam BarkeshliHyperparameter Optimization Large Models
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
May 19, 2026Daniel Barley, Jonathan Leis, Benjamin Klenk +1Parallel Large Language Model Pipelines
MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenMuon Optimizer Muon
Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
May 19, 2026Brandon Cui, Ximing Lu, Jaehun Jung +7Large Models Large Language Model Pipelines
Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training
May 18, 2026Yanru Wu, Jianning Wang, Chongxin Gan +1Large Audio Language Models Audio Representation
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
May 18, 2026Aaron DefazioLarge Models Learning Rates
Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference
May 16, 2026Mengtian Yang, Zhekun Zhang, Mingheng Wu +3Large Models Large Language Model Performance
New Wide-Net-Casting Jailbreak Attacks Risk Large Models
May 16, 2026Qiuchi Xiang, Haoxuan Qu, Hossein Rahmani +1Adversarial Jailbreaks Jailbreak Attacks
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM
May 15, 2026Shaoke Xi, ChonLam Lao, Boyi Jia +11Large Models Graphics Processing Unit Memory

Jun 23, 2026 · cs.AIJ/K move · Enter open · S save

Decentralised AI Training and Inference with BlockTrain

Peter Toth, Dan Oprisa

Spheroid Labs∗

Frontier AI training is increasingly shaped by access to dense, centrally controlled accelerator clusters. This creates a structural advantage for hyperscalers and large centralized laboratories, and makes open or independent AI efforts depend on scarce capital, privileged infrastructure, and data-center geography. We present Spheroid BlockTrain, a decentralized training protocol in which a model is partitioned into independently trainable blocks, each optimized on a local objective derived from the same global target and composed at inference into one model. On byte-level WikiText, BlockTrain reaches cross entropy 1.359 (perplexity 3.89), within about 0.04 CE of a same-setup end-to-end Transformer reference, while each active worker trains only one block and avoids full-model optimizer state. A shared six-worker block training run reaches CE 1.385 by averaging same-block updates into one assembled model. HTTP/TCP transport experiments move real serialized checkpoints and updates, including a public-IP three-host run that improves CE from 5.580 to 1.811 while moving 15.22 GB. For inference, the current BlockTrain path uses one block-stack traversal per full output and serves over direct TCP across three public-network GPU hosts up to a 75.80B-parameter logical fp16 shape, outperforming a matched plain-autoregressive TCP pipeline baseline because it emits a full sequence per WAN pipeline traversal rather than one token per traversal.

Accelerator Large Models Parallel Sphere

Open paper Ask about it PDF