Sequential Scaling

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 74

All topics

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
Oct 1, 2026Haotian Gu, Yizhou Xu, Lenka ZdeborováIn-Context Learning Linear Attention
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-Of-Experts Large Language Models Mixture-Of-Experts
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Does Scaling Reinforcement Learning Really Require More Training?
Oct 1, 2026Bangji Yang, Jiajun Fan, Hongba Ma +2Offline Reinforcement Learning Sequential Scaling
University of Illinois at Urbana-Champaign · Tsinghua University
Overcoming Kernel Redundancy for Scaling Logic Gate Networks
Oct 1, 2026Sejin Park, Hongjae Lee, Changwoo Han +1Differentiable Logic Gate Networks Sequential Scaling
Department of Electrical Engineering Korea University
Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Sep 30, 2026Minki Kang, Ryo Hachiuma, Shaokun Zhang +8Test-Time Scaling Agent Harness
NVIDIA · KAIST
TRACE: Trajectory Selection for Parallel Scaling of Search Agents
Sep 30, 2026Qisheng Zhou, Zhen Xiong, Qiaoyu TanSearch Agents Traces
New York University Shanghai · New York University
Towards Better Exploration in Sequential Test-Time Scaling
Sep 30, 2026Joseph Rance, Fabio Pizzati, Juil Sock +4Test-Time Scaling Sequential Scaling
University of Oxford · MBZUAI · BBC R&D
SteerQuant: Steering Quantization Error with Action-Guided Scaling in World-Action Models
Sep 30, 2026Yunhan Wang, Haodong Wang, Zhiming Liu +7Steering Sequential Scaling
Tongji · HKUST · HIT +1
Mitigating the Length-Scaling Tax with Online Distillation
Sep 30, 2026Xu Wan, Wenyue Xu, Shengjie Zhao +1Sequential Scaling Symbolic Distillation
ByteDance Seed · Tongji University · Peking University
Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs
Sep 29, 2026Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi ShekharCharacter Error Rate Disparities
School of Computer Science and Electronic Engineering, University of Essex, UK
Port-Hamiltonian Latent Deliberation: Mitigating the Deliberation Drift Cliff in Test-Time Compute Scaling
Sep 29, 2026Zeyu JiaTest-Time Scaling Efficient Latent Reasoning
School of Biomedical Engineering and Technology, Tianjin Medical University, Medical School, Tianjin University
Longer Records, Broader Invariance: The Hidden Scaling Problem in Longitudinal Contrastive Learning
Sep 29, 2026Rameen Mahmood, Xuhai "Orson" Xu, Zachary Beattie +2Longitudinal Data Contrastive Learning
New York University · Columbia University · Oregon Health & Science University
From Search to Research: Exploring Search Scaling in Autonomous Quantitative Factor Mining
Sep 28, 2026Kangcheng Deng, Hui Cai, Jiacheng Lu +6Research Automation Sequential Scaling
StepFun · Shanghai Jiao Tong University · University of California, Los Angeles +1
DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation
Sep 27, 2026Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee +5Efficient Long-Context Inference LLM Reasoning Strategies
National University of Singapore · Adobe Research
KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling
Sep 23, 2026Zhiheng Hu, Yixun Wei, Jian Zhou +8LLM Inference Optimization Sequential Scaling
StepFun
NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation
Sep 23, 2026Chuanlin Lan, Yanwei Zheng, Yuxi Jing +6Object Goal Navigation Navigation
School of Computer Science and Technology, Shandong University, Qingdao, China
Taking a Second Look: Correcting Sea Ice Forecasts with Sparse Observations
Sep 21, 2026Tianshuo Zhang, Xianglei Xing, Aowen Yang +4Sea-Surface Temperature Data Forecast
College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, China
Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation
Sep 14, 2026Jiajun Peng, Fengrui Liu, Xinyu Liu +1Neural Audio Procedural
School of Data Science, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China
Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling
Sep 14, 2026Xinchen Du, Elizaveta Rebrova, Micha\l Dereziński +1Newton Covariance
School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Operations Research and Financial Engineering, Princeton University · Computer Science and Engineering, University of Michigan
Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
Sep 7, 2026Guangneng Chen, David Garcia Selfa, Pablo Quesada BarriusoKolmogorov-Arnold Networks High-Performance Computing
University of Santiago de Compostela, 15782 Santiago de Compostela, Spain · Galicia Supercomputing Center (CESGA), 15705 Santiago de Compostela, Spain
Correlated initialization of deep residual networks
Sep 3, 2026Felix Benning, Ivan Nourdin, Giovanni PeccatiResidual Networks Infinities
From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs
Sep 1, 2026Jie Chen, Xiangqian Yu, Yanchao Lian +9Recommendation Sequential Scaling
ByteDance
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Aug 31, 2026Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6Gradient Quantization LLM Inference Optimization
Cornell University New York, NY, USA · Intel Corporation Santa Clara, CA, USA · Intel Corporation Hillsboro, OR, USA
RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Search
Aug 31, 2026Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato +2Residual Vector Quantization Approximate Nearest-Neighbor Search
Google Zurich, Switzerland · Technical University of Munich Munich, Germany · University of Copenhagen Copenhagen, Denmark +1
Ladders in Chaos: When, How, (and Perhaps Why) Does Test-Time Scaling Improve LLM Machine Translation
Aug 28, 2026Di Wu, Sergey Troshin, Christof Monz +2Test-Time Scaling Machine Translation
University of Amsterdam · Vrije Universiteit Amsterdam
XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling
Aug 12, 2026Jinxiu Liu, Xuanming Liu, Kangfu Mei +2Generative Flow Networks Diffusion Models
1CUHK · 2Westlake University · 3Johns Hopkins University
Online Learning of Scale Parameters in Score-Driven Filters
Aug 10, 2026Fabrizio Lillo, Giulia Livieri, Gianluca PalmariMirror Descent Sequential Scaling
Scuola Normale Superiore, Piazza dei Cavalieri 7, Pisa, Italy · Department of Statistics, London School of Economics and Political Science, Columbia House, 69 Aldwych, London, United Kingdom
Spatiotemporal Context-dependent Personalized Movement Compensation in Delayed Telemanipulation
Aug 8, 2026Sai Jiang, Zonghe ChuaTeleoperation Sequential Scaling
Department of ECSE, Case Western Reserve University, Cleveland, OH 44106 USA
ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
Aug 4, 2026Yang Yang, Qinyu Zhao, Mouxiang Chen +5Multimodal Large Language Models Vision Transformer
1The Australian National University · 2Shanghai AI Laboratory · 3Zhejiang University +2
How fine a change can moments see? A scale law for detecting distribution shift, with a kernel calibration rule
Aug 2, 2026Adel KalechePersistent Homology Two-Sample Testing
Independent researcher, France
KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models
Jul 30, 2026Sparsh Roy, Samuel Girmachew, Nishita ChavanFairness Audits Risk Stratification
Massachusetts Institute of Technology · Hopewell Valley Central High School · East Brunswick High School
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
Jul 27, 2026Jianlin Yu, Jing Lin, Linghui Kong +13Text-To-Video Diffusion Models Attention Layers
Huawei Technologies Co., Ltd.
Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
Jul 14, 2026Xinyu Tang, Qianggang Cao, Yurou Liu +13Reinforcement Learning With Verifiable Reward Test-Time Scaling
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Ant Group · 3Tsinghua University +1
An End-to-End Hybrid Quantum--Classical Sampling Workflow for Discrete Markov Random Fields: A Reproducible Case Study
Jul 10, 2026Arul Rhik MazumderMarkov Chain Monte Carlo Hybrid Quantum-Classical Pipeline
School of Computer Science
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Batch Sequential Scaling
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London
Will Scaling Improve Social Simulation with LLMs?
Jul 2, 2026Caleb Ziems, William Held, Su Doga Karaca +3Scaling Improve Social Simulation Sequential Scaling
Stanford University · Open Athena
Optimal scaling of MCMC algorithms: the Hamiltonian approach
Jul 1, 2026P. Dobson, J. M. Sanz-Serna, K. C. ZygalakisMarkov Chain Monte Carlo Sequential Scaling
Maxwell Institute for Mathematical Sciences and Mathematics Department Heriot-Watt University, Edinburgh, EH14 4AS, UK · Departamento de Matem´aticas, Universidad Carlos III de Madrid Avenida Universidad 30, 28911 Legan´es, Madrid · Maxwell Institute for Mathematical Sciences and School of Mathematics University of Edinburgh, Peter Guthrie Tait Rd, EH9 3FD, Edinburgh
ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies
Jun 30, 2026Andrew Zou Li, Gokul Swamy, Yonatan Bisk +1Generative Control Policies Test-Time Scaling
Carnegie Mellon University
MixTTA: Low-Rank Cross-Channel Mixing for Reliable Test-Time Adaptation
Jun 26, 2026Mansoo Jung, Youngwook Kim, Jungwoo LeeStable Test-Time Adaptation Test-Time Adaptation
Seoul National University, Seoul, Republic of Korea · Kookmin University, Seoul, Republic of Korea · HodooAI Labs, Seoul, Republic of Korea
Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis
Jun 24, 2026Lianbo Liu, Shiao Zhu, Kai Washizaki +10Flow-Matching Text-To-Speech Speaker Similarity
SB Intuitions
Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score
Jun 23, 2026Giorgio Corani, Stefano Damato, Dario Azzimonti +1Model Selection Probabilistic Forecasting
SUPSI, Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA), Lugano, Switzerland
Self-Adaptive Scale Handling for Forecasting Time Series with Scale Heterogeneity
Jun 18, 2026Xu Zhang, Zhengang Huang, Yunzhi Wu +6Time Series Forecasting Time Series
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Ant Group, Shanghai, China
Recursive Scaling in Masked Diffusion Models
Jun 16, 2026Alba Carballo-Castro, Julianna Piskorz, Paulius Rauba +2Masked Diffusion Language Models Diffusion Models
LTS4, EPFL, Lausanne, Switzerland · University of Cambridge, Cambridge, UK.
Training and Evaluating Diffusion Policies with Long Context Lengths
Jun 15, 2026Abhinav Agarwal, Adam Wei, Taylan Kargin +6Imitation Learning Diffusion Policies
Massachusetts Institute of Technology
The Hidden Power of Scaling Factor in LoRA Optimization
Jun 11, 2026Zicheng Zhang, Haoran Li, Jiaxing Wang +10Batch Multi-Lora
JD · School of Mathematical Sciences, UCAS · School of Mathematical Sciences, NKU +1
A Riemannian Approach to Low-Rank Optimal Transport
Jun 10, 2026Pratik Jawanpuria, Bamdev MishraLow-Rank Structure Riemannian Optimization
Centre for Machine Intelligence and Data Science, IIT Bombay · Microsoft India
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
Jun 10, 2026Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8Fast Inference Efficient Inference
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1
Exploring the Scale and Diversity of Speech Anti-spoofing Datasets: Experiments and Analysis
Jun 6, 2026Zhuolin Yi, Jun Xue, Yanzhen Ren +5Cross-Dataset Benchmark Sequential Scaling
School of Cyber Science and Engineering, Wuhan University, China
Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training
Jun 4, 2026Yucheng ChenForward Propagation Singular Learning Theory
Amplimit Toronto, ON, Canada
HORIZON: Recoverability-Governed Curriculum for Physical-Domain Scaling
Jun 3, 2026Chenhao Bai, Liqin Lu, Kaijun Wang +5Robot Policies Curriculum
1Zhejiang University, State Key Lab of CAD & CG · 2Zhejiang University of Technology
Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers
Jun 3, 2026Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh +3Automatic Speech Recognition Transformer Architectures
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Jun 2, 2026Zekun Qi, Xuchuan Chen, Dairu Liu +10Generative Pretrained Transformers Sequential Scaling
Tsinghua University · Galbot Inc. · Shanghai Jiao Tong University +2
Provable Data Scaling Law for Meta Learning via Complexity Minimization
Jun 1, 2026Kazuto Fukuchi, Ryuichiro Hataya, Kota MatsuiMeta-Learning Pretraining
University of Tsukuba, Japan · RIKEN AIP, Japan · SB Intuitions Corp., Japan +3
Conditional Collapse in Sign Language Production: A Diagnostic and a Scaling Argument
Jun 1, 2026Rui Hong, Jana KošeckáSign Language Translation Speech Language Models
George Mason University
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Large Language Model Training Sparsity
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Batch Two-Layer Neural Networks
Dartmouth College · Stanford University · Virginia Tech
Snippet-Driven Supply Chain Discovery with LLMs: Scaling Visibility in China
May 27, 2026Hiroto Fukada, Takayuki MizunoSupply Chain Optimization Knowledge Graphs
Graduate Institute for Advanced Studies (SOKENDAI) · National Institute of Informatics · Tokyo, Japan
FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing
May 26, 2026Jingbin Qian, Congwen Yi, Min Xia +3Semiconductor Manufacturing Multi-Hop Reasoning
FutureFab.AI
RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation
May 21, 2026Minseok Jung, Abhas Ricky, Muhammad Rameez ChatniInference-Time Text2Cypher
Cloudera 6220 America Center Dr. San Jose, CA 95002
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeAutoregressive Transformers Transformer Architectures
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · DEKE Lab, Renmin University of China, Beijing, China

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

Haotian Gu, Yizhou Xu, Lenka Zdeborová

Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

In-Context Learning Linear Attention Memorization Sequential Scaling

Open paper AskPDF arXiv

In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.

Momentum

Related topics

Latest papers 74