Iterative Refinement

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, down 17% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 38

All topics

Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Softmax Attention Transformer Interpretability
Department of Mathematics, University of California, Santa Barbara, CA
Alternating Target-Path Planning for Scalable Multi-Agent Coordination
May 8, 2026Yu Kumagai, Keisuke OkumuraMulti-Agent Coordination Multi-Robot Task Allocation
1Hitotsubashi University, Japan · 2National Institute of Advanced Industrial Science and Technology (AIST), Japan
Exact ReLU realization of tensor-product refinement iterates
May 5, 2026Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology
Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements
May 3, 2026Boldsaikhan Bolorkhuu, Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences
Refinement via Regeneration: Enlarging Modification Space Boosts Image Refinement in Unified Multimodal Models
Apr 28, 2026Jiayi Guo, Linqing Wang, Jiangshan Wang +6Unified Multimodal Models T2I Generation
1Tsinghua University · 2Tencent HY
FlowRefiner: Flow Matching-Based Iterative Refinement for 3D Turbulent Flow Simulation
Apr 18, 2026Yilong Dai, Yiming Sun, Yiheng Chen +3Flow Matching Neural PDE Solvers
University of Alabama · University of Pittsburgh
Iterative Diffusion-Refined Neural Attenuation Fields for Multi-Source Stationary CT Reconstruction: NAF Meets Diffusion Model
Nov 18, 2025Jiancheng Fang, Shaoyu Wang, Junlin Wang +3Diffusion Models for Image Restoration Sparse-View CT Reconstruction
School of Information Engineering, Nanchang University, Nanchang 330031, China
MSRNet: A Multi-Scale Recursive Network for Camouflaged Object Detection
Nov 16, 2025Leena Alghamdi, Muhammad Usman, Hafeez Anwar +2Multi-Scale Feature Fusion Iterative Refinement
King Fahd University of Petroleum and Minerals, KSA · Ontario Tech University, Canada · National University of Computer and Emerging Sciences, Pakistan +2

May 8, 2026, cs.LGJ/K move · Enter open · S save

Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

Mingsong Yan, Dongyang Li, Charles Kulick, Sui Tang

Department of Mathematics, University of California, Santa Barbara, CA

Softmax Attention Transformer Interpretability Kernel Ridge Regression Kernel Regression Self-Attention In-Context Learning+2

Open paper AskPDF arXiv

Mechanistic accounts of in-context learning (ICL) have identified iterative algorithms for linear regression and related linear prediction tasks, often using linear or ReLU attention variants. For nonlinear ICL, prior work has related softmax and kernelized attention to functional-gradient-type dynamics, but it remains unclear whether a standard transformer with softmax attention can implement a convergent solver with an end-to-end prediction-error guarantee. In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass by implementing preconditioned Richardson iteration on the associated kernel linear system. Under bounded-data assumptions, we construct a single-head transformer with

O(\log(1/ε))

blocks and MLP width

O(\sqrt{N/ε})

that achieves

ε

-accurate prediction for prompts of length

N

. Our construction reveals a functional decomposition within the transformer architecture: softmax attention produces a row-normalized Gaussian-kernel operator needed for cross-token interactions, while ReLU MLP layers act locally to approximate the intra-token scalar arithmetic required by the update. Empirically, we train GPT-2-style transformers on Gaussian-process regression tasks to further test the preconditioned Richardson interpretation. Through linear probing, we compare the transformer's layer-wise predictions with the step-wise outputs of classical KRR solvers and find that its error profiles align most consistently with preconditioned Richardson iteration. Ablation studies further support this interpretation. Together, our theory and experiments identify preconditioned Richardson iteration as a concrete mechanism that softmax-attention transformers can realize for nonlinear in-context Gaussian-kernel regression.

Momentum

Related topics

Latest papers 38