Iteration Head

Iteration heads represent a specialized attention mechanism within transformer models that facilitates chain-of-thought reasoning, a crucial aspect of improving large language models' performance on complex tasks. Current research focuses on understanding the emergence and function of these mechanisms, analyzing their transferability across tasks, and exploring their role in various iterative algorithms, including those used in optimization, regression, and reinforcement learning. This research contributes to a deeper understanding of how complex reasoning emerges in artificial intelligence and has implications for improving the efficiency and accuracy of numerous machine learning applications.

Papers

November 7, 2024

Asymptotic regularity of a generalised stochastic Halpern scheme with applications
Nicholas Pischke, Thomas Powell
Financial Application Stochastic Optimization Iteration Head Halpern Iteration

November 1, 2024

Re-thinking Richardson-Lucy without Iteration Cutoffs: Physically Motivated Bayesian Deconvolution
Zachary H. Hendrix, Peter T. Brown, Tim Flanagan, Douglas P. Shepherd, Ayush Saurabh, Steve Pressé
Image Deconvolution Iteration Head

October 30, 2024

Bregman implementation of Meyer's $G-$norm for cartoon + textures decomposition
Jerome Gilles, Stanley Osher
Gradient Norm Microbial Decomposition Bregman Information Texture Quality Iteration Head Simple Algorithm

October 14, 2024

FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
Jingfeng Yao, Wang Cheng, Wenyu Liu, Xinggang Wang
Supervised ImageNet Diffusion Transformer Signal to Noise Ratio Slow Convergence Iteration Head

October 7, 2024

Transition of $α$-mixing in Random Iterations with Applications in Queuing Theory
Attila Lovas
Financial Application Time Series Global Mixing Late Time Transition Iteration Head Exogenous Variable Central Limit Theorem Random Model

June 4, 2024

Iteration Head: A Mechanistic Study of Chain-of-Thought
Vivien Cabannes, Charles Arnal, Wassim Bouaziz, Alice Yang, Francois Charton, Julia Kempe
Attention Mechanism Chain of Thought Iterative Reasoning Mechanistic Study Iteration Head

April 27, 2024

Uncertainty quantification for iterative algorithms in linear models with application to early stopping
Pierre C. Bellec, Kai Tan
Application Proficiency High Dimensional Uncertainty Quantification Linear Model Iterative Algorithm Iteration Head Novel Estimator

March 9, 2024

IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image- and Video-Quality Metrics
Ekaterina Shumitskaya, Anastasia Antsiferova, Dmitriy Vatolin
Adversarial Attack Reference Image Video Benchmark Video Quality Metric Iteration Head Replay Attack

February 18, 2024

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations
Yao Shu, Jiongfeng Fang, Ying Tiffany He, Fei Richard Yu
Optimization Purpose Stochastic Gradient Descent Gradient Estimation First Order Optimization Iteration Head Gradient Prediction

May 24, 2023

Test like you Train in Implicit Deep Learning
Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré, Thomas Moreau
Deep Learning Overparametrization Bound Implicit Neural Network Deep Equilibrium Iteration Head

May 17, 2023

A proof of imitation of Wasserstein inverse reinforcement learning for multi-objective optimization
Akira Kitaoka, Riki Eto
Multi Objective Optimization Feature Imitation Partial Proof Optimal Solution Iteration Head Reward Value

March 22, 2023

Three iterations of $(d-1)$-WL test distinguish non isometric clouds of $d$-dimensional points
Valentino Delle Rose, Alexander Kozachinskiy, Cristóbal Rojas, Mircea Petrache, Pablo Barceló
Iterative Algorithm Iteration Head Isometric Mapping

February 24, 2023

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning
Edwige Cyffers, Aurélien Bellet, Debabrota Basu
Industrial Disturbing Noise Differential Privacy SGD Private Gradient Private Optimization Centralized Control Iteration Head Fixed Point Iteration

September 13, 2022

Warm Start Active Learning with Proxy Labels \& Selection via Semi-Supervised Fine-Tuning
Vishwesh Nath, Dong Yang, Holger R. Roth, Daguang Xu
Semi Supervised Medical Image Datasets Supervised Fine Tuning View Selection Medical Segmentation Decathlon Iteration Head Start Active Learning Proxy Label

August 19, 2022

Federated Learning of Neural ODE Models with Different Iteration Counts
Yuto Hoshino, Hiroki Kawakami, Hiroki Matsutani
Neural ODE Machine Learning Approach ResNet Based Communication Cost Client Heterogeneity Iteration Head

May 19, 2022

How catastrophic can catastrophic forgetting be in linear regression?
Itay Evron, Edward Moroshko, Rachel Ward, Nati Srebro, Daniel Soudry
Continual LEArning Catastrophic Forgetting Anti Forgetting Linear Regression Iteration Head Kaczmarz Algorithm

February 4, 2022

Polynomial convergence of iterations of certain random operators in Hilbert space
Soumyadip Ghosh, Yingdong Lu, Tomasz J. Nowicki
Stochastic Gradient Descent Random Sequence Hilbert Space Iteration Head Convergence Property Stochastic Operator

December 8, 2021

Convergence Results For Q-Learning With Experience Replay
Liran Szlak, Ohad Shamir
Q Learning Specific Heuristic Experience Replay Iteration Head Tabular Q Learning

Iteration Head

Papers

Asymptotic regularity of a generalised stochastic Halpern scheme with applications

Re-thinking Richardson-Lucy without Iteration Cutoffs: Physically Motivated Bayesian Deconvolution

Bregman implementation of Meyer's $G-$norm for cartoon + textures decomposition

FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification

Transition of $α$-mixing in Random Iterations with Applications in Queuing Theory

Iteration Head: A Mechanistic Study of Chain-of-Thought

Uncertainty quantification for iterative algorithms in linear models with application to early stopping

IOI: Invisible One-Iteration Adversarial Attack on No-Reference Image- and Video-Quality Metrics

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations

Test like you Train in Implicit Deep Learning

A proof of imitation of Wasserstein inverse reinforcement learning for multi-objective optimization

Three iterations of $(d-1)$-WL test distinguish non isometric clouds of $d$-dimensional points

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Warm Start Active Learning with Proxy Labels \& Selection via Semi-Supervised Fine-Tuning

Federated Learning of Neural ODE Models with Different Iteration Counts

How catastrophic can catastrophic forgetting be in linear regression?

Polynomial convergence of iterations of certain random operators in Hilbert space

Convergence Results For Q-Learning With Experience Replay