Backward Passes

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 32

All topics

Backward-State Policy Is Part of the Learning Algorithm
Sep 30, 2026Shuxiao Xie, Shuyang Xie, Dezhi Ran +2Backward Passes Gradient
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuBlock Sparse Flash Attention Gumbel-Softmax Relaxation
Revisiting Thinning Methods for Kernel Learning Problems
Sep 7, 2026Blanca Cano-Camarero, Yago R. Aguado-Carrillo-de-Albornoz, Ángela Fernández-Pascual +1Kernel Method Reproducing Kernel Hilbert Spaces
Departamento de Ingeniería Informática, Universidad Autónoma de Madrid, Madrid, Spain
When Quantization Breaks Memory: Recurrent-State Write-Back in Low-Precision Temporal Inference
Sep 7, 2026Ismail Erbas, Xavier Intes, Vikas PandeyRecurrent State Backward Passes
Department of Biomedical Engineering Center for Modeling, Simulation, and Imaging in Medicine Rensselaer Polytechnic Institute Troy, NY 12180
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
Aug 10, 2026Tong CheMuon Newton
NVIDIA Research
Chess on Ice: Curling Tactical Decision-Making via Backward Induction and Deep Reinforcement Learning
Aug 3, 2026Patrick Oberlin, Matteo Cederle, Aren Karapetyan +3Chess Tactics
Automatic Control Laboratory, ETH Zurich, Switzerland · Department of Information Engineering, University of Padova, Italy · Belimo Automation AG, Hinwil, Switzerland
CC-AOS: Cost- and Horizon-Conditioned Amortized Backward Induction for Finite-Horizon Optimal Stopping
Jul 24, 2026Tianwei YuOptimal Stopping Stochastic Optimization
World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation
Jul 20, 2026Xinchen Yao, Leixin Chang, Hua ChenSim-To-Real Reinforcement Learning Inverse Dynamics
Zhejiang University · LimX Dynamics
Backpropagation as a Nilpotent Linear System
Jul 13, 2026Ahmed BoughammouraBackpropagation Backward Passes
Higher Institute of Informatics and Mathematics of Monastir, University of Monastir, 5000 Monastir, Tunisia.
A Posteriori Error Analysis for Decoupled Neural Approximations of Fully Coupled FBSDEs with Control Mismatch
Jun 28, 2026Xichuan ZhangStochastic Differential Equations Neural Approximations
Intelligent Game and Decision Lab, Beijing 100091, China
Deep numerical schemes for systems of Ergodic BSDEs with applications to regime-switching forward utilities
Jun 23, 2026Guillaume Broux-Quemerais, Sarah Kaakai, Anis Matoussi +1Stochastic Differential Equations Classical Numerical Solver
LMM · 1Laboratoire Manceau de Mathématiques & FR CNRS No 2962, Institut du Risque et de l’Assurance, Le Mans Université · LAGA +1
Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
Jun 18, 2026Nico Harder, Daniel Becking, Karsten Mueller +1Large Language Model Compression Low-Rank Structure
Fraunhofer HHI, Berlin, Germany
Backward Coherence and Hidden-State Stability in Recurrent Neural Networks: A Quasi-Reverse-Martingale Theory
Jun 8, 2026Yuan-chin Ivan ChangRecurrent Neural Networks Backward Passes
Institute of Statistical Science, Academia Sinica · Academia Road, Section 2, Nankang, Taipei 115, Taiwan
Evaluating the Impact of Task Granularity on Catastrophic Forgetting in Continual Learning
Jun 6, 2026Emre Alyamac, Himanshu Janmeda, Shashwat Krishna +1Catastrophic Forgetting Continual Learning
College of Engineering · College of Natural Science
Turning Back Without Forgetting: Selective Backward Refinement for Parameter-Efficient Continual Learning
May 31, 2026Anushka Tiwari, Kaiyi JiReplay-Based Continual Learning Cross-Task Knowledge Transfer
Institute for Artificial Intelligence and Data Science, University at Buffalo, Buffalo, NY, USA · Department of Computer Science and Engineering, University at Buffalo, Buffalo, NY, USA
Time-Optimal Collision Avoidance Via a Greedy Polynomial Backward Sweep
May 31, 2026Zeno Pavanello, Frank De Veld, Roberto ArmellinCollision Avoidance Intent-Aligned Autonomous Spacecraft Guidance
Department of Aerospace Science and Technology (DAER), Politecnico di Milano, via Privata Giuseppe La Masa 34, Milano, 20156, Italy · Netherlands Aerospace Centre (NLR), Anthony Fokkerweg 2, Amsterdam, 1059 CM, The Netherlands · Te Pūnaha ¯Atea Auckland - Space Institute (TPA-SI), The University of Auckland, 20 Symonds Street, Auckland, 1010, New Zealand
Self-Improving Language Models with Bidirectional Evolutionary Search
May 27, 2026Guowei Xu, Zhenting Qi, Huangyuan Su +4Evolutionary Search Backward Passes
1Harvard University · 2MIT
Can Transformers Learn to Verify During Backtracking Search?
May 21, 2026Yin Jun Phua, Tony Ribeiro, Tuan Nguyen +1Transformer Architectures Backward Passes
Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, Japan · Centrale Nantes, CNRS, Laboratoire des Sciences du Num´erique de Nantes, LS2N, UMR 6004, F-44000 Nantes, France · National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan +2
Fast Adversarial Attacks with Gradient Prediction
May 14, 2026Kamil Ciosek, Aleksandr V. Petrov, Nicolò Felicioni +1Gradient-Based Attacks Black-Box Adversarial Attacks
Spotify
Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
May 11, 2026Yao Shu, Zilin ZhuZeroth-Order Zeroth-Order Optimization
HKUST-GZ
BCJR-QAT: A Differentiable Relaxation of Trellis-Coded Weight Quantization
May 11, 2026Venugopalan IyengarQuantization-Aware Training Backward Passes
DySurface: Consistent 4D Surface Reconstruction via Bridging Explicit Gaussians and Implicit Functions
May 11, 2026Minje Kim, Younghyun Noh, Jaesoon Kim +1Neural Radiance Fields Volume
KAIST · KT · Sungkyunkwan University
The Position Curse: LLMs Struggle to Locate the Last Few Items in a List
May 8, 2026Zhanqi Zhang, Hua-Dong Xiong, Robert C. Wilson +3Position Backward Passes
UC San Diego · Georgia Tech · New York University
Playing the network backward: A Game Theoretic Attribution Framework
May 7, 2026Jakob Paul Zimmermann, Jim Berend, Georg Loho +2Backward Passes Mechanistic Interpretability
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany. · Technische Universität Berlin, Berlin, Germany. · Freie Universität Berlin, Berlin, Germany. +2
NeuroViz: Real-time Interactive Visualization of Forward and Backward Passes in Neural Network Training
May 3, 2026Tanvi Sharma, Reza RawassizadehNeural Network Training Visualization
Boston University · Boston, MA, USA
Network Digital Untwinning: Towards Backward Optimization of Digital Twins
Apr 30, 2026Zifan Zhang, Dianwei Chen, Anjun Gao +5Digital Twins Rollback
North Carolina State University, USA · University of Maryland, USA · University of Louisville, USA +2
Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation
Apr 21, 2026Yaowei Zheng, Richong Zhang, Shenxi Wu +5Bellman Equation Generators
School of Computer Science and Engineering, Beihang University, China · Fudan University, China · Faculty of Humanities and Social Sciences, University of Nottingham, China +1
Towards Universal Convergence of Backward Error in Linear System Solvers
Apr 17, 2026Michał Dereziński, Yuji Nakatsukasa, Elizaveta RebrovaSolver Iterations Convergence
University of Michigan, Ann Arbor, MI · University of Oxford, Oxford, UK · Princeton University, Princeton, NJ
SkipGS: Post-Densification Backward Skipping for Efficient 3DGS Training
Mar 9, 2026Jingxing Li, Yongjae Lee, Deliang FanSkip Backward Passes
Arizona State University, Tempe, AZ 85281, USA
Small Reward Models via Backward Inference
Feb 14, 2026Yike Wang, Faeze Brahman, Shangbin Feng +3Llm-As-A-Judge Small Reasoning Model
University of Washington · Allen Institute for Artificial Intelligence
Recursive Entropic Variational Inference for Nonlinear State-Space Models
Nov 19, 2025Hany Abdulsamad, Ángel F. García-Fernández, Simo SärkkäVariational Inference State Space Models
Amsterdam Machine Learning Lab University of Amsterdam, Netherlands · Information Processing and Telecommunications Center Universidad Politécnica de Madrid, Spain · ELLIS Institute Finland Department of Electrical Engineering and Automation Aalto University, Finland
Synthetic Blips: Generalizing Synthetic Controls for Dynamic Treatment Effects
Date pendingAnish Agarwal, Sukjin Han, Dwaipayan Saha +2Heterogeneous Treatment Effects Treatment Allocation
Columbia University · University of Bristol · Stanford University +1

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Backward-State Policy Is Part of the Learning Algorithm

Shuxiao Xie, Shuyang Xie, Dezhi Ran, Wei Yang+1

Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China

Backward Passes Gradient Full-Precision Performance

Open paper AskPDF arXiv

Low-precision training rounds tensors that the backward pass reads again, often for several gradients; each use can read the forward's rounded value, the original, or a new random rounding. This backward-state policy looks like a memory and precision detail, settled by copy accuracy and final loss. We argue that it is part of the learning algorithm, and that neither check shows whether it is right. Copy accuracy does not decide the outcome: in three pairs of 390M runs with an emulated FP8 backward, training fails when attention's backward reuses the forward's rounded output and succeeds with a new rounding from the same distribution. Even the most accurate copy, the original itself, can be wrong by our reference: the gradient of the forward pass as it actually ran, with gradients passed through rounding unchanged. For example, a normalization output stored in low precision feeds two gradients: the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied. Final loss, the other check, does not rule out the error of reading the original for both: it persists in models trained with such a store, while planned loss comparisons stay within a margin fixed in advance. We therefore derive from this reference which value each use must read, or which substitute gives the same gradient on average with the forward held fixed, and check these per-use requirements on single operators, without training. In three tests using PyTorch and Transformer Engine, the requirements predicted beforehand whether reuse changes what the backward computes on average relative to an independent copy, and every prediction held. Backward-state policy is thus part of the learning algorithm: it should be specified and checked use by use, not settled by copy accuracy and final loss.

Momentum

Related topics

Latest papers 32