Stationary Point

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 17

Sep 20, 2026math.OC

The Exponential Price of Determinism in Nonsmooth Nonconvex Optimization

We study the complexity of finding

(δ,ε)

-Goldstein stationary points of nonsmooth nonconvex Lipschitz functions. By now, it is known that randomized first-order algorithms can solve this task with a dimension-free oracle complexity [Zhang et al., 2020], whereas deterministic algorithms cannot, as their complexity must scale at least linearly with the dimension

d

[Jordan et al., 2023, Tian and So, 2024]. This leaves open whether deterministic algorithms can nevertheless solve the problem with oracle complexity polynomial in

d

. We answer this question negatively by proving a lower bound of order

(1/ε)^{Ω(d)}

for deterministic algorithm, closing the exponential gap between the previously known lower and upper bounds and resolving an open problem posed by Jordan et al. [2023]. We further discuss several extensions and implications of this result to weaker stationarity notions, finding a descent direction and deterministic smoothing. Overall, our results establish an exponential computational advantage in nonsmooth nonconvex optimization offered by randomization.

Guy Kornowski

Stochastic Convex Optimization First Order Oracle Complexity Lipschitz Continuity Stationary Point Exponents

Aug 12, 2026math.OC

A Local-Linearly Convergent Algorithm for Nonconvex Equality-Constrained Optimization

For solving nonconvex equality-constrained optimization problems, a recent Gradient-Eigenstep Algorithm by Goyens et al.~is an iteration-efficient approach, based on minimizing Fletcher's augmented Lagrangian function, for finding an approximate second-order stationary point from an arbitrary starting point. In this paper, the analysis of this algorithm is extended, offering a two-fold contribution. First, it is shown that a local-linear rate of convergence can be obtained by this method if it is initiated sufficiently close to a strong second-order stationary point and employs a sufficiently small step-size parameter and sufficiently large penalty parameter. In this case, the algorithm reduces to a gradient descent algorithm applied to minimize Fletcher's augmented Lagrangian. Second, as a particularly useful application of the first result, it is shown that the Gradient-Eigenstep algorithm can be used as an iteration-efficient subproblem solver in the context of a progressive sampling strategy for solving equality-constrained optimization problems when the objective and constraint functions are defined by large sample averages, ultimately offering an algorithm with an improved worst-case sample complexity when compared to an approach that solves a full-sample problem directly.

Frank E. Curtis, Lingjun Guo, Daniel P. Robinson

Department of Industrial and Systems Engineering, Lehigh University

Nonconvex Convergence Stationary Point

Aug 3, 2026stat.ML

Detecting Nonproperness of Likelihood Equations

Given an algebraic statistical model, a challenging problem is classifying the data according to the number of positive critical points of the likelihood function. The positive critical points are the positive solutions to an algebraic system, say likelihood equations. So, identifying the number of positive critical points is a real root classification problem for the likelihood equations. A discriminant variety of a likelihood-equation system geometrically describes the data for which the number of real solutions becomes unusual. As an essential component of the discriminant variety, the nonproperness set collects the data such that the likelihood-equation system has a solution at infinity. So, the number of real solutions varies when the data passes the nonproperness set, and identifying the nonproperness set plays a crucial role in the real root classification. In this work, we develop a novel method for computing nonproperness sets of likelihood-equation systems. We prove the correctness of this method. We show experimentally that it is far more efficient than the known methods in the literature.

Xiaoxian Tang, Bican Xia, Tianqi Zhao

School of Mathematical Sciences, Beihang University, Xueyuan Road, 100191, Beijing, China · School of Mathematical Sciences, Peking University, Yiheyuan Road, 100871, Beijing, China · Zhongguancun Laboratory, Zhongguancun East Road, 100084, Beijing, China

Stationary Point

Jul 26, 2026cs.IR

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

Large language models (LLMs) used as listwise rerankers in recommendation systems suffer from position bias when serializing candidate sets into prompts. We show this order sensitivity creates an exploitable attack surface: an attacker can promote a label-0 target into the top-

k

solely by reordering candidates, without changing item content, labels, or model parameters. We introduce

\mathrm{promo}@k

to quantify this vulnerability, measuring the fraction of label-0 targets that can be elevated into top-

k

rankings via permutation. Evaluating across three domains (MovieLens, Amazon Books, and Amazon Fashion),

\mathrm{promo}@5

reaches up to 0.57 at an attack budget of

R

= 50 orderings. Furthermore, ordinary permutation stability predicts vulnerability without running the attack. While a bidirectional T5 encoder scorer reduces exposure, permutation-consistency regularization and architectural invariance effectively mitigate it. Pointwise scoring avoids the bias issue but degrades ranking quality. These results demonstrate that input candidate order in listwise LLM reranking is a security-relevant attack vector. Code and data are available at https://github.com/geoz-lab/position_bias_attack.

Ge Zhang, Jingru Cheng, Huiyuan Chen

Stanford University · Independent Researcher

Large Language Model Reranking Order Matters Cross-Encoder Cyberattacks Position Stationary Point

Jul 25, 2026cs.LG

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure. Predominant knowledge distillation (KD) methods favor replication: smaller students mimic teacher output logits, yet empirically yield low task performance, hamper convergence, and act merely as regularization rather than substantive knowledge transfer. We propose Saddle Point Recruitment for Knowledge Distillation (SPRKD), reframing distillation from replication to employing teachers as optimization-curvature and domain proxies, characterizing saddle points as regions of strong further-descent potential via embedding and basin-fractal properties. Using Hessian eigenvalue spectral density (ESD), SPRKD identifies low-loss saddle regions for student re-exploration; weak-teacher ensembles are aggregated into an Approximated Saddle Region (ASR), re-parameterized into the student via Transfer Learning by Injection, and approached with exponentially decaying Euclidean transformations, Negative Hessian Eigensteps, and Gaussian perturbations. On malaria blood smear classification with a 6,430-parameter CNN distilled from a weak 25,546-parameter teacher, SPRKD reaches 94.8% validation accuracy, outperforming Response KD by 24.70 percentage points (McNemar p = 6.3e-87) and matching scratch-trained baselines of the same architecture to statistical equivalence (p = 1.0). Across MNIST, CIFAR-100, and TinyImageNet, SPRKD exceeds scratch-trained baselines by up to 8 percentage points on preliminary benchmarks. Hessian ESD and 2-D loss landscape analysis show convergence to wider minima with substantially smaller Hessian trace and spectral radius than Response KD and control students, indicating smoother descent and greater noise robustness.

Aditya Dewan, Arjun Yogeswaran, Benjamin Fedoruk

Carnegie Mellon University · Ontario Tech University

Knowledge Distillation Hessian Stationary Point Jobs

Jul 17, 2026cs.LG

DebrisTracer: Reliable Tracking in Hypervelocity Impact Fast Imaging

This application paper presents DebrisTracer, a framework for the reliable tracking of debris in hypervelocity impact fast imaging. These noisy and highly specific datasets capture the ejection of a large number of debris fragments after the impact of a projectile launched at hypervelocity into a target material. The reliable estimation of debris mass and speed distributions is of major importance in aerospace applications. We document how to extend an off-the-shelf topology tracking framework based on critical point extraction and matching, in order to incorporate domain knowledge and physical assumptions. Our approach automatically produces an accurate and reliable debris tracking, enabling an interpretable visual analysis of this complex space-time phenomenon. Extensive experiments demonstrate the accuracy improvements provided by our approach over established tools used by domain experts in terms of physical validation, specifically via the prediction of the experimental ejected mass and crater depth profiles. We illustrate the utility of our approach across several use cases (with varying impact angles and physics). We show that our statistical summaries enable the visual identification of distinct regimes within the debris population, corroborating and refining prior expectations of domain experts. Our database and our C++ implementation are available at this address: https://github.com/tloloum/DebrisTracer.

Théophane Loloum, Fabien Vivodtzev, David Hébert +3

CNRS and Sorbonne University

Speeds Multi-Object Tracking Stationary Point

Jul 8, 2026stat.ML

Finding a stationary point of a stochastic convex problem

We consider the problem of finding stationary points for stochastic convex optimization problems. Rather than surrogates to stationarity, such as a proximity-to-stationarity guarantee or small gradient of the Moreau envelope, we ask for a stronger notion: that the subdifferential of the objective actually contains a small element. This criterion is non-trivial, because subdifferentials of convex functions fail to converge uniformly, even in arbitrarily small neighborhoods of the optimum. Our convergence guarantees rely on dimension theory to decompose the graph of the subdifferential of a convex function, showing how stochastic sampling preserves "pieces" of these graphs, and allowing effective application of proximal-point-like methods.

Felipe Areces, John Duchi, Malo Sommers

Stanford University

Stochastic Convex Optimization Stationary Point Proximal Operators

Jun 26, 2026cs.LG

Singular Learning and Occam's Razor in Deep Monomial Networks

In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture. These critical points occur where the Jacobian of the model's parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason's Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.

Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir +2

KTH Stockholm · No Affiliation · Umeå University +1

Activation Functions Stationary Point Singular Learning Theory

Jun 25, 2026cs.LG

Finding Stationary Points by Comparisons

We study the problem of finding stationary points of non-convex functions when access to the objective is provided only through a comparison oracle that, given two points, outputs which has the larger function value. For a twice differentiable

f\colon\mathbb R^n\to\mathbb R

with Lipschitz gradient and Hessian, we develop an algorithm that visits an

ε

-stationary point using

\widetilde O(n^2/ε^{1.5})

queries. Our approach uses a subroutine that estimates the normalized Hessian to accuracy

δ

using

\widetilde O(n^2\log(1/δ))

queries. We further study this problem with a quantum comparison oracle model where queries can be made in superpositions, and develop the first quantum algorithm that finds an

ε

-stationary point, which takes

\widetilde O(n/ε^{1.5})

queries.

Helin Wang, Chenyi Zhang, Xiwen Tao +2

School of Electronics Engineering and Computer Science, Peking University · Center on Frontiers of Computing Studies, Peking University · Computer Science Department, Stanford University +1

Stationary Point First Order Oracle Complexity Nonconvex Lipschitz Continuity Comparison

Jun 15, 2026cs.CC

The Complexity of Min-Max Optimization for Quadratic Polynomials

We prove that computing approximate stationary points of min-max optimization over the hypercube is PPAD-hard for quadratic polynomials. This holds even when the polynomials are multilinear, each variable appears in at most three monomials, and the approximation factor is inverse polynomial. As a direct consequence, we obtain the first PPAD-hardness results for two-team zero-sum polymatrix games.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli +1

Bocconi University · Politecnico di Milano · University of Oxford

Np-Hard Polynomials Stationary Point Minimax Imperfect-Information Games Inverse Design

Jun 6, 2026cs.GR

MS-COOT: Comparing Morse-Smale Complexes with Co-Optimal Transport

Understanding and comparing structures in scalar fields is a central challenge in scientific visualization, with applications ranging from feature analysis to temporal and structural comparison. The Morse-Smale (MS) complex provides a natural representation by decomposing a scalar field into regions induced by gradient flow. However, existing approaches typically rely on graph-based representations, capturing relationships between critical points while discarding region-level structure. In this work, we represent the MS complex as a hypergraph, where critical points form nodes and regions define hyperedges. We introduce MS-COOT, a co-optimal transport distance that jointly computes correspondences between critical points and regions. This formulation enables explicit region-to-region matching within a distance-based framework, allowing identification of region-level events such as splitting and merging. We instantiate this framework with domain-specific components, including a hypernetwork function encoding critical point-region relationships, persistence-based probability measures that emphasize topologically significant features, and a sample cost term that incorporates critical point attributes. We evaluate MS-COOT on five datasets spanning 2D simulations, 3D surface meshes, and volumetric data. Our results show that MS-COOT captures region-level structural changes that are not reflected by graph-based distances, while achieving strong performance in downstream tasks such as classification and resolution discrimination.

Guangyu Meng, Mingzhe Li, Erin Wolf Chambers

Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, United States

Multiscale Differentiable Optimal Transport Scalar Field Stationary Point Visual Analytics Hypernetwork

Jun 3, 2026cs.LG

When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function. Concretely, we consider a realizable setting where inputs are drawn i.i.d. from a Gaussian distribution and labels follow a planted linear model. This stylized framework captures salient features of end-to-end training in inverse problems and certain auto-encoder models. Despite its apparent simplicity, the dynamics remain poorly understood, in part because the loss landscape contains multiple non-strict saddle points, making it unclear why gradient descent from random initialization reliably escapes bad stationary regions. We provide a detailed characterization of the optimization landscape and prove that gradient descent from a moderately small random initialization-simultaneously training both layers-converges to a global minimizer at a linear rate with order-wise optimal sample complexity. Our analysis tracks the trajectory through three phases: an alignment phase in which hidden weights progressively align with the planted direction while the output weights maintain the correct sign pattern; a growth phase in which the norms of both layers increase while preserving alignment; and a local refinement phase in which the aligned neurons rapidly converge to the planted direction, yielding fast local convergence. To rigorously show that GD avoids non-strict saddles, we develop trajectory-level control arguments for the end-to-end dynamics. In addition, we establish novel uniform concentration results that hold along the entire trajectory, and are essential for obtaining order-wise optimal sample complexity. We corroborate our theory with extensive experiments across a range of configurations.

Berk Tinaz, Changzhi Xie, Mahdi Soltanolkotabi

Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA

Rectified Linear Unit Networks Gradient Descent Stationary Point Neurons

May 26, 2026cs.LG

Bilevel Optimization over Saddle Points of Zero-Sum Markov Games

Reinforcement learning (RL) often has a hierarchical structure, where an upper-level (UL) learner selects model parameters and a lower-level (LL) decision-making process responds, naturally leading to a bilevel optimization problem. Most existing bilevel RL methods assume a single-policy LL Markov decision process (MDP), and therefore fail to capture competitive structures arising in applications such as incentive design, where multiple policies interact. We study bilevel optimization problems in which the LL problem is a regularized min-max zero-sum Markov game and the UL objective is optimized through the saddle-point equilibrium induced by the LL game. In this work, we propose penalty-augmented Nikaido-Isoda descent-ascent (PANDA), a penalty-based first-order policy-gradient method based on the Nikaido-Isoda function. By exploiting the min-max game structure, PANDA avoids computing UL hypergradients and does not require second-order information. We prove that PANDA converges to stationary points without convexity assumptions on either the UL or LL objectives. Moreover, PANDA reaches an

ε

-stationary point in

\tilde{\mathcal{O}}(ε^{-1})

iterations with sample complexity

\tilde{\mathcal{O}}(ε^{-3})

, matching the best-known rates for bilevel RL with single-policy LL MDPs. Experiments demonstrate the superior performance of PANDA over closely related baselines.

Zihao Zheng, Irwin King, Songtao Lu

Department of Computer Science and Engineering, The Chinese University of Hong Kong · Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong

Bilevel Optimization Mean Field Games Stationary Point

May 17, 2026cs.LG

WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points

Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performance plateau, particularly below 4-bit-widths. While this problem has been observed in prior work, its precise cause remains unclear. In this paper, we analyze the convergence of QAT by estimating the spectrum of the loss-surface Hessians. We find that the weights converge to flat regions around saddle points, where a large fraction of the Hessian eigenvalues are both positive and negative. During training, an increasing fraction of Hessian eigenvalues concentrates around zero, whose magnitude decreases. At lower bit-widths, the magnitude of eigenvalues in the Hessian spectrum is significantly smaller. To mitigate these issues, we propose an algorithm called WinQ to accelerate QAT, which involves: (1) periodically resetting weights to the linear interpolation of full-precision and quantized weights, reducing the distance to the quantization grid and increasing eigenvalue magnitude, and (2) computing gradients of noise-injected weights to regularize the Hessian. Extensive experiments show that WinQ accelerates QAT by up to 4 times across various quantization methods and models. Under the same training cost, WinQ improves state-of-the-art sub-4-bit quantization by up to 8.8%. These results are consistent across 16 settings with different language models, quantization methods, and bit widths.

Dongyue Li, Zechun Liu, Kai Yi +6

1Northeastern University, MA · 2Meta AI, CA.

Quantization-Aware Training Large Language Model Training Model Weights Language Modeling Hessian Stationary Point

May 13, 2026cs.DS

Min-Max Optimization Requires Exponentially Many Queries

We study the query complexity of min-max optimization of a nonconvex-nonconcave function

f

over

[0,1]^d \times [0,1]^d

. We show that, given oracle access to

f

and to its gradient

\nabla f

, any algorithm that finds an

\varepsilon

-approximate stationary point must make a number of queries that is exponential in

1/\varepsilon

or

d

.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli +1

Bocconi University · Politecnico di Milano · University of Oxford

Convex Optimization Nonconvex Stationary Point Exponents Queries

May 11, 2026cs.LG

Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $k$ -step Policy Gradients

This work revisits standard policy gradient methods used on restricted policy classes, which are known to get stuck in suboptimal critical points. We identify an important cause for this phenomenon to be that the policy gradient is itself fundamentally myopic, i.e. it only improves the policy based on the one-step

Q

-function. In this work, we propose a generalized

k

-step policy gradient method that couples the randomness within a

k

-step time window and can escape the myopic local optima in MDPs with restricted policy classes. We show this new method is theoretically guaranteed to converge to a solution that is exponentially close in performance to the optimal deterministic policy with respect to

k

. Further, we show projected gradient descent and mirror descent with this

k

-step policy gradient can achieve this exponential guarantee in

O(\frac{1}{T})

iterations, despite only assuming smoothness and differentiability of the value function. This will provide near optimal solutions to previously elusive applications like state aggregation and partially observable cooperative multi-agent settings. Moreover, our bounds avoid the ubiquitous distribution mismatch factors

||d_μ^{π^*} / d_μ^π||_\infty

and

||d_μ^{π^*} / μ||_\infty

enabling the

k

-step policy gradient method to escape suboptimal critical points that emerge from poor exploration in fully observable settings.

Alex DeWeese, Guannan Qu

Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213-3890, USA

Policy Gradient Optimal Policies Stationary Point Flat Minima Kernel-Resident Tool Governance Gateway

May 7, 2026math.OC

Stability of the Monge Map in Semi-Dual Optimal Transport

This paper shows that the semi-dual formulation of the optimal transport problem has a degenerate saddle-point structure, and that its numerical solution is equivalent to solving a constrained optimization problem. We derive necessary and sufficient conditions for the convergence of Monge maps without requiring optimality of the dual potential. This analysis helps explain why, in practice, numerical algorithms often require more iterations to update the transport map than the potential.

Anton Selitskiy, David Millard

Dept. of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11790, USA · Dept. of Electrical and Computer Engineering University of Rochester Rochester, NY 14627, USA

Differentiable Optimal Transport Primal-Dual Methods Stationary Point Convex Relaxation Classical Numerical Solver