Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization

Junjie Xiao; Huiwen Jia

cs.LGOct 2, 2026

Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization

Authors: Junjie Xiao, Huiwen Jia

Organizations: Department of Mathematics, Peking University · Department of Industrial Engineering and Operations Research, University of California, Berkeley

Abstract

Modern training on a new task often starts from a previously trained model rather than from scratch, raising the question of how this initialization affects the subsequent training trajectory. Classical implicit-bias results characterize the direction selected by prolonged training, but this direction alone does not provide information regarding the intermediate behavior. We address this question through a geometric approximation of full-batch logistic gradient descent (GD) trajectories on strictly linearly separable data, with large initialization of scale $R$ motivated by prior training. From any limiting normalized initial position, we use minimum-norm projection rules to construct a unique continuous ideal path consisting of finitely many linear segments. The path has two stages: negative-margin correction followed by minimum-margin growth. We prove that, after an explicit two-stage time reparameterization, the fixed-step GD trajectory divided by $R$ converges uniformly to this path on every fixed parameter interval as $R\to\infty$ . Further, our quantitative error bounds account for initialization perturbations and the transition between stages. This approximation provides asymptotic formulas for peak evaluation loss and cumulative training loss. In particular, peak evaluation loss can grow linearly in $R$ even when both endpoint losses tend to zero. The cumulative losses in the correction and margin-growth stages, normalized by $R^2$ and $R$ , respectively, converge to explicit limits. Experiments on controlled geometries and fixed image features complement our theoretical results.

Figures & tables

Figure 1: The two-stage ideal path. The analytic example uses

z_{1}=(2,0)

z_{2}=(1,2)

p_{1}=p_{2}=1/2

, and

u_{0}=(-1,-1/5)

. (a) The dashed correction path turns when the second margin reaches zero and reaches the shaded nonnegative region

C

E=(0,1/5)

. The solid continuation follows

z_{1}

until the margins meet, then follows their equal-margin line with velocity

(8/5,4/5)

. (b) The same path’s margins,

z_{i}^{\top}P(r)

, show the correction turn at

r=2/5

, entry at

\tau_{*}=4/5

, and persistent equality from

r=1

. The horizontal axis uses the two-stage parameter defined in ( 5 ).

Figure 2: An intermediate peak in evaluation loss. The analytic example follows the training geometry of Figure 1 in the invariant plane

x_{3}=1

. (a) The path crosses the shaded evaluation region

a^{\top}x<0

; correction is dashed and negative evaluation margins are marked in orange. (b) The limiting normalized evaluation loss peaks at

1/25

, although it vanishes at both endpoints. Both panels are computed from the ideal path.

Figure 3: GD approaches the complete ideal path. (a) A coordinate projection of the four-sample example, omitting most of the initial approach to highlight the turns. Colored curves connect saved GD iterates divided by

R

; the dashed line shows the ideal path, with its entry

E

and endpoint

P(S)

marked. (b) Uniform path errors for the analytic and four-sample examples. (c) The median and 10th–90th percentile band over all 40 random instances. Errors in (b,c) use full parameter vectors over each problem’s complete window

[0,S]

Figure 4: Functional predictions and a test on image features. Panels (a,b) use the four-sample example. (a) Evaluation loss divided by

R

, compared with

H_{\mathrm{ev}}(P(r))

. Curves connect saved GD iterates; shading marks ideal correction. (b) Cumulative training losses divided by

0.38277R^{2}

before entry and

0.10131R

after entry; the dashed line at one is the limit. (c) Uniform path error for the 32-image digit task over its fixed window.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Runs	Number	Total seconds	Largest update count	Budget per run
G1, short window	5	0.63	$5.395\times 10^{6}$	$4\times 10^{9}$
G2, main window	5	8.09	$5.962\times 10^{7}$	$4\times 10^{9}$
Random instances	200	187.93	$1.415\times 10^{9}$	$2\times 10^{9}$
32-image digits task	5	452.75	$2.691\times 10^{7}$	$2\times 10^{8}$
Larger digits tasks, entry	12	382.82	$4.785\times 10^{6}$	$5\times 10^{7}$

Appendix

Table 1: Recorded computation for the selected numerical results. The last column is the allowed number of updates per run; every run in this table completed its specified stopping rule.

$j$	$t_{j}$	$t_{j+1}$	$v_{j}$
0	$0$	$4/15$	$(-3/4,7/4,5/4)$
1	$4/15$	$121/360$	$(-1/4,5/4,3/2)$
2	$121/360$	$151/360$	$(1/2,1/2,5/4)$
3	$151/360$	$383/864$	$(2/9,7/9,10/9)$
4	$383/864$	$28369/54432$	$(1/2,1/4,3/4)$
5	$28369/54432$	$85637/163296$	$(2,1,3)$

Appendix

Table 2: Finite construction of G2 on the reported window. All event times and velocities shown as fractions are exact; the final time is the original grid-selected

S

Instance	$\eta$	$S$	Instance	$\eta$	$S$
00	2.383566011883	0.812550874730	20	3.239836363412	0.687360271366
01	4.988250994041	3.192085672009	21	3.359344012717	1.983970959559
02	4.079246859906	3.703531488345	22	2.326955393044	2.986875270919
03	3.378190352060	2.424884905424	23	3.180895784073	0.292845536333
04	1.769894663824	0.691047805794	24	3.970855233719	1.771797801208
05	1.661883647012	2.979381305143	25	3.614814813406	0.875733263005

Appendix

Table 3: Step sizes and fixed windows for all random instances, rounded to twelve decimal places. The generation sequence and grid rule specify their full floating-point values.

$R$	After entry	Entry at stop	No entry	10%	Median	90%
5	14	20	6	0.07693	0.18467	0.36983
10	20	15	5	0.05373	0.13004	0.21629
20	26	11	3	0.04282	0.09769	0.15039
40	36	3	1	0.03021	0.05787	0.09460
80	39	1	0	0.01850	0.03701	0.06194

Appendix

Table 4: All

40

random runs complete the fixed window at each value of

R

. “After entry” counts runs with at least one post-entry update; the next two columns give entry at the stopping node and no entry by that node.

Geometry	$\eta$	$f(R)$	$\eta\Lambda$	$\mathcal{E}_{R}(S)$	Entry error
G1	0.25	0	0.20505	0.00795	0.00528
G1	0.5	0	0.41010	0.01160	0.00664
G1	1	0	0.82019	0.00885	0.00799
G1	2	0	1.64039	0.01433	0.01383
G1	4	0	3.28078	0.02796	0.02790
G1	1	1	0.82019	0.01398	0.00945

Appendix

Table 5: Completed stability checks at

R=80

. The entry error is

\|w_{\kappa_{R}}/R-E\|

relative to the unperturbed ideal entry.

Source	Target	Source size	Train/test	Source margin	$\eta$	$\tau_{*}$
$3/8$	$5/6$	357	254/109	0.20773	0.328481301706	6.473229321622
$4/9$	$2/3$	361	251/109	0.37357	0.317649989551	14.787703491645
$0/1$	$7/9$	360	251/108	0.59168	0.342935275073	16.407770420857

Appendix

Table 6: Larger digits tasks used for the entry-location comparison. The source margin is

\min_{i}z_{i}^{A\top}u_{0}

. The target split and feature map are specified above.

Figure 5: Additional functional and entry-location results. (a) The maximum evaluation loss divided by

R

in the full-rank four-sample geometry G2 approaches the ideal value

553/(1440\sqrt{17})

; maxima include every GD iterate through the stopping iterate. (b) Mean loss on all

357

source images of digits

3/8

, divided by

R

, along GD trained on the fixed

32

images of digits

5/6

. The black curve is the ideal evaluation profile, and shading marks the ideal correction interval. Colored curves connect saved nodes using each run’s actual time parameterization. (c) Entry-position errors

\|w_{\kappa_{R}}/R-E\|

for the three larger target tasks in Table 6 . Each run stops at actual entry; this panel measures entry convergence rather than a completed post-entry path.

Explore similar work

Aug 5, 2026cs.LG

Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within

O(\exp(\exp(-δ)))

iterations, where

δ>0

is the permissible alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment.

Han Bao

The Institute of Statistical Mathematics · Tohoku University · RIKEN AIP

Gradient Descent Overparameterization Empirical Risk Minimization Convergence Logistic Regression Biases

Ask PDF

Jun 7, 2024cs.LG

Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes

We study gradient descent (GD) dynamics on logistic regression problems with large, constant step sizes. For linearly-separable data, it is known that GD converges to the minimizer with arbitrarily large step sizes, a property which no longer holds when the problem is not separable. In fact, the behaviour can be much more complex -- a sequence of period-doubling bifurcations begins at the critical step size

2/λ

, where

λ

is the largest eigenvalue of the Hessian at the solution. Using a smaller-than-critical step size guarantees convergence if initialized nearby the solution: but does this suffice globally? In one dimension, we show that a step size less than

1/λ

suffices for global convergence. However, for all step sizes between

1/λ

and the critical step size

2/λ

, one can construct a dataset such that GD converges to a stable cycle. In higher dimensions, this is actually possible even for step sizes less than

1/λ

. Our results show that although local convergence is guaranteed for all step sizes less than the critical step size, global convergence is not, and GD may instead converge to a cycle depending on the initialization.

Si Yi Meng, Antonio Orvieto, Daniel Yiming Cao +1

Department of Computer Science Cornell University Ithaca, NY, USA · ELLIS Institute Tübingen MPI for Intelligent Systems Tübingen AI Center, Germany

Step Accuracy Gradient Descent Logistic Regression Bifurcation Separability High-Dimensional+1

Ask PDF

Oct 5, 2026cs.LG

Improved Convergence of Large Stepsize Gradient Descent for Logistic Regression

We study gradient descent (GD) with a large constant stepsize for logistic regression on linearly separable data. Existing analysis shows an accelerated rate of

\widetilde{O}(1/\sqrtε)

to reach loss

ε

with an aggressive stepsize, although the loss may initially oscillate. Tighter control of the oscillatory dynamics has been available only for two-dimensional data. We prove a substantially faster rate in arbitrary dimension: GD with a large stepsize

η=1/ε

reaches loss

ε

within

O(\ln^{p}(1/ε))

steps, where

p

depends only on the margin and the rank of the data. Our proof improves the bound on the transition time of GD from the oscillatory to the stable phase, after which the loss decreases monotonically. We split the oscillatory phase into recursively nested intervals. The margin and the rank bound the nesting depth, and a counting argument bounds the number of intervals at each depth, together yielding the polylogarithmic step complexity.

Xiaochuan Gong, Ang Li

University of Maryland, College Park

Logistic Regression

Ask PDF