cs.LGSep 30, 2026

Same Loss, Different Gradients

Authors: Ningkang Peng, Xiaoqian Peng, Yifan He, Anjie Hu, Chao Tan, Peirong Ma, Yanhui Gu

Organizations: Nanjing Normal University · Nanjing University of Chinese Medicine

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Differentiable learning typically assumes that the scalar objective evaluated in the forward pass and the gradient supplied to the optimizer in the backward pass describe the same mathematical object. We show that this correspondence can fail when probabilistic objectives rely on finite special-function recurrences, custom backward rules, and numerical clipping. In high-dimensional von Mises-Fisher learning, real numerical implementations can produce identical forward scores and losses at the same learning state while supplying different gradients and following different optimization trajectories. We characterize the structure of this mismatch in finite-start Bessel recurrence and show that classwise radial mismatch can compose through probabilities into a locally nonconservative update field. Evaluating the accuracy of special-function values and derivatives separately is therefore insufficient to characterize the realized learning objective. Motivated by this observation, we introduce AR/FR, a fixed-depth analytic realization that constructs a potential and its derivative jointly, ensuring forward-backward coherence by construction. We establish a uniform cubic-order error bound relative to the exact Bessel ratio over the entire nonnegative concentration axis and propagate this guarantee to learning scores and objectives. As representation dimension increases, the original finite recurrence becomes sequentially deeper, whereas the worst-case AR/FR error guarantee tightens cubically, jointly providing coherence, certified fidelity, and fixed-depth computation. These results suggest that a differentiable numerical primitive is defined by both the values it realizes and the derivatives it actually supplies to the optimizer; together, they constitute the numerical realization of the learning algorithm.

Figures & tables

Figure 1: Same loss, different gradients: diagnosis and paired realization. (a) A common forward objective can supply different updates when backward rules differ. (b) The coherence defect compares the supplied backward with the derivative of that forward. (c) AR/FR constructs the potential and derivative together, combining coherence with a uniform fidelity bound.
Figure 2: Local geometry of coherent and mismatched update fields. Both panels share the same scalar forward surface. A matched backward yields a conservative gradient field with symmetric Jacobian, whereas a mismatched backward can introduce a local rotational component and asymmetric cross-derivatives. The figure is an analytic illustration.
Figure 3: Same-state gradient mismatch becomes trajectory separation. Original and consistent recurrence share random or intermediate-training starting states and augmented inputs. Left: local gradient gaps with bitwise-identical forward values. Right: parameter and feature distances after separate updates.
Figure 4: Concentration and depth jointly determine coherence. Colour shows log⁡10(∣gsupplied−Φ~′∣/∣Φ~′∣)\log_{10}(|g_{\rm supplied}-\widetilde{\Phi}^{\prime}|/|\widetilde{\Phi}^{\prime}|) , with a 10−1610^{-16} display floor. The diamond locates captured PATT queries at p=512p=512 and M/ν=2M/\nu=2 .
Figure 5: Cubic error decay. Blue: sampled maximum error. Orange: the all-axis certificate ν−3\nu^{-3} .
DatasetIFMetricOriginal recurrenceConsistent recurrenceAR/FR
CIFAR-10-LT10Acc. ↑\uparrow92.04±0.1792.04\pm 0.1792.12‾±0.18\underline{92.12}\pm 0.1892.16±0.19\mathbf{92.16}\pm 0.19
  AUROC ↑\uparrow93.07‾±0.13\underline{93.07}\pm 0.1392.30±0.2792.30\pm 0.2793.43±0.46\mathbf{93.43}\pm 0.46
CIFAR-10-LT50Acc. ↑\uparrow87.30±0.2987.30\pm 0.2987.58±0.35\mathbf{87.58}\pm 0.3587.42‾±0.15\underline{87.42}\pm 0.15
  AUROC ↑\uparrow90.18±2.4690.18\pm 2.4690.95‾±1.57\underline{90.95}\pm 1.5791.02±1.13\mathbf{91.02}\pm 1.13
CIFAR-10-LT100Acc. ↑\uparrow83.82±0.2183.82\pm 0.2184.67‾±0.41\underline{84.67}\pm 0.4184.73±0.31\mathbf{84.73}\pm 0.31
  AUROC ↑\uparrow90.66±0.13\mathbf{90.66}\pm 0.1389.97‾±0.39\underline{89.97}\pm 0.3989.95±0.3489.95\pm 0.34
Table 1: Classification and OOD detection under different numerical realizations. CIFAR results report mean ±\pm standard deviation over three runs. Metrics are percentages; IF denotes the imbalance factor. Full FPR95 results appear in Appendix G.1 . Bold and underline indicate the best and second-best values within each setting.
Figure 6: Local runtime–memory trade-off.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: ProCo coherence defects depend on the visited state. Absolute differences compare the supplied radial rule with FP64 autodiff at M=2νM=2\nu . Boxes show median and interquartile range; whiskers span extrema. (a) CIFAR queries, (b) ImageNet queries, and (c) iNaturalist class-bank concentrations. The dashed diagnostic scale is 10−610^{-6} ; values below 10−1610^{-16} share a display floor.
ViewMax score ∣Δ∣|\Delta|Loss ∣Δ∣|\Delta|Max ∣Dclip∣|D_{\rm clip}|Gradient relative ℓ2\ell_{2}Gradient max ∣Δ∣|\Delta|
200002.55×10−32.55\times 10^{-3}2.56×10−32.56\times 10^{-3}1.40×10−51.40\times 10^{-5}
300002.55×10−32.55\times 10^{-3}2.56×10−32.56\times 10^{-3}1.35×10−51.35\times 10^{-5}
Appendix
Table 2: Changing only the backward rule reveals a PATT coherence defect. Original and consistent recurrence use the same frozen CIFAR100-LT state and two augmented views (seed 3407). Score and loss columns report absolute differences; gradient columns compare the complete adjusted cross-entropy feature gradients. Dclip=Φ~′−g=−dD_{\rm clip}=\widetilde{\Phi}^{\prime}-g=-d is the signed radial residual.
ν\nuppRaw R~\widetilde{R}SignPhaseScaledLimitRel. error
631281.645×1051.645\times 10^{5}++linear1.645×10−41.645\times 10^{-4}1.645×10−41.645\times 10^{-4}1.2×10−111.2\times 10^{-11}
641306.208×10−66.208\times 10^{-6}−-inverse6.208×1036.208\times 10^{3}6.208×1036.208\times 10^{3}1.3×10−111.3\times 10^{-11}
2555121.020×1041.020\times 10^{4}++linear1.020×10−51.020\times 10^{-5}1.020×10−51.020\times 10^{-5}3.2×10−93.2\times 10^{-9}
2565149.856×10−59.856\times 10^{-5}−-inverse9.856×1049.856\times 10^{4}9.856×1049.856\times 10^{4}3.3×10−93.3\times 10^{-9}
Appendix
Table 3: Finite-recurrence asymptotics follow the predicted parity. At x=109x=10^{9} and M=2νM=2\nu , odd and even orders are compared with their analytic limits using 100-digit arithmetic. The scaled quantity is R~/x\widetilde{R}/x for odd ν\nu and xR~x\widetilde{R} for even ν\nu ; sign refers to R~−Rν\widetilde{R}-R_{\nu} .
ComparisonReferenceMaximum discrepancy
Finite-potential scoreExact score3.32×10−33.32\times 10^{-3}
AR/FR same-state scoreExact score1.17×10−91.17\times 10^{-9}
Clipped backward radial factorExact Bessel ratio2.55×10−32.55\times 10^{-3}
Forward–backward coherence defectFinite-forward derivative2.56×10−32.56\times 10^{-3}
Appendix
Table 4: Score fidelity and backward coherence at captured PATT states. Maximum absolute discrepancies over six CIFAR100-LT views. Score and radial fidelity use the exact Bessel reference; coherence compares the supplied backward with the derivative of its own finite forward.
DimensionCUSFAR/FRAR/FR certificate
643.37×10−123.37\times 10^{-12}2.74×10−62.74\times 10^{-6}3.36×10−53.36\times 10^{-5}
1282.38×10−112.38\times 10^{-11}3.29×10−73.29\times 10^{-7}4.00×10−64.00\times 10^{-6}
2562.29×10−112.29\times 10^{-11}4.03×10−84.03\times 10^{-8}4.88×10−74.88\times 10^{-7}
5121.85×10−111.85\times 10^{-11}4.99×10−94.99\times 10^{-9}6.03×10−86.03\times 10^{-8}
10242.45×10−112.45\times 10^{-11}6.21×10−106.21\times 10^{-10}7.49×10−97.49\times 10^{-9}
20488.87×10−118.87\times 10^{-11}7.74×10−117.74\times 10^{-11}9.34×10−109.34\times 10^{-10}
Appendix
Table 5: Ratio fidelity across feature dimensions. Maximum absolute errors use 61 log-spaced concentrations x/ν∈[10−3,103]x/\nu\in[10^{-3},10^{3}] per dimension and a 60-digit Bessel reference. CUSF forms a ratio from adjacent log-Bessel values in CPU FP64; AR/FR is evaluated in 60-digit arithmetic. The certificate ν−3\nu^{-3} covers the full nonnegative axis.
DimensionLog-MillerCUSFAR/FR
643.36×10−43.36\times 10^{-4}1.89×10−121.89\times 10^{-12}2.76×10−62.76\times 10^{-6}
1288.29×10−58.29\times 10^{-5}8.47×10−128.47\times 10^{-12}3.30×10−73.30\times 10^{-7}
2562.06×10−52.06\times 10^{-5}2.31×10−112.31\times 10^{-11}4.04×10−84.04\times 10^{-8}
5125.13×10−65.13\times 10^{-6}9.72×10−129.72\times 10^{-12}5.00×10−95.00\times 10^{-9}
10241.28×10−61.28\times 10^{-6}3.21×10−113.21\times 10^{-11}6.21×10−106.21\times 10^{-10}
20482.11×10−72.11\times 10^{-7}9.67×10−119.67\times 10^{-11}7.74×10−117.74\times 10^{-11}
Appendix
Table 6: Fidelity of unit-length potential differences. Maximum absolute errors in Φν(x+1)−Φν(x)\Phi_{\nu}(x+1)-\Phi_{\nu}(x) use the common 61-point concentration grid at each dimension. Log-Miller retains the finite start; CUSF subtracts separately evaluated potentials; AR/FR uses its paired analytic potential. All errors use the high-precision Bessel reference.
Figure 8: Score and loss errors remain below their certificates as class count varies. The 15 fixed-state configurations combine five class counts with prior ratios 1, 100, and 1000. (a) The three maximum score-error curves coincide and are drawn once. (b) Mean cross-entropy error is shown separately for each prior ratio. Dashed lines denote the respective theoretical bounds. Class counts are equally spaced discrete settings.
Figure 9: A frozen PATT state exhibits mainly gradient-magnitude distortion. The first CIFAR-10 query is varied over a 25×2525\times 25 slice along two principal feature directions, with class statistics fixed. Scores and losses agree bitwise at every location. (a–b) Negative gradients from the finite forward and supplied backward, using common contours and arrow scales. (c) Their difference magnified 400×400\times ; each field displays 64 interior arrows. (d–e) Empirical cumulative distributions over all 625 projected-gradient pairs. Relative mismatch has median 0.2561%0.2561\% ; 1−cos⁡1-\cos resolves the small directional difference.
Figure 10: Classwise mismatch controls the local Jacobian obstruction. (a) Signed antisymmetry versus concentration ratio at 29 constructed states: lines show the analytic prediction and circles the differentiated field; the consistent field remains zero. (b) Gradient-gap norm and absolute antisymmetry across nine mismatch strengths, with predictions shown as lines and computed values as markers. All interventions retain forward loss log⁡2\log 2 .
Figure 11: Concentration and gradient scaling delimit the coherence defect. (a) Absolute clipped defect over 130 conditions; dashed segments show the asymptote (ν+3/2)/x(\nu+3/2)/x , and the grey interval marks saved PATT query radii. Values below 10−1610^{-16} share a display floor. (b) Cumulative distributions for 256 frozen PATT queries in the full feature space with FP64 geometry. Relative differences use the consistent gradient norm before scaling and the supplied gradient norm after optimal per-query scaling.
Numerical realizationRuntime (ms)Peak increment (MiB)
Original recurrence266.40504.89
Consistent recurrence412.801755.42
Log-domain Miller120.28504.89
AR/FR4.90508.12
Structure-aware AR/FR0.20632.00
Appendix
Table 7: Local H100 forward–backward costs. Median runtime and incremental peak allocation under the protocol in Appendix F.3 .
DatasetIFMetricOriginal recurrenceConsistent recurrenceAR/FR
CIFAR-10-LT10Acc. ↑\uparrow92.04±0.1792.04\pm 0.1792.12‾±0.18\underline{92.12}\pm 0.1892.16±0.19\mathbf{92.16}\pm 0.19
  AUROC ↑\uparrow93.07‾±0.13\underline{93.07}\pm 0.1392.30±0.2792.30\pm 0.2793.43±0.46\mathbf{93.43}\pm 0.46
  FPR95 ↓\downarrow26.20‾±1.19\underline{26.20}\pm 1.1929.52±1.0629.52\pm 1.0625.52±1.01\mathbf{25.52}\pm 1.01
CIFAR-10-LT50Acc. ↑\uparrow87.30±0.2987.30\pm 0.2987.58±0.35\mathbf{87.58}\pm 0.3587.42‾±0.15\underline{87.42}\pm 0.15
  AUROC ↑\uparrow90.18±2.4690.18\pm 2.4690.95‾±1.57\underline{90.95}\pm 1.5791.02±1.13\mathbf{91.02}\pm 1.13
  FPR95 ↓\downarrow34.51±6.9634.51\pm 6.9632.84‾±1.94\underline{32.84}\pm 1.9432.23±2.89\mathbf{32.23}\pm 2.89
Appendix
Table 8: Complete classification and OOD detection results. Metrics are percentages; CIFAR results report mean ±\pm standard deviation over three runs. Bold and underline indicate the best and second-best values within each setting.

Explore similar work

CardsList

  1. How Accurate Is Accurate Enough?

    Sep 30, 2026Ningkang Peng, Qianfeng Yu, Jingyang Mao +2Loss FunctionProbability

    Nanjing Normal University · Nanjing University of Chinese Medicine

  2. Learn-Then-Differentiate Gradient Estimation

    Sep 30, 2026Nifei Lin, Qingkai Zhang, L. Jeff HongGradientImplicit Differentiation

    Research Institute for Interdisciplinary Sciences, School of Information Management and Engineering, Shanghai University of Finance and Economics, Shanghai 200433, China · Department of Decision Analytics and Operations, City University of Hong Kong, Hong Kong, China · Department of Industrial and Systems Engineering, University of Minnesota, Minneapolis, Minnesota 55455

  3. Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling

    Jul 9, 2026Yiwei ZhouDiffusion SamplingWasserstein Gradient Flows