Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional p-th moments, p≥2. Our main result gives explicit accuracy and confidence conditions under which, for p>2, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At p=2, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
Figures & tables
Figure 1: OBD off-policy click prediction. Lower is better in (b)–(c). (a) Empirical survival functions of the importance weight and stochastic-gradient norm at the IPS optimum, each normalized by its median; the effective-sample fraction is 0.16% and the largest weight is 12500. (b) Direct target-error comparison: GM(N) from ( 28 ) for U-SGD and WG-Clip with the single source-validation-selected threshold λval=5 , held fixed throughout every trajectory. (c) Target-error sensitivity GM(4194304) across every predeclared finite threshold; the dashed horizontal line is U-SGD. Points below it favor WG-Clip, whereas points above it favor U-SGD. This post-selection sweep is a diagnostic and is not used for tuning. Bands and bars are pointwise distribution-free 95% order-statistic intervals conditional on the fixed logs; paired-bootstrap intervals are reported in the text.