cs.LGNov 1, 2026

Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization

Authors: Bum Jun Kim

Organizations: The University of Tokyo

Abstract

Gradients are no longer used only to train models: Gradient magnitudes are now measurement instruments, used to rank attention heads for pruning, to decide per-example clipping in private training, to allocate adapter rank budgets, and to debug models. Across these workflows, larger gradient magnitudes are taken to mean greater importance. Even for a fixed input--output function, these gradients are computed in one of many possible parameter coordinate systems. For example, attention scores depend on the query and key projections only through the paired product WQWK⊤W_QW_K^\top. Scaling a head's query projection by cc and the corresponding key projection by 1/c1/c therefore leaves every output logit unchanged. This change in parameter coordinates rescales the corresponding query and key gradient norms by 1/c1/c and cc, respectively. To test whether gradient-based decisions remain unchanged under these coordinate changes, we introduce Gradient Reliability under Reparameterization (GRR). GRR rewrites a trained model in new parameter coordinates that compute exactly the same function, verifies that predictions are unchanged, and then tests whether the decisions derived from the gradients are unchanged too. Our audits span GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter Llama-style model on WikiText-2, and six trained LoRA adapters. Empirically, predictions are preserved throughout, but both the gradients and the gradient-based decisions change. Across 66 GLUE attention audits, the median top-3 overlap of raw head-gradient rankings is only 1/31/3, and rankings and pruning masks change even under the mildest rescalings we test. In the largest of 14 global-pruning audits, the shifted ranking selects head masks that differ in four of 22 pruned heads and reach 50.92%50.92\% and 94.84%94.84\% validation accuracy on one ELECTRA SST-2 model.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards

    Jun 3, 2026Tej Deep Pala, Vernon Toh, Soujanya PoriaRL for Language Model ReasoningCredit Assignment in RL

  2. ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

    Oct 14, 2026Fanyao Li, Yuxi Liu, Xiaohao Liu +3Low-Rank AdaptationParameter-Efficient Fine-Tuning

  3. Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift

    Jun 14, 2026Weihang Su, Jiacheng Kang, Jingyan Xu +7Retrieval-Augmented GenerationContinual Learning for LLMs