Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization
Organizations: The University of Tokyo
Abstract
Gradients are no longer used only to train models: Gradient magnitudes are now measurement instruments, used to rank attention heads for pruning, to decide per-example clipping in private training, to allocate adapter rank budgets, and to debug models. Across these workflows, larger gradient magnitudes are taken to mean greater importance. Even for a fixed input--output function, these gradients are computed in one of many possible parameter coordinate systems. For example, attention scores depend on the query and key projections only through the paired product . Scaling a head's query projection by and the corresponding key projection by therefore leaves every output logit unchanged. This change in parameter coordinates rescales the corresponding query and key gradient norms by and , respectively. To test whether gradient-based decisions remain unchanged under these coordinate changes, we introduce Gradient Reliability under Reparameterization (GRR). GRR rewrites a trained model in new parameter coordinates that compute exactly the same function, verifies that predictions are unchanged, and then tests whether the decisions derived from the gradients are unchanged too. Our audits span GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter Llama-style model on WikiText-2, and six trained LoRA adapters. Empirically, predictions are preserved throughout, but both the gradients and the gradient-based decisions change. Across 66 GLUE attention audits, the median top-3 overlap of raw head-gradient rankings is only , and rankings and pruning masks change even under the mildest rescalings we test. In the largest of 14 global-pruning audits, the shifted ranking selects head masks that differ in four of 22 pruned heads and reach and validation accuracy on one ELECTRA SST-2 model.
Figures & tables
| Measurement | Agreement or flip rate | Mean ratio |
|---|---|---|
| Raw head-gradient norm | ||
| Raw paired-product tangent | ||
| Per-example clipping | Not applicable | |
| Function-space ablation | ||
| Path-normalized gradient | ||
| Canonically balanced gradient |
| Decision rule | Jaccard median | Jaccard range | Max accuracy gap (pp) |
|---|---|---|---|
| Single-layer raw bottom-3 pruning | 0.20 | 1.47 | |
| Single-layer raw top-3 ablation | 0.20 | 10.05 | |
| Single-layer function-space top-3 | 1.00 | 0.00 | |
| Global raw bottom-15% pruning | 0.517 | 43.92 | |
| Global raw top-15% ablation | 0.294 | 27.18 | |
| Global canonically balanced bottom-15% | 1.00 | 0.00 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| Model, loss, and audit | |
| Network with parameters | |
| Parameter vector and parameter space | |
| Input space and finite audit set | |
| Logits of at input | |
| Per-example loss and audited loss | |
| Model | Task | Metric | Audit count | Eval |
| DistilBERT fine-tuned | SST-2 | 91.06% | 12 | 872 |
| BERT-base fine-tuned | 92.43% | 24 | ||
| RoBERTa-base fine-tuned | 94.04% | 6 | ||
| ELECTRA-base fine-tuned | 95.18% | 6 | ||
| RoBERTa-base fine-tuned | MNLI-m | 88.10% | 6 | 9,815 |
| RoBERTa-base fine-tuned | QNLI | 92.68% | 6 | 5,463 |
| Transformation family | Audits | Max logit difference | Prediction and argmax flips |
|---|---|---|---|
| GLUE layer Q–K and V–O attention gauges | 66 | 0 | |
| ViT attention gauges on CIFAR-10 | 24 | 0 | |
| ViT global V–O profiles on CIFAR-10 | 3 | 0 | |
| GPT-2-medium attention gauges on WikiText-2 | 48 | 0 | |
| GPT-2-large attention gauges on WikiText-2 | 72 | max | |
| GPT-2-medium global V–O profiles on WikiText-2 | 3 | 0 |
| Measurement | DistilBERT on SST-2 | RoBERTa on MRPC |
|---|---|---|
| Baseline accuracy | mean 89.87% | mean 90.28% |
| Raw rank-gradient top-12 overlap | med. 0.417 | med. 0.333 |
| Raw paired-product tangent top-12 overlap | med. 0.250 | med. 0.250 |
| Raw paired-product tangent mean ratio | med. 5.25 | med. 5.08 |
| Canonical gradient and product top-12 overlap | 1.000, 1.000 | 1.000, 1.000 |
| Per-example clipping flip rate | med. 8.33% | med. 14.58% |
| Gauge range | Raw top-3 overlap | Raw Spearman | Paired-product ratio |
|---|---|---|---|
| Mild | |||
| Medium | |||
| Aggressive |
| Clipping flip rate (%) | ||||
|---|---|---|---|---|
| Gauge range | All subsets | Median-norm ratio | ||
| Mild | ||||
| Medium | ||||
| Aggressive | ||||
| Model and task | Gauge range | Raw top-3 overlap | Raw Spearman | Paired-product ratio |
|---|---|---|---|---|
| BERT on SST-2 | Mild | |||
| Aggressive | ||||
| ELECTRA on SST-2 | Mild | |||
| Aggressive | ||||
| RoBERTa on SST-2 | Mild | |||
| Aggressive |
| Gauge profile | Decision | Raw mask Jaccard | Raw accuracy gap (pp) |
|---|---|---|---|
| Mild | bottom-15% pruning | ||
| top-15% ablation | |||
| Medium | bottom-15% pruning | ||
| top-15% ablation | |||
| Aggressive | bottom-15% pruning | ||
| top-15% ablation |
| Model | Measurement | Top-3 overlap or flip rate | Mean ratio | Decision effect |
|---|---|---|---|---|
| GPT-2-medium | Raw head-gradient norm | pruning Jaccard med. | ||
| Raw paired-product tangent | max ppl. gap | |||
| GPT-2-large | Raw head-gradient norm | pruning Jaccard med. | ||
| Raw paired-product tangent | max ppl. gap | |||
| TinyLlama 1.1B GQA | Raw head-gradient norm | pruning Jaccard med. | ||
| Raw paired-product tangent | max ppl. gap |
| Model | Gauge profile | Decision | Raw mask Jaccard | Raw ppl. gap |
|---|---|---|---|---|
| GPT-2-medium | Mild | bottom-15% pruning | ||
| top-15% ablation | ||||
| Medium | bottom-15% pruning | |||
| top-15% ablation | ||||
| Aggressive | bottom-15% pruning | |||
| top-15% ablation |
| Gauge profile | Decision | Raw mask Jaccard | Raw accuracy gap (pp) |
|---|---|---|---|
| Mild | bottom-15% pruning | ||
| top-15% ablation | |||
| Medium | bottom-15% pruning | ||
| top-15% ablation | |||
| Aggressive | bottom-15% pruning | ||
| top-15% ablation |
| Mild bottom-15% pruning | Mild top-15% ablation | |||||
|---|---|---|---|---|---|---|
| Task and checkpoint | Baseline acc. (%) | Mask Jaccard | Acc. gap (pp) | Mask Jaccard | Acc. gap (pp) | Max raw gap (pp) |
| RoBERTa on SST-2 | ||||||
| RoBERTa on QNLI | ||||||
| RoBERTa on MRPC | ||||||
Gradient-Based Attribution
Gradient-Based Attribution uses model derivatives to estimate which inputs, features, parameters, or training examples influence predictions and behaviors.
See all 120 papers