Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Yufan Zhang; Sagnik Mukherjee; Hao Peng

cs.LGSep 28, 2026

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Authors: Yufan Zhang, Sagnik Mukherjee, Hao Peng

Organizations: University of Illinois Urbana-Champaign

Abstract

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training.

Figures & tables

Figure 1: Logical chain of how on-policy training gives rise to low Fisher and concentrated learning. Section 4.1 studies the first two links, showing how on-policy sampling can produce small token Fisher and how these token-level contributions accumulate into small model Fisher. Section 3.1 establishes the third link, showing theoretically how small model Fisher limits expected gradients. Sections 3.2 and 3.3 empirically verify whether Fisher identifies where gradients and effective learning are concentrated.

Figure 2: Fisher identifies a small set of parameters that supports learning. (a) Before training, the top 1% Fisher coordinates contain 47.40% of the top 1% gradient coordinates, compared with a 1.00% random baseline. (b) Updating only this Fisher-selected subset achieves much of the performance gain from updating all parameters (Full) and substantially outperforms an equally sized random subset. Results are for Qwen3-1.7B-Base trained with RL for mathematical reasoning.

Method	Response distribution $\mu$	Learning signal $R_{\theta}$
SFT	$\nu(y\mid x)$	$\log\pi_{\theta}(y\mid x)$
On-policy RL	$\pi_{\theta}(y\mid x)$	$r(x,y)$
PG-style OPD	$\pi_{\theta}(y\mid x)$	$R_{T}(x,y)$
Full-vocabulary OPD	$\pi_{\theta}(y\mid x)$	$-\sum_{t}D_{\mathrm{KL}}(\pi_{T}(\cdot\mid s_{t})\\|p_{\theta}(\cdot\mid s_{t}))$

Table 1: Post-training objectives and update conventions. The demonstration distribution

\nu

, teacher

\pi_{T}

, and reward

r

are fixed, and

R_{T}(x,y)=\sum_{t}\log[\pi_{T}(y_{t}\mid s_{t})/p_{\theta}(y_{t}\mid s_{t})]

. For PG-style OPD,

R_{T}

is treated as a fixed reward in the policy-gradient estimator. For full-vocabulary OPD, the rollout distribution is held fixed during differentiation, and gradients are taken only through the student’s next-token probabilities.

Method	Effective signal $a(x,y)$	Squared scale
SFT	$w_{\nu}(x,y)-1$	$\mathbb{E}_{x,\,y\sim\pi_{\theta}}[(w_{\nu}-1)^{2}]$
On-policy RL	$r(x,y)-\bar{r}(x)$	$\mathbb{E}_{x}\operatorname{Var}_{\pi_{\theta}}(r\mid x)$
PG-style OPD	$R_{T}(x,y)-\bar{R}_{T}(x)$	$\mathbb{E}_{x}\operatorname{Var}_{\pi_{\theta}}(R_{T}\mid x)$
Full-vocabulary OPD	$A_{T}(x,y)$	$\mathbb{E}_{x,\,y\sim\pi_{\theta}}[A_{T}(x,y)^{2}]$

Table 2: Effective signals and squared scales

\mathbb{E}_{x,\,y\sim\pi_{\theta}}[a^{2}]

in Equation ( 4 ). Here

w_{\nu}(x,y)=\nu(y\mid x)/\pi_{\theta}(y\mid x)

\bar{r}(x)=\mathbb{E}_{\pi_{\theta}}[r\mid x]

\bar{R}_{T}(x)=\mathbb{E}_{\pi_{\theta}}[R_{T}\mid x]

, and

A_{T}(x,y)=\sum_{t}\bigl(\pi_{T}(y_{t}\mid s_{t})/p_{\theta}(y_{t}\mid s_{t})-1\bigr)

Figure 3: (a) Gradient-support recall (%) across selection percentages $q$ . Initial denotes the pretrained policy. Checkpoint rows average independent full-parameter training runs. Random is the analytic baseline. (b) Final MATH-500 mean reward across fixed-mask budgets. Fisher and Random denote fixed Fisher-selected and uniformly random masks, respectively, while Full updates all parameters. The dashed line marks the initial policy’s score ( $22.65\%$ ).

			Current low-Fisher group (%)
Checkpoint	Metric	All	99.95	99.9	99.5	99	95
Initial	Real	22.19	4.48	3.98	3.06	2.72	1.87
	Uniform	141.21	41.81	39.36	34.43	32.48	27.87
	Ratio	0.16	0.11	0.10	0.09	0.08	0.07
Step 250	Real	11.57	3.00	2.71	2.13	1.90	1.32
	Uniform	184.21	46.77	43.07	35.72	32.89	26.01

Table 3: Average token Fisher under the model’s probabilities (Real), the uniform reference (Uniform), and their ratio. Fisher values are in units of

10^{-6}

. Low-Fisher groups are selected from each checkpoint. Checkpoint values average four training runs. The initial policy is shared.

		Selection $q$ (%)
Checkpoint	Metric	All	0.05	0.1	0.5	1	5
Step 250	Retention	–	99.99 / 75.76	99.98 / 76.96	99.89 / 77.67	99.76 / 76.70	98.78 / 76.89
	$M_{k}/M_{0}$	0.70	0.92	0.92	0.93	0.93	0.92
Step 500	Retention	–	99.99 / 73.02	99.97 / 74.36	99.88 / 75.30	99.74 / 74.37	98.66 / 74.62
	$M_{k}/M_{0}$	0.73	0.99	1.00	1.00	1.01	0.99
Step 750	Retention	–	99.99 / 72.02	99.97 / 73.48	99.87 / 74.61	99.73 / 73.68	98.63 / 73.96

Table 4: Persistence of the initial Fisher pattern during full-parameter training. Retention reports low/high-Fisher set overlap (%), and

M_{k}/M_{0}

reports the Fisher magnitude of the fixed initial low-Fisher set relative to initialization. Values are averaged over four runs. Random gives the analytic retention baseline.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Model	Qwen3-1.7B-Base
Policies	Shared Initial; Full training seeds 0–3 at steps 250, 500, 750, and 1000
Prompt source	The 8,513-problem RL training set
Fisher sample size	800 prompts, one response per prompt
Gradient sample size	300 prompts, eight responses per prompt
Generation	Temperature 1, top- $p=1$ , no top- $k$ truncation

Appendix

Table 5: Test 1 measurement settings.

Setting	RL	OPD
Initial model	Qwen3-1.7B-Base	Qwen3-1.7B (post-trained)
Teacher	None	Qwen3-4B-Instruct-2507, frozen
Training dataset	SimpleRL-Zoo, Abel levels 3–5	DAPO-Math-17k
Training data size	8,513 problems	17,183 problems after deduplication and removal of exact MATH-500 matches
Objective	Dr. GRPO	Token-local PG-style distillation
Optimizer updates	1,000	20

Appendix

Table 6: Training and evaluation settings for the reported Test 2 comparisons. All conditions within each column share these settings.

Setting	Value
Model and policies	Qwen3-1.7B-Base; shared Initial and Full seeds 0–3 at steps 250/500/750/1000
Prompts	200 prompts sampled without replacement from Test 1’s 300 gradient prompts, shared across policies
Responses	Each policy’s first saved on-policy response per prompt
Response generation	Temperature 1, top- $p=1$ , no top- $k$ truncation; prompt/response limits 1,024/4,096
Positions	Eight uniformly selected positions per response; identical for Real and Uniform
Token pairs	Eight independent pairs per position per condition, sampled with replacement

Appendix

Table 7: Settings for the Real–Uniform token-Fisher comparison.

$q$ (%)	Full	Fisher	R0	R1	Random	Recovery	$\Delta$
0.05	56.41	49.65	22.75	23.66	23.21	79.98	26.44
0.1	56.41	51.41	24.36	23.44	23.90	85.21	27.51
0.5	56.41	54.00	26.39	26.45	26.42	92.87	27.58
1	56.41	55.03	30.93	31.21	31.07	95.90	23.96
5	56.41	56.01	47.58	47.60	47.59	98.82	8.43

Appendix

Table 8: RL on MATH-500: four-training-seed mean response accuracy (%). The initial model is Qwen3-1.7B-Base. All models are evaluated after 1,000 updates. Recovery is a percentage and

\Delta

is in percentage points.

Checkpoint	$q=0.5\%$	$q=1\%$	$q=5\%$
Initial (step 0)	37.12	35.66	37.84
Step 5	43.26	41.05	41.90
Step 10	47.64	45.63	47.64
Step 15	47.75	45.80	48.25
Step 20	49.98	47.86	50.11
Random	0.50	1.00	5.00

Appendix

Table 9: OPD Fisher–gradient support recall (%), supplementing Section 3.2 . At each policy, recall is the fraction of the top-

q\%

gradient-magnitude coordinates captured by the top-

q\%

Fisher coordinates. Checkpoint rows are equal-weight means over full-training seeds 0 and 1; the shared initial policy is measured once. Random gives the analytic expectation for a uniform subset of the same size.

$q$ (%)	Metric	Full	Fisher	R0	R1	Random	Recovery	$\Delta$
0.5	Avg@8	26.67	25.10	14.69	15.42	15.05	89.49	10.05
0.5	pass@8	47.50	52.50	33.33	30.83	32.08	156.25	20.42
1	Avg@8	26.67	26.15	15.00	16.46	15.73	99.47	10.42
1	pass@8	47.50	55.00	29.17	38.33	33.75	185.42	21.25
5	Avg@8	26.67	26.25	24.27	23.33	23.80	106.59	2.45
5	pass@8	47.50	50.83	49.17	45.83	47.50	147.92	3.33

Appendix

Table 10: OPD on AIME24: means over four training seeds after 20 updates. Scores and Recovery are percentages;

\Delta

is in percentage points. Both metrics use the same eight responses per problem.

Explore similar work

On the Geometry of On-Policy Distillation
Jun 5, 2026Zhennan Shen, Yanshu Li, Qingyu Yin +6Language Model Distillation On-Policy Distillation
1HKUST · 2Brown University · 3Zhejiang University +5
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-Tuning On-Policy Distillation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group
On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training
Sep 29, 2026Shufan Shen, Zhongni Hou, Junshu Sun +5LLM Post-Training Fine-Tuning
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy Sciences · Meituan

Jun 5, 2026, cs.LGJ/K move · Enter open · S save

On the Geometry of On-Policy Distillation

Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong+5

1HKUST · 2Brown University · 3Zhejiang University+5

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.

Abstract

Explore similar work

On the Geometry of On-Policy Distillation

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training