Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

Junning Shao; Siwei Wang; Zhixuan Fang

cs.CLSep 28, 2026

Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

Authors: Junning Shao, Siwei Wang, Zhixuan Fang

Organizations: Tsinghua University, Beijing, China · Microsoft Research Asia, Beijing, China · Shanghai Qi Zhi Institute, Shanghai, China

Abstract

In recent years, the performance of large language models (LLMs) on reasoning tasks has been remarkable, even surpassing human capabilities on various benchmarks. However, there remains a lack of clear understanding in the academic community regarding how the structure and internal parameters of LLMs progressively solve complex reasoning problems. In this study, we investigate the inference process of LLMs on cross-linguistic materials and propose the hypothesis that LLM layers exhibit a structured division of labor across conceptualization, reasoning, and textualization. Based on this hypothesis, we introduce a bottleneck identification mechanism using sensitivity analysis to pinpoint the most critical functional stage for a specific task. Leveraging this insight, we propose a novel approach, Layer-Informed Fine-Tuning (LIFT), which achieves efficient and effective fine-tuning by selectively updating only these functionally critical layers. We then conduct extensive experiments to show that the LIFT method not only accelerates the training process but also significantly improves model performance.

Figures & tables

Figure 1: The upper part of the figure illustrates how an LLM processes a simple question under the Three-Stage Functional Segmentation hypothesis. The lower part presents a toy experiment validating this hypothesis by comparing hidden state similarities across conceptualization, reasoning, and textualization layers. We have also added a curve showing the cosine similarity between adjacent layers, with higher values indicating greater redundancy in those layers. In the figure, Q2 and O2 are the Chinese expressions of Q1 and O1, respectively.

Panel of Figure 2 — Figure 2: Cosine similarity between Hidden States in the Same Layer.

Panel of Figure 3 — Figure 3: Cosine Similarity between Hidden Stated in Adjacent Layers

Llama-3-8B-Instruct (Drop)
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	44.04% ( $\pm$ 2.03%)	42.44% ( $\pm$ 1.70%)	44.95% ( $\pm$ 1.54%)	44.18% ( $\pm$ 1.89%)
FOLIO	52.88% ( $\pm$ 0.97%)	51.33% ( $\pm$ 0.83%)	52.80% ( $\pm$ 1.36%)	52.48% ( $\pm$ 1.17%)
LogicalDeduction	47.48% ( $\pm$ 0.76%)	43.18% ( $\pm$ 1.45%)	48.86% ( $\pm$ 1.43%)	48.58% ( $\pm$ 0.89%)
gsm8k	60.30% ( $\pm$ 1.73%)	58.95% ( $\pm$ 2.48%)	61.96% ( $\pm$ 1.00%)	61.39% ( $\pm$ 1.50%)

Table 1: Performance of quantization probing (Drop and Recovery) over Various Layer Ranges of Llama-3-8B-Instruct (Averaged over 8 Seeds).

	Llama-3-8B-Instruct				Qwen2-7B-Instruct
ProofWriter	Base model	FullFT	AdaLoRA	LIFT[2,20)	Base model	FullFT	AdaLoRA	LIFT[3,26)
Accuracy	45.67%	56.12% (±2.02%)	47.83% (±0.49%)	57.67% (±0.36%)	48.00%	53.17% (±3.10%)	48.25% (±1.68%)	53.46% (±2.36%)
Training Time(s)	-	1383	1923	1215	-	1288	1839	1180
FOLIO	Base model	FullFT	AdaLoRA	LIFT[2,20)	Base model	FullFT	AdaLoRA	LIFT[3,26)
Accuracy	55.39%	52.08% (±1.67%)	57.11% (±2.21%)	58.21% (±2.35%)	34.31%	44.98% (±0.74%)	18.01% (±1.57%)	44.73% (±1.98%)
Training Time(s)	-	332	516	290	-	310	496	284

Table 2: Performance and Training Time Comparison of Fine-Tuning Methods over 4 Seeds

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Layers	Model	Layers
Llama-3-8B-Instruct	32	Qwen2-7B-Instruct	28

Appendix

Table 3: Number of layers in each LLM

Panel of Figure 4 — Figure 4: Performance of Llama-3-8B-Instruct across languages and datasets.

Panel of Figure 5 — Figure 5: Performance of Qwen2-7B-Instruct across languages and datasets.

Panel of Figure 6 — Figure 6: Cosine Similarity between Hidden Stated in Adjacent Layers of Llama-3-8B-Instruct

Panel of Figure 7 — Figure 7: Cosine Similarity between Hidden Stated in Adjacent Layers of Qwen2-7B-Instruct

Llama-3-8B-Instruct
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	[0,2)	[2,20)	[20,30)	[30,32)
FOLIO	[0,2)	[2,20)	[20,30)	[30,32)
LogicalDeduction	[0,2)	[2,22)	[22,30)	[30,32)
GSM8K	[0,2)	[2,22)	[22,30)	[30,32)

Appendix

Table 4: Functional Layer Segmentation of Llama-3-8B-Instruct

Qwen2-7B-Instruct
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	[0,3)	[3,26)	-	[26,28)
FOLIO	[0,3)	[3,26)	-	[26,28)
LogicalDeduction	[0,3)	[3,24)	[24,26)	[26,28)
GSM8K	[0,3)	[3,26)	-	[26,28)

Appendix

Table 5: Functional Layer Segmentation of Qwen2-7B-Instruct

Qwen2-7B-Instruct (Drop)
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	48.45% ( $\pm$ 1.51%)	43.15% ( $\pm$ 2.05%)	-	46.68% ( $\pm$ 1.83%)
FOLIO	42.18% ( $\pm$ 1.45%)	41.71% ( $\pm$ 3.67%)	-	42.16% ( $\pm$ 1.17%)
LogicalDeduction	50.50% ( $\pm$ 1.54%)	49.51% ( $\pm$ 0.86%)	50.31% ( $\pm$ 1.56%)	49.56% ( $\pm$ 0.78%)
gsm8k	84.16% ( $\pm$ 0.78%)	82.79% ( $\pm$ 0.54%)		84.03% ( $\pm$ 0.76%)

Appendix

Table 6: Performance of quantization probing (Drop and Recovery) over Various Layer Ranges of Qwen2-7B-Instruct (Averaged over 8 Seeds)

Llama-3-8B-Instruct
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	37.25% (±1.04%)	57.67% (±0.36%)	48.08% (±0.75%)	46.38% (±0.89%)
FOLIO	53.43% (±1.65%)	58.21% (±2.35%)	56.74% (±1.29%)	55.51% (±2.31%)
LogicalDeduction	45.67% (±0.72%)	53.58% (±2.47%)	46.83% (±1.67%)	45.25% (±0.88%)
GSM8K	73.96% (±0.98%)	67.87% (±0.64%)	65.22% (±0.53%)	56.48% (±0.58%)

Appendix

Table 7: Performance of Fine-Tuning over Various Layer Ranges of Llama-3-8B-Instruct

Qwen2-7B-Instruct
Datasets	conceptualization layers	active reasoning layers	idle reasoning layers	textualization layers
ProofWriter	45.50% (±0.93%)	53.46% (±2.36%)	-	47.46% (±0.85%)
FOLIO	27.08% (±4.19%)	44.73% (±1.98%)	-	28.55% (±1.62%)
LogicalDeduction	48.08% (±0.83%)	57.08% (±3.29%)	49.00% (±0.47%)	52.58% (±2.15%)
GSM8K	75.34% (±0.66%)	75.78% (±0.24%)	-	72.29% (±0.72%)

Appendix

Table 8: Performance of Fine-Tuning over Various Layer Ranges of Qwen2-7B-Instruct

Panel of Figure 8 — Figure 8: T-SNE visualizations for layers 1-32 of Llama-3-8B-Instruct on the GSM8K dataset.

Panel of Figure 9 — Figure 9: T-SNE visualizations for layers 1-32 of Llama-3-8B-Instruct on the FOLIO dataset.

Panel of Figure 10 — Figure 10: T-SNE visualizations for layers 1-32 of Llama-3-8B-Instruct on the LogicalDeduction dataset.

Panel of Figure 11 — Figure 11: T-SNE visualizations for layers 1-32 of Llama-3-8B-Instruct on the ProofWriter dataset.

Panel of Figure 12 — Figure 12: PCA visualizations for layers 1-32 of Llama-3-8B-Instruct on the GSM8K dataset.

Panel of Figure 13 — Figure 13: PCA visualizations for layers 1-32 of Llama-3-8B-Instruct on the FOLIO dataset.

Panel of Figure 14 — Figure 14: PCA visualizations for layers 1-32 of Llama-3-8B-Instruct on the LogicalDeduction dataset.

Panel of Figure 15 — Figure 15: PCA visualizations for layers 1-32 of Llama-3-8B-Instruct on the ProofWriter dataset.

Panel of Figure 16 — Figure 16: UMAP visualizations for layers 1-32 of Llama-3-8B-Instruct on the GSM8K dataset.

Panel of Figure 17 — Figure 17: UMAP visualizations for layers 1-32 of Llama-3-8B-Instruct on the FOLIO dataset.

Panel of Figure 18 — Figure 18: UMAP visualizations for layers 1-32 of Llama-3-8B-Instruct on the LogicalDeduction dataset.

Panel of Figure 19 — Figure 19: UMAP visualizations for layers 1-32 of Llama-3-8B-Instruct on the ProofWriter dataset.

Panel of Figure 20 — Figure 20: T-SNE visualizations for layers 1-28 of Qwen2-7B-Instruct on the GSM8K dataset.

Panel of Figure 21 — Figure 21: T-SNE visualizations for layers 1-28 of Qwen2-7B-Instruct on the FOLIO dataset.

Panel of Figure 22 — Figure 22: T-SNE visualizations for layers 1-28 of Qwen2-7B-Instruct on the LogicalDeduction dataset.

Panel of Figure 23 — Figure 23: T-SNE visualizations for layers 1-28 of Qwen2-7B-Instruct on the ProofWriter dataset.

Panel of Figure 24 — Figure 24: PCA visualizations for layers 1-28 of Qwen2-7B-Instruct on the GSM8K dataset.

Panel of Figure 25 — Figure 25: PCA visualizations for layers 1-28 of Qwen2-7B-Instruct on the FOLIO dataset.

Panel of Figure 26 — Figure 26: PCA visualizations for layers 1-28 of Qwen2-7B-Instruct on the LogicalDeduction dataset.

Panel of Figure 27 — Figure 27: PCA visualizations for layers 1-28 of Qwen2-7B-Instruct on the ProofWriter dataset.

Panel of Figure 28 — Figure 28: UMAP visualizations for layers 1-28 of Qwen2-7B-Instruct on the GSM8K dataset.

Panel of Figure 29 — Figure 29: UMAP visualizations for layers 1-28 of Qwen2-7B-Instruct on the FOLIO dataset.

Panel of Figure 30 — Figure 30: UMAP visualizations for layers 1-28 of Qwen2-7B-Instruct on the LogicalDeduction dataset.

Panel of Figure 31 — Figure 31: UMAP visualizations for layers 1-28 of Qwen2-7B-Instruct on the ProofWriter dataset.

Explore similar work

Integrated and Cross-Architecture Interpretation of LLM Reasoning
May 27, 2026Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu YangTransformer Interpretability LLM Interpretability
Tsinghua Shenzhen International Graduate School · Harbin Institute of Technology, Shenzhen
Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
Jun 15, 2026Xiandong Zou, Jing Huang, Jianshu Li +1RL for Language Model Reasoning Efficient Language Model Reasoning
Singapore Management University · Ant Group
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China

May 27, 2026, cs.CLJ/K move · Enter open · S save

Integrated and Cross-Architecture Interpretation of LLM Reasoning

Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu Yang

Tsinghua Shenzhen International Graduate School · Harbin Institute of Technology, Shenzhen

Transformer Interpretability LLM Interpretability LLM Reasoning

Open paper AskPDF arXiv

Understanding how LLMs reason is hindered by a practical asymmetry: while their generated outputs are observable, the underlying reasoning patterns remain opaque. Relying on single probes, such as Mutual Information Peak (MIP) or Deep-Thinking Ratio (DTR), risks underestimating the genuine inferential structure. To response this deficiency, we present an Integrated, cross-Architecture Reasoning (IAR) framework, designed to provide a unified approach to LLM reasoning interpretability. Specifically, we first propose to use bandwidth-calibrated MIP coupled with Tukey IQR peak-detection to isolate reasoning-crucial tokens at the output layer. Second, we performed an overlap analysis between MIP-picked tokens and DTR-deep tokens to trace the cross-layer trajectories of those tokens. This also discloses whether reasoning-crucial tokens are computation-intensive as well, further facilitating to understand how reasoning patterns evolve across model layers. Finally, we apply a Jaccard stability metric over multi-domain problems to verify if the MIP-identified tokens are reasoning quality-guaranteed. Extensive experiments on six models across four domains (mathematics, code, logic, and common sense) demonstrate IAR's generalizable interpretation capabilities across architectures. Our code is available at https://github.com/LeonardoMatthew/IAR.

Abstract

Explore similar work

Integrated and Cross-Architecture Interpretation of LLM Reasoning

Latent Thought Flow: Efficient Latent Reasoning in Large Language Models

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains