HyperRepair: Recovering Forgotten General Capabilities of Fine-Tuned Language Models

Junrong Tang; Jianghui Wang; Hao Kong; Yang Lin; Yibo Yang

cs.CLOct 30, 2026

HyperRepair: Recovering Forgotten General Capabilities of Fine-Tuned Language Models

Authors: Junrong Tang, Jianghui Wang, Hao Kong, Yang Lin, Yibo Yang

Organizations: Shanghai Jiao Tong University · The University of Queensland

Abstract

Fine-tuning improves a language model's performance in a target domain but can weaken its general capabilities. Existing methods often face a trade-off between general capabilities and domain specialization, making it difficult to preserve both. We introduce HyperRepair, a framework for recovering general capabilities while retaining domain specialization at only inference cost. To make recovery generalizable across models sharing a pretrained backbone, HyperRepair learns a hypernetwork from repairs trained on source models. The hypernetwork uses activation fingerprints that summarize changes from the pretrained backbone to generate a tailored Repairing Side Net (RSN), whose prompt-conditioned gate applies corrections selectively. This enables recovery on models from unseen adaptation domains without target-domain examples or additional optimization. We evaluate HyperRpair across two model families, 12 LoRA adaptation datasets, and 15 general benchmarks. On the default split, HyperRepair achieves 93.4% of the base models' average general score while retaining 99.5% of the fine-tuned models' average domain score.

Figures & tables

Figure 1: Comparison of prior approaches and HyperRepair. (a) Training-time mitigation requires control over fine-tuning. (b) Post-hoc weight modification may trade domain specialization for general recovery. (c) HyperRepair selectively corrects a frozen model with a gated RSN to recover general capabilities while retaining domain specialization, without target-domain data or optimization at deployment.

Figure 2: Overview of HyperRepair. (1) Domain fine-tuning produces source models, whose activations on general prompts are used to construct fingerprints. (2) The gated RSN is trained for each source model while keeping its backbone frozen. (3) A HyperNet learns to generate RSN parameters from activation fingerprints using weight, representation-alignment, and functional supervision. (4) For a fine-tuned model from an unseen adaptation domain, the frozen HyperNet generates RSN parameters without further optimization or target-domain data. The gate controls whether the repair is applied.

Figure 3: Architectures of the RSN and HyperNet. (A) The RSN processes intermediate states from the frozen backbone. The prompt-conditioned gate $\lambda(x)$ scales its final output before addition to the backbone state. The backbone and language-model head remain frozen throughout repair training. (B) HyperNet encodes layerwise activation shifts with a Transformer and pools the encodings into a global representation. It adds generated local and global updates to a trainable shared template.

Method	ARC-Challenge	HellaSwag	MMLU	OpenBookQA	PIQA	SciQ	WinoGrande	ANLI-R1	SocialIQA	LogiQA	WiC	COPA	MATH	HumanEval	MBPP
Qwen2.5-7B
Base	51.4	78.8	71.8	47.4	79.8	95.2	73.1	53.0	54.7	35.8	57.8	91	39.3	56.1	63.8
LEDGAR-Top10
Fine-tuned	53.8	73.1	55.0	46.0	79.4	88.1	70.8	48.5	49.0	33.6	50.0	88	3.6	22.0	28.8
CorDA-KPM	45.7	65.7	24.0	44.2	75.7	78.1	69.6	33.0	46.8	27.0	50.0	90	0.0	4.3	0.0
FAPM	55.4	75.0	63.1	48.0	80.5	91.3	71.9	49.0	51.1	33.3	50.0	89	28.5	34.8	44.0

Table 1: General benchmark test scores for Qwen2.5-7B and Llama-3.1-8B on Split D. Base is shared across held-out domains within each backbone. HyperRepair uses RSNs generated by a HyperNet trained on the other nine domains.

Figure 4: Domain test performance across the 12 held-out adaptation domains. HyperRepair generally retains the scores of the fine-tuned models on both backbones.

Figure 5: Ablation study on Qwen2.5-7B across the three held-out domains in Split D. Values are changes in general and domain scores relative to the full model, measured in score points.

Base (correct)	Fine-tuned (incorrect)	HyperRepair (correct)
ARC-Challenge (discriminative) Adaptation: HateXplain (Split A)
Prompt. An astronomer observes that a planet rotates faster after a meteorite impact. Which is the most likely effect of this increase in rotation?
Planetary days will become shorter.	Planetary gravity will become stronger.	Planetary days will become shorter.
MATH (generative) Adaptation: MASSIVE-en (Split B)
Prompt. Determine the number of ways to arrange the letters of the word ELLIPSE.
The word ELLIPSE has 7 letters, with 2 L’s and 2 E’s. The number of ways to arrange the letters is $\frac{7!}{2!2!}=\boxed{1260}.$	The word ELLIPSE has 7 letters, but the letter E appears twice. Therefore, the number of ways to arrange the letters is $\frac{7!}{2!}=\boxed{2520}.$	The word ELLIPSE has 7 letters, with 2 L’s and 2 E’s. The number of ways to arrange the letters is $\frac{7!}{2!2!}=\boxed{1260}$ .

Table 2: Recovery examples on Qwen2.5-7B for ARC-Challenge and MATH. HyperRepair restores the correct outputs after fine-tuning errors.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Repairing Side Net (RSN)
Hidden dimension / layers	192 / 6
Attention heads / feedforward dimension	4 / 768
Backbone layers (Qwen / Llama)	5, 10, 15, 19, 24, 28 / 6, 11, 17, 22, 27, 32
Prompt gate	Mean pooling, MLP (hidden dimension 384), sigmoid
Ladder gate	Learned per layer; sigmoid temperature 0.1

Appendix

Table 3: Architecture, training, and inference settings of HyperRepair.

Dataset	Task	Train	Test	Metric
HateXplain ( Mathew et al., 2021 )	Hate-speech detection	4,000	1,000	Macro-F1
SemEval-2010 T8 ( Hendrickx et al., 2010 )	Relation classification	4,000	2,717	Macro-F1
SCOTUS ( Chalkidis et al., 2022 )	Legal issue classification	5,000	1,400	Macro-F1
MedMCQA ( Pal et al., 2022 )	Medical QA	4,000	1,000	Accuracy
MASSIVE-en ( FitzGerald et al., 2023 ; Bastianelli et al., 2020 )	Intent recognition	4,000	1,000	Accuracy
WikiSQL ( Zhong et al., 2017 )	Text-to-SQL	4,000	1,000	Execution accuracy

Appendix

Table 4: The 12 adaptation datasets.

Dataset	Category	Train	Test
Discriminative tasks
ARC-Challenge ( Clark et al., 2018 )	Science QA	889	1,172
HellaSwag ( Zellers et al., 2019 )	Commonsense reasoning	1,970	10,042
MMLU ( Hendrycks et al., 2021a )	Multitask knowledge	21,752	14,042
OpenBookQA ( Mihaylov et al., 2018 )	Science QA	1,970	500
PIQA ( Bisk et al., 2020 )	Physical reasoning	1,970	1,838

Appendix

Table 5: The 15 general benchmarks.

Split	Domain 1	Domain 2	Domain 3
A	HateXplain	SemEval-2010 T8	SCOTUS
B	MedMCQA	MASSIVE-en	WikiSQL
C	RACE	Banking77	CodeXGLUE-JS
D	LEDGAR-Top10	SNIPS-7	DROP

Appendix

Table 6: Held-out adaptation domains in each split.

Base (correct)	Fine-tuned (incorrect)	HyperRepair (correct)
HellaSwag Adaptation: CodeXGLUE-JS (Split C)
Prompt. Clean and jerk: A woman in a black shirt is bent over. She
picks up a heavy weight.	is putting lotion onto her back.	picks up a heavy weight.
MMLU Adaptation: SCOTUS (Split A)
Prompt. A wave transfers
energy	amplitude	energy

Appendix

Table 7: Recovery examples on 13 additional benchmarks, showing abbreviated prompts, selected answers, and verbatim response excerpts.

Fine-Tuning Language Models

See all 63 papers

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation
Aug 31, 2026Kwangmin Ki, Yunhun Nam, Jongheon Jeong +1LLM Fine-Tuning Domain Adaptation for LLMs
Korea University · Yonsei University
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
May 19, 2026Aarash Abro, Muhammad TahirLLM Fine-Tuning Spectral Methods
Zeta Labs · Lahore University of Management Sciences
HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models
Jul 13, 2026Aznaur Aliev, Carlos Hinojosa, Abdelrahman Eldesokey +3LLM Fine-Tuning LLM Safety Alignment
King Abdullah University of Science and Technology, Saudi Arabia

Aug 31, 2026J/K move · Enter open · S save

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Kwangmin Ki, Yunhun Nam, Jongheon Jeong, Jaehyung Kim

Korea University · Yonsei University

LLM Fine-Tuning Domain Adaptation for LLMs LLM Routing Catastrophic Forgetting

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.

Abstract

Fine-Tuning Language Models

Explore similar work

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models