COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Kavinda Kehelella; Sanghamitra Dutta

cs.CLOct 9, 2026

COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Authors: Kavinda Kehelella, Sanghamitra Dutta

Organizations: University of Maryland, College Park

Abstract

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.

Figures & tables

Figure 1: Counterfactual-guided few-shot distillation framework. We begin with (1) a few-shot factual QA dataset $\mathcal{D}_{f}$ and (2) generate a counterfactual candidate for a factual sample. (3) The generated counterfactual is evaluated using the teacher model with (4) our proposed Context Following Difficulty (CFD) metric and accepted if it satisfies the CFD threshold $\tau$ . (5) The CFD-validated counterfactuals are then paired with their corresponding factuals and used to distill the student model.

Panel of Figure 2 — Figure 2: Counterfactual context generation prompts. System and user prompts used to minimally modify the factual context $C$ into a counterfactual context $C^{\prime}$ that supports the target answer $A^{\prime}$ .

Figure 3: Visualizing the Hausdorff distance between

\mathcal{M}_{t}

and

\mathcal{M}_{s}

Figure 4: Illustrative examples of CFD-based counterfactual evaluation from the NQ dataset. The four cases demonstrate how Answer Agreement (AA) and Contextual Loss Ratio (CLR) complement each other in evaluating generated counterfactual candidates.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	61.83 ${}_{\scalebox{0.6}{0.49}}$	61.44 ${}_{\scalebox{0.6}{2.08}}$	60.34 ${}_{\scalebox{0.6}{1.91}}$	56.91 ${}_{\scalebox{0.6}{3.20}}$	58.07 ${}_{\scalebox{0.6}{1.33}}$	59.72 ${}_{\scalebox{0.6}{2.15}}$	66.91 ${}_{\scalebox{0.6}{0.44}}$	66.01 ${}_{\scalebox{0.6}{0.59}}$	65.71 ${}_{\scalebox{0.6}{0.39}}$	64.68 ${}_{\scalebox{0.6}{0.61}}$	65.66 ${}_{\scalebox{0.6}{0.69}}$	65.79 ${}_{\scalebox{0.6}{0.80}}$
	+ CofFeQA	62.16 ${}_{\scalebox{0.6}{0.27}}$	62.09 ${}_{\scalebox{0.6}{1.30}}$	60.85 ${}_{\scalebox{0.6}{1.37}}$	59.08 ${}_{\scalebox{0.6}{2.01}}$	58.63 ${}_{\scalebox{0.6}{2.30}}$	60.56 ${}_{\scalebox{0.6}{1.65}}$	66.27 ${}_{\scalebox{0.6}{0.27}}$	68.10 ${}_{\scalebox{0.6}{0.20}}$	67.23 ${}_{\scalebox{0.6}{0.32}}$	67.25 ${}_{\scalebox{0.6}{0.73}}$	67.19 ${}_{\scalebox{0.6}{0.99}}$	67.21 ${}_{\scalebox{0.6}{0.65}}$
	LWD	61.81 ${}_{\scalebox{0.6}{0.56}}$	61.23 ${}_{\scalebox{0.6}{2.49}}$	60.37 ${}_{\scalebox{0.6}{1.56}}$	58.16 ${}_{\scalebox{0.6}{1.81}}$	58.72 ${}_{\scalebox{0.6}{2.29}}$	60.06 ${}_{\scalebox{0.6}{1.57}}$	65.62 ${}_{\scalebox{0.6}{5.03}}$	66.32 ${}_{\scalebox{0.6}{0.94}}$	65.90 ${}_{\scalebox{0.6}{1.18}}$	65.61 ${}_{\scalebox{0.6}{0.85}}$	66.62 ${}_{\scalebox{0.6}{0.86}}$	66.01 ${}_{\scalebox{0.6}{0.45}}$
	+ CofFeQA	62.12 ${}_{\scalebox{0.6}{0.22}}$	61.46 ${}_{\scalebox{0.6}{2.57}}$	60.04 ${}_{\scalebox{0.6}{2.09}}$	60.03 ${}_{\scalebox{0.6}{1.54}}$	58.93 ${}_{\scalebox{0.6}{2.71}}$	60.52 ${}_{\scalebox{0.6}{1.27}}$	63.98 ${}_{\scalebox{0.6}{4.96}}$	68.16 ${}_{\scalebox{0.6}{0.35}}$	67.78 ${}_{\scalebox{0.6}{0.64}}$	68.07 ${}_{\scalebox{0.6}{0.56}}$	67.96 ${}_{\scalebox{0.6}{0.77}}$	67.19 ${}_{\scalebox{0.6}{1.80}}$
	SeqKD	60.68 ${}_{\scalebox{0.6}{1.24}}$	59.19 ${}_{\scalebox{0.6}{1.07}}$	60.80 ${}_{\scalebox{0.6}{1.12}}$	59.78 ${}_{\scalebox{0.6}{1.57}}$	62.09 ${}_{\scalebox{0.6}{1.54}}$	60.51 ${}_{\scalebox{0.6}{1.11}}$	65.89 ${}_{\scalebox{0.6}{0.34}}$	67.55 ${}_{\scalebox{0.6}{0.55}}$	66.84 ${}_{\scalebox{0.6}{0.31}}$	67.33 ${}_{\scalebox{0.6}{0.35}}$	67.73 ${}_{\scalebox{0.6}{0.76}}$	67.07 ${}_{\scalebox{0.6}{0.74}}$

Table 1: In-domain F1 score comparison across

k

-shots for T5 and Qwen2.5 models using baselines. Mean F1 scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. baseline). Blue shading highlights aggregate results.

Figure 5: Teacher-student F1 agreement of standard KD and CofFeQA across datasets and

k

-shot settings using T5. CofFeQA generally achieves higher agreement than the baseline. Spokes denote

k

-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
DROP	KD	38.56 ${}_{\scalebox{0.6}{1.72}}$	35.81 ${}_{\scalebox{0.6}{3.76}}$	32.48 ${}_{\scalebox{0.6}{2.47}}$	28.06 ${}_{\scalebox{0.6}{3.65}}$	27.78 ${}_{\scalebox{0.6}{1.41}}$	32.54 ${}_{\scalebox{0.6}{4.73}}$	49.58 ${}_{\scalebox{0.6}{1.13}}$	44.92 ${}_{\scalebox{0.6}{1.31}}$	42.07 ${}_{\scalebox{0.6}{0.45}}$	41.28 ${}_{\scalebox{0.6}{0.67}}$	42.57 ${}_{\scalebox{0.6}{0.94}}$	44.08 ${}_{\scalebox{0.6}{3.36}}$
DROP	+ CofFeQA	39.24 ${}_{\scalebox{0.6}{1.14}}$	37.54 ${}_{\scalebox{0.6}{1.20}}$	32.41 ${}_{\scalebox{0.6}{2.71}}$	32.29 ${}_{\scalebox{0.6}{2.00}}$	29.44 ${}_{\scalebox{0.6}{1.53}}$	34.19 ${}_{\scalebox{0.6}{4.06}}$	48.58 ${}_{\scalebox{0.6}{0.79}}$	49.53 ${}_{\scalebox{0.6}{1.23}}$	45.20 ${}_{\scalebox{0.6}{1.31}}$	42.65 ${}_{\scalebox{0.6}{1.43}}$	41.36 ${}_{\scalebox{0.6}{0.60}}$	45.46 ${}_{\scalebox{0.6}{3.58}}$
DuoRC	KD	50.38 ${}_{\scalebox{0.6}{1.23}}$	48.49 ${}_{\scalebox{0.6}{3.56}}$	46.98 ${}_{\scalebox{0.6}{2.07}}$	42.52 ${}_{\scalebox{0.6}{1.73}}$	44.46 ${}_{\scalebox{0.6}{1.80}}$	46.57 ${}_{\scalebox{0.6}{3.13}}$	41.73 ${}_{\scalebox{0.6}{0.77}}$	44.84 ${}_{\scalebox{0.6}{1.29}}$	43.38 ${}_{\scalebox{0.6}{1.30}}$	42.77 ${}_{\scalebox{0.6}{1.16}}$	42.71 ${}_{\scalebox{0.6}{1.49}}$	43.08 ${}_{\scalebox{0.6}{1.15}}$
DuoRC	+ CofFeQA	51.66 ${}_{\scalebox{0.6}{0.56}}$	50.58 ${}_{\scalebox{0.6}{1.02}}$	44.55 ${}_{\scalebox{0.6}{2.64}}$	46.14 ${}_{\scalebox{0.6}{1.23}}$	44.72 ${}_{\scalebox{0.6}{1.51}}$	47.53 ${}_{\scalebox{0.6}{3.36}}$	41.04 ${}_{\scalebox{0.6}{1.11}}$	45.01 ${}_{\scalebox{0.6}{0.92}}$	43.11 ${}_{\scalebox{0.6}{0.98}}$	43.25 ${}_{\scalebox{0.6}{1.00}}$	44.39 ${}_{\scalebox{0.6}{1.42}}$	43.36 ${}_{\scalebox{0.6}{1.52}}$
RelationExtraction	KD	82.96 ${}_{\scalebox{0.6}{0.99}}$	83.30 ${}_{\scalebox{0.6}{0.46}}$	82.65 ${}_{\scalebox{0.6}{0.61}}$	78.30 ${}_{\scalebox{0.6}{2.66}}$	77.53 ${}_{\scalebox{0.6}{1.47}}$	80.95 ${}_{\scalebox{0.6}{2.79}}$	85.32 ${}_{\scalebox{0.6}{0.45}}$	85.34 ${}_{\scalebox{0.6}{0.49}}$	84.36 ${}_{\scalebox{0.6}{0.25}}$	82.77 ${}_{\scalebox{0.6}{0.71}}$	82.74 ${}_{\scalebox{0.6}{0.19}}$	84.11 ${}_{\scalebox{0.6}{1.30}}$

Table 2: Out-of-domain F1 score comparison across

k

-shots for T5 and Qwen2.5 models using KD. Mean F1 scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: KD performance as a function of the teacher prediction margin. Factual samples are grouped into three bins according to their average token-level prediction margins, with Bin 0 containing the smallest margin samples and Bin 2 containing the largest margin samples. Smaller margins are used as a proxy for greater proximity to the teacher’s decision boundary. Knowledge distillation is performed separately using 100 factual samples from each bin, maintaining the same distillation budget across all three settings. Panels (a) and (b) report EM and F1 averaged across the five in-domain datasets, respectively, while panels (c) and (d) report the corresponding dataset-wise results. Distillation using samples from the smallest margin bin generally results in lower performance than using samples from the larger margin bins.

Panel of Figure 7 — Figure 7: Prompts used for counterfactual generation. The generation process consists of entity type classification, alternative entity generation, and counterfactual context generation. The final stage uses a system prompt and a user prompt to minimally modify the original context to support the alternative answer.

Figure 8: Visualization of the distances between

\mathcal{M}_{t}

and

\mathcal{M}_{s}

c_{i}

and

c^{\prime}_{i}

are factual and counterfactual pairs with

\|c_{i}-c_{i}^{\prime}\|=\alpha^{\prime}\leq\alpha

t

is an arbitrary point in

\mathcal{M}_{t}

that finds some

t_{i}^{*}

cut point in

\mathcal{M}_{t}

within

\epsilon^{\prime}

distance (

\epsilon^{\prime}\leq\epsilon

	Teacher Dataset		Student Dataset
Dataset	Train	Val.	Train	Val.	Test
NQ	93,663	10,408	34,383	3,821	2,283
SQuAD	77,847	8,651	16,341	1,816	2,790
NewsQA	66,655	7,407	10,956	1,218	690
TriviaQA	55,519	6,170	4,415	491	607
HotpotQA	65,630	7,294	28,956	3,218	2,052

Appendix

Table 3: Statistics of teacher and student datasets used in our experiments. Teacher datasets contain the original training/validation splits, while student datasets contain available factual instances for distillation and evaluation.

Test Set Type	Model	Score	NQ	SQuAD	NewsQA	TriviaQA	HotpotQA
Factual	T5	EM	74.90	79.64	73.19	67.38	77.19
	T5	F1	77.76	87.78	78.97	69.27	85.49
	Qwen2.5	EM	79.46	83.91	75.65	73.48	79.34
	Qwen2.5	F1	82.83	91.04	82.50	76.37	88.52
Counterfactual	T5	EM	67.94	79.75	66.38	58.32	65.84
	T5	F1	70.79	87.04	71.35	60.58	71.37

Appendix

Table 4: Teacher model performance (EM and F1) on factual and counterfactual test sets across five in-domain QA benchmarks for T5 and Qwen2.5 models.

Panel of Figure 9 — Figure 9: Effect of CFD verification on CofFeQA . Performance gains over standard KD ( $\Delta=\mathrm{\textsc{CofFeQA}}-\mathrm{KD}$ ) are shown across different few-shot budgets $k$ . We compare CofFeQA without CFD verification against the full CofFeQA framework with CFD verification ( $\tau=0.3$ ). Panels (a) and (b) report accuracy EM and F1 gains, while panels (c) and (d) report teacher-student agreement EM and F1 gains. CFD verification generally yields larger improvements, demonstrating the benefit of validating generated counterfactuals before distillation.

Figure 10: CFD threshold ablation. Effect of the CFD threshold

\tau

on CofFeQA using T5 on TriviaQA. The bars show the improvement of CofFeQA over standard KD (

\Delta=\mathrm{\textsc{CofFeQA}}-\mathrm{KD}

), averaged across

k\in\{16,32,64,128,256\}

and five random seeds. The left plot shows the improvement in accuracy measured by EM and F1, while the right plot shows the improvement in teacher-student agreement measured by EM and F1.

Model	Duration/candidate (s)	CPU/candidate (kWh)	GPU/candidate (kWh)
T5	2.884	0.000083	0.000055
Qwen2.5	2.877	0.000071	0.000067

Appendix

Table 5: Average duration and CPU/GPU energy consumption per counterfactual generation candidate under the T5 and Qwen2.5 settings. Results are averaged across all generation attempts.

Objective	Method	$k$	EM (%)	Duration (s)	CPU (kWh)	GPU (kWh)
KD	Baseline	16	61.88	62.86	0.00201	0.01342
		32	60.82	50.34	0.00153	0.00980
		64	54.83	56.15	0.00170	0.01259
		128	56.07	75.90	0.00224	0.01662
		256	55.56	128.67	0.00378	0.02777
	CofFeQA	16	63.36	45.19	0.00143	0.01016

Appendix

Table 6: Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few-shot budgets under the T5 setting.

Objective	Method	$k$	EM (%)	Duration (s)	CPU (kWh)	GPU (kWh)
KD	Baseline	16	61.22	748.41	0.02378	0.11453
		32	60.24	636.82	0.01930	0.11645
		64	59.13	774.16	0.02278	0.15832
		128	58.51	1202.28	0.03660	0.27748
		256	59.07	1793.38	0.05119	0.35929
	CofFeQA	16	61.73	704.12	0.02244	0.11386

Appendix

Table 7: Comparison of EM score, distillation duration, and CPU/GPU energy consumption between baseline methods and COFFEQA across different few-shot budgets under the Qwen2.5.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	58.10 ${}_{\scalebox{0.6}{0.44}}$	57.73 ${}_{\scalebox{0.6}{3.52}}$	54.73 ${}_{\scalebox{0.6}{3.79}}$	47.84 ${}_{\scalebox{0.6}{6.31}}$	50.32 ${}_{\scalebox{0.6}{3.61}}$	53.75 ${}_{\scalebox{0.6}{4.54}}$	61.95 ${}_{\scalebox{0.6}{0.36}}$	61.24 ${}_{\scalebox{0.6}{0.61}}$	60.89 ${}_{\scalebox{0.6}{0.44}}$	60.18 ${}_{\scalebox{0.6}{0.53}}$	62.07 ${}_{\scalebox{0.6}{0.47}}$	61.27 ${}_{\scalebox{0.6}{0.78}}$
	+ CofFeQA	58.89 ${}_{\scalebox{0.6}{0.33}}$	58.49 ${}_{\scalebox{0.6}{1.50}}$	56.92 ${}_{\scalebox{0.6}{2.34}}$	49.86 ${}_{\scalebox{0.6}{6.34}}$	51.20 ${}_{\scalebox{0.6}{5.30}}$	55.07 ${}_{\scalebox{0.6}{4.24}}$	61.79 ${}_{\scalebox{0.6}{0.27}}$	63.12 ${}_{\scalebox{0.6}{0.35}}$	63.06 ${}_{\scalebox{0.6}{0.47}}$	62.66 ${}_{\scalebox{0.6}{0.82}}$	62.84 ${}_{\scalebox{0.6}{1.32}}$	62.69 ${}_{\scalebox{0.6}{0.54}}$
	LWD	58.20 ${}_{\scalebox{0.6}{0.48}}$	57.47 ${}_{\scalebox{0.6}{3.89}}$	54.80 ${}_{\scalebox{0.6}{3.04}}$	50.15 ${}_{\scalebox{0.6}{3.73}}$	50.90 ${}_{\scalebox{0.6}{5.43}}$	54.30 ${}_{\scalebox{0.6}{3.68}}$	60.19 ${}_{\scalebox{0.6}{6.36}}$	61.91 ${}_{\scalebox{0.6}{0.79}}$	61.10 ${}_{\scalebox{0.6}{1.33}}$	60.71 ${}_{\scalebox{0.6}{0.44}}$	61.82 ${}_{\scalebox{0.6}{1.03}}$	61.15 ${}_{\scalebox{0.6}{0.73}}$
	+ CofFeQA	58.84 ${}_{\scalebox{0.6}{0.27}}$	57.45 ${}_{\scalebox{0.6}{3.65}}$	55.00 ${}_{\scalebox{0.6}{4.16}}$	52.58 ${}_{\scalebox{0.6}{4.70}}$	51.16 ${}_{\scalebox{0.6}{6.81}}$	55.01 ${}_{\scalebox{0.6}{3.22}}$	58.95 ${}_{\scalebox{0.6}{6.05}}$	62.53 ${}_{\scalebox{0.6}{0.63}}$	63.49 ${}_{\scalebox{0.6}{0.56}}$	63.48 ${}_{\scalebox{0.6}{0.69}}$	63.39 ${}_{\scalebox{0.6}{1.46}}$	62.37 ${}_{\scalebox{0.6}{1.95}}$
	SeqKD	54.75 ${}_{\scalebox{0.6}{1.75}}$	54.47 ${}_{\scalebox{0.6}{1.07}}$	56.44 ${}_{\scalebox{0.6}{1.11}}$	55.85 ${}_{\scalebox{0.6}{1.57}}$	58.59 ${}_{\scalebox{0.6}{1.65}}$	56.02 ${}_{\scalebox{0.6}{1.64}}$	59.85 ${}_{\scalebox{0.6}{0.41}}$	61.60 ${}_{\scalebox{0.6}{0.61}}$	61.96 ${}_{\scalebox{0.6}{0.40}}$	62.37 ${}_{\scalebox{0.6}{0.58}}$	62.94 ${}_{\scalebox{0.6}{1.21}}$	61.74 ${}_{\scalebox{0.6}{1.17}}$

Appendix

Table 8: In-Domain EM score comparison across

k

-shots for T5 and Qwen2.5 models using baselines. Mean EM scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. baseline). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	56.23 ${}_{\scalebox{0.6}{4.87}}$	60.20 ${}_{\scalebox{0.6}{2.87}}$	54.47 ${}_{\scalebox{0.6}{6.56}}$	50.26 ${}_{\scalebox{0.6}{7.34}}$	52.62 ${}_{\scalebox{0.6}{5.00}}$	54.76 ${}_{\scalebox{0.6}{3.76}}$	61.88 ${}_{\scalebox{0.6}{0.68}}$	62.93 ${}_{\scalebox{0.6}{0.40}}$	62.84 ${}_{\scalebox{0.6}{0.62}}$	62.53 ${}_{\scalebox{0.6}{0.22}}$	63.73 ${}_{\scalebox{0.6}{0.83}}$	62.78 ${}_{\scalebox{0.6}{0.67}}$
	+ CofFeQA	61.34 ${}_{\scalebox{0.6}{0.56}}$	60.64 ${}_{\scalebox{0.6}{0.49}}$	57.97 ${}_{\scalebox{0.6}{1.36}}$	52.94 ${}_{\scalebox{0.6}{6.72}}$	53.46 ${}_{\scalebox{0.6}{4.92}}$	57.27 ${}_{\scalebox{0.6}{3.93}}$	61.46 ${}_{\scalebox{0.6}{0.53}}$	62.97 ${}_{\scalebox{0.6}{0.32}}$	64.52 ${}_{\scalebox{0.6}{0.13}}$	64.94 ${}_{\scalebox{0.6}{0.11}}$	64.72 ${}_{\scalebox{0.6}{1.17}}$	63.72 ${}_{\scalebox{0.6}{1.48}}$
	LWD	59.86 ${}_{\scalebox{0.6}{0.63}}$	58.94 ${}_{\scalebox{0.6}{4.18}}$	56.42 ${}_{\scalebox{0.6}{3.20}}$	51.35 ${}_{\scalebox{0.6}{4.46}}$	52.22 ${}_{\scalebox{0.6}{5.87}}$	55.76 ${}_{\scalebox{0.6}{3.85}}$	60.08 ${}_{\scalebox{0.6}{0.64}}$	63.05 ${}_{\scalebox{0.6}{0.76}}$	62.79 ${}_{\scalebox{0.6}{1.26}}$	62.50 ${}_{\scalebox{0.6}{0.77}}$	62.66 ${}_{\scalebox{0.6}{1.13}}$	62.21 ${}_{\scalebox{0.6}{1.21}}$
	+ CofFeQA	61.31 ${}_{\scalebox{0.6}{0.40}}$	58.95 ${}_{\scalebox{0.6}{3.37}}$	56.57 ${}_{\scalebox{0.6}{4.41}}$	53.98 ${}_{\scalebox{0.6}{4.65}}$	52.74 ${}_{\scalebox{0.6}{7.23}}$	56.71 ${}_{\scalebox{0.6}{3.52}}$	58.92 ${}_{\scalebox{0.6}{0.62}}$	63.87 ${}_{\scalebox{0.6}{0.70}}$	64.87 ${}_{\scalebox{0.6}{0.51}}$	65.51 ${}_{\scalebox{0.6}{0.56}}$	65.47 ${}_{\scalebox{0.6}{1.02}}$	63.73 ${}_{\scalebox{0.6}{2.77}}$
	SeqKD	57.52 ${}_{\scalebox{0.6}{1.96}}$	57.19 ${}_{\scalebox{0.6}{1.23}}$	59.24 ${}_{\scalebox{0.6}{1.76}}$	58.92 ${}_{\scalebox{0.6}{1.83}}$	62.41 ${}_{\scalebox{0.6}{1.97}}$	59.06 ${}_{\scalebox{0.6}{2.07}}$	61.45 ${}_{\scalebox{0.6}{0.52}}$	64.81 ${}_{\scalebox{0.6}{0.70}}$	65.01 ${}_{\scalebox{0.6}{0.38}}$	65.86 ${}_{\scalebox{0.6}{0.44}}$	66.30 ${}_{\scalebox{0.6}{1.21}}$	64.69 ${}_{\scalebox{0.6}{1.91}}$

Appendix

Table 9: In-domain teacher-student agreement EM score comparison across

k

-shots for T5 and Qwen2.5 models using baselines. Mean agreement EM scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. baseline). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	62.31 ${}_{\scalebox{0.6}{3.15}}$	64.50 ${}_{\scalebox{0.6}{1.70}}$	61.75 ${}_{\scalebox{0.6}{2.79}}$	60.11 ${}_{\scalebox{0.6}{3.06}}$	60.98 ${}_{\scalebox{0.6}{2.11}}$	61.93 ${}_{\scalebox{0.6}{1.66}}$	68.95 ${}_{\scalebox{0.6}{0.66}}$	69.19 ${}_{\scalebox{0.6}{0.31}}$	68.87 ${}_{\scalebox{0.6}{0.59}}$	68.30 ${}_{\scalebox{0.6}{0.55}}$	68.94 ${}_{\scalebox{0.6}{0.85}}$	68.85 ${}_{\scalebox{0.6}{0.33}}$
	+ CofFeQA	65.08 ${}_{\scalebox{0.6}{0.40}}$	64.91 ${}_{\scalebox{0.6}{0.42}}$	62.65 ${}_{\scalebox{0.6}{0.70}}$	61.96 ${}_{\scalebox{0.6}{2.63}}$	61.36 ${}_{\scalebox{0.6}{2.37}}$	63.19 ${}_{\scalebox{0.6}{1.71}}$	68.18 ${}_{\scalebox{0.6}{0.32}}$	70.29 ${}_{\scalebox{0.6}{0.31}}$	70.47 ${}_{\scalebox{0.6}{0.34}}$	70.63 ${}_{\scalebox{0.6}{0.28}}$	70.45 ${}_{\scalebox{0.6}{0.76}}$	70.01 ${}_{\scalebox{0.6}{1.03}}$
	LWD	64.22 ${}_{\scalebox{0.6}{0.62}}$	63.55 ${}_{\scalebox{0.6}{2.69}}$	62.55 ${}_{\scalebox{0.6}{1.45}}$	60.32 ${}_{\scalebox{0.6}{2.31}}$	61.12 ${}_{\scalebox{0.6}{2.68}}$	62.35 ${}_{\scalebox{0.6}{1.63}}$	67.54 ${}_{\scalebox{0.6}{0.52}}$	69.09 ${}_{\scalebox{0.6}{1.01}}$	69.07 ${}_{\scalebox{0.6}{1.11}}$	68.78 ${}_{\scalebox{0.6}{0.64}}$	69.19 ${}_{\scalebox{0.6}{0.52}}$	68.73 ${}_{\scalebox{0.6}{0.68}}$
	+ CofFeQA	65.11 ${}_{\scalebox{0.6}{0.37}}$	63.84 ${}_{\scalebox{0.6}{2.50}}$	62.65 ${}_{\scalebox{0.6}{2.03}}$	62.31 ${}_{\scalebox{0.6}{1.45}}$	61.58 ${}_{\scalebox{0.6}{3.01}}$	63.10 ${}_{\scalebox{0.6}{1.39}}$	66.10 ${}_{\scalebox{0.6}{0.51}}$	71.11 ${}_{\scalebox{0.6}{0.54}}$	70.93 ${}_{\scalebox{0.6}{0.65}}$	71.54 ${}_{\scalebox{0.6}{0.24}}$	71.35 ${}_{\scalebox{0.6}{0.52}}$	70.21 ${}_{\scalebox{0.6}{2.30}}$
	SeqKD	63.33 ${}_{\scalebox{0.6}{1.54}}$	62.02 ${}_{\scalebox{0.6}{1.16}}$	63.64 ${}_{\scalebox{0.6}{1.50}}$	62.60 ${}_{\scalebox{0.6}{1.83}}$	65.66 ${}_{\scalebox{0.6}{1.78}}$	63.45 ${}_{\scalebox{0.6}{1.39}}$	68.48 ${}_{\scalebox{0.6}{0.35}}$	71.63 ${}_{\scalebox{0.6}{0.55}}$	70.87 ${}_{\scalebox{0.6}{0.41}}$	71.58 ${}_{\scalebox{0.6}{0.11}}$	71.85 ${}_{\scalebox{0.6}{0.78}}$	70.88 ${}_{\scalebox{0.6}{1.39}}$

Appendix

Table 10: In-domain teacher-student agreement F1 score comparison across

k

-shots for T5 and Qwen2.5 models using baselines. Mean agreement F1 scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. baseline). Blue shading highlights aggregate results.

Figure 11: Teacher-student F1 agreement of KD, LWD, and SeqKD with and without CofFeQA across datasets and

k

-shot settings using T5. CofFeQA generally achieves higher agreement across the distillation baselines. Spokes denote

k

-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization.

Figure 12: Teacher-student F1 agreement of KD, LWD, and SeqKD with and without CofFeQA across datasets and

k

-shot settings using Qwen2.5. CofFeQA generally achieves higher agreement across the distillation baselines. Spokes denote

k

-shot settings, and radial axis F1 score ranges are adjusted independently by dataset for visualization.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
DROP	KD	34.75 ${}_{\scalebox{0.6}{1.78}}$	32.00 ${}_{\scalebox{0.6}{4.64}}$	27.56 ${}_{\scalebox{0.6}{3.50}}$	22.63 ${}_{\scalebox{0.6}{4.29}}$	23.80 ${}_{\scalebox{0.6}{1.97}}$	28.15 ${}_{\scalebox{0.6}{5.20}}$	44.73 ${}_{\scalebox{0.6}{0.78}}$	39.19 ${}_{\scalebox{0.6}{1.71}}$	36.93 ${}_{\scalebox{0.6}{0.33}}$	35.03 ${}_{\scalebox{0.6}{0.90}}$	36.57 ${}_{\scalebox{0.6}{1.04}}$	38.49 ${}_{\scalebox{0.6}{3.79}}$
DROP	+ CofFeQA	35.68 ${}_{\scalebox{0.6}{1.01}}$	33.86 ${}_{\scalebox{0.6}{1.47}}$	28.36 ${}_{\scalebox{0.6}{3.54}}$	25.98 ${}_{\scalebox{0.6}{3.56}}$	24.65 ${}_{\scalebox{0.6}{2.56}}$	29.71 ${}_{\scalebox{0.6}{4.85}}$	44.04 ${}_{\scalebox{0.6}{0.73}}$	44.32 ${}_{\scalebox{0.6}{1.35}}$	39.56 ${}_{\scalebox{0.6}{1.51}}$	37.58 ${}_{\scalebox{0.6}{1.03}}$	35.19 ${}_{\scalebox{0.6}{1.10}}$	40.14 ${}_{\scalebox{0.6}{4.00}}$
DuoRC	KD	44.63 ${}_{\scalebox{0.6}{1.56}}$	43.12 ${}_{\scalebox{0.6}{3.14}}$	41.47 ${}_{\scalebox{0.6}{1.91}}$	38.17 ${}_{\scalebox{0.6}{1.43}}$	39.33 ${}_{\scalebox{0.6}{1.84}}$	41.35 ${}_{\scalebox{0.6}{2.65}}$	33.99 ${}_{\scalebox{0.6}{0.80}}$	38.53 ${}_{\scalebox{0.6}{1.06}}$	36.66 ${}_{\scalebox{0.6}{1.60}}$	36.62 ${}_{\scalebox{0.6}{1.10}}$	36.62 ${}_{\scalebox{0.6}{1.80}}$	36.48 ${}_{\scalebox{0.6}{1.62}}$
DuoRC	+ CofFeQA	46.01 ${}_{\scalebox{0.6}{0.84}}$	44.63 ${}_{\scalebox{0.6}{0.87}}$	39.02 ${}_{\scalebox{0.6}{2.37}}$	40.94 ${}_{\scalebox{0.6}{1.17}}$	40.04 ${}_{\scalebox{0.6}{1.08}}$	42.13 ${}_{\scalebox{0.6}{3.03}}$	33.68 ${}_{\scalebox{0.6}{0.84}}$	37.55 ${}_{\scalebox{0.6}{1.39}}$	36.88 ${}_{\scalebox{0.6}{0.78}}$	36.70 ${}_{\scalebox{0.6}{0.87}}$	38.58 ${}_{\scalebox{0.6}{1.48}}$	36.68 ${}_{\scalebox{0.6}{1.83}}$
RelationExtraction	KD	74.71 ${}_{\scalebox{0.6}{1.74}}$	75.13 ${}_{\scalebox{0.6}{0.68}}$	74.49 ${}_{\scalebox{0.6}{1.19}}$	66.70 ${}_{\scalebox{0.6}{5.61}}$	66.65 ${}_{\scalebox{0.6}{3.71}}$	71.53 ${}_{\scalebox{0.6}{4.44}}$	78.38 ${}_{\scalebox{0.6}{0.36}}$	78.95 ${}_{\scalebox{0.6}{0.41}}$	78.00 ${}_{\scalebox{0.6}{0.17}}$	76.38 ${}_{\scalebox{0.6}{1.13}}$	76.88 ${}_{\scalebox{0.6}{0.52}}$	77.72 ${}_{\scalebox{0.6}{1.06}}$

Appendix

Table 11: Out-of-domain EM score comparison across

k

-shots for T5 and Qwen2.5 models using KD. Mean EM scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	54.90 ${}_{\scalebox{0.6}{4.68}}$	58.16 ${}_{\scalebox{0.6}{2.79}}$	53.18 ${}_{\scalebox{0.6}{5.88}}$	49.00 ${}_{\scalebox{0.6}{6.84}}$	51.18 ${}_{\scalebox{0.6}{4.85}}$	53.28 ${}_{\scalebox{0.6}{3.51}}$	52.10 ${}_{\scalebox{0.6}{0.64}}$	52.98 ${}_{\scalebox{0.6}{0.66}}$	52.90 ${}_{\scalebox{0.6}{0.36}}$	51.53 ${}_{\scalebox{0.6}{0.57}}$	53.04 ${}_{\scalebox{0.6}{0.68}}$	52.51 ${}_{\scalebox{0.6}{0.67}}$
NQ	+ CofFeQA	58.70 ${}_{\scalebox{0.6}{0.78}}$	59.33 ${}_{\scalebox{0.6}{0.62}}$	56.20 ${}_{\scalebox{0.6}{1.23}}$	51.83 ${}_{\scalebox{0.6}{6.47}}$	52.34 ${}_{\scalebox{0.6}{4.46}}$	55.68 ${}_{\scalebox{0.6}{3.49}}$	51.65 ${}_{\scalebox{0.6}{0.51}}$	53.90 ${}_{\scalebox{0.6}{1.04}}$	56.10 ${}_{\scalebox{0.6}{0.50}}$	56.73 ${}_{\scalebox{0.6}{0.33}}$	56.93 ${}_{\scalebox{0.6}{1.20}}$	55.06 ${}_{\scalebox{0.6}{2.26}}$
SQuAD	KD	78.14 ${}_{\scalebox{0.6}{0.84}}$	75.71 ${}_{\scalebox{0.6}{0.88}}$	74.52 ${}_{\scalebox{0.6}{2.49}}$	75.63 ${}_{\scalebox{0.6}{1.17}}$	73.43 ${}_{\scalebox{0.6}{5.21}}$	75.49 ${}_{\scalebox{0.6}{1.75}}$	60.37 ${}_{\scalebox{0.6}{0.35}}$	59.74 ${}_{\scalebox{0.6}{0.66}}$	59.18 ${}_{\scalebox{0.6}{0.35}}$	60.34 ${}_{\scalebox{0.6}{0.78}}$	60.98 ${}_{\scalebox{0.6}{0.41}}$	60.12 ${}_{\scalebox{0.6}{0.69}}$
SQuAD	+ CofFeQA	78.42 ${}_{\scalebox{0.6}{0.30}}$	78.49 ${}_{\scalebox{0.6}{0.37}}$	77.05 ${}_{\scalebox{0.6}{1.20}}$	76.98 ${}_{\scalebox{0.6}{0.61}}$	75.77 ${}_{\scalebox{0.6}{1.62}}$	77.34 ${}_{\scalebox{0.6}{1.14}}$	58.48 ${}_{\scalebox{0.6}{0.40}}$	60.80 ${}_{\scalebox{0.6}{0.38}}$	61.08 ${}_{\scalebox{0.6}{0.74}}$	62.57 ${}_{\scalebox{0.6}{1.35}}$	64.77 ${}_{\scalebox{0.6}{1.63}}$	61.54 ${}_{\scalebox{0.6}{2.32}}$
NewsQA	KD	64.29 ${}_{\scalebox{0.6}{2.04}}$	64.58 ${}_{\scalebox{0.6}{3.03}}$	63.19 ${}_{\scalebox{0.6}{2.52}}$	54.84 ${}_{\scalebox{0.6}{3.97}}$	50.49 ${}_{\scalebox{0.6}{0.94}}$	59.48 ${}_{\scalebox{0.6}{6.43}}$	39.54 ${}_{\scalebox{0.6}{1.29}}$	34.99 ${}_{\scalebox{0.6}{1.04}}$	34.99 ${}_{\scalebox{0.6}{1.38}}$	33.42 ${}_{\scalebox{0.6}{1.22}}$	32.90 ${}_{\scalebox{0.6}{2.06}}$	35.17 ${}_{\scalebox{0.6}{2.61}}$

Appendix

Table 12: EM score comparison on modified-context test sets across

k

-shot settings for T5 and Qwen2.5 models using KD. Mean EM scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	60.03 ${}_{\scalebox{0.6}{3.26}}$	61.70 ${}_{\scalebox{0.6}{1.69}}$	59.73 ${}_{\scalebox{0.6}{2.45}}$	57.84 ${}_{\scalebox{0.6}{2.95}}$	58.49 ${}_{\scalebox{0.6}{2.03}}$	59.56 ${}_{\scalebox{0.6}{1.49}}$	54.52 ${}_{\scalebox{0.6}{0.43}}$	55.36 ${}_{\scalebox{0.6}{0.65}}$	55.40 ${}_{\scalebox{0.6}{0.54}}$	54.07 ${}_{\scalebox{0.6}{0.58}}$	55.29 ${}_{\scalebox{0.6}{0.50}}$	54.93 ${}_{\scalebox{0.6}{0.60}}$
NQ	+ CofFeQA	62.05 ${}_{\scalebox{0.6}{0.53}}$	62.65 ${}_{\scalebox{0.6}{0.65}}$	60.04 ${}_{\scalebox{0.6}{0.70}}$	59.79 ${}_{\scalebox{0.6}{2.52}}$	59.30 ${}_{\scalebox{0.6}{1.90}}$	60.76 ${}_{\scalebox{0.6}{1.49}}$	53.74 ${}_{\scalebox{0.6}{0.50}}$	56.26 ${}_{\scalebox{0.6}{0.96}}$	58.43 ${}_{\scalebox{0.6}{0.45}}$	59.07 ${}_{\scalebox{0.6}{0.30}}$	59.51 ${}_{\scalebox{0.6}{1.25}}$	57.40 ${}_{\scalebox{0.6}{2.40}}$
SQuAD	KD	86.28 ${}_{\scalebox{0.6}{0.78}}$	84.15 ${}_{\scalebox{0.6}{0.74}}$	83.72 ${}_{\scalebox{0.6}{1.19}}$	84.55 ${}_{\scalebox{0.6}{1.04}}$	82.90 ${}_{\scalebox{0.6}{2.76}}$	84.32 ${}_{\scalebox{0.6}{1.26}}$	65.26 ${}_{\scalebox{0.6}{0.33}}$	64.49 ${}_{\scalebox{0.6}{0.59}}$	64.19 ${}_{\scalebox{0.6}{0.38}}$	65.67 ${}_{\scalebox{0.6}{0.74}}$	66.22 ${}_{\scalebox{0.6}{0.60}}$	65.17 ${}_{\scalebox{0.6}{0.83}}$
SQuAD	+ CofFeQA	86.52 ${}_{\scalebox{0.6}{0.19}}$	86.51 ${}_{\scalebox{0.6}{0.35}}$	85.28 ${}_{\scalebox{0.6}{0.98}}$	85.34 ${}_{\scalebox{0.6}{0.62}}$	84.31 ${}_{\scalebox{0.6}{1.31}}$	85.59 ${}_{\scalebox{0.6}{0.94}}$	63.82 ${}_{\scalebox{0.6}{0.44}}$	65.57 ${}_{\scalebox{0.6}{0.28}}$	66.12 ${}_{\scalebox{0.6}{0.75}}$	67.46 ${}_{\scalebox{0.6}{1.36}}$	69.79 ${}_{\scalebox{0.6}{1.74}}$	66.55 ${}_{\scalebox{0.6}{2.23}}$
NewsQA	KD	71.99 ${}_{\scalebox{0.6}{2.01}}$	73.56 ${}_{\scalebox{0.6}{1.97}}$	72.39 ${}_{\scalebox{0.6}{1.33}}$	69.03 ${}_{\scalebox{0.6}{3.06}}$	66.17 ${}_{\scalebox{0.6}{0.96}}$	70.63 ${}_{\scalebox{0.6}{3.00}}$	46.19 ${}_{\scalebox{0.6}{1.28}}$	42.10 ${}_{\scalebox{0.6}{1.00}}$	42.91 ${}_{\scalebox{0.6}{1.56}}$	39.78 ${}_{\scalebox{0.6}{1.20}}$	39.19 ${}_{\scalebox{0.6}{2.34}}$	42.03 ${}_{\scalebox{0.6}{2.80}}$

Appendix

Table 13: F1 score comparison on modified-context test sets across

k

-shot settings for T5 and Qwen2.5 models using KD. Mean F1 scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	56.23 ${}_{\scalebox{0.6}{4.87}}$	60.20 ${}_{\scalebox{0.6}{2.87}}$	54.47 ${}_{\scalebox{0.6}{6.56}}$	50.26 ${}_{\scalebox{0.6}{7.34}}$	52.62 ${}_{\scalebox{0.6}{5.00}}$	54.76 ${}_{\scalebox{0.6}{3.76}}$	53.69 ${}_{\scalebox{0.6}{0.61}}$	55.75 ${}_{\scalebox{0.6}{0.87}}$	55.44 ${}_{\scalebox{0.6}{0.53}}$	54.31 ${}_{\scalebox{0.6}{0.82}}$	55.25 ${}_{\scalebox{0.6}{0.27}}$	54.89 ${}_{\scalebox{0.6}{0.86}}$
NQ	+ CofFeQA	61.34 ${}_{\scalebox{0.6}{0.56}}$	60.64 ${}_{\scalebox{0.6}{0.49}}$	57.97 ${}_{\scalebox{0.6}{1.36}}$	52.94 ${}_{\scalebox{0.6}{6.72}}$	53.46 ${}_{\scalebox{0.6}{4.92}}$	57.27 ${}_{\scalebox{0.6}{3.93}}$	52.62 ${}_{\scalebox{0.6}{0.58}}$	56.01 ${}_{\scalebox{0.6}{1.03}}$	57.34 ${}_{\scalebox{0.6}{0.62}}$	58.57 ${}_{\scalebox{0.6}{0.68}}$	58.50 ${}_{\scalebox{0.6}{0.62}}$	56.61 ${}_{\scalebox{0.6}{2.46}}$
SQuAD	KD	85.35 ${}_{\scalebox{0.6}{0.75}}$	81.94 ${}_{\scalebox{0.6}{1.37}}$	80.52 ${}_{\scalebox{0.6}{3.17}}$	82.17 ${}_{\scalebox{0.6}{2.32}}$	78.67 ${}_{\scalebox{0.6}{6.21}}$	81.73 ${}_{\scalebox{0.6}{2.46}}$	61.86 ${}_{\scalebox{0.6}{0.51}}$	61.42 ${}_{\scalebox{0.6}{0.73}}$	61.10 ${}_{\scalebox{0.6}{0.59}}$	61.37 ${}_{\scalebox{0.6}{1.08}}$	61.69 ${}_{\scalebox{0.6}{0.77}}$	61.49 ${}_{\scalebox{0.6}{0.29}}$
SQuAD	+ CofFeQA	85.73 ${}_{\scalebox{0.6}{0.39}}$	85.55 ${}_{\scalebox{0.6}{0.58}}$	83.33 ${}_{\scalebox{0.6}{1.33}}$	83.96 ${}_{\scalebox{0.6}{0.45}}$	81.28 ${}_{\scalebox{0.6}{2.43}}$	83.97 ${}_{\scalebox{0.6}{1.82}}$	60.45 ${}_{\scalebox{0.6}{0.43}}$	62.22 ${}_{\scalebox{0.6}{0.32}}$	62.22 ${}_{\scalebox{0.6}{1.01}}$	63.67 ${}_{\scalebox{0.6}{0.99}}$	64.48 ${}_{\scalebox{0.6}{0.86}}$	62.61 ${}_{\scalebox{0.6}{1.55}}$
NewsQA	KD	63.28 ${}_{\scalebox{0.6}{2.18}}$	63.10 ${}_{\scalebox{0.6}{3.40}}$	60.61 ${}_{\scalebox{0.6}{2.80}}$	51.19 ${}_{\scalebox{0.6}{4.13}}$	46.58 ${}_{\scalebox{0.6}{1.56}}$	56.95 ${}_{\scalebox{0.6}{7.62}}$	41.80 ${}_{\scalebox{0.6}{1.25}}$	37.86 ${}_{\scalebox{0.6}{0.82}}$	37.51 ${}_{\scalebox{0.6}{0.50}}$	36.87 ${}_{\scalebox{0.6}{1.29}}$	35.97 ${}_{\scalebox{0.6}{1.75}}$	38.00 ${}_{\scalebox{0.6}{2.24}}$

Appendix

Table 14: Teacher-student agreement EM score comparison on modified-context test sets across

k

-shot settings for T5 and Qwen2.5 models using KD. Mean agreement EM scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

		T5						Qwen2.5
Dataset	Method	16	32	64	128	256	Avg	16	32	64	128	256	Avg
NQ	KD	62.31 ${}_{\scalebox{0.6}{3.15}}$	64.50 ${}_{\scalebox{0.6}{1.70}}$	61.75 ${}_{\scalebox{0.6}{2.79}}$	60.11 ${}_{\scalebox{0.6}{3.06}}$	60.98 ${}_{\scalebox{0.6}{2.11}}$	61.93 ${}_{\scalebox{0.6}{1.66}}$	59.30 ${}_{\scalebox{0.6}{0.46}}$	61.21 ${}_{\scalebox{0.6}{0.76}}$	60.87 ${}_{\scalebox{0.6}{0.35}}$	59.53 ${}_{\scalebox{0.6}{0.70}}$	60.45 ${}_{\scalebox{0.6}{0.37}}$	60.27 ${}_{\scalebox{0.6}{0.83}}$
NQ	+ CofFeQA	65.08 ${}_{\scalebox{0.6}{0.40}}$	64.91 ${}_{\scalebox{0.6}{0.42}}$	62.65 ${}_{\scalebox{0.6}{0.70}}$	61.96 ${}_{\scalebox{0.6}{2.63}}$	61.36 ${}_{\scalebox{0.6}{2.37}}$	63.19 ${}_{\scalebox{0.6}{1.71}}$	58.04 ${}_{\scalebox{0.6}{0.48}}$	61.55 ${}_{\scalebox{0.6}{1.04}}$	62.83 ${}_{\scalebox{0.6}{0.51}}$	63.71 ${}_{\scalebox{0.6}{0.58}}$	64.03 ${}_{\scalebox{0.6}{0.59}}$	62.03 ${}_{\scalebox{0.6}{2.43}}$
SQuAD	KD	90.29 ${}_{\scalebox{0.6}{0.71}}$	87.73 ${}_{\scalebox{0.6}{0.93}}$	87.20 ${}_{\scalebox{0.6}{1.46}}$	88.17 ${}_{\scalebox{0.6}{1.55}}$	85.90 ${}_{\scalebox{0.6}{3.23}}$	87.86 ${}_{\scalebox{0.6}{1.60}}$	67.69 ${}_{\scalebox{0.6}{0.47}}$	67.24 ${}_{\scalebox{0.6}{0.39}}$	66.98 ${}_{\scalebox{0.6}{0.45}}$	68.04 ${}_{\scalebox{0.6}{0.86}}$	68.34 ${}_{\scalebox{0.6}{0.57}}$	67.66 ${}_{\scalebox{0.6}{0.56}}$
SQuAD	+ CofFeQA	90.45 ${}_{\scalebox{0.6}{0.25}}$	90.43 ${}_{\scalebox{0.6}{0.40}}$	88.88 ${}_{\scalebox{0.6}{0.85}}$	89.13 ${}_{\scalebox{0.6}{0.39}}$	87.34 ${}_{\scalebox{0.6}{1.48}}$	89.25 ${}_{\scalebox{0.6}{1.29}}$	66.49 ${}_{\scalebox{0.6}{0.57}}$	67.98 ${}_{\scalebox{0.6}{0.16}}$	68.37 ${}_{\scalebox{0.6}{0.80}}$	69.49 ${}_{\scalebox{0.6}{0.87}}$	71.00 ${}_{\scalebox{0.6}{1.13}}$	68.66 ${}_{\scalebox{0.6}{1.69}}$
NewsQA	KD	72.06 ${}_{\scalebox{0.6}{1.88}}$	72.84 ${}_{\scalebox{0.6}{2.12}}$	71.26 ${}_{\scalebox{0.6}{1.39}}$	67.68 ${}_{\scalebox{0.6}{2.77}}$	64.94 ${}_{\scalebox{0.6}{1.45}}$	69.76 ${}_{\scalebox{0.6}{3.34}}$	50.55 ${}_{\scalebox{0.6}{1.41}}$	47.13 ${}_{\scalebox{0.6}{1.02}}$	47.65 ${}_{\scalebox{0.6}{1.17}}$	45.17 ${}_{\scalebox{0.6}{1.52}}$	44.41 ${}_{\scalebox{0.6}{2.23}}$	46.98 ${}_{\scalebox{0.6}{2.40}}$

Appendix

Table 15: Teacher-student agreement F1 score comparison on modified-context test sets across

k

-shot settings for T5 and Qwen2.5 models using KD. Mean agreement F1 scores are reported with standard deviations as subscripts (

Mean_{std}

). Bold values indicate the better-performing method ( CofFeQA vs. KD). Blue shading highlights aggregate results.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 522 papers

Sep 30, 2026

K2P: Label-Free Knowledge to Prompt Distillation

Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.

Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1

Department of Statistics University of Georgia Athens, GA 30602

Prompt Learning Data-Free Knowledge Distillation Language Model Distillation Knowledge Distillation

Ask PDF

Feb 18, 2026

Task-Specific Knowledge Distillation via Intermediate Probes

Knowledge distillation from large language models (LLMs) assumes that the teacher's output distribution is a high-quality training signal. On reasoning tasks, this assumption is frequently violated. A model's intermediate representations may encode the correct answer, yet this information is lost or distorted through the vocabulary projection, where prompt formatting and answer-token choices creates brittle, noisy outputs. We introduce \method{}, a distillation framework that bypasses this bottleneck by training lightweight probes on frozen teacher hidden states and using the probe's predictions, rather than output logits, as supervision for student training. This simple change yields consistent improvements across four reasoning benchmarks (AQuA-RAT, ARC Easy/Challenge, and MMLU), with gains most pronounced under limited data. Probes trained on intermediate representations provide cleaner labels than the teacher's own outputs, effectively denoising the distillation signal. \method{} requires no architectural changes to student or teacher, is architecture-agnostic, and adds minimal compute since probe training is cheap and teacher representations can be cached. By exploiting internal representations, \method{} enables practitioners to extract more value from large teacher models without additional training data or architectural complexity.

Ryan Brown, Chris Russell

Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.

Language Model Distillation Knowledge Distillation

Ask PDF

Oct 14, 2026

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance. We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing. Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers. Our approach consistently outperforms naive distillation with comparable data budgets (

57.15 \pm 1.01

vs 51.61 on average) and offers a much steeper learning curve. Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.

Andrey Goncharov, Daniil Vyazhev, Simon Epanov +1

Applied AI Institute

Online Knowledge Distillation Language Model Distillation Knowledge Distillation CoT Distillation Selective Knowledge Distillation

Ask PDF

Abstract

Language Model Distillation

Explore similar work

K2P: Label-Free Knowledge to Prompt Distillation

Task-Specific Knowledge Distillation via Intermediate Probes

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing