Understanding the Benefits of Scaffolds through Computational Separations

Jingchu Gai; Adam Block; Andrej Risteski

cs.LGOct 19, 2026

Understanding the Benefits of Scaffolds through Computational Separations

Authors: Jingchu Gai, Adam Block, Andrej Risteski

Organizations: Machine Learning Department Carnegie Mellon University · Departments of Computer Science and Electrical Engineering Columbia University

Abstract

Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness $\mathcal{A}$ as an arbitrary wrapper around a base model $M$ and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is $\NL$ -complete even when the primitive is a trivial $\AC^0$ lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that $\NL\not\subseteq\TC^0$ . Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the gap, which we verify empirically. Finally, we scale to an agentic harness on competition math, where the same gap reappears. Overall, our results show that a harness confers on the base model a profound gain in representational power, one that training a same-class student on its traces cannot transfer back into the weights.

Figures & tables

Figure 1: Same-size students cannot recover the teacher. ID (blue) / OOD (magenta) accuracy of the base model and the two students; dashed lines denote the teacher.

Figure 2: Bigger students and a simpler scaffold narrow the gap. (a) The two scaffolds. (b) Accuracy against the student’s parameter count. (c) Accuracy against the distribution the traces were drawn from; dashed lines mark the teacher.

Figure 3: A long enough chain-of-thought matches the scaffold. (a) The task. (b) Accuracy against the training chain-of-thought length

L

; the dashed line marks the phase transition.

Figure 4: Illustration of the directed

s

–

t

reachability task. (a) The path-finding task. (b) With an

\mathsf{AC}^{0}

edge oracle, a harness decides it in polynomial time. (c) A model with a bounded chain-of-thought cannot.

Figure 5: Larger trees widen the gap. (a) Teacher and student accuracy on Qwen3-1.7B per tree shape

(d,L)

. (b) Trace length and gain over the base model on QED-Nano.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: No fine-tuning setting reaches the teacher. A same-size student fine-tuned on VGB traces (Sokoban, MC-oracle score, score tokens kept). Top row: clean-data ratio (a), number of iterations (b) and number of traces (c). Bottom row: the clean-data sweep repeated with training puzzles drawn from

P_{\alpha}

, for

\alpha=0.25

(d),

0.50

(e) and

0.75

(f). Dashed lines mark the teacher.

Figure 7: Fine-tuned A* students stay at the base model. Fine-tuning the base model on A* traces (Sokoban). For every clean-data ratio, number of iterations and amount of data, the student stays close to the base model and well below the teacher (dashed).

Figure 8: From-scratch students stay far below the teacher. Students trained from scratch, with VGB in the top row and A* search in the bottom row. No seed, trace source or rough change in size brings either student close to its teacher (dashed; VGB

0.765/0.605

, A*

0.99/0.98

scale	tiny	xsmall	small	s_plus	teacher	t_plus
$(d,L,H)$	32/2/4	48/2/4	64/2/4	96/2/4	128/2/4	160/2/8
params	$0.10$ M	$0.16$ M	$0.24$ M	$0.44$ M	$0.68$ M	$0.98$ M
scale	mini	m_m1	m_m2	medium	m_plus	large
$(d,L,H)$	192/3/8	224/3/8	256/3/8	256/4/8	320/4/8	384/6/8
params	$1.77$ M	$2.32$ M	$2.94$ M	$3.73$ M	$5.65$ M	$11.5$ M

Appendix

Table 1: The

12

student scales of Section 3.1 , plus large . VGB is run on the

8

scales from tiny to m_plus ; A* search and Dyck use all of them.

Figure 9: Larger students close the gap on Sokoban. Best accuracy at each scale in the two Sokoban settings.

Figure 10: Accuracy increases with size at every clean ratio and checkpoint. The sweeps of Figure 9 shown separately for each clean-data ratio and each checkpoint. Within each block, the rows are A*, VGB-Sokoban and VGB-Dyck from top to bottom.

	reference		distilled, trace source			synthetic BT, errors from
	base	oracle-BT	ID	$\alpha{=}0.25$	$\alpha{=}0.50$	OOD	random	ID
one pass, ID	$0.947$	—	$0.936$	$0.957$	$0.942$	$0.947$	$0.977$	$0.840$
one pass, OOD	$0.650$	—	$0.133$	$0.470$	$0.313$	$0.250$	$0.637$	$0.450$
with <BT> , ID	—	$0.996$	$0.966$	$0.970$	$0.957$	$0.957$	$0.963$	$0.933$
with <BT> , OOD	—	$0.873$	$0.543$	$0.677$	$0.680$	$0.637$	$0.683$	$0.697$

Appendix

Table 2: Full results of simple-BT distillation on Dyck, in one pass and with the student’s own <BT> tokens. Figure 2 c in the main text uses the <BT> rows of the distilled students.

Figure 11: The same phase transition on an easier task. (a) The verifier-backtracking harness with

K{=}16

. (b) Accuracy against the training chain-of-thought length

L

for

n_{\mathrm{ops}}{=}10

Figure 12: No distillation setting reaches the teacher, and only the teacher gains from more calls. Branching distillation on Qwen3-1.7B. Top row: clean-data ratio (a), number of traces (b) and number of epochs (c); the teacher is dashed and the base model dotted. Bottom row: number of calls per problem for every tree shape

(d,L)

(d), and accuracy with the shapes sorted by the teacher’s accuracy (e).

shape	$(1,4)$	$(1,5)$	$(1,6)$	$(2,2)$	$(3,3)$
leaves	$4$	$5$	$6$	$4$	$27$
aggregations	$1$	$1$	$1$	$2{+}1$	$13$
calls	$5$	$6$	$7$	$7$	$40$
median trace	$208$ k	$253$ k	$309$ k	—	$\approx 1.4$ M

Appendix

Table 3: The five tree shapes we run and the median length of their full traces.

Figure 13: A larger backbone closes the gap. (a) Student accuracy with the

1.7

B and

4

B backbones. (b) Student accuracy with LoRA ranks

16

and

64

and with full fine-tuning. (c) Gain of the harness over the base model on QED-Nano for each shape. Only the

40

-call shape has a gain whose interval excludes zero, and its trace is much longer than the context window.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 829 papers

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning
May 9, 2026Zhengyang Zhao, Lu Ma, Wentao ZhangOn-Policy Self-Distillation Language Model Distillation
Peking University
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Aug 12, 2026Cheng Qian, Wenting Zhao, Liangwei Yang +6Knowledge Distillation
Salesforce AI Research · University of Illinois Urbana-Champaign
Harnesses for Inference-Time Alignment over Execution Trajectories
May 15, 2026Boyuan Wang, Bochao Li, Minghan Wang +2LLM Planning Agent Harness Optimization
Southern University of Science and Technology

May 9, 2026J/K move · Enter open · S save

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

Zhengyang Zhao, Lu Ma, Wentao Zhang

Peking University

On-Policy Self-Distillation Language Model Distillation LLM Reasoning

Open paper AskPDF arXiv

Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce On-Policy Harness Self-Distillation (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.

Abstract

Knowledge Distillation

Explore similar work

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

Harnesses for Inference-Time Alignment over Execution Trajectories