Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness A as an arbitrary wrapper around a base model M and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is \NL-complete even when the primitive is a trivial \AC0 lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that \NL⊆\TC0. Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the gap, which we verify empirically. Finally, we scale to an agentic harness on competition math, where the same gap reappears. Overall, our results show that a harness confers on the base model a profound gain in representational power, one that training a same-class student on its traces cannot transfer back into the weights.
Figures & tables
Figure 1: Same-size students cannot recover the teacher. ID (blue) / OOD (magenta) accuracy of the base model and the two students; dashed lines denote the teacher.
Figure 2: Bigger students and a simpler scaffold narrow the gap. (a) The two scaffolds. (b) Accuracy against the student’s parameter count. (c) Accuracy against the distribution the traces were drawn from; dashed lines mark the teacher.
Figure 3: A long enough chain-of-thought matches the scaffold. (a) The task. (b) Accuracy against the training chain-of-thought length L ; the dashed line marks the phase transition.
Figure 4: Illustration of the directed s – t reachability task. (a) The path-finding task. (b) With an AC0 edge oracle, a harness decides it in polynomial time. (c) A model with a bounded chain-of-thought cannot.
Figure 5: Larger trees widen the gap. (a) Teacher and student accuracy on Qwen3-1.7B per tree shape (d,L) . (b) Trace length and gain over the base model on QED-Nano.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: No fine-tuning setting reaches the teacher. A same-size student fine-tuned on VGB traces (Sokoban, MC-oracle score, score tokens kept). Top row: clean-data ratio (a), number of iterations (b) and number of traces (c). Bottom row: the clean-data sweep repeated with training puzzles drawn from Pα , for α=0.25 (d), 0.50 (e) and 0.75 (f). Dashed lines mark the teacher.
Figure 7: Fine-tuned A* students stay at the base model. Fine-tuning the base model on A* traces (Sokoban). For every clean-data ratio, number of iterations and amount of data, the student stays close to the base model and well below the teacher (dashed).
Figure 8: From-scratch students stay far below the teacher. Students trained from scratch, with VGB in the top row and A* search in the bottom row. No seed, trace source or rough change in size brings either student close to its teacher (dashed; VGB 0.765/0.605 , A* 0.99/0.98 ).
scale
tiny
xsmall
small
s_plus
teacher
t_plus
(d,L,H)
32/2/4
48/2/4
64/2/4
96/2/4
128/2/4
160/2/8
params
0.10 M
0.16 M
0.24 M
0.44 M
0.68 M
0.98 M
scale
mini
m_m1
m_m2
medium
m_plus
large
(d,L,H)
192/3/8
224/3/8
256/3/8
256/4/8
320/4/8
384/6/8
params
1.77 M
2.32 M
2.94 M
3.73 M
5.65 M
11.5 M
Appendix
Table 1: The 12 student scales of Section 3.1 , plus large . VGB is run on the 8 scales from tiny to m_plus ; A* search and Dyck use all of them.
Figure 9: Larger students close the gap on Sokoban. Best accuracy at each scale in the two Sokoban settings.
Figure 10: Accuracy increases with size at every clean ratio and checkpoint. The sweeps of Figure 9 shown separately for each clean-data ratio and each checkpoint. Within each block, the rows are A*, VGB-Sokoban and VGB-Dyck from top to bottom.
reference
distilled, trace source
synthetic BT, errors from
base
oracle-BT
ID
α=0.25
α=0.50
OOD
random
ID
one pass, ID
0.947
—
0.936
0.957
0.942
0.947
0.977
0.840
one pass, OOD
0.650
—
0.133
0.470
0.313
0.250
0.637
0.450
with <BT> , ID
—
0.996
0.966
0.970
0.957
0.957
0.963
0.933
with <BT> , OOD
—
0.873
0.543
0.677
0.680
0.637
0.683
0.697
Appendix
Table 2: Full results of simple-BT distillation on Dyck, in one pass and with the student’s own <BT> tokens. Figure 2 c in the main text uses the <BT> rows of the distilled students.
Figure 11: The same phase transition on an easier task. (a) The verifier-backtracking harness with K=16 . (b) Accuracy against the training chain-of-thought length L for nops=10 .
Figure 12: No distillation setting reaches the teacher, and only the teacher gains from more calls. Branching distillation on Qwen3-1.7B. Top row: clean-data ratio (a), number of traces (b) and number of epochs (c); the teacher is dashed and the base model dotted. Bottom row: number of calls per problem for every tree shape (d,L) (d), and accuracy with the shapes sorted by the teacher’s accuracy (e).
shape
(1,4)
(1,5)
(1,6)
(2,2)
(3,3)
leaves
4
5
6
4
27
aggregations
1
1
1
2+1
13
calls
5
6
7
7
40
median trace
208 k
253 k
309 k
—
≈1.4 M
Appendix
Table 3: The five tree shapes we run and the median length of their full traces.
Figure 13: A larger backbone closes the gap. (a) Student accuracy with the 1.7 B and 4 B backbones. (b) Student accuracy with LoRA ranks 16 and 64 and with full fine-tuning. (c) Gain of the harness over the base model on QED-Nano for each shape. Only the 40 -call shape has a gain whose interval excludes zero, and its trace is much longer than the context window.
Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce On-Policy Harness Self-Distillation (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.
Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
Cheng Qian, Wenting Zhao, Liangwei Yang +6
Salesforce AI Research · University of Illinois Urbana-Champaign
Harness engineering has emerged as an important inference-time technique for large language model (LLM) agents, aiming to improve long-term performance through task decomposition and guided execution. However, more elaborate harnesses are not uniformly better: increasing decomposition or guidance can sometimes improve execution, but can also reduce final task success. We study harness design through the lens of inference-time trajectory alignment. This perspective separates harness into two mechanisms: task decomposition, which structures a task into sub-goals, and guided execution, which reshapes local action distributions during execution. This decomposition allows us to quantify how workflow granularity, retry budgets, and guidance-induced action reweighting shape the performance limits of harness design. It further reveals concrete failure modes, including over-decomposition, over-pruning, and hallucinated execution. We validate these predictions through controlled synthetic experiments and real terminal agent benchmarks. Inspired by the theory, we further show that effective harnesses can be partial: specifying only the initial steps and leaving the remaining execution to agent can achieve higher pass rate than fully structured workflows.