Teaching the Harness: Student-Aware Skill Transfer for LLM Agents
Organizations: Colorado School of Mines, Golden, CO, USA
Abstract
Effective teaching requires reasoning about the learner as well as the task, a capacity that Theory of Mind describes as inferring another agent's latent state from its behavior. We ask whether a stronger LLM teaches a fixed student better when it applies this principle. The student is an LLM agent acting through a harness that fixes its tools, context, and action budget, so a procedure that helps one model can exhaust the budget of another. We introduce Student-Aware Teaching (SAT). The teacher builds a learner model that pairs an empirical profile with hypotheses about the student's limitations, tests these hypotheses with diagnostic probes, and writes reusable skills for the diagnosed failures. A skill enters the library only after it raises success on matched development tasks. With Claude-Sonnet-5 teaching DeepSeek-V4-Flash, SAT reaches 86.46% task success across four primary runs, leading both a profile-free teacher with the same framework (81.04%) and the untaught student (75.00%) in each run. Removing the analytical learner model lowers both prediction and success, and under misleading learner profiles none of the 48 candidate skills passes admission. When DeepSeek-V4-Flash and Gemini-3.8-Flash exchange libraries, each performs better with its own. Saved libraries keep average gains of up to 11.1 points after the teacher leaves. The benefit depends on a student that can follow instructions and a deployment route that delivers them.
Figures & tables
| Family | Examples | What changes | Signal that shapes the change |
|---|---|---|---|
| Machine teaching | Knowledge tracing, IMT | Training examples | Estimated learner state |
| Distillation | KD, Step-by-Step, GKD | Student weights | Teacher outputs |
| Prompt optimization | ProTeGi, DSPy, TextGrad, GEPA | Prompts and pipelines | Task metric and execution feedback |
| Experience memory | Reflexion, ExpeL, AWM, ReasoningBank, ACE | Memories, workflows, playbooks | The agent’s own experience |
| Skill optimization | SkillOpt, EvoSkill, Trace2Skill | Natural-language skills for a frozen agent | Trajectories and validation scores |
| Skills and harnesses | Voyager, SkillWeaver, Meta-Harness, AI4AI | Code skills, agents, or harness code | Environment tests, scores, traces |
| No skill | Agnostic | Aware | Aware vs. none | Aware vs. agnostic | |
| Per run | |||||
| S1 | 79.17 | 75.83 | 85.83 | ||
| S2 | 77.50 | 85.00 | 87.50 | ||
| S3 | 70.83 | 86.67 | 87.50 | ||
| S4 | 72.50 | 76.67 | 85.00 | ||
| Mean | 75.00 | 81.04 | 86.46 | ||
| Configuration | Aware acceptance | Teacher Brier |
|---|---|---|
| Component ablations, run S2 | ||
| Full SAT | 87.5 | 0.041 |
| Without fine probes | 83.3 | 0.036 |
| Without analytical learner model | 83.3 | 0.180 |
| Misleading profiles, mean of two seeds | ||
| Overestimating profile | 75.0 | 0.238 |
| Curriculum record | Procedure encoded in the skill |
|---|---|
| Agnostic plan | List items and inspect each item’s stock. Restock deficient items to the threshold. Track processed items and write the completion marker after processing all items. |
| Aware skill 1 | Retrieve all stock in one query. Restock items below the threshold and inspect each updated item. Retain intermediate state for later checks. |
| Aware skill 2 | Retrieve all stock in one query. Restock the required quantities and verify final levels after all updates. |
| Student | Route | Own | Exchanged | Own minus exchanged |
|---|---|---|---|---|
| DeepSeek-V4-Flash | Student selection | 79.2 | 75.0 | |
| Category routing | 93.3 | 83.3 | ||
| Flat injection | 86.7 | 82.5 | ||
| Gemini-3.8-Flash | Student selection | 91.7 | 88.3 | |
| Category routing | 93.3 | 90.0 | ||
| Flat injection | 93.3 | 91.7 |
| Student | Library | Student selection | Category routing | Flat injection |
|---|---|---|---|---|
| DeepSeek-V4-Flash | L1 | |||
| DeepSeek-V4-Flash | L2 | |||
| DeepSeek-V4-Flash | L3 | |||
| DeepSeek-V4-Flash | Mean of L1 to L3 | |||
| DeepSeek-V4-Flash | Extended | |||
| Qwen2.5-7B-Instruct | Local |
| Student | Teacher | No skill | Agnostic | Aware |
|---|---|---|---|---|
| Qwen3-8B | Qwen3-14B | 38.3 | 33.3 | 53.3 |
| Qwen3-8B | Qwen3-32B | 38.3 | 45.0 | 50.0 |
| Qwen2.5-7B | Qwen2.5-32B | 43.3 | 45.0 | 46.7 |
| Qwen2.5-3B, run A | Qwen2.5-32B | 16.7 | 21.7 | 20.0 |
| Qwen2.5-3B, run B | Qwen2.5-32B | 15.0 | 13.3 | 15.0 |
| Qwen3-1.7B | Qwen3-32B | 11.7 | 11.7 | 11.7 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Suite | Categories | Verifier checks |
|---|---|---|
| Terminal | Word and line counting, extension renaming (with nested directories), sorted numeric merge, log deduplication, CSV filtering, staged file pipelines | Final files and directory state, or the reported count |
| Inventory API | Restocking every item below a threshold through list, query, and restock tools | Final stock levels and the completion marker |
| Arithmetic | Compound discounts, mixtures, relative rates | The final numeric answer |
| Python code | Function implementation and bug repair | The submitted function or repaired file |
| Dependent workflows | Chains of stages in which each stage reads the previous stage’s output file | The output of every stage |
| Experiment | Teaching | Development | Test | Suites |
|---|---|---|---|---|
| Main setting (S1 to S4) and controls | 12 | 6 | 12 | All five |
| Extended eight-round run | 40 | 12 | 30 | All five |
| Production harnesses | 12 | 6 | 12 | All except inventory API |
| Condition | Learned | Process | Content | Plan | Affordance | Complete bodies |
|---|---|---|---|---|---|---|
| Agnostic | 13 | 6 | 3 | 1 | 3 | 10 |
| Aware | 15 | 10 | 3 | 2 | 0 | 12 |
| Label | Inclusion rule |
|---|---|
| Execution economy | Batch operations, reduce redundant tool calls, or allocate a limited action budget. |
| Environment coverage | Enumerate the complete target set or preserve coverage across directories. |
| State and progress | Track intermediate state or completed work to avoid stale or repeated actions. |
| Task semantics | Specify computation, exact transformations, or output-format requirements. |
| Verification and completion | Inspect a result or postcondition before proceeding or finishing. Conditional and deferred checks qualify. |
| Function | Agnostic ( ) | Aware ( ) |
|---|---|---|
| Execution economy | 5 (50.0) | 5 (41.7) |
| Environment coverage | 1 (10.0) | 6 (50.0) |
| State and progress | 3 (30.0) | 2 (16.7) |
| Task semantics | 6 (60.0) | 8 (66.7) |
| Verification and completion | 10 (100.0) | 10 (83.3) |
| Main artifacts | Qwen3 | |||
|---|---|---|---|---|
| Category | Agnostic | Aware | Agnostic | Aware |
| Extension rename | 1 | 6 | 0 | 0 |
| Dependent workflow | 4 | 2 | 0 | 1 |
| Inventory restock | 2 | 2 | 1 | 2 |
| Sorted merge | 1 | 1 | 0 | 0 |
| File pipeline | 1 | 1 | 0 | 0 |
| Case | Agnostic curriculum | Aware curriculum | Agnostic | Aware |
|---|---|---|---|---|
| S4 inventory | Generic stock-check instruction remains active | Bulk stock snapshot replaces per-item discovery | 50.0 | 79.2 |
| Qwen3 inventory | Learned plan checks stock item by item | Two learned skills use bulk stock retrieval and explicit checks | 0.0 | 91.7 |
| S3 inventory | Learned plan skips routine verification | Learned procedure checks stock after all updates | 91.7 | 95.8 |
| S1 workflows | Skill for simple single-purpose commands | Skill for computing chain stages in one call | 75.0 | 95.8 |
| Extended renaming | Generic instruction with shallow glob | Recursive discovery with a final extension check | 20.0 | 100.0 |
| Student and library | Route | Library | No skill | Difference and interval |
|---|---|---|---|---|
| DeepSeek-V4-Flash, L1 | Student | 85.8 | 78.3 | |
| Category | 90.0 | 80.8 | ||
| Flat | 90.8 | 81.7 | ||
| DeepSeek-V4-Flash, L2 | Student | 78.3 | 79.2 | |
| Category | 85.0 | 77.5 | ||
| Flat | 84.2 | 73.3 |
| Route | Acceptance | Gain | 95% interval for gain |
|---|---|---|---|
| Student selection | 83.7 | ||
| Category routing | 85.7 | ||
| Flat injection | 85.0 |
| Subset | None | Agn. | Aware |
|---|---|---|---|
| MATH (3 runs) | 75.0 | 73.6 | 81.9 |
| GSM8K (2 runs) | 87.5 | 89.6 | 97.9 |
| Dataset | Run | No skill | Agnostic | Aware |
|---|---|---|---|---|
| MATH | M1 | 79.2 | 70.8 | 83.3 |
| MATH | M2 | 70.8 | 79.2 | 83.3 |
| MATH | M3 | 75.0 | 70.8 | 79.2 |
| GSM8K | G1 | 91.7 | 83.3 | 100.0 |
| GSM8K | G2 | 83.3 | 95.8 | 95.8 |
| Final task success | Final Brier | ||||
|---|---|---|---|---|---|
| Run | No skill | Agnostic | Aware | Agnostic | Aware |
| P1 | 90.8 | 88.3 | 88.3 | 0.077 | 0.051 |
| P2 | 89.2 | 90.0 | 90.0 | 0.137 | 0.055 |
| P3 | 86.7 | 93.3 | 93.3 | 0.065 | 0.066 |
| Mean | 88.9 | 90.6 | 90.6 | ||
| Test acceptance | Brier score | AUROC | |||||
|---|---|---|---|---|---|---|---|
| Round | No skill | Agnostic | Aware | Agnostic | Aware | Agnostic | Aware |
| 1 | 74.3 | 78.3 | 78.3 | 0.175 | 0.132 | 0.938 | 0.938 |
| 2 | 75.3 | 79.0 | 78.7 | 0.204 | 0.071 | 0.812 | 1.000 |
| 3 | 78.0 | 80.0 | 80.7 | 0.181 | 0.046 | 0.938 | 1.000 |
| 4 | 74.3 | 80.0 | 78.7 | 0.128 | 0.064 | 0.857 | 1.000 |
| 5 | 76.7 | 79.0 | 80.3 | 0.111 | 0.063 | 0.844 | 1.000 |
| Suite | No skill | Agnostic | Aware |
|---|---|---|---|
| Terminal | 45.0 | 50.0 | 73.3 |
| Inventory tool use | 56.7 | 56.7 | 55.0 |
| Mathematics | 91.7 | 96.7 | 95.0 |
| Code | 100.0 | 100.0 | 100.0 |
| Dependent workflows | 96.7 | 100.0 | 98.3 |
| Overall | 78.0 | 80.7 | 84.3 |
| Condition | Round | Skill | Type | Utility |
| Agnostic | 1 | Restock to the exact threshold | Content | |
| Agnostic | 2 | Minimize API calls to fit the step budget | Process | |
| Agnostic | 4 | Verify merge output line count and format | Process | |
| Agnostic | 4 | Sanitize and cross-check merge inputs | Content | |
| Agnostic | 7 | Compute sequential percentage discounts | Content | |
| Aware | 2 | Compute the exact restock delta per SKU | Process |
| Configuration | No skill | Agnostic | Aware | Aware Brier |
|---|---|---|---|---|
| Full SAT | 77.5 | 85.0 | 87.5 | 0.041 |
| No fine probes | 81.7 | 73.3 | 83.3 | 0.036 |
| No analytical learner model | 78.3 | 80.8 | 83.3 | 0.180 |
| Acceptance | Brier | AUROC | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Profile | Seed | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
| Overest. | 6 | 79.2 | 76.7 | 78.3 | 71.7 | .288 | .284 | .380 | .190 | .50 | .83 | .79 | .69 |
| Overest. | 26 | 79.2 | 81.7 | 74.2 | 78.3 | .193 | .190 | .191 | .285 | .38 | .72 | .63 | .74 |
| Underest. | 6 | 78.3 | 78.3 | 76.7 | 82.5 | .644 | .558 | .646 | .746 | .50 | .29 | .41 | .13 |
| Underest. | 26 | 78.3 | 80.8 | 80.8 | 75.8 | .650 | .746 | .840 | .747 | .55 | .31 | .11 | .28 |
| Student and route | Own minus none | Swapped minus none | Own minus swapped |
|---|---|---|---|
| DeepSeek-V4-Flash, student | |||
| DeepSeek-V4-Flash, category | |||
| DeepSeek-V4-Flash, flat | |||
| Gemini-3.8-Flash, student | |||
| Gemini-3.8-Flash, category | |||
| Gemini-3.8-Flash, flat |
| Harness | Agnostic | Aware |
|---|---|---|
| Codex | ||
| Pi | ||
| OpenCode | ||
| Claude Code |
| Harness | Condition | Round 1 | Round 2 | Round 3 | Round 4 |
|---|---|---|---|---|---|
| Codex | Agnostic | ||||
| Aware | |||||
| Pi | Agnostic | ||||
| Aware | |||||
| OpenCode | Agnostic | ||||
| Aware |
| Harness | Condition | Terminal | Arithmetic | Code | Workflows |
|---|---|---|---|---|---|
| Codex | Agnostic | ||||
| Aware | |||||
| Pi | Agnostic | ||||
| Aware | |||||
| OpenCode | Agnostic | ||||
| Aware |
| Condition | Role | Calls | Input tokens | Output tokens | Estimated USD |
|---|---|---|---|---|---|
| No skill | Student | 3725.33 | 1361517.3 | 1644371.3 | 0.443 |
| Agnostic | Student | 4185.00 | 1794873.3 | 805769.7 | 0.285 |
| Agnostic | Teacher | 76.00 | 54743.7 | 12194.0 | 0.231 |
| Aware | Student | 4355.33 | 1885968.7 | 724544.7 | 0.273 |
| Aware | Teacher | 82.67 | 119400.3 | 15304.0 | 0.392 |
| Category | Records | No skill | Agnostic | Aware |
|---|---|---|---|---|
| Exact-word counting | 6 | 0.00 | 0.00 | 0.00 |
| Extension rename | 6 | 0.00 | 33.33 | 100.00 |
| Sorted merge | 6 | 50.00 | 75.00 | 75.00 |
| Log deduplication | 6 | 91.67 | 100.00 | 100.00 |
| CSV filtering | 6 | 100.00 | 100.00 | 100.00 |
| File pipeline | 6 | 50.00 | 66.67 | 50.00 |
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 620 papers