Organizations: Peking University · Sichuan University · Beijing Institute of Technology · Renmin University of China · The Hong Kong University of Science and Technology
Language-model agents increasingly tackle long-horizon tasks whose interaction histories exceed the model's active context. They must therefore decide what to write down and what to read back, and when to do either. Recent work has begun to use on-policy distillation (OPD) to make memory control part of the policy, relying on memory tools and compaction protocols that lie outside either model's pre-training. However, this weakens the distillation on both sides: the teacher cannot give useful guidance on an interface it has never used, and the student samples too few meaningful memory actions for that guidance to take effect. To study this problem, we introduce Coding Agent Memory Gym II (CAMG-II), a suite of long-horizon agentic-RL environments spanning Shop, Coding, AutoResearch, General, and Visual. Alongside each environment's native task interface, CAMG-II provides ordinary Bash access and an episode-persistent workspace, enabling the policy to create, revise, search, and reuse files as memory across context replacements. We also introduce Coding Agent Memory Post-Training II (CAMP-II), which trains a single policy jointly across all five environments with fully asynchronous OPD to learn this file-based memory behavior directly from a frozen teacher with a prior for file operations. On CAMG-II, CAMP-II obtains 2.97 times the gain of the strongest OPD baseline. The memory behavior CAMP-II learns generalizes to two external benchmarks, Terminal-Bench 2.1 and SWE-Bench Pro V2, with improved success rates.
Figures & tables
Figure 1: File-operation priors improve supervision for memory distillation. A frozen teacher scores student-generated memory responses that retain ( Keep ) or omit ( Omit ) the same Download requirement. Compared with dedicated memory tools and context-compaction protocols, ordinary Bash/file operations draw on the teacher’s existing prior, yielding a clearer preference for retaining task-critical information and thus more effective supervision. The student’s general notes and bounded continuation file persist across context replacements, so it can recover the saved Download requirement through Bash.
Figure 2: CAMP-II distils file-based memory across five environments. Top: the student acts in the environment and writes and revises memory files. At each context boundary it compresses its state into the continuation file, which a later read restores. Bottom: completed episodes enter the queue, a frozen teacher scores the tokens the student actually sampled, and the OPD learner updates the student weights from the teacher–student log-probability gap, closing the loop back to the rollout.
Task success (%)
Method
Shop
Coding
AutoResearch
General
Visual
Avg. ↑
Frozen control
Qwen3.5-4B
77.3
27.3
46.1
39.8
36.0
45.3
Qwen3.5-9B
90.6
26.6
56.2
60.9
36.5
54.2
Frozen teacher (Qwen3.8-27B)
100.0
63.3
83.6
93.0
40.3
76.0
Learned-memory methods
Table 1: CAMP-II attains the highest average success rate across the CAMG-II test set. Among the 4B systems it outperforms CAMP-I, every memory-learning baseline, and the frozen student. The largest gain lands in General, and Shop saturates. Scaling the same recipe to a 9B student lifts the average further.
Figure 3: Both the teacher and the student are most familiar with the file interface, and that interface carries the most memory reads and writes. (a) Surprisal, the mean negative log-likelihood per generated token, over the five environments under the three memory interfaces. (b) Memory actions as a share of all actions. The file interface draws roughly three times the share of the dedicated tools.
Figure 4: A Visual episode continues from its own saved file. We follow one episode from a late training update on the fitness-product showcase task. In (a) the student writes down what is wrong with the page and how it will rewrite it, before its context is replaced. In (b) it lists the workspace afterwards and reads that note back. In (c) it delivers a page that carries the sections the note listed.
Figure 5: The negative-advantage problem worsens as the interface becomes less familiar. We score every token of the memory actions the student produces under each interface, with the teacher and the student. The mean size of the negative advantage rises from the file interface through the memory tools to compaction, (a) on interface decisions and (b) on content tokens.
Figure 6: An episode’s later success is caused by the memory it wrote. We continue each episode three ways at its first context replacement, leaving the file memory intact, installing another episode’s memory, or emptying it, so the runs differ only in the memory that crosses the boundary.
Figure 7: Each component of CAMP-II is necessary. On the CAMG-II test set we compare the mean success of CAMP-II with that of variants that (i) replace the file interface by compaction, (ii) replace the file interface by the dedicated memory tools, (iii) mask the loss on the memory tokens, and (iv) mask the same number of tokens at random. Every variant that touches the memory supervision loses most of the gain, and matching the masked budget does not.
Policy
TB 2.1
SWE-Bench Pro V2
Avg.
Qwen3.5-4B
23.1
11.2
17.1
Qwen3.5-9B
23.1
23.1
23.1
Qwen3.5-35B-A3B
33.8
27.4
30.6
CAMP-II-4B
30.0
18.1
24.1
CAMP-II-9B
38.0
30.1
34.1
Table 2: Both trained policies beat a larger same-generation model on the average, with fewer parameters. Success rate on Terminal-Bench 2.1 and SWE-Bench Pro V2 for the initial students, same-generation larger models, and the trained policies.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dependency family
Information needed later
Attribute chain
An earlier product attribute constrains another category.
Latent preference
Earlier choices reveal an unstated later preference.
Recency override
A new preference replaces an earlier value.
Distractor robustness
Relevant state is mixed with stale or unrelated records.
Compositional recall
Two earlier relations jointly identify the target.
Negative constraint
Earlier exclusions determine the admissible option.
Appendix
Table 3: Shop’s eight dependency families, inherited from CAMG. Paired variants change earlier state while keeping the later request and candidates fixed.
Figure 8: The policy solves every CAMG-II task across a context replacement. Each panel shows one task input and the interaction available to the policy.
Q0
Q∗
ρ
( 7 )
( 9 )
0.001
0.5
0.5
82.7
72.4
10−5
0.5
0.5
4780.0
4342.9
0.001
0.0012
0.5
329.9
2739.4
0.001
0.5
0.95
21.7
72.4
Appendix
Table 4: Where the rare-action form holds. The two columns are λN from the exact integral and from the asymptotic form, for a student that rarely emits the token under a confident teacher, under a teacher nearly as rare as the student, and under a small relative contraction.
teacher prefers it
teacher does not
student emits it
supervision: the intersection
excess: negative advantage, growing with the gap
student does not
deficit: no direct supervision of the ungenerated action
empty: absent from the loss
Appendix
Table 5: Interface supervision has four cells. Rows split by whether the student emits the action, columns by whether the teacher prefers it. Only the intersection provides direct supervision on the desired action and its content. Eq. 12 prices it.
Group
Hyperparameter
Value
Models
Student (full parameters)
Qwen3.5-4B
Frozen external teacher
Qwen3.8-27B
Objective
Signal
Sampled reverse KL ( k1 )
Teacher target support
Student-sampled token
Target temperature
1
Supervised positions
Full sampled response
Appendix
Table 6: Training configuration.
Environment
Reward
Shop
rt=71 per correct purchase of the session target under the cumulative budget; a wrong purchase pays −0.5 and terminates the episode with earlier payments retained. Success requires the full six-session sequence.
Coding
R=1[hidden tests pass on the submitted repository change] at the submission sentinel; rt=−0.01 per recoverable invalid action and at horizon failure.
AutoResearch
R=max(0,m)+0.051[first execution]+0.101[first validation] , where m is the private grader’s native-metric score of the submitted file and milestone payments are capped at 0.15 per episode; rt=−0.01 at action-budget exhaustion or wall-clock termination, and invalid submissions pay 0 .
General
R=w⊤r , a weighted rubric over the final answer, delivered files, and post-action system state, paid once at finish with required rule checks as gates; rt=−0.1 per invalid action and −0.01 when the budget is exhausted without a submission.
Visual
R=mean(rrubric,rfulfilment,rassets) , an absolute rubric over screenshots of the delivered page at finish (layout, typography, color, whitespace, content), averaged with query fulfilment and asset quality; invalid actions pay 0 .
Appendix
Table 7: Task rewards.
Seed
Shop
Coding
AutoResearch
General
Visual
Avg.
1
100.0
37.5
64.1
81.2
40.2
64.6
2
100.0
39.1
65.6
82.0
41.0
65.5
3
100.0
38.3
63.3
80.5
39.7
64.4
4
100.0
36.7
64.8
82.8
40.5
65.0
5
100.0
38.3
66.4
81.2
40.0
65.2
Mean ± Std.
100.0 ± 0.0
38.0 ± 0.9
64.8 ± 1.2
81.5 ± 0.9
40.3 ± 0.5
64.9 ± 0.5
Appendix
Table 8: Five training seeds of CAMP-II-4B. Task success on the CAMG-II held-out panels, one row per seed, with the mean and the standard deviation over seeds. Seed 1 is the run the main text reports. The average column weights the five domains equally, and the Visual column follows the paper’s definition of that column.
Figure 9: CAMP-I training dynamics. The average in panel (f) weights the five domains equally.
Figure 10: CAMP-II training dynamics. Panel y-axes match Fig. 9 for a common scale. The average in (f) weights the five domains equally.
Figure 11: The file interface keeps the teacher’s organization of memory content. Each interface writes its own note ( 160 notes per interface, initial Qwen3.5-4B). Panels (a)–(c) plot the similarity the teacher assigns to a note pair against the similarity the student assigns, over the 12,720 pairs of each interface, so a pair on the dashed line is one the two checkpoints treat alike. Panel (d) gives the rank correlation of the two similarity matrices with delete-half jackknife intervals over the notes, highest for the file interface.
Figure 12: The file interface generates the least corrective signal. (a) Norm of the gradient of the distillation loss restricted to memory-content tokens, per supervised token, backpropagated through the initial checkpoint. Markers are individual notes ( n=12 per interface) and the bar is their mean. (b) Teacher-minus-student log-probability gap on the same tokens, which is the loss weight. (c) Per-layer gradient norms: the three interfaces share one profile and differ in scale. (d) Nearly independent directions the twelve note-level gradients span, for the MLP gate projection of layers 8 , 16 and 23 (participation ratio of the Gram spectrum, where one direction per note is the ceiling of 12 ): the file interface needs less correction per token, and that correction is less redundant.
Interface
Correct
95% interval
Margin
p vs file
File
60/60
[0.940,1.000]
+0.577
—
Compaction
55/60
[0.816,0.972]
+0.528
0.029
Memory tools
50/60
[0.715,0.917]
+0.480
0.0007
Appendix
Table 9: Replacing a note’s key information by another task’s. Each note is scored against itself with its key segments replaced, matched in character count. Intervals are exact binomial, the margin is per memory-content token, and the last column is a one-sided Fisher test against the file interface.
Figure 13: The teacher ranks the note written for its own task above a foreign note, most often under the file interface. (a) Share of decisions in which the teacher scores the note written for the prefix’s own context higher, with intervals over decisions and the chance level of one half. Hollow markers restrict the decisions to pairs whose two notes differ by at most a fifth in content tokens. (b) Margin per decision, the difference in mean log-probability per memory-content token, markers are single decisions and the bar is their mean. (c) The two candidates’ scores for the same decision, so a point above the diagonal is a decision the teacher gets right.
Method
Shop
Coding
AutoResearch
General
Visual
Avg.
CAMP-II
100.0
37.5
64.1
81.2
40.2
64.6
without the requirement, as trained
100.0
35.9
61.7
78.1
37.5
62.6
without the requirement, retrained
100.0
36.7
62.5
80.5
40.2
64.0
Appendix
Table 10: Removing the requirement to write at a boundary. Task success on the CAMG-II test set when the context is replaced whether or not a valid continuation write exists. The second row evaluates the trained CAMP-II-4B in that environment, the third trains the same recipe there.
Figure 14: A note written before a replacement is read back after it. (a) Writes ( ▲ ), read-backs ( ▼ ) and context replacements ( ∣ ) along 14 CAMP-II-4B Terminal-Bench episodes, one row per episode. (b) Writes and read-backs per episode for CAMP-II-4B and for the initial checkpoint on the same episodes. (c) Share of writes that land at least one, two or three steps before the replacement.
Language-model agents increasingly tackle long-horizon tasks whose interaction histories exceed the model's active context. Recent work has begun to use reinforcement learning to make memory control part of the policy, often relying on predefined memory tools within domain-specific training environments of relatively short horizons. This setup ties learned memory behavior to environment-specific interfaces that lie outside the base model's pre-training and must be learned from scratch, so even after post-training, agents struggle to use memory in long-horizon tasks. To address these limitations, we introduce Coding Agent Memory Gym (CAMG), a suite of long-horizon agentic-RL environments spanning Shop, Coding, DeepResearch, and AutoResearch. Alongside each environment's native task interface, CAMG provides executable shell access and an episode-persistent workspace, enabling agents to create, revise, search, and reuse files as memory throughout an episode. We also introduce CAMG-RL, which trains a single policy jointly across all four environments with fully asynchronous PPO, learning this file-based memory behavior directly from downstream task reward, and we train CAMG-RL-4B and CAMG-RL-9B from Qwen3.5 models of matching size. On SWE-bench Verified and MLE-bench Lite, CAMG-RL-4B and CAMG-RL-9B are competitive with Qwen3.5-35B-A3B and Qwen3.5-122B-A10B, respectively.
Long-horizon agents accumulate growing contexts during interaction, impairing performance and stability. Compact memory mitigates this problem by compressing and rewriting the history retained between model invocations. Learning what to retain typically relies on proximal policy optimization (PPO) with final task rewards, but sparse rewards provide little guidance for individual memory updates. This limitation motivates on-policy distillation (OPD), which supplies dense teacher supervision on student rollouts. For such supervision to be valid, the teacher must evaluate each sampled action under the same state in which it was generated. However, the context rewriting performed during memory compression can break this alignment. When sampled responses are retained and re-encoded for later invocations, flattening the interaction into a persistent history may cause the teacher to score the action under a state that the student never visited during rollout. The action therefore remains on-policy by provenance, but not necessarily by state. We therefore propose Memory-Aligned On-Policy Distillation (MemOPD). MemOPD records the inputs and sampled outputs of each model invocation, restores its original token positions and causal visibility, and packs the reconstructed invocations for efficient teacher scoring. The teacher provides full-vocabulary supervision at the sampled action positions, while PPO preserves the final task objective. Experiments verify state alignment across several context updates and show that it improves F1 by 7.0% over persistent-history teacher scoring in a matched control. Overall, MemOPD-3B improves F1 over PPO by up to 416.2%, while packing yields up to a 1.63x speedup in actor computation during training. The code for this work is publicly available at: https://github.com/TPssp/MemOPD.
Zhiyuan Liu, Tinghong Ye, Chenghao Liu +2
School of Advanced Manufacturing and Robotics, Peking University · Zhejiang University · Harbin Institute of Technology
Memory operations of long-horizon LLM agents are hard to supervise: an operation's value is unobservable when it is taken. But they are special -- they leave machine-readable evidence in the trajectory: retrieval hits and answer-time citations. Hindsight Memory-PRM exploits this audit trail twice: offline to train an operation-conditioned memory-utility critic, and online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit, propagated along version chains as an action-level proxy reward -- no per-operation human labels, no Monte-Carlo replay of continuations. On held-out LoCoMo a local 8B policy reaches 77.5% under a fixed shared reader, surpassing its API teacher (65.1%) and all reproduced external systems, at one eighth the context of Mem0's official operating point; on LongMemEval, 79.0%. Ablations attribute the gain to causal calibration rather than signal density, and the policy converges to a multi-version memory organization whose gains no tested open-loop baseline reproduces.
Haoxuan Jia, Yang Liu, Yingguang Yang +14
Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com +7