cs.AISep 26, 2026
SaveSCLATE: a Substrate for Continual-Learning Agent Training and Evaluation
Organizations: Apple
Abstract
Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requires interleaving tasks with agent-side events such as session stop and start, crons, and memory consolidation. Yet existing benchmarks and training frameworks schedule only the benchmark's own events, leaving each benchmark and agent pair to build a custom scheduling loop. We present SCLATE, an execution substrate where benchmarks and unmodified agents each add their events to one open event scheduler through an adapter. A hybrid simulated clock runs these events on a shared timeline, flowing in real time while the agent works and skipping idle gaps, which compresses a month-long scenario into hours. SCLATE also serves as a rollout engine that runs any agent's harness and memory unmodified, recording the tokens and log probabilities of every model call through an in-container proxy. We port seven benchmarks to SCLATE and compare ten unmodified harness and memory configurations head to head on ten models. The comparison shows that an added memory system does not reliably beat the harness's native memory and that models differ widely in how they use the same harness and memory. We then post-train Qwen3.5-4B through unmodified harnesses and memory systems. The model learns to use both, reading 6.8x fewer file lines with a 16.7-point higher SWE-bench Verified pass rate, and writing richer memory records, while its held-out MetaClaw accuracy rises by up to 11.8 points.
Figures & tables
Figure 1 : Continual-learning lifecycle across multi-session horizons. (a) Existing frameworks run only a built-in assistant or schedule only the benchmark’s own events, leaving agent-side events off the timeline. (b) SCLATE ’s open event scheduler lets the benchmark and the agent register their events on one simulated clock, so the same runs serve multi-session evaluation and post-training.
Figure 2 : SCLATE architecture. Adapters decouple benchmarks and agent harnesses into a uniform event stream, while the host orchestration manages container lifecycles and training batches. Inside replicated sandboxes, an open event scheduler coordinates benchmark and agent events on a hybrid simulated clock, while an in-container proxy captures token-faithful trajectories for post-training.
Figure 3 : Parity between the upstream Gaia2-CLI stack and SCLATE , with the same agent and judge. (a) Every scheduled environment event is delivered on time on all three stacks. (b) On the Time scenarios, idle-time jumps compress simulated time by up to 4.3 ; the label gives the share of runs that clock jumped. (c) Pass rate on 25 Time and 25 Adaptability scenarios with Haiku-4.5, each scenario repeated 7 times on every stack (175 runs per bar, 1,050 runs in total). (d) An SCLATE run and a GAIA-2 run of the same scenario disagree about as often as two runs of the same stack.
| Claude Code | Hermes | Codex | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | No-Mem | Native | Mem0 | Hindsight | GBrain | No-Mem | Native | Mem0 | Hindsight | Native |
| Haiku-4.5 | 40.2% | 46.0% | 69.7% | 58.7% | 56.7% | 43.6% | 59.8% | 61.9% | 61.0% | 44.8% |
| Sonnet-5 | 43.9% | 51.2% | 57.5% | 50.9% | 62.1% | 43.1% | 47.4% | 59.0% | 50.3% | 56.7% |
| Opus-4.6 | 48.8% | 65.0% | 66.5% | 71.1% | 74.0% | 46.5% | 68.5% | 65.0% | 66.8% | 51.5% |
| Opus-4.8 | 48.6% | 60.4% | 66.8% | 74.0% | 67.6% | 45.4% | 64.2% | 64.7% | 61.9% | 48.6% |
| Gemini-3.7F | 63.9% | 79.2% | 79.2% | 81.8% | 79.5% | 73.4% | 77.8% | 79.5% | 80.1% | 73.4% |
Table 1 : MetaClaw pass accuracy over 30-workday arc (346 rounds) across ten models and ten harness and memory configurations. Bold marks the best accuracy within each harness-model pair.
Figure 4 : Pass rate (%) of each model under each harness and memory configuration; CC is Claude Code. Left: GAIA-2 and AppWorld, with each harness on its native memory. No harness is best for every model. Right: LongMemEval-s and PersonaMem-128k, with Claude Code and Hermes on native memory or an added memory system. Claude Code recalls more than Hermes on LongMemEval-s, and the two are close on PersonaMem-128k. Hermes with Hindsight is not shown on PersonaMem-128k, where its per-turn memory updates exceed the 900 s turn limit.
| In-distribution (days 1–20) | Held-out (days 21–30) | |||||||
|---|---|---|---|---|---|---|---|---|
| Harness & Memory | Base | Trained | (pp) | Base | Trained | (pp) | ||
| CC & no-memory | 37.4% | 52.4% | 0.001 | 9.2% | 16.8% | 0.039 | ||
| CC & Mem0 | 42.7% | 67.4% | 0.001 | 10.1% | 21.8% | 0.023 | ||
| CC & Hindsight | 51.1% | 52.0% | 0.91 | 16.0% | 16.8% | 0.83 | ||
| Hermes & no-memory | 29.1% | 36.6% | 0.27 | 8.4% | 18.5% | 0.047 | ||
| Hermes & Mem0 | 31.3% | 58.6% | 0.004 | 8.4% | 16.0% | 0.17 | ||
Table 2 : MetaClaw post-training on a Qwen3.5-4B model across six configurations. Columns show base and post-trained (epoch 5) accuracy and the improvement ( ) on the in-distribution and held-out partitions. is a paired wild cluster bootstrap- test with simulated days as clusters.
Figure 5 : Memory facts written by the base and trained models as Hindsight’s fact extractor on Hermes (5,027 and 3,617 facts). (a) Median length: training lengthens the metadata (+111%) but not the statement (+2%). (b) Facts containing the optional purpose clause or a procedural rule.
Figure 6 : SWE-Gym post-training and downstream SWE-bench Verified generalization on SCLATE . (a) GRPO rollout mean reward across 73 training iterations with exponential moving averages ( ). (b) Pass@1 on SWE-bench Verified comparing base and post-trained checkpoints across harnesses. (c) Context use under Claude Code on 42 matched tasks, where the trained model reads fewer lines, avoids whole-file reads, and rarely hits the context limit.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| As deployed | Long horizon | Faithful execution | ||||
| System | Harness | Memory | Multi-session | Time accel. | Shared clock | Agent-side events |
| Evaluation | ||||||
| Harbor ( Harbor Framework Team, 2026 ) | ||||||
| GAIA-2 / ARE ( Froger et al., 2026 ) | ||||||
| MetaClaw ( Xia et al., 2026 ) | ||||||
| Post-training | ||||||
Table 3 : Requirements for evaluating and training a continual-learning agent as deployed, ordered so that each depends on those to its left. Long horizons need multi-session scenarios and time acceleration, an accelerated clock must be shared by the benchmark and the agent, and agent-side events, such as session stop and start, state save and restore, crons, and memory consolidation, must then run on that shared clock, scheduled alongside the benchmark’s events. supported, partial, not supported or not described in the system’s paper or documentation. A partial shared clock is real wall-clock time; GAIA-2 evaluates only its own scaffold, and rLLM agents are written against its own agent interface; MetaClaw runs only OpenClaw with a rewritten prompt, carries skills but not agent processes across workdays, and schedules its own training in idle windows on real time.
| Method & Signature | Systems Semantics |
|---|---|
| BenchmarkAdapter Substrate (Scenario Environment and Evaluation) | |
| setup(scenario, state_dir) | Provisions initial mock application state (mail, calendar, git). |
| source_manifest(ctx) / event_sources(ctx) | Yields scenario tasks, inter-session world updates, session boundaries, and grading, either as declarations the scheduler instantiates or as ready-made sources. |
| world_provider() | Optionally exposes a read-only view of application state as .world to event sources and graders. |
| grading_policy() | Declares which turns are graded and when, how turns that are not graded are discharged, and whether a graded turn may be regraded, so the scheduler carries no benchmark-specific grading logic. |
| Grader.evaluate(ctx) -> Judgment | Independent oracle that scores each turn the grading policy selects against the task rubric, reading the dispatch context including the world state, and returns the verdict and feedback. |
Table 4 : Plug-in interface contracts in SCLATE . The substrate communicates with benchmark adapters, agent adapters, and event sources through uniform interfaces, decoupling environment definitions and agent harnesses from the open event scheduler. An Agent Adapter covers a harness together with its memory system, so it also declares how their durable state is saved and restored. Every turn is delivered without blocking, so events can reach the agent while a turn is still running.
| Event Source Class | Contributor | Systems Responsibility and Fire Dynamics |
| UserInputSource | Benchmark | Yields multi-turn task prompts and user instructions dynamically as prior turns are graded. |
| SessionStartSource | Shared | Coordinates multi-phase session opens, advancing time, invoking StartAgent , and folding session-init turns. |
| SessionEndSource | Shared | Manages session closes in two stages: delivering the handoff notification, then invoking ShutdownAgent once settled. |
| AppStateConditionedSource | Benchmark | Polls ctx.world during active flow mode, firing immediately when benchmark application predicates are satisfied. |
| GradingSource | Benchmark | Schedules evaluation fires across turns and session boundaries, invoking the independent oracle grader. |
| CronSource / WakeupSource | Agent | Emits recurring wakeup targets on the timeline to pace native agent background crons and periodic maintenance. |
Table 5 : Shared event source catalog in SCLATE .
| Effect Primitive | Operational Semantics and State Mutations |
|---|---|
| Deliver | Delivers structured task instructions, user prompts, or system notifications directly to the agent over filesystem IPC channels. |
| RunHook | Executes a custom callback defined by an adapter to mutate environment state, seed database tables, or evaluate grading rubrics. |
| AsyncReserve | Switches the scheduler into real-time flow mode to wait for an asynchronous hook to finish, advancing time continuously without blocking the scheduler loop. |
| AwaitWakeup | Pauses timeline progression until the agent process settles into an idle state, becomes alive, or triggers an internal wakeup alarm. |
| StartAgent | Spawns the agent harness process along with any configured memory system inside the execution sandbox. |
| ShutdownAgent | Sends graceful termination signals to shut down the running agent harness and sidecar processes before closing the session. |
Table 6 : Effect primitives in SCLATE ’s Effect Runner.
| MetaClaw | |
|---|---|
| Runs | 100 |
| Simulated span, median | 40.0 days |
| Real span, median | 5.9 h |
| Simulated over real time (all runs combined) | 142 |
| Benchmark-side events dispatched | 77,700 |
| Agent-side events dispatched | 12,400 |
Table 7 : SCLATE scheduler measurements over every counted final-eval run, from the raw scheduler traces. Benchmark-side events are user turns, grading, and GAIA-2 environment events; agent-side events are session starts and stops, including the handoff and init turns, and GBrain’s nightly consolidation. GAIA-2 scenarios are single-session, so their runs dispatch no agent-side events. The scenario-specified delay is checked on the 6,999 GAIA-2 environment events whose parent the judge matched.
Figure 8 : Multi-layer simulated time grounding evidence across harnesses. Verbatim excerpts from MetaClaw 30-workday evaluation runs showing synchronized simulated time across (a) harness system prompts injected at session boundaries, (b) active agent tool executions discovering time via shell commands under libfaketime , and (c) memory systems recording timestamped entries.
Figure 9 : MetaClaw under Claude Code with GBrain and Sonnet-5. (a) Simulated calendar day versus real hours over the full arc, with one dot per work day. (b) One work day on a broken time axis, where flow segments are drawn at real scale, init turns at one-third scale, and jumps collapsed into labeled breaks, showing benchmark rounds and agent-side events such as the handoff, the per-round recall hook, and GBrain’s nightly consolidation on the same scheduler.
Figure 10 : GAIA-2 Adaptability scenario 155 under Claude Code with Sonnet-5. (a) Simulated versus real time from the scheduler trace, flowing at real-time rate throughout. (b) The oracle event DAG, the scheduler’s dispatches and notifications, and the agent’s turns and tool calls on one simulated time axis, where bold ticks are writes to application state labeled with the oracle node the judge matched.
Figure 11 : GAIA-2 Time scenario 80 under Claude Code with Sonnet-5. (a) Simulated versus real time, where the clock follows the real-time diagonal while the agent works and jumps vertically across idle gaps. (b) The agent’s message A5 arms timers for four scheduled replies, and the scheduler jumps the clock to each reply that falls due while the agent is idle, with hatched bands marking the jumps.
Figure 12 : GAIA-2 Adaptability scenario 148 under Claude Code with Sonnet-5. (a) Simulated versus real time, flowing at real-time rate throughout. (b) The oracle event DAG, the scheduler’s dispatches, and the agent’s turns and tool calls, with the dashed box marking the window shown in (c). (c) Four events dispatched within 1.2 seconds, where the first opens a turn and the other three wait in Claude Code’s input queue until the harness consumes them at its next step boundary.
| Model | Reasoning effort |
|---|---|
| Haiku-4.5 | high |
| Sonnet-5 | high |
| Opus-4.6 | high |
| Opus-4.8 | high |
| Gemini-3.7F | provider default |
| GPT-5.4 | none |
Table 8 : Reasoning effort per model, the same under every harness.
| Benchmark | Upstream Size | Evaluated Tasks ( ) | Task Domain & Environment |
|---|---|---|---|
| MetaClaw ( Xia et al., 2026 ) | 346 tasks (30 workdays) | All 346 tasks | Multi-session developer workspace |
| GAIA-2 ( Froger et al., 2026 ) | 800 tasks (5 categories) | All 800 tasks | Asynchronous personal assistant apps |
| AppWorld ( Trivedi et al., 2024 ) | 750 tasks (585 test) | All 585 test tasks (168 normal, 417 challenge) | Multi-app transactions |
| LongMemEval ( Wu et al., 2025 ) | 500 tasks (oracle/s/m) | 100 tasks ( split) | Multi-session conversational recall |
| PersonaMem ( Jiang et al., 2025 ) | 5,990 tasks (32k–1M) | 100 tasks (128k tier) | Long-horizon persona tracking |
| SWE-Gym ( Pan et al., 2024 ) | 2,438 tasks | 293 tasks (SkyRL-v0 subset) | GitHub pull request resolution |
Table 9 : Comprehensive specifications of the seven ported benchmark workloads evaluated in SCLATE . Subsampled recall suites evaluate the first items of the listed split, and every arm of a benchmark evaluates the same items.
Figure 13 : Frequency of harness session-search tool invocations across models and driving harnesses on MetaClaw. Gemini-3.7F calls session search substantially more than other models across all harnesses, consistent with its elevated no-memory baseline performance.
| Claude Code | Hermes | Codex | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | No-Mem | Native | Mem0 | Hindsight | GBrain | No-Mem | Native | Mem0 | Hindsight | Native |
| Haiku-4.5 | 34 | 38 | 77 | 67 | 99 | 78 | 242 | 270 | 256 | 42 |
| Sonnet-5 | 51 | 67 | 159 | 104 | 179 | 76 | 197 | 442 | 310 | 70 |
| Opus-4.6 | 45 | 57 | 122 | 100 | 107 | 59 | 148 | 252 | 203 | 31 |
| Opus-4.8 | 41 | 42 | 176 | 109 | 144 | 65 | 166 | 209 | 198 | 34 |
| Gemini-3.7F | 227 | 155 | 349 | 228 | 295 | 317 | 588 | 625 | 528 | 322 |
Table 10 : MetaClaw total token usage (millions) over the 30-workday, 346-round arc for each model and harness–memory configuration of Table 1 . Totals count uncached input, cache reads, cache writes, and output across all agent and memory-system calls recorded by the SCLATE proxy, which attributes each call to the agent or to the memory system itself. Across the 38 memory-backend cells with complete usage records, the memory system’s own calls account for a median of 4.2% of tokens, at most 13.8% for Claude Code with Hindsight and under 0.3% for GBrain. Cell color intensity scales with tokens within each harness family. “–” marks Claude Code on self-hosted models. The proxy records these calls, but the self-hosted endpoint’s streamed responses omit the usage fields this table sums.
| Harness | Model | Execution | Search | Adaptability | Ambiguity | Time | Mean |
|---|---|---|---|---|---|---|---|
| Claude Code | Haiku-4.5 | 36.9% | 70.0% | 18.8% | 13.1% | 9.4% | 29.6% |
| Hermes | Haiku-4.5 | 53.8% | 56.2% | 15.0% | 14.4% | 13.8% | 30.6% |
| Codex | Haiku-4.5 | 41.2% | 62.5% | 16.2% | 12.5% | 10.0% | 28.5% |
| Claude Code | Sonnet-5 | 81.9% | 92.5% | 37.5% | 73.8% | 28.1% | 62.8% |
| Hermes | Sonnet-5 | 73.8% | 92.5% | 26.9% | 58.8% | 29.4% | 56.2% |
| Codex | Sonnet-5 | 78.1% | 86.9% | 28.8% | 56.9% | 13.8% | 52.9% |
Table 11 : Complete GAIA-2 Pass@1 (%) breakdown across production harnesses and models on the full 800-scenario benchmark (160 scenarios per capability category). Cell color intensity scales continuously with accuracy within each harness family (terracotta for Claude Code, blue for Hermes, and slate for Codex). Bold indicates the highest accuracy for each model across harnesses.
| Harness | Model | Normal (168) | Challenge (417) | Overall (585) |
|---|---|---|---|---|
| Claude Code | Haiku-4.5 | 60.7% | 43.4% | 48.4% |
| Hermes | Haiku-4.5 | 60.1% | 44.6% | 49.1% |
| Codex | Haiku-4.5 | 64.9% | 43.4% | 49.6% |
| Claude Code | Sonnet-5 | 70.8% | 58.0% | 61.7% |
| Hermes | Sonnet-5 | 73.2% | 52.0% | 58.1% |
| Codex | Sonnet-5 | 61.3% | 48.4% | 52.1% |
Table 12 : AppWorld Pass@1 (%) on the complete 585-task test suite (168 normal and 417 challenge tasks) across production harnesses for Haiku-4.5, Sonnet-5, and DS-V4.1. Cell color intensity scales continuously with accuracy within each harness family (terracotta for Claude Code, blue for Hermes, and slate for Codex). Bold indicates the highest score per split for each model.
| Configuration Field | Setting / Value |
|---|---|
| Base Model | Qwen3.5-4B |
| Reinforcement Learning Algorithm | GRPO |
| Harnesses Evaluated | Claude Code, Codex |
| Learning Rate | |
| Context Window ( ctx ) | 50,000 tokens |
| Prompts per Batch | 4 |
Table 13 : Hyperparameter and infrastructure configuration for SWE-Gym post-training on SCLATE .
Figure 14 : Example of tokens and log probabilities captured by the in-container capture proxy, from one model call of a SWE-Gym GRPO training rollout with Qwen3.5-4B (instance conan-io__conan-15931 , training step 20, model call 4 of 36, rollout reward 1.0). The proxy marks which tokens the model generated and which it did not. Grey tokens were not generated by the model. They are the prompt tokens added since the previous call, the result of the agent’s last tool call and the chat template, and the trainer masks them out of the loss (345 tool-result tokens are elided). The remaining tokens are the model’s completion, each shaded by the log probability the proxy recorded for it at sampling time, and these are the tokens that receive the grader’s reward during training.
Figure 15 : Task completion accuracy against OPD training epoch for all six agent and memory configurations. The top row shows in-distribution performance (days 1–20) for (a) Claude Code and (b) Hermes, and the bottom row shows held-out generalization (days 21–30) for (c) Claude Code and (d) Hermes. Each point is a separate evaluation over the full 346-round arc, at the base model and at the end of each training epoch.
| Configuration Field | Setting / Value |
|---|---|
| Student Model | Qwen3.5-4B |
| Teacher Model | Qwen3.5-35B-A3B (MoE), served via SGLang, 65,536-token context |
| Training Algorithm | On-policy distillation via advantage shaping |
| OPD KL Coefficient ( ) | 1.0 |
| Auxiliary Loss Terms | Reference KL 0.001 ( low_var_kl ); entropy coefficient 0.0 |
| Learning Rate | , constant, no warmup |
Table 14 : Configuration and infrastructure for MetaClaw OPD post-training on SCLATE .
Figure 16 : Reverse KL between student and teacher logged across training for all six agent and memory configurations, captured through unmodified harnesses by the in-container proxy. Raw per-step values are shown under an exponential moving average.
Figure 17 : Task completion accuracy against OPD training epoch under the within-day partition, Hermes paired with Mem0. The model trains on the first two-thirds of each day’s rounds (242 rounds, gray) and is evaluated on each day’s final third (104 rounds, green).
Figure 18 : The procedural-rule gain is structural rather than a by-product of longer facts, Hermes agent with the Hindsight backend, both arms with the memory tools forced on. (a) Facts naming at least one procedural rule within fact-length strata; the trained store leads in every stratum. (b) The distribution over the number of distinct rules a fact names shifts rather than scaling, with facts naming no rule falling pp. (c) Per-rule incidence; all five rise, least for the backup rule (P4).
Figure 19 : Verbatim memory writes from the two arms on three of the days where the distilled model’s gain in rounds passed is largest, Hermes agent with the Hindsight backend. Within each day, each cell is the most completely filled record that store wrote, scored identically for both arms. The distilled records name the artifact produced, attribute it to the agent, and state a purpose; the untrained records restate the instruction or describe the user. The round tally counts the rounds passed on that day and is not tied to the round that produced the record. Fact text is verbatim but not de-duplicated.