Continual-ARC: Measuring Skill Acquisition over a Working Lifetime
Organizations: Unify
Abstract
Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.
Figures & tables
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Kept in the harness’s notes | Kept in the student’s weights | |
|---|---|---|
| Taught by the student | Q: the harness as shipped QE: given expert guidance, the program library’s procedure and playing rules | – |
| Taught by GLM-5.3 Flash | QT: the teacher plays the stream and writes the notes, and the student plays from them QTE: the teacher given the guidance QETE: both given it | QDE: imitates the teacher’s move in each state the student met, the teacher reading expert guidance |
| track | tasks | stream | held-out | ||||
|---|---|---|---|---|---|---|---|
| ARC-AGI-1 | 25 | 200 | 100 | 4 | 3 | 30 | 2 |
| file | the texts it holds |
|---|---|
| systems/protocol.py | the grid protocol’s other messages: the replies to an answer or a request, the verdicts, the re-prompts and the budget notice; the ask after every verdict of methods QA, GA and H-A ; the expert guidance of methods QE, GE, H-E , H-E 0 and QETE and its asks |
| systems/prime_agent.py | prime-agent’s one addition to the rules, its fact about its store |
| systems/bare.py | the demo-cache floor’s cached pairs, and the program library’s addition to the rules, its kept program, its checks and its re-prompt |
| systems/arc3_protocol.py | the third track’s protocol: its rules, messages, feedback and re-prompt |
| references.py | c1, the page of playing rules that DAgger’s teacher reads |
| oracle.py | the rules and the note with which DAgger’s teacher labels the student’s calls |
| method | ||||||
| Qwen3.5-9B, notes in the harness | ||||||
| Q | -0.02 | [-0.09, 0.04] | -0.01 | [-0.04, 0.03] | -0.01 | [-0.05, 0.03] |
| QA | -0.05 | [-0.11, 0.01] | +0.02 | [-0.02, 0.07] | 0.00 | [-0.05, 0.04] |
| QE | +0.05 | [-0.06, 0.16] | +0.03 | [-0.08, 0.13] | -0.01 | [-0.05, 0.03] |
| Qwen3.5-9B, notes the teacher writes | ||||||
| QT | +0.05 | [-0.02, 0.14] | +0.06 | [-0.01, 0.14] | +0.03 | [-0.02, 0.07] |
| method | users | ||||||
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B, weights | |||||||
| QDE | 1 | +0.43 | [0.33, 0.54] | +0.57 | [0.48, 0.67] | +0.29 | [0.23, 0.35] |
| QDE | 2 | +0.47 | [0.38, 0.56] | +0.55 | [0.46, 0.64] | +0.40 | [0.34, 0.47] |
| QDE | 4 | +0.53 | [0.46, 0.60] | +0.51 | [0.45, 0.57] | +0.39 | [0.33, 0.46] |
| Qwen3.5-9B, baselines | |||||||
| QP | 1 | +0.43 | [0.25, 0.64] | +0.65 | [0.46, 0.86] | +0.03 | [-0.02, 0.09] |
| stream | held out | all | |||||
|---|---|---|---|---|---|---|---|
| model, effort | private | public | private | public | private | public | calibrated |
| GPT-6 Luna, low | 31.5 | 31.5 | 21.0 | 20.0 | 26.3 | 25.8 | 31.4 |
| GPT-6 Luna, medium | 54.0 | 54.0 | 56.0 | 56.0 | 55.0 | 55.0 | 58.7 |
| GPT-6 Luna, high | 59.0 | 59.0 | 66.5 | 66.0 | 62.8 | 62.5 | 67.7 |
| Claude Opus 5.5, max | 97.0 | 97.0 | 100.0 | 100.0 | 98.5 | 98.5 | 97.8 |
| GLM-5.3 Flash, medium | 23.0 | 29.5 | 30.0 | 24.0 | 26.5 | 26.8 | 29.2 |
| ARC-AGI-2 | ARC-AGI-3 | |||
|---|---|---|---|---|
| model | solved | instances | solved | mean charge |
| Kimi K3.1 Pro | 29 | 100 | 25 | 6.8 |
| Kimi-K3 | 56 | 50 | 35 | 6.0 |
| Qwen3.5-9B | 22 | 100 | – | – |
| stream | first visits | returns | ||||
|---|---|---|---|---|---|---|
| method | solved | charge | solved | charge | solved | charge |
| KC 0 | 25.8 | 6.69 | 5 | 6.29 | 24.8 | 6.75 |
| KC | 32.5 | 6.04 | 6 | 5.99 | 31.4 | 6.05 |
| KP | 25.8 | 6.32 | 1 | 7.82 | 28.6 | 6.11 |
| K | 65.8 | 4.88 | 12 | 4.13 | 63.8 | 4.98 |
| K 0 | 65.8 | 4.79 | 9 | 5.11 | 66.7 | 4.75 |
| Model | Provider | Input | Cached input | Output |
|---|---|---|---|---|
| Qwen3.5-9B | PrimeIntellect | 0.020 | 0.018 | 0.030 |
| GLM-5.3-Flash | Z.ai | 0.150 | 0.030 | 0.500 |