Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.
Figures & tables
Figure 1: CLE at a Glance. (A) Illustration of continual learning: an agent accumulates experience en through interaction and uses the update operator Un to modify its model parameters θn , persistent harness state sn , or both. (B) Summary statistics of CLE. (C) Completion rates across 17 harness–model configurations, measured as the percentage of all 600 levels passed. Trial 1 allows one submission per level without diagnostic feedback; Trial 2 allows one revision and resubmission following diagnostic feedback on an initial failure. (D) The change in completion rate from Trial 1 to Trial 2, in percentage points (pp).
Figure 2: Laddered Task Progression in CLE . In CLE , each task comprises five sequential levels. The agent advances to a new assignment only after the current level is passed; retained knowledge is reused and refined across levels. A level Li can contain multiple learning stages n,…,m , followed by stage m+1 in Li+1 . The expanded stage separates two interaction loops: actions produce environment observations, while submissions trigger rubric-based evaluation. After the first failed submission in K=2 , diagnostic feedback returns to the actor to support revision. Dashed arrows collect the resulting interaction history and available feedback as experience en . Together with the current learner state, en informs Un , producing the state for the next learning stage. Stage counts are schematic: i indexes assignments, and n indexes learning updates, not levels or submissions.
Figure 3: Continual Learning Capability Tiers. The tiers are cumulative and distinguish learning autonomy, cross-domain adaptation, research design, and problem selection.
Figure 4: Overview of CLE . Each human-authored task comprises five sequential levels linked by transferable knowledge. Agents interact with stateful applications through APIs, and an evaluation agent verifies submissions against human-authored rubrics. A per-level submission budget of K=1 allows one submission without diagnostic feedback; K=2 provides diagnostic feedback after a failed first submission and permits one resubmission .
Figure 5: A Five-Level Task Example: Ibex RISC-V Core Maintenance and Verification. The five levels present distinct assignments within a shared professional context, requiring the agent to apply and adapt accumulated knowledge and skills as task objectives and constraints change.
Figure 6: Agent–Environment Interaction in CLE. The policy selects actor action at from harness state st . The environment evolves from xt to xt+1 and returns observation ot+1 and reward rt+1 , which the harness uses with at to update st to st+1 through Ut . Here et=(at,ot+1,rt+1) is the interaction experience. On submission, the evaluator’s verdict determines rt+1 . Red arrows show environment dependencies and rewards; blue arrows show agent information and control.
Figure 7: Data Construction Pipeline. A domain expert drafts each task from a workflow in their own professional practice. The draft is audited by the review agent and cross-reviewed by two randomly assigned human reviewers, and it enters CLE only after clearing both gates. A rejection at either gate returns the task to its author for revision.
Figure 8: CLE Statistics. (A) The 120 tasks (600 levels) grouped into twelve occupational families and counted in tasks. (B) The 23,981 input files grouped by format family, counted in files. (C) The 16,125 human-authored checklist items by level, counted in items summed over the 120 tasks.
Table 1: Main results on CLE . Rows are grouped by agent harness, with one backbone model per row. Trial 1 allows one submission per level. Trial 2 allows a second submission after diagnostic feedback on a failed first submission. Completion and level pass rates are reported as percentages. / indicate whether higher/lower is preferred.
Figure 9: Completion Rates and Feedback Gains.
Figure 10: Completion versus Cost and Runtime. A–B show all 17 Trial 2 configurations; colours identify harnesses. Upper left is better. Bold labels and larger markers identify the observed Pareto frontier, traced by the step. Resource totals cover full evaluation runs.
Table 2: Comparison of agent benchmarks. ✓ indicates that the benchmark meets the criterion in that column. indicates that it meets the criterion only partially or for some tasks or settings. ✗ indicates that the feature is absent or is not a focus of the benchmark.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Concept
Perspective on Continual Learning
Update in (θn,sn)
Learning Settings and Paradigms
Continual Learning
Acquiring, refining, retaining, and applying knowledge from experience to improve later capabilities while preserving useful prior knowledge [ 52 , 49 ] .
θn→θn+1 and/or sn→sn+1 through Un .
Lifelong Learning
Emphasises the lifetime over which CL accumulates and transfers knowledge; often used interchangeably with CL [ 67 , 49 ] .
θn→θn+1 and/or sn→sn+1 across a continuing stream of experience.
Online Learning
Learns through sequential prediction, feedback, and update, typically aiming to minimise cumulative loss [ 55 ] . In CL, online updates combine adaptation to incoming experience with retention of useful prior knowledge.
θn→θn+1 and/or sn→sn+1 as experience en arrives.
Transfer Learning
Uses knowledge from source tasks or domains to improve learning or performance on target tasks or domains [ 48 ] . In CL, knowledge transfer can benefit both subsequent and previously learned tasks [ 35 ] .
Knowledge in θn or sn is reused and may be refined through Un to support other tasks.
Multi-Task Learning
Jointly learns multiple tasks through shared representations or parameters. In CL, joint training on current and replayed experience can support knowledge sharing and retention [ 11 , 57 ] .
θn→θn+1 using multi-task experience en ; sn may retain examples or task information.
Appendix
Table 3: Related Perspectives on Continual Learning. Selected learning paradigms and mechanisms, their relation to CL, and representative updates in Equation 1 . Arrows indicate update targets; equality indicates a fixed component. The memory, reflection, and skill entries show frozen-backbone examples.
Figure 11: Review-Agent Interface Used during Task Construction. A task contributor uploads a complete five-level task together with its supporting files and receives a conference-style verdict, review comments, and revision suggestions after the review agent completes its execution and design checks.
Judge
1
2
3
4
1 Human Annotators
100.00%
96.86%
97.27%
95.49%
2 Ours w/ DeepSeek V4 Flash
96.86%
100.00%
96.58%
92.90%
3 Claude Code w/ Claude Opus 5
97.27%
96.58%
100.00%
95.22%
4 Codex w/ GPT-5.6-Sol
95.49%
92.90%
95.22%
100.00%
Appendix
Table 4: Agreement on Individual Rubric Items. Each cell reports the percentage of rubric items receiving the same label from the corresponding pair of judges. Column numbers correspond to the numbered rows. Human annotators independently assess each rubric item and resolve disagreements to establish the reference labels. Judge 2 is CLE ’s evaluation agent with DeepSeek V4 Flash. Judges 3 and 4 provide additional comparisons using representative frontier agent–model combinations: Claude Code with Claude Opus 5 and Codex with GPT-5.6-Sol, respectively.
Judge
Total checklist items
Consistent items
Agreement (%)
Ours w/ DeepSeek V4 Flash
556
553
99.46
Appendix
Table 5: Stability of the evaluation agent across repeated runs. A fixed set of submissions is graded in three fresh sessions. Agreement is the percentage of the 556 rubric items that receive the same label in all three runs.
L1
L2
L3
L4
L5
Checklist Items (Mean)
18.74
22.07
25.83
30.90
36.83
Checklist Items (Median)
16.00
19.00
22.00
27.50
34.50
Checklist Items (Max)
75
85
82
87
112
Brief Length (Mean Words)
246
271
300
375
466
Rubric Sections (Mean)
4.10
4.39
4.52
5.21
5.74
Appendix
Table 6: Task Statistics by Level. Each column summarises checklist size, task-brief length, and the number of rubric sections across the 120 tasks at the corresponding level.
Figure 12: Cross-Level File References. Number and percentage of the 120 task ladders whose task specification or evaluation rubric at each level refers to files from an earlier level.
Figure 13: Task Progression across Five Levels. (A–B) Distribution of tasks by the number of levels completed (0–5) in Trial 1 ( K=1 ) and Trial 2 ( K=2 ). Each row sums to 120 tasks; darker cells indicate larger counts. (C) Cumulative pass rates in Trial 2, measured over all 120 tasks at each level. Colours identify harnesses. The black curve shows the median across the 17 agent configurations. Claude Code with Opus 5, the configuration with the highest overall completion, is highlighted.
Figure 14: Conditional Progression between Adjacent Levels. Each cell shows the percentage of tasks that passed Lk+1 among those that passed Lk . Values are rounded to one decimal place. A dash indicates that no task passed Lk .
Figure 15: Feedback and Revision Gains at Each Level. Cells show Trial 2 minus Trial 1 pass rate in percentage points across all 120 tasks. Blue indicates an increase, red a decrease, and the neutral midpoint no change. Differences use integer pass counts before rounding.
Figure 16: Harness Comparison on Six Shared Backbones. Grouped bars compare Hermes-Agent and OpenClaw under the same submission budget. (A–B) Overall completion out of 600 levels. (C–D) Full-task completion out of 120 tasks.
Model
Trial 1
Trial 2
Compl.
Per-Level Pass
Actor Tok.
Time
Cost
Compl.
Per-Level Pass
Actor Tok.
Time
Cost
L1
L2
L3
L4
L5
L1
L2
L3
L4
L5
Gemini 3.6 Flash
21.00%
50.00%
27.50%
13.33%
9.17%
5.00%
389.18M
28.40h
$198.62
23.50%
48.33%
31.67%
19.17%
10.83%
7.50%
602.72M
51.70h
$328.79
Gemini 3.7 Flash
25.33%
55.00%
33.33%
19.17%
10.83%
8.33%
486.25M
46.10h
$215.64
33.67%
62.50%
40.83%
30.00%
19.17%
15.83%
901.82M
102.20h
$394.84
Gemini 3.8 Flash
21.17%
51.67%
28.33%
14.17%
8.33%
3.33%
795.78M
62.90h
$269.20
30.67%
61.67%
39.17%
25.00%
15.83%
11.67%
1741.21M
142.50h
$569.26
Appendix
Table 7: Gemini model-generation comparison with Hermes-Agent. All variants use the full 120-task protocol and the metrics of Table 1 . Gemini 3.7 Flash reuses the main-table runs.
Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.
Parth Asawa, Christopher M. Glaze, Gabriel Orlanski +7
UC Berkeley · Snorkel AI · University of Wisconsin-Madison +1
Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.