Large language models (LLMs) can improve task performance during learning, yet these gains do not necessarily reflect learners' ability to thereafter apply the knowledge and skills independently. We present AI Cosmos, an agentic infrastructure for situated learning that organizes learning around two complementary principles: scenario-based participation and cumulative assets of practice. AI Cosmos organizes learners and role-specialized agents within learning scenarios that integrate task-relevant knowledge and, when needed, executable workbenches. An asset-memory loop preserves the processes and outcomes of practice as reusable assets with explicit provenance and dependencies. We evaluate scenario-based interaction through two preregistered randomized controlled studies examining the acquisition of legal reasoning and programming competence, holding the same underlying LLM across conditions. AI Cosmos achieves greater learning gains than a conventional LLM in legal reasoning and higher mastery of unseen programming problems than an LLM-as-Teacher baseline.
Figures & tables
Figure 1: Overview of the Scenario–Asset Framework. AI Cosmos shifts AI-assisted learning from request–artifact exchange to scenario-based participation, and from one-off outputs to cumulative assets of practice, supporting competence development through sustained practice.
Figure 2: Architecture of AI Cosmos. The scenario control plane configures learning environments, the task execution plane coordinates participant interactions and workbench operations, and the asset–memory substrate preserves and reuses validated assets across scenarios.
Figure 3: Reorganizing the interaction as a structured scene improves bar-exam learning over a vanilla assistant. Three arms ( n=37 each) shared a single base model, differing only in interaction architecture. (a) Pre-test scores. Dots are individuals, markers denote mean ±95% CI. The arms were equivalent at baseline ( F(2,108)=0.88 , p=0.42 ), with AI Cosmos being numerically the lowest. (b) Immediate learning gain (post − pre). Gains increased monotonically across arms of increasing interaction structure ( 15.7→18.5→22.8 pp; linear-trend contrast p=0.018 ). The bracket highlights the a priori primary contrast, AI Cosmos vs. vanilla ( Δ=+7.0 pp, Cohen’s d=0.56 , bootstrap 95% CI [0.10,1.05] ; p=0.018 ); an ANCOVA adjusting for the pre-test reproduced this finding ( +4.5 pp, p=0.018 , partial η2=0.053 ). The LLM-as-Teacher arm did not differ significantly from vanilla ( p=0.12 ), locating the performance effect in the architecture rather than the persona prompt. (c) Interaction dynamics. Each point is one participant, and × are group centroids. AI Cosmos elicited ∼1.7× more dialogue turns ( 51.0 vs. 29.2 / 29.6 ; F(2,108)=21.8 , p<10−7 , η2=0.29 , g≈1.2 ) while requiring a statistically identical total input ( ∼7,100 characters, p=0.88 ). This yielded ∼40% fewer characters per turn ( 162 vs. 276 ; p≈0.001 , g=−0.80 ), which is a behavioral signature of active participation replacing one-shot delivery. (d) Cumulative gain over each arm’s own baseline at three time points within the delayed-test subsample ( n=18 per arm; mean, ± , 95% ,CI). AI Cosmos retained the largest net gain at one month ( +18.9 pp). Note: this subsample analysis is exploratory, and the omnibus was not significant ( p=0.62 ).
Figure 4: The advantage concentrates in the applied scenario-based choice, rather than open-ended argumentation. Each test was split into objective (scenario-based choice) and subjective (open-response) sections, each renormalized to 0 – 100 . (a) Learning gain by arm and section (bars indicate mean ±95% CI; points indicate individuals). On the objective section gains were ordered 18.2<21.2<26.5 pp ( F(2,108)=2.89 , p=0.060 , η2=0.051 ; ANCOVA-adjusted AI Cosmos vs. vanilla p=0.038 ). The subjective section improved equivalently across arms ( 5.6 – 7.8 pp; F(2,108)=0.23 , p=0.80 ). (b) AI Cosmos − baseline gain for each section against both controls (point estimate, 95% bootstrap CI; grey indicates vs. vanilla; blue indicates vs. LLM-as-Teacher). On the objective section the advantage over vanilla excludes zero ( +8.3 pp [1.8,14.9] ) and over the teacher baseline is positive but unresolved ( +5.3 pp [−1.5,12.2] ). On the subjective section, neither contrast significantly differs from zero ( +2.1[−4.9,9.3] ; −0.1[−7.1,7.1] ). The group × section interaction in a mixed-design ANOVA was not significant ( F(2,108)=0.95 , p=0.39 ). Thus, the localization is descriptive. Inter-rater reliability for subjective scoring was high (Cronbach’s α=0.95 – 0.96 ).
Figure 5: The advantage over the vanilla assistant is directionally consistent across learner subgroups. AI Cosmos − baseline difference in immediate learning gain (Cohen’s d , 95% bootstrap CI) against a, the vanilla assistant and b, the LLM-as-Teacher , for the full sample (diamond; shaded band and dotted line mark the overall estimate) and four strata from median splits on prior knowledge (median =46.7 ) and AI literacy. Against vanilla, the effect is positive in every stratum and most precisely estimated overall ( d=0.56[0.10,1.05] ). Against the teacher baseline, it remains positive throughout but lacks statistical resolution at this sample size (overall d=0.34[−0.10,0.82] ). Subgroup cells contain 15 – 22 participants per arm and are uncorrected for multiplicity.
Figure 6: AI Cosmos shows the largest learning gains across both aptitude strata. Participants were split at the median pre-test score ( 46.7 ) into (a) lower- and (b) higher-prior-knowledge strata . Translucent points are individual pre/post scores; filled markers are group means ±95% CI. The slope of each connecting line is the mean learning gain (annotated). Within each stratum, the arms were statistically equivalent at pre-test ( p=0.92 and p=0.39 ), yet AI Cosmos gained most in both ( +28.4 vs. +26.0 / +21.8 pp, lower; +14.5 vs. +9.6 / +10.6 pp, higher). As expected given their greater headroom, lower-prior-knowledge learners gained more across all arms (aptitude main effect F(1,105)=46.8 , p<10−4 ); the group × aptitude interaction was not significant ( F(2,105)=0.55 , p=0.58 ). Because the arms were baseline-equivalent within each stratum, the consistent advantage of AI Cosmos indicates the overall effect is not a statistical artifact of its slightly lower aggregate starting point.
Figure 7: Results of pre- and post-test. Three arms ( n=17 – 19 ) shared a single base model; both AI Cosmos and the teacher baseline withheld complete code, whereas the vanilla assistant was unconstrained. (a) Pre-test scores (dots represent individuals; markers denote mean ±95% CI). The arms were statistically equivalent at baseline ( F(2,51)=0.12 , p=0.89 ). (b) Post-test scores . AI Cosmos outperformed the teacher baseline by a near-significant margin ( Δ=18.2 points, g=0.56 ; Welch p=0.092 ) and performed comparably to the unconstrained vanilla assistant ( g=0.21 , n.s.). The LLM-as-Teacher baseline was the weakest arm overall. Given the sample size, the omnibus test was not significant ( F(2,51)=1.65 , p=0.20 ).
Figure 8: Post-test submission outcomes by arm. (a) Rates of scored submissions ( >0 ; pale bars) versus criterion mastery (grade ≥ G4; solid bars). AI Cosmos and the vanilla assistant earned partial credit at nearly identical rates ( 82% vs. 85% ) but diverged sharply on true mastery ( 55% vs. 38% ), explaining their converging mean scores despite disparate learning outcomes. (b) Proportion of scored submissions achieving genuine mastery. Among credit-earning submissions, 68% under AI Cosmos reached mastery, compared to 45% for vanilla and 32% for the LLM-as-Teacher baseline (Fisher’s exact: AI Cosmos vs. LLM-as-Teacher p=0.009 ; vs. vanilla p=0.12 ). Equivalently, AI Cosmos roughly halved the rate of hollow submissions—code that secures points without actually implementing the target algorithm.
Figure 9: Criterion mastery and failure modes. (a) Proportion of learners achieving criterion mastery (grade ≥ G4) on 0 , 1 , or 2 post-test problems. AI Cosmos learners mastered both problems (darkest band) at more than double the rate of controls ( 36.8% vs. 17.6% vanilla, 11.1% LLM-as-Teacher), significantly outperforming the LLM-as-Teacher baseline in per-student mastery count (Mann–Whitney p=0.024 ). (b) Post-test failure-mode rates. AI Cosmos achieved the highest flawless-implementation rate ( EOK , 52.6% ; Fisher vs. teacher p=0.019 ) and the lowest incomplete-template rate ( E02 ). Brute-force degeneration ( E06 , i.e., exploiting test cases without using the intended algorithm) appeared only under the unconstrained vanilla arm ( 5.9% ), never under either code-withholding condition.
Figure 10: Differential impact of the execution workbench across learner aptitudes. (a) AI Cosmos advantage over the LLM-as-Teacher baseline (Hedges’ g , 95% CI) on post-test scores and criterion mastery, by track. The advantage is significant for mastery among novices in the basic-algorithm track (Mann–Whitney p=0.039 ). Conversely, the effect is small and statistically unresolved for the stronger data-structure track, despite the latter covering harder algorithms (arm × track interaction p=0.34 ). (b)&(c) Pre- to post-test trajectories by arm within each track (faint lines represent individuals; markers denote mean ±95% CI; annotated values indicate mean change). In the basic-algorithm track, the LLM-as-Teacher baseline actually regressed from its pre-test level ( −22 points) while AI Cosmos achieved the highest gains. In the data-structure track, all arms improved, and performance gaps narrowed.
Figure 11: Within-arm correlation between prior aptitude (within-track pre-test z -score) and criterion mastery. Lines indicate OLS fits, and points represent vertically jittered individuals. (a) Vanilla and (b) LLM-as-Teacher baselines exhibit no predictive relationship ( r=0.08 and r=−0.06 , respectively; both n.s.). When an assistant can freely supply code, recorded outcomes decouple from actual learner ability. (c) AI Cosmos shows a strong, stable correlation ( r=0.59 , p=0.008 ; robust to leave-one-out resampling, r∈[0.53,0.65] ) that significantly differs from the LLM-as-Teacher baseline (Fisher’s z , p=0.040 ). Requiring learners to construct and repair algorithms restores construct validity, making the score a faithful measure of the person.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Vanilla LLM
LLM-as-Teacher
AI Cosmos
Test ( p )
( n=37 )
( n=37 )
( n=37 )
Female (%)
73.0
81.1
86.5
χ2,0.34
Prior legal training (%)
24.3
27.0
21.6
χ2,0.86
Age (years)
23.2±2.1
23.8±3.0
23.0±2.0
F,0.41
AI literacy (/170)
119.7±15.0
119.8±13.8
116.1±14.3
F,0.44
Self-efficacy, GSES
15.1±4.0
15.1±4.3
13.6±3.7
F,0.27
Appendix
Table 1: Participant statistics by arm, with tests of equivalence. Values are presented as mean ± s.d..
Vanilla LLM
LLM-as-Teacher
AI Cosmos
Test ( p )
( n=17 )
( n=18 )
( n=19 )
Female (%)
11.8
11.1
0.0
χ2 , 0.31
Age (years)
18.5 ± 0.6
18.7 ± 0.7
19.3 ± 1.9
F , 0.20
First-year (%)
94.1
88.9
84.2
χ2 , 0.64
Basic- algorithm track (%)
47.1
38.9
47.4
χ2 , 0.85
Data-structure track (%)
52.9
61.1
52.6
χ2 , 0.85
Appendix
Table 2: Participant characteristics by arm, with tests of equivalence. Values are mean ± s.d.. Pre-test scores are within-track comparable. The two tracks were assessed on different target algorithms. Pre-test problems below criterion reports the share of pre-test submissions an expert graded below G4 (not yet a substantially complete, correct implementation of the target algorithm).
Grade
Anchor
Description
G1
Target algorithm not formed
Implementation direction is inconsistent with the target algorithm, or no valid algorithmic intent can be identified. A brute-force or non-target solution is graded here even when it earns partial credit.
G2
Intent present, key errors
The target algorithm is attempted, but with defects in complexity, boundary handling, memory safety, or a critical invariant.
G3
Main flow present, details unstable
The target algorithm’s main framework exists but still contains key implementation errors.
G4
Framework essentially complete
Only minor, non-fundamental defects remain. (Criterion-mastery threshold.)
G5
Correct and complete
The target algorithm is implemented correctly and passes; corresponds to error code EOK .
Appendix
Table 3: The five-level mastery scale used by the expert re-graders. Each grade is anchored to how completely and correctly a submission realizes the problem’s target algorithm. The criterion-mastery threshold used in the main text is grade ≥ G4.
Code
Name
Definition
Severity
E00
Not submitted / incomplete
No valid submission, or no assessable result was produced.
medium
E01
Compilation / syntax error
Code does not compile, or the submitted file is not executable source.
high
E02
Incomplete skeleton
The basic structure of the target algorithm was never formed.
high
E03
Critical maintenance error
A core invariant or key maintenance step is wrong, yielding incorrect results.
high
E04
State / boundary error
The main flow is present, but a conditional branch or boundary case is mishandled.
medium
E05
Complexity / termination error
Complexity is inadequate or the termination strategy is unstable, failing some test cases.
high
Appendix
Table 4: Error codebook. Every submission is tagged [code:name] ; the code records the specific failure mode underlying the grade. “Severity” reflects how far the failure is from a correct target-algorithm implementation. Codes E01 – E07 and E00 are mutually informative failure modes; N01 / N02 are annotations rather than errors; sub-variant suffixes (e.g., -S0 / -S1 ) distinguish source-availability and intent sub-cases and are merged into their base code for the aggregate analysis.
Every major LLM agent framework gives the LLM the role of orchestrator; the model decides what to do next, when to call tools, and when to stop. We argue that token explosion, control-flow hallucination, and unreliable completion are not implementation bugs but architectural consequences of assigning the deterministic work of looping, branching, and sequencing to a probabilistic system. A better prompt or a stronger model cannot guarantee the reliability of the LLM agent. We therefore propose Agentic Programming, in which the program governs all control flow, and the LLM is itself part of it, an adaptive component we call LLM-as-Code and invoke only where a task calls for reasoning or generation. Within each call the model keeps full flexibility, but it cannot alter the program's execution path. With control in the program, the LLM's context is built from the execution history's call tree and forms a directed acyclic graph (DAG). Each call's context length is then determined by its call depth rather than by accumulation over steps. A case study of computer-use agents shows that the design is practical, not just a theoretical stance, substantially improving the stability of long visual operation sequences.
Large language models (LLMs) have become a central foundation of modern artificial intelligence, yet their lifecycle remains constrained by a rigid separation between training and deployment, after which learning effectively ceases. This limitation contrasts with natural intelligence, which continually adapts through interaction with its environment. In this paper, we formalise deployment-time learning (DTL) as the third stage in the LLM lifecycle that enables LLM agents to improve from experience during deployment without modifying model parameters. We present CASCADE (CASe-based Continual Adaptation during DEployment), a general and principled framework that equips LLM agents with an explicit, evolving episodic memory. CASCADE formulates experience reuse as a contextual bandit problem, enabling principled exploration-exploitation trade-offs and establishing no-regret guarantees over long-term interactions. This design allows agents to accumulate, select, and refine task-relevant cases, transforming past experience into actionable knowledge. Across 16 diverse tasks spanning medical diagnosis, legal analysis, code generation, web search, tool use, and embodied interaction, CASCADE improves macro-averaged success rate by 20.9% over zero-shot prompting while consistently outperforming gradient-based and memory-based baselines. By reframing deployment as an adaptive learning process, this work establishes a foundation for continually improving AI systems.
Siyuan Guo, Yali Du, Hechang Chen +2
aSchool of Artificial Intelligence, Jilin University. · bEngineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University. · cInternational Center of Future Science, Jilin University. +3
Learning to complete tasks in unfamiliar environments with unknown rules remains a key challenge for LLM agents. Current LLM agents often record their discoveries in prose, which may not provide a compact, explicit account of how the environment works. Inspired by how scientists organize observations into testable, predictive theories, we introduce Schema, an agent harness that organizes learning and action through interactive program induction. The LLM agent decides what to investigate and how to act, expressing its evolving understanding of the environment as executable programs. The harness consists of a persistent program workspace and a small set of interfaces for checking these programs against the interaction history, planning within them, and executing plans under step-by-step verification. Schema raises ARC-AGI-3 RHAE from 58.7% to 99.2% with the same base model, solves 100% of the public DiG-bench games, and reaches the median performance of the top-50 human players on MazeBench. Extensive analysis shows the effectiveness of Schema in unknown mechanism discovery, and ablations confirm the contribution of each component.
Guanning Zeng, Jiani Wang, Wenjie Ma +8
Carnegie Mellon University · Impossible Research · UC Berkeley