Can Large Language Model Agents Learn by Watching Tutorial Videos?
Organizations: University College London
Abstract
Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduce Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that consolidates tutorial videos into a persistent, structured tutorialbook memory through a two-stage adversarial process that harvests hard, video-grounded questions and uses them to refine the memory. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model's reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL outperforms all evaluated baselines and improves monotonically as more videos are consumed, achieving up to 113% and 91% relative gains over the corresponding vanilla MLLMs on the training and test splits, respectively. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in 69% of head-to-head games. Evaluation in a second domain further demonstrates the generality of TAL. Together, these results highlight instructional videos as a scalable resource for parameter-update-free agent continual learning.
Figures & tables
| Method | Training Set | Test Set | ||||||
| # Demonstration Videos | 12 | 24 | 36 | 48 | 12 | 24 | 36 | 48 |
| Vanilla MLLMs (Direct Answering) | ||||||||
| GPT-5.2 | 31.61% | 31.61% | 31.61% | 31.61% | 37.27% | 37.27% | 37.27% | 37.27% |
| Kimi K2.5 | 38.51% | 38.51% | 38.51% | 38.51% | 31.82% | 31.82% | 31.82% | 31.82% |
| Gemini 3 Flash | 47.70% | 47.70% | 47.70% | 47.70% | 48.18% | 48.18% | 48.18% | 48.18% |
| Claude Sonnet 4.6 | 41.95% | 41.95% | 41.95% | 41.95% | 37.27% | 37.27% | 37.27% | 37.27% |
| Method | Training Set (174 QA) | Test Set (110 QA) | ||||||
| # Demonstration Videos | 12 | 24 | 36 | 48 | 12 | 24 | 36 | 48 |
| Vanilla (no memory) | 53.1 1.0 | 57.6 2.3 | ||||||
| Transcript-RAG | 55.9 0.5 | 58.2 1.1 | 60.0 2.3 | 64.6 3.1 | 59.7 1.9 | 59.4 0.4 | 59.7 1.5 | 60.0 2.6 |
| Video-RAG (caption retrieval) [ 38 ] | 58.0 0.9 | 59.6 0.3 | 59.4 2.7 | 62.1 1.7 | 58.5 0.9 | 57.9 0.9 | 62.1 1.5 | 64.5 1.5 |
| Video-RAG (+retrieved frames) [ 38 ] | 59.2 2.0 | 57.3 2.2 | 60.0 2.7 | 62.1 2.6 | 60.3 1.5 | 61.8 3.7 | 62.4 2.3 | 63.6 0.7 |
| mem0 [ 17 ] | 57.1 0.5 | 55.7 1.4 | 58.2 1.0 | 59.6 0.7 | 62.7 2.0 | 63.0 1.7 | 62.4 0.4 | 60.9 1.5 |
| Method | Training Set (53 QA) | Test Set (55 QA) | ||||||
| # Demonstration Videos | 4 | 8 | 12 | 16 | 4 | 8 | 12 | 16 |
| Vanilla (no memory) | 38.4 5.4 | 21.2 3.7 | ||||||
| Transcript-RAG | 62.3 2.7 | 66.7 0.9 | 72.3 0.9 | 78.6 0.9 | 27.3 2.6 | 35.8 0.9 | 29.7 3.1 | 30.9 2.6 |
| Video-RAG (caption retrieval) [ 38 ] | 68.6 0.9 | 72.3 2.4 | 81.8 0.9 | 84.3 3.2 | 33.9 0.9 | 38.2 8.3 | 29.7 3.1 | 38.8 4.8 |
| Video-RAG (+retrieved frames) [ 38 ] | 67.3 1.8 | 74.8 3.2 | 84.3 0.9 | 86.2 1.8 | 32.1 0.9 | 36.4 1.5 | 27.3 1.5 | 33.9 2.3 |
| mem0 [ 17 ] | 54.1 3.6 | 57.2 0.9 | 67.9 1.5 | 71.7 1.5 | 29.7 5.6 | 33.9 1.7 | 33.9 4.8 | 33.9 1.7 |
| Horizon | Win Rate (TAL/Van) | Mean ACPL (TAL/Van) | Median ACPL (TAL/Van) | 5% Trimmed Mean ACPL (TAL/Van) |
| move2 | 50.83% / 49.17% | 63.14 / 85.26 | 11.50 / 24.50 | 47.98 / 74.36 |
| move3 | 69.17% / 30.83% | 87.62 / 142.98 | 28.83 / 135.17 | 75.01 / 130.75 |
| move4 | 67.92% / 32.08% | 107.34 / 153.51 | 66.25 / 146.75 | 99.96 / 145.38 |
| move5 | 62.50% / 37.50% | 125.74 / 162.81 | 112.60 / 156.60 | 119.24 / 159.01 |
| Setting | # Demonstration Videos | |||
| Top- | 12 | 24 | 36 | 48 |
| 54.55% | 57.27% | 69.09% | 71.82% | |
| 58.18% | 60.00% | 60.91% | 66.36% | |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Opening | ||
| The Caro-Kann Defense | The French Defense | The Grünfeld Defense |
| The Italian Game | The King’s Indian Defense | The London System |
| The Nimzo-Indian Defense | The Queen’s Gambit | The Ruy Lopez |
| The Scotch Game | The Sicilian Defense | The Slav Defense |
| Against Train Questions (n=174) | Against Train Subtitle Chunks (n=2564) | |||||||||
| Metric | Mean SD | 95% CI | Median | P90 | Mean SD | 95% CI | Median | P90 | ||
| Ngram-2 Overlap | 0.295 0.116 | [0.273, 0.317] | 0.282 | 0.452 | 0.0% | 0.208 0.062 | [0.196, 0.219] | 0.200 | 0.286 | 0.0% |
| Max Jaccard | 0.389 0.073 | [0.377, 0.403] | 0.393 | 0.468 | 0.0% | 0.184 0.029 | [0.179, 0.190] | 0.185 | 0.219 | 0.0% |
| TF-IDF Cosine | 0.210 0.065 | [0.198, 0.223] | 0.207 | 0.288 | 0.0% | 0.100 0.024 | [0.095, 0.104] | 0.097 | 0.131 | 0.0% |
| LSA Cosine (128d) | 0.419 0.115 | [0.397, 0.442] | 0.407 | 0.574 | 0.0% | 0.426 0.079 | [0.411, 0.440] | 0.423 | 0.513 | 0.0% |
Continual Learning for LLM Agents
Continual learning for LLM agents studies how agents adapt from experience while retaining skills and knowledge across tasks.
See all 218 papers