Agentic Large Language Models

Max van Duijn; Michiel van der Meer; Aske Plaat; Niki van Stein

Figure 1.1 : Recursive self-improvement cycle connecting three categories of agentic LLM: reasoning, acting, and interacting (in red, corresponding to Parts II, III/IV and V). Concepts that influence a category are in green.

Course Goal	Parts	Case Study	Chapters	Optional
Comprehensive	I, II, III, IV, V	1, 2, 3, 4	1-18	19-22
Core Assistants	II, III, IV	2, 3	6-14	2-5, 19
Core Social Simulation	II, III, V	2, 4	6-10, 11, 15-18	2-5, 22

Table 1.1 : Mapping different Course Goals to Parts and Chapters in this Book

Figure 2.1 : A typical conversation with ELIZA

Figure 2.2 : The standard NLP pipeline, as visualized by Manning et al. [2014]

Figure 2.3 : Illustration of vector arithmetic with word2vec embeddings [ Singerep, 2022 ] .

Figure 2.4 : Reinforcement Learning, after Sutton and Barto [2018]

Figure 2.5 : Visualizations of the skipgram (left) and continuous bag-of-words (right) approaches to constructing word embeddings [ Zhang et al., 2023 ] .

Figure 2.6 : Unrolled RNN (illustration from Chris Olah’s blog [ Olah and others, 2015 ] )

Figure 2.7 : Repeating module of an LSTM (illustration from Chris Olah’s blog [ Olah and others, 2015 ] )

Figure 2.8 : Illustration of the Encoder-Decoder architecture

Figure 2.9 : The bilingual evaluation understudy (BLEU) scores of generated translations with respect to the lengths of the sentences [ Bahdanau et al., 2015 ]

Figure 2.10 : Visualization of the Transformer architecture [ Godoy, 2020 , Vaswani et al., 2017 ]

Figure 3.2 : Overview of the data sources inside The Pile dataset [ Gao et al., 2020 ] .

Figure 3.3 : Example data mixtures for training SmolLM3 [ Bakouch et al., 2025 ] .

Figure 3.4 : Training loss for the BERT model using the original architecture (a) and a rearranged architecture (b) [ Shoeybi et al., 2020 ]

Figure 3.5 : High-level overview of RLHF: training a reward model (left), and optimizing a language model (the policy) with respect to that reward model Dong et al. [2024]

Figure 3.6 : Comparison of RLHF (specifically, PPO) to RLVR (specifically, GRPO) method [ Shao et al., 2024 ]

Benchmark Name	Capability Tested	Task Format	Metric
Humanity’s Last Exam (HLE)	General Intelligence, Multimodal Reasoning	Short-Answer, Multiple-Choice	Pass@1
Abstract Reasoning Challenge (ARC)	General Intelligence	Visual Grid Pattern Completion	Pass@1 (Private Set)
GPQA Diamond	Expert-Level STEM Reasoning	Multiple-Choice	Pass@1
AIME 2025	Advanced Mathematical Reasoning	Integer Answer (0-999)	Pass@1
SWE-Bench	Agentic Coding, Real-World Software Engineering	Code Patch Generation	Pass@1
Fortress	AI Safety, National Security Risk Assessment	Varies (Proprietary)	Avg Risk Score (ARS) & Over-Refusal Score (ORS)

Table 3.1 : Overview of common LLM benchmarks with the metrics reported on them

Figure 4.1 : Loss on language modeling objective is a function of training compute, dataset size, and model size [ Kaplan et al., 2020 ] .

Figure 4.2 : Larger models reach a given loss using fewer training tokens, illustrating greater sample efficiency [ Kaplan et al., 2020 ]

Model	$N$	$D$	$D/N$	Regime
Gopher [ Rae et al., 2021 ]	280B	300B	$\approx 1$	undertrained
GPT-3 [ Brown et al., 2020 ]	175B	300B	$\approx 1.7$	undertrained
Chinchilla [ Hoffmann et al., 2022 ]	70B	1.4T	$20$	compute-optimal
LLaMA-1 7B [ Touvron et al., 2023a ]	7B	1.0T	$\approx 143$	overtrained
LLaMA-2 7B [ Touvron et al., 2023b ]	7B	2.0T	$\approx 286$	overtrained
LLaMA-3 8B [ Grattafiori et al., 2024 ]	8B	15T	$\approx 1875$	overtrained

Table 4.1 : Token-to-parameter ratios of several well-known models relative to the Chinchilla rule of thumb

D\approx 20N

.

Figure 4.3 : IsoFLOP curves [ Hoffmann et al., 2022 ]

Figure 4.4 : Eight examples of emergence in the few-shot prompting setting, each point is a separate model [ Wei et al., 2022 ] .

Figure 4.5 : Claimed emergent abilities evaporate upon changing the metric from a nonlinear (top) to a smooth (bottom) metric [ Schaeffer et al., 2023 ]

Figure 4.6 : Scaling behavior for the Hindsight Neglect (left) and Repetitive Algebra (right) tasks [ McKenzie et al., 2023 ]

Figure 4.7 : Early memorization of the training set (blue) versus late generalization (red: grokking) on a modular addition task [ Nanda et al., 2023 ]

Figure 4.8 : Low-Rank Adaptation reparametrization [ Hu et al., 2021 ] . During finetuning, only

A

and

B

are trained

Figure 4.9 : Illustration of the sparsely gated Mixture of Experts (MoE) architecture on top of a decoder-only transformer [ Zhang et al., 2025 ]

Model ID	Organization
allenai/OLMo-1B	Allen Institute for AI
HuggingFaceTB/SmolLM2-360M	Hugging Face
Qwen/Qwen2.5-0.5B	Alibaba Cloud

Table 5.1 : Model IDs for the Hub warm-up exercise

Figure 6.1 : Different Chain of Though (CoT) prompting techniques [ Wei et al., 2022b , Kojima et al., 2022 , Gao et al., 2023 , Plaat et al., 2025 ] . At the top the prompts, at the bottom the answers.

Figure 6.2 : Self Consistency (Adapted from [ Wang et al., 2022 ] ), showing majority voting

Figure 6.3 : Tic Tac Toe is a Simple Board Game, in which State is important

Figure 6.4 : Tree of Thoughts. Left Breadth First Search, Right Depth First Search. The arrows indicate the LLM prompts (after [ Yao et al., 2024 ] )

Figure 6.5 : Tree of Thoughts Example prompts for Game of 24 [ Yao et al., 2024 ]

Figure 6.6 : Reasoning structure of Chain of Thought (b), Self Consistency (c), Tree of Thoughts (d)

Figure 6.7 : Self Refine Architecture [ Madaan et al., 2023 ]

Figure 6.8 : Architecture of Reflexion [ Shinn et al., 2024 ] , showing a close resemblance to the agent/environment structure of reinforcement learning

Figure 6.9 : Buffer of Thoughts [ Yang et al., 2024 ]

Figure 7.1 : A Question in a Visual-Question Answering Tasks might be: What is the mustache made of?

Figure 7.2 : A Captioning Task, possible Caption: A happy dog is standing in the ocean

Figure 7.3 : Architecture of a VLM (after LLaVa [ Liu et al., 2023 ] , image from Huggingface)

Figure 7.4 : Architecture of Stable Diffusion [ Esser et al., 2024 ]

Figure 7.5 : DALL-E: Avocado Armchairs

Figure 7.6 : Flava Pretraining Image-Text Dataset Examples [ Singh et al., 2022 ]

Figure 7.7 : Prompt Following Text-to-Image Examples from DALL-E 3 [ Betker et al., 2023 ]

Figure 7.8 : Stable Diffusion Examples [ Esser et al., 2024 ]

Figure 7.9 : VLA has a different meaning in different languages. So has Vlas. (A hint of which language this is flagged in one of the pictures)

Figure 7.10 : Example from CLIPort: put the gray letter E in the left letter E shape hole [ Shridhar et al., 2022 ]

Figure 7.11 : Another Example from CLIPort: sweep the beans into the blue zone [ Shridhar et al., 2020 ]

Figure 7.12 : Mobility VLA Architecture: Long Context VLM and Topological Maps [ Chiang et al., 2024 ]

Figure 7.13 : Three-LLM Architecture of LM-Nav [ Shah et al., 2023 ]

Figure 7.14 : RT-2 Architecture: VLM trained using Robotic Actions as Text Tokens [ Brohan et al., 2023 ]

Figure 7.15 : Magma Pretraining Pipeline [ Yang et al., 2025 ]

Figure 7.16 : Architecture of

\pi_{0}

. The model is based on a VLM, enhanced with a cross-embodiment dataset [ Black et al., 2024 ]

Figure 7.17 : Sokoban for WorldCoder [ Tang et al., 2024 ]

Figure 7.18 : AlfWorld for WorldCoder [ Tang et al., 2024 ]

Figure 7.19 : Architecture of WorldGPT [ Ge et al., 2024 ]

Figure 7.20 : WebDreamer pseudocode: using LLM for score and sim function [ Gu et al., 2024 ]

Figure 7.21 : WorldCoder Architecture [ Tang et al., 2024 ]

Figure 7.22 : The standard reinforcement learning loop. The agent learns a

\theta

-parameterized policy

\pi_{\theta}

from feedback rewards

r

from the environment. [ Sutton and Barto, 2018 ]

Figure 7.23 : Model-based reinforcement learning where environment feedback is used to improve both the policy

\pi

and the local dynamics model

\hat{M}=(\hat{T},\hat{R})

from which the policy

\pi

is also sampled ( planning )

Figure 9.1 : The reinforcement learning loop where the agent learns a parameterized policy

\pi_{\theta}

from feedback rewards

r

from the environment’s transition

T

and reward function

R

, upon the agent’s actions

a

[ Sutton and Barto, 2018 ]

Figure 9.2 : Reinforcement learning where the agent uses an LLM to parameterize policy

\pi_{LLM_{P\pi}}

to use feedback

r

from the environment to choose its actions

a

Figure 9.3 : Reinforcement learning where both the agent and the environment use LLMs for

\pi

,

T

, and

R

. The LLMs are each given a prompt

Px

to return an action, state, or reward value

Agent type	Prompt	LLM	Example
A : Query	zero-shot user-written	pretrained	GPT-2 [ Vaswani et al., 2017 ]
	query or instruction
B : In-Context Learning	few-shot	pretrained	GPT-3 [ Brown et al., 2020 ]
(supervised example)	examples
C : In-Context Learning	chain-of-thought-steps	pretrained	Chain of Thought [ Wei et al., 2022 ]
(reasoning-steps)	example		Zero-shot CoT [ Kojima et al., 2022 ]

Table 9.1 : Five Agent Styles

Figure 9.4 : Style A: Static Query

Figure 9.5 : Style B: In Context Supervised Learning with a prompt based on few examples

Figure 9.6 : Style C: Top: Chain of Thought: In-Context Learning with an LLM based on examples of the reasoning steps; Bottom: Zero-shot Chain of Thought with an instruction that often works

Figure 9.7 : Style D: Self-reflection with an LLM; the prompt is being optimized in context with answers from an LLM, first to generate subproblems, then to evaluate them; controlled by search (top), or reinforcement learning (bottom)

Figure 9.8 : Style E: Reinforcement Finetuning of the LLM parameters

LLM^{\psi}

in addition to reinforcement learning of the policy

\pi

parameters

\pi_{\theta}

[ Luong et al., 2024 ] .

Figure 10.1 : List of games in BALROG [ Paglieri et al., 2025 ]

Figure 10.2 : Architecture of BALROG [ Paglieri et al., 2025 ]

Figure 10.7 : ReAct example Performing Search [ Yao et al., 2022 ]

Figure 10.8 : Reflexion Architecture [ Shinn et al., 2024 ]

Figure 10.9 : Reflexion Algorithm [ Shinn et al., 2024 ]

Figure 11.1 : Multi LLM agent Framework with a Planner, Caller, and Summarizer [ Shen et al., 2024 ]

Figure 11.5 : Adversarial prompts generated by Rainbow Teaming [ Samvelyan et al., 2024 ]

Figure 11.6 : Quality Diversity Mutation Architecture of Rainbow Teaming [ Samvelyan et al., 2024 ]

Figure 13.1 : AssistGPT system architecture [ Neszlényi et al., 2024 ]

Figure 13.2 : Medical Education Copilot [ Wei et al., 2024a ]

Figure 13.3 : SWE-Agent for Software Engineering [ Yang et al., 2024b ]

Figure 15.1 : Various scenarios in EgoSocialArena [ Hou et al., 2024 ]

Figure 15.2 : The social settings and scenarios in ToMBench categorized in 9 primary topics, #S: Number of stories, #Q: Number of questions, ASL(En/Zh): Average story length (English/Chinese), Agr.: agreement on the correct answer between two raters [ Chen et al., 2024 ]

Figure 16.1 : LLM’s Performance on various Metrics [ Xu et al., 2024 ]

Figure 16.2 : Illustration from the LLM Coordination Benchmark [ Agashe et al., 2025 ] , which uses two types of measures to assess agentic coordination: behavior in pure coordination games in which agents must cooperate to maximize gains (middle) and detailed questions about those games (right)

Figure 16.3 : Alympics Water Allocation Challenge Game [ Mao et al., 2023 ]

Figure 16.4 : Role-playing in CAMEL [ Li et al., 2023 ]

Figure 16.5 : Three Different Communication Strategies in ChatEval [ Chan et al., 2023 ]

Figure 16.6 : The Chain-Structured Workflow in ChatDev [ Qian et al., 2024 ] based on Multi-Turn Dyadic Communication between Agents Powered by ChatGPT-3.5

Figure 16.7 : Illustration of a Scenario with a Conflicting Social Goal from SOTOPIA [ Zhou et al., 2023 ]

Figure 17.1 : Agent-based Modeling and LLM-agents [ Gao et al., 2024 ]

Figure 17.2 : Structure of LLM-agents for Multi-Agent Modeling [ Gao et al., 2023 ]

Figure 17.3 : The Memory–Reflection–Planning loop is the enabling design of the Generative Agents system: without retrieval over an ever-growing experience record and periodic reflection, agents lose coherence within hours of simulated time [ Park et al., 2023 ]

Figure 17.4 : Illustration of the Generative Agents simulation featuring 25 agents and an interactive design that allows users to set behavior in motion [ Park et al., 2023 ]

Figure 17.5 : Components of OASIS [ Yang et al., 2024 ]

Figure 17.6 : Four Stages for Decision Making in AgentVerse [ Chen et al., 2023 ]

Figure 17.7 : Overview of EvolutionaryAgent [ Li et al., 2024a ]

Figure 17.8 : Agents with Different Traits make up Diverse Machine Societies [ Zhang et al., 2023 ]

Figure 19.1 : Projections of the available human-generated public text and dataset sizes to train LLMs [ Villalobos et al., 2024 ] . The intersection of the stock and dataset size projection lines indicates the median year (2028) in which the stock is expected to be fully utilized.

Figure 19.2 : The synthetic-data loop that organizes this chapter. A generator produces raw synthetic data from a small seed set, followed by curation filtering and scoring. The data trains or distills a model, which is then evaluated and can be used as the generator of the next iteration (dashed).

Figure 19.3 : Visualization of the Small Model Learnability Gap, where small student models do not consistently benefit from long CoT reasoning and instead perform better when fine-tuned on shorter CoT reasoning from smaller teachers [ Li et al., 2025b ]

Figure 19.4 : Scaling experiments comparing Unnatural Instructions with Super-Natural Instructions [ Honovich et al., 2023 ] , with model performance controlling for dataset size, tested on two benchmarks: Super-Natural Instructions (left) and LMentry (right).

Figure 19.5 : Self-Instruct pipeline [ Wang et al., 2023b ] .

Figure 20.1 : The Cartpole Problem: Balance the Pole just by Moving the Cart Left or Right

Figure 20.2 : For reference, the standard reinforcement learning loop where the agent learns a

\theta

-parameterized policy

\pi_{\theta}

from feedback rewards

r

from the environment upon its actions

a

, selected from policy

\pi_{\theta}

, etc. [ Sutton and Barto, 2018 ]

Figure 20.3 : ICRL: Prompt Optimization with Reinforcement Learning

Figure 20.4 : Illustration of the Meta Harness approach [ Lee et al., 2026 ]

Figure 20.5 : Illustration of the Shepherd Meta Agent Environment [ Yu et al., 2026 ]

Figure 21.1 : Taxonomy of Interpretability Paradigms . Behavioral interpretability treats the model as a black box, mapping input-output relations (

f(x)=y

). Attributional methods quantify the influence of specific input features on the final output using gradients or saliency. Concept-based interpretability employs probes to identify high-level, human-understandable representations within hidden layers. Mechanistic interpretability aims to reverse-engineer the specific causal circuits and internal algorithms that govern model behavior.

Figure 21.2 : The concept of polysemanticity in LLMs and other transformer based models

Figure 21.3 : Visualization of the logit lens . The residual stream is intercepted at an intermediate layer (e.g., Layer 2) and decoded using the model’s final unembedding matrix (

W_{U}

). This reveals the current belief of the model at that specific step, though it may be noisy in early layers.

Figure 21.4 : Visualization of the Tuned Lens . Unlike the naive lens, this method uses a layer-specific, learned transformation (

\phi_{\ell}

) to decode the residual stream. This transformation is trained to align the intermediate representation with the final output distribution, providing a clearer picture of information emergence in early layers.

Figure 21.5 : Mechanics of an Induction Head Circuit. Induction heads implement a prefix-matching algorithm where the model identifies a previous occurrence of the current token (the prefix) and attends to the subsequent token to predict a repeat sequence.

Figure 21.6 : Activation Patching. Activations from a clean run (where the model succeeds) are swapped into a corrupted run (where the model fails). By observing whether this interchange intervention restores the correct output, we can causally verify if the patched component is responsible for the specific reasoning task.

Figure 21.7 : Architecture of a Sparse Autoencoder. The model projects lower-dimensional input activations (

x\in\mathbb{R}^{d}

) into a much higher-dimensional latent space (

c\in\mathbb{R}^{N}

, where

N\gg d

). A sparsity penalty (e.g.,

L_{1}

regularization on

c

) forces most hidden units to be inactive (grey), resulting in a sparse set of active, interpretable features (teal) that are used to reconstruct the original input.

Table 22.1 : Sources of evidence for the topics of this chapter at the behavioral, representational, and mechanistic level, for (1) bare LLMs and (2) agentic LLMs. The amount of available work is indicated along with a few representative studies. For agentic LLMs, evidence available at the time of writing is overall sparse.

Figure 22.1 : Schematic of the Othello-GPT experiments [ Li et al., 2023 , Nanda et al., 2023 ] , which provide evidence at all three levels of Section 22.1.1 . (1) The model, trained only on move sequences, predicts legal moves. (2) A probe decodes the board state from the activations

h_{\ell}

at an intermediate layer. (3) When the decoded board is edited (here one piece is flipped) and the activations are changed accordingly, the model predicts moves that are legal for the edited board. The board is drawn as

4\times 4

for readability; the real board is

8\times 8

.

Theory	Indicator (simplified)	Bare LLM	Agentic system
Recurrent processing	Feedback loops in processing	Debated: feedforward per token, recurrent only through its own output	Same
Global workspace	Limited-capacity workspace with global broadcast	Debated	Closer: shared memory buffer feeding perception, planning, and action modules
Higher-order theories	Monitoring of own representations	Weak and unreliable (Sect. 22.5 )	Verification modules monitor outputs, not internal states
Attention schema	Model of own attention	No evidence	No evidence
Agency and embodiment	Goal pursuit with feedback; body model	Largely absent	Agency partly; embodiment only in robotics
Biological views	Life, metabolism, bodily regulation	Absent	Absent

Table 22.2 : Simplified overview of indicator properties for consciousness and the extent to which bare LLMs and agentic systems satisfy them. After Butlin et al. [2023] and Goldstein and Kirk-Giannini [2024] ; the assessments in the last two columns are debated.

Agentic Large Language Models

Abstract

Figures & tables

LLM Agents

Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates

Agentic Tool Use in Large Language Models

Small Language Models are the Future of Agentic AI

Abstract

LLM Agents

Explore similar work

Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates

Agentic Tool Use in Large Language Models

Small Language Models are the Future of Agentic AI