Large language models (LLMs) have taken the world by storm. Their ability to communicate with us in our own language, and to reason about our world, has made them a favorite tool for writing, answering questions, and much more. Their answers already influence our decisions and our actions. What would happen if LLMs themselves could act in the world directly? What would happen if LLMs could interact with each other? Agentic LLMs combine the power of large language models with the ability to act in the real world. In this book we will discuss how agentic LLMs work, how they act, how they interact, and what this means for us, as users.
Figures & tables
Figure 1.1 : Recursive self-improvement cycle connecting three categories of agentic LLM: reasoning, acting, and interacting (in red, corresponding to Parts II, III/IV and V). Concepts that influence a category are in green.
Course Goal
Parts
Case Study
Chapters
Optional
Comprehensive
I, II, III, IV, V
1, 2, 3, 4
1-18
19-22
Core Assistants
II, III, IV
2, 3
6-14
2-5, 19
Core Social Simulation
II, III, V
2, 4
6-10, 11, 15-18
2-5, 22
Table 1.1 : Mapping different Course Goals to Parts and Chapters in this Book
Figure 2.1 : A typical conversation with ELIZA
Figure 2.2 : The standard NLP pipeline, as visualized by Manning et al. [2014]
Figure 2.3 : Illustration of vector arithmetic with word2vec embeddings [ Singerep, 2022 ] .
Figure 2.4 : Reinforcement Learning, after Sutton and Barto [2018]
Figure 2.5 : Visualizations of the skipgram (left) and continuous bag-of-words (right) approaches to constructing word embeddings [ Zhang et al., 2023 ] .
Figure 2.6 : Unrolled RNN (illustration from Chris Olah’s blog [ Olah and others, 2015 ] )
Figure 2.7 : Repeating module of an LSTM (illustration from Chris Olah’s blog [ Olah and others, 2015 ] )
Figure 2.8 : Illustration of the Encoder-Decoder architecture
Figure 2.9 : The bilingual evaluation understudy (BLEU) scores of generated translations with respect to the lengths of the sentences [ Bahdanau et al., 2015 ]
Figure 2.10 : Visualization of the Transformer architecture [ Godoy, 2020 , Vaswani et al., 2017 ]
Figure 3.2 : Overview of the data sources inside The Pile dataset [ Gao et al., 2020 ] .
Figure 3.3 : Example data mixtures for training SmolLM3 [ Bakouch et al., 2025 ] .
Figure 3.4 : Training loss for the BERT model using the original architecture (a) and a rearranged architecture (b) [ Shoeybi et al., 2020 ]
Figure 3.5 : High-level overview of RLHF: training a reward model (left), and optimizing a language model (the policy) with respect to that reward model Dong et al. [2024]
Figure 3.6 : Comparison of RLHF (specifically, PPO) to RLVR (specifically, GRPO) method [ Shao et al., 2024 ]
Benchmark Name
Capability Tested
Task Format
Metric
Humanity’s Last Exam (HLE)
General Intelligence, Multimodal Reasoning
Short-Answer, Multiple-Choice
Pass@1
Abstract Reasoning Challenge (ARC)
General Intelligence
Visual Grid Pattern Completion
Pass@1 (Private Set)
GPQA Diamond
Expert-Level STEM Reasoning
Multiple-Choice
Pass@1
AIME 2025
Advanced Mathematical Reasoning
Integer Answer (0-999)
Pass@1
SWE-Bench
Agentic Coding, Real-World Software Engineering
Code Patch Generation
Pass@1
Fortress
AI Safety, National Security Risk Assessment
Varies (Proprietary)
Avg Risk Score (ARS) & Over-Refusal Score (ORS)
Table 3.1 : Overview of common LLM benchmarks with the metrics reported on them
Figure 4.1 : Loss on language modeling objective is a function of training compute, dataset size, and model size [ Kaplan et al., 2020 ] .
Figure 4.2 : Larger models reach a given loss using fewer training tokens, illustrating greater sample efficiency [ Kaplan et al., 2020 ]
Model
N
D
D/N
Regime
Gopher [ Rae et al., 2021 ]
280B
300B
≈1
undertrained
GPT-3 [ Brown et al., 2020 ]
175B
300B
≈1.7
undertrained
Chinchilla [ Hoffmann et al., 2022 ]
70B
1.4T
20
compute-optimal
LLaMA-1 7B [ Touvron et al., 2023a ]
7B
1.0T
≈143
overtrained
LLaMA-2 7B [ Touvron et al., 2023b ]
7B
2.0T
≈286
overtrained
LLaMA-3 8B [ Grattafiori et al., 2024 ]
8B
15T
≈1875
overtrained
Table 4.1 : Token-to-parameter ratios of several well-known models relative to the Chinchilla rule of thumb D≈20N .
Figure 4.4 : Eight examples of emergence in the few-shot prompting setting, each point is a separate model [ Wei et al., 2022 ] .
Figure 4.5 : Claimed emergent abilities evaporate upon changing the metric from a nonlinear (top) to a smooth (bottom) metric [ Schaeffer et al., 2023 ]
Figure 4.6 : Scaling behavior for the Hindsight Neglect (left) and Repetitive Algebra (right) tasks [ McKenzie et al., 2023 ]
Figure 4.7 : Early memorization of the training set (blue) versus late generalization (red: grokking) on a modular addition task [ Nanda et al., 2023 ]
Figure 4.8 : Low-Rank Adaptation reparametrization [ Hu et al., 2021 ] . During finetuning, only A and B are trained
Figure 4.9 : Illustration of the sparsely gated Mixture of Experts (MoE) architecture on top of a decoder-only transformer [ Zhang et al., 2025 ]
Model ID
Organization
allenai/OLMo-1B
Allen Institute for AI
HuggingFaceTB/SmolLM2-360M
Hugging Face
Qwen/Qwen2.5-0.5B
Alibaba Cloud
Table 5.1 : Model IDs for the Hub warm-up exercise
Figure 6.1 : Different Chain of Though (CoT) prompting techniques [ Wei et al., 2022b , Kojima et al., 2022 , Gao et al., 2023 , Plaat et al., 2025 ] . At the top the prompts, at the bottom the answers.
Figure 6.2 : Self Consistency (Adapted from [ Wang et al., 2022 ] ), showing majority voting
Figure 6.3 : Tic Tac Toe is a Simple Board Game, in which State is important
Figure 6.4 : Tree of Thoughts. Left Breadth First Search, Right Depth First Search. The arrows indicate the LLM prompts (after [ Yao et al., 2024 ] )
Figure 6.5 : Tree of Thoughts Example prompts for Game of 24 [ Yao et al., 2024 ]
Figure 6.6 : Reasoning structure of Chain of Thought (b), Self Consistency (c), Tree of Thoughts (d)
Figure 6.8 : Architecture of Reflexion [ Shinn et al., 2024 ] , showing a close resemblance to the agent/environment structure of reinforcement learning
Figure 6.9 : Buffer of Thoughts [ Yang et al., 2024 ]
Figure 7.1 : A Question in a Visual-Question Answering Tasks might be: What is the mustache made of?
Figure 7.2 : A Captioning Task, possible Caption: A happy dog is standing in the ocean
Figure 7.3 : Architecture of a VLM (after LLaVa [ Liu et al., 2023 ] , image from Huggingface)
Figure 7.4 : Architecture of Stable Diffusion [ Esser et al., 2024 ]
Figure 7.22 : The standard reinforcement learning loop. The agent learns a θ -parameterized policy πθ from feedback rewards r from the environment. [ Sutton and Barto, 2018 ]
Figure 7.23 : Model-based reinforcement learning where environment feedback is used to improve both the policy π and the local dynamics model M^=(T^,R^) from which the policy π is also sampled ( planning )
Figure 9.1 : The reinforcement learning loop where the agent learns a parameterized policy πθ from feedback rewards r from the environment’s transition T and reward function R , upon the agent’s actions a [ Sutton and Barto, 2018 ]
Figure 9.2 : Reinforcement learning where the agent uses an LLM to parameterize policy πLLMPπ to use feedback r from the environment to choose its actions a
Figure 9.3 : Reinforcement learning where both the agent and the environment use LLMs for π , T , and R . The LLMs are each given a prompt Px to return an action, state, or reward value
Agent type
Prompt
LLM
Example
A : Query
zero-shot user-written
pretrained
GPT-2 [ Vaswani et al., 2017 ]
query or instruction
B : In-Context Learning
few-shot
pretrained
GPT-3 [ Brown et al., 2020 ]
(supervised example)
examples
C : In-Context Learning
chain-of-thought-steps
pretrained
Chain of Thought [ Wei et al., 2022 ]
(reasoning-steps)
example
Zero-shot CoT [ Kojima et al., 2022 ]
Table 9.1 : Five Agent Styles
Figure 9.4 : Style A: Static Query
Figure 9.5 : Style B: In Context Supervised Learning with a prompt based on few examples
Figure 9.6 : Style C: Top: Chain of Thought: In-Context Learning with an LLM based on examples of the reasoning steps; Bottom: Zero-shot Chain of Thought with an instruction that often works
Figure 9.7 : Style D: Self-reflection with an LLM; the prompt is being optimized in context with answers from an LLM, first to generate subproblems, then to evaluate them; controlled by search (top), or reinforcement learning (bottom)
Figure 9.8 : Style E: Reinforcement Finetuning of the LLM parameters LLMψ in addition to reinforcement learning of the policy π parameters πθ [ Luong et al., 2024 ] .
Figure 10.1 : List of games in BALROG [ Paglieri et al., 2025 ]
Figure 10.2 : Architecture of BALROG [ Paglieri et al., 2025 ]
Figure 10.7 : ReAct example Performing Search [ Yao et al., 2022 ]
Figure 11.1 : Multi LLM agent Framework with a Planner, Caller, and Summarizer [ Shen et al., 2024 ]
Figure 11.5 : Adversarial prompts generated by Rainbow Teaming [ Samvelyan et al., 2024 ]
Figure 11.6 : Quality Diversity Mutation Architecture of Rainbow Teaming [ Samvelyan et al., 2024 ]
Figure 13.1 : AssistGPT system architecture [ Neszlényi et al., 2024 ]
Figure 13.2 : Medical Education Copilot [ Wei et al., 2024a ]
Figure 13.3 : SWE-Agent for Software Engineering [ Yang et al., 2024b ]
Figure 15.1 : Various scenarios in EgoSocialArena [ Hou et al., 2024 ]
Figure 15.2 : The social settings and scenarios in ToMBench categorized in 9 primary topics, #S: Number of stories, #Q: Number of questions, ASL(En/Zh): Average story length (English/Chinese), Agr.: agreement on the correct answer between two raters [ Chen et al., 2024 ]
Figure 16.1 : LLM’s Performance on various Metrics [ Xu et al., 2024 ]
Figure 16.2 : Illustration from the LLM Coordination Benchmark [ Agashe et al., 2025 ] , which uses two types of measures to assess agentic coordination: behavior in pure coordination games in which agents must cooperate to maximize gains (middle) and detailed questions about those games (right)
Figure 16.3 : Alympics Water Allocation Challenge Game [ Mao et al., 2023 ]
Figure 16.4 : Role-playing in CAMEL [ Li et al., 2023 ]
Figure 16.5 : Three Different Communication Strategies in ChatEval [ Chan et al., 2023 ]
Figure 16.6 : The Chain-Structured Workflow in ChatDev [ Qian et al., 2024 ] based on Multi-Turn Dyadic Communication between Agents Powered by ChatGPT-3.5
Figure 16.7 : Illustration of a Scenario with a Conflicting Social Goal from SOTOPIA [ Zhou et al., 2023 ]
Figure 17.1 : Agent-based Modeling and LLM-agents [ Gao et al., 2024 ]
Figure 17.2 : Structure of LLM-agents for Multi-Agent Modeling [ Gao et al., 2023 ]
Figure 17.3 : The Memory–Reflection–Planning loop is the enabling design of the Generative Agents system: without retrieval over an ever-growing experience record and periodic reflection, agents lose coherence within hours of simulated time [ Park et al., 2023 ]
Figure 17.4 : Illustration of the Generative Agents simulation featuring 25 agents and an interactive design that allows users to set behavior in motion [ Park et al., 2023 ]
Figure 17.5 : Components of OASIS [ Yang et al., 2024 ]
Figure 17.6 : Four Stages for Decision Making in AgentVerse [ Chen et al., 2023 ]
Figure 17.7 : Overview of EvolutionaryAgent [ Li et al., 2024a ]
Figure 17.8 : Agents with Different Traits make up Diverse Machine Societies [ Zhang et al., 2023 ]
Figure 19.1 : Projections of the available human-generated public text and dataset sizes to train LLMs [ Villalobos et al., 2024 ] . The intersection of the stock and dataset size projection lines indicates the median year (2028) in which the stock is expected to be fully utilized.
Figure 19.2 : The synthetic-data loop that organizes this chapter. A generator produces raw synthetic data from a small seed set, followed by curation filtering and scoring. The data trains or distills a model, which is then evaluated and can be used as the generator of the next iteration (dashed).
Figure 19.3 : Visualization of the Small Model Learnability Gap, where small student models do not consistently benefit from long CoT reasoning and instead perform better when fine-tuned on shorter CoT reasoning from smaller teachers [ Li et al., 2025b ]
Figure 19.4 : Scaling experiments comparing Unnatural Instructions with Super-Natural Instructions [ Honovich et al., 2023 ] , with model performance controlling for dataset size, tested on two benchmarks: Super-Natural Instructions (left) and LMentry (right).
Figure 19.5 : Self-Instruct pipeline [ Wang et al., 2023b ] .
Figure 20.1 : The Cartpole Problem: Balance the Pole just by Moving the Cart Left or Right
Figure 20.2 : For reference, the standard reinforcement learning loop where the agent learns a θ -parameterized policy πθ from feedback rewards r from the environment upon its actions a , selected from policy πθ , etc. [ Sutton and Barto, 2018 ]
Figure 20.3 : ICRL: Prompt Optimization with Reinforcement Learning
Figure 20.4 : Illustration of the Meta Harness approach [ Lee et al., 2026 ]
Figure 20.5 : Illustration of the Shepherd Meta Agent Environment [ Yu et al., 2026 ]
Figure 21.1 : Taxonomy of Interpretability Paradigms . Behavioral interpretability treats the model as a black box, mapping input-output relations ( f(x)=y ). Attributional methods quantify the influence of specific input features on the final output using gradients or saliency. Concept-based interpretability employs probes to identify high-level, human-understandable representations within hidden layers. Mechanistic interpretability aims to reverse-engineer the specific causal circuits and internal algorithms that govern model behavior.
Figure 21.2 : The concept of polysemanticity in LLMs and other transformer based models
Figure 21.3 : Visualization of the logit lens . The residual stream is intercepted at an intermediate layer (e.g., Layer 2) and decoded using the model’s final unembedding matrix ( WU ). This reveals the current belief of the model at that specific step, though it may be noisy in early layers.
Figure 21.4 : Visualization of the Tuned Lens . Unlike the naive lens, this method uses a layer-specific, learned transformation ( ϕℓ ) to decode the residual stream. This transformation is trained to align the intermediate representation with the final output distribution, providing a clearer picture of information emergence in early layers.
Figure 21.5 : Mechanics of an Induction Head Circuit. Induction heads implement a prefix-matching algorithm where the model identifies a previous occurrence of the current token (the prefix) and attends to the subsequent token to predict a repeat sequence.
Figure 21.6 : Activation Patching. Activations from a clean run (where the model succeeds) are swapped into a corrupted run (where the model fails). By observing whether this interchange intervention restores the correct output, we can causally verify if the patched component is responsible for the specific reasoning task.
Figure 21.7 : Architecture of a Sparse Autoencoder. The model projects lower-dimensional input activations ( x∈Rd ) into a much higher-dimensional latent space ( c∈RN , where N≫d ). A sparsity penalty (e.g., L1 regularization on c ) forces most hidden units to be inactive (grey), resulting in a sparse set of active, interpretable features (teal) that are used to reconstruct the original input.
Table 22.1 : Sources of evidence for the topics of this chapter at the behavioral, representational, and mechanistic level, for (1) bare LLMs and (2) agentic LLMs. The amount of available work is indicated along with a few representative studies. For agentic LLMs, evidence available at the time of writing is overall sparse.
Figure 22.1 : Schematic of the Othello-GPT experiments [ Li et al., 2023 , Nanda et al., 2023 ] , which provide evidence at all three levels of Section 22.1.1 . (1) The model, trained only on move sequences, predicts legal moves. (2) A probe decodes the board state from the activations hℓ at an intermediate layer. (3) When the decoded board is edited (here one piece is flipped) and the activations are changed accordingly, the model predicts moves that are legal for the edited board. The board is drawn as 4×4 for readability; the real board is 8×8 .
Theory
Indicator (simplified)
Bare LLM
Agentic system
Recurrent processing
Feedback loops in processing
Debated: feedforward per token, recurrent only through its own output
Same
Global workspace
Limited-capacity workspace with global broadcast
Debated
Closer: shared memory buffer feeding perception, planning, and action modules
Higher-order theories
Monitoring of own representations
Weak and unreliable (Sect. 22.5 )
Verification modules monitor outputs, not internal states
Attention schema
Model of own attention
No evidence
No evidence
Agency and embodiment
Goal pursuit with feedback; body model
Largely absent
Agency partly; embodiment only in robotics
Biological views
Life, metabolism, bodily regulation
Absent
Absent
Table 22.2 : Simplified overview of indicator properties for consciousness and the extent to which bare LLMs and agentic systems satisfy them. After Butlin et al. [2023] and Goldstein and Kirk-Giannini [2024] ; the assessments in the last two columns are debated.
Complexity and interpretability rarely coincide: systems rich enough for complex behaviours to emerge are usually too opaque to question, while transparent ones are too simple for anything complex to emerge. A single large language model (LLM) is a static artefact, hardly exhibiting any of the emergent properties we associate with life. This changes through interaction: populations of LLMs display emergent dynamics absent from isolated models. Furthermore, LLMs can be endowed with persistent memory, tools and shared skills, and the capacity to initiate actions unprompted, i.e., turning LLMs agentic. In this paper, we argue that such collectives of agents can serve as a computational substrate for Artificial Life (ALife) research. Critically, since the agents communicate in natural language, their collective behaviour can be directly interrogated by examining textual traces and asking the agents themselves. We outline the notion of interpretability in language-model research and extend it for collectives of agents. Lastly, we survey recent examples of agentic LLM collectives that already instantiate the idea of agentic substrates, from controlled experiments to deployments in the wild.
Elias Najarro, Ane Espeseth, Eleni Nisioti +2
IT University of Copenhagen, Denmark · University of Oslo, Norway · Sakana AI, Japan +1
Large language models are increasingly being deployed as autonomous agents yet their real world effectiveness depends on reliable tools for information retrieval, computation and external action. Existing studies remain fragmented across tasks, tool types, and training settings, lacking a unified view of how tool-use methods differ and evolve. This paper organizes the literature into three paradigms: prompting as plug-and-play, supervised tool learning and reward-driven tool policy learning, analyzes their methods, strengths and failure modes, reviews the evaluation landscape and highlights key challenges, aiming to address this fragmentation and provide a more structured evolutionary view of agentic tool use.
Jinchao Hu, Meizhi Zhong, Kehai Chen +2
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China
Large language models (LLMs) are often praised for exhibiting near-human performance on a wide range of tasks and valued for their ability to hold a general conversation. The rise of agentic AI systems is, however, ushering in a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation. Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice. We discuss the potential barriers for the adoption of SLMs in agentic systems and outline a general LLM-to-SLM agent conversion algorithm. Our position, formulated as a value statement, highlights the significance of the operational and economic impact even a partial shift from LLMs to SLMs is to have on the AI agent industry. We aim to stimulate the discussion on the effective use of AI resources and hope to advance the efforts to lower the costs of AI of the present day. Calling for both contributions to and critique of our position, we commit to publishing all such correspondence at https://research.nvidia.com/labs/lpr/slm-agents.