Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

Ryuki Hyodo

cs.MAAug 24, 2026

Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

Authors: Ryuki Hyodo

Organizations: SpaceData Inc., 1-17-1 Toranomon, Minato-ku, Tokyo 105-6490, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo 171-8501, Japan · Earth-Life Science Institute, Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan · Université Paris Cité, Institut de Physique du Globe de Paris, CNRS, F-75005 Paris, France

Abstract

Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.

Figures & tables

Figure 1: Snapshot of an example SD-AgentFoundry-2D simulation. The two-dimensional grid contains 20 LLM-driven agents, two places (a cafe and a library), and two active fire events. Colored markers represent the agents, shaded disks show the fire-perception radii, and gray lines connect pairs eligible for local communication. Such configurable environments may support social-simulation studies and characterization of LLM-agent behavior.

Figure 2: Snapshot of an example SD-AgentFoundry-3D simulation in the Kibo digital twin. The Kibo module is loaded as the USD environment, where a single VLM-driven agent, shown as a red-clad human avatar, attempts the assigned task. The lower-right inset is a same-viewpoint re-rendering matched to the VLM camera pose and settings. Users can replace Kibo with their own USD scenes to run VLM-driven embodied simulations in custom three-dimensional spaces.

Setting	Value
LLM backend and model	Ollama; qwen3:4b-instruct-2507-q4_K_M
Inference settings	Temperature 0.2; thinking disabled; maximum 512 response tokens
Agents and inspected point	20 agents; state recorded after step 7
World	Integer grid with both axes ranging from $-25$ to $25$
Places	Cafe at $(-15,0)$ and library at $(15,0)$ , with capacities 12 and 10
Fire events	fire1 : step 3, intensity 0.8, radius 15; fire2 : step 5, intensity 0.5, radius 8

Table 1: Configuration and recorded artifacts for an example SD-AgentFoundry-2D run.

Setting	Value
VLM backend and model	Ollama; qwen3.6:27b
Scene	assets/kibou/KIBOU.usd , excluding the distant Earth backdrop
Task	Find JAXA logo and get close to it
Initial pose	Position $(4.0,20.1)$ m; yaw $180^{\circ}$ ; pitch $0^{\circ}$ ; eye height 1.6 m
Run length	10 VLM decisions for the representative short run
Camera	$512\times 384$ pixels; $70^{\circ}$ vertical field of view

Table 2: Configuration for an example SD-AgentFoundry-3D run.

Agent-Based Modeling

Agent-based modeling simulates how interacting, heterogeneous entities and their environments produce patterns in social, economic, biological, and engineered systems.

See all 160 papers

AgoraSim: A Hybrid Agent-Based Modeling Framework
Jul 7, 2026Chung-Chi ChenHuman Behavior Simulation LLM Agent Orchestration
Human-Agent Ally Lab (HAA Lab) National Institute of Informatics, Japan
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
May 8, 2026Yi Liu, TingFeng Hui, Wei Zhang +4AI Agent Evaluation AI Agent Benchmarks
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology · Chongqing University
SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
May 10, 2026Haoqiang Kang, Xiaokang Ye, Yuhan Liu +5Embodied Navigation AI Coding Agents
1UC San Diego · 2New York University

Jul 7, 2026J/K move · Enter open · S save

AgoraSim: A Hybrid Agent-Based Modeling Framework

Chung-Chi Chen

Human-Agent Ally Lab (HAA Lab) National Institute of Informatics, Japan

Human Behavior Simulation LLM Agent Orchestration Agent-Based Modeling LLM Agents Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

LLM-agent simulations make natural-language social scenarios easy to instantiate, but their outputs can be overread as predictions and are often difficult to compare with explicit social dynamics. We present AgoraSim, a hybrid agent-based modeling framework for scenario-oriented social reaction analysis. AgoraSim resolves textual or multimodal artifacts into editable ABM configurations, runs ratio-controlled populations that mix LLM, vision-language, custom-endpoint, random, and classical agents, and compares the same scenario against matched classical reference dynamics. All agents emit a shared structured decision object, enabling common action spaces, interaction protocols, metrics, and audit records. Exposed through a local UI, Python SDK/CLI, and REST API, AgoraSim helps users inspect scenario trajectories, compare modeling assumptions, and identify cases that warrant empirical validation.

Abstract

Agent-Based Modeling

Explore similar work

AgoraSim: A Hybrid Agent-Based Modeling Framework

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning