A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans

Alessandro Emmanuel Pecora; Stefano Calzolari; Francesco Strada; Andrea Bottino

cs.AISep 28, 2026

A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans

Authors: Alessandro Emmanuel Pecora, Stefano Calzolari, Francesco Strada, Andrea Bottino

Organizations: DAUIN, Politecnico di Torino, Torino, Italy

Abstract

Creating believable vh requires the coherent integration of perception, reasoning, and action mediated by language. A central challenge is to combine these components into a control loop grounded in interactive 3D environments. To this end, we present A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations), a visual-symbolic framework for language-driven vh that leverages a pretrained vlm with tool calling to unify perception, reasoning, and action within a single control loop. A.D.A.M.O. maintains a dual visual-symbolic world model that combines egocentric visual input and synchronized symbolic state to support grounded task-oriented behavior from natural language prompts. To support diagnostic evaluation, we introduce a controlled task suite organized by a cd taxonomy that breaks down spatial tasks into procedural and linguistic complexity. Experiments in controlled scenes show that semantic labeling strongly influences task completion and failure modes, reducing perceptual ambiguity while shifting failures toward downstream execution, whereas reasoning errors remain comparatively rare.

Figures & tables

Figure 2: Infrastructure architecture. Numbered labels indicate dataflow and execution sequence across modules; dotted steps (3–6) implement the Thought–Act–Observation cycle.

Figure 3: Scene in our benchmark (top left: S1 ; bottom left: S2 ) with closeups of the respective objects arrangement (right).

Dim	Unit	Specification	Level

Act	pick	Pick an object.	1
	drop	Release or place an object.	1

Ref	id	Numerical identifier, unique in scene.	0
	tag	Semantic tag or name.	1

Table 1: Capability–Difficulty (C–D) taxonomy: units, specification, and levels.

Task ID	Task Prompt	ACT	REF	REL	TOT
T1	Pick the object tagged as 34	1	0	0	1
T2	Pick the plate	1	1	0	2
T3	Pick the blue circular object	1	1	0	2
T4	Pick the plate and place it on the floor	2	1	1	4
T5	Pick the object tagged as 4 and place it near the indoor plant	2	1	2	5
T6	Pick the smallest object then place it on the floor	2	3	1	6

Table 2: Full list of task prompts and corresponding C-D levels of our taxonomy.

Model	CR ↑	Time per Call (s) ↓	#Walk	#Look	#Total
Labeling Scheme: OPAQ
G4O	0.48	7.63	0.53	1.61	4.80
S35	0.38	8.31	0.38	2.10	4.57
Labeling Scheme: SEM
G4O	0.88	6.06	0.62	1.41	4.60
S35	0.83	7.72	0.23	1.25	3.80

Table 3: Average performances across all episodes for labeling scheme (top blocks) and, for SEM scheme only, per-scene (bottom blocks).

Figure 4: Completion rate across all tasks for different scenes and models using SEM labeling scheme.

k	CR ↑	# Look	# Model Call ↓	Time per Call (s) ↓
4	0.82	1.27	6.34	3.94
6	0.84	1.20	9.20	3.85
10	0.86	1.47	6.99	4.05
50 †	0.93	1.08	4.67	4.71

Table 4: MSTM rolling-buffer size.

Configuration	(Temp ; Top-p)	CR ↑
Quasi-deterministic †	(0.0 ; 0.0)	0.93
Moderately stochastic	(0.3 ; 0.5)	0.82
Highly stochastic	(0.8 ; 1.0)	0.85

Table 5: VLM sampling configurations.

# Ref. Points	CR ↑	# Model Call ↓	Tokens per Ep. ↓
3x3	0.92	5.71	42963.89
4x4	0.90	5.48	41023.82
5x5	0.89	5.77	44816.31
6x6 †	0.93	4.67	32977.61
7x7	0.87	5.34	43372.84

Table 6: Number of 3D reference points.

Resolution	CR ↑	Time per Call (s) ↓
$640\times 360$	0.76	2.65
$1280\times 720$	0.85	3.52
$1920\times 1080$ †	0.93	4.71

Table 7: Input image resolution.

Explore similar work

DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model
Jun 10, 2026Pankhuri Vanjani, Zhuoyue Li, Jakub Suliga +4Diffusion-Based Vision-Language-Actions Action Generation
Intuitive Robots Lab, Karlsruhe Institute of Technology (KIT), Germany · NVIDIA · Robotics Institute of Germany
AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation
Sep 20, 2026Yicheng Jiang, Zesen Gan, Xiaobo Wang +8Efficient World-Action Model Robotic Manipulation
The Hong Kong University of Science and Technology · Shenzhen University of Advanced Technology · Sangfor Technologies Inc. +4
ChunkVLA-AM: Parallel Action Chunking for Vision-Language-Action Robot Control in Additive Manufacturing
Oct 1, 2026Zhugang Liu, Kaichuang Zhang, Jinman Zhang +7Vision-Language-Action Framework Robot Systems
Department of Computer Science, The University of Texas Rio Grande Valley · Department of Electrical Engineering, University of South Florida · Department of Computer Science, San Diego State University +1

Jun 10, 2026, cs.ROJ/K move · Enter open · S save

DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model

Pankhuri Vanjani, Zhuoyue Li, Jakub Suliga, Moritz Reuss+3

Intuitive Robots Lab, Karlsruhe Institute of Technology (KIT), Germany · NVIDIA · Robotics Institute of Germany

Diffusion-Based Vision-Language-Actions Action Generation Photometric Supervision Vision-Language Foundation Models Cross-Attention

Open paper AskPDF arXiv

Vision-language-action (VLA) models inherit a shared synchronous clock from vision-language pretraining, processing every input at one rate. This is misaligned with physical interaction, where a high-frequency modality changes at hundreds of hertz, vision evolves more slowly, and language stays constant across an episode. A synchronous VLA oversamples slow modalities, undersamples fast ones, and caps action generation at the lowest effective frequency. We hypothesize that decoupling temporal processing per modality, letting each update and retain information at its own sensor rate, yields stronger representations and more robust control. We present DAM-VLA, which maintains per-modality latent buffers refreshed at sensor rates and read continuously by the action head, integrating new high-frequency modalities through gated cross-attention that leaves the pretrained backbone intact. Across seven contact-rich real-world manipulation tasks, DAM-VLA more than doubles the average success rate of the strongest synchronous baseline (95.2% vs.\ 40.95%) while sustaining smooth, reactive 100,Hz control. Project website: intuitive-robots.github.io/DAM-VLA/

Abstract

Explore similar work

DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model

AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation

ChunkVLA-AM: Parallel Action Chunking for Vision-Language-Action Robot Control in Additive Manufacturing