Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

Tommaso Galliena; Stefano Rosa; Tommaso Apicella; Pietro Morerio; Alessio Del Bue; Lorenzo Natale

cs.CVMar 25, 2026

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

Authors: Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale

Organizations: Italian Institute of Technology, Genoa, Italy · University of Genoa, Genoa, Italy

Abstract

Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.

Figures & tables

Figure 1: Memory-driven multi-view exploration progressively resolves ambiguous object captions into a consistent object-level description. (a) the agent predicts a caption for the observed object; (b) a different caption is predicted from a different viewpoint; (c) a consistent caption is predicted based on the episodic object memory; (d) the predicted caption for the object remains consistent.

Figure 2: Overview of the proposed EPOS-VLM model. Structured episodic memory, RGB observations (with detected object bounding boxes and IDs overlayed), and top-down maps are jointly encoded and processed by a Vision–Language transformer, which autoregressively outputs object association, object-level captions, and actions.

KEYS – $LAN$ : language, $RGB$ : image, $B4$ : BLEU, $M$ : METEOR, $RL$ : ROUGE-L, $CI$ : CIDEr, $SP$ : SPICE, $CS$ : cosine similarity between SBERT embedding of prediction and annotation.
Pseudo-captioner	LAN	RGB	B4	M	RL	CI	SP	CS
IC3 [ 5 ]			1.36	13.56	15.17	0.03	11.87	57.89
LD-CPS [ 8 ]			21.02	26.86	53.10	1.36	35.81	73.00
ECO-score [ 13 ]			18.36	23.27	47.32	1.30	34.98	72.16
3D-CPS			28.54	31.97	61.13	1.92	44.73	78.26

Table 1: Pseudo-captioning performance on manually annotated HM3D [ 21 ] test set.

	Accuracy						Consistency
Model	B4	M	RL	CI	SP	CS	Mean CS $\uparrow$	Median CS $\uparrow$	IQR $\downarrow$
Qwen3-VL [ 39 ]	16.00	15.97	41.15	0.84	29.88	60.01	59.65	60.01	29.12
BLIP-2 [ 14 ]	10.99	14.51	39.99	0.57	26.16	56.88	60.15	64.12	25.29
Intern-VL [ 34 ]	17.21	16.87	46.02	0.98	30.02	61.18	55.19	58.76	27.34
CoCa [ 40 ]	15.23	16.13	43.33	0.83	29.95	57.47	57.87	57.26	29.01
Florence [ 41 ]	14.19	17.26	44.03	0.78	31.45	59.12	52.56	55.00	26.98

Table 2: Caption accuracy (

B4-CS

) and caption consistency across multiple views of the same object (Mean

CS-IQR

) on HM3D [ 21 ] manually annotated test set.

	Accuracy						Consistency
Model	B4	M	RL	CI	SP	CS	Mean CS $\uparrow$	Median CS $\uparrow$	IQR $\downarrow$
Galliena et al. [ 8 ]	17.03	19.65	46.88	1.05	35.58	66.03	78.18	80.10	11.92
EPOS-VLM	19.88	22.94	53.87	1.63	39.14	75.86	85.83	85.94	4.17
KEYS – $B4$ : BLEU-4, $M$ : METEOR, $RL$ : ROUGE-L, $CI$ : CIDEr, $SP$ : SPICE, $CS$ : cosine similarity between SBERT embedding of prediction and annotation. Mean CS / Median CS: cosine similarity between SBERT embeddings of captions for the same object (higher, better), IQR: interquartile range of cosine similarity (lower, better).

Table 3: Caption accuracy (

B4

–

CS

) and caption consistency across multiple views of the same object (Mean

CS

–IQR) on Gibson [ 37 ] manually annotated test set. Methods are trained on HM3D [ 21 ] .

KEYS – $Acc$ : matching accuracy, $F1-M$ : F1-score for matching already observed objects, $F1-N$ : F1-score for detecting newly observed objects (NEW), $IDSW$ : ID switches, $Frag$ : track fragmentation
Method	Acc $\uparrow$	F1-M $\uparrow$	F1-N $\uparrow$	IDSW $\downarrow$	Frag $\downarrow$
Galliena et al. [ 8 ]	0.91	0.95	0.90	0.11	0.08
EPOS-VLM	0.93	0.97	0.89	0.09	0.06

Table 4: Data association performance over exploration trajectories on HM3D [ 21 ] test set.

Panel of Figure 3 — Figure 3: Mean and standard deviation of inference time (left) and memory occupancy (right) averaged over HM3D [ 21 ] test episodes for EPOS-VLM and the dense point cloud baseline [ 8 ] . KEY: Point cloud, Ours

	Accuracy						Consistency
Policy	B4	M	RL	CI	SP	CS	Mean CS $\uparrow$	Median CS $\uparrow$	IQR $\downarrow$
Random Goal	21.98	24.12	52.78	1.38	33.88	74.02	80.33	80.65	9.03
Frontier Exploration [ 38 ]	22.87	25.01	53.67	1.50	35.36	74.89	80.21	81.03	8.45
EPOS-VLM	25.86	27.89	59.88	1.87	41.82	77.12	89.37	90.32	2.79
KEYS – $B4$ : BLEU-4, $M$ : METEOR, $RL$ : ROUGE-L, $CI$ : CIDEr, $SP$ : SPICE, $CS$ : cosine similarity between SBERT embeddings of prediction and annotation. Mean $CS$ and Median $CS$ measure cross-view consistency (higher is better), $IQR$ measures variability (lower is better).

Table 5: Caption accuracy (

B4

–

CS

) and caption consistency across multiple views of the same object (Mean

CS

–IQR) varying exploration policy on HM3D [ 21 ] manually annotated test set.

Figure 4: Comparison of object captions predicted by a VLM baseline (Qwen3-VL) and our method along successive viewpoints of an exploration. Mistakes highlighted in red.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description	Value
$\alpha$	Weighting coefficient for viewpoint selection	0.7
$A_{\min}$	Minimum area of high-disagreement regions	100 pixels
$\mathcal{R}$	Viewpoint sampling radii	$\{0.5,1.0,2.0\}$ m
$N_{r}$	Candidate viewpoints per radius	30
$N_{\min}$	Minimum viewpoints per target object	5
$N_{\max}$	Maximum viewpoints per target object	20

Appendix

Table 6: Parameters used during trajectory collection.

Parameter	Value
Rank ( $r$ )	16
Scaling factor ( $\alpha$ )	8
Dropout	0.05
Target modules	q_proj , k_proj , v_proj , o_proj , language_model.embed_tokens , lm_head

Appendix

Table 7: LoRA configuration used for fine-tuning the Unified VLM.

Panel of Figure 5 — Figure 5: Mean and standard deviation of memory token count (left) and number of objects (right) per step, averaged over all HM3D test episodes.

Panel of Figure 6 — Figure 6: Examples of episodic trajectories on scenes from the HM3D test set. (a),(b),(c): the agent successfully revisits one or more objects from different viewpoints, while avoiding obstacles. (d): partial navigation failure where the agent hits a wall.

Figure 7: Additional qualitative examples for the vanilla and fine-tuned model of captions predicted over three views of the same object. Errors are highlighted in red.

KEYS – $B4$ : BLEU, $M$ : METEOR, $RL$ : ROUGE-L, $CI$ : CIDEr, $SP$ : SPICE, $CS$ : cosine similarity between SBERT embedding of prediction and annotation.
VLM	B4	M	RL	CI	SP	CS
InternVL3.5-30B [ 34 ]	27.76	30.11	61.06	1.87	44.53	76.19
Gemma3-27B [ 30 ]	26.54	29.12	58.95	1.68	42.06	73.36
Qwen3-VL-30B [ 39 ]	28.54	31.97	61.13	1.92	44.73	78.26

Appendix

Table 8: Pseudo-captioning performance on manually annotated HM3D [ 21 ] test set.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
Aug 5, 2026Houze Xu, Jizhong Li, Ziyi YeMemory-Augmented VLMs Vision-Language-Action Models
Fudan University
Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
Jun 20, 2026Sourabh Sharma, Sonam Gupta, SadbhawnaVLM Reasoning VLM Hallucination Mitigation
Malaviya National Institute of Technology Jaipur, India · IBM Software Innovation Lab, India
PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
Aug 22, 2026Jihyung Ko, Eunji Jung, Hyeongsub Kim +4Image Captioning Vision-Language Models
Seoul National University, Korea · LG CNS, Korea · Konkuk University, Korea +1

Aug 5, 2026J/K move · Enter open · S save

Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models

Houze Xu, Jizhong Li, Ziyi Ye

Fudan University

Memory-Augmented VLMs Vision-Language-Action Models VLMs for Robotics Reasoning for VLA Models Temporal Modeling for VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models provide a unified paradigm for connecting visual perception, language understanding, and robotic control. However, existing VLA models still face major challenges in long-horizon tasks: sparse expert demonstrations constrain cross-task compositional generalization; the non-Markovian nature of long-horizon tasks makes it difficult for policies conditioned only on current observations to maintain temporal consistency; limited closed-loop error correction allows execution errors to accumulate; and end-to-end action fine-tuning may weaken the high-level semantic representations of vision-language model (VLM) backbones. To address these issues, we propose a hierarchical long-horizon VLA architecture with an explicit language-memory module. The central idea is to convert discrete temporal observations into a coherent textual memory sequence with temporal logic. The system is decoupled into a high-level VLM and a low-level VLA: the high-level VLM performs semantic reasoning through a visual question answering training paradigm, while the low-level VLA executes precise continuous control conditioned on subtask instructions and visual observations. The high-level VLM recursively updates both language memory and subtask instructions using the previous memory as a contextual anchor, enabling persistent temporal tracking and dynamic correction during long-horizon execution. We evaluate the proposed method in multiple simulation environments and conduct sim-to-real experiments on a real robotic platform. The results demonstrate that explicit language memory improves the success rate and robustness of VLA models on complex long-horizon tasks while providing an interpretable semantic account of the decision process.

Abstract

Memory-Augmented VLMs

Explore similar work

Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models

Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models