MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences

Shijian Wang; Jiarui Jin; Runhao Fu; Zexuan Yan; Xingjian Wang; Mengkang Hu; Eric Wang; Xiaoxi Li

cs.CVMar 29, 2026

MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences

Authors: Shijian Wang, Jiarui Jin, Runhao Fu, Zexuan Yan, Xingjian Wang, Mengkang Hu, Eric Wang, Xiaoxi Li, +6 more

Organizations: Southeast University · Monash University · Xiaohongshu Inc. · Shanghai Jiao Tong University · University of Hong Kong · Zhejiang University · Renmin University of China

Abstract

Research agents have recently achieved significant progress in information seeking and synthesis across heterogeneous textual and visual sources. In this paper, we introduce MuSEAgent, a multimodal reasoning agent that enhances decision-making by extending the capabilities of research agents to discover and leverage stateful experiences. Rather than relying on trajectory-level retrieval, we propose a stateful experience learning paradigm that abstracts interaction data into atomic decision experiences through hindsight reasoning. These experiences are organized into a quality-filtered experience bank that supports policy-driven experience retrieval at inference time. Specifically, MuSEAgent enables adaptive experience exploitation through complementary wide- and deep-search strategies, allowing the agent to dynamically retrieve multimodal guidance across diverse compositional semantic viewpoints. Extensive experiments demonstrate that MuSEAgent consistently outperforms strong trajectory-level experience retrieval baselines on both fine-grained visual perception and complex multimodal reasoning tasks. These results validate the effectiveness of stateful experience modeling in improving multimodal agent reasoning.

Figures & tables

Figure 1: The overview of MuSEAgent. The framework consists of two phases: (1) Experience Abstraction , which extracts state-level experiences via hindsight evaluation and builds multi-viewpoint embeddings for each experience; (2) Experience Exploitation , where the agent performs a deep-and-wide search over the experience bank to determine the next action.

\mathcal{E}^{(j)}\leftarrow\operatorname{Top\text{-}K}_{e_{m}\in\mathcal{B}}\Bigl(\operatorname{sim}\!\left(\mathbf{q}_{t}^{(i_{j})},\,\mathbf{z}_{m}^{(i_{j})}\right)\Bigr)

Algorithm 2 Experience Exploitation via Deep-and-Wide Search

Panel of Table 2 — Table 2: Performance comparison of MuSEAgent against baselines on various benchmarks under three base models. We report the accuracy (%) for all tasks. Best results are highlighted in bold .

Panel of Figure 2 — Figure 2: Scaling behavior of the Deep-and-Wide search mechanism. We evaluate model performance by scaling (a) the maximum iterations of experience search (Deep Search) and (b) the number of experiences per search (Wide Search), with the alternate dimension fixed at 3 in each setting.

Table 3: Generalization of OOD Stateful Experiences. We evaluate whether stateful experiences generalize well to unseen domains. Unlike baselines that use in-domain experiences, our method uses OOD experiences sourced from the other three datasets. We report the accuracy (%) for all tasks. Best results are highlighted in bold .

Figure 3: A Case of Stateful Experience Abstraction and Deep-and-Wide Search on Stateful Experiences.

Table 4: Ablation of experience sources for MuSEAgent. We compare the impact of using correct, incorrect, or both types of trajectories as the experience source. We report the accuracy (%) for all tasks. Best results are highlighted in bold .

Panel of Table 5 — Table 5: Ablation of hindsight models for MuSEAgent. We compare the impact of using different MLLMs (e.g., Qwen3-VL series and GPT-4o) for hindsight experience abstraction.

Table 6: Ablation of quality score thresholds for MuSEAgent. We report the accuracy (%) for all tasks. Best results are highlighted in bold .

Explore similar work

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents
Apr 21, 2026Shengqin Wang, Wentao Yan, Huichi Zhou +4Reward Shaping RL for Language Model Reasoning
East China Normal University · University College London · Huawei Noah’s Ark Lab +2
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
Apr 22, 2026Wentao Yan, Shengqin Wang, Huichi Zhou +4Multimodal IR Process Reward Models
East China Normal University · Huawei Noah’s Ark Lab · University College London +2
GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning
Sep 12, 2026Zhongyu WangMultimodal RAG Retrieval-Augmented Generation
Beihang University

Apr 21, 2026J/K move · Enter open · S save

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

Shengqin Wang, Wentao Yan, Huichi Zhou, Yihang Chen+3

East China Normal University · University College London · Huawei Noah’s Ark Lab+2

Reward Shaping RL for Language Model Reasoning Multimodal QA Multimodal Search Agents Multimodal Agents RL for LLM Agents

Open paper AskPDF arXiv

Agentic multimodal models have garnered significant attention for their ability to leverage external tools to tackle complex tasks. However, it is observed that such agents often meet premature interaction collapse, caused by two primary reasons: 1) the terminal reward often appending on the last token prevents the advantage from distinguishing trajectories with exploratory behavior; 2) excessively redundant context hinders the agent from absorbing useful feedback. To address these issues, we propose the Deepening Reasoning MMSearchAgent, the framework leverages the structural proximity to derive advantage signals from the whole rollout trajectories in an entire batch, such that trajectories of different lengths are further encouraged to be generated, even when containing the same correct answer. Additionally, differentiated gaussian rewards are employed to dynamically calibrate interaction tolerance, thereby ensuring information reliability and reduce redundancy. To support multi-turn interaction training, we have constructed a multi-step deep-reasoning dataset including 3602 high-quality QA pair with at least 3 reasonning steps. Extensive experiments demonstrate that our method achieves state-of-the-art performance, outperforming the MMSearch-R1 by 8.4

\%

on FVQA-test.

Abstract

Explore similar work

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards

GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning