Organizations: Northwestern Polytechnical University, Xi’an, China · AI Business, Alibaba Group, Hangzhou, China · Xi’an University of Architecture and Technology, Xi’an, China · Zhejiang University, Hangzhou, China
Multimodal Large Language Models (MLLMs) based agents have demonstrated remarkable potential in autonomous web navigation. However, handling long-horizon tasks remains a critical bottleneck. Prevailing strategies often rely heavily on extensive data collection and model training, yet still struggle with high computational costs and insufficient reasoning capabilities when facing complex, long-horizon scenarios. To address this, we propose M2, a training-free, memory-augmented framework designed to optimize context efficiency and decision-making robustness. Our approach incorporates a dual-tier memory mechanism that synergizes Dynamic Trajectory Summarization (Internal Memory) to compress verbose interaction history into concise state updates, and Insight Retrieval Augmentation (External Memory) to guide the agent with actionable guidelines retrieved from an offline insight bank. Extensive evaluations across WebVoyager and OnlineMind2Web demonstrate that M2 consistently surpasses baselines, yielding up to a 19.6% success rate increase and 58.7% token reduction for Qwen3-VL-32B, while proprietary models like Claude achieve accuracy gains up to 12.5% alongside significantly lower computational overhead.
Figures & tables
Figure 1
Figure 2: Overview of the proposed framework. (a) The baseline agent operates with raw context containing redundant visual history and verbose interaction text. This creates high computational overhead and introduces noise that may impair decision-making. Our method M 2 incorporates two key mechanisms: In-Mem and Ex-Mem. (b) The In-Mem module prompts the agent to self-summarize historical steps into a concise context. Simultaneously, (c) the Ex-Mem module retrieves actionable insights from historically successful trajectories based on query similarity. (d) This dual-memory approach provides explicit guidance for the current execution step, significantly reducing errors and yielding tangible performance gains.
Figure 3: Comparison between Full Context and In Mem interaction paradigms. The Full Context (left) retains the entire sequence of raw screenshots and exhaustive historical text, leading to high redundancy. In contrast, the In Mem approach (right) employs a specific prompt to synthesize historical steps into a concise textual summary.
Figure 4: Overview of Insight Retrieval Augmentation pipeline. The workflow begins with a Trajectory Insight Database populated by successful trajectories from diverse models. For a given user query , the system filters insights based on semantic similarity, retrieving the Top- i relevant insights. These insights are then injected into the System Prompt. This mechanism ensures the Web Agent generates goal-centric and error-proof actions, effectively navigating complex UI environments by leveraging cross-trajectory experiences.
Model
Config
AR
AM
AP
AX
BN
CD
CO
ES
GH
GM
HF
WA
Avg.step
Avg.token
Avg.acc.
Claude-3.7-Sonnet
Normal
71.1
65.9
69.8
74.4
78.6
83.7
83.3
77.3
75.6
63.4
44.2
76.1
14.6
121.2k
72.0
In Mem
73.3
70.7
69.8
62.8
78.6
88.4
83.3
68.2
78.1
63.4
58.1
80.4
20.3
82.9k
72.9
Ex Mem
82.2
78.1
76.7
76.7
83.3
86.1
85.7
68.2
90.2
80.5
67.4
82.6
15.7
164.9k
79.8
In & Ex Mem
84.4
80.5
76.7
86.1
85.7
93.0
95.2
84.1
92.7
82.9
74.4
78.3
16.9
84.5k
84.5 ( + 12.5%)
Claude-Sonnet-4
Normal
82.2
70.7
69.8
79.1
83.3
93.0
85.7
84.1
87.8
78.1
69.8
82.6
15.4
128.2k
80.5
In Mem
84.4
75.6
79.1
81.4
78.6
88.4
85.7
79.6
82.9
70.7
67.4
82.6
16.8
64.5k
79.7
Table 2: Performance Comparison of Claude and Qwen3VL Model across Multiple WebSite on WebVoyager. (Abbreviations: AR: Allrecipes, AM: Amazon, AP: Apple, AX: ArXiv, BN: BBC News, CD: Cambridge Dict., CO: Coursera, ES: ESPN, GH: GitHub, GM: Google Map, HF: Huggingface, WA: Wolfram Alpha.)
Model
Config
Easy
Medium
Hard
Average
Step
Token
Acc. (%)
Step
Token
Acc. (%)
Step
Token
Acc (%)
Step
Token
Acc. (%)
Claude-3.7-Sonnet
Normal
21.72
219.6k
74.07
29.50
310.9k
58.27
34.51
371.6k
53.52
28.56
300.3k
61.95
In & Ex
19.89
98.0k ( − 55.4%)
79.01 (+4.9%)
27.40
144.0k ( − 53.7%)
65.47 (+7.2%)
31.03
163.9k ( − 55.9%)
61.97 (+8.5%)
26.05
135.2k ( − 55.0%)
68.82 (+6.9%)
Claude-Sonnet-4
Normal
16.23
154.7k
74.07
21.24
205.9k
66.91
21.24
292.1k
61.97
21.54
212.7k
67.70
In & Ex
18.77
95.7k ( − 38.1%)
82.72 (+8.7%)
27.00
141.2k ( − 31.4%)
73.38 (+6.5%)
37.24
207.2k ( − 29.1%)
64.79 (+2.8%)
27.21
144.6k ( − 32.0%)
73.88 (+6.2%)
Qwen3-VL-32B
Normal
25.41
220.9k
44.44
35.38
325.8k
30.22
42.11
403.7k
21.13
34.25
315.6k
31.96
Table 3: Evaluation Results on OnlineMind2Web. Green values in parentheses represent the relative reduction percentage for Tokens and absolute percentage point increase for Accuracy.
Ablation Category
Settings
Acc. (%)
Historical Window k
1
51.6
3
56.0
5 (Default)
57.8
7
52.2
Similarity Function
TF-IDF ( Salton and Buckley, 1988 )
61.1
Word2Vec ( Mikolov et al., 2013 )
64.4
Table 4: Ablation study of M 2 components on WebVoyager by Qwen3-VL-32B. S.T. denotes Sentence Transformer (Default).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of the 55k trajectory data across 12 distinct web domains in the Insight Bank.
Figure 6: Cumulative token consumption comparison on an Apple.com navigation task (16 steps). The agent uses Qwen3-VL-32B.
Figure 7: Comparison of agent responses between the original interaction form and the M 2 interaction form.
Figure 8: Qualitative comparison on the GitHub task. Left (Original): Agent gets trapped in a loop of irrelevant repository results due to context redundancy. Right (M 2 ): Agent navigates hierarchically to the documentation page and uses a precise search, maintaining a clear task progress via memory summarization.
Figure 9: Qualitative comparison on a geographic disambiguation task. Left (Original): The agent suffers from context hallucination, ignoring the established UK starting point and selecting a US destination (North Plymouth, MA), resulting in a routing failure. Right (M 2 ): Guided by the external insight to prioritize the first dropdown suggestion and maintaining internal context consistency, the agent correctly identifies the local UK destination.
Figure 10: Original system prompt used by the agent.
Figure 11: Update of Dynamic Trajectory Summarization compared to the original system prompt.
Figure 12: Prompt Pext used by Abstractor to extract insight.
Figure 13: System prompts Psys′ used in the final M 2 framework.
Large language model (LLM)-based agents have emerged as powerful autonomous controllers for digital environments, including mobile interfaces, operating systems, and web browsers. Web navigation, for example, requires handling dynamic content and long sequences of actions, making it particularly challenging. Existing LLM-based agents struggle with long-horizon planning in two main ways. During online execution, they often lose track as new information arrives, lacking a clear and adaptive path toward the final goal. This issue is further exacerbated during reinforcement learning (RL) fine-tuning, where sparse and delayed rewards make it difficult for agents to identify which actions lead to success, preventing them from maintaining coherent reasoning over extended tasks. To address these challenges, we propose two contributions. First, we introduce an agent framework that leverages proprietary models for online planning through subgoal decomposition. Second, we present MiRA (Milestoning your Reinforcement Learning Enhanced Agent), an RL training framework that uses dense, milestone-based reward signals. The real-time planning mechanism improves proprietary models such as Gemini by approximately a 10% absolute increase in success rate (SR) on the WebArena-Lite benchmark. Meanwhile, applying MiRA to the open Gemma3-12B model increases its success rate from 6.4% to 43.0%. This performance surpasses proprietary systems such as GPT-4-Turbo (17.6%) and GPT-4o (13.9%), as well as the previous open-model state of the art, WebRL (38.4%). Overall, our findings demonstrate that combining explicit inference-time planning with milestone-based rewards significantly improves an agent's long-horizon capabilities, paving the way for more robust and general-purpose autonomous systems.
Modern language-model agents increasingly interact with external environments over long-horizon, multi-step trajectories, where the accumulated interaction history can quickly exceed practical context budgets. To ensure reliability, agents must maintain factual information over long horizons, remember execution errors and corrective feedback, and track progress across actions. Several approaches have been proposed to achieve this without the need for maintaining the entire execution history in context, such as using the reasoning and action history, learning to maintain a fixed-size memory through an overwrite mechanism, and periodic summarization. Although overwrite memory can in principle retain anything an append-only memory can, it must learn to carry each fact through every subsequent rewrite, which is difficult to learn from sparse outcome rewards; for interactive applications like web agents, we find that trained overwrite memories delete key information required by the trajectory, as well as corrective feedback received from the environment. We introduce AMBER (Append-only Memory Bank for Evidence Retention) - a simple and scalable framework where an agent jointly learns to reason, act, and write free-form memory, while an append-only rule guarantees retention by construction. This allows AMBER to be trained end-to-end with reinforcement learning from outcome rewards without the need for extensive curated SFT data. On WebArena Lite, AMBER improves average success over overwrite-based memory by 4.09 percentage points, increases the fraction of tasks solved in five repeated runs by 4.8 percentage points, and matches an overwrite baseline trained on substantially more expensive curated supervision. AMBER achieves these improvements while maintaining a practical token budget, providing a strong balance between context efficiency, task performance, and reliable long-horizon execution.
Large Language Models (LLMs) struggle with long-horizon tasks due to the "context bottleneck" and the "lost-in-the-middle" phenomenon, where accumulated noise from verbose environments degrades reasoning over multi-turn interactions. To address this issue, we introduce a symbiotic framework that decouples context management from task execution. Our architecture pairs a lightweight, specialized policy model, ContextCurator, with a powerful frozen foundation model, TaskExecutor. Trained via reinforcement learning, ContextCurator actively reduces information entropy in the working memory. It aggressively prunes environmental noise while preserving reasoning anchors, that is, sparse data points that are critical for future deductions. On WebArena, our framework improves the success rate of Gemini-3.0-flash from 36.4% to 41.2% while reducing token consumption by 8.8% (from 47.4K to 43.3K). On DeepSearch, it achieves a 57.1% success rate, compared with 53.9%, while reducing token consumption by a factor of 8. Remarkably, a 7B ContextCurator matches the context management performance of GPT-4o, providing a scalable and computationally efficient paradigm for autonomous long-horizon agents.
Xiaozhe Li, Tianyi Lyu, Yizhao Yang +6
Tongji University · CurrentsAI Research · Stanford University +1