MM-ContextFold: Context Folding for Multimodal Agentic Retrieval

Yang Tian; Fan Liu; Jingyuan Zhang; Zhenyang Li; Yupeng Hu; Liqiang Nie

cs.CVSep 19, 2026

MM-ContextFold: Context Folding for Multimodal Agentic Retrieval

Authors: Yang Tian, Fan Liu, Jingyuan Zhang, Zhenyang Li, Yupeng Hu, Liqiang Nie

Organizations: Shandong University Jinan, China · Southeast University Nanjing, China · Kuaishou Beijing, China · Hong Kong University of Science and Technology Hong Kong, Hong Kong ·

Abstract

Multimodal Agentic Retrieval (MAR) requires agents to solve complex information-seeking tasks by iteratively invoking external tools. Typical frameworks such as ReAct maintain raw multimodal inputs and the accumulating interaction history in a single, ever-growing context, leading to the context explosion problem. While existing methods alleviate this issue by compressing redundant text, effective strategies for managing token-intensive visual content remain largely underexplored. To address this gap, we first conduct a systematic empirical study of approximately 10,000 trajectories. The results show that as visual cues are progressively extracted through external tools and textualized into the context, raw images become increasingly redundant. Continued image retention is associated with higher output entropy and can even degrade task accuracy. Motivated by these findings, we propose MM-ContextFold, a training-free framework that loads raw images only when needed. It maintains a persistent, text-only main context for high-level planning and spawns ephemeral branch contexts for image-dependent subtasks. Within each branch, the agent loads the relevant images, completes the subtask, and folds the result back into the main context as a concise textual summary; the images and branch trace are then discarded. Experiments on seven MAR benchmarks across five backbone models show that MM-ContextFold improves average accuracy by 6.3 percentage points over ReAct while reducing the working context length by 27.5%.

Figures & tables

Figure 1. Per-step tool-call counts (bars) and proportions (lines) for Visual Search , Text Retrieval , and Web Visit . Bar chart with overlaid line plots showing, for each trajectory step, the number and share of Visual Search, Text Retrieval, and Web Visit tool calls. Visual Search dominates at step 1 with 69.7 percent of calls and declines sharply afterwards, while Text Retrieval and Web Visit dominate from step 2 onward.

Figure 2. Empirical analysis on the role of images in MAR trajectories. (a) Per-step breakdown of reasoning texts into Image Needed (grounded in new visual information directly extracted from the raw images), Image-Derived (based on visual information already textualized in earlier steps), and Text Only ; the dashed line shows the cumulative proportion of Image Needed instances. (b) Step-level entropy change induced by image removal, measured as $\Delta H_{t}=\mathbf{H}(\tau_{t}^{\text{IR}})-\mathbf{H}(\tau_{t})$ ; box plots show the per-step distributions and the dashed line connects the step-level means. (c) $\Delta$ Accuracy (%) relative to the with-image baseline when images are removed from step $T$ onward; grouped bars show the three benchmark categories, and the dashed line shows the weighted average over all seven benchmarks. Three-panel figure. Panel (a): per-step proportions of reasoning texts labeled Image Needed, Image-Derived, and Text Only, with a dashed cumulative Image Needed curve that saturates within the first five steps. Panel (b): box plots of the entropy change caused by image removal at each step; the mean change is slightly negative at steps 1 and 2, stays near zero during steps 3 to 6, and shifts to more negative values from step 7 onward. Panel (c): grouped bars of accuracy change when images are removed from step T onward for three benchmark categories, with a dashed weighted-average line rising from negative values to slightly positive as T increases.

Figure 3. Overview of MM-ContextFold. Top : the standard MAR framework, where raw query images $\mathcal{I}$ persist in the context throughout the entire trajectory. Bottom : MM-ContextFold maintains a text-only main state that plans subtasks using coarse descriptions from initialization; the agent injects only the assigned images into an ephemeral branch state for visual grounding and tool use, and RETURN folds the branch result back as a textual summary, after which the branch context is discarded. Two-panel schematic. Top: the standard MAR framework keeps raw query images in a single growing context at every reasoning step. Bottom: MM-ContextFold keeps a text-only main state for planning, spawns an ephemeral branch state that loads only the assigned images for visual grounding and tool calls, and folds the branch result back into the main state as a textual summary before the branch context is discarded.

Method	BC-VL #300	LiveVQA #245	MMSearch #171	MMS-Plus #311	SimpleVQA #300	VDR-Bench #200	WorldVQA #200	Avg.
Gemini-3-Flash	44.3 $\pm$ 0.8	73.5 $\pm$ 0.7	59.5 $\pm$ 0.9	22.5 $\pm$ 0.8	75.0 $\pm$ 0.6	14.5 $\pm$ 0.7	40.5 $\pm$ 0.8	47.5
ReAct	54.7 $\pm$ 1.6	79.2 $\pm$ 1.4	74.1 $\pm$ 1.4	37.3 $\pm$ 1.6	78.0 $\pm$ 1.1	23.0 $\pm$ 1.4	57.0 $\pm$ 1.6	57.6
AgentFold	50.3 $\pm$ 1.4	82.0 $\pm$ 1.1	77.6 $\pm$ 1.2	30.7 $\pm$ 1.5	73.7 $\pm$ 1.1	22.0 $\pm$ 1.1	56.0 $\pm$ 1.4	55.4
ContextFold	52.3 $\pm$ 1.4	82.9 $\pm$ 1.1	76.6 $\pm$ 1.1	34.6 $\pm$ 1.4	80.3 $\pm$ 0.9	21.0 $\pm$ 1.1	57.5 $\pm$ 1.4	57.7
Ours	62.7 $\pm$ 1.1	85.4 $\pm$ 0.9	82.8 $\pm$ 1.1	40.3 $\pm$ 1.4	82.7 $\pm$ 0.9	28.5 $\pm$ 1.2	61.0 $\pm$ 1.1	63.2
GPT-5.2	47.0 $\pm$ 0.7	61.6 $\pm$ 0.9	46.8 $\pm$ 0.8	12.6 $\pm$ 0.7	67.3 $\pm$ 0.8	12.5 $\pm$ 0.6	35.5 $\pm$ 0.9	41.1

Table 1. Main results on seven multimodal agentic retrieval benchmarks (accuracy, %). Subscripts give the number of evaluated instances per benchmark, identical for all methods and trials. All methods use the same tool set

\mathcal{U}

and the same inference budget of up to 20 LLM calls. Within each backbone block, the best results are bold and the second-best underlined . Avg. is weighted by these counts. Each entry reports mean

\pm

standard deviation over three independent trials.

Config	BC-VL	LiveVQA	MMSearch	MMS-Plus	SimpleVQA	VDR-Bench	WorldVQA	Avg.
MM-ContextFold (Gemini)	62.7	85.4	82.8	40.3	82.7	28.5	61.0	63.2
$\hookrightarrow$ Image Caption	49.0 ( $-$ 13.7)	73.5 ( $-$ 11.9)	65.5 ( $-$ 17.3)	23.0 ( $-$ 17.3)	64.7 ( $-$ 18.0)	13.0 ( $-$ 15.5)	39.5 ( $-$ 21.5)	46.9 ( $-$ 16.3)
w/o Init	53.3 ( $-$ 9.4)	75.6 ( $-$ 9.8)	76.6 ( $-$ 6.2)	35.3 ( $-$ 5.0)	73.3 ( $-$ 9.4)	22.5 ( $-$ 6.0)	53.5 ( $-$ 7.5)	55.5 ( $-$ 7.7)
w/o Image in Branch	52.7 ( $-$ 10.0)	80.4 ( $-$ 5.0)	76.0 ( $-$ 6.8)	32.3 ( $-$ 8.0)	76.7 ( $-$ 6.0)	24.5 ( $-$ 4.0)	55.5 ( $-$ 5.5)	56.5 ( $-$ 6.7)
w/ Image in Main	51.7 ( $-$ 11.0)	78.6 ( $-$ 6.8)	78.9 ( $-$ 3.9)	35.0 ( $-$ 5.3)	78.7 ( $-$ 4.0)	20.5 ( $-$ 8.0)	56.5 ( $-$ 4.5)	56.8 ( $-$ 6.4)
MM-ContextFold (Qwen3.5-35)	57.3	80.5	76.2	36.3	77.7	22.5	56.5	58.1

Table 2. Ablation results (accuracy, %). Parenthesized values indicate the drop relative to MM-ContextFold on the same backbone.

Backbone	R@1	R@2	R@3	R@4	R@5	ReAct	Ours
Gemini	47.3	55.2	56.4	58.7	59.4	57.6	63.2
Qwen3.5-35	27.6	46.4	48.8	51.7	52.4	51.1	58.1

Table 3. Average accuracy (%) of fixed-step image-removal heuristics. R@

k

denotes ReAct-Removal@

k

, which removes the query images from step

k

onward.

Metric	ReAct	AgentFold	ContextFold	Ours
Cumulative Tokens	54,551	15,993	49,594	44,088
Working Context	8,846	3,415	7,691	6,411
Tool Calls	9.6	6.8	8.7	8.1
$\hookrightarrow$ Text Retrieval	7.4	4.4	5.4	5.8
Web Visit	1.5	1.8	2.2	1.4
Visual Search	0.7	0.6	1.1	0.9

Table 4. Token usage and tool calls averaged per sample across five backbones and seven benchmarks. Working Context denotes the maximum tokens passed to a single LLM call. Tool calls are broken down by type.

Method	1 Image (222)		2 Images (54)		3+ Images (35)
Method	Acc.	Tools	Acc.	Tools	Acc.	Tools
ReAct	31.8	0.9 / 10.1	28.8	0.6 / 9.8	24.6	0.7 / 9.2
AgentFold	24.9	0.7 / 7.3	21.4	1.2 / 8.2	20.9	2.2 / 9.2
ContextFold	30.2	0.7 / 5.3	24.1	2.1 / 12.4	22.8	2.7 / 16.3
MM-ContextFold	38.1	1.0 / 8.9	35.6	1.7 / 11.4	27.6	2.3 / 14.4

Table 5. MMSearch-Plus: accuracy(%) and tool calls ( Visual Search / total) by the number of query images, averaged over five backbones. Parenthesized counts denote the number of samples in each group.

Figure 4. Context growth (a) and accuracy (b) as a function of trajectory length (total LLM calls), averaged across five backbones and seven benchmarks. Two line charts averaged across five backbones and seven benchmarks. Panel (a): working-context tokens versus trajectory length; ReAct grows nearly linearly, ContextFold and MM-ContextFold flatten after early growth, and AgentFold stays lowest. Panel (b): accuracy versus trajectory length; all methods decline on longer trajectories while MM-ContextFold remains highest throughout.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Category	Image Regime	Role of the Query Images	External Retrieval
SimpleVQA	Atomic factuality	Single image	Identifies an entity, artifact, landmark, or product whose name must be recognized.	Answer depends on encyclopedic or attribute knowledge not reliably stored in parametric memory.
WorldVQA	Atomic factuality	Single image	Provides the entity to be identified, often involving long-tail objects, places, or culturally specific items.	Retrieval is needed to ground rare or tail knowledge once the visual referent is identified.
MMSearch	Dynamic info-seeking	Single image	Anchors a current event, organization, scene, or object that must first be identified.	Answer depends on fresh or specialized web information, making static memorization insufficient.
LiveVQA	Dynamic info-seeking	Single image	Serves as an entry point to a time-sensitive or domain-specific query.	Required evidence often comes from recent webpages or niche sources that must be actively verified.
BC-VL	Visual deep research	Single image	Contains a key clue, but the referent is visually ambiguous, indirect, or difficult to name from a single glance.	Requires chained retrieval and cross-source synthesis rather than one-shot identification.
MMS-Plus	Visual deep research	Single/multi-image	In multi-image queries, relevant clues are distributed across images; the agent must determine which image matters for which subtask.	Needs to combine evidence across images and sources, not just identify each clue.

Appendix

Table 6. Benchmark taxonomy used in this work. We summarize the aspects of each benchmark that are most relevant to multimodal agentic retrieval: how the query image contributes, how much external retrieval is required, and why the benchmark belongs to a particular category.

Benchmark	#Samples	Avg. #Imgs	Max #Imgs
SimpleVQA	300	1.00	1
WorldVQA	200	1.00	1
MMSearch	171	1.00	1
LiveVQA	245	1.00	1
BC-VL	300	1.00	1
MMS-Plus	311	1.42	5

Appendix

Table 7. Reported benchmark sizes and image statistics. The reported set is either the released benchmark split or a stratified subset from the public pool. For MMS-Plus, statistics are over the full normalized release.

Benchmark	Source	Pool	Reported	Sampling Policy
SimpleVQA	( Cheng et al., 2025 ) , WebWatcher ( Geng et al., 2026 )	300	300	Used directly without additional sampling.
MMSearch	( Jiang et al., 2025 ) , WebWatcher ( Geng et al., 2026 )	171	171	Used directly without additional sampling.
LiveVQA	( Fu et al., 2025 ) , WebWatcher ( Geng et al., 2026 )	245	245	Used directly without additional sampling.
MMS-Plus	( Tao et al., 2026 )	311	311	Used directly after normalization. Image statistics computed over the full release preserving category proportions.
BC-VL	WebWatcher ( Geng et al., 2026 )	400	300	All 200 Level-2 examples adopted directly; 100 Level-1 examples sampled from the 200 Level-1 pool.
WorldVQA	( Zhou et al., 2026a )	3,000	200	Stratified subset by us. Proportional sampling across the 8 non-People categories preserving original ratios.

Appendix

Table 8. Data source and sampling policy. “Pool” denotes the released source set available after normalization. “Reported” denotes the set size used in this paper.

Backbone models
Gemini-3-Flash	OpenRouter API
GPT-5.2	OpenRouter API
Qwen3.5-9B	OpenRouter API
Qwen3.5-27B	OpenRouter API
Qwen3.5-35B-A3B	OpenRouter API
Shared hyperparameters

Appendix

Table 9. Backbone models, serving configuration, and shared hyperparameters.

Backbone	Total	Excluded	Valid
Qwen3.5-9B	1,727	45	1,682
Qwen3.5-27B	1,727	21	1,706
Qwen3.5-35B-A3B	1,727	28	1,699
Total	5,181	94	5,087

Appendix

Table 10. Trajectory counts for the local entropy analysis. Each backbone is run on all 1,727 samples; invalid trajectories are excluded.

Method	Tool-call format
ReAct (WebWatcher)	JSON inside <tool_call> tags. Parameter key: "arguments" . {"name":"search","arguments": {"queries":["..."]}}
AgentFold	Same JSON <tool_call> syntax as ReAct. Parameter key: "arguments" . Additionally uses <compress> and <motivation> blocks for multi-scale memory folding.
ContextFold	XML-style tags: <function= name > with nested <parameter= key >value</parameter> . Parsed into a flat string-valued argument dict.
MM-Context- Fold	Main state: <subtask> containing a JSON block with description , prompt , and assigned_images . Branch state: JSON inside <tool_call> tags. Parameter key: "parameters" .

Appendix

Table 11. Tool-call format comparison. Each method retains its action grammar; all are routed to the same backend tool implementations.

Method	P1	P2	P3	How it manages visual context
ReAct	✗	✗	✗	Append-only mixed-modality history. Raw query images remain globally visible throughout the trajectory; planning and grounding share a single interleaved context.
AgentFold	$\boldsymbol{\sim}$	$\boldsymbol{\sim}$	✗	Multi-scale textual compression. Planning operates over compressed text (partial P1), and raw images are replaced by textual summaries (partial P2). Compression is lossy, so fine-grained visual cues may be lost and cannot be selectively reloaded later.
ContextFold	$\boldsymbol{\sim}$	✗	✗	Branch-and-return with modality-symmetric inheritance. Branching provides subtask isolation (partial P1), but raw images in the main context propagate into every subsequent branch, and image assignment is implicit rather than per-subtask.
MM-ContextFold	✓	✓	✓	Text-only main state for planning (P1); raw images confined to ephemeral branches and discarded after return (P2); each BRANCH action explicitly specifies which images to load, including $\varnothing$ for text-only subtasks (P3).

Appendix

Table 12. Design-principle comparison. Each method is assessed against the three principles (P1–P3) motivating MM-ContextFold. ✓ = fully satisfied,

\boldsymbol{\sim}

= partially satisfied, ✗ = not satisfied. All agentic methods share the backbone, tools, and budget described in § 4.1 .

Backbone	Direct	ReAct	AgentFold	ContextFold	Ours
Gemini-3-Flash	27.4	32.7	29.8	33.0	36.8
Qwen3.5-35B-A3B	20.3	30.0	27.4	32.0	34.7

Appendix

Table 13. Results on HLE-VL (accuracy, %). Each entry reports the mean over three independent runs. All agentic methods use the same tool set and budget.

	$B_{\max}{=}2$	$B_{\max}{=}3$	$B_{\max}{=}4$	$B_{\max}{=}5$	$B_{\max}{=}6$
Qwen3.5-35B-A3B	56.9	57.8	58.4	58.7	58.2

Appendix

Table 14. Sensitivity of MM-ContextFold to the branch budget

B_{\max}

(average accuracy (%) on the 700-sample subset, Qwen3.5-35B-A3B). All other hyperparameters are held fixed.

Multimodal Search Agents

Multimodal search agents retrieve and reason across text, images, video, and other sources to answer complex questions.

See all 47 papers

POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management
Apr 15, 2026Yikun Liu, Yuan Liu, Haicheng Wang +6Multimodal Search Agents Multimodal Agents
School of Artificial Intelligence, Shanghai Jiao Tong University · CMIC, Shanghai Jiao Tong University · WeChat AI, Tencent
WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IR Multimodal Search Agents
Weixin AI, Tencent · Sun Yat-sen University
ActICL: Post-Training for Agentic Multimodal In-Context Learning
Oct 14, 2026Yanshu Li, Ruixiang Tang, Junyi Jessy Li +1Multimodal ICL Retrieval-Augmented ICL
University of Texas at Austin · 2Rutgers University

Apr 15, 2026J/K move · Enter open · S save

POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

Yikun Liu, Yuan Liu, Haicheng Wang, Zhongyin Zhao+5

School of Artificial Intelligence, Shanghai Jiao Tong University · CMIC, Shanghai Jiao Tong University · WeChat AI, Tencent

Multimodal Search Agents Multimodal Agents Memory-Augmented VLMs

Open paper AskPDF arXiv

Large Multimodal Models (LMMs) excel at visual perception but struggle with real-time, knowledge-intensive queries due to their reliance on static parametric knowledge. While multimodal search agents offer a promising solution, developing them from vanilla LMMs presents two major challenges: the lack of open recipes to cultivate search agency from scratch, and the severe context explosion (attention dilution) that occurs during long-horizon multi-turn investigations. In this paper, we address both challenges by developing a native multimodal search agent. First, we introduce an open training recipe featuring Agentic Seeding, a formative training stage that bootstraps tool-use and planning abilities directly from a non-agentic foundation. Second, to overcome the context bottleneck, we propose V-Fold, an adaptive memory management mechanism. V-Fold retains recent interactions as high-fidelity text while folding stale historical context into the visual space via rendering, exploiting the model's cross-modal alignment to preserve raw evidence without text token redundancy. Combining these innovations, we present POINTS-Seeker-8B, which achieves state-of-the-art performance among models of comparable scale across six multimodal search benchmarks.

Abstract

Multimodal Search Agents

Explore similar work

POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

ActICL: Post-Training for Agentic Multimodal In-Context Learning