VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning

Hexiong Yang; Mingrui Chen; Jie Cao; Ran He

cs.AISep 21, 2026

VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning

Authors: Hexiong Yang, Mingrui Chen, Jie Cao, Ran He

Organizations: NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Science, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy

Abstract

Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language models to vision-language models (VLMs) introduces a distinct state-management problem. Visual reasoning produces intermediate image-valued evidence---crops, masks, overlays, zoomed regions, and analytic renderings---that must remain addressable without accumulating unboundedly in multimodal context. We introduce VLM-in-Sandbox, a training-free framework for agentic multimodal reasoning in controlled computer environments. Its Visual Workspace registers generated artifacts in an image ledger, maintains a bounded active visual context, and lets the model explicitly promote selected evidence for subsequent inspection. This separates visual evidence generation, performed by sandbox tools, from visual evidence management. Across seven benchmarks and four base VLMs, VLM-in-Sandbox achieves the highest sample-weighted average accuracy among Vanilla VLM, Append-only Sandbox, and the proposed method. A compiler-matched $2\times2$ study on 1,260 examples further separates model-directed visibility from bounded retention: VLM-in-Sandbox reaches 66.27% accuracy with 18.6% fewer total tokens than the automatic, retain-all control. Over all 6,350 submitted GPT-4.1-mini examples, it produces 302 rescues and 142 regressions relative to Original Append-only. A local vLLM study with prefix caching confirms that the smaller request workload also reduces uncached tokens, time to first token, and end-to-end latency. These results identify explicit visual evidence state as a central abstraction for sandboxed VLM agents.

Figures & tables

Figure 1: Motivating HRBench-4K example. Vanilla VLM reasons directly from the image; Append-only Sandbox accumulates generated views in its multimodal history; VLM-in-Sandbox keeps visual artifacts on disk and selects evidence for two active slots. The color-mask and bicycle-crop pairing illustrates view selection; the displayed answers are from the original runs, whose trace is discussed in § 3 .

Figure 2: Overview of VLM-in-Sandbox . (a) A frozen VLM uses harness-defined tools in a Docker sandbox to generate visual evidence. Visual Workspace preserves files with stable identifiers and provenance; model-issued Promote operations select the focus and aux images for the next request. (b) Replacing an active view leaves its file available for later selection. The illustrative, nonconsecutive snapshots show an earlier artifact returning to active context without replaying all intermediate images.

Model	Setting	Search		Math		General
Model	Setting	HRBench-4K	V ∗ Bench	WeMath	MathVision	MMStar	RealWorldQA	MMMU	Avg.
Closed-source Models
GPT-4.1-mini	Vanilla VLM	78.00	68.06	62.64	41.12	64.40	71.76	66.19	65.81
	+ Sandbox Only	77.63	68.06	62.76	44.08	60.47	70.33	66.86	64.95
	+ Ours	79.25	70.68	64.08	46.71	65.67	72.94	68.10	67.47
Gemini-2.5-Flash	Vanilla VLM	83.50	79.06	81.38	39.14	76.87	74.77	71.81	76.11

Table 1: Main results across closed-source and open-source MLLMs. We compare three inference settings (Vanilla VLM, + Sandbox Only, and + Ours) across seven multimodal benchmarks grouped into three task families: high-resolution visual search (HRBench-4K, V ∗ Bench), mathematical reasoning (WeMath, MathVision), and general multimodal reasoning (MMStar, RealWorldQA, MMMU). Each cell reports accuracy (%). Within each model block, the best accuracy is shown in bold . The Avg. column reports a sample-size-weighted average across the seven benchmarks.

Figure 3: Average total-token usage across inference settings. Bars report the average total tokens per instance for each benchmark on a logarithmic scale; the shaded Avg. column is the sample-size-weighted average over the seven benchmarks. Sandbox access costs roughly an order of magnitude more total tokens than a single Vanilla VLM call, and the Visual Workspace removes part of that overhead: at the aggregate level, + Ours uses fewer total tokens than + Sandbox Only for every base model while achieving higher accuracy.

Figure 4: Matched visual-state controls. Left: the

2\times 2

design, with sample-weighted accuracy and total tokens per example in each cell. Right: the four settings on complete V ∗ Bench (

N=191

), MathVision-mini (

N=304

), and RealWorldQA (

N=765

), using GPT-4.1-mini. Accuracy panels use separate, labeled scales; token bars start at zero. Original Append-only is not a factorial cell.

Figure 5: Reliability across examples and repeated runs. (a–b) GPT-4.1-mini versus Original Append-only on seven benchmarks (

N=6{,}350

): paired accuracy gains with 95% bootstrap intervals, and corresponding rescue/regression counts. Pooled resampling is benchmark-stratified with HRBench variants grouped by base question. (c–f) Three runs per setting at temperature 1.0 on complete V ∗ Bench and RealWorldQA splits. Points are individual runs; horizontal bars and labels show means. Accuracy scales differ across panels.

Task	Wrong ✗	Wrong2Right ✗ $\rightarrow$ ✓	Rate	Avg. Tool Calls	With Promote
Search	202	70	34.7%	5.64	27.1%
Math	579	331	57.2%	6.15	4.2%
General	934	333	35.7%	5.49	6.3%
Overall	1,715	734	42.8%	5.80	7.4%

Table 2: Wrong2Right analysis and promotion use by task family for Qwen3.5-9B. Wrong2Right and Rate report the number and percentage of vanilla errors corrected by sandbox inference. Avg. Tool Calls reports the mean number of tool invocations per recovered example, and With Promote is the percentage invoking promote at least once.

Figure 6: Stored artifacts versus model-visible context. Open-loop replays of 100 successful trajectories, with actions, observations, selections, and termination fixed. Pixel-identical copies increase the artifact count without adding visual information. Left: tokens summed over compiled requests per replay. Right: mean visible derived images per request, excluding the source image. No model calls are made.

Figure 7: Active visual capacity. Sample-weighted GPT-4.1-mini results on the matched-study collection (

N=1{,}260

). Separate panels show accuracy and mean total tokens per example; the two-slot default is highlighted. Differences are descriptive point estimates.

Figure 8: Inference with prefix caching. Qwen3.5-9B served by vLLM on

8\times

H20 (tensor parallelism 8, request concurrency 1), on the same three complete benchmark splits (

N=1{,}260

). Stacked bars separate cached and uncached prompt tokens per example. TTFT is measured per request; E2E is measured per example. The dashed divider separates Original Append-only from the matched configurations.

Figure 9: Complementary views for visual matching (Qwen3.5-9B, MathVision-mini). The agent retains three panels as files and selects the stickers and candidates for the two active slots. Their comparison supports answer A.

Figure 10: Refining evidence for road-line counting (Claude Haiku 4.5, RealWorldQA). Selected stages of a longer trajectory show a broad mask followed by road-specific diagnostics. The promoted yellow mask is considered together with the crop, edges, and white mask before the answer of zero painted lines.

Figure 11: Inspecting rendered alternatives (Qwen3.5-9B, WeMath). The agent renders the three candidate moves and promotes the comparison image to focus . Moving A to $(3,2)$ fails to produce reflection symmetry; the other two moves succeed. The original generated rendering is preserved.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	State transition or returned information
Register $(f,\tau,p,P)$	Register file $f$ with type $\tau$ , provenance $p$ , and optional parents $P$ ; return a stable asset identifier.
Resolve/List	Resolve an identifier to metadata or enumerate registered artifacts using metadata filters.
Promote $(a,s)$	Put asset $a$ into active slot $s\in\{\texttt{focus},\texttt{aux}\}$ and return a promotion receipt.
ActiveSlots	Return the identifiers and metadata currently rendered as visual inputs.
EvictPayload $(a)$	Release non-active inline image data while preserving the file, identifier, and metadata.

Appendix

Table 3: Minimal Visual Workspace interface. The interface is independent of the concrete sandbox executor.

Block	Contents
Fixed task block	System instructions, benchmark question, output location, and tool-use contract.
Older-turn recap	Deterministic summaries of earlier actions, observations, and relevant artifact references.
Recent history	The three most recent assistant/action/observation turns replayed without compression in the submitted and matched configurations.
Workspace summary	Current active slots and a compact index of available visual artifacts.
Runtime guidance	Remaining-step count, answer-file presence, and deterministic deadline instruction.
Active visual input	Image payloads for the current focus and effective aux assets.

Appendix

Table 4: Compiled request structure. Blocks appear in the listed logical order; active images are rendered as multimodal parts rather than text paths.

Interface	Function
execute_bash	Execute code or shell commands and return text observations.
str_replace_editor	Read, create, and edit files; its promote command selects an existing image for an active slot.
submit	Finalize the answer stored in the designated output file.

Appendix

Table 5: Model-visible tool interfaces.

Setting	Value
Maximum agent steps $T_{\max}$	20
Deadline horizon $d$	3
Submitted/matched recent-turn window	Three turns plus deterministic recap
Active visual slots	focus , aux
Temperature	1.0
Maximum token limit	131,072 in submitted and matched runs

Appendix

Table 6: Core inference settings.

Evaluation splits and scoring
Dataset	Split/release	$N$	Scorer
HRBench-4K	hr_bench_4k.tsv ; 200 questions $\times$ four cyclic option orders	800	Official LLM judge
V ∗ Bench	test	191	Deterministic
WeMath	testmini	1,740	Deterministic
MathVision-mini	testmini	304	Official equivalence
MMStar	val	1,500	Deterministic

Appendix

Table 7: Datasets and scoring protocols. Public source identifiers and split names specify the evaluated releases. License statements follow the corresponding official dataset cards; mixed-source benchmarks remain subject to their upstream material terms.

Study	Models	Evaluation scope	Design
Main evaluation	Four base VLMs	Seven benchmarks	Vanilla, Append-only, and VLM-in-Sandbox
Matched controls	GPT-4.1-mini	Three benchmarks; 1,260 examples	Visibility $\times$ retention
Repeated evaluation	GPT-4.1-mini; Qwen3.5-9B	V ∗ Bench; RealWorldQA	Three evaluations per setting
Local efficiency	Qwen3.5-9B	Three benchmarks; 1,260 examples	Five settings with prefix caching on $8\times$ H20
Context scaling	Recorded trajectories	100 trajectories	Fixed-trace replay with 4–32 artifacts

Appendix

Table 8: Overview of experimental settings. Common inference parameters are listed in Table 6 .

Measurement	Value
Container lifecycle / example	0.82 s
Original Append-only E2E / example	40.1 s
Lifecycle share of E2E	2.0%
Container lifecycle failures	0 / 1,260
Out-of-memory or resource-limit failures	0 / 1,260
Episodes with a failed tool call	10 / 1,260 (0.8%)

Appendix

Table 9: Executor overhead and reliability (

N=1{,}260

). Lifecycle time includes container creation, startup, and removal.

Figure 12: Full matched-control results. GPT-4.1-mini on the three complete splits and their sample-size-weighted pooling, with accuracy (top row) and mean total tokens per example (bottom row) on separate axes. The four matched settings cross visibility (automatic or model-directed) with retention (all images or two active slots). Original Append-only uses the original append-only history instead of the shared prompt compiler, so it is a reference rather than a factorial cell; the dashed line marks its level in every panel. Brackets in the pooled column compare the compiler-matched control with VLM-in-Sandbox.

Comparison	$\Delta$ Acc.	95% CI	$p$	$\Delta$ Tokens
Matched-Auto-All $\rightarrow$ VLM-in-Sandbox	+1.83 pp	[0.40, 3.25]	0.014	$-18.6\%$
Matched-Auto-All $\rightarrow$ Auto-Recent-2	+1.19 pp	[0.00, 2.38] †	0.072	$-16.7\%$
Matched-Append $\rightarrow$ VLM-in-Sandbox	+1.27 pp	[0.24, 2.38]	0.026	$-11.8\%$
Auto-Recent-2 $\rightarrow$ VLM-in-Sandbox	+0.63 pp	[ $-0.16$ , 1.51]	0.185	$-2.3\%$

Appendix

Table 10: Paired comparisons among matched settings. Confidence intervals use 50,000 fixed-seed paired percentile-bootstrap resamples;

p

-values use exact McNemar tests. Token change is the right-hand setting relative to the left-hand setting.

Step budget	Acc. (%)	Total tok.	Mean steps	Exhausted
10	65.71	10.74K	4.3	26 (2.06%)
20	66.27	11.53K	5.0	0
30	66.27	11.61K	5.1	0

Appendix

Table 11: Agent-step budget sensitivity on the matched-study collection (

N=1{,}260

), GPT-4.1-mini. Only the maximum number of agent steps varies; the default budget is highlighted. Exhausted counts episodes that reach the limit before an accepted submit call.

Model	Reference	Stable correct	Rescue	Regression	Stable wrong
GPT-4.1-mini	Original Append-only	3,982	302	142	1,924
Qwen3.5-9B	Vanilla VLM	4,224	734	411	981

Appendix

Table 12: Complete outcome transitions to VLM-in-Sandbox. Each row covers all seven benchmarks (

N=6{,}350

). Stable correct/wrong means both settings agree on correctness; rescue and regression mean wrong-to-correct and correct-to-wrong, respectively. The reference setting differs by model.

Dataset	$N$	Rescue	Regression	$\Delta$ Acc.	95% CI	$p$
HRBench-4K	800	30	17	+1.63 pp	[ $-0.75$ , 4.13]	0.242
V ∗ Bench	191	14	9	+2.62 pp	[ $-2.09$ , 7.85]	0.405
WeMath	1,740	48	25	+1.32 pp	[0.34, 2.30]	0.0095
MathVision-mini	304	17	9	+2.63 pp	[ $-0.66$ , 5.92]	0.169
MMStar	1,500	121	43	+5.20 pp	[3.53, 6.87]	$<0.001$
RealWorldQA	765	36	16	+2.61 pp	[0.78, 4.44]	0.0078

Appendix

Table 13: GPT-4.1-mini paired results by benchmark. Rescue and regression are defined relative to Original Append-only.

Family	Persistent artifacts	Identity / provenance	Active visual budget	Reactivation	Training
Visual scratchpads [ 7 ]	Episode canvas/state	Method-specific	Method-specific	Update or regenerate	Varies
PyVision [ 42 ]	Session artifacts	Session/tool-specific	No general slot bound	Tool view or rerun	No
Multimodal memory [ 35 ]	Model or memory state	Not a general file ledger	Method-specific	Retrieval/update	Usually trained
VLM-in-Sandbox	Sandbox filesystem	Ledger ID + provenance	Two active slots	promote by ID	No

Appendix

Table 15: Interface-level comparison with nearby visual-reasoning families. “Method-specific” indicates that the property depends on the particular canvas, memory, or tool protocol rather than a general file-backed artifact lifecycle.

Figure 13: Correct answer with an unsupported visual rationale (Claude Haiku 4.5, RealWorldQA). The promoted overlay highlights vehicles, whereas the model describes pavement arrows. The original detection overlay is retained to expose the mismatch between evidence and interpretation.

Explore similar work

VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
Aug 3, 2026Yizheng Wu, Jiashen Hua, Bing Deng +1Tool Use in VLMs Tool-Using Vision-Language Agents
Alibaba Group
Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
Aug 21, 2026Lars Benedikt Kaesberg, Tianyu Yang, Florian Valentin Wunderlich +4VLM Reasoning Visual Spatial Reasoning
University of Göttingen, Germany
ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination
Aug 5, 2026Lei Peng, Shuai Lv, Wei HuVLM Reasoning Visually Grounded Reasoning
University of Science and Technology of China School of Artificial Intelligence and Data Science Hefei, Anhui, China · University of Science and Technology of China State Key Laboratory of Precision and Intelligent Chemistry Hefei, Anhui, China

Aug 3, 2026, cs.CVJ/K move · Enter open · S save

VC-Tooler: Learning Compositional and Adaptive Visual Tool Use

Yizheng Wu, Jiashen Hua, Bing Deng, Jieping Ye

Alibaba Group

Tool Use in VLMs Tool-Using Vision-Language Agents RL for Tool Use Multimodal Tool Use

Open paper AskPDF arXiv

Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including

95.8\%

on V* and

35.3\%

on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler

Abstract

Explore similar work

VC-Tooler: Learning Compositional and Adaptive Visual Tool Use

Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds

ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination