VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning
Authors: Hexiong Yang, Mingrui Chen, Jie Cao, Ran He
Organizations: NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Science, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy
Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language models to vision-language models (VLMs) introduces a distinct state-management problem. Visual reasoning produces intermediate image-valued evidence---crops, masks, overlays, zoomed regions, and analytic renderings---that must remain addressable without accumulating unboundedly in multimodal context. We introduce VLM-in-Sandbox, a training-free framework for agentic multimodal reasoning in controlled computer environments. Its Visual Workspace registers generated artifacts in an image ledger, maintains a bounded active visual context, and lets the model explicitly promote selected evidence for subsequent inspection. This separates visual evidence generation, performed by sandbox tools, from visual evidence management. Across seven benchmarks and four base VLMs, VLM-in-Sandbox achieves the highest sample-weighted average accuracy among Vanilla VLM, Append-only Sandbox, and the proposed method. A compiler-matched 2×2 study on 1,260 examples further separates model-directed visibility from bounded retention: VLM-in-Sandbox reaches 66.27% accuracy with 18.6% fewer total tokens than the automatic, retain-all control. Over all 6,350 submitted GPT-4.1-mini examples, it produces 302 rescues and 142 regressions relative to Original Append-only. A local vLLM study with prefix caching confirms that the smaller request workload also reduces uncached tokens, time to first token, and end-to-end latency. These results identify explicit visual evidence state as a central abstraction for sandboxed VLM agents.
Figures & tables
Figure 1: Motivating HRBench-4K example. Vanilla VLM reasons directly from the image; Append-only Sandbox accumulates generated views in its multimodal history; VLM-in-Sandbox keeps visual artifacts on disk and selects evidence for two active slots. The color-mask and bicycle-crop pairing illustrates view selection; the displayed answers are from the original runs, whose trace is discussed in § 3 .
Figure 2: Overview of VLM-in-Sandbox . (a) A frozen VLM uses harness-defined tools in a Docker sandbox to generate visual evidence. Visual Workspace preserves files with stable identifiers and provenance; model-issued Promote operations select the focus and aux images for the next request. (b) Replacing an active view leaves its file available for later selection. The illustrative, nonconsecutive snapshots show an earlier artifact returning to active context without replaying all intermediate images.
Model
Setting
Search
Math
General
HRBench-4K
V ∗ Bench
WeMath
MathVision
MMStar
RealWorldQA
MMMU
Avg.
Closed-source Models
GPT-4.1-mini
Vanilla VLM
78.00
68.06
62.64
41.12
64.40
71.76
66.19
65.81
+ Sandbox Only
77.63
68.06
62.76
44.08
60.47
70.33
66.86
64.95
+ Ours
79.25
70.68
64.08
46.71
65.67
72.94
68.10
67.47
Gemini-2.5-Flash
Vanilla VLM
83.50
79.06
81.38
39.14
76.87
74.77
71.81
76.11
Table 1: Main results across closed-source and open-source MLLMs. We compare three inference settings (Vanilla VLM, + Sandbox Only, and + Ours) across seven multimodal benchmarks grouped into three task families: high-resolution visual search (HRBench-4K, V ∗ Bench), mathematical reasoning (WeMath, MathVision), and general multimodal reasoning (MMStar, RealWorldQA, MMMU). Each cell reports accuracy (%). Within each model block, the best accuracy is shown in bold . The Avg. column reports a sample-size-weighted average across the seven benchmarks.
Figure 3: Average total-token usage across inference settings. Bars report the average total tokens per instance for each benchmark on a logarithmic scale; the shaded Avg. column is the sample-size-weighted average over the seven benchmarks. Sandbox access costs roughly an order of magnitude more total tokens than a single Vanilla VLM call, and the Visual Workspace removes part of that overhead: at the aggregate level, + Ours uses fewer total tokens than + Sandbox Only for every base model while achieving higher accuracy.
Figure 4: Matched visual-state controls. Left: the 2×2 design, with sample-weighted accuracy and total tokens per example in each cell. Right: the four settings on complete V ∗ Bench ( N=191 ), MathVision-mini ( N=304 ), and RealWorldQA ( N=765 ), using GPT-4.1-mini. Accuracy panels use separate, labeled scales; token bars start at zero. Original Append-only is not a factorial cell.
Figure 5: Reliability across examples and repeated runs. (a–b) GPT-4.1-mini versus Original Append-only on seven benchmarks ( N=6,350 ): paired accuracy gains with 95% bootstrap intervals, and corresponding rescue/regression counts. Pooled resampling is benchmark-stratified with HRBench variants grouped by base question. (c–f) Three runs per setting at temperature 1.0 on complete V ∗ Bench and RealWorldQA splits. Points are individual runs; horizontal bars and labels show means. Accuracy scales differ across panels.
Task
Wrong ✗
Wrong2Right ✗ → ✓
Rate
Avg. Tool Calls
With Promote
Search
202
70
34.7%
5.64
27.1%
Math
579
331
57.2%
6.15
4.2%
General
934
333
35.7%
5.49
6.3%
Overall
1,715
734
42.8%
5.80
7.4%
Table 2: Wrong2Right analysis and promotion use by task family for Qwen3.5-9B. Wrong2Right and Rate report the number and percentage of vanilla errors corrected by sandbox inference. Avg. Tool Calls reports the mean number of tool invocations per recovered example, and With Promote is the percentage invoking promote at least once.
Figure 6: Stored artifacts versus model-visible context. Open-loop replays of 100 successful trajectories, with actions, observations, selections, and termination fixed. Pixel-identical copies increase the artifact count without adding visual information. Left: tokens summed over compiled requests per replay. Right: mean visible derived images per request, excluding the source image. No model calls are made.
Figure 7: Active visual capacity. Sample-weighted GPT-4.1-mini results on the matched-study collection ( N=1,260 ). Separate panels show accuracy and mean total tokens per example; the two-slot default is highlighted. Differences are descriptive point estimates.
Figure 8: Inference with prefix caching. Qwen3.5-9B served by vLLM on 8× H20 (tensor parallelism 8, request concurrency 1), on the same three complete benchmark splits ( N=1,260 ). Stacked bars separate cached and uncached prompt tokens per example. TTFT is measured per request; E2E is measured per example. The dashed divider separates Original Append-only from the matched configurations.
Figure 9: Complementary views for visual matching (Qwen3.5-9B, MathVision-mini). The agent retains three panels as files and selects the stickers and candidates for the two active slots. Their comparison supports answer A.
Figure 10: Refining evidence for road-line counting (Claude Haiku 4.5, RealWorldQA). Selected stages of a longer trajectory show a broad mask followed by road-specific diagnostics. The promoted yellow mask is considered together with the crop, edges, and white mask before the answer of zero painted lines.
Figure 11: Inspecting rendered alternatives (Qwen3.5-9B, WeMath). The agent renders the three candidate moves and promotes the comparison image to focus . Moving A to (3,2) fails to produce reflection symmetry; the other two moves succeed. The original generated rendering is preserved.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Operation
State transition or returned information
Register (f,τ,p,P)
Register file f with type τ , provenance p , and optional parents P ; return a stable asset identifier.
Resolve/List
Resolve an identifier to metadata or enumerate registered artifacts using metadata filters.
Promote (a,s)
Put asset a into active slot s∈{focus,aux} and return a promotion receipt.
ActiveSlots
Return the identifiers and metadata currently rendered as visual inputs.
EvictPayload (a)
Release non-active inline image data while preserving the file, identifier, and metadata.
Appendix
Table 3: Minimal Visual Workspace interface. The interface is independent of the concrete sandbox executor.
Block
Contents
Fixed task block
System instructions, benchmark question, output location, and tool-use contract.
Older-turn recap
Deterministic summaries of earlier actions, observations, and relevant artifact references.
Recent history
The three most recent assistant/action/observation turns replayed without compression in the submitted and matched configurations.
Workspace summary
Current active slots and a compact index of available visual artifacts.
Runtime guidance
Remaining-step count, answer-file presence, and deterministic deadline instruction.
Active visual input
Image payloads for the current focus and effective aux assets.
Appendix
Table 4: Compiled request structure. Blocks appear in the listed logical order; active images are rendered as multimodal parts rather than text paths.
Interface
Function
execute_bash
Execute code or shell commands and return text observations.
str_replace_editor
Read, create, and edit files; its promote command selects an existing image for an active slot.
submit
Finalize the answer stored in the designated output file.
Appendix
Table 5: Model-visible tool interfaces.
Setting
Value
Maximum agent steps Tmax
20
Deadline horizon d
3
Submitted/matched recent-turn window
Three turns plus deterministic recap
Active visual slots
focus , aux
Temperature
1.0
Maximum token limit
131,072 in submitted and matched runs
Appendix
Table 6: Core inference settings.
Evaluation splits and scoring
Dataset
Split/release
N
Scorer
HRBench-4K
hr_bench_4k.tsv ; 200 questions × four cyclic option orders
800
Official LLM judge
V ∗ Bench
test
191
Deterministic
WeMath
testmini
1,740
Deterministic
MathVision-mini
testmini
304
Official equivalence
MMStar
val
1,500
Deterministic
Appendix
Table 7: Datasets and scoring protocols. Public source identifiers and split names specify the evaluated releases. License statements follow the corresponding official dataset cards; mixed-source benchmarks remain subject to their upstream material terms.
Study
Models
Evaluation scope
Design
Main evaluation
Four base VLMs
Seven benchmarks
Vanilla, Append-only, and VLM-in-Sandbox
Matched controls
GPT-4.1-mini
Three benchmarks; 1,260 examples
Visibility × retention
Repeated evaluation
GPT-4.1-mini; Qwen3.5-9B
V ∗ Bench; RealWorldQA
Three evaluations per setting
Local efficiency
Qwen3.5-9B
Three benchmarks; 1,260 examples
Five settings with prefix caching on 8× H20
Context scaling
Recorded trajectories
100 trajectories
Fixed-trace replay with 4–32 artifacts
Appendix
Table 8: Overview of experimental settings. Common inference parameters are listed in Table 6 .
Measurement
Value
Container lifecycle / example
0.82 s
Original Append-only E2E / example
40.1 s
Lifecycle share of E2E
2.0%
Container lifecycle failures
0 / 1,260
Out-of-memory or resource-limit failures
0 / 1,260
Episodes with a failed tool call
10 / 1,260 (0.8%)
Appendix
Table 9: Executor overhead and reliability ( N=1,260 ). Lifecycle time includes container creation, startup, and removal.
Figure 12: Full matched-control results. GPT-4.1-mini on the three complete splits and their sample-size-weighted pooling, with accuracy (top row) and mean total tokens per example (bottom row) on separate axes. The four matched settings cross visibility (automatic or model-directed) with retention (all images or two active slots). Original Append-only uses the original append-only history instead of the shared prompt compiler, so it is a reference rather than a factorial cell; the dashed line marks its level in every panel. Brackets in the pooled column compare the compiler-matched control with VLM-in-Sandbox.
Comparison
Δ Acc.
95% CI
p
Δ Tokens
Matched-Auto-All → VLM-in-Sandbox
+1.83 pp
[0.40, 3.25]
0.014
−18.6%
Matched-Auto-All → Auto-Recent-2
+1.19 pp
[0.00, 2.38] †
0.072
−16.7%
Matched-Append → VLM-in-Sandbox
+1.27 pp
[0.24, 2.38]
0.026
−11.8%
Auto-Recent-2 → VLM-in-Sandbox
+0.63 pp
[ −0.16 , 1.51]
0.185
−2.3%
Appendix
Table 10: Paired comparisons among matched settings. Confidence intervals use 50,000 fixed-seed paired percentile-bootstrap resamples; p -values use exact McNemar tests. Token change is the right-hand setting relative to the left-hand setting.
Step budget
Acc. (%)
Total tok.
Mean steps
Exhausted
10
65.71
10.74K
4.3
26 (2.06%)
20
66.27
11.53K
5.0
0
30
66.27
11.61K
5.1
0
Appendix
Table 11: Agent-step budget sensitivity on the matched-study collection ( N=1,260 ), GPT-4.1-mini. Only the maximum number of agent steps varies; the default budget is highlighted. Exhausted counts episodes that reach the limit before an accepted submit call.
Model
Reference
Stable correct
Rescue
Regression
Stable wrong
GPT-4.1-mini
Original Append-only
3,982
302
142
1,924
Qwen3.5-9B
Vanilla VLM
4,224
734
411
981
Appendix
Table 12: Complete outcome transitions to VLM-in-Sandbox. Each row covers all seven benchmarks ( N=6,350 ). Stable correct/wrong means both settings agree on correctness; rescue and regression mean wrong-to-correct and correct-to-wrong, respectively. The reference setting differs by model.
Dataset
N
Rescue
Regression
Δ Acc.
95% CI
p
HRBench-4K
800
30
17
+1.63 pp
[ −0.75 , 4.13]
0.242
V ∗ Bench
191
14
9
+2.62 pp
[ −2.09 , 7.85]
0.405
WeMath
1,740
48
25
+1.32 pp
[0.34, 2.30]
0.0095
MathVision-mini
304
17
9
+2.63 pp
[ −0.66 , 5.92]
0.169
MMStar
1,500
121
43
+5.20 pp
[3.53, 6.87]
<0.001
RealWorldQA
765
36
16
+2.61 pp
[0.78, 4.44]
0.0078
Appendix
Table 13: GPT-4.1-mini paired results by benchmark. Rescue and regression are defined relative to Original Append-only.
Family
Persistent artifacts
Identity / provenance
Active visual budget
Reactivation
Training
Visual scratchpads [ 7 ]
Episode canvas/state
Method-specific
Method-specific
Update or regenerate
Varies
PyVision [ 42 ]
Session artifacts
Session/tool-specific
No general slot bound
Tool view or rerun
No
Multimodal memory [ 35 ]
Model or memory state
Not a general file ledger
Method-specific
Retrieval/update
Usually trained
VLM-in-Sandbox
Sandbox filesystem
Ledger ID + provenance
Two active slots
promote by ID
No
Appendix
Table 15: Interface-level comparison with nearby visual-reasoning families. “Method-specific” indicates that the property depends on the particular canvas, memory, or tool protocol rather than a general file-backed artifact lifecycle.
Figure 13: Correct answer with an unsupported visual rationale (Claude Haiku 4.5, RealWorldQA). The promoted overlay highlights vehicles, whereas the model describes pavement arrows. The original detection overlay is retained to expose the mismatch between evidence and interpretation.
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler
Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet recent work shows that their failures often reflect an interaction between visual grounding and downstream reasoning. What remains less clear is how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged. We study this question in SPaRC, a benchmark for grid-based visual spatial planning, by introducing lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible. Across multiple VLMs, these scaffolds improve task accuracy over the original visual setting by up to 34.0 percentage points and further complement GRPO-based training, yielding up to 4.6 additional accuracy points compared with near-zero gains on the original visual input. Analyses on both end-to-end task solving and object detection show that these gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging. We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.
Lars Benedikt Kaesberg, Tianyu Yang, Florian Valentin Wunderlich +4
Vision-Language Models (VLMs) often lose visual grounding during multi-step reasoning: as reasoning chains grow longer, later inference steps rely increasingly on language priors rather than image evidence. We identify a consistent benchmark-level signature associated with this degradation: across 2,510 re-examined samples from four benchmarks, attention entropy over image tokens typically decreases during Round 1 and rises again after image re-injection. However, we find that effective visual re-examination requires two complementary ingredients: image re-injection and targeted self-diagnosis. Without targeted diagnosis, re-examination can even hurt performance, whereas accurate self-diagnosis yields substantial gains -- a swing of several points on key benchmarks, indicating that diagnostic quality is a key factor in whether re-examination helps or hurts in our setting. We present ReGround, a two-stage framework that teaches VLMs to self-diagnose grounding failures and selectively re-examine visual evidence, without architectural modifications or external tools. Through capability bootstrapping, a stronger variant from the same model family provides diagnostic scaffolding only during data construction, while the policy model learns to diagnose autonomously at inference time and retains most of the assisted gains. Experiments on eight benchmarks across two VLM backbones demonstrate consistent gains, especially on visually intensive multi-step reasoning tasks, while incurring only modest inference overhead relative to tool-augmented baselines. Project page: https://sespoir.github.io/reground-page/ . Code: https://github.com/sespoir/ReGround .
Lei Peng, Shuai Lv, Wei Hu
University of Science and Technology of China School of Artificial Intelligence and Data Science Hefei, Anhui, China · University of Science and Technology of China State Key Laboratory of Precision and Intelligent Chemistry Hefei, Anhui, China