As agentic systems increasingly rely on reinforcement learning from verifiable rewards, standardized ``gym'' infrastructure has become essential for rapid iteration, reproducibility, and fair comparison. Vision agents lack such infrastructure, limiting systematic study of what drives their learning and where current models fall short. We introduce Gym-V, a unified platform of 179 procedurally generated visual environments across 10 domains with controllable difficulty, enabling controlled experiments that were previously infeasible across fragmented toolkits. Using it, we find that observation scaffolding is more decisive for training success than the choice of RL algorithm, with captions and game rules determining whether learning succeeds at all. Cross-domain transfer experiments further show that training on diverse task categories generalizes broadly while narrow training can cause negative transfer, with multi-turn interaction amplifying all of these effects. Gym-V is released as a convenient foundation for training environments and evaluation toolkits, aiming to accelerate future research on agentic VLMs.
Figure 1: Overview of Gym-V. Top: 105 single-turn and 74 multi-turn environments across 10 categories. Bottom: a unified reset / step interface shared by interactive environments, offline datasets, and evaluation benchmarks.
Figure 2: Evaluation fidelity of Gym-V against official pipelines. (a) VLM evaluation on Qwen2.5-VL-7B-Instruct reproduces official VLMEvalKit scores. (b) Text-to-image and image-editing evaluation closely matches official GenExam, RISE, and GenEval results.
Category
#Envs
Description
Single-Turn (105)
Algorithmic
21
Grid-based tasks that require algorithmic reasoning—BFS traversal, cellular automata simulation, and game-theoretic computation—over visually rendered grids (e.g. GridBFS ).
ARC
3
Abstract pattern induction tasks that challenge models to infer transformation rules from few visual input–output grid pairs and generalize to unseen instances (e.g. ArcAgi ).
Cognition
10
Perceptual inference tasks involving 3D mental rotation, visual pattern completion, and spatial reasoning from rendered scenes (e.g. RubiksCube-QA ).
Geometry
8
Geometric reasoning tasks requiring computation of convex hulls, enclosing circles, and areas over visually rendered point sets and shapes (e.g. ConvexHull ).
Graphs
23
Graph-theoretic reasoning tasks including traversal, shortest-path computation, spanning trees, coloring, and matching over visually rendered graphs (e.g. ShortestPath ).
Table 2: Overview of the Gym-V environment suite (179 environments across 10 categories). All environments share the reset / step interface and support deterministic seeding.
Table 3: Left: Single-turn evaluation ( ×100 ) across 7 domain categories . Right: Multi-turn evaluation ( ×100 ) across Games (12 envs), Spatial/2D (6 Minigrid envs), and Spatial/3D (3 MiniWorld envs). Avg: mean over all 10 columns reported in this table. Best result per column in bold . Note that Minigrid (Sp./2D) uses shaped episodic returns that include negative penalties (e.g., stepping into hazards), so mean@3 returns can be below zero.
Figure 4: Training reward curves for GRPO, GSPO, and SAPO across 12 single-turn (rows 1–3) and 4 multi-turn (row 4) environments. Smoothed curves (bold) are overlaid on raw trajectories (translucent). All environments exhibit learnable reward signals; no single algorithm dominates uniformly. Multi-turn games show slower convergence and lower absolute returns, reflecting the compounding difficulty of sequential decision-making.
Figure 5: Training reward curves for context modeling (top row) and rules injection (bottom row) ablations on four multi-turn games. Top : with context (3-turn, red; 5-turn, blue) vs. without context (green). Bottom : with rules (green) vs. without rules (red). Smoothed curves (bold) are overlaid on raw trajectories (translucent).
Figure 6: Training reward curves comparing image only (green) vs. image + caption (red) on two single-turn and two multi-turn environments. Adding textual captions yields a substantial and consistent improvement across all four environments. The benefit is most pronounced on perception-heavy tasks such as longest_path , where caption narrows the visual grounding gap early in training, and on multi-turn games ( minesweeper , frozenlake ), where natural-language state descriptions help the agent maintain situational awareness across interaction steps.
Train \ Eval
Algo.
Cogn.
Geom.
Graph
Logic
Puzzle
Baseline
3.6
31.6
5.3
1.2
36.7
7.5
Algorithmic
4.5 +0.9
47.1 +15.5
7.4 +2.1
1.6 +0.5
45.8 +9.2
8.1 +0.5
Cognition
5.7 +2.2
48.3 +16.7
8.5 +3.2
1.7 +0.5
42.5 +5.8
12.2 +4.7
Geometry
2.5 -1.1
36.2 +4.6
11.9 +6.6
0.8 -0.3
42.6 +5.9
4.1 -3.4
Graphs
3.7 +0.1
40.3 +8.7
8.1 +2.8
1.3 +0.1
45.1 +8.4
11.0 +3.5
Logic
5.8 +2.3
42.6 +11.0
7.2 +1.9
1.1 -0.1
45.6 +9.0
10.5 +3.0
Table 4: Cross-domain transfer matrices ( ×100 ) for (a) single-turn categories and (b) multi-turn games. Each row is a training source; each column an evaluation target. Subscripts show absolute change from zero-shot baseline. Diagonal entries (bold) denote in-domain performance.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Training reward curves comparing three prompt conditions across four multi-turn games. Top row : 3-turn context window; Bottom row : 5-turn context window. Green: no context, no rules; Red: context only, no rules; Blue: context with rules. Providing interaction context without rules substantially outperforms the no-context baseline and, in several environments ( Game2048 , FrozenLake , Minesweeper ), approaches or matches the with-rules setting, demonstrating that context enables implicit rule discovery through interaction experience.
Figure 8: Sampled observations from 8 retro arcade environments over 200 VLM agent steps. Despite exhausting the full step budget, the game state barely progresses in any environment.
Figure 9: Algorithmic. Left: BinaryMatrix — the agent performs an algorithmic operation (e.g., counting connected components) on a rendered binary grid; output is a single integer or transformed grid. Right: RottenOranges — the agent determines the minimum time for all oranges to rot via BFS propagation on a grid; output is a single integer.
Figure 10: Cognition. Left: RubiksCube-QA — the agent observes a rendered 3D Rubik’s Cube and answers questions about face colors after rotations. Right: OddOneOutPoly — the agent identifies which polygon differs from the rest based on a visual property (e.g., symmetry, number of sides).
Figure 11: Geometry. Left: LargestIsland — the agent observes a binary grid (blue = water, green = land) and must find the maximum area of a 4-directionally connected island; output is a single integer. Right: VisibleLine — given N lines y=Ax+B plotted on a 2D plane, the agent identifies which lines are visible from y=+∞ (i.e., lie on the upper envelope); output is the space-separated indices of visible lines.
Figure 12: Graphs. Left: ShortestPath — the agent computes the shortest path between two nodes in a weighted graph; output is the node sequence. Right: LongestPath — the agent finds the longest simple path in a graph; output is the path length.
Figure 13: Logic. Left: Thermometers — the agent fills thermometer-shaped regions in a grid to satisfy row/column sum constraints. Right: CircuitLogic — the agent traces a Boolean logic circuit and determines the output given input values.
Figure 14: Puzzles. Left: KnightSwap — the agent swaps the positions of two sets of knights on a small chessboard using legal knight moves; output is the move sequence. Right: TowerOfHanoi — the agent produces the move sequence to transfer all discs to the target peg; output is a sequence of (source, target) moves.
Figure 15: Games. Left: Sokoban — the agent pushes boxes onto goal positions by issuing directional commands ( up/down/left/right ) in a multi-turn grid puzzle. Right: Chess — the agent plays against a built-in opponent, outputting moves in UCI notation (e.g., e2e4 ) each turn.
Figure 16: Spatial. Left: DoorKey (MiniGrid, 2D) — the agent navigates a top-down grid-world to find a key, unlock a door, and reach the goal. Right: CollectHealth (MiniWorld, 3D) — the agent navigates a first-person 3D room to collect health items; actions include move_forward and turn .
Figure 17: Temporal. Left: StreetsOfRage2 — a side-scrolling beat-em-up requiring continuous combat inputs. Right: GoldenAxe — a hack-and-slash arcade game.
Figure 18: Per-environment zero-shot evaluation heatmap across all evaluated environments and 9 models. Rows are grouped by category; columns correspond to models ordered by overall performance. Scores are percentages ( ×100 ); darker shades indicate higher performance.
Environment
Description
Algorithmic (21 environments)
AdditionTable
Find the correct base N and digit mappings for a system where distinct letters represent digits, given addition equations.
BinaryMatrix
Compute the taxicab (Manhattan) distance to the nearest 0 for each cell in a binary matrix.
CirculatingGrid
Modify the minimum number of directional arrows in a toroidal grid so that every cell eventually loops back to itself.
CoinSquareGame
Determine the optimal strategy in a two-player coin-removal game where players take leftmost coins and sum their values.
FaceRightWay
Make all arrows in a binary array face right using the minimum number of fixed-window flip operations.
Appendix
Table 5: Complete catalog of single-turn environments in Gym-V, grouped by category.
Environment
Description
Games (31 environments)
Alquerque
Capture all opponent pieces or block their moves on a 5 × 5 connected board.
Breakthrough
Advance pieces to the opponent’s home row; move forward or capture diagonally.
Chess
Play standard chess against an opponent, outputting moves in UCI notation.
ConnectFour
Drop pieces into a 7-column, 6-row grid; connect four in a line to win.
ConnectFourMultiAgent
Two-agent Connect Four where both sides are controlled by separate models.
Appendix
Table 6: Complete catalog of multi-turn environments in Gym-V, grouped by category.
Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.
Advait Gupta, Sachin Konan, Tom Yin +2
University of Maryland, College Park · Princeton University
Training agents with reinforcement learning requires a gym, comprising a task, an executable environment in which the task can be attempted, and a verifier that reliably distinguishes success from failure. Constructing such gyms remains manual, expensive, and static. Task sets saturate as models improve and are increasingly exposed to contamination. Synthetic generation offers scale, but single-pass synthesis produces tasks whose difficulty is largely cosmetic. Models comparable in capability solve them despite convoluted phrasing, and correctness must be adjudicated post-hoc by unreliable LLM judges. We present AutoGym, a framework that generates complete gyms (tasks, executable environments, and verifiers) from a minimal domain seed or prior model trajectories. AutoGym introduces three mechanisms. (1) Blueprint-first generation specifies the valid solution space, environment requirements, and verification criteria before the environment is materialized, making solvability a construction prerequisite rather than a property verified after the fact. (2) Explicit generation parameters control task topology, interaction depth, capability axes, question obfuscation, and distractor composition, enabling fine-grained difficulty steering. (3) Active curriculum synthesis uses performance-informed calibration to adjust the distribution over these parameters as model capabilities evolve. Across productivity and temporal-reasoning settings, AutoGym generates gyms spanning the capability spectrum, including instances that challenge frontier models.
Aarati Andrea Noronha, Kavya Ravikumar, Carly Xiaoyu Lin
Large Vision-Language Models (VLMs) now act as agents in interactive environments, where success requires coherent reasoning and decision-making across turns. Although end-to-end training in agentic environments can improve such multi-turn decision-making abilities, current methods mainly rely on either token-wise optimization over concatenated token trajectories or turn-wise optimization with uniform within-turn credit. In this work, we establish theoretical formulations for the two levels of optimization and derive a hybrid advantage that serves both objectives. Furthermore, with an appropriate choice of discount factor and learning target, we prove that a unified critic model can estimate values for both turn-wise and token-wise. As such, we propose HyGAE, an actor-critic framework that jointly optimizes token- and turn-level objectives with the hybrid advantage and unified critic. We conduct extensive evaluations of HyGAE across five multi-turn decision-making environments, where it achieves an average success rate of 91% and a significant improvement of 10% over other methods. Furthermore, we provide an in-depth analysis showing that the exact analytic form of the hybrid advantage and return is crucial for optimization. Project Page: https://wx-zhang.github.io/hygae-web/.