Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Chenhao Zheng; Zixian Ma; Kai Li; Kanchana Ranasinghe; Jieyu Zhang; Artemis Panagopoulou; Weikai Huang; Jianxu Shangguan

cs.CVNov 5, 2026

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Authors: Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang, Artemis Panagopoulou, Weikai Huang, Jianxu Shangguan,

Organizations: University of Washington · Salesforce AI Research

Abstract

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.

Figures & tables

Figure 1: Interactive Visual Gym. (a) System overview: The visual interaction model under evaluation is tasked with guiding a simulated user to complete a physical task in a visual environment. (b) Examples showing how different task execution can be simulated based on different assistant response.

Figure 2: Example tasks in our environments. We show the simulated tasks across three domains, with representative action commands and corresponding visual observations during task execution. Our semantic action spaces express actions at the level of everyday human communication.

A Scheduled out-of-plan events (CookSim) Category Event types Execution errors Repeat, forget, or skip a step; wrong plating order; wrong ingredient; add unwanted ingredients; plate raw food; unattended cooking. Questions Action order; counts; elapsed time; time duration; event recall Proactive requests Notify when a specified condition holds. Plan changes Modify the recipe; perform a side task.

B User personas Axis Levels / behavior Skill Low / Medium / High Planning and anticipation. Trust in AI Low / Medium / High Evidence needed to accept advice. Execution speed Cautious / Balanced / Quick Careful versus rapid execution. Interruption Step-by-step / Moderate / Necessary only Receptiveness to frequent guidance.

Table 1: Controllable user simulation. (A) Scheduled out-of-plan events, illustrated with CookSim, test assistant responses to different types of out-of-plan events. Events are triggered by conditions on task progress. (B) Four persona axes, each with three levels, simulate how different simulated users perform tasks and respond to guidance differently .

	CookSim								VHSim								ScreenSim
	In-time success	Det. recall	Interaction quality						In-time success	Det. recall	Interaction quality						In-time success	Det. recall	Interaction quality
Assistant	In-time success	Det. recall	Fac. grnd.	Sit. rel.	Act. guid.	User intent	Guid. conc.	Overall	In-time success	Det. recall	Fac. grnd.	Sit. rel.	Act. guid.	User intent	Guid. conc.	Overall	In-time success	Det. recall	Fac. grnd.	Sit. rel.	Act. guid.	User intent	Guid. conc.	Overall
Frontier API models
Gemini 4 Argon	60.9	80.2	70.2	90.1	71.4	82.2	76.9	63.8	15.6	81.2	60.3	90.7	60.8	83.2	84.8	55.6	83.7	100.0	91.0	95.5	86.5	87.1	95.1	87.1
Gemini 3.8 Flash	46.4	77.9	83.5	83.5	65.5	66.2	90.9	73.4	14.7	72.2	79.2	88.4	63.4	75.3	92.7	71.2	78.5	97.8	92.4	95.7	89.4	80.2	99.0	89.4
GPT-6.1 Sol	42.9	79.9	80.4	92.4	78.1	91.1	91.7	75.8	15.6	60.3	77.8	92.9	57.2	89.1	92.2	70.5	87.4	100.0	89.0	96.6	91.7	93.4	97.3	86.5

Table 2: Quantitative results across three environments. Results are averaged across three independent runs. Det. recall: error detection recall; Fac. grnd.: factual grounding; Sit. rel.: situational relevance; Act. guid.: actionable guidance; User intent: user intent uptake; Guid. conc.: guidance conciseness. Scores are percentages; bold and underline indicate the best and second-best scores in each column.

Figure 3: Task success versus interaction quality. Scores are averaged over all three environments.

	Question answering				Proactive request	Overall
Assistant	Temp. order	Act. count	Time dur.	Event recall	Proactive request	Overall
Frontier API models
Gemini 4 Argon	66.8	52.8	82.3	85.8	85.3	79.0
GPT-6 Astra	51.1	34.8	89.5	88.9	86.8	78.6
GPT-6.1 Sol	58.9	22.9	81.7	76.5	88.2	74.3
Gemini 3.8 Flash	67.3	30.0	59.1	55.8	69.7	59.4

Table 3: Question answering and proactive requests in CookSim. Results are averaged across three runs with measured latency. Temp. order: temporal order; Act. count: action count; Time dur.: duration.

CookSim

VHSim

In-time success

Det. recall

Interaction quality

In-time success

Det. recall

Interaction quality

Assistant

Fac. grnd.

Sit. rel.

Act. guid.

User intent

Guid. conc.

Overall

Fac. grnd.

Sit. rel.

Act. guid.

User intent

Guid. conc.

Overall

Frontier API models

Gemini 4 Argon

33.1 [-1pt] (-27.8)

66.9 [-1pt] (-13.3)

65.3 [-1pt] (-4.9)

84.9 [-1pt] (-5.2)

63.0 [-1pt] (-8.4)

86.0 [-1pt] (+3.7)

71.1 [-1pt] (-5.8)

57.9 [-1pt] (-6.0)

12.9 [-1pt] (-2.7)

71.4 [-1pt] (-9.7)

49.5 [-1pt] (-10.7)

79.1 [-1pt] (-11.6)

43.7 [-1pt] (-17.1)

77.0 [-1pt] (-6.2)

76.0 [-1pt] (-8.8)

43.1 [-1pt] (-12.4)

Gemini 3.8 Flash

29.8 [-1pt] (-16.7)

61.3 [-1pt] (-16.6)

77.6 [-1pt] (-5.9)

77.8 [-1pt] (-5.7)

58.5 [-1pt] (-7.0)

66.9 [-1pt] (+0.7)

88.4 [-1pt] (-2.5)

66.9 [-1pt] (-6.5)

10.7 [-1pt] (-4.0)

57.8 [-1pt] (-14.4)

70.0 [-1pt] (-9.2)

82.2 [-1pt] (-6.2)

53.2 [-1pt] (-10.2)

72.2 [-1pt] (-3.2)

90.2 [-1pt] (-2.5)

61.4 [-1pt] (-9.7)

GPT-6.1 Sol

30.0 [-1pt] (-12.9)

67.5 [-1pt] (-12.5)

74.8 [-1pt] (-5.5)

87.6 [-1pt] (-4.8)

69.2 [-1pt] (-8.9)

90.6 [-1pt] (-0.6)

86.3 [-1pt] (-5.4)

68.4 [-1pt] (-7.5)

8.0 [-1pt] (-7.6)

59.0 [-1pt] (-1.2)

71.7 [-1pt] (-6.1)

86.2 [-1pt] (-6.7)

44.3 [-1pt] (-13.0)

84.0 [-1pt] (-5.1)

88.3 [-1pt] (-3.9)

62.4 [-1pt] (-8.1)

Table 4: Results with measured assistant latency. Parentheses show changes from the fixed two-tick setting in percentage points. Results are averaged across three independent runs.

Figure 4: Effect of assistant latency on task success and interaction quality. Open markers show the fixed two-tick setting; filled markers show measured latency. Lines connect the same model under the two settings. Scores are averaged across three independent runs.

Figure 5: Behavior comparison of real human and simulated human agent. See Tab. 13 in the appendix for measure definitions.

Figure 6: Qualitative examples of assistant interaction trajectories. It illustrates recurring problems across assistant types that our dynamic evaluation expose.

Figure 7: Qualitative example from RL. The base assistant misperceives the task state and fails to prevent the error. After RL, the assistant gives timely guidance that prevents the error.

User	Assistant	In-time success	Det. recall	Interaction quality
Gemini 3.7 Flash	Qwen3.5-4B	$33.3\pm 3.3$	$\mathbf{49.0}\pm 11.9$	$29.6\pm 4.8$
Gemini 3.7 Flash	+ RL	$\mathbf{42.2}\pm 6.9$	$34.9\pm 6.7$	$\mathbf{31.1}\pm 0.4$
Gemini 3.8 Flash	Qwen3.5-4B	$64.4\pm 6.9$	$\mathbf{43.0}\pm 14.5$	$24.3\pm 2.1$
Gemini 3.8 Flash	+ RL	$\mathbf{67.8}\pm 7.7$	$28.7\pm 3.4$	$\mathbf{30.2}\pm 0.3$

Table 5: CookSim validation results before and after RL, by simulated user model. + RL denotes 12 updates. Scores are percentages, reported as mean

\pm

SD. Bold marks higher means within pairs.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

User model	In-time success (%)	Any success (%)	Words/episode
Gemini 3.1 Pro	28.9	56.7	228.6
GPT-6 Astra	23.3	48.9	215.2
Gemini 3.7 Flash	21.1	52.2	147.1
Qwen3.8-27B	2.2	27.8	423.7

Appendix

Table 6: Sensitivity to the simulated-user model. Any success ignores the completion deadline. Word counts refer to the simulated user, not the assistant.

	In-time success (%)		Detection recall (%)		Interaction quality (%)
Assistant	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD
Fixed two-tick delay
Gemini 4 Argon	64.0 / 58.7 / 60.0	60.9 $\pm$ 2.3	81.1 / 80.6 / 79.0	80.2 $\pm$ 0.9	63.5 / 64.5 / 63.5	63.8 $\pm$ 0.5
Gemini 3.1 Pro	46.0 / 51.3 / 47.3	48.2 $\pm$ 2.3	83.4 / 87.0 / 84.8	85.1 $\pm$ 1.5	36.0 / 37.1 / 38.2	37.1 $\pm$ 0.9
Gemini 3.8 Flash	48.0 / 46.0 / 45.3	46.4 $\pm$ 1.1	77.7 / 77.7 / 78.3	77.9 $\pm$ 0.3	73.7 / 72.8 / 73.6	73.4 $\pm$ 0.4
GPT-6.1 Sol	45.3 / 42.7 / 40.7	42.9 $\pm$ 1.9	79.6 / 80.6 / 79.6	79.9 $\pm$ 0.5	76.2 / 75.1 / 76.1	75.8 $\pm$ 0.5

Appendix

Table 7: Independent-run results in CookSim. Scores are percentages for three independent runs (R1–R3), with their mean and population standard deviation (SD).

	In-time success (%)		Detection recall (%)		Interaction quality (%)
Assistant	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD
Fixed two-tick delay
Gemini 4 Argon	20.0 / 13.3 / 13.3	15.6 $\pm$ 3.1	81.4 / 79.8 / 82.3	81.2 $\pm$ 1.0	57.5 / 54.8 / 54.4	55.6 $\pm$ 1.4
Gemini 3.1 Pro	17.3 / 12.0 / 14.7	14.7 $\pm$ 2.2	89.6 / 90.1 / 89.3	89.6 $\pm$ 0.3	53.8 / 56.4 / 54.8	55.0 $\pm$ 1.1
Gemini 3.8 Flash	16.0 / 17.3 / 10.7	14.7 $\pm$ 2.9	73.5 / 73.7 / 69.4	72.2 $\pm$ 2.0	69.5 / 71.4 / 72.6	71.2 $\pm$ 1.3
GPT-6.1 Sol	17.3 / 16.0 / 13.3	15.6 $\pm$ 1.7	56.7 / 64.1 / 60.0	60.3 $\pm$ 3.0	70.0 / 70.8 / 70.8	70.5 $\pm$ 0.4

Appendix

Table 8: Independent-run results in VHSim. Scores are percentages for three independent runs (R1–R3), with their mean and population standard deviation (SD).

	In-time success (%)		Detection recall (%)		Interaction quality (%)
Assistant	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD	R1 / R2 / R3	Mean $\pm$ SD
ScreenSim
Gemini 4 Argon	82.2 / 86.7 / 82.2	83.7 $\pm$ 2.1	100.0 / 100.0 / 100.0	100.0 $\pm$ 0.0	84.7 / 87.6 / 89.1	87.1 $\pm$ 1.8
Gemini 3.1 Pro	85.6 / 84.4 / 88.9	86.3 $\pm$ 1.9	99.4 / 100.0 / 99.4	99.6 $\pm$ 0.3	85.7 / 83.7 / 85.2	84.9 $\pm$ 0.9
Gemini 3.8 Flash	78.9 / 75.6 / 81.1	78.5 $\pm$ 2.3	97.6 / 97.6 / 98.3	97.8 $\pm$ 0.3	89.4 / 89.5 / 89.3	89.4 $\pm$ 0.1
GPT-6.1 Sol	85.6 / 88.9 / 87.8	87.4 $\pm$ 1.4	100.0 / 100.0 / 100.0	100.0 $\pm$ 0.0	86.7 / 85.7 / 87.2	86.5 $\pm$ 0.6

Appendix

Table 9: Independent-run results in ScreenSim. Scores are percentages for three independent runs (R1–R3), with their mean and population standard deviation (SD).

Assistant	User	Split	Checkpoint	In-time task success	Detection recall	Interaction quality
Qwen3.5-4B	Gemini 3.7 Flash	Train	Base	$32.2\pm 4.3$	$\mathbf{49.1}\pm 7.0$	$23.5\pm 0.3$
		Train	+ RL	$\mathbf{42.2}\pm 2.9$	$43.8\pm 6.8$	$\mathbf{32.5}\pm 1.9$
		Validation	Base	$33.3\pm 3.3$	$\mathbf{49.0}\pm 11.9$	$29.6\pm 4.8$
		Validation	+ RL	$\mathbf{42.2}\pm 6.9$	$34.9\pm 6.7$	$\mathbf{31.1}\pm 0.4$
		Overall	Base	$32.4\pm 4.1$	$\mathbf{49.0}\pm 4.4$	$23.7\pm 0.5$
		Overall	+ RL	$\mathbf{42.2}\pm 2.5$	$42.1\pm 4.9$	$\mathbf{32.5}\pm 1.8$

Appendix

Table 10: CookSim RL results by split. Scores are percentages, reported as mean

\pm

SD. + RL denotes 12 updates; bold marks higher means within pairs. Success and detection SDs are sample SDs over three passes. Gemini-judged interaction quality uses a matched subset, with population SD over three passes for train and overall and two for validation.

Assistant	User	Split	Checkpoint	Strict in-time task success	Native detection recall	Interaction quality
Qwen3.5-4B	Baseline	Development	Base	$24.0\pm 11.8$	$91.2\pm 1.1$	$\mathbf{30.0}\pm 0.6$
		Development	+ RL	$\mathbf{27.1}\pm 9.5$	$\mathbf{94.9}\pm 4.7$	$29.7\pm 2.9$
		Frozen test	Base	$\mathbf{27.8}\pm 6.9$	$85.4\pm 3.2$	$\mathbf{36.4}\pm 1.7$
		Frozen test	+ RL	$25.6\pm 1.9$	$\mathbf{88.7}\pm 6.1$	$31.4\pm 1.3$
	Classic novice	Development	Base	$2.1\pm 1.8$	—	$\mathbf{27.0}\pm 3.2$
		Development	+ RL	$\mathbf{4.2}\pm 1.8$	—	$22.1\pm 1.5$

Appendix

Table 11: ScreenSim no-manual RL results. Scores are percentages, reported as mean

\pm

SD over three passes. + RL denotes 12 updates; bold marks the higher mean within each user, split, and metric. Success and recall SDs are sample SDs; interaction-quality SDs are population SDs. Interaction quality uses the primary GPT-5.4-mini judge; dashes indicate unavailable recall.

Reported family	Example form	Grading evidence
Temporal order	Did I prepare $A$ before $B$ ?	Completion times of both events; the stated relation must agree.
Action count	How many times have I visited $X$ ?	Number of realized visits; the integer must match exactly.
Duration	How long has $X$ been cooking?	Loading, ignition, and completion times; numeric duration answers allow five ticks of tolerance. The family also includes longest-on-heat and past-completion queries.
Event recall	Did I already do $A$ ? Does $X$ exist?	Recorded event occurrence or world inventory; the answer must reject an incorrect premise.
Proactive request	Tell me when $X$ is ready.	A correct readiness notification within the state-derived window, not merely an acknowledgement of the request.

Appendix

Table 12: Question and request families in the main evaluation. Concrete items, stations, and actions are bound from the current episode.

Measure	Episode-level statistic
Utterance length	Mean number of words per nonempty user utterance.
Speech rate	Spontaneous user utterances per 100 ticks, with the time normalization described below.
Responsiveness	Fraction of assistant utterances followed by a user reply within ten ticks.
Suggestion uptake	Fraction of assistant utterances naming an ingredient followed by pickup or scooping of that ingredient within 15 ticks.
Disagreement	Fraction of user utterances that contest an assistant claim.
Guidance requests	Fraction of user utterances asking for the next action or help; text patterns are used for humans and response labels for simulated users.

Appendix

Table 13: Behavioral measures for human–simulation comparisons. Episode means exclude cases without an eligible observation. Interruption ignoring combines absent responses and explicit requests for fewer interruptions; the two counts may overlap.

Measure	Human	Simulated	Similarity
Words per utterance	5.87	7.38	0.89
Utterances per 100 ticks	0.39	16.10	0.05
Replies within ten ticks (%)	2.0	79.0	0.05
Skill: task success (%)	50.0 $\to$ 79.3	25.0 $\to$ 100.0	0.56
Trust: suggestion uptake (%)	32.3 $\to$ 24.3	32.9 $\to$ 20.1	0.77
Trust: disagreement (%)	8.7 $\to$ 30.1	9.3 $\to$ 36.7	0.88

Appendix

Table 14: Human and simulated behavior. Arrows show group means in the following directions: first to last play / novice to expert (skill), high to low trust, step-by-step to minimal interruption, and cautious to faster execution. Similarity compares the unrounded differences between groups. Speech-rate normalization is described in Appendix F .

Speech act	Human (%)	Simulated (%)
Greeting	10.4	0.0
Comment on the assistant or its silence	2.6	0.1
Expression of frustration or hostility	3.2	0.1
Request for a future notification	5.2	0.0
Strategy or planning question	6.5	0.0
Question about location or next action	22.1	15.9

Appendix

Table 15: Distribution of conversational behaviors. We apply the same rule-based classifier to 154 human and 63,512 simulated utterances. The resulting distributions have similarity 0.61 under Equation 9 .

Error type	Annotations	Episodes
Incorrect preparation	42	31
Unattended cooking	20	16
Forgotten next step	19	19
Wrong assembly order	13	13
Serving raw food	8	7
Repeated step	5	5

Appendix

Table 16: Action errors in human task execution. An LLM-assisted audit identifies procedural errors in 55 unassisted episodes. An episode may contain multiple error types. The final row reports behaviors outside the scheduled-error taxonomy.

Environment	Evaluated case construction	Cases	Episodes/run
CookSim	Five recipes, two kitchen layouts, five schedules per recipe–layout pair; three personas per case.	50	150
VHSim	Household tasks with multiple progress-triggered deviations; three assigned personas per case.	25	75
ScreenSim	Multi-step phone configurations with two scheduled mistakes; three assigned personas per case.	30	90

Appendix

Table 17: IVG-Bench benchmark composition. Each case is paired with three personas. VHSim and ScreenSim select these personas from nine available profiles.

Environment	Cases	Errors	Questions	Requests	Plan changes	Events/case, mean [range]
CookSim	50	140	88	29	3	5.20 [2–7]
VHSim	25	51	0	0	0	2.04 [2–3]
ScreenSim	30	60	0	0	0	2.00 [2–2]
Total	105	251	88	29	3	3.53 [2–7]

Appendix

Table 18: Scheduled-event distribution. Counts are per case before persona assignment. Requests denotes proactive requests. Temporal questions, proactive requests, and plan changes are evaluated in CookSim. The number of events per case is 2, 3, 4, 5, 6, or 7 in 55, 12, 1, 11, 16, and 10 cases, respectively.

	Task Goal	Action classes
CookSim	loaded_burrito Cook rice and sliced steak; chop tomato and lettuce; plate in that order. shrimp_pasta Boil noodles, chop tomato, and sear shrimp; plate in that order. steak_frites Sear steak slices; fry two chip portions; plate steak first.	Navigate: go to <food source> ; go to <cook station> ; go to <prep station> ; go to <counter> ; go to plates/pass/trash . Manipulate: pick up; put down; scoop; chop; turn on; trash; serve.
VHSim	post_meal_cleanup Load dishes into the dishwasher; return the apple to the fridge; close both. collect_dishes Gather three plates and two glasses; load and close the dishwasher. return_belongings Return toilet paper to the bathroom counter, a book to the desk, and a pillow to the bed.	Navigate: turn left/right; go straight; look up/down; go through the visible door; turn to <obj> ; go to <visible obj/place> . Manipulate: pick up <obj> ; put <obj> on/in <target> ; open/close <obj> ; switch on/off <obj> ; drink <obj> ; sit on <obj> ; stand up .
ScreenSim	app_privacy Disable one app’s location, tracking, and background refresh only. readability Enlarge and bold text; clarify buttons; increase contrast; reduce motion. lock_screen_security Block Control Center, Today View, Wallet, and missed-call return.	Input: tap <label> ; type <text> . Navigate and adjust: swipe up/down; swipe left/right on <label> ; press back/home.

Appendix

Table 19: Example tasks and semantic action spaces. Angle brackets indicate object or place references resolved by the grounding module. Tasks illustrate the supported environments, including configurations beyond the evaluated subset.

	CookSim			VHSim
Assistant	Delay (s)	Fixed	Measured	Delay (s)	Fixed	Measured
Gemini 4 Argon	6.5	62.3	39.9	8.1	46.2	27.0
Gemini 3.1 Pro	13.2	57.1	19.3	11.7	59.0	32.3
Gemini 3.8 Flash	5.1	48.6	31.2	6.1	37.9	21.5
GPT-6.1 Sol	5.2	49.0	35.7	5.5	31.0	23.6
GPT-6 Astra	6.6	48.5	36.8	8.2	33.1	15.4

Appendix

Table 20: Assistant response time and error prevention. Delay is the mean observed response time across all three measured-delay runs, before conversion to simulation ticks and capping. Fixed and measured columns report error prevention (%) under each delay setting.

Category	Item	Pass criterion
Factual grounding	T1	Claims about the current state and the user’s completed actions are true.
	T2	Referenced entities, places, controls, rules, and task requirements exist.
Situational relevance	S1	The response addresses the current action, subgoal, live concern, pending request, or announced intent.
	S2	A pending question, request, or objection is answered, accepted, rejected with a reason, or deferred for a stated urgent reason.
Actionable guidance	H1	The first advised action can start now, or its prerequisite is supplied.
	H2	Advice advances the current plan, answers the user’s request, or repairs the error.

Appendix

Table 21: Interaction-quality rubric. A category passes when all applicable items pass. Failure on either factual-grounding item gates the overall score. Items unrelated to the response’s context are marked not applicable.

Axis	Lower-level disposition	Higher-level disposition
Skill	Broad goals are difficult to translate into a reliable sequence.	Maintains the workflow and reasons ahead about timing, tools, and dependencies.
Trust	Compares disputed claims with memory and visible evidence before changing course.	Takes plausible, situation-linked guidance seriously, while noticing clear contradictions.
Speed	Prefers methodical, cautious execution.	Values momentum, overlapping work, and avoiding unnecessary delay.
Interruption	Welcomes close, step-by-step guidance.	Finds non-urgent commentary intrusive and keeps replies brief while working.

Appendix

Table 22: Persona instructions. The two endpoint levels for each axis are summarized above; the middle level describes an intermediate preference or ability.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 1,015 papers

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
Oct 5, 2026Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini +5AI Agent Benchmarks
University of Amsterdam · Eindhoven University of Technology · Eberhard Karls Universität Tübingen
AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University
WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation
May 25, 2026Kaining Ying, Hengrui Hu, Siyu Ren +6World Model Evaluation Interactive World Models
Fudan University · Meituan Longcat Team

Oct 5, 2026J/K move · Enter open · S save

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini, Michelle Lorena Acevedo Callejas+4

University of Amsterdam · Eindhoven University of Technology · Eberhard Karls Universität Tübingen

AI Agent Benchmarks

Open paper AskPDF arXiv

Recent advances in multimodal foundation models yield strong performance on static perception and reasoning benchmarks, yet such evaluations largely overlook a central aspect of intelligence: acting competently in dynamic environments over extended time horizons. We introduce PlaySuite, a large-scale benchmark for evaluating interactive visual intelligence across more than 5K open-source video games curated from PyWeek and itch.io. Spanning diverse genres and engines, including Pygame, HTML5, Godot, and Unity, these independent games are largely out-of-distribution for current models, reducing the likelihood that success can be achieved by retrieving memorized walkthroughs or web-scale training artifacts. To enable scalable evaluation across heterogeneous titles, we develop a unified closed-loop interaction framework optimized for HPC clusters alongside a Video-LLM-as-a-judge protocol that maps observable gameplay milestones to standardized progress levels. We evaluate fourteen recent open models spanning vision-language models, computer-use agents, and vision-language-action models. Our results yield strong evidence of a perception-action gap: despite strong reasoning capabilities, current models struggle to make sustained progress and exhibit recurring failures in spatial grounding, action execution, and self-correction. PlaySuite provides a reproducible and extensible testbed for measuring progress from visual perception to goal-directed interaction, and a foundation for developing models that can act, adapt, and generalize in dynamic visual environments.

Abstract

VLM Evaluation

Explore similar work

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation