Vis2State: Visual Reasoning through Executable State Abstractions

Sheng Jin; Wentao Liu; Chen Qian

cs.CVNov 5, 2026

Vis2State: Visual Reasoning through Executable State Abstractions

Authors: Sheng Jin, Wentao Liu, Chen Qian

Organizations: SenseTime Research and Tetras.AI

Abstract

Solving visual puzzles requires both accurate interpretation of visual evidence and reliable computation under task-specific rules. Embedding visual assumptions and task constraints directly in solving code makes the represented problem difficult to inspect and revise independently. We introduce Vis2State, a training-free agentic framework that abstracts visual puzzles into explicit, revisable problem states linking visual inspection and execution. Visual inspection guides state construction and refinement, while generated programs solve the represented problem. The model constructs both the state and the program without predefined puzzle schemas or solving procedures. Inspection and execution feedback allow it to revise the problem representation and solving procedure independently. Vis2State achieves the state-of-the-art performance on four benchmarks. Cross-model evaluation further demonstrates consistent gains. Ablations, error analyses, and fixed-program state interventions further support the effectiveness of the state-centered design. The code will be made publicly available to facilitate future research.

Figures & tables

Figure 1: Vis2State turns visual evidence into explicit problem states for computational solving. Two illustrative puzzles show how (a) inspection corrects a misread coordinate and (b) execution over explicit state finds the shortest path.

Figure 2: Inspection, State, and Execution in Vis2State. Inspection gathers visual evidence to construct the State , and Execution solves the represented problem. The model refines the state or program through feedback before submitting an answer.

Tool	Output
crop_region	Cropped image with coordinates.
pixel_probe	RGB values at queried points and regional color statistics.
edge_view	Image of local RGB differences.
color_extent	Bounding box and count of matched pixels, with an annotated image.
color_components	Bounding boxes and sizes of connected color components, with an annotated image.

Table 1: Visual inspection tools, with interaction design informed by V* [ 44 ] and Visual Sketchpad [ 19 ] .

	Method	Overall	Board	Calendar	Chain	Checker	Clock	Colour	Map	Maze	Move box
	GPT-4V [ 27 ]	30.3	48.0	57.0	21.0	33.0	25.0	27.0	34.0	30.0	29.0
	Claude 3 Opus [ 1 ]	30.9	48.0	39.0	23.0	35.0	37.0	16.0	33.0	23.0	19.0
	GPT-5.2 [ 29 ]	54.6	49.0	70.0	23.0	56.0	76.0	38.0	51.0	38.0	39.0
	o3 (high) [ 30 ]	62.3	60.0	96.0	75.0	56.0	98.0	47.0	5.0	39.0	52.0
	o4-mini (high) [ 30 ]	65.4	52.0	99.0	77.0	97.0	91.0	47.0	3.0	42.0	50.0
Closed-source	Gemini 3 Pro Preview [ 16 ]	81.3	93.0	99.0	61.0	94.0	98.0	43.0	61.0	49.0	87.0

Table 2: Accuracy (%) across 18 task families on AlgoPuzzleVQA [ 14 ] . Bold indicates the best model result in each column.

Method	Overall	Algorithmic	Analogical	Deductive	Inductive	Spatial
Human (50th Percentile)	75.0	88.0	66.0	80.0	50.0	90.0
Claude 3.7 Sonnet [ 2 ]	48.3	64.5	48.3	65.0	26.8	37.4
Claude Opus 4.1 (thinking) [ 3 ]	48.8	–	–	–	–	–
Gemini 2.5 Pro [ 10 ]	49.5	60.0	64.0	60.0	29.7	36.4
o3 [ 30 ]	54.0	64.5	68.3	69.5	27.3	42.7
o4-mini [ 30 ]	57.0	65.3	68.7	75.5	33.0	45.5

Table 3: Accuracy (%) by reasoning category on VisualPuzzles [ 34 ] . Bold indicates the best model result in each column.

Method	Overall	Quantity	Spatiality	Position	Attribute	Style	Other
Human	51.4	45.3	52.7	71.1	50.0	47.5	44.2
GPT-4o [ 20 ]	26.3	28.6	24.7	27.2	26.8	20.0	25.9
Claude Opus 4.1 (thinking) [ 3 ]	27.9	–	–	–	–	–	–
Gemini 2.0 Pro [ 22 ]	28.0	29.7	24.2	27.9	30.5	22.2	33.3
GPT-5 (high) [ 32 ]	28.5	–	–	–	–	–	–
o3 [ 30 ]	29.5	24.2	31.0	34.6	27.0	25.0	41.9

Table 4: Accuracy (%) across six reasoning categories on VisuLogic [ 47 ] . Bold indicates the best model result in each column.

		Single-concept					Dual-concept
Method	Overall	N	C	Z	S	Avg.	N $+$ S	C $+$ N	N $+$ Z	C $+$ S	Z $+$ S	C $+$ Z	Avg.
Claude 3 Opus [ 1 ]	42.0	47.0	32.5	33.5	44.5	39.4	48.0	54.5	34.0	42.5	33.0	50.0	43.7
GPT-4V [ 27 ]	45.9	67.5	42.0	35.0	41.0	46.4	52.5	56.0	31.0	47.0	31.5	55.0	45.5
GPT-5.2 [ 29 ]	67.3	69.5	98.0	66.0	71.5	76.3	75.0	68.5	37.0	61.0	68.5	58.5	61.4
o3 (high) [ 30 ]	84.0	99.5	99.0	87.5	74.0	90.0	92.0	97.5	45.0	92.0	97.0	57.0	80.1
o4-mini (high) [ 30 ]	85.7	99.0	99.5	86.0	70.5	88.8	96.0	99.5	62.0	86.5	85.5	72.5	83.7

Table 5: Accuracy (%) on single- and dual-concept puzzles in PuzzleVQA [ 9 ] . N/C/S/Z denote numbers/colors/shapes/size. Bold indicates the best model result in each column.

Figure 3: Category-level accuracy and gains over Direct CoT. Blue bars show Direct CoT accuracy. Orange segments show the accuracy gain from Vis2State. Each full bar and its numeric label give Vis2State accuracy (%). In PuzzleVQA, N/C/S/Z denote numbers/colors/shapes/size. Vis2State improves 36 of 39 categories and ties the remaining three at 100.0%.

Benchmark	# Wins	# Losses	Gain [95.0% CI]
AlgoPuzzleVQA	295	16	15.5 [13.7, 17.3]
VisualPuzzles	237	80	13.4 [10.5, 16.4]
VisuLogic	238	102	13.6 [10.1, 17.1]
PuzzleVQA	208	47	8.1 [6.6, 9.6]

Table 6: Paired comparison with Direct CoT. # Wins count questions answered correctly only by Vis2State. # Losses count those answered correctly only by Direct CoT. Intervals are paired 95.0% bootstrap confidence intervals for the accuracy gain.

Configuration	Visual Inspection	Explicit State	Code Execution	AlgoPuzzleVQA	VisualPuzzles	VisuLogic	PuzzleVQA
Direct CoT	$\times$	$\times$	$\times$	82.3	61.6	35.5	85.2
Inspection only	$\checkmark$	$\times$	$\times$	92.1	72.0	43.0	92.1
Execution only	$\times$	$\times$	$\checkmark$	90.0	64.0	37.7	89.5
Inspection + Execution	$\checkmark$	$\times$	$\checkmark$	96.9	73.5	43.1	91.7
Vis2State	$\checkmark$	$\checkmark$	$\checkmark$	97.8	75.0	49.1	93.3

Table 7: Ablation study. We compare visual inspection, code execution, and their combination using DeepSeek V4.1 Flash. Vis2State connects inspection and execution through an explicit state–program interface and achieves the highest accuracy on all four benchmarks. Checkmarks indicate the components included in each configuration. Results are accuracy (%), with the best score in each column in bold.

Figure 4: Direct CoT error profiles and corrections by Vis2State. Inner rings show Direct CoT error-group proportions. Outer rings show cases corrected by Vis2State (light, hatched) or not corrected (dark, solid), with within-group correction rates. Centers report corrected/total errors. State / Reasoning combines representation and rule errors; Search / Computation combines search, state updates, arithmetic, and answer comparison. Compound includes cross-group, answer-extraction, random-parser, and unattributed errors.

Figure 5: Case study on Move Box. Direct CoT misreads the goal position and overlooks a wall, producing an incorrect two-push solution. Vis2State explicitly represents the puzzle state and searches legal push sequences, finding the minimum of four pushes.

State condition	AlgoPuzzleVQA	PuzzleVQA
Original state	200 (100.0%)	200 (100.0%)
Counterfactual	196 (98.0%)	187 (93.5%)
Answer-preserving control	200 (100.0%)	197 (98.5%)

Table 8: State interventions. Correct counts and accuracies on 200 questions per dataset.

Method	AP	VP	VL	PQ
Direct CoT (GLM-5.3-Flash)	82.3	67.2	36.2	89.4
Vis2State (GLM-5.3-Flash)	94.5	75.1	44.2	96.3
Gain (pp)	+12.2	+7.9	+8.0	+6.9

Table 9: Cross-model evaluation. Accuracy (%) with GLM-5.3-Flash on AlgoPuzzleVQA (AP), VisualPuzzles (VP), VisuLogic (VL), and PuzzleVQA (PQ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Limit
Model calls per attempt ( $B$ )	22
Active runtime per attempt ( $T$ )	1,800 s
Output tokens per model call	65,536
HTTP timeout per request	1,200 s
Python time per execution	3 s
Python memory per execution	256 MB

Appendix

Table S1: Runtime settings. Request timeouts are also bounded by the remaining per-attempt time budget.

Method	Overall	Board	Calendar	Chain	Checker	Clock	Colour	Map	Maze	Move box
Questions	1,800	100	100	100	100	100	100	100	100	100
Direct CoT	82.3	100.0	99.0	56.0	92.0	86.0	40.0	65.0	56.0	84.0
Vis2State (Ours)	94.5	100.0	100.0	74.0	100.0	100.0	99.0	71.0	92.0	98.0
Gain (pp)	+12.2	0.0	+1.0	+18.0	+8.0	+14.0	+59.0	+6.0	+36.0	+14.0

Appendix

Table S3 : GLM-5.3-Flash on AlgoPuzzleVQA . Accuracy (%) across 18 task families. Bold marks the better result, including ties.

Method	Overall	Algorithmic	Analogical	Deductive	Inductive	Spatial
Questions	1,168	262	211	200	209	286
Direct CoT	67.2	80.9	81.0	83.5	35.4	56.3
Vis2State (Ours)	75.1	91.2	81.0	86.0	53.1	64.3
Gain (pp)	+7.9	+10.3	0.0	+2.5	+17.7	+8.0

Appendix

Table S4 : GLM-5.3-Flash on VisualPuzzles . Accuracy (%) across five reasoning categories. Bold marks the better result, including ties.

Method	Overall	Quantity	Spatiality	Position	Attribute	Style	Other
Questions	1,000	353	231	136	82	90	108
Direct CoT	36.2	37.7	29.9	45.6	39.0	24.4	40.7
Vis2State (Ours)	44.2	47.0	29.9	55.9	37.8	51.1	50.0
Gain (pp)	+8.0	+9.3	0.0	+10.3	-1.2	+26.7	+9.3

Appendix

Table S5 : GLM-5.3-Flash on VisuLogic . Accuracy (%) across six reasoning categories. Bold marks the better result, including ties.

		Single-concept					Dual-concept
Method	Overall	N	C	Z	S	Avg.	N $+$ S	C $+$ N	N $+$ Z	C $+$ S	Z $+$ S	C $+$ Z	Avg.
Questions	2,000	200	200	200	200	800	200	200	200	200	200	200	1,200
Direct CoT	89.4	93.0	100.0	95.0	68.5	89.1	91.0	99.0	85.0	86.0	98.5	78.0	89.6
Vis2State (Ours)	96.3	99.0	100.0	100.0	74.5	93.4	99.5	99.5	94.5	97.5	100.0	98.5	98.3
Gain (pp)	+6.9	+6.0	0.0	+5.0	+6.0	+4.3	+8.5	+0.5	+9.5	+11.5	+1.5	+20.5	+8.7

Appendix

Table S6 : GLM-5.3-Flash on PuzzleVQA . Accuracy (%) across single- and dual-concept categories. Bold marks the better result, including ties. N/C/S/Z denote numbers/colors/shapes/size.

Figure S1 : Accuracy and output-token cost with DeepSeek V4.1 Flash. Full-benchmark accuracy is plotted against mean output tokens on sampled questions. Panels use a shared logarithmic cost scale and accuracy ranges spanning 20 percentage points. Stars denote Vis2State, dashed lines indicate Direct CoT accuracy, and annotations show the corresponding accuracy gains.

Figure S2 : Inspection, execution, and their combination. (a) Color probes refine tile correspondences into a six-tile cycle. (b) Search over the maze state computes the shortest path. (c) Revised coordinates and search yield the shortest WEEKLY walk. Task accuracies in (a,b) compare Direct CoT with Vis2State over 100 questions each. Only decisive steps are shown. Coordinates are one-based (row, column) in (a) and zero-based $(x,y)$ from the top-left in (c).

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 349 papers

VisualCoding: Learning Executable Skills for Reliable Visual Reasoning
Oct 9, 2026Jiaqi Liu, Ruinan Jin, Xiaoxiao LiVisual Program Synthesis Visually Grounded Reasoning
Department of Electrical and Computer Engineering, University of British Columbia · Vector Institute
LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles
Sep 28, 2026Bingo Zhang, Haochuan Lu, Zongjie Li +3GUI Agents Long-Horizon Agent Evaluation
Vera Praxis · Tencent · The Hong Kong University of Science and Technology +2
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Jul 24, 2026Yan Yang, Xiangru Jian, Ziyang Luo +7Computer-Use Agents Tool-Using Agents
Salesforce AI Research

Oct 9, 2026J/K move · Enter open · S save

VisualCoding: Learning Executable Skills for Reliable Visual Reasoning

Jiaqi Liu, Ruinan Jin, Xiaoxiao Li

Department of Electrical and Computer Engineering, University of British Columbia · Vector Institute

Visual Program Synthesis Visually Grounded Reasoning

Open paper AskPDF arXiv

Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and produce the final answer. Errors and hallucinations at these steps can prevent acquired experience from translating into reliable single-run performance. We propose VisualCoding, which combines execution-guided acquisition of reusable visual programs with direct adoption of their answers on applicable inputs. Execution-Guided Program Acquisition composes visual primitives and additional code into family-level solvers, then iteratively tests and revises them on labeled training instances. Scope-Aware Direct Execution runs a retained solver and returns its valid in-scope answer. We evaluate VisualCoding on four models across four benchmarks spanning 69 task families. VisualCoding exceeds the strongest evaluated baseline in mean accuracy by 4.45 percentage points with Gemini-3.1-Pro and by 14.78 points when reusing the frozen Pro-acquired artifacts with Flash-Lite. Ablations support execution-guided acquisition and show that direct execution outperforms textual or callable reuse of the same programs. Code is available at https://github.com/jiaq-liu/VisualCoding.

Abstract

Visually Grounded Reasoning

Explore similar work

VisualCoding: Learning Executable Skills for Reliable Visual Reasoning

LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents