cs.CVOct 9, 2026

VisualCoding: Learning Executable Skills for Reliable Visual Reasoning

Authors: Liu Jiaqi, Jin Ruinan, Li Xiaoxiao

Organizations: Department of Electrical and Computer Engineering, University of British Columbia · Vector Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and produce the final answer. Errors and hallucinations at these steps can prevent acquired experience from translating into reliable single-run performance. We propose VisualCoding, which combines execution-guided acquisition of reusable visual programs with direct adoption of their answers on applicable inputs. Execution-Guided Program Acquisition composes visual primitives and additional code into family-level solvers, then iteratively tests and revises them on labeled training instances. Scope-Aware Direct Execution runs a retained solver and returns its valid in-scope answer. We evaluate VisualCoding on four models across four benchmarks spanning 69 task families. VisualCoding exceeds the strongest evaluated baseline in mean accuracy by 4.45 percentage points with Gemini-3.1-Pro and by 14.78 points when reusing the frozen Pro-acquired artifacts with Flash-Lite. Ablations support execution-guided acquisition and show that direct execution outperforms textual or callable reuse of the same programs. Code is available at https://github.com/jiaq-liu/VisualCoding.

Figures & tables

Figure 1: From text skills to executable experience. Agents can still make errors when applying text skills distilled from rollouts (top). VisualCoding refines reusable programs through execution feedback and directly adopts their valid answers for more reliable visual reasoning (bottom).
Figure 2: VisualCoding overview. Program acquisition: training examples, baseline statistics, and visual primitives support program search. Execution feedback guides revision, and scope fitting and selection retain a frozen program PP and scope gg . Direct execution: valid in-scope outputs directly supply the answer through an adapter.
MethodTIRMIRAGQAPVQAAvg.
Direct (TF)48.933.368.393.661.0
GEPA (TF)52.9 +4.035.0 +1.773.7 +5.497.8 +4.364.9 +3.8
TextGrad (TF)54.0 +5.133.3 0.075.1 +6.896.7 +3.164.8 +3.8
Tool agent62.937.273.393.766.8
XSkill68.9 +6.042.4 +5.278.4 +5.196.9 +3.271.7 +4.9
Trace2Skill65.8 +2.945.2 +8.074.9 +1.695.7 +2.070.4 +3.6
Table 1: Main comparison with Gemini 3.1 Pro. Accuracy (%); Avg. equally weights four task-weighted dataset accuracies. GQA: GameQA-17; PVQA: PuzzleVQA; TF: tool-free inference. Bold/underline mark best/second-best distinct displayed accuracies within each executor (ties included). Small green/red numbers are gains/losses in percentage points: GEPA/TextGrad versus Direct; XSkill/Trace2Skill/SkillOpt/VC versus Tool agent. Protocol details: Appendix C.4 .
MethodTIR-BenchMIRAGameQAPuzzleVQAAvg.
(a) Gemini 3.1 Flash-Lite
Direct (TF)28.317.928.678.838.4
GEPA (TF)36.3 +8.018.5 +0.640.9 +12.391.8 +12.946.9 +8.4
TextGrad (TF)33.5 +5.216.3 -1.732.5 +3.984.4 +5.541.7 +3.3
Tool agent39.020.930.483.443.4
XSkill48.7 +9.728.4 +7.445.1 +14.893.0 +9.653.8 +10.4
Table 2: Cross-executor reuse. Accuracy (%); annotations follow Table 1 , ranked separately per executor. (a) External Pro-learned artifacts deploy with Flash-Lite; ours uses Flash-Lite training-side calibration. (b) DeepSeek/GPT reuse frozen FL routes and program readings without target calibration. Judge assignments and evaluation protocol: Appendix C.4 .
Figure 3: Representative cases. (a) TIR-Bench: lighter cells indicate lower boundary-matching costs. (b) MIRA: scores show the difference between reflected and original ORB match counts; C has the largest increase. (c) GameQA: the displayed path has 16 turns and is verified on the extracted grid. (d) PuzzleVQA: measured component areas identify six (A) as the answer.
VariantTIR-BenchMIRA
Tool agent62.937.2
(a) Acquisition
Full execution feedback75.946.6
Independent search73.546.0
No scope during acquisition68.342.0
(b) Deployment
Table 3: Acquisition and deployment ablations with Gemini-3.1-Pro. Complete-panel accuracy (%): TIR-Bench has 746 questions and MIRA 363. Panel (a) compares feedback conditions on five re-acquired families under controlled deployment settings. Panel (b) changes how frozen main-experiment programs are used. Both panels retain full-system outputs on unaffected questions. Feedback-granularity and deployment-configuration controls appear in the appendix.
Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

RoleDocumented entries
Image preparationarr , gray , background , fg_mask , ink_mask , hue_mask , stretch , upscale , rotate
Structure and geometrycomponents , fill_holes , holes , contour_stats , classify , min_area_rect
Color measurementscolor_name , n_colors , color_share , mean_color
Spatial patternsprojection , period , panels8 , apply_variant
Perceptual interfacesdetect , count_objects , ask
Question processingquestion , numbers_in
Appendix
Table 4: The fixed generic primitive library. Diagnostic returns can be included in the program’s diag ; debug images and substep checks support offline feedback.
CriterionHigh-precision tierGain tier
Minimum fitted precision0.950.50
Minimum fitted support per route35
Minimum pooled leave-one-out precision0.900.50
Minimum rule stability0.800.50
Maximum learned atoms per route22
Minimum numeric separation0.020.02
Appendix
Table 5: Acceptance hyperparameters.
MethodCi=0C_{i}=00<Ci<Kte0<C_{i}<K_{\mathrm{te}}Ci=KteC_{i}=K_{\mathrm{te}}Full panel
Task count194211341746
Tool agent (reference pool)0.0052.25100.0060.49
VisualCoding CORE41.7562.80100.0074.33
VisualCoding full50.9065.7697.2176.27
Appendix
Table 6: Success stratified by four observed Pro attempts on TIR. Entries are pass@1 (%) on common baseline-defined strata. CORE disables text skills but keeps the same programs, scopes, and family eligibility; the full policy retains skill fallback. The tool-agent row uses the reference pool that defines the strata.
Figure 4: Single-run accuracy across observed sampling strata. TIR-Bench tasks are grouped by the number of correct answers among four stored Pro tool-agent attempts. Bars reproduce Table 6 ; CORE retains programs and scopes with text fallback disabled. Grouping fixes the reference rates at 0% and 100% in the extreme strata.
FamilyNNBase @1Base @4Full @1
Color6771.6485.0772.39
Contrast3463.2479.4163.97
Instrument5422.6937.0425.00
Jigsaw804.3811.2595.00
Math8074.6983.7575.62
Maze8088.1295.00100.00
Appendix
Table 7: Family-level Pro sampling reference and full-policy accuracy on TIR. Percentages; all families are reported. OCR uses stored model-dependent program readings. Other rows can include program, skill, or base routes; these are complete-family scores, not in-scope precision.
DatasetFamiliesTrainTest
TIR-Bench12369746
MIRA20183363
GameQA17259514
PuzzleVQA206631337
Appendix
Table 8: Training and evaluation partitions.
Dataset ( nn )Tool agentPro policyPro no checkFL policyPro −- FL [95% CI]
TIR-Bench (746)39.061.565.068.5−7.0  [−9.4,−4.7]-7.0\;[-9.4,-4.7]
MIRA (363)20.936.438.341.6−5.2  [−8.0,−3.0]-5.2\;[-8.0,-3.0]
GameQA (514)30.452.758.665.4−12.6  [−16.0,−9.5]-12.6\;[-16.0,-9.5]
PuzzleVQA (1,337)83.497.198.798.8−1.7  [−2.5,−0.9]-1.7\;[-2.5,-0.9]
Avg. (datasets)43.461.965.168.6−6.6-6.6
Pooled (2,960)55.373.075.878.3−5.4  [−6.3,−4.5]-5.4\;[-6.3,-4.5]
Appendix
Table 9: Flash-Lite sensitivity to the family deployment policy. Accuracy (%); every column uses Flash-Lite as the executor. Pro and FL denote the executor used for deployment calibration; no check removes the additional executor-specific check from the Pro configuration. The final column reports Pro-policy minus FL-policy accuracy in percentage points, with paired task-bootstrap 95% intervals from 4,000 resamples for each dataset and the pooled sample. Avg. equally weights datasets; Pooled weights all 2,960 questions.
VariantTIR-BenchMIRA
Full method76.347.1
No text fallback76.046.3
Appendix
Table 10: Text-skill fallback control. Complete-panel accuracy (%) with Pro on the same TIR-Bench and MIRA panels as Table 3 .
FamilynnDir.GEPATGToolXST2SSOVC
color6750.759.7 +9.052.2 +1.574.674.6 0.077.6 +3.073.1 -1.570.1 -4.5
contrast3464.747.1 -17.658.8 -5.964.773.5 +8.864.7 0.067.7 +2.952.9 -11.8
instrument5424.125.9 +1.924.1 0.024.125.9 +1.924.1 0.031.5 +7.425.9 +1.9
jigsaw801.23.8 +2.55.0 +3.85.02.5 -2.57.5 +2.510.0 +5.095.0 +90.0
math8058.858.8 0.063.7 +5.077.577.5 0.072.5 -5.072.5 -5.081.2 +3.7
maze8091.298.8 +7.598.8 +7.590.097.5 +7.588.8 -1.396.2 +6.2100.0 +10.0
Appendix
Table 11: TIR-Bench: Gemini 3.1 Pro. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
color6741.846.3 +4.552.2 +10.461.264.2 +3.068.7 +7.573.1 +11.956.7 -4.5
contrast3441.244.1 +2.947.1 +5.955.961.8 +5.964.7 +8.858.8 +2.955.9 0.0
instrument5414.820.4 +5.614.8 0.013.024.1 +11.124.1 +11.127.8 +14.816.7 +3.7
jigsaw800.00.0 0.00.0 0.00.00.0 0.00.0 0.00.0 0.095.0 +95.0
math8033.845.0 +11.238.8 +5.051.256.2 +5.053.8 +2.555.0 +3.841.2 -10.0
maze8023.832.5 +8.823.8 0.037.535.0 -2.535.0 -2.528.7 -8.8100.0 +62.5
Appendix
Table 12: TIR-Bench: Gemini 3.1 Flash-Lite. Complete-family accuracy (%).
FamilynnDS ToolDS VCGPT ToolGPT VC
color6774.664.2 -10.450.762.7 +11.9
contrast3458.861.8 +2.926.532.4 +5.9
instrument5414.825.9 +11.116.731.5 +14.8
jigsaw805.095.0 +90.01.295.0 +93.8
math8081.277.5 -3.737.535.0 -2.5
maze8068.8100.0 +31.257.5100.0 +42.5
Appendix
Table 13: TIR-Bench: DeepSeek and GPT with frozen FL routes. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
billiards1414.328.6 +14.350.0 +35.735.764.3 +28.650.0 +14.350.0 +14.371.4 +35.7
convex hull2055.055.0 0.025.0 -30.090.090.0 0.095.0 +5.095.0 +5.090.0 0.0
cubes count205.05.0 0.00.0 -5.05.05.0 0.05.0 0.00.0 -5.05.0 0.0
cubes missing205.00.0 -5.05.0 0.05.00.0 -5.00.0 -5.05.0 0.00.0 -5.0
defuse a bomb1717.617.6 0.035.3 +17.623.535.3 +11.823.5 0.029.4 +5.917.6 -5.9
electric charge1478.685.7 +7.185.7 +7.178.685.7 +7.178.6 0.085.7 +7.185.7 +7.1
Appendix
Table 14: MIRA: Gemini 3.1 Pro. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
billiards1428.60.0 -28.628.6 0.035.742.9 +7.128.6 -7.128.6 -7.171.4 +35.7
convex hull2015.020.0 +5.010.0 -5.085.090.0 +5.090.0 +5.080.0 -5.085.0 0.0
cubes count200.00.0 0.00.0 0.00.05.0 +5.00.0 0.00.0 0.00.0 0.0
cubes missing205.00.0 -5.00.0 -5.00.05.0 +5.00.0 0.00.0 0.05.0 +5.0
defuse a bomb1735.347.1 +11.817.6 -17.617.629.4 +11.841.2 +23.523.5 +5.911.8 -5.9
electric charge1464.357.1 -7.171.4 +7.185.764.3 -21.478.6 -7.185.7 0.078.6 -7.1
Appendix
Table 15: MIRA: Gemini 3.1 Flash-Lite. Complete-family accuracy (%).
FamilynnDS ToolDS VCGPT ToolGPT VC
billiards1457.171.4 +14.314.371.4 +57.1
convex hull2095.090.0 -5.065.060.0 -5.0
cubes count205.05.0 0.00.00.0 0.0
cubes missing200.00.0 0.00.00.0 0.0
defuse a bomb1735.317.6 -17.623.529.4 +5.9
electric charge1471.471.4 0.050.064.3 +14.3
Appendix
Table 16: MIRA: DeepSeek and GPT with frozen FL routes. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
jewel2 q54090.087.5 -2.590.0 0.087.590.0 +2.587.5 0.087.5 0.087.5 0.0
jewel2 q62060.065.0 +5.070.0 +10.075.075.0 0.070.0 -5.065.0 -10.080.0 +5.0
maze q43551.445.7 -5.760.0 +8.645.777.1 +31.460.0 +14.357.1 +11.4100.0 +54.3
minesweeper q54087.590.0 +2.590.0 +2.590.090.0 0.090.0 0.080.0 -10.090.0 0.0
pacman q12020.020.0 0.015.0 -5.035.050.0 +15.045.0 +10.095.0 +60.095.0 +60.0
pacman q32055.060.0 +5.060.0 +5.065.090.0 +25.055.0 -10.075.0 +10.0100.0 +35.0
Appendix
Table 17: GameQA-17: Gemini 3.1 Pro. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
jewel2 q54072.562.5 -10.032.5 -40.030.057.5 +27.552.5 +22.570.0 +40.065.0 +35.0
jewel2 q62020.035.0 +15.030.0 +10.045.050.0 +5.045.0 0.040.0 -5.050.0 +5.0
maze q43520.02.9 -17.125.7 +5.714.314.3 0.08.6 -5.728.6 +14.3100.0 +85.7
minesweeper q54050.082.5 +32.575.0 +25.055.070.0 +15.062.5 +7.572.5 +17.585.0 +30.0
pacman q12010.010.0 0.05.0 -5.010.020.0 +10.030.0 +20.045.0 +35.095.0 +85.0
pacman q32010.015.0 +5.030.0 +20.010.015.0 +5.015.0 +5.015.0 +5.0100.0 +90.0
Appendix
Table 18: GameQA-17: Gemini 3.1 Flash-Lite. Complete-family accuracy (%).
FamilynnDS ToolDS VCGPT ToolGPT VC
jewel2 q54055.065.0 +10.060.065.0 +5.0
jewel2 q62060.060.0 0.035.020.0 -15.0
maze q43597.1100.0 +2.934.3100.0 +65.7
minesweeper q54070.085.0 +15.035.072.5 +37.5
pacman q12095.095.0 0.05.095.0 +90.0
pacman q32070.0100.0 +30.00.0100.0 +100.0
Appendix
Table 19: GameQA-17: DeepSeek and GPT with frozen FL routes. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
circle size number6770.195.5 +25.4100.0 +29.977.6100.0 +22.488.1 +10.4100.0 +22.4100.0 +22.4
color grid67100.0100.0 0.0100.0 0.0100.0100.0 0.0100.0 0.0100.0 0.0100.0 0.0
color hexagon67100.0100.0 0.0100.0 0.0100.0100.0 0.0100.0 0.0100.0 0.0100.0 0.0
color number hexagon6792.5100.0 +7.597.0 +4.591.0100.0 +9.0100.0 +9.0100.0 +9.0100.0 +9.0
color overlap squares6798.5100.0 +1.5100.0 +1.5100.0100.0 0.0100.0 0.0100.0 0.0100.0 0.0
color size circle6698.5100.0 +1.5100.0 +1.589.498.5 +9.187.9 -1.598.5 +9.1100.0 +10.6
Appendix
Table 20: PuzzleVQA: Gemini 3.1 Pro. Complete-family accuracy (%).
FamilynnDir.GEPATGToolXST2SSOVC
circle size number6774.6100.0 +25.4100.0 +25.471.692.5 +20.971.6 0.0100.0 +28.4100.0 +28.4
color grid6779.192.5 +13.479.1 0.095.5100.0 +4.597.0 +1.5100.0 +4.5100.0 +4.5
color hexagon67100.0100.0 0.0100.0 0.0100.0100.0 0.0100.0 0.0100.0 0.0100.0 0.0
color number hexagon6788.191.0 +3.086.6 -1.591.098.5 +7.595.5 +4.5100.0 +9.0100.0 +9.0
color overlap squares6782.1100.0 +17.9100.0 +17.991.0100.0 +9.098.5 +7.5100.0 +9.0100.0 +9.0
color size circle6650.0100.0 +50.089.4 +39.472.795.5 +22.769.7 -3.081.8 +9.1100.0 +27.3
Appendix
Table 21: PuzzleVQA: Gemini 3.1 Flash-Lite. Complete-family accuracy (%).
FamilynnDS ToolDS VCGPT ToolGPT VC
circle size number6774.6100.0 +25.429.9100.0 +70.1
color grid6795.5100.0 +4.552.2100.0 +47.8
color hexagon67100.098.5 -1.517.938.8 +20.9
color number hexagon6791.098.5 +7.537.397.0 +59.7
color overlap squares6782.188.1 +6.014.937.3 +22.4
color size circle6690.9100.0 +9.147.0100.0 +53.0
Appendix
Table 22: PuzzleVQA: DeepSeek and GPT with frozen FL routes. Complete-family accuracy (%).
FamilynnToolFULLCIIDSCALARNo scope
color6774.670.170.170.170.1
contrast3464.752.952.952.952.9
instrument5424.125.925.925.925.9
jigsaw ∗805.090.475.047.181.2
math8077.581.281.281.281.2
maze ∗8090.096.797.190.443.8
Appendix
Table 23: TIR-Bench: complete-family acquisition ablations. Accuracy (%).
FamilynnToolFullTEXTCODEINJECTNo text fallback
color6774.670.170.170.170.174.6
contrast3464.752.952.952.952.964.7
instrument5424.125.925.925.925.924.1
jigsaw805.095.033.872.592.595.0
math8077.581.281.281.281.277.5
maze8090.0100.097.5100.0100.0100.0
Appendix
Table 24: TIR-Bench: complete-family deployment ablations. Accuracy (%).
FamilynnToolFULLCIIDSCALARNo scope
billiards ∗1435.747.642.961.950.0
convex hull2090.090.090.090.090.0
cubes count205.05.05.05.05.0
cubes missing205.00.00.00.00.0
defuse a bomb1723.517.617.617.617.6
electric charge1478.685.785.785.785.7
Appendix
Table 25: MIRA: complete-family acquisition ablations. Accuracy (%).
FamilynnToolFullTEXTCODEINJECTNo text fallback
billiards1435.771.457.157.171.471.4
convex hull2090.090.090.090.090.090.0
cubes count205.05.05.05.05.05.0
cubes missing205.00.00.00.00.05.0
defuse a bomb1723.517.617.617.617.623.5
electric charge1478.685.785.785.785.778.6
Appendix
Table 26: MIRA: complete-family deployment ablations. Accuracy (%).
Figure 5: Additional executable procedures. (a) Minimum-area frame orientation yields four angle candidates; the answer adapter matches the provided options to select C ( 310∘310^{\circ} clockwise). (b) Measured hand directions, mirror/dial correction, and time arithmetic yield 9:15; the central dial depicts corrected angles schematically. (c) Centroid offset determines Pac-Man’s direction; the enlarged scanline counts four beans before a wall. (d) Component fill heights map the query bar to yellow (B). All four examples use programs without model calls.
RoundRevisionCorrect / 40
1Boundary matching with constrained Kruskal/Prim-style assembly.35
2Run Prim-style assembly from every starting piece and compare complete grids.37
4Jointly trim two boundary pixels and replace RMSE costs with MSE; rejected after regression.18
6Combine multiple cost variants and fixed-seed perturbations; rank complete grids by RMS boundary cost.39
Appendix
Table 27: Selected candidates in the deployed jigsaw solver’s acquisition history. Correct counts use all 40 training puzzles before scope filtering; missing outputs count as incorrect. These are candidates within one historical acquisition run, separate from the controlled acquisition ablations.
More in this topic

Visual Program Synthesis

See all 10 papers

Explore similar work

CardsList

  1. Self-Evolving Code-with-Image Reasoning

    Aug 11, 2026Tianze Yang, Liang Wu, Ruitong Sun +6Tool Use in VLMsMultimodal Reasoning

    Nokia · University of Georgia · The Hong Kong Polytechnic University

  2. PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides

    Aug 31, 2026Xiaoqiu Wang, Yizhe Chi, Wenyi Li +7Coding AgentsAI Agent Benchmarks

    Navers Lab, Einsia.AI · Tsinghua University

  3. Vero: An Open RL Recipe for General Visual Reasoning

    Apr 6, 2026Gabriel Sarch, Linrong Cai, Qunzhong Wang +3Reasoning for VLMsVisual Reasoning

    Princeton University