TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

Daniel Nobrega Medeiros

cs.CVFeb 27, 2026

TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

Authors: Daniel Nobrega Medeiros

Organizations: Independent Researcher

Abstract

Existing visual reasoning benchmarks predominantly rely on natural language prompts, evaluate narrow reasoning modalities, or depend on subjective scoring procedures such as LLM-as-judge. We introduce the TACIT Benchmark, a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains: spatial navigation, abstract pattern completion, causal simulation, logical constraint satisfaction, graph theory, and topology. The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors. Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences rather than exploit superficial cues. Version 0.1.0 distributes 6,000 puzzles (108,000 PNG images across three resolutions) with fully deterministic seeded generation and reproducible verification. The dataset, generation code, and evaluation harness are released under the Apache 2.0 license on HuggingFace (DOI: 10.57967/hf/7904).

Figures & tables

Benchmark	Domains	Tasks	Lang-min	Gen	Disc	Det-V
RAVEN	1	1	✓	×	✓	✓
ARC	1	vary	✓	×	✓	✓
Bongard-LOGO	1	1	✓	×	✓	×
MMMU	many	900+	×	×	✓	×
MathVista	many	6k+	×	×	✓	×
CLEVR	1	1	×	×	✓	✓

Table 1: Comparison of TACIT Benchmark with related visual reasoning benchmarks. Lang-min : language-minimal task specification. Gen : generative evaluation track. Disc : discriminative evaluation track. Det-V : deterministic verification (no human/LLM judge).

Panel of Figure 1 — Figure 1: Dual-track evaluation architecture. Track 1 (generative) requires the model to produce a solution image verified by a deterministic CV pipeline. Track 2 (discriminative) presents five candidates for multiple-choice selection.

#	Task	Domain	Bin.	Verification	Difficulty Axes	Easy	Hard
1	Multi-Layer Maze	Spatial	No	BFS path tracing	grid, layers, portals	$8{\times}8$ , 1L, 0P	$32{\times}32$ , 3L, 5P
2	Raven’s Matrices	Pattern	No	SSIM ${\geq}\,0.997$	rules, complexity	1 rule, additive	3 rules, compositional
3	CA Forward	Pattern	No	Pixel sampling	grid, states, steps	$8^{2}$ , 2-state, 1-step	$32^{2}$ , 8-state, 5-step
4	CA Inverse	Pattern	No	Pixel sampling	grid, states, steps	$8^{2}$ , 4-state, 1-step	$32^{2}$ , 16-state, 3-step
5	Logic Grid	Logical	No	Pixel sampling	grid, constraints, types	$4{\times}4$ , 6C, 2T	$6{\times}6$ , 16C, 4T
6	Graph $k$ -Coloring	Graph	No	Node sampling	nodes, density, $k$	6N, 0.3, $k{=}4$	20N, 0.5, $k{=}3$

Table 2: Overview of the 10 TACIT Benchmark tasks. Bin. : binary classification task. Verification : CV-based verification strategy. Difficulty parameters shown for easy/hard levels (200 puzzles each; medium omitted for space).

Panel of Figure 2 — Figure 2: Representative puzzle instances from all 10 TACIT Benchmark tasks, grouped by domain. Row 1: Spatial and Pattern tasks. Row 2: Logical, Graph, and Topology tasks. Row 3: Geometric Projection tasks. Each puzzle is specified entirely through visual encoding with minimal text annotations.

Panel of Figure 3 — Figure 3: Generation and verification pipeline. SVGs are generated deterministically, rasterized to multi-resolution PNGs, and verified against candidates using task-specific CV pipelines.

Panel of Figure 4 — Figure 4: Puzzle–solution pairs for four representative tasks. (a–b) Multi-layer maze with blue path from start (green) to end (red). (c–d) Graph $k$ -coloring: gray uncolored nodes and their proper coloring. (e–f) Orthographic projection: 3D voxel solid and its 2D silhouette. (g–h) Raven’s matrix with missing tile completed.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

#	Strategy	Threshold	Failure Diagnostic
1	BFS path tracing	4 structural checks	Failed check ID
2	SSIM	$\geq 0.997$	SSIM score
3	Pixel sampling	All cells match	Diff. cells / total
4	Pixel sampling	All entries match	Diff. entries / total
5	Color voting	Latin sq. + exact	Constraint ID
6	Node sampling	Proper $k$ -coloring	Violating node(s)

Appendix

Table 3: Verification specifications for all 10 tasks.

#	Violation Types
1	wall_breach , portal_skip , disconnected , wrong_exit
2	wrong_shape , wrong_color , wrong_rotation , wrong_count
3	wrong_cell , wrong_step_count , wrong_rule
4	off_by_one_rule , transposed_rule , partial_rule
5	constraint_violation , symbol_swap , non_unique
6	adjacent_conflict , missing_color , wrong_k

Appendix

Table 4: Distractor violation types per task.

#	Task	Easy	Medium	Hard
1	Maze	$8^{2}$ , 1L, 0P	$16^{2}$ , 2L, 2P	$32^{2}$ , 3L, 5P
2	Raven	1r, add.	2r, add.	3r, comp.
3	CA Fwd	$8^{2}$ , 2s, 1t	$16^{2}$ , 4s, 3t	$32^{2}$ , 8s, 5t
4	CA Inv	$8^{2}$ , 4s, 1t	$16^{2}$ , 8s, 2t	$32^{2}$ , 16s, 3t
5	Logic	$4^{2}$ , 6c, 2ty	$5^{2}$ , 10c, 3ty	$6^{2}$ , 16c, 4ty
6	Color	6n, .3, $k{=}4$	12n, .4, $k{=}4$	20n, .5, $k{=}3$

Appendix

Table 5: Release parameters (200 puzzles per task per level).

Visual Reasoning

Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.

See all 161 papers

GenPuzzle: Benchmarking Visual Reasoning in Image Generation Models
Sep 5, 2026Changpeng Zhao, Yiren Song, Jinpeng WangImage Generation Image Generation Evaluation
Central South University · National University of Singapore
Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning
Oct 1, 2026Yuzhou Wang, Emile Anand, Ijay NarangVLM Evaluation Visual Question Answering
Georgia Institute of Technology
LogicVal:A Logic-Formalized Benchmark for Constraint Satisfaction in Text-to-Image Generation
Oct 16, 2026Savitha Sam Abraham, Marjan Alirezaie, Hossein RahnamaImage Generation Evaluation T2I Generation Evaluation
City St George’s, University of London · Toronto Metropolitan University · Flybits Labs, Creative AI Hub, Toronto +1

Sep 5, 2026J/K move · Enter open · S save

GenPuzzle: Benchmarking Visual Reasoning in Image Generation Models

Changpeng Zhao, Yiren Song, Jinpeng Wang

Central South University · National University of Singapore

Image Generation Image Generation Evaluation Spatial Reasoning Benchmarks

Open paper AskPDF arXiv

Recent image generation systems increasingly combine multimodal understanding, reasoning, and synthesis, suggesting that they may do more than render plausible scenes. Yet existing evaluations emphasize aesthetics, prompt alignment, compositionality, or text-based answers, leaving unclear whether these systems can solve visual problems and faithfully express solutions in pixels. We introduce GenPuzzle, a benchmark for reasoning-centric image generation. GenPuzzle contains 2,005 problems across 12 tracks, spanning pattern completion, spatial construction, mazes, Sudoku, nonograms, tangrams, board games, matchstick puzzles, orthographic projection, and mathematical visual proof. Each task provides a visual puzzle and requires an image output that preserves the input state while executing a logically valid solution. GenPuzzle uses task-specific evaluation protocols: discrete grid outputs are transcribed and verified programmatically, while visually complex outputs are assessed with tiered, multidimensional, or binary multimodal large language model (MLLM) rubrics. We further select the automatic judge by measuring agreement with human reference scores. Across three frontier generators, the strongest model reaches only 40.57 Macro Overall, revealing frequent failures in logic, geometry, state preservation, and instruction execution. GenPuzzle provides a testbed for measuring progress from image rendering toward visual problem solving.

Abstract

Visual Reasoning

Explore similar work

GenPuzzle: Benchmarking Visual Reasoning in Image Generation Models

Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

LogicVal:A Logic-Formalized Benchmark for Constraint Satisfaction in Text-to-Image Generation