Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

Yuzhou Wang; Emile Anand; Ijay Narang

cs.CVOct 1, 2026

Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

Authors: Yuzhou Wang, Emile Anand, Ijay Narang

Organizations: Georgia Institute of Technology

Abstract

Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.

Figures & tables

Panel of Figure 1 — Figure 1: Simplified illustrations of the two tasks. The Yes/No example uses logically equivalent Boolean expressions with the same image and answer, while changing the wording. The identification example asks for the label of the object satisfying the description. We provide instances of our released examples in Figure 2 and Appendix C.1 .

Track	Images	Questions	Yes / No	Prompts
Yes/No, generated scenes	1,000	4,000	2,000 / 2,000	8,000
Yes/No, photographs	46	2,000	1,000 / 1,000	4,000
Identification, photographs	46	1,000	—	2,000
Total	1,046	7,000	—	14,000

Table 1: Hob-VL counts. Each question has one symbolic and one structured natural-language prompt. Yes/No counts include base expressions and equivalent rewrites, and each Yes/No track has equal Yes and No counts.

Panel of Figure 2 — Figure 2: Released Hob-VL examples. (a) One of eight clauses in a ten-statement sparse conjunctive normal form (CNF) question. (b) A complete object-identification condition. Instructions are omitted and notation adapted for readability. Reference answers are shown only for the reader. Appendix C.1 provides exact prompts

	Symbolic	Natural language
Base question	$F$	$\mathrm{NL}(F)$
Rewritten question	$\widetilde{F}$	$\mathrm{NL}(\widetilde{F})$

Table 2: Base/De Morgan-rewritten expressions in symbolic vs structured natural-language formats.

Family ( $n$ )	Operation and example	Diagnostic role
Parity of conjunctions ( $1{,}000$ )	Parity of five AND-pair outputs: $(1,1,0,0,0)$ gives false; $(1,1,1,0,0)$ gives true.	Cross-fact dependence; combining local results rather than counting true statements.
Parity of majorities ( $1{,}000$ )	XOR two five-input majorities. Groups with three and one true inputs yield different majorities, hence true.	Composing threshold decisions while preserving fact grouping.
Nested multiplexers ( $1{,}500$ )	Select branches via $M(1,u,v)=u$ and $M(0,u,v)=v$ ; nest selections and reuse inputs.	Nested evaluation; consistent reuse of facts across branches.
Dense CNF ( $1{,}000$ )	Conjoin 24 five-literal clauses. Any clause with five false literals makes the formula false.	Overlapping constraints coupled through shared variables.
Sparse chain CNF ( $500$ )	Conjoin eight three-literal clauses; adjacent clauses share two variables. Any false clause makes the formula false.	Local-interaction control against dense CNF; short false-answer certificates.
Polarity-trap composition ( $500$ )	Core $D$ forces a signed input false despite three unnegated and two negated mentions. Compose with parity $P$ : $D\oplus P$ .	Occurrence-majority cue favors the wrong value for satisfying the core.

Table 3: Construction families, evaluation examples, and diagnostic roles. Question counts include base and De Morgan variants. Examples precede optional output negation; signed inputs are variables or their negations. Appendix B.2 provides all templates, sampling procedures, and evaluations.

Thinking disabled or minimized
	Boolean evaluation		Object identification
Configuration	Symbolic	Natural Language	Symbolic	Natural Language
GPT-5.6 Luna	49.05	48.67	35.70	40.00
Claude Haiku 4.5	49.37	50.22	32.20	43.00
Grok 4.3	48.75	49.57	21.60	25.70
Gemini 3.5 Flash-Lite	48.60	48.52	27.10	41.10

Table 4: Accuracy (%) for each configuration and presentation format. Each Boolean column contains 6K questions, and each identification column contains 1K. Invalid responses count as incorrect. Bold indicates the largest observed score in the group, including ties. Thinking-enabled GLM uses a

4{,}096

-token output cap, and other configurations use

2{,}048

tokens. Appendix A documents the configurations, model identifiers, prompt policies, and scoring.

Certificate	$N$	Sym. off	Sym. on	NL off	NL on
3–4	1,772	45.71	82.62	52.88	62.19
5–6	2,856	49.02	57.35	48.98	47.41
7–10	1,372	51.38	51.60	50.00	44.90

Table 5: GLM-4.6V-FlashX accuracy (%) by instance certificate size.

N

counts questions per format, including both equivalent-expression variants. Invalid outputs count as incorrect. Off/on denote thinking-disabled/enabled configurations with output-token caps of 2,048/4,096, respectively. Each group contains a different mixture of construction families.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Provider	Requested model identifier	Mode	Cap	Init. dates
OpenAI	gpt-5.6-luna	none	2,048	14
Anthropic	claude-haiku-4-5-20251001	disabled	2,048	14
xAI	grok-4.3	none	2,048	14
Google	gemini-3.5-flash-lite	minimal	2,048	14–15
DeepSeek	deepseek-flash	disabled	2,048	15
Z.ai	glm-4.6v-flashx	disabled	2,048	15

Appendix

Table 6: Recorded inference configurations. Mode denotes the requested thinking setting or selection of an Instruct variant; it is not a measurement of internal reasoning. Cap denotes the requested output-token limit, including thinking tokens where the provider counts them toward that limit. Initialization dates are days in September 2026 (UTC), spanning creation of the four task–format run manifests; they are not completion or retry dates.

Configuration	Sym. BC	Conflict	$n_{v}$	NL BC	Conflict	$n_{v}$
Luna	35.27	27.55	2998	41.83	13.67	3000
Haiku 4.5	47.70	2.44	2945	44.67	10.92	2985
Grok 4.3	38.67	20.17	3000	31.37	36.40	3000
Gemini Flash-Lite	37.57	22.07	3000	35.83	25.37	3000
DeepSeek Flash	47.63	3.27	2936	47.93	3.44	2997
GLM FlashX (off)	39.13	18.90	3000	49.53	1.67	3000

Appendix

Table 7: Equivalent-expression outcomes. BC (both-correct accuracy, %) uses all 3,000 base–De Morgan pairs per format; conflict (%) uses only pairs with two valid outputs, counted by

n_{v}

. Invalid outputs never count as correct. A constant-label predictor achieves

50\%

BC and zero conflict; interpret these metrics alongside item accuracy and label frequencies.

Configuration	Bool. sym.	Bool. NL	Ident. sym.	Ident. NL	Token limit
GPT-5.6 Luna	2	0	0	0	0
Claude Haiku 4.5	57	15	31	44	0
Grok 4.3	0	0	13	12	0
Gemini 3.5 Flash-Lite	0	0	183	1	10
DeepSeek Flash	72	3	317	247	81
GLM-4.6V-FlashX (off)	0	0	0	0	0

Appendix

Table 8: Invalid-response counts for each task–format condition. Each Boolean column contains

6{,}000

questions, and each identification column contains

1{,}000

. The final column counts token-limit failures across all four conditions; these are included in the preceding invalid-response counts. Every question has a final response after retries. Completed responses containing extra prose and responses cut off at the token limit are both scored as invalid, but represent different failure modes.

Family	$N$	Sym. off	Sym. on	NL off	NL on
XOR of conjunctions	1000	50.60	56.80	51.80	47.30
XOR of majorities	1000	47.50	52.00	47.80	44.50
Nested multiplexers	1500	50.60	80.53	52.33	58.53
Dense CNF	1000	45.90	58.40	47.90	47.80
Sparse CNF control	500	40.80	86.20	52.00	67.60
Polarity trap + parity	500	53.40	47.40	53.60	49.60

Appendix

Table 9: GLM-4.6V-FlashX Boolean accuracy (%) by construction family.

N

is the number of questions per format. Off and on denote the thinking-disabled and thinking-enabled configurations, with output-token caps of

2{,}048

and

4{,}096

, respectively.

Configuration	Sym. synth.	Sym. photo	NL synth.	NL photo
GPT-5.6 Luna	49.15	48.85	48.23	49.55
Claude Haiku 4.5	49.03	50.05	50.10	50.45
Grok 4.3	48.48	49.30	49.40	49.90
Gemini 3.5 Flash-Lite	48.08	49.65	48.15	49.25
DeepSeek Flash	50.00	48.85	49.55	49.90
GLM-4.6V-FlashX (off)	48.13	49.50	50.10	50.90

Appendix

Table 10: Boolean accuracy (%) by image source, using

4{,}000

questions over generated scenes and

2{,}000

questions over photographs per format. Photographic scores are computed against the supplied reference annotations.

Configuration	Symbolic Yes (%)	NL Yes (%)
GPT-5.6 Luna	74.98	89.10
Claude Haiku 4.5	1.35	13.62
Grok 4.3	13.12	29.87
Gemini 3.5 Flash-Lite	81.23	80.08
DeepSeek Flash	1.78	2.08
GLM-4.6V-FlashX (off)	78.32	98.60

Appendix

Table 11: Valid Yes responses as a percentage of all

6{,}000

Boolean questions in each format. Reference labels are exactly balanced. The remaining percentage includes both No responses and invalid outputs; Table 8 reports invalid-output counts.

Configuration	Bool. BC	S-only	N-only	Ident. BC	S-only	N-only
GPT-5.6 Luna	35.52	812	789	27.20	85	128
Claude Haiku 4.5	42.73	398	449	27.20	50	158
Grok 4.3	33.12	938	987	13.50	81	122
Gemini 3.5 Flash-Lite	35.75	771	766	21.50	56	196
DeepSeek Flash	47.53	125	128	8.30	82	160
GLM-4.6V-FlashX (off)	38.12	628	735	24.10	72	109

Appendix

Table 12: Matched symbolic and natural-language outcomes for the same question IDs. Here, BC denotes the percentage of questions answered correctly in both formats, rather than both members of an equivalent-expression pair. S-only and N-only count questions answered correctly only in the symbolic or natural-language format, respectively. The Boolean comparison contains

6{,}000

matched questions, and the identification comparison contains

1{,}000

. Invalid outputs count as incorrect.

Figure 3: Balanced Boolean accuracy (%) by logical-property group. Columns show symbolic and structured natural-language formats; rows show certificate size, best two-fact predictability, and total influence. Groups differ in their construction-family composition, so the figure reports descriptive associations rather than isolated effects of each property.

Family	$N$	$c_{F}(a)$	$A_{2}(F)$	$I(F)$
Parity of conjunctions	1,000	5–10	0.5312	5.0000
Parity of majorities	1,000	6	0.5703	3.7500
Nested multiplexers	1,500	4–5	0.6250	2.8594
Dense CNF	1,000	4–8	0.5889–0.7158	2.5508–3.3047
Sparse chain CNF	500	3–6	0.6816–0.8350	1.7656–2.4766
Polarity-trap composition	500	7–10	0.5000	6.5000

Appendix

Table 13: Observed logical-property ranges within construction families.

N

counts questions per format. A single value indicates no within-family variation; influence and predictability are rounded to four decimal places. Ranges describe the released questions, not all possible formulas in a family.

Parity of conjunctions
Certificate	$N$	Symbolic	NL
5	232	74.14	51.72
6	436	47.71	48.17
7	238	62.18	39.92
8	74	43.24	56.76
9	14	50.00	21.43

Appendix

Table 14: Thinking-enabled GLM accuracy (%) by certificate size within each family that exhibits certificate variation.

N

counts questions per format; each equivalent-expression pair contributes two questions. Invalid outputs remain incorrect. Small groups, including those with fewer than 50 questions, should be interpreted cautiously.

Figure 4: A symbolic Yes/No example from the sparse-CNF control family. The prompt is reproduced verbatim except for one instruction sentence about spatial conventions, marked [...] . The image is synthetic_0362 , and the expression contains 54 AST nodes, the smallest expression size in the release. The answer is No : D is yellow, G is a circle, and J is a triangle, so $p_{6}=p_{7}=p_{8}=0$ and the clause $p_{6}\lor p_{7}\lor p_{8}$ is false. These three values certify the answer. A false width-three CNF always has a certificate of size at most three (Appendix B.5 ), so this control does not meet the benchmark’s broad-dependence criterion. Its De Morgan partner rewrites two subtrees and contains 60 AST nodes; one rewrite replaces $(p_{6}\lor(p_{7}\lor p_{8}))\land R$ with $\neg\bigl(\neg(p_{6}\lor(p_{7}\lor p_{8}))\lor\neg R\bigr)$ , where $R$ denotes the other child of that conjunction.

Figure 6: An identification question for photo_001 . The upper block shows the complete NL prompt; the lower block shows the symbolic statements and condition, with the shared header and candidate list abbreviated. The correct label is C: $p_{2}$ is false because C is not above itself, and $p_{3}$ is true because C lies left of D. Thus, both disjunctions in $(p_{1}\lor\neg p_{2})\land(p_{3}\lor p_{4})$ hold. D’s vertical relation to C falls within the 6% uncertainty margin and is recorded as unknown, but D is excluded because it is neither left of itself nor above E, making both $p_{3}$ and $p_{4}$ false. The values of $p_{2}$ and $p_{3}$ distinguish C from every other candidate.

Explore similar work

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Multimodal Reasoning Benchmarks Multimodal Large Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints
Jul 23, 2026Liqiang Jing, Xiong Zhou, Siddharth Varia +3Visual Reasoning Recent Vision-Language Models
University of Texas at Dallas · Amazon Web Services
A Study of Commonsense Reasoning over Visual Object Properties
Aug 14, 2025Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang +2Visual Question Answering Benchmarks Visual Reasoning
Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands · Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA

May 25, 2026, cs.CVJ/K move · Enter open · S save

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Longteng Guo, Yifan Wang, Pengkang Huo, Tailai Chen+3

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Multimodal Reasoning Benchmarks Multimodal Large Language Models Recent Vision-Language Models Visual Evidence

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Abstract

Explore similar work

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints

A Study of Commonsense Reasoning over Visual Object Properties