RuleForge: Inferring and Instantiating Latent Visual Rules as Executable Programs
Authors: Weifeng Xu, Yaxiong Wang, Zhedong Zheng
Organizations: Institute of Artificial Intelligence and Brain Sciences, University of Macau Macau, China · Faculty of Information Science and Computing, University of Macau Macau, China · Hefei University of Technology Hefei, China
Visual rule reasoning asks whether multimodal models can recover an abstract concept from a few examples. Existing studies either infer rules from visual examples but evaluate only classification or verbalization, or supervise executable generation with rules provided as input. Consequently, the relationship between visual rule understanding and reliable rule instantiation remains insufficiently explored. We present Bongard-Forge100k, a controlled benchmark, together with RuleForge, a backbone-agnostic framework that jointly learns visual rule understanding and rule-conditioned executable generation. The benchmark contains 108,797 problems: 86,000 for training, 19,297 for testing familiar rules, and 3,500 for testing held-out rule compositions. The framework progresses through rule-understanding training, joint understanding and scene-program supervision, and refinement via execution-grounded reinforcement learning with Understanding Replay. Executing generated programs under multiple random seeds provides verifiable feedback on executability, rule satisfaction, diversity, and robustness. With UniLIP as the primary backbone, RuleForge improves multiple-choice accuracy over the base model by 20.93 percentage points on familiar rules and 28.42 percentage points on held-out rule compositions, while increasing scene-level rule satisfaction on familiar rules from 2.91% to 70.99%. Additional experiments on InternVL3.5, Qwen3-VL, and Show-o2 demonstrate the effectiveness of the proposed training data and framework across diverse multimodal architectures.
Figures & tables
Figure 1. The proposed RuleForge framework for jointly learning visual rule understanding and executable generation. Given a Bongard-Forge100k problem containing six positive and six negative panels, the model selects the separating rule and provides a natural-language explanation. Through a separate generation interface, the corresponding target rule is provided as input, and the model generates an executable scene program that produces new rule-consistent instances under five fixed seeds. “What I cannot create, I do not understand.” — Richard Feynman
62 registered base rules; 24 held-out compositions
Table 1. Comparison with representative Bongard benchmarks. Inventory entries follow the native definitions of the respective benchmarks and are therefore not directly equivalent measures of concept complexity. “Executable generation” indicates whether generating executable scene-construction programs constitutes an evaluated model task. Bongard-LOGO uses rendering programs to construct its benchmark, but its standard evaluation task is binary classification.
Figure 2. Overview of Bongard-Forge100k. (a) Representative problems covering counting, attributes and size, spatial relations, and containment. (b) The controlled construction pipeline, a structured problem record, and dataset composition. The donut chart’s inner ring represents the training, familiar-rule test, and held-out-composition test splits. The outer ring partitions each split into eight display groups aggregated from finer rule-family metadata for legibility; sector sizes are proportional to problem counts. (c) Held-out compositions combine rules observed individually during training, requiring both constituent conditions to hold in positive examples.
Figure 3. Overview of RuleForge. A unified VLM supports visual rule understanding from examples and rule-conditioned program generation from a separately given rule. Stage I trains rule understanding; Stage II jointly performs Understanding SFT and Program SFT; and Stage III applies multi-seed execution feedback through GRPO with understanding replay. Six candidate programs are executed under three shared training seeds, while final evaluation uses five fixed seeds. The shared connector indicates that the same VLM is progressively updated across all stages.
Model
Stage
Familiar Rules
Held-Out Rule Compositions
MCQ
Exp.
Joint
MCQ
Exp.
Joint
Commercial zero-shot references
Gemini-2.5 Flash-Lite ( Google, 2025 )
–
30.95
10.37
3.61
37.63
0.74
0.23
GPT-6-Sol
–
68.81
21.32
16.18
84.09
5.31
4.63
Backbones trained with RuleForge
InternVL3.5-2B ( Wang et al., 2025a )
Base
33.31
3.81
1.61
35.00
0.97
0.76
Table 2. Visual rule understanding results (%). MCQ, Exp., and Joint denote multiple-choice, explanation, and joint accuracy. Bold marks the best RuleForge-trained result.
Model
Stage
Familiar Rules
Held-Out Rule Compositions
Exec.
RuleSat
Exec.
RuleSat
API-documented commercial zero-shot references
Gemini-2.5 Flash-Lite ( Google, 2025 )
–
45.47
30.37
33.33
20.76
GPT-6-Sol
–
94.71
85.41
91.67
83.40
Backbones trained with RuleForge
Qwen3-VL-4B-Instruct ( Bai et al., 2025 )
Base
4.04
2.69
0.00
0.00
Table 3. Rule-conditioned executable generation results (%). Exec. denotes program executability, and RuleSat denotes scene-level rule satisfaction over five seeds. Bold marks the best RuleForge-trained result.
Figure 4. Matched-seed comparison between Stages II and III. Stage II produces executable but rule-violating positive scenes, whereas Stage III satisfies the target rules. Each displayed case uses one seed; RuleSat is computed over five fixed seeds.
Familiar
Held-Out
Stage
Source
Exec. ↑
RuleSat ↑
Exec. ↑
RuleSat ↑
II
Ref.
77.50
71.98
62.40
43.65
III
Ref.
88.70
79.37
79.20
50.09
II
Pred.
59.20
34.43
25.60
12.96
III
Pred.
69.50
38.69
42.60
21.20
Table 4. Inference-time rule chaining (%). Ref. uses the reference rule; Pred. uses the model-predicted rule. Bold indicates the better stage under the same rule source.
(a) Stage II task mixture
Metric
U-only
G-only
1:1
2:1
Familiar MCQ
70.80
61.50
72.90
84.95
Held-out MCQ
60.77
49.74
59.73
66.60
Familiar Exp.
24.80
5.40
25.90
25.80
Held-out Exp.
8.22
0.10
17.07
16.83
Familiar RuleSat
2.91
68.76
61.89
60.42
Table 5. UniLIP-3B ablations (%). (a) Stage II task mixtures; (b) Stage III reward components.
Understanding and program batches 8; accumulation 4
Understanding:Program =2:1
III
Stage II checkpoint; LM LoRA only
1,000 policy updates
1×10−6
Six candidates per group; replay batch 8
Three shared execution seeds; understanding replay at every update
Appendix
Table 6. Training configuration used for the three RuleForge stages. Logical batch sizes are measured before gradient accumulation. Stage III samples one group of six candidate programs per policy update.
Familiar
Held-Out
Configuration
MCQ
Exp.
Joint
MCQ
Exp.
Joint
Full III
85.01
24.63
21.75
66.00
16.46
7.00
III w/o Replay
84.91
25.30
22.35
67.29
13.09
5.60
Appendix
Table 7. Understanding Replay ablation on UniLIP-3B (%). The ablated Stage III model removes Lrep . Bold indicates the better result in each column.
Figure 5. Stage-wise performance. (a) Familiar-rule MCQ accuracy and (b) held-out RuleSat for the base model and Stages I–III. Stage III annotations show percentage-point gains over the corresponding base models.
Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.
Xixiang He, Baiqi Wu, Xingming Li +4
National University of Defense Technology · Zhejiang University · Intelligent Game and Decision Lab
Existing visual reasoning benchmarks predominantly rely on natural language prompts, evaluate narrow reasoning modalities, or depend on subjective scoring procedures such as LLM-as-judge. We introduce the TACIT Benchmark, a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains: spatial navigation, abstract pattern completion, causal simulation, logical constraint satisfaction, graph theory, and topology. The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors. Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences rather than exploit superficial cues. Version 0.1.0 distributes 6,000 puzzles (108,000 PNG images across three resolutions) with fully deterministic seeded generation and reproducible verification. The dataset, generation code, and evaluation harness are released under the Apache 2.0 license on HuggingFace (DOI: 10.57967/hf/7904).
Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.