RuleForge: Inferring and Instantiating Latent Visual Rules as Executable Programs

Weifeng Xu; Yaxiong Wang; Zhedong Zheng

cs.CVOct 28, 2026

RuleForge: Inferring and Instantiating Latent Visual Rules as Executable Programs

Authors: Weifeng Xu, Yaxiong Wang, Zhedong Zheng

Organizations: Institute of Artificial Intelligence and Brain Sciences, University of Macau Macau, China · Faculty of Information Science and Computing, University of Macau Macau, China · Hefei University of Technology Hefei, China

Abstract

Visual rule reasoning asks whether multimodal models can recover an abstract concept from a few examples. Existing studies either infer rules from visual examples but evaluate only classification or verbalization, or supervise executable generation with rules provided as input. Consequently, the relationship between visual rule understanding and reliable rule instantiation remains insufficiently explored. We present Bongard-Forge100k, a controlled benchmark, together with RuleForge, a backbone-agnostic framework that jointly learns visual rule understanding and rule-conditioned executable generation. The benchmark contains 108,797 problems: 86,000 for training, 19,297 for testing familiar rules, and 3,500 for testing held-out rule compositions. The framework progresses through rule-understanding training, joint understanding and scene-program supervision, and refinement via execution-grounded reinforcement learning with Understanding Replay. Executing generated programs under multiple random seeds provides verifiable feedback on executability, rule satisfaction, diversity, and robustness. With UniLIP as the primary backbone, RuleForge improves multiple-choice accuracy over the base model by 20.93 percentage points on familiar rules and 28.42 percentage points on held-out rule compositions, while increasing scene-level rule satisfaction on familiar rules from 2.91% to 70.99%. Additional experiments on InternVL3.5, Qwen3-VL, and Show-o2 demonstrate the effectiveness of the proposed training data and framework across diverse multimodal architectures.

Figures & tables

Figure 1. The proposed RuleForge framework for jointly learning visual rule understanding and executable generation. Given a Bongard-Forge100k problem containing six positive and six negative panels, the model selects the separating rule and provides a natural-language explanation. Through a separate generation interface, the corresponding target rule is provided as input, and the model generates an executable scene program that produces new rule-consistent instances under five fixed seeds. “What I cannot create, I do not understand.” — Richard Feynman

Benchmark	Understanding task	Executable generation	Visual domain	# Problems	Concept inventory
Original Bongard Problems ( Bongard, 1970 )	Verbal rule induction	No	Diagrams	100	Problem-specific visual concepts
Bongard-LOGO ( Nie et al., 2020 )	Binary classification	No	Drawings	12,000	627 shape categories; 25 abstract attributes
Bongard-HOI ( Jiang et al., 2022 )	Binary classification	No	HOI images	${\sim}53{,}000$	242 HOI concepts
Bongard-OpenWorld ( Wu et al., 2024 )	Binary classification	No	Web images	1,010	1,010 free-form concepts
Bongard-Forge100k	Multiple-choice rule recognition; free-form rule explanation	Yes (scene programs)	Rendered scenes	108,797	62 registered base rules; 24 held-out compositions

Table 1. Comparison with representative Bongard benchmarks. Inventory entries follow the native definitions of the respective benchmarks and are therefore not directly equivalent measures of concept complexity. “Executable generation” indicates whether generating executable scene-construction programs constitutes an evaluated model task. Bongard-LOGO uses rendering programs to construct its benchmark, but its standard evaluation task is binary classification.

Figure 2. Overview of Bongard-Forge100k. (a) Representative problems covering counting, attributes and size, spatial relations, and containment. (b) The controlled construction pipeline, a structured problem record, and dataset composition. The donut chart’s inner ring represents the training, familiar-rule test, and held-out-composition test splits. The outer ring partitions each split into eight display groups aggregated from finer rule-family metadata for legibility; sector sizes are proportional to problem counts. (c) Held-out compositions combine rules observed individually during training, requiring both constituent conditions to hold in positive examples.

Figure 3. Overview of RuleForge. A unified VLM supports visual rule understanding from examples and rule-conditioned program generation from a separately given rule. Stage I trains rule understanding; Stage II jointly performs Understanding SFT and Program SFT; and Stage III applies multi-seed execution feedback through GRPO with understanding replay. Six candidate programs are executed under three shared training seeds, while final evaluation uses five fixed seeds. The shared connector indicates that the same VLM is progressively updated across all stages.

Commercial zero-shot references
Model	Stage	Familiar Rules			Held-Out Rule Compositions
Model	Stage	MCQ	Exp.	Joint	MCQ	Exp.	Joint
Gemini-2.5 Flash-Lite ( Google, 2025 )	–	30.95	10.37	3.61	37.63	0.74	0.23
GPT-6-Sol	–	68.81	21.32	16.18	84.09	5.31	4.63
Backbones trained with RuleForge
InternVL3.5-2B ( Wang et al., 2025a )	Base	33.31	3.81	1.61	35.00	0.97	0.76

Table 2. Visual rule understanding results (%). MCQ, Exp., and Joint denote multiple-choice, explanation, and joint accuracy. Bold marks the best RuleForge-trained result.

API-documented commercial zero-shot references
Model	Stage	Familiar Rules		Held-Out Rule Compositions
Model	Stage	Exec.	RuleSat	Exec.	RuleSat
Gemini-2.5 Flash-Lite ( Google, 2025 )	–	45.47	30.37	33.33	20.76
GPT-6-Sol	–	94.71	85.41	91.67	83.40
Backbones trained with RuleForge
Qwen3-VL-4B-Instruct ( Bai et al., 2025 )	Base	4.04	2.69	0.00	0.00

Table 3. Rule-conditioned executable generation results (%). Exec. denotes program executability, and RuleSat denotes scene-level rule satisfaction over five seeds. Bold marks the best RuleForge-trained result.

Figure 4. Matched-seed comparison between Stages II and III. Stage II produces executable but rule-violating positive scenes, whereas Stage III satisfies the target rules. Each displayed case uses one seed; RuleSat is computed over five fixed seeds.

		Familiar		Held-Out
Stage	Source	Exec. $\uparrow$	RuleSat $\uparrow$	Exec. $\uparrow$	RuleSat $\uparrow$
II	Ref.	77.50	71.98	62.40	43.65
III	Ref.	88.70	79.37	79.20	50.09
II	Pred.	59.20	34.43	25.60	12.96
III	Pred.	69.50	38.69	42.60	21.20

Table 4. Inference-time rule chaining (%). Ref. uses the reference rule; Pred. uses the model-predicted rule. Bold indicates the better stage under the same rule source.

(a) Stage II task mixture
Metric	U-only	G-only	$1{:}1$	$2{:}1$
Familiar MCQ	70.80	61.50	72.90	84.95
Held-out MCQ	60.77	49.74	59.73	66.60
Familiar Exp.	24.80	5.40	25.90	25.80
Held-out Exp.	8.22	0.10	17.07	16.83
Familiar RuleSat	2.91	68.76	61.89	60.42

Table 5. UniLIP-3B ablations (%). (a) Stage II task mixtures; (b) Stage III reward components.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Initialization and trainable components	Training budget	Learning rate	Batching	Sampling configuration
I	Base model; LM LoRA and multimodal projector	2 epochs; 6,500 updates	$1{\times}10^{-5}$ LoRA; $2{\times}10^{-6}$ projector	Logical batch 8; accumulation 6; effective batch 48	MCQ/Exp./Verify/Fact $=0.60/0.25/0.10/0.05$
II	Stage I checkpoint; LM LoRA only	1 epoch; 7,313 updates	$1{\times}10^{-5}$	Understanding and program batches 8; accumulation 4	Understanding:Program $=2{:}1$
III	Stage II checkpoint; LM LoRA only	1,000 policy updates	$1{\times}10^{-6}$	Six candidates per group; replay batch 8	Three shared execution seeds; understanding replay at every update

Appendix

Table 6. Training configuration used for the three RuleForge stages. Logical batch sizes are measured before gradient accumulation. Stage III samples one group of six candidate programs per policy update.

	Familiar			Held-Out
Configuration	MCQ	Exp.	Joint	MCQ	Exp.	Joint
Full III	85.01	24.63	21.75	66.00	16.46	7.00
III w/o Replay	84.91	25.30	22.35	67.29	13.09	5.60

Appendix

Table 7. Understanding Replay ablation on UniLIP-3B (%). The ablated Stage III model removes

\mathcal{L}_{\mathrm{rep}}

. Bold indicates the better result in each column.

Figure 5. Stage-wise performance. (a) Familiar-rule MCQ accuracy and (b) held-out RuleSat for the base model and Stages I–III. Stage III annotations show percentage-point gains over the corresponding base models.

Visual Reasoning

Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.

See all 165 papers

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
May 29, 2026Xixiang He, Baiqi Wu, Xingming Li +4VLM Evaluation Visual Reasoning
National University of Defense Technology · Zhejiang University · Intelligent Game and Decision Lab
TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models
Feb 27, 2026Daniel Nobrega MedeirosVisual Reasoning Spatial Reasoning Benchmarks
Independent Researcher
Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning
Oct 1, 2026Yuzhou Wang, Emile Anand, Ijay NarangVLM Evaluation Visual Question Answering
Georgia Institute of Technology

May 29, 2026J/K move · Enter open · S save

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

Xixiang He, Baiqi Wu, Xingming Li, Ao Cheng+3

National University of Defense Technology · Zhejiang University · Intelligent Game and Decision Lab

VLM Evaluation Visual Reasoning Multimodal Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.

Abstract

Visual Reasoning

Explore similar work

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning