Weakly supervised concept Bottleneck Learning for Robust Two stage Object centric visual reasoning

Sparsh Tiwari; Gesina Schwalbe; Bettina Finzel

cs.AIAug 23, 2026

Weakly supervised concept Bottleneck Learning for Robust Two stage Object centric visual reasoning

Authors: Sparsh Tiwari, Gesina Schwalbe, Bettina Finzel

Organizations: University of Lübeck, Germany · Ulm University, Germany · University of Bamberg, Germany

Abstract

Two-stage neuro-symbolic architectures provide an elegant paradigm for visual problem solving by cleanly separating connectionist perception of predefined symbols from possibly later defined relational reasoning thereon. However, anchoring high-level predicates into visual frames typically necessitates annotations that are expensive to acquire. In this work, we introduce the Dynamic Orthogonal Concept Bottleneck (D-OCB), an object-centric slot- VAE framework designed to extract human-aligned symbolic predicates under extremely weak supervision. D-OCB eliminates the arduous manual tuning of loss-balancing coef- ficients by dynamically learning optimal hyperparameter allocations during training. To infuse prior knowledge on independence of concept categories, in addition to standard re- construction self-supervision we penalize correlation across concept subspaces. Crucially, to combat the instability of very low supervision regimes, D-OCB incorporates a dynamic di- mensionality allocation mechanism; this adaptive formulation allows well-represented con- cepts to yield latent dimensions to underperforming concepts that are lagging behind, effectively preventing representation collapse and significantly improving overall concept accuracy. Through an extensive empirical evaluation, we demonstrate that our framework achieves high concept alignment and downstream visual reasoning accuracy using minimal label budgets, matching or outperforming end-to-end paradigms.

Figures & tables

Figure 1: Overview of our D-OCB architecture and training scheme

Panel of Figure 2 — Figure 2: Correleation MATRIX for Clevr HANS7 Showing our DDA assign dimension from one concept to another for epoch 5 (left) to epoch 100 (right)

		CNN Backbone					DINO Backbone
Sup.	Method	Size	Shape	Color	Material	Mean	Size	Shape	Color	Material	Mean
1%	Vanilla+SA	$51.2\pm 0.0$	$49.3\pm 0.0$	$10.7\pm 0.0$	$62.2\pm 0.0$	$46.4\pm 0.0$	$72.3\pm 0.0$	$63.4\pm 0.0$	$15.8\pm 0.0$	$70.6\pm 0.0$	$57.3\pm 0.0$
	CEM	$51.2\pm 1.3$	$33.9\pm 0.5$	$17.2\pm 6.3$	$50.9\pm 0.5$	$38.3\pm 2.1$	$73.4\pm 2.2$	$41.3\pm 3.9$	$12.3\pm 0.1$	$70.8\pm 6.6$	$49.5\pm 2.6$
	ProbCBM	$48.2\pm 3.9$	$34.5\pm 2.1$	$20.9\pm 8.2$	$50.7\pm 0.7$	$38.6\pm 3.3$	$72.3\pm 4.4$	$38.4\pm 2.2$	$12.5\pm 0.3$	$67.0\pm 7.4$	$47.5\pm 3.2$
	SlotFormer	$25.2\pm 3.1$	$25.0\pm 2.0$	$17.5\pm 5.1$	$50.1\pm 1.1$	$29.4\pm 1.7$	$48.0\pm 2.8$	$38.3\pm 2.8$	$17.7\pm 3.9$	$55.0\pm 3.1$	$39.8\pm 2.1$
	SlotVAE-NoVAE	$52.8\pm 0.0$	$51.5\pm 0.0$	$8.9\pm 0.0$	$60.5\pm 0.0$	$48.6\pm 0.0$	$76.2\pm 0.0$	$62.6\pm 0.0$	$12.1\pm 0.0$	$75.6\pm 0.0$	$56.9\pm 0.0$

Table 1: Concept classification acc. on the CLEVR dataset across varying levels of weak supervision. Results are reported as mean

\pm

standard deviation over 5 folds.

Backbone	Sup. (%)	Dx	Localization	Sex	Age	Mean
CNN	1	$55.3\pm 0.0$	$84.2\pm 0.0$	$22.1\pm 0.0$	$50.0\pm 0.0$	$52.9\pm 0.0$
	5	$55.0\pm 0.0$	$84.4\pm 0.0$	$60.6\pm 0.0$	$61.7\pm 0.0$	$65.4\pm 0.0$
	15	$59.3\pm 0.0$	$86.0\pm 0.0$	$64.3\pm 0.0$	$64.6\pm 0.0$	$68.6\pm 0.0$
DINO	1	$61.8\pm 0.0$	$82.5\pm 0.0$	$67.4\pm 0.0$	$60.2\pm 0.0$	$68.0\pm 0.0$
	5	$64.4\pm 0.0$	$78.5\pm 0.0$	$69.4\pm 0.0$	$69.1\pm 0.0$	$70.4\pm 0.0$
	15	$64.2\pm 0.0$	$85.5\pm 0.0$	$72.8\pm 0.0$	$70.9\pm 0.0$	$73.4\pm 0.0$

Table 2: Concept classification acc. of D-OCB on the HAM10000 dataset across varying levels of weak supervision. Results are reported as mean

\pm

standard deviation over five folds.

	ID	Rule Type	Description
CLEVR	R1	4-way Conjunction	The scene contains an object that is simultaneously large, red, metal, and a cube.
	R2	3-way Conjunction	The scene contains a small rubber sphere.
	R3	Spatial Relation	A blue cylinder is spatially to the left of a red object (evaluated via 3D x-coordinates).
	R4	Cardinality	The scene contains $\geq 2$ metal objects; requires the model to count distinct slots satisfying this constraint.
	R5	Proximity Relation	A large green object is within a continuous Euclidean distance threshold ( $\leq 2.5$ ) of a yellow object.
HAM	R1	Simple	The lesion is malignant.

Table 3: Downstream evaluation rules for the CLEVR and HAM10000 datasets.

Backb.	Percept. Model	Decision Tree (DT)					Bayes Net (BN)					NS-CL					DILP
Backb.	Percept. Model	1%	15%	25%	50%	75%	1%	15%	25%	50%	75%	1%	15%	25%	50%	75%	1%	15%	25%	50%	75%
CNN	D-OCB (Ours)	63.1	70.8	72.0	71.9	69.8	85.0	71.6	72.1	71.8	71.8	62.2	71.0	71.5	71.3	71.2	67.1	97.0	97.1	97.4	97.1
	SlotVAE-KL	56.3	33.3	66.6	67.7	71.6	52.9	32.7	83.3	84.5	85.5	45.9	31.4	50.1	74.0	74.2	45.0	47.1	69.5	80.8	82.1
	SlotVAE-NoVAE	55.6	48.3	59.8	68.1	69.4	69.3	53.6	70.0	60.2	61.5	61.4	56.7	66.6	72.2	72.9	70.9	74.6	89.0	95.6	96.0
	Vanilla+SA	51.3	51.8	54.1	55.5	58.9	51.4	53.6	53.9	53.8	52.4	62.2	60.3	63.8	62.7	66.8	64.2	76.1	85.2	85.2	92.1
DINO	D-OCB (Ours)	67.9	68.3	70.3	74.0	69.9	55.6	71.1	71.0	71.0	71.0	70.0	70.8	70.6	70.6	70.8	93.9	96.9	97.3	97.4	97.6

Table 4: Downstream reasoning acc. (%) averaged across rules R1-R5 on the CLEVR dataset. Results over multiple folds (complete table in the supplementary)

BB	Sup%	Model	DILP (%)			Decision Tree (%)			Bayes Net (%)			NS-CL (%)
BB	Sup%	Model	R1	R2	R3	R1	R2	R3	R1	R2	R3	R1	R2	R3
CNN	1	D-OCB (Ours)	$\mathbf{31.8\pm 0.0}$	$\mathbf{34.8\pm 0.0}$	$\mathbf{86.6\pm 0.0}$	$\mathbf{59.7\pm 0.0}$	$\mathbf{69.2\pm 0.0}$	$\mathbf{72.6\pm 0.0}$	$\mathbf{76.6\pm 0.0}$	$\mathbf{82.6\pm 0.0}$	$\mathbf{92.5\pm 0.0}$	$\mathbf{52.7\pm 0.0}$	$\mathbf{60.2\pm 0.0}$	$\mathbf{92.5\pm 0.0}$
	1	Slot-VAE	-	-	-	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$15.6\pm 1.0$	$18.6\pm 1.0$	$2.6\pm 1.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
	15	D-OCB (Ours)	$\mathbf{69.7\pm 0.0}$	$\mathbf{72.6\pm 0.0}$	$\mathbf{80.1\pm 0.0}$	$\mathbf{63.2\pm 0.0}$	$\mathbf{58.2\pm 0.0}$	$\mathbf{72.1\pm 0.0}$	$\mathbf{84.6\pm 0.0}$	$\mathbf{82.6\pm 0.0}$	$\mathbf{95.0\pm 0.0}$	$\mathbf{69.2\pm 0.0}$	$\mathbf{64.2\pm 0.0}$	$\mathbf{84.1\pm 0.0}$
	15	Slot-VAE	-	-	-	$27.2\pm 3.0$	$15.4\pm 3.0$	$0.4\pm 1.0$	$30.0\pm 1.0$	$18.5\pm 1.0$	$0.0\pm 0.0$	$43.3\pm 0.0$	$37.9\pm 0.0$	$8.8\pm 0.0$
	75	D-OCB (Ours)	$\mathbf{82.6\pm 2.5}$	$\mathbf{82.3\pm 2.2}$	$\mathbf{92.4\pm 3.3}$	$\mathbf{79.9\pm 2.3}$	$\mathbf{73.8\pm 3.9}$	$\mathbf{84.6\pm 1.6}$	$\mathbf{86.4\pm 2.3}$	$\mathbf{83.9\pm 0.5}$	$\mathbf{93.5\pm 1.1}$	$\mathbf{76.3\pm 1.6}$	$\mathbf{73.1\pm 3.9}$	$\mathbf{89.7\pm 2.9}$

Table 5: Reasoning acc. (%) of D-OCB vs. Slot-VAE baseline on HAM10000 (RQ2) for three diagnostic rules across different backbones, supervision levels, and reasoning frameworks. Baseline F1 scores have been converted to % for direct comparison. Mean

\pm

std. dev. over 5 folds.

CLEVR-Hans3 Benchmark
Method	Sup.	Train Acc $\uparrow$	Test Acc $\uparrow$	OOD Gap $\downarrow$
E2E CNN Basel.	0%	$100.0\pm 0.0$	$70.3\pm 0.0$	$29.7\pm 0.0$
NeSy	100%	$98.5\pm 0.0$	$81.7\pm 0.0$	$16.8\pm 0.0$
NeSy-XIL	100%	$100.0\pm 0.0$	$91.3\pm 0.0$	$8.7\pm 0.0$
$\alpha$ ILP	100%	$97.5\pm 0.0$	$97.5\pm 0.0$	$-0.0\pm 0.0$
NEUMANN	100%	$96.7\pm 0.0$	$97.4\pm 0.0$	$-0.8\pm 0.0$

Table 6: Confounder Resistance on CLEVR-Hans3 and CLEVR-Hans7. Comparison of D-OCB against end-to-end architectures and dense-supervision baselines. We report train acc. (in-distribution), test acc. (out-of-distribution), and the OOD Gap (

\downarrow

). Mean

\pm

std. dev. over 5 folds.

Concept Bottleneck Models

Concept Bottleneck Models route predictions through human-understandable concepts that people can inspect, correct, or intervene on.

See all 88 papers

Weakly Supervised Concept Learning for Object-centric Visual Reasoning
May 5, 2026Sparsh Tiwari, Bettina Finzel, Gesina SchwalbeVisual Reasoning Object-Centric Representation Learning
University of Lübeck, Germany · University of Bamberg, Germany · University of Ulm, Germany
AnyBottle: A Recipe to Only Keep the Concepts You Really Need
Oct 6, 2026Wolfgang Stammer, Sukrut Rao, Hevra Petekkaya +2Efficient Neural Network Inference Concept Bottleneck Models
Max Planck Institute for Informatics, Saarland Informatics Campus · Technical University of Darmstadt
Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
May 13, 2026Nicola Debole, Andrea Passerini, Stefano Teso +2Concept Bottleneck Models Human-in-the-Loop Annotation
DISI, University of Trento, Italy · CIMeC, University of Trento, Italy

May 5, 2026J/K move · Enter open · S save

Weakly Supervised Concept Learning for Object-centric Visual Reasoning

Sparsh Tiwari, Bettina Finzel, Gesina Schwalbe

University of Lübeck, Germany · University of Bamberg, Germany · University of Ulm, Germany

Visual Reasoning Object-Centric Representation Learning Weakly Supervised Learning Visually Grounded Reasoning Neuro-Symbolic Reasoning

Open paper AskPDF arXiv

Neurosymbolic systems promise to combine deep neural network's (DNN) processing of raw sensor inputs with few-shot performance of symbolic artificial intelligence. Two-stage approaches explicitly decouple DNN based perception from subsequent rule based reasoning. This avoids optimization and interpretability issues of end to end differentiable approaches, but requires costly labels for the perception output. This paper introduces an efficient weak supervision scheme for the perception stage to ground its output symbols for logical induction in object-centric reasoning tasks. It combines a slot-based architecture for object-centricity with a Variational Autoencoder (VAE) for self-supervision, competing with concept guidance on latent dimensions for human interpretable grounding. The resulting predictions are translated into symbolic background knowledge for reasoning frameworks, such as Inductive Logic Programming (ILP), Decision Trees, and Bayesian Networks. Our extensive empirical evaluation on synthetic and real world datasets shows that our approach can discover complex, abstract rules for object centric reasoning whilst reducing supervision to as little as 1% of labels, and being robust even under substantial domain shift. Notably, at 1% supervision it even outperforms state of the art foundation model baselines in domain generalization

Abstract

Concept Bottleneck Models

Explore similar work

Weakly Supervised Concept Learning for Object-centric Visual Reasoning

AnyBottle: A Recipe to Only Keep the Concepts You Really Need

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations