AnyBottle: A Recipe to Only Keep the Concepts You Really Need

Wolfgang Stammer; Sukrut Rao; Hevra Petekkaya; David Steinmann; Bernt Schiele

cs.AIOct 6, 2026

AnyBottle: A Recipe to Only Keep the Concepts You Really Need

Authors: Wolfgang Stammer, Sukrut Rao, Hevra Petekkaya, David Steinmann, Bernt Schiele

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus · Technical University of Darmstadt

Abstract

Concept bottleneck models (CBMs) make predictions inspectable and intervenable by routing them through human-interpretable concepts, but originally required concept annotations. Annotation-free variants remove this requirement, but typically use large concept vocabularies, static at both training and inference, producing bottlenecks larger than any task or prediction needs and harder to inspect. We propose AnyBottle, a single recipe for building compact, task-specific CBMs. AnyBottle assumes only a frozen backbone and an unsupervised concept pool, such as a sparse autoencoder. A black-box teacher trained on the same backbone then guides selection: each round adds the concept that best explains the bottleneck's current failures, with candidates restricted to regions of teacher/student disagreement. Trained with nested dropout over this selection order, the final bottleneck predicts accurately from any concept prefix, so inference spends fewer concepts on inputs it is confident about early and more on hard ones. Since no stage is modality-specific, a new domain and task requires swapping only the backbone and concept pool. Across six vision and two text datasets and two teacher paradigms, AnyBottle yields bottlenecks with fewer concepts and higher concept consistency than annotation-free baselines, while staying close to the black-box reference. Overall, AnyBottle shows that going annotation-free need not mean going large: a small, discovered vocabulary can be as expressive as a much larger, fixed one.

Figures & tables

Figure 1: Only as many concepts as needed. Left: existing annotation-free concept bottlenecks draw from a large, static concept pool, using far more concepts than a given task needs, and the same fixed vocabulary for every sample regardless of difficulty. AnyBottle instead selects a concept set sized to the task, and lets inference use only as many of those concepts as each sample actually requires. Right: two test samples illustrating this sample-level adaptivity.

Figure 2: Overview of AnyBottle . An input $x$ is encoded by the frozen backbone $g$ into token embeddings $z^{\ell}$ , which a sparse autoencoder $a$ maps to concept activations $c^{\ell}$ . Each selection round repeats four steps: (1) a forward pass through the teacher $f_{T}$ and current student $f_{\mathcal{S}}$ ; (2) teacher attributions over the input; (3) the resulting teacher/student blind-spots (green), used to restrict the SAE to blind-spot tokens; and (4) scoring the resulting candidates by least-squares residual $R$ against the teacher’s output $Y_{T}$ to pick the next concept $j^{\star}$ , added to $\mathcal{S}$ . This repeats until selection converges on $\mathcal{S}^{\star}$ . Once selection terminates, (5) the final predictor $f$ is trained once on rescaled $\tilde{C}_{\mathcal{S}^{\star}}$ with nested dropout, yielding a head that predicts accurately from any concept prefix.

Figure 3: Top: AnyBottle matches or exceeds black-box teacher accuracy at a fraction of the concepts used. Accuracy gap (

\Delta

(pp)) to the teacher, plotted against concept count (log scale); AnyBottle reaches the teacher across all six datasets with orders of magnitude fewer concepts than the baselines. Bottom: AnyBottle ’s smaller bottleneck is also the more interpretable one. Concept consistency (

C^{2}_{\mathrm{img}}

) at the same budgets matches or exceeds every baseline, showing the compactness above is not simply discarded capacity. Points are seed means.

Panel of Figure 4 — Figure 4: Left: AnyBottle reaches near-full accuracy with a fraction of its concept budget. Concepts needed to reach $90$ / $95$ / $98\%$ of the student’s full-budget accuracy, per dataset; $K$ shown for reference. Right: Early rounds close large blind spots; later rounds refine with diminishing returns. Round-by-round validation accuracy on Imagenette, with two exemplary rounds detailed.

Figure 5: Adaptive inference uses more concepts only on samples that need them. Two easy samples (top) and one hard sample (bottom) given the task and selected concept set. For each sample, the activating concepts to reach 90% confidence are shown, together with three exemplars. Boxed numbers mark firing concepts left out for visibility.

LS	Loc	PF	K $\downarrow$	C ${}^{2}_{\text{patch}}$ $\uparrow$	$\Delta$ $\uparrow$ (%)
✓	✓	✓	$145.0$	$0.362$	$-0.4$
✓	✓	–	$165.7$	$0.345$	$-0.9$
✓	–	–	$190.5$	$0.314$	$-1.3$
–	–	–	$231.3$	$0.272$	$-2.1$

Table 1: Every component of AnyBottle ’s selection facilitates compact, consistent, and accurate concept sets. Ablation over selection strategy, averaged over vision datasets. LS: least-squares scoring (vs. random selection). Loc: localized candidate search. PF: prefiltering.

	Method	K $\downarrow$	AIS $\uparrow$	$\Delta$ (%) $\uparrow$	Acc (%) $\uparrow$
AG News	SAE probe	$31477$	$0.327{\scriptstyle\pm 0.009}$	$-0.12{\scriptstyle\pm 0.38}$	$90.42{\scriptstyle\pm 0.38}$
AG News	AnyBottle (mlp)	$93.7{\scriptstyle\pm 6.4}$	$0.469{\scriptstyle\pm 0.006}$	$-0.18{\scriptstyle\pm 0.25}$	$90.36{\scriptstyle\pm 0.25}$
[2pt/2pt]	SAE probe	$9710$	$0.157{\scriptstyle\pm 0.001}$	$+18.57{\scriptstyle\pm 0.98}$	$88.25{\scriptstyle\pm 0.98}$
	AnyBottle (verb)	$30.0{\scriptstyle\pm 5.0}$	$0.094{\scriptstyle\pm 0.020}$	$+0.82{\scriptstyle\pm 1.89}$	$70.50{\scriptstyle\pm 1.89}$
PubMed	SAE probe	$25893$	$0.453{\scriptstyle\pm 0.000}$	$-2.47{\scriptstyle\pm 0.87}$	$73.58{\scriptstyle\pm 0.87}$
PubMed	AnyBottle (mlp)	$100.0{\scriptstyle\pm 2.6}$	$0.526{\scriptstyle\pm 0.004}$	$-0.02{\scriptstyle\pm 0.23}$	$76.03{\scriptstyle\pm 0.23}$

Table 2: AnyBottle generalizes across domains and teacher paradigms. Evaluation of AnyBottle on two text datasets with a trained mlp-clf teacher and a zero-shot verbalizer teacher, verb-clf. We report concept count

K

, AutoInterp score (AIS), teacher gap (

\Delta

), and absolute balanced accuracy (Acc).

Figure 6: Raw accuracy of CBM baselines across vision datasets.

Figure 7: Patch-level concept consistency (C

{}^{2}_{\text{patch}}

) versus number of concepts, across six datasets. Top row : Linear SAE probe evaluated at its full dictionary size (8192) compared to AnyBottle. Bottom row : the same comparison restricted to the Linear SAE probe’s active features only, shown at a finer x-axis scale. AnyBottle achieves comparable or higher C

{}^{2}_{\text{patch}}

at substantially fewer concepts.

Dataset	LS	Loc	PF	K $\downarrow$	C ${}^{2}_{\text{patch}}$ $\uparrow$	$\Delta$ $\uparrow$ (%)
Imagenette	✓	✓	✓	$34.0{\scriptstyle\pm 2.0}$	$0.442{\scriptstyle\pm 0.005}$	$+0.3{\scriptstyle\pm 0.6}$
	✓	✓	–	$47.3{\scriptstyle\pm 3.5}$	$0.402{\scriptstyle\pm 0.008}$	$-0.9{\scriptstyle\pm 0.4}$
	✓	–	–	$87.3{\scriptstyle\pm 1.5}$	$0.348{\scriptstyle\pm 0.003}$	$+0.4$
	–	–	–	$184.7{\scriptstyle\pm 22.8}$	$0.269{\scriptstyle\pm 0.015}$	$+0.2$
Dogs-10	✓	✓	✓	$37.0{\scriptstyle\pm 1.0}$	$0.341{\scriptstyle\pm 0.002}$	$-1.8{\scriptstyle\pm 0.3}$
	✓	✓	–	$49.0{\scriptstyle\pm 0.0}$	$0.345{\scriptstyle\pm 0.000}$	$-2.0{\scriptstyle\pm 0.2}$

Table 3: Per-dataset results underlying the averaged ablation in Table Tab. 1 (main text). Ablations of least-squares scoring (LS), localized candidate search (Loc), and prefiltering (PF).

\Delta

= balanced test accuracy

-

balanced teacher accuracy. C

{}^{2}_{\text{patch}}

is the patch-level concept consistency. Mean

\pm

std over 3 seeds.

Dataset	Config	#Concepts $\downarrow$	C ${}^{2}_{\text{patch}}$ $\uparrow$	$\Delta$ $\uparrow$ (%)
Imagenette	Discrete	$90.0{\scriptstyle\pm 9.0}$	$0.38{\scriptstyle\pm 0.012}$	$+0.1{\scriptstyle\pm 0.4}$
Imagenette	Continuous	$34.0{\scriptstyle\pm 2.0}$	$0.442{\scriptstyle\pm 0.005}$	$+0.3{\scriptstyle\pm 0.6}$
Dogs-10	Discrete	$89.0{\scriptstyle\pm 3.0}$	$0.317{\scriptstyle\pm 0.004}$	$+0.3{\scriptstyle\pm 0.8}$
Dogs-10	Continuous	$37.0{\scriptstyle\pm 1.0}$	$0.341{\scriptstyle\pm 0.002}$	$-1.8{\scriptstyle\pm 0.3}$
Machines-15	Discrete	$141.0{\scriptstyle\pm 38.0}$	$0.405{\scriptstyle\pm 0.015}$	$-0.9{\scriptstyle\pm 1.1}$
Machines-15	Continuous	$110.0{\scriptstyle\pm 14.0}$	$0.399{\scriptstyle\pm 0.009}$	$+0.3{\scriptstyle\pm 0.4}$

Table 4: Ablation studies on concept representation (left) and SAE fine-tuning (right). Left : effect of concept representation with a Task-aware: discrete vs continuous concept scores. Right : effect of SAE fine-tuning with continuous concepts: task-agnostic pretrained vs. task-aware finetuned.

\Delta

= balanced test accuracy

-

balanced teacher accuracy; C

{}^{2}_{\text{patch}}

is the patch-level concept consistency. Mean

\pm

std over 3 seeds.

Dataset	$\delta_{acc}$	#Concepts	C ${}^{2}_{\text{patch}}$ $\uparrow$	C ${}^{2}_{\text{img}}$ $\uparrow$	$\Delta_{\text{val}}$ (%)	$\Delta_{\text{test}}$ (%)
Imagenette	$0.0$	$33.7{\scriptstyle\pm 2.1}$	$0.442{\scriptstyle\pm 0.005}$	$0.324{\scriptstyle\pm 0.003}$	$+0.09$	$+0.33$
	$0.1$	$33.7{\scriptstyle\pm 2.1}$	$0.442{\scriptstyle\pm 0.005}$	$0.324{\scriptstyle\pm 0.003}$	$+0.09$	$+0.33$
	$0.2$	$25.0{\scriptstyle\pm 4.6}$	$0.453{\scriptstyle\pm 0.007}$	$0.335{\scriptstyle\pm 0.007}$	$-0.19$	$-0.27$
Dogs-10	$0.0$	$36.7{\scriptstyle\pm 0.6}$	$0.341{\scriptstyle\pm 0.002}$	$0.105{\scriptstyle\pm 0.001}$	$+0.05$	$-1.80$
	$0.1$	$36.7{\scriptstyle\pm 0.6}$	$0.341{\scriptstyle\pm 0.002}$	$0.105{\scriptstyle\pm 0.001}$	$+0.05$	$-1.80$
	$0.2$	$36.7{\scriptstyle\pm 0.6}$	$0.341{\scriptstyle\pm 0.002}$	$0.105{\scriptstyle\pm 0.001}$	$+0.05$	$-1.80$

Table 5: Effect of the stopping tolerance

\delta_{acc}

on AnyBottle, across all six vision datasets.

\Delta

= Accuracy

-

Teacher (balanced accuracy). C

{}^{2}_{\text{patch}}

is the patch-level (localised) consistency score; C

{}^{2}_{\text{img}}

is the image-level score. Mean

\pm

std over 3 seeds.

Dataset	Method	#Concepts $\downarrow$	C@90 $\downarrow$	C@95 $\downarrow$	AIS $\uparrow$	$\Delta$ (%) $\uparrow$	Acc (%) $\uparrow$
AG News	Linear SAE probe	$31477$	–	–	$0.327{\scriptstyle\pm 0.009}$	$-0.12{\scriptstyle\pm 0.38}$	$90.42{\scriptstyle\pm 0.38}$
AG News	mlp-clf (gemma-2-2B)	$93.7{\scriptstyle\pm 6.4}$	$63{\scriptstyle\pm 1}$	$64{\scriptstyle\pm 0}$	$0.469{\scriptstyle\pm 0.006}$	$-0.18{\scriptstyle\pm 0.25}$	$90.36{\scriptstyle\pm 0.25}$
[2pt/2pt]	Linear SAE probe	$9710$	–	–	$0.157{\scriptstyle\pm 0.001}$	$+18.57{\scriptstyle\pm 0.98}$	$88.25{\scriptstyle\pm 0.98}$
	verb-clf (gemma-3-1B-it)	$30.0{\scriptstyle\pm 5.0}$	$27{\scriptstyle\pm 10}$	$30{\scriptstyle\pm 6}$	$0.094{\scriptstyle\pm 0.020}$	$+0.82{\scriptstyle\pm 1.89}$	$70.50{\scriptstyle\pm 1.89}$
PubMed-20k	Linear SAE probe	$25893$	–	–	$0.453{\scriptstyle\pm 0.000}$	$-2.47{\scriptstyle\pm 0.87}$	$73.58{\scriptstyle\pm 0.87}$
PubMed-20k	mlp-clf (gemma-2-2B)	$100.0{\scriptstyle\pm 2.6}$	$47{\scriptstyle\pm 10}$	$62{\scriptstyle\pm 1}$	$0.526{\scriptstyle\pm 0.004}$	$-0.02{\scriptstyle\pm 0.23}$	$76.03{\scriptstyle\pm 0.23}$

Table 6: Across domains and teacher paradigms, AnyBottle matches near-teacher accuracy with orders of magnitude fewer concepts than a linear probe over the full SAE dictionary. We report concept count

K

, concepts needed to reach

90\%

and

95\%

of the student accuracy (C@90/95), AutoInterp score (AIS), and balanced accuracy gap to teacher (

\Delta

) and in absolute terms (Acc).

Figure 8: Concept selection round by round on all six vision datasets. For each dataset, we show the validation accuracy of the current predictor $f_{\mathcal{S}}$ (left) and the concepts and blind spots for three rounds (right). The dashed line is the teacher’s validation accuracy. For each round, we show one training image from its focus set on which the added concept fires, with the teacher’s attribution and the concept’s activation. Green outlines mark the round’s blind spots, i.e. the tokens the teacher attends to, but $f_{\mathcal{S}}$ does not. Below each image are its class and the probability $f_{\mathcal{S}}$ assigns to it before $\to$ after the round.

Figure 9: Adaptive inference on the other five datasets. Two test images per dataset, one per row, as Fig. 5 shows them for Mixed-20. Imagenette, Dogs-10 and Machines-15 here; Flowers-102 and ISIC-7 in Fig. 10 .

Figure 10: Adaptive inference on Flowers-102 and ISIC-7 , continued from Fig. 9 .

Figure 11: Concept examples. Five concepts per dataset, drawn at random, one per row, each with the ten test images it fires on most strongly. Each image keeps its colour where the concept fires, scaled to its own strongest patch, and fades elsewhere. Imagenette, Dogs-10 and Machines-15 here; Mixed-20, Flowers-102 and ISIC-7 in Fig. 12 .

Figure 12: Concept examples , continued from Fig. 11 : Mixed-20, Flowers-102 and ISIC-7.

Subset	Classes
Dogs (10)	Golden retriever, Labrador retriever, Chesapeake Bay retriever, German shepherd, Rottweiler, Doberman, Boxer, Great Dane, Siberian husky, Eskimo dog
Machines (15)	Sports car, Station wagon, Convertible, Jeep, Pickup, Garbage truck, Trailer truck, Steam locomotive, Streetcar, Freight car, Airliner, Warplane, Speedboat, Submarine, Forklift
Mixed (20)	Animals: Lion, Tiger, African elephant, Zebra, Giant panda, Polar bear, Gorilla, Flamingo, Ostrich, Hippopotamus; Machines: Sports car, Warplane, Submarine, Steam locomotive, Forklift, Ambulance, Tank, Space shuttle, Fire engine, Motor scooter

Table 7: ImageNet-1k classes used in our three curated subsets.

Dataset	Prompt	Class $\rightarrow$ token
AG News	Article: "{text}" Which topic best fits this article: World, Sports, Business, or Technology? Answer with exactly one word.	World, Sports, Business, Sci/Tech $\rightarrow$ Technology
PubMed 20k RCT	Sentence from a medical research abstract: "{text}" Does this sentence state the study’s background, objective, methods, results, or conclusion? Answer with exactly one word.	Background, Objective, Methods, Results, Conclusion

Table 8: Verbalizer prompts (wrapped in the Gemma-3 chat template) and class-to-token mappings.

Stage	Parameter	Vision	Text
Backbone / Teacher	Teacher backbone	CLIP-DINOiser (ViT-B/16), CLS	G2: Gemma-2-2B, last-token resid. (layer 19) / G3: Gemma-3-1B-it (frozen verbalizer)
	Teacher head	MLP, hidden dim 512, dropout 0.1 (G3: none, frozen LM)
	Teacher optimizer	Adam, lr $10^{-3}$ , weight decay $10^{-4}$ (G3: n/a)
	Teacher batch size / epochs	256 / 40	full-batch / 60 (early stop, patience 12)
	Teacher selection metric	balanced val. accuracy
SAE	Architecture	Matryoshka BatchTopK	G2: TopK (SAEBench) / G3: JumpReLU (Gemma Scope 2)

Table 9: Hyperparameters used throughout AnyBottle ’s selection and training pipeline. Where vision and text differ, both values are given; a single value applies to both. For text, G2 = Gemma-2-2B MLP-teacher runs, G3 = Gemma-3-1B-it verbalizer runs. Accuracy thresholds are in percentage points.

Explore similar work

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
May 13, 2026Nicola Debole, Andrea Passerini, Stefano Teso +2Concept Bottleneck Models Annotations
DISI, University of Trento, Italy · CIMeC, University of Trento, Italy
Caption Bottleneck Models
Jul 1, 2026Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli +1Concept Bottleneck Models Image Difference Captioning
Dept. of Computer Eng., Middle East Technical University (METU) · Robotics & AI Center (ROMER), METU
Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation
Jul 1, 2026Md Mohasin Hossain, Anar Amirli, Robert Leist +2Heatmap Gradient-Weighted Class Activation Mapping
German Research Center for Artificial Intelligence · Saarland University, Saarbrücken, Germany · 1German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany +5

May 13, 2026, cs.CVJ/K move · Enter open · S save

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

Nicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana+1

DISI, University of Trento, Italy · CIMeC, University of Trento, Italy

Concept Bottleneck Models Annotations Bottlenecks

Open paper AskPDF arXiv

Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts -- and the predictions they entail -- by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from Vision-Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM's embedding space, which captures useful global information about the target domain, to propagate the expert's supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data, while sporting better concept calibration and supporting active learning.

Abstract

Explore similar work

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

Caption Bottleneck Models

Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation