Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Emanuel Ben Baruch; Sivan Biham; Alon Shoshan; Yochai Zvik; Avi Ben-Cohen

cs.CVOct 15, 2026

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Authors: Emanuel Ben Baruch, Sivan Biham, Alon Shoshan, Yochai Zvik, Avi Ben-Cohen

Organizations: Amazon

Abstract

When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model's own zero-shot performance. In this regime, we uncover a surprising matched-prompt penalty whereby the standard choice of reusing the descriptive finetuning prompt at inference can yield lower rare-class recall than using a different descriptive prompt. We call this inference-time strategy cross-prompt inference. We evaluate diverse VLMs across six datasets and find that cross-prompt inference consistently recovers rare-class recall, with varying trade-offs in frequent-class recall. The effect reflects a train-eval prompt interaction rather than prompt quality alone, and does not arise with non-descriptive prompt changes. It is strongest when imbalance and visual confusability produce substantial rare-class suppression, and is weak or absent when such suppression does not occur. Using Tuning Contribution (TuCo), which decomposes a model's computation into pretrained and finetuning parts, we find that cross-prompt inference reduces reliance on finetuning-induced computation and shifts the model toward its pretrained component. Reducing that finetuning contribution also substantially reduces the cross-prompt advantage. These results show that rare-class knowledge suppressed by imbalanced finetuning can remain accessible under a different inference prompt.

Figures & tables

Panel of Figure 1 — Figure 1: The matched-prompt penalty under class-imbalanced VLM finetuning. (a) Finetuning with prompt $P$ on an imbalanced distribution favors frequent classes and suppresses rare ones. Reusing $P$ at inference preserves this suppression, while a different descriptive prompt $P^{\prime}$ mitigates it without retraining by enabling the model to draw less on the computation introduced by finetuning. (b) A $7\times 7$ Food-101 cross-prompt matrix, in which the matched cell is never its row maximum.

Figure 3: Cross-prompt recovery of rare recall across models . Each panel corresponds to one dataset and shows, for each model, the recall change relative to matched for rare (green) and frequent (grey) classes. The dot is the averaged mismatch and the bar is the best mismatch. The rare gain is positive on every model and dataset, with a trade-off in frequent-class recall.

Dataset	Regime	Match	Mism.(avg)	Mism.(best)	Match	Mism.(avg)	Mism.(best)	ZS
		Rare ( $\Delta$ vs. matched)			Overall
Food-101	Imbalanced	$65.7$	$71.7$ $\,(+6.0)$	$75.7$ $\,(+10.0)$	$81.9$	$84.8$	$86.8$	$86.4$
Food-101	Oversampled	$69.9$	$75.5$ $\,(+5.6)$	$78.9$ $\,(+9.0)$	$83.9$	$86.5$	$88.1$	$86.4$
CUB-200	Imbalanced	$40.8$	$46.3$ $\,(+5.5)$	$59.5$ $\,(+18.7)$	$66.2$	$65.6$	$72.3$	$59.7$
CUB-200	Oversampled	$57.4$	$58.3$ $\,(+0.9)$	$71.0$ $\,(+13.6)$	$74.1$	$70.1$	$76.9$	$59.7$
FGVC-Aircraft	Imbalanced	$35.5$	$41.0$ $\,(+5.5)$	$46.1$ $\,(+10.6)$	$64.8$	$65.7$	$67.4$	$60.8$

Table 1: Rare and overall recall (%). Matched, average mismatched, and best mismatched results are shown, with

\Delta

vs. matched in parentheses. ZS is zero-shot overall recall averaged over the seven prompts. Cross-prompt inference recovers rare recall under imbalanced and oversampled training.

Panel of Figure 4 — Figure 4: Rare-class degradation: Dashed lines show the average accuracy over matched prompts. Solid lines show the average accuracy over best mismatch prompts.

	Descriptive			Non-descriptive
Dataset	Match	Mism.	$\Delta$	Match	Mism.	$\Delta$
Food-101	$65.7$	$71.7$	$+6.0$	$64.8$	$60.7$	$-4.1$
CUB-200	$40.8$	$46.3$	$+5.5$	$39.0$	$37.7$	$-1.3$
FGVC-Aircraft	$35.5$	$41.0$	$+5.5$	$35.4$	$33.5$	$-1.9$
Stanford Dogs	$57.0$	$59.5$	$+2.5$	$55.7$	$55.9$	$+0.2$

Table 2: Non-descriptive prompt setting. Rare-class recall (%), averaged over prompts, for matched vs. mismatched evaluation, with the cross-prompt gain

\Delta

		Mism.(avg)		Ens.(mism)		Ens.(all)
Dataset	Match	Rare	Freq.	Rare	Freq.	Rare	Freq.
Food-101	$65.7$	$+6.0$	$-0.2$	$+7.0$	$-0.3$	$+6.1$	$-0.2$
CUB-200	$40.8$	$+5.5$	$-6.6$	$+5.5$	$-3.7$	$+3.7$	$-2.9$
FGVC-Aircraft	$35.5$	$+5.5$	$-3.6$	$+3.9$	$-1.3$	$+2.4$	$-0.6$
Stanford Dogs	$57.0$	$+2.5$	$-4.7$	$+4.6$	$-2.5$	$+4.1$	$-1.8$

Table 3: Prompt ensembling. Changes in rare and frequent recall relative to the matched prompt (gain

\Delta

) for Mism.(avg) and prompt ensembling.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n_{f}$	$n_{r}$	ratio	train / test
Food-101	$750$	$5$	$150{:}1$	$7550$ / $2000$
CUB-200	$29$	$1$	$29{:}1$	$300$ / $220$
FGVC-Aircraft	$60$	$2$	$30{:}1$	$620$ / $600$
Stanford Dogs	$100$	$2$	$50{:}1$	$1020$ / $600$

Appendix

Table 4: Dataset configurations.

n_{f}

n_{r}

are images per frequent/rare class.

n_{f}

is the largest count the source pool supports.

Dataset	Training prompt	Matched	Mismatched	$\Delta$ ( $\times 10^{-3}$ )	Drop / Rank
Food-101	detailed_all__contrastive	$0.0529$	$0.0518$	$-1.07$	$89.6\%$ / 2
	detailed_all__appearance	$0.0525$	$0.0515$	$-0.95$	$90.4\%$ / 1
	detailed_all__attribute	$0.0547$	$0.0531$	$-1.61$	$99.4\%$ / 1
	detailed_rare__appearance	$0.0523$	$0.0513$	$-1.01$	$93.4\%$ / 1
CUB-200	detailed_all__contrastive	$0.0254$	$0.0243$	$-1.09$	$91.5\%$ / 1
	detailed_all__appearance	$0.0251$	$0.0241$	$-1.05$	$92.0\%$ / 1

Appendix

Table 5: Finetuning contribution under matched and mismatched prompts. Mean TuCo for each finetuned model under its own training prompt (matched) and averaged over the six remaining prompts (mismatched).

Dataset	Model	Matched	Mismatch	$\Delta$ r/ovr	$\Delta$ best r
		R/F $\cdot$ Ovr	R/F $\cdot$ Ovr	rare/ovr	rare
Food-101	Qwen3-VL-2B	$65.7/98.2\cdot 81.9$	$71.7/98.0\cdot 84.8$	$+6.0/+2.9$	$+10.0$
	Qwen3-VL-4B	$65.4/98.4\cdot 81.9$	$70.2/98.2\cdot 84.2$	$+4.8/+2.3$	$+8.4$
	Qwen3-VL-8B	$61.0/97.5\cdot 79.2$	$70.1/97.3\cdot 83.7$	$+9.1/+4.5$	$+13.6$
	Qwen2.5-VL-3B	$53.6/96.3\cdot 74.9$	$62.0/95.7\cdot 78.9$	$+8.5/+3.9$	$+12.1$
	Gemma-3-4B	$58.9/97.6\cdot 78.2$	$60.2/97.5\cdot 78.9$	$+1.3/+0.6$	$+2.9$

Appendix

Table 6: Breakdown across all six models. Each cell reports rare / frequent

\cdot

overall recall (%), and the

\Delta

columns are relative to matched.

with oversampling
Method	Food-101	CUB-200	FGVC-Aircraft	Stanford Dogs
Matched prompt	$65.7$ / $98.2$ / $81.9$	$40.8$ / $91.5$ / $66.2$	$35.5$ / $94.1$ / $64.8$	$57.0$ / $95.7$ / $76.4$
Mism.(avg)	$71.7$ / $98.0$ / $84.8$	$46.3$ / $84.9$ / $65.6$	$41.0$ / $90.5$ / $65.7$	$59.5$ / $91.0$ / $75.3$
Mism.(ens)	$72.7$ / $97.9$ / $85.3$	$46.3$ / $87.8$ / $67.1$	$39.4$ / $92.8$ / $66.1$	$61.6$ / $93.2$ / $77.4$
Mism.(sel, $k{=}1$ )	$73.8$ / $97.8$ / $85.8$	$56.4$ / $84.1$ / $70.3$	$42.9$ / $88.8$ / $65.8$	$61.0$ / $89.9$ / $75.4$
Mism.(sel, $k{=}5$ )	$74.8$ / $97.9$ / $86.3$	$58.6$ / $83.5$ / $71.0$	$45.3$ / $88.8$ / $67.1$	$62.4$ / $89.5$ / $75.9$

Appendix

Table 7: Method comparison. In each cell we report rare / frequent / overall recall (%). The best overall recall per dataset is in bold. Oversampling resamples each rare class up to the frequent count without adding new images.

full finetuning
Dataset	Match	Mism.(avg)	Mism.(best)
Food-101	$65.7$	$71.7$ $\,(+6.0)$	$75.7$ $\,(+10.0)$
CUB-200	$40.8$	$46.3$ $\,(+5.5)$	$59.5$ $\,(+18.7)$
FGVC-Aircraft	$35.5$	$41.0$ $\,(+5.5)$	$46.1$ $\,(+10.6)$
Stanford Dogs	$57.0$	$59.5$ $\,(+2.5)$	$63.1$ $\,(+6.1)$
LoRA

Appendix

Table 8: LoRA vs. full finetuning. Rare-class recall (%) with the mismatch gain (

\Delta

) vs. matched in parentheses, for both full finetuning and LoRA.

Dataset	Group#1	Group#2	Group#3	Group#4	Group#5	Group#6
	(similar)					(different)
Food-101	0.733	0.718	0.705	0.694	0.679	0.667
FGVC-Aircraft	0.803	0.793	0.772	0.760	0.754	0.734
Stanford Dogs	0.775	0.764	0.751	0.743	0.729	0.713
CUB-200	0.849	0.817	0.788	0.782	0.761	0.741
ImageNet	0.798	0.770	0.749	0.722	0.689	0.675

Appendix

Table 9: Average rare-to-frequent SigLIP2 cosine similarity for each of the six frequent-rare partitions, ordered from most to least similar. All groups use the same 20 classes per dataset.

Dataset	P $\to$ P	P $\to$ D	D $\to$ P	D $\to$ D matched	D $\to$ D ′ mismatched
Food-101	$65.5$	$70.8$	$72.7$	$65.7$	$71.5$
CUB-200	$37.9$	$52.1$	$28.5$	$40.8$	$49.8$
FGVC-Aircraft	$34.3$	$42.6$	$39.8$	$35.5$	$41.2$
Stanford Dogs	$52.3$	$55.7$	$60.2$	$57.0$	$59.4$

Appendix

Table 10: Where the descriptions appear. Rare-class recall (%) averaged over prompts and seeds, with P the plain prompt and D a descriptive one.

Dataset	Regime	Match	Mism.(avg)	Mism.(best)
Food-101	Imbalanced	$65.7$	$71.7$ $(+6.0\pm 0.8)$	$75.7$ $(+10.0\pm 1.2)$
Food-101	Oversampled	$69.9$	$75.5$ $(+5.6\pm 0.3)$	$78.9$ $(+9.0\pm 0.6)$
CUB-200	Imbalanced	$40.8$	$46.3$ $(+5.5\pm 0.8)$	$59.5$ $(+18.7\pm 0.9)$
CUB-200	Oversampled	$57.4$	$58.3$ $(+0.9\pm 1.1)$	$71.0$ $(+13.6\pm 1.6)$
FGVC-Aircraft	Imbalanced	$35.5$	$41.0$ $(+5.5\pm 0.7)$	$46.1$ $(+10.6\pm 0.5)$
FGVC-Aircraft	Oversampled	$41.7$	$48.8$ $(+7.1\pm 0.6)$	$56.2$ $(+14.5\pm 0.6)$

Appendix

Table 11: Rare-class recall (%) with seed variability. Values in brackets are the gain over the matched prompt and the gain’s standard error across seeds.

Class	appearance	contrastive	attribute
filet mignon	a thick round cut of grilled beef steak	a tall, round, boneless beef medallion (unlike the flat wide steak or bone-in pork chop)	beef; boneless; shape: tall round medallion; color: dark seared outside, red inside
prime rib	a large roasted beef rib cut, pink in the middle	a roasted beef slab with a bone and a pink center (unlike a seared boneless filet)	beef; bone-in; shape: thick roasted slab; color: brown crust, pink center
pork chop	a grilled bone-in pork cut	a pale bone-in pork cut, lighter than beef (unlike the dark red steak)	pork; bone-in; shape: flat chop; color: pale tan
steak	a grilled slab of beef	a flat, wide, dark-seared beef slab (unlike the tall round filet mignon)	beef; boneless; shape: flat wide slab; color: dark grilled
bolognese spaghetti	spaghetti with a red meat sauce	spaghetti under a chunky red tomato-meat sauce (unlike the pale creamy carbonara)	pasta; sauce: red tomato-meat; texture: chunky; color: red
carbonara spaghetti	spaghetti with a creamy egg-and-bacon sauce	spaghetti in a pale glossy egg-cream sauce, no red (unlike red bolognese)	pasta; sauce: egg-cream; texture: glossy; color: pale yellow

Appendix

Table 12: Food-101: per-class descriptions for all three styles. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rows for rare classes.

Class	appearance	contrastive	attribute
california gull	a medium gull with a yellow bill, dark eye, and grey back	a MEDIUM gull with yellow legs and a dark eye (unlike the pink-legged herring gull)	gull; size: medium; legs: yellow; eye: dark; back: grey
herring gull	a large pale-grey gull with pink legs and a red-spotted yellow bill	a LARGE pale gull with PINK legs (unlike the yellow-legged california/ring-billed gulls)	gull; size: large; legs: pink; bill: yellow with red spot; back: pale grey
ring-billed gull	a small gull with a yellow bill circled by a black ring, yellow legs	a small gull whose yellow bill has a clean BLACK RING (unlike the red-spotted herring bill)	gull; size: small; bill: yellow with black ring; legs: yellow
western gull	a large gull with a very dark slate back and pink legs	a gull with a very DARK slate back (unlike the pale-grey herring and california gulls)	gull; size: large; back: dark slate; legs: pink
arctic tern	a slender grey tern with a full black cap and short red legs	a tern with an ALL-RED short bill and very short legs (unlike caspian’s thick bill or common’s black-tipped bill)	tern; bill: all red; legs: very short red; cap: full black
caspian tern	a large tern with a thick bright-red bill and a black cap	a large tern with a THICK bright-red carrot bill (unlike the slender arctic/common terns)	tern; size: large; bill: thick bright red; cap: black

Appendix

Table 13: CUB-200: per-class descriptions for all three styles. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rows for rare classes.

Class	appearance	contrastive	attribute
737-300	a narrow-body Boeing 737 jet, mid-length fuselage	a Boeing 737 with a MID-length fuselage (longer than the short 737-500)	maker: Boeing; family: 737; fuselage: medium; engines: 2 underwing
737-500	a narrow-body Boeing 737 jet, short fuselage	a Boeing 737 with the SHORTEST fuselage of the family (shorter than the 737-300)	maker: Boeing; family: 737; fuselage: short; engines: 2 underwing
737-700	a narrow-body Boeing 737NG jet	a Boeing 737NG, SHORTER than the 737-800	maker: Boeing; family: 737NG; fuselage: short-medium; engines: 2 underwing
737-800	a narrow-body Boeing 737NG jet, long fuselage	a Boeing 737NG, the LONGEST fuselage (longer than the 737-700)	maker: Boeing; family: 737NG; fuselage: long; engines: 2 underwing
747-200	a four-engine Boeing 747 jumbo jet with a short upper deck	a Boeing 747 with a SHORT upper deck and NO winglets (unlike the 747-400)	maker: Boeing; family: 747; deck: short hump; winglets: no; engines: 4
747-400	a four-engine Boeing 747 jumbo jet with winglets	a Boeing 747 with WINGLETS and an extended upper deck (unlike the 747-200)	maker: Boeing; family: 747; deck: extended hump; winglets: yes; engines: 4

Appendix

Table 14: FGVC-Aircraft: per-class descriptions for all three styles. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rows for rare classes.

Class	appearance	contrastive	attribute
miniature schnauzer	a small wiry-coated schnauzer with a bearded muzzle and eyebrows	the SMALLEST schnauzer (vs the medium standard and large giant)	type: schnauzer; size: small; coat: wiry; face: bearded
standard schnauzer	a medium wiry schnauzer with a bearded muzzle	the MEDIUM schnauzer (bigger than miniature, smaller than giant)	type: schnauzer; size: medium; coat: wiry; face: bearded
giant schnauzer	a large black wiry schnauzer with a bearded muzzle	the LARGEST, black schnauzer (vs the small miniature/standard)	type: schnauzer; size: large; color: black; coat: wiry
golden retriever	a medium dog with a dense wavy golden coat	a retriever with a WAVY GOLDEN coat (vs the short-coated labrador)	type: retriever; coat: wavy; color: golden
labrador retriever	a short-coated retriever, solid color, otter tail	a SHORT-coated solid-color retriever (vs the wavy golden retriever)	type: retriever; coat: short; tail: otter
flat-coated retriever	a black retriever with a long flat glossy coat	a BLACK LONG-FLAT-coated retriever (vs the golden’s wavy gold coat)	type: retriever; coat: long flat; color: black

Appendix

Table 15: Stanford Dogs: per-class descriptions for all three styles. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rows for rare classes.

Class	appearance	contrastive	attribute
european fire salamander	a glossy black salamander with bold irregular yellow blotches	black salamander whose yellow markings are IRREGULAR blotches (unlike the spotted salamander’s neat round spots)	salamander; ground colour: black; markings: irregular yellow blotches; skin: smooth and glossy
spotted salamander	a dark grey salamander with two neat rows of round yellow spots	grey salamander with two rows of ROUND yellow spots (unlike the european fire salamander’s irregular blotches)	salamander; ground colour: dark grey or slate; markings: two rows of round yellow spots; skin: smooth
black and gold garden spider	a large orb-weaver with a boldly black-and-yellow patterned abdomen	orb-weaver with a BOLD BLACK-AND-YELLOW abdomen (unlike the garden spider’s brown cross mark)	orb-weaver spider; abdomen: bold black and yellow bands; size: large; web: circular with a zigzag centre
garden spider	a brown orb-weaver with a pale cross-shaped mark on its abdomen	BROWN orb-weaver with a pale CROSS on the abdomen (unlike the black and gold garden spider’s yellow bands)	orb-weaver spider; abdomen: mottled brown with a pale cross; size: medium; web: circular
red-backed sandpiper	a small shorebird with a short slightly drooping bill and a rufous back	small sandpiper with a SHORT drooping bill (unlike the dowitcher’s very long straight bill)	sandpiper; size: small; bill: short, drooping at the tip; back: rufous; belly: black patch in breeding plumage
dowitcher	a medium shorebird with a very long straight bill probing mud	shorebird with a VERY LONG straight bill (unlike the red-backed sandpiper’s short drooping bill)	shorebird; size: medium; bill: very long and straight; feeding: vertical sewing-machine probing

Appendix

Table 16: ImageNet-20: per-class descriptions for all three styles. Rows are ordered as the ten similarity-matched pairs, frequent class first. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rare rows.

Class	appearance	contrastive	attribute
Ophrys apifera	an orchid whose furry brown lip looks like a bee, with a tucked-under yellow tip	bee orchid whose lip appendage is TUCKED UNDER and hidden (unlike Ophrys fuciflora’s forward-pointing one)	genus: Ophrys; lip: brown velvety, bee-like; appendage: reflexed and hidden; sepals: pink to white
Ophrys fuciflora	an orchid with a broad square furry lip and a forward-pointing green appendage	bee orchid with a BROAD SQUARE lip and a FORWARD-pointing green appendage (unlike Ophrys apifera’s hidden tip)	genus: Ophrys; lip: broad and squarish; appendage: forward-pointing, visible; sepals: pink, broad
Lamium hybridum	a small dead-nettle with pink-purple hooded flowers and deeply cut toothed leaves	dead-nettle with DEEPLY CUT sharply toothed leaves (unlike Lamium purpureum’s shallowly toothed heart-shaped leaves)	genus: Lamium; flower: pink-purple hooded; leaf: deeply and sharply toothed; habit: low
Lamium purpureum	a small low plant with pinkish-purple hooded flowers and heart-shaped purple-tinged leaves crowded near the top	dead-nettle with HEART-SHAPED shallowly toothed purple-tinged leaves (unlike Lamium hybridum’s deeply cut leaves)	genus: Lamium; flower: pink-purple hooded; leaf: heart-shaped purple-tinged; habit: low
Trifolium pratense	a clover with dense rounded pink-purple flower heads and pale V-marked leaflets	clover whose leaflets carry a PALE V MARK and whose head is pink-purple (unlike Trifolium medium’s unmarked leaflets)	genus: Trifolium; head: dense globe, pink-purple; leaflet: oval with a pale V; habit: upright
Trifolium medium	a clover with deep red-purple heads and narrow leaflets lacking a pale mark	clover with NARROW UNMARKED leaflets and a deep red-purple head (unlike Trifolium pratense’s V-marked leaflets)	genus: Trifolium; head: deep red-purple; leaflet: narrow, unmarked; habit: sprawling, zigzag stem

Appendix

Table 17: PlantNet-20: per-class descriptions for all three styles. Rows are ordered as the ten similarity-matched pairs, frequent class first. A detailed_all prompt supplies the whole column; detailed_rare supplies only the rare rows.

VLM Inference

Vision-language model inference studies how multimodal models turn images and video into answers efficiently, accurately, and reliably.

See all 82 papers

Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangVision-Language Models VLM Adaptation
Beijing Jiaotong University, Beijing, China.
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Sep 20, 2026Weihan Cai, Hao Tan, Xinping Gao +2Zero-Shot Learning VLM Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Purple Mountain Laboratories, Nanjing, China · Beijing University of Posts and Telecommunications, Beijing, China
AdaBoosting Text Prompts for Vision-Language Models
Jul 1, 2026Seokhee Jin, Changhwan Sung, Sunung Mun +2Vision-Language Models Ensemble Learning
KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea

May 15, 2026J/K move · Enter open · S save

Neutral-Reference Prompting for Vision-Language Models

Senmao Tian, Xiang Wei, Shunli Zhang

Beijing Jiaotong University, Beijing, China.

Vision-Language Models VLM Adaptation Few-Shot Learning Zero-Shot Image Classification Prompt Learning for VLMs

Open paper AskPDF arXiv

Efficient transfer learning of vision-language models (VLMs) commonly suffers from a Base-New Trade-off (BNT): improving performance on unseen (new) classes often degrades accuracy on known (base) classes. Addressing how to boost recognition of unseen classes without sacrificing known-class performance remains a central challenge. Existing work often simplistically attributes the BNT to overfitting on known classes. We observe an interesting phenomenon: VLMs frequently exhibit asymmetric confusion on certain downstream data, i.e., samples of class A are systematically mispredicted as class B, while the reverse confusion (B to A) rarely occurs. For known classes, this kind of bias can be mitigated by tuning using a cross-entropy loss, but for unseen classes, such pretraining-induced bias persists and harms generalization. Motivated by this, we propose NeRP, a plug-and-play prompting correction strategy that improves discrimination on unseen classes without modifying model parameters. NeRP leverages neutral text prompts and reference images to measure class-wise prior preferences along the pre-trained inter-class geometry, and combines them with the sample likelihood to obtain the model's surrogate score. If, for a given sample, the prior strongly favors the current prediction while the observed evidence is clearly insufficient, we perform a local flip between easily confusable class pairs, thereby correcting prior-dominated mispredictions. Extensive experiments across multiple backbones and 15 few-shot and cross-domain benchmarks show that NeRP substantially improves accuracy on unseen classes while preserving known-class prediction performance.

Abstract

VLM Inference

Explore similar work

Neutral-Reference Prompting for Vision-Language Models

PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models

AdaBoosting Text Prompts for Vision-Language Models