RSAJEV: Training-Free Abstaining Prototype Memory for Eliminating Silent Hallucination in JEV-Based Remote Sensing Classification

Man Chen; Antai Chen; Shuai Xu; Chengjie Gu; Huaping Zhou; Kelei Sun; Tao Wu; Yongjie Huang

cs.CVOct 11, 2026

RSAJEV: Training-Free Abstaining Prototype Memory for Eliminating Silent Hallucination in JEV-Based Remote Sensing Classification

Authors: Man Chen, Antai Chen, Shuai Xu, Chengjie Gu, Huaping Zhou, Kelei Sun, Tao Wu, Yongjie Huang

Organizations: School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China, and also with the School of Electrical Automation and Information Engineering, Tianjin University, Tianjin 300072, China, and the State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Huainan 232001, Anhui, China · School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China · School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China, and also with the State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Huainan 232001, Anhui, China · Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin 541004, Guangxi, China

Abstract

Scene classification of remote sensing imagery with multimodal large language models (MLLMs) is formulated as a single-forward-pass decision problem: candidate class names are lettered options in the prompt, a registered answer slot is conditioned on the prompt, and classification is read out by scoring letter tokens with the language-modeling head. We show that although this letter binding is highly effective within its training vocabulary (97.71% OA on UCM), it behaves erratically out of vocabulary: zero-adaptation accuracy across four cross-dataset directions fluctuates between 7.35% and 96.40%, and under out-of-vocabulary weights its confidence fails inversely on open-set evaluation, assigning higher confidence to unseen than to seen classes (AUROC 0.38), a failure we term silent hallucination. We retain the slot representation and replace the readout with RSAJEV, a training-free abstaining prototype memory that performs retrieval-based prototype matching: K reference images per class are cached as prototypes via their slot representations under the same prompt, and classification becomes nearest-neighbor retrieval over the prototypes, refined by a shrinkage-Mahalanobis geometric correction. This readout is mechanistically isomorphic to retrieval-augmented generation (RAG) but operates on discriminative decisions; it is the first retrieval-based readout for MLLM slot representations. The retrieval readout reaches 89.43% on the UCM cross-dataset gate with 16 references per class and zero gradient updates, and 96.90% on AID. Its ID-ness scores attain 0.91-0.99 AUROC on open-set evaluation over two datasets, exceeding symbolic confidence (0.38-0.96) in all four open-set settings, restoring a calibratable abstention channel and supporting class addition/removal at zero extra inference cost via offline rescoring of the cached matrix.

Figures & tables

Fig. 1: Comparison of three families of methods: structural differences, strengths, and weaknesses of conventional visual classifiers, CLIP/generative MLLMs, and this paper’s approach. Our column is the main line of the method; the symbolic and text readouts serve only as controls.

Fig. 2: Pipeline overview: one shared forward pass produces the slot representation; the retrieval readout completes the decision over cached prototypes and provides the abstention channel.

Fig. 3: Coverage–accuracy curves: subset accuracy after accepting samples in descending order of ID-ness. (a) UCM $\to$ AID gate; (b) AID $\to$ NWPU gate—95.77 at 10% coverage versus 87.30 for cosine and 14.29 for the symbolic head.

Setting	UCM OA $\uparrow$	UCM F1 $\uparrow$	AID OA $\uparrow$	AID F1 $\uparrow$
Official report [ 6 ]	97.71	97.72	96.56	96.30
Ours: official-weights evaluation	–	–	97.39	97.16

TABLE I: Closed-Set Evaluation and Officially Released Numbers (%). The Two AID Rows Use Different Test-Split Draws and Serve as Magnitude References

Direction	letters	cos_t2	proto	maha	$\Delta$
AID $\to$ UCM (cross, 21 classes)	21.52	57.33	78.76	89.43	$+10.7$
UCM $\to$ AID (cross, 30 classes)	96.40	91.55	96.46	96.90	$+0.4$
AID $\to$ NWPU (cross, 45 classes)	7.35	39.47	64.55	78.52	$+14.0$
UCM $\to$ NWPU (cross, 45 classes)	34.29	22.80	64.50	77.99	$+13.5$
UCM $\to$ UCM (in-domain)	78.10	38.10	82.86	89.05	$+6.2$
AID $\to$ AID (in-domain)	97.39	96.81	97.51	97.40	$+0.1$

TABLE II: Cross-Dataset Decision Gate: One Frozen Forward Pass and Reference Cache, Swapping Only the Scoring Head, Comparing Four Readouts Row by Row (OA, %;

\lambda

Selected on the Shared Validation Grid)

Fig. 4: Per-Class F1 of the Prototype Readout ( $K{=}16$ ) on the UCM Cross-Dataset Gate (Derived From the Confusion Matrix): 13/21 Classes Exceed the 78.64 Macro-Average

Fig. 5: AUROC of Five ID-ness Scorers Under Three Open-Set Settings: UCM 14/7, AID 20/10 Cross-Weights, AID 20/10 Clean Weights

ID-ness score	AUROC $\uparrow$	FPR@95TPR $\downarrow$	OSCR $\uparrow$
letters (max softmax)	0.3853	1.0000	0.1291
cos_t2 (text control)	0.7593	0.6214	0.2304
proto (cosine proto)	0.8826	0.6029	0.2661
maha (ours, $\lambda{=}0.05$ )	0.9147	0.4671	0.2680

TABLE III: UCM 14 Seen/7 Unseen, Official Weights; Out-of-Vocabulary Weight Setting (Deployed Weights Whose Training Vocabulary Excludes the Target Labels)

ID-ness score	AUROC $\uparrow$	FPR@95TPR $\downarrow$	OSCR $\uparrow$
letters (max softmax)	0.7642	0.6729	0.7151
cos_t2 (text control)	0.6545	0.6486	0.5394
proto (cosine proto)	0.9121	0.3814	0.8123
maha (ours, $\lambda{=}0.1$ )	0.9426	0.2900	0.8304

TABLE IV: UCM 14 Seen/7 Unseen, Weights Retrained on the 14 Seen Classes Only (2 Epochs, Never Seeing the 7 OOD Classes); Clean-Denominator Setting, Testing Whether the Conclusions of Table III Hold

ID-ness score	AUROC $\uparrow$	FPR@95TPR $\downarrow$	OSCR $\uparrow$
letters (max softmax)	0.7323	0.9208	0.5755
cos_t2 (text control)	0.3750	0.9990	0.2897
proto (cosine proto)	0.9936	0.0219	0.7524
maha (ours, $\lambda{=}0.1$ )	0.9865	0.0627	0.7486

TABLE V: AID 20 Seen/10 Unseen (Seed-42 Split, 20-Option Prompts,

n{=}8000

), UCM-Trained Weights Never Having Seen AID Classes; Out-of-Vocabulary Weight Setting

ID-ness score	AUROC $\uparrow$	FPR@95TPR $\downarrow$	OSCR $\uparrow$
letters (max softmax)	0.9630	0.1176	0.9444
cos_t2 (text control)	0.1806	1.0000	0.1763
proto (cosine proto)	0.9911	0.0318	0.9699
maha (ours, $\lambda{=}0.5$ )	0.9937	0.0249	0.9715

TABLE VI: AID 20 Seen/10 Unseen, Weights Retrained on the 20 Seen Classes Only (2 Epochs, Never Seeing the 10 OOD Classes); Clean-Denominator Setting

Fig. 6: Readout Ablation on the Shared Representation (a) and Shrinkage-Mahalanobis $\lambda$ Sensitivity (b)

Fig. 7: Prototype-Readout Accuracy Versus the Number of References $K$ : Mean and Medoid Aggregation (UCM Gate)

Readout variant	OA $\uparrow$
Cosine prototype (= L2 on normalized slots)	78.76
$k$ -means multi-prototype ( $m{=}5$ )	80.10
Typicality-filtered references (top-12)	78.00
Prototype + cosine score fusion	77.71
Shrinkage Mahalanobis ( $\lambda{=}0.1$ )	89.43
Shrinkage Mahalanobis ( $\lambda{=}0.02$ , default)	89.90

TABLE VII: Readout Ablation on UCM Closed-Set With 16 References Per Class (OA, %). Geometric Correction Contributes About

+11

Points; Every Cache-Construction and Score-Fusion Variant Falls Short of the Naive Prototype—the Gain Comes From Metric Geometry, Not Cache Content

Remote Sensing Scene Classification

See all 11 papers

RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models
Oct 6, 2026Dongchen Si, Di Wang, Mingzhen Xu +3Remote Sensing Image Understanding Efficient Multimodal Inference
School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing
Mar 3, 2026Yi Liu, Jing Zhang, Di Wang +3Remote Sensing VQA VLM Hallucination Mitigation
School of Computer Science, Wuhan University, China · Zhongguancun Academy, China · School of Computer Science, Chongqing University, China
Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting
Jun 15, 2026Eirini Baltzi, Dionysis Christopoulos, Sotiris Spanos +2Remote Sensing Image Understanding Zero-Shot Image Classification
Remote Sensing Lab, National Technical University of Athens Athens, Greece · Department of Engineering and Sciences, Universitas Mercatorum Rome, Italy

Oct 6, 2026J/K move · Enter open · S save

RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models

Dongchen Si, Di Wang, Mingzhen Xu, Jing Zhang+2

School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

Remote Sensing Image Understanding Efficient Multimodal Inference Multimodal Large Language Models Multimodal Classification Remote Sensing Scene Classification

Open paper AskPDF arXiv

Remote sensing scene classification is a fundamental task in Earth observation and geospatial analysis. Existing approaches mainly follow three paradigms: task-specific visual classification, vision-language similarity matching, and autoregressive multimodal generation. However, visual classifiers rely on predefined label spaces, CLIP-based methods perform recognition through static image-text alignment, and multimodal large language models (MLLMs) introduce unnecessary token-level generation for classification tasks with explicit candidate categories. To address these limitations, we propose RSJEV, a one-pass multimodal decision framework for remote sensing scene classification. Unlike conventional MLLMs that formulate classification as autoregressive text generation, RSJEV reformulates scene classification as a candidate-conditioned multimodal discriminative decision process, where visual representations, task instructions, and candidate category semantics are jointly modeled. Specifically, we introduce a OnePass Decider that extracts multimodal decision states and directly estimates category probabilities within the candidate category space, eliminating autoregressive decoding while preserving vision-language interactions. Extensive experiments on three widely used remote sensing scene classification benchmarks, including UC Merced, AID, and NWPU-RESISC45, demonstrate that RSJEV achieves superior classification performance compared with representative CNN-, Transformer-, Mamba-, CLIP-, and MLLM-based methods. Moreover, RSJEV significantly reduces inference costs and achieves a better accuracy-efficiency trade-off with only a compact 0.8B-parameter model. These results demonstrate the effectiveness of state-conditioned multimodal decision making for efficient remote sensing image understanding. The code will be available at https://github.com/Dongtcs/RSJEV.

Abstract

Remote Sensing Scene Classification

Explore similar work

RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models

Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing

Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting