Oct 11, 2026
Scene classification of remote sensing imagery with multimodal large language models (MLLMs) is formulated as a single-forward-pass decision problem: candidate class names are lettered options in the prompt, a registered answer slot is conditioned on the prompt, and classification is read out by scoring letter tokens with the language-modeling head. We show that although this letter binding is highly effective within its training vocabulary (97.71% OA on UCM), it behaves erratically out of vocabulary: zero-adaptation accuracy across four cross-dataset directions fluctuates between 7.35% and 96.40%, and under out-of-vocabulary weights its confidence fails inversely on open-set evaluation, assigning higher confidence to unseen than to seen classes (AUROC 0.38), a failure we term silent hallucination. We retain the slot representation and replace the readout with RSAJEV, a training-free abstaining prototype memory that performs retrieval-based prototype matching: K reference images per class are cached as prototypes via their slot representations under the same prompt, and classification becomes nearest-neighbor retrieval over the prototypes, refined by a shrinkage-Mahalanobis geometric correction. This readout is mechanistically isomorphic to retrieval-augmented generation (RAG) but operates on discriminative decisions; it is the first retrieval-based readout for MLLM slot representations. The retrieval readout reaches 89.43% on the UCM cross-dataset gate with 16 references per class and zero gradient updates, and 96.90% on AID. Its ID-ness scores attain 0.91-0.99 AUROC on open-set evaluation over two datasets, exceeding symbolic confidence (0.38-0.96) in all four open-set settings, restoring a calibratable abstention channel and supporting class addition/removal at zero extra inference cost via offline rescoring of the cached matrix.
Man Chen, Antai Chen, Shuai Xu +5
School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China, and also with the School of Electrical Automation and Information Engineering, Tianjin University, Tianjin 300072, China, and the State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Huainan 232001, Anhui, China · School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China · School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, Anhui, China, and also with the State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Huainan 232001, Anhui, China +1