Seeing Through Extreme Visual Sparsity: Surface Understanding from a Single Random Visual Patch

Sindhuja Penchala; Sudip Mittal; Noorbakhsh Amiri Golilarz

cs.CVAug 30, 2026

Seeing Through Extreme Visual Sparsity: Surface Understanding from a Single Random Visual Patch

Authors: Sindhuja Penchala, Sudip Mittal, Noorbakhsh Amiri Golilarz

Organizations: Department of Computer Science, The University of Alabama, Tuscaloosa, AL, USA

Abstract

Surface material recognition from incomplete visual observations remains a challenging problem in robotic perception and environmental understanding. This paper discusses Sparse Surface Understanding Framework (SSUF), a unified dual-task learning framework that adapts four pretrained architectures-Convolutional Autoencoder (ConvAE), Vision Transformer (ViT), Swin Transformer, and Masked Autoencoder (MAE) for si-multaneous surface reconstruction and material classification. Experiments were conducted on the Touch-and-Go dataset using a sparse observation protocol in which only 10% of the original image remained visible while the remaining regions were masked. To enable a fair comparison, reconstruction-oriented models were extended with classification heads, whereas classification- oriented models were augmented with reconstruction decoders. The resulting architectures were assessed using reconstruction quality, classification performance, model complexity, and in-ference efficiency metrics. Experimental results revealed distinct strengths across the models. Swin Transformer achieved the best classification performance with an accuracy of 89.21%, an F1-score of 0.8922, and a ROC-AUC of 0.9813. In contrast, MAE produced the highest reconstruction scores among evaluated models, with a PSNR of 16.06 dB and an SSIM of 0.4501, while ViT provided the best overall balance between reconstruction and classification performance. Furthermore, all models achieved real-time inference, requiring less than 5 ms per image. Over-all, the results show that pretrained architectures can support material recognition under severe visual sparsity, while accurate image reconstruction remains challenging.

Figures & tables

Component	ConvAE	ViT	Swin	MAE
Primary Objective	Reconstruction	Classification	Classification	Self-Supervised Reconstruction
Encoder Type	ResNet50	Transformer	Hierarchical Transformer	Transformer
Patch Embedding	×	✓	✓	✓
Attention Mechanism	×	Global	Shifted Window	Global
Original Decoder	✓	×	×	✓
Original Classifier	×	✓	✓	×

TABLE I : Comparison of the evaluated architectures and the modifications introduced in this study.

Fig. 1 : Overview of the Sparse Surface Understanding Framework (SSUF) for simultaneous surface reconstruction and material classification from sparse visual observations. Images from the Touch-and-Go dataset are converted into sparse inputs by retaining only a random continuous patch corresponding to 10% of the original image area. The resulting observations are processed by four adapted architectures: SSUF-ConvAE, SSUF-ViT, SSUF-Swin, and SSUF-MAE. Learned feature representations are jointly utilized for image reconstruction and classification of four surface material categories (Grass, Concrete, Wood, and Rock). Task-specific modules introduced to enable dual-task learning are highlighted in red (reconstruction decoders) and blue (classification heads).

Model	PSNR (dB) $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	MSE $\downarrow$
SSUF-ConvAE	15.91	0.4380	0.13123	0.03056
SSUF-ViT	16.03	0.4269	0.12861	0.02940
SSUF-Swin	15.52	0.4255	0.13941	0.03307
SSUF-MAE	16.06	0.4501	0.12924	0.03009

TABLE II : Reconstruction performance of the evaluated SSUF architectures on the Touch-and-Go dataset using only 10% visible image content. Best results are shown in bold.

Model	Acc. (%) $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
SSUF-ConvAE	83.90	0.8405	0.8390	0.8385
SSUF-ViT	86.30	0.8640	0.8630	0.8630
SSUF-Swin	89.21	0.8930	0.8921	0.8922
SSUF-MAE	76.71	0.7650	0.7671	0.7651

TABLE III : Classification performance of the evaluated SSUF architectures under the 10% visible-patch setting. Precision, recall, and F1-score are weighted averages. Best results are shown in bold.

Model	Class	Precision	Recall	F1-Score
SSUF-ConvAE	Grass	0.9091	0.9375	0.9231
	Concrete	0.8596	0.7656	0.8099
	Wood	0.8390	0.8866	0.8622
	Rock	0.7431	0.7941	0.7678
SSUF-ViT	Grass	0.9381	0.9479	0.9430
	Concrete	0.8715	0.8125	0.8410

TABLE IV : Class-wise classification performance of the evaluated SSUF architectures on the Touch-and-Go test set under the 10% visible patch setting. Precision, recall, and F1-score are reported separately for each material category.

Model	Total Parameters	Trainable Parameters	Non-Trainable Parameters
SSUF-ConvAE	38,923,847	15,336,711	23,587,136
SSUF-ViT	92,028,039	5,635,335	86,392,704
SSUF-Swin	32,300,049	4,718,375	27,581,674
SSUF-MAE	109,822,471	24,020,295	85,802,176

TABLE V : Model complexity comparison of the evaluated SSUF architectures. Lower values indicate lower model complexity.

Model	Total Time (s) $\downarrow$	Avg. Time/Img (s) $\downarrow$	Avg. Time/Img (ms) $\downarrow$
SSUF-ConvAE	2.329	0.003988	3.988
SSUF-ViT	2.784	0.004766	4.766
SSUF-Swin	2.257	0.003865	3.865
SSUF-MAE	1.481	0.002536	2.536

TABLE VI : Inference time comparison of SSUF-ConvAE, SSUF-ViT, SSUF-Swin, and SSUF-MAE on the Touch-and-Go test set.

Panel of Figure 8 — Fig. 2 : ROC curves (top row) and confusion matrices (bottom row) for the evaluated SSUF architectures on the Touch-and-Go dataset under the 10% visible-patch setting. SSUF-Swin achieved the strongest overall classification performance, followed by SSUF-ViT, SSUF-ConvAE, and SSUF-MAE.

Robotic Perception

Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.

See all 320 papers

Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox
May 22, 2026Allan Kazakov, Duygu Cakir, Hilal Kurt İrfanoğlu +1Image Segmentation Vision Transformer
Bahcesehir University, Istanbul, Turkey · Poder Bilişim Teknolojileri Sanayi ve Ticaret A.Ş., Istanbul, Turkey · Galatasaray University, Istanbul, Turkey
Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism
Mar 21, 2026Qinghui Chen, Zekai Zhang, Zaigui Zhang +5Industrial Anomaly Detection Industrial Inspection
School of Control Science and Engineering, Shandong University, Jinan, China · Laoshan Laboratory, Qingdao, China · Jinan Inspur Data Technology Co., Ltd, Jinan, China +2
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
Aug 12, 2026Yan Di, Chengxi Li, Yaoxing Wang +73D Reconstruction Sparse-View 3D Reconstruction
HIT (Shenzhen) · Tsinghua University · NWPU +2

May 22, 2026J/K move · Enter open · S save

Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox

Allan Kazakov, Duygu Cakir, Hilal Kurt İrfanoğlu, Yavuz İrfanoğlu

Bahcesehir University, Istanbul, Turkey · Poder Bilişim Teknolojileri Sanayi ve Ticaret A.Ş., Istanbul, Turkey · Galatasaray University, Istanbul, Turkey

Image Segmentation Vision Transformer OOD Generalization Semantic Segmentation

Open paper AskPDF arXiv

Material segmentation, the pixel-wise classification of physical surface properties, remains a challenging problem in computer vision, requiring physicochemical understanding distinct from object-centric parsing. Despite the introduction of the rigorous Apple Dense Material Segmentation (DMS) dataset, the benchmark has suffered from attrition and stagnation, increasingly overshadowed by geometry-biased foundation models. In this paper, we revive the Apple-DMS benchmark to establish a modern Vision Transformer baseline. We conduct an exhaustive evaluation of SegFormer and Mask2Former architectures, revealing that standard training paradigms fail on amorphous texture fields due to high-variance gradients. To address this, we introduce a stabilized training recipe featuring High-Fidelity Logit Projection, Query Entropy Regularization, and a domain-specific, physics-compliant augmentation pipeline. Our optimized SegFormer-B5 achieves a new State-of-the-Art (SOTA) of 0.4572 mIoU on the original dataset split, significantly surpassing the prior convolutional baseline. Furthermore, we identify a critical "Generalization Paradox": while re-partitioning the dataset into a data-rich 80/10/10 split inflates the metric to 0.5276 mIoU, expert qualitative analysis reveals this induces distributional homogenization, severely degrading real-world, out-of-distribution performance. By releasing our recovered dataset index and robust training framework, we demonstrate that material perception is far from solved and urge the community to leverage the rigorous original split to drive genuine progress in physically grounded artificial intelligence.

Abstract

Robotic Perception

Explore similar work

Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox

Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction