GViT: Representing Images as Gaussians for Visual Recognition

Jefferson Hernandez; Ruozhen He; Guha Balakrishnan; Alexander C. Berg; Vicente Ordonez

cs.CVJun 30, 2025

GViT: Representing Images as Gaussians for Visual Recognition

Authors: Jefferson Hernandez, Ruozhen He, Guha Balakrishnan, Alexander C. Berg, Vicente Ordonez

Organizations: Rice University · University of California, Irvine

Abstract

We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.

Figures & tables

Figure 1: GViT replaces fixed patch tokens with a compact set of predicted 2D Gaussians. The encoder fits these primitives in one forward pass, making Gaussian tokens practical as the input interface of a ViT classifier.

Figure 2: Overview of GViT. A Denoising Gaussian Encoder predicts 2D Gaussians from image patches. The Gaussians are rendered for reconstruction and passed as tokens to a ViT classifier. Classifier-gradient guidance shapes the primitives during training. Algorithm 1 gives the training procedure.

Panel of Figure 3 — Figure 3: Visualization of Gaussians for an image under three regimes. (a) Original image, (b) Gaussians fitted via SGD reconstruction-only, (c) End-to-end predicted Gaussians without classifier guidance, and (d) Gaussians steered by classifier-gradient guidance. We use an ellipse sized to 2 standard deviations for each Gaussian to visualize.

Model	Input		Fine-grained Benchmarks
Model	Input	IN-1K	AVG	C10	C100	CUB	SUN	Flwr	Pets	DTD
ViT-B/32 ( Dosovitskiy et al., 2021 )	Patches	71.4	85.6	-	87.6	92.6	73.8	94.4	92.2	72.7
ViT-B/16 ( Dosovitskiy et al., 2021 )	Patches	77.9	92.1	98.1	87.1	-	-	89.5	93.8	-
ViT-B/16 ( Steiner et al., 2022 )	Patches	78.7	89.0	98.1	87.8	93.0	75.3	97.6	93.2	77.8
GViT-B (no guid)	Gaussians	73.6	84.9	96.6	85.7	88.3	67.1	92.3	92.2	71.9
GViT-B (w/guid)	Gaussians	76.9	87.1	98.6	87.7	91.2	68.3	96.0	94.5	73.4

Table 1: Top-1 classification accuracies (%) on ImageNet-1K (IN-1K) and seven transfer datasets for Gaussian ViT variants and literature ViT baselines. C10, C100, SUN, and Flwr denote CIFAR-10, CIFAR-100, SUN397, and Flowers. AVG is the mean of the positive values in the seven transfer columns, excluding IN-1K and missing entries; dashes denote unreported results. Literature baselines use their source protocols ( Dosovitskiy et al., 2021 ; Steiner et al., 2022 ) .

		ImageNet-Seg			Perturbation
Method	Model	Pixel Acc	mIoU	mAP	Neg ( $\uparrow$ )	Pos ( $\downarrow$ )
	ViT	–	–	–	–	–
$\det(\Sigma)$	GViT	55.45	38.65	66.75	36.4	41.3
Raw Attn	ViT	67.84	46.37	80.24	50.00	23.70
Raw Attn	GViT	61.09	42.89	75.68	42.0	29.9
Rollout	ViT	73.54	55.42	84.76	51.3	20.4

Table 2: Explainability performance on ImageNet-Seg and ImageNet-Val under perturbations. Perturbation deletes classifier input tokens on a 10–70% schedule. ViT uses black input patches. GViT uses a position-only null: the center remains, covariance is fixed and isotropic, and color and opacity are zero. ViT segmentation numbers are from Chefer et al. (2021) ; perturbation results use the stated token-deletion protocol. Bold values mark the best probe within each model for each metric.

Figure 4: Gradient-guided Gaussians exhibit a natural interpretability component. (a) Original input images. (b) Determinant of the covariance ( $\det(\Sigma)$ ), which clusters Gaussian primitives in class-salient regions. (c) GradCam over the learned Gaussians. (d) Rollout of the attentions.

Model	Foreg. Segm.	Obj. Det.
BEiT	0.79	0.32
MAE	4.28	1.98
MAE-VQGAN	5.26	3.21
VQGAN	9.13	4.99
Copy	14.91	13.50
GMAE	18.16	20.26

Table 3: Geometric foreground diagnostic on PASCAL VOC validation, following GMAE ( Rajasegaran et al., 2025 ) (IoU; higher is better). GViT uses geometry learned through supervised classifier guidance. Its threshold is selected from the VOC training distribution.

Table 4: Design choices. Mini-ImageNet top-1 accuracy (%), reported as mean

\pm

sample SD over three seeds per setting. Each sweep varies only the named setting; all other parameters use the selected defaults in the paper.

Panel of Figure 5 — Figure 5: Token budget and resolution. (a) Sensitivity to the number of Gaussians. (b) Accuracy across input resolutions for the resolution sweep. The matched-recipe comparisons and inference costs are reported in the text.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table C.1 : Implementation details. Left: ImageNet-1k pre-training settings. Right: fine-tuning hyper-parameters for downstream transfer.

Method	Representation	Res.	Token sequence	Params (M)	GMAC	Top-1
DeiT-Ti/16 ( Touvron et al., 2021 )	RGB patches	224	196	5.72	1.25	72.2
DeiT-S/16	RGB patches	224	196	22.05	4.60	79.8
DeiT-B/16	RGB patches	224	196	86.57	17.56	81.8
ByteFormer-Ti ( Horton et al., 2024 )	JPEG bytes; windowed attention	224	Variable; $\approx 6{,}069$ initially, then halving	16.22	$\approx 11.57$	64.9
JPEG-Ti ( Park and Johnson, 2023 )	Grouped JPEG DCT	224	196	5.64	1.23	75.1
JPEG-S	Grouped JPEG DCT	224	196	21.98	4.55	76.5

Appendix

Table D.1 : Classifier input interfaces and complete-system comparisons. ImageNet-1K top-1 accuracy (%). Methods differ in capacity, resolution, token budget, and training. Token counts exclude CLS. GMAC counts one multiply-accumulate once; costs cover the stated model operations, not a uniform end-to-end image pipeline. Dashes indicate unavailable counts. Panel (b) separates tokenizer parameters from the remaining model.

Model	Input		Fine-grained Benchmarks
Model	Input	IN-1K	AVG	C10	C100	CUB	SUN	Flwr	Pets	DTD
ViT-S/32 ( Dosovitskiy et al., 2021 )	Patches	69.2	84.6	-	86.4	92.7	72.9	93.6	91.2	70.7
ViT-S/16 ( Steiner et al., 2022 )	Patches	77.5	87.8	97.9	86.9	93.1	74.3	95.7	93.8	72.8
GViT-S (no guid)	Gaussians	68.3	81.3	92.3	79.4	86.3	63.9	89.3	86.4	71.4
GViT-S (w/guid)	Gaussians	72.4	83.8	95.3	80.2	88.5	65.9	94.8	89.0	72.9

Appendix

Table D.2 : Top-1 classification accuracies (%) on ImageNet-1K (IN-1K) and seven transfer datasets for Small classifiers. Dataset abbreviations and AVG follow Table 1 : AVG uses only positive values in the transfer columns and excludes IN-1K and missing entries. Literature ViT-S baselines use their source protocols ( Dosovitskiy et al., 2021 ; Steiner et al., 2022 ) .

Table E.1 : Shortcut diagnostics. (a) Comparable GS/IMG rankings across training settings argue against hidden information leakage. (b) GViT is substantially more shape-biased than ResNet-50 and close to patch-based ViTs under the Geirhos et al. (2018) cue-conflict test.

Figure F.1 : Gradient-guided Gaussians exhibit a natural interpretability component. (a) Original input images. (b) Determinant of the covariance ( $\det(\Sigma)$ ), which clusters Gaussian primitives in class-salient regions. (c) GradCAM over the learned Gaussians. (d) Rollout of the attentions.

Image Classification

Image classification assigns semantic labels to images, enabling recognition of objects, scenes, conditions, and fine-grained categories.

See all 398 papers

$\text{VG}^2$ GT: Voxel-Gaussian Splatting Visual Geometry Grounded Transformer
Jun 1, 2026Yibin Zhao, Yihan Pan, Jun Nan +33D Gaussian Splatting Novel View Synthesis
East China University of Science and Technology, 130 Meilong Road, Shanghai, 200237, China · Shanghai Open University, 288 Guoshun Road, 200433, Shanghai, China · Shanghai Xiaoyuan Innovation Center, 599 Xingmei Road, 200237, Shanghai, China
Subtoken Vision Transformer for Fine-grained Recognition
Jul 10, 2026Jie Zhu, Ivy Zhang, Minchul Kim +1Vision Transformer Visual Tokenization
Michigan State University · Cranbrook Kingswood School · University of North Carolina at Chapel Hill
Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers
Jul 1, 2026Cong Liu, Xiaofang Li, Simon X. YangVision Transformer
School of Engineering, Yancheng Institute of Technology, No. 1 Xiwang Middle Avenue, Yancheng, 224051, Jiangsu, PR China · School of Computer Information Engineering, Changzhou Institute of Technology, No. 666 Liaohe Road, Xinbei District, Changzhou, 213032, Jiangsu, PR China · Advanced Robotics and Intelligent Systems Laboratory, School of Engineering, University of Guelph, 50 Stone Road East, Guelph, N1G 2W1, Ontario, Canada

Jun 1, 2026J/K move · Enter open · S save

$\text{VG}^2$ GT: Voxel-Gaussian Splatting Visual Geometry Grounded Transformer

Yibin Zhao, Yihan Pan, Jun Nan, Wenli Yang+2

East China University of Science and Technology, 130 Meilong Road, Shanghai, 200237, China · Shanghai Open University, 288 Guoshun Road, 200433, Shanghai, China · Shanghai Xiaoyuan Innovation Center, 599 Xingmei Road, 200237, Shanghai, China

3D Gaussian Splatting Novel View Synthesis Geometric Representation Learning Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Gaussian splatting has shown strong potential for 3D reconstruction and novel view synthesis. However, most existing methods require accurate camera parameters and per-scene optimization, while feed-forward methods with pixel-aligned Gaussian primitives often suffer from artifacts and non-uniform primitives. In this paper, we propose

\text{VG}^2

GT, a Voxel-Gaussian Splatting Visual Geometry-Grounded Transformer.

\text{VG}^2

GT leverages a frozen pretrained visual foundation model (VFM), incorporates a multi-scale differentiable voxel module to enhance geometric understanding, and directly splits and regresses Gaussian primitive parameters from voxel features. During training, depth maps are supervised through stochastic solid volume rendering, enabling geometrically accurate Gaussian scene reconstruction while keeping the visual foundation model fully frozen. This design enables

\text{VG}^2

GT to be seamlessly plugged into any patch-feature-based VFM, while substantially reducing the required training cost.

\text{VG}^2

GT outperforms current state-of-the-art methods on widely used DTU, Replica, TAT, and ScanNet datasets.

Abstract

Image Classification

Explore similar work

VG2\text{VG}^2VG2GT: Voxel-Gaussian Splatting Visual Geometry Grounded Transformer

Subtoken Vision Transformer for Fine-grained Recognition

Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers

$\text{VG}^2$ GT: Voxel-Gaussian Splatting Visual Geometry Grounded Transformer