cs.CVFeb 19, 2026
SaveHuman-level 3D shape perception emerges from multi-view learning
Organizations: University of California, Berkeley
Abstract
Humans can infer the three-dimensional structure of objects from two-dimensional visual inputs. Modeling this ability has been a longstanding goal for the science and engineering of visual intelligence, yet decades of computational methods have fallen short of human performance. Here we develop a modeling framework that predicts human 3D shape inferences for arbitrary objects, directly from experimental stimuli. We achieve this with a novel class of neural networks trained using a visual-spatial objective over naturalistic sensory data; given a set of images taken from different locations within a natural scene, these models learn to predict spatial information related to these images, such as camera location and visual depth, without relying on any object-related inductive biases. Notably, these visual-spatial signals are analogous to sensory cues readily available to humans. We design a zero-shot evaluation approach to determine the performance of these 'multi-view' models on a well established 3D perception task, then compare model and human behavior. Our modeling framework is the first to match human accuracy on 3D shape inferences, even without task-specific training or fine-tuning. Remarkably, independent readouts of model responses predict fine-grained measures of human behavior, including error patterns and reaction times, revealing a natural correspondence between model dynamics and human perception. Taken together, our findings indicate that human-level 3D perception can emerge from a simple, scalable learning objective over naturalistic visual-spatial data. Code, images, and human data needed to reproduce all analyses can be found at https://tzler.github.io/human_multiview/
Figures & tables
Figure 1: Schematic of multi-view model training approach and 3D perceptual testing protocol. We evaluate a novel class of multi-view transformers (VGGT-1B, Wang et al., 2025a ), which is trained on large-scale, multi-view, naturalistic scene data. During training, VGGT receives sets of images depicting the same scene from different viewpoints (top left) and must learn to predict the relative depth, camera position, and aleatoric uncertainty associated with these images (bottom left). These multi-modal signals are analogous to information that is available to humans through stereo vision and proprioception. Notably, VGGT uses a general transformer architecture with no hand-coded geometric priors: any understanding of 3D structure emerges from learning the predictive relationship between images and these multi-modal cues. To evaluate these multi-view transformers alongside human observers on a 3D perception task, we use a standard experimental design from the cognitive sciences: a concurrent visual discrimination (‘oddity’) task. This design requires zero-shot visual inference about object shape: given two images of an object from different viewpoints (A and A ′ ), and another image of a different object (B), the task is to determine which image contains the non-matching object (e.g., right). We evaluate humans and models on diverse object types, including real-world objects (e.g., chairs, tables) as well as procedurally generated abstract shapes (i.e., ‘nonsense’ objects).
Figure 2: Evaluation approach used to estimate model performance on each trial. We develop a series of zero-shot evaluation metrics to determine the behavior of multi-view models on human 3D perceptual tasks. To estimate accuracy we leverage the model’s internal estimate of aleatoric uncertainty (top): we encode all pairwise combinations of images on a trial, extract model uncertainty estimates for each pair, average, then select the oddity to be the object with the lowest paired confidence scores. We compare this model-selected non-match with the ground truth, resulting in a single binary (correct/incorrect) outcome. Next, we compute the margin between matching/non-matching objects to determine the model’s confidence for this decision ( top right). We visualize the norm of model responses across several layers, going from early to final layers (left to right). Note that the activation patterns for matching (AA ′ ) objects is visibly distinct from those of the non-matching (AB/BA ′ ) objects in the final layers. When conducting a similarity analysis across all layers (bottom), for each encoded pair of images, we find that the features of matching objects (AA ′ ) become more correlated (orange, bottom), while non-matching objects (AB/BA ′ ) become less correlated (black/purple, bottom). We describe the earliest layer where the non-matching object can be identified as the model ‘solution layer’.
Figure 3: Multi-view models match human 3D perception accuracy, error patterns, and reaction times. When comparing normalized accuracy of humans and models across all conditions of this 3D perception benchmark (left), VGGT matches human performance, while both humans and multi-view models significantly outperform standard vision models like DINOv2. Critically, VGGT’s human-level performance does not depend on task-specific training or fine-tuning; the model used only its pre-trained representations learned from multi-view prediction. These findings demonstrate that multi-view learning is sufficient to achieve human-level 3D object perception. Beyond average accuracy, we find that model confidence (i.e., the margin between matching and non-matching objects in each trial) is significantly correlated with human choice behaviors. This indicates that the aleatoric uncertainty used during training provides a natural analogue for human perceptual judgments. Finally, we observe a clear correspondence between model solution layer and human reaction time; as the number of layers required to solve this perceptual task increases, so too does human reaction time needed for correct responses. These data reveal an emergent correspondence between model dynamics and human perception.
Figure 4: Investigating between-image activations reveals location-based object correspondence. How does this multi-view model represent the similarity or difference between objects? To address this question we provide a qualitative visualization of the information present in intermediate model layers. For an example trial (e.g., abstract image in the top half; chair in the bottom half), we encoded image pairs A, A ′ and A, B separately, then extract the cross-image block of the attention matrix. Here we manually select keypoints in the reference image, A (different color dots on A, far left), and identify the corresponding patch token in the target images A ′ and B. We then retrieve the attention distribution over all the target patch tokens. In intermediate model layers (e.g., here we visual attention from layer 15) we find that different query locations from the reference image A elicit distinct attention patterns across points and across target images A ′ and B. Concretely, it appears that each query location on A elicits a pattern of attention in A ′ that correspondence to the same location on the object, albeit a different location in xyz coordinates. This qualitative analysis indicates that the model can represent the object-object similarity via the correspondence between spatial locations on each object.
Figure S1: Standard vision models fail on 3D shape inferences. We evaluate the ability of large vision models on the MOCHI benchmark ( Bonnen et al., 2024 ) using a cosine distance metric. For each trial, the image with the lowest average cosine distance to all other images is selected as the oddity. We compare to ground truth and average across all trials. Evaluating DINOv2, CLIP, and MAE models at multiple scales, we find that no model approaches human-level performance. Even the best-performing model (DINOv2-giant) performs at half the accuracy of humans, indicating that the geometric structure of these feature spaces does not naturally separate 3D object identity.
Figure S2: 3D shape performance of all vision models evaluated. We begin our analysis by evaluating the performance of state of the art multi-view transformers on the MOCHI ( Bonnen et al., 2024 ) benchmark. This includes VGGT ( Wang et al., 2025a ), Pi3 ( Wang et al., 2025b ), MAST3R ( Leroy et al., 2024 ), DUST3R ( Wang et al., 2024 ), as well as a zero-shot evaluation of DINOv2 ( Oquab et al., 2023 ). Because VGGT is the only model that achieves human-level performance on this 3D shape inference benchmark we restrict all subsequent analyses to this model.
Figure S3: Visualizing model performance as a function of model size.
Figure S4: Condition-level performance of humans and vision models. We visualize the performance of humans (orange), multi-view vision models (red), and image models (grey) for each individual condition on this benchmark. These conditions contain both abstract (e.g., ’greebles (high similarity)’; far left) and real-world objects (e.g., ’familiar objects (low similarity)’ far left) rendered in color and greyscale. Across these abstract and real-world conditions there is a close correspondence between multi-view models and human observers, while both substantially outperform image models. We also visualize a single image from a trial in each of these conditions which are, from left to right: ’greebles (high similarity)’, ’familiar objects (high similar)’, ’telephone’, ’car’, ’table’, ’display’, ’loudspeaker’, ’watercraft’, ’familiar objects (low similarity)’, ’cabinet’, ’lamp’, ’sofa’, ’airplane’, ’bench’, ’chair’, ’greebles (low similarity)’, and ’abstract objects’.
Figure S5: Comparing the relationship between humans and vision models per condition. Here we visualize the correlation between human and model performance per condition (i.e., averaging across all trials within a given condition) for both VGGT (left) and DINOv2 (right). Human-VGGT performance is largely on diagonal, indicating that VGGT is predicting human-level performance across conditions. In contrast, Human-DINOv2 performance is significantly above the diagonal, indicating that humans are significantly outperforming the DINOv2 across conditions.
Figure S6: Direct relationship between model confidence and human accuracy/RT. Because human accuracy and reaction time are correlated, the ability to predict one measure typically implies the ability to predict the other. Here we visualize the relationship between model confidence and both human error patterns (left) and reaction time (right), demonstrating this clear correspondence. It is in part because of this circularity that we developed an independent model readout to evaluate the model’s ability to predict reaction time data; model ‘solution time’ is independent of confidence measures.
Figure S7: The relationship between accuracy and reaction time, ordered by model solution time. To independently corroborate the utility of model solution time as a predictor of human reaction time, here we use the binned trials identified by our solution layer metric in order to visualize the relationship between human reaction time (x axis, left) and human accuracy (y axis, left): there is a clear relationship between RT and accuracy when grouping trials according to model solution layer, corroborating the validity of this measure. We also plot the relationship between model solution time (x axis, right) and model confidence (y axis, right).
Figure S8: Evaluating the performance of VGGT across multiple layers/readouts . In addition to the confidence-based performance metric, we also develop layer-wise performance metrics to observe how performance evolves across the model’s layer-by-layer processes using the max (green), mean (yellow) and global average pooling of the layer features. For each metric we simple determine, at each layer, whether A and A ′ are more similar in representational space than A and B.
Figure S9: Pattern of global attention across different objects in layer zero.
Figure S10: Pattern of global attention across different objects in layer 15.
More in this topic
Multi-View Learning
Multi-view learning combines complementary representations of the same objects to improve prediction, clustering, and robust data integration.
See all 182 papers