Quantifying and extending the coverage of spatial categorization data sets

Wanchun Li; Alexandra Carstensen; Yang Xu; Terry Regier; Charles Kemp

cs.CLMar 10, 2026

Quantifying and extending the coverage of spatial categorization data sets

Authors: Wanchun Li, Alexandra Carstensen, Yang Xu, Terry Regier, Charles Kemp

Organizations: Faculty of Engineering and Information Technology, The University of Melbourne · Psychology, Arizona State University · Department of Computer Science, Cognitive Science Program, University of Toronto · Department of Linguistics, University of California, Berkeley ·

Abstract

Variation in spatial categorization across languages is often studied by eliciting human labels for the relations depicted in a set of scenes known as the Topological Relations Picture Series (TRPS). We demonstrate that labels generated by large language models (LLMs) align relatively well with human labels, and show how LLM-generated labels can help to decide which scenes and languages to add to existing spatial data sets. To illustrate our approach we extend the TRPS by adding 42 new scenes, and show that this extension achieves better coverage of the space of possible scenes than two previous extensions of the TRPS. Our results provide a foundation for scaling towards spatial data sets with dozens of languages and hundreds of scenes.

Figures & tables

Figure 1: (a) Spatial relations stimulus sets. Each stimulus shows the relation between a focal object (shown in gold, or marked with an arrow) and a background object. Some of the original images have been cropped for this figure. (i) The Topological Relations Picture Series (TRPS; Bowerman and Pederson, 1992) includes 71 scenes that were chosen to explore the boundaries of terms for on and in relations. (ii) The Zhang set ( Zhang, 2013 ) includes 63 scenes illustrating configurations that are absent from the TRPS but relevant to the expression of on and in in Chinese. 1 1 1 Zhang (2013) mentions that she developed 64 new pictures but 63 were included in the set that she kindly shared with us. The examples here include “girls in line,” “shadow on wall” and “food on plate.” (iii) The LJSP set ( Landau et al., 2017 ) includes 44 scenes that were designed to span six subtypes of containment and five subtypes of support. The examples here include “paper in box,” “cup in sleeve,” and “cup on counter.” (iv) The LCXRK set developed by us includes 42 images designed to illustrate spatial terms in English and Chinese that are not represented by the TRPS, and to include negations and reversals of TRPS scenes. The examples here include “cat among flowers.” b) Extending a cross-linguistic data set using LLMs. (i) The original human labels are organized into a dark gray matrix of languages (rows) by scenes (columns). (ii) We add candidate languages (new rows) and candidate scenes (new columns) and generate LLM labels for all new combinations of languages and scenes. (iii) A coverage measure based on LLM labels is used to prioritize languages and scenes for subsequent human labeling.

Figure 2: Evaluation of LLM labels against data collected by (a) Carstensen et al. (2019) and (b) Xu and Kemp (2010) . The binary score for an image label is 1 if a human provided the same label, and the graded score is the proportion of humans who provided the label. In (a), black points show the maximum possible graded score for each language. In (b), black points show the average result when a single human from the Carstensen et al. (2019) data is scored with respect to a different speaker of the same language, and error bars show the 95% interquantile range.

Stimulus set	Size	Coverage score	95% CI
TRPS	71	0.914	[0.82, 0.95]
TRPS + Zhang	134	0.918	[0.85, 0.95]
TRPS + LJSP	115	0.918	[0.87,0.95]
TRPS + LCXRK	113	0.964	[0.96, 0.995]

Table 1: Coverage scores achieved by four stimulus sets relative to a universe

U

of 220 scenes.

Figure 3: MDS visualization of a space that includes scenes from all stimulus sets considered in this paper. The four panels show the coverage of this space achieved by (a) the TRPS (b) the Zhang set (c) the LJSP set and (d) our new stimulus set. Points labeled in panels (a) and (d) correspond to images shown in Figures 1 a.i and 1 a.iv.

Figure 4: MDS visualization of the similarity between spatial systems from the languages represented in Figure 2 . Languages appearing in the Carstensen et al. (2019) data set are shown in red.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
Oct 13, 2026Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh +2Multimodal Spatial Reasoning Multimodal Large Language Models
Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada. · Alberta Machine Intelligence Institute (Amii) Fellow, CIFAR Fellow.
From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
Jun 3, 2026Chen Chu, Bita Azarijoo, Li Xiong +2Spatial Reasoning Benchmarks Multimodal Large Language Models
University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Spatial Reasoning Multilingual QA
Data and Web Science Group, University of Mannheim, Germany

Oct 13, 2026J/K move · Enter open · S save

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh, Sheila Schoepp+1

Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada. · Alberta Machine Intelligence Institute (Amii) Fellow, CIFAR Fellow.

Multimodal Spatial Reasoning Multimodal Large Language Models Spatial Reasoning Benchmarks Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks. However, MLLMs suffer from limitations in spatial reasoning and perception. Post-training techniques have shown promise for improving these capabilities; yet, there are limited data-driven surveys of such approaches. In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the datasets used. We investigate themes of dataset construction, data modality and annotation, and evaluation of the resulting models. Based on these themes, we categorize research into three groups: dataset design, dataset construction and evaluation. We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop. We also observe substantial variation in evaluation approaches, including different scoring methods, tolerance metrics, and benchmark types. Finally, we identify opportunities for future research in unified dynamic annotation, efficient spatial reasoning, standardization of evaluation, expanded spatial range, and human-centered spatial interactions.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning