Benchmarking PhD-Level Coding in 3D Geometric Computer Vision

Wenyi Li; Renkai Luo; Yue Yu; Huan-ang Gao; Mingju Gao; Li Yuan; Chaoyou Fu; Hao Zhao

cs.CVMar 31, 2026

Benchmarking PhD-Level Coding in 3D Geometric Computer Vision

Authors: Wenyi Li, Renkai Luo, Yue Yu, Huan-ang Gao, Mingju Gao, Li Yuan, Chaoyou Fu, Hao Zhao

Organizations: AIR, Tsinghua University · Qiuzhen College, Tsinghua University · BAAI · University of Toronto ·

Abstract

AI-assisted coding has rapidly reshaped software practice and research workflows, yet today's models still struggle to produce correct code for complex 3D geometric vision. If models could reliably write such code, the research of our community would change substantially. To measure progress toward that goal, we introduce GeoCodeBench, a PhD-level benchmark that evaluates coding for 3D vision. Each problem is a fill-in-the-function implementation task curated from representative papers at recent venues: we first let a tool propose candidate functions from official repositories, then perform careful human screening to select core 3D geometric components. For every target, we generate diverse, edge-case unit tests, enabling fully automatic, reproducible scoring. We evaluate eight representative open- and closed-source models to reflect the current ecosystem. The best model, GPT-5, attains only 36.6% pass rate, revealing a large gap between current capabilities and dependable 3D scientific coding. GeoCodeBench organizes tasks into a two-level hierarchy: General 3D capability (geometric transformations and mechanics/optics formulation) and Research capability (novel algorithm implementation and geometric logic routing). Scores are positively correlated across these axes, but research-oriented tasks are markedly harder. Context ablations further show that "more paper text" is not always better: cutting off at the Method section statistically outperforms full-paper inputs, highlighting unresolved challenges in long-context scientific comprehension. Together, these findings position GeoCodeBench as a rigorous testbed for advancing from generic coding to trustworthy 3D geometric vision coding.

Figures & tables

Capability	MMMU [ 74 ]	VR-B [ 72 ]	MBPP [ 3 ]	SWE [ 26 ]	Paper-B [ 52 ]	RCB [ 22 ]	Ours
Func. synthesis	$\times$	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$
Paper-to-code	$\times$	$\times$	$\times$	$\times$	$\surd$	$\surd$	$\surd$
Format discipline	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Domain knowledge	$\surd$	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
Hidden tests	$\times$	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
3D impl.	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\surd$

Table 1: Capability Coverage across Representative Benchmarks and GeoCodeBench. (

\surd

strong/explicit;

\surd

partial/incidental;

\times

not a primary focus.)

Figure 2: The Benchmark Curation and Evaluation Pipeline of GeoCodeBench.

					General 3D Capability		Research Capability
Model	Company	Overall	General	Research	Geo. Trans.	Mech./Opt.	Algorithm	Routing
GPT-5	OpenAI	36.6%	42.8%	29.1%	41.7%	43.7%	29.1 %	28.9%
Claude-Sonnet-4.5	Anthropic	31.1%	37.2%	23.7%	38.3%	36.5%	19.7%	35.9%
Gemini-2.5-Pro	Google	30.4%	33.8%	26.2%	41.9%	27.6%	25.3%	29.1%
Kimi-K2-Instruct	Moonshot	30.4%	34.6%	25.1%	36.7%	33.1%	23.1%	31.4%
Doubao-Seed-1.6	ByteDance	26.9%	29.7%	23.4%	40.9%	21.0%	22.9%	25.2%

Table 2: The Performance of Representative LLMs on GeoCodeBench. Performance is assessed across two aspects: General 3D Capability (General) and Research Capability (Research), four dimensions: Geometric Transformations (Geo. Trans.), Mechanics/Optics Formulation (Mech./Opt.), Novel Algorithm Implementation (Algorithm), and Geometric Logic Routing (Routing).

Figure 3: Comparison of General and Research Capability.

Figure 4: Case Study: Consistent Failure Across LLMs on a Simple Function. The function forward_event approximates “event accumulation” using the logarithmic intensity difference derived from two event-camera frames. Despite its brevity and simplicity, all tested LLMs failed.

Figure 5: Case Study: Creativity Correctness. The function compute_epipolar_distance requires calculating the symmetric epipolar distance between corresponding image points $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$ given $T_{21}$ and $\mathbf{K}$ . GPT-5 uses the Fundamental Matrix ( $\mathbf{F}$ ) method ( $\mathbf{l}_{2}=\mathbf{F}\mathbf{p}_{1}$ ), operating directly on pixel coordinates. DeepSeek-R1, conversely, first transforms the inputs to normalized coordinates ( $\mathbf{x}_{1},\mathbf{x}_{2}$ ) and then applies the Essential Matrix ( $\mathbf{E}$ ) method ( $\mathbf{l}^{\prime}_{2}=\mathbf{E}\mathbf{x}_{1}$ ). Both approaches are mathematically equivalent ( $\mathbf{F}=\mathbf{K}^{-T}\mathbf{E}\mathbf{K}^{-1}$ ) and yield the correct final distance, thus demonstrating Creative Correctness : models select distinct, valid pathways to achieve the required $3\text{D}$ geometry constraint.

Figure 6: Case Study: Divergence between General and Research Capability. Left: DeepSeek-R1 correctly solves the basic geometric problem of mapping 2D image coordinates to 3D spherical coordinates using trigonometric functions, demonstrating strong foundational knowledge. Right: In contrast, the model fails to correctly implement the paper-specific function cross_warp_with_pose_depth_candidates . The error involves misinterpreting the required mutual projection logic (Yin $\leftrightarrow$ Yang) as separate projections, highlighting the difficulty LLMs face in procedural reasoning and fine-grained algorithmic synthesis based on paper content.

Figure 7: Impact of Paper Length on Detailed Dimension Performance.

Figure 8: Impact of Paper Length on Overall Performance.

Figure 9: Distribution of Common Failure Types across Models on GeoCodeBench.

Source	Repos Nums	Problems Nums
CVPR’25	28	55
ICCV’25	15	33
ICLR’25	4	12
Total	47	100

Table 3: Data Source Statistic of GeoCodeBench.

Component	Mean	Min	Max
Structured Paper Content	20,232	14,052	31,079
Code w/ Masked Function	3,802	68	14,036
Golden Implementation Function	753	222	2,917

Table 4: Tokens Statistic of Different Components in GeoCodeBench.

Figure 10: Problem Type and Representative Functions in GeoCodeBench.

Figure 12: Case Study: Incorrect Type of Distance Implemented.

Figure 13: Case Study: Errors of De Casteljau Split When Degrading to Lines.

Figure 14: Case Study: MGadapter Fidelity Differs with Text Length.

Figure 15: Case Study: Incorrect Implementation of the BRDF Formulas.

Figure 16: Case Study: Incorrect Normalization that Ignores Area-weighting Coefficients.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

Sep 6, 2026

3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models

We introduce 3DHarnessBench, a benchmark that evaluates the agentic ability of frontier vision-language models (VLMs) to recover 3D geometry as Blender Python code from a variety of inputs. Unlike previous frameworks that prompt the VLMs with a fixed input (e.g., a single rendering or a text description), 3DHarnessBench evaluates four separate harness settings that progressively enable active agentic exploration, facilitated by recent Blender MCP functionality. Our hierarchy from Single-view, Multi-view, Active Visual (arbitrary viewpoint access), and Full 3D Interaction (complete access to the target object through Blender function calls) probes the models' abilities in both visual perception and active inference, tool calling, and self-correction. We observe that the ability of all frontier models to recover 3D geometry improves significantly with richer function call access, although the improvements are strongly model-dependent, revealing highly uneven agentic 3D-to-code capabilities. We will release the benchmark, code, outputs, and agent trajectories for reproducible 3D evaluation.

Ling Liu, Bingchen Gong, Amal Dev Parakkat +1

Institut Polytechnique de Paris · École Polytechnique · Télécom Paris

3D Reconstruction Benchmarks 3D VLMs Tool Use in VLMs 3D Reconstruction

Ask PDF

Apr 2, 2026

VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation

Evaluating code generation models for 3D spatial reasoning requires executing generated code in realistic environments and assessing outputs beyond surface-level correctness. We introduce a platform VoxelCode, for analyzing code generation capabilities for 3D understanding and environment creation. Our platform integrates natural language task specification, API-driven code execution in Unreal Engine, and a unified evaluation pipeline supporting both automated metrics and human assessment. To demonstrate its utility, we construct VoxelCodeBench, a benchmark of voxel manipulation tasks spanning three reasoning dimensions: symbolic interpretation, geometric construction, and artistic composition. Evaluating leading code generation models, we find that producing executable code is far easier than producing spatially correct outputs, with geometric construction and multi-object composition proving particularly challenging. By open-sourcing our platform and benchmark, we provide the community with extensible infrastructure for developing new 3D code generation benchmarks and probing spatial reasoning in future models.

Yan Zheng, Florian Bordes

FAIR at Meta · The University of Texas at Austin

Code Generation Benchmarks 3D Spatial Reasoning 3D Scene Generation

Ask PDF

May 31, 2026

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack. Authoring such procedural content, however, demands deep expertise in 3D software APIs, parametric design, and code-level geometric reasoning. In this paper, we propose 3DCodeBench, a systematic benchmark for evaluating vision-language model (VLM) agents for procedural 3D generation in 3D modeling software. Specifically, 3DCodeBench evaluates how effectively 12 advanced VLMs can serve as procedural 3D modelers by translating text and image references into procedural code for 3D modeling software. Recognizing that automated metrics may not fully capture the perceptual quality of 3D shapes, we build 3DCodeArena, a ranking platform based on pairwise human preferences over generated 3D outputs. From extensive evaluations and results, we observe that: (1) Failures mostly arise from API mismatches, while successful renders still suffer from disconnected or floating 3D geometric components. (2) Test-time scaling, such as higher thinking budgets and multi-turn refinement, improves performance overall. Our findings highlight a critical need for high-quality procedural coding data to advance commercial VLMs. Furthermore, effective procedural 3D modeling requires a robust execution environment that provides high-fidelity feedback for iterative refinement. We release 3DCodeBench, including the curated large-scale dataset of multimodal (text/image) prompts, procedural code, 3D object triplets, evaluation protocol, and the public 3DCodeArena platform as a foundational toolkit for exploring VLM-based procedural 3D modelers.

Yipeng Gao, Lei Shu, Genzhi Ye +5

Google DeepMind · University of Southern California · Google Research

VLM Evaluation 3D Asset Generation Procedural Content Generation Procedural 3D Generation Code Generation Benchmarks

Ask PDF

Abstract

Code Generation Benchmarks

Explore similar work

3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models

VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code