Benchmarking PhD-Level Coding in 3D Geometric Computer Vision

Wenyi Li; Renkai Luo; Yue Yu; Huan-ang Gao; Mingju Gao; Li Yuan; Chaoyou Fu; Hao Zhao

cs.CVMar 31, 2026

Benchmarking PhD-Level Coding in 3D Geometric Computer Vision

Authors: Wenyi Li, Renkai Luo, Yue Yu, Huan-ang Gao, Mingju Gao, Li Yuan, Chaoyou Fu, Hao Zhao

Organizations: AIR, Tsinghua University · Qiuzhen College, Tsinghua University · BAAI · University of Toronto ·

Abstract

AI-assisted coding has rapidly reshaped software practice and research workflows, yet today's models still struggle to produce correct code for complex 3D geometric vision. If models could reliably write such code, the research of our community would change substantially. To measure progress toward that goal, we introduce GeoCodeBench, a PhD-level benchmark that evaluates coding for 3D vision. Each problem is a fill-in-the-function implementation task curated from representative papers at recent venues: we first let a tool propose candidate functions from official repositories, then perform careful human screening to select core 3D geometric components. For every target, we generate diverse, edge-case unit tests, enabling fully automatic, reproducible scoring. We evaluate eight representative open- and closed-source models to reflect the current ecosystem. The best model, GPT-5, attains only 36.6% pass rate, revealing a large gap between current capabilities and dependable 3D scientific coding. GeoCodeBench organizes tasks into a two-level hierarchy: General 3D capability (geometric transformations and mechanics/optics formulation) and Research capability (novel algorithm implementation and geometric logic routing). Scores are positively correlated across these axes, but research-oriented tasks are markedly harder. Context ablations further show that "more paper text" is not always better: cutting off at the Method section statistically outperforms full-paper inputs, highlighting unresolved challenges in long-context scientific comprehension. Together, these findings position GeoCodeBench as a rigorous testbed for advancing from generic coding to trustworthy 3D geometric vision coding.

Figures & tables

Capability	MMMU [ 74 ]	VR-B [ 72 ]	MBPP [ 3 ]	SWE [ 26 ]	Paper-B [ 52 ]	RCB [ 22 ]	Ours
Func. synthesis	$\times$	$\surd$	$\surd$	$\surd$	$\times$	$\surd$	$\surd$
Paper-to-code	$\times$	$\times$	$\times$	$\times$	$\surd$	$\surd$	$\surd$
Format discipline	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Domain knowledge	$\surd$	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
Hidden tests	$\times$	$\times$	$\times$	$\surd$	$\surd$	$\surd$	$\surd$
3D impl.	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\surd$

Table 1: Capability Coverage across Representative Benchmarks and GeoCodeBench. (

\surd

strong/explicit;

\surd

partial/incidental;

\times

not a primary focus.)

Figure 2: The Benchmark Curation and Evaluation Pipeline of GeoCodeBench.

					General 3D Capability		Research Capability
Model	Company	Overall	General	Research	Geo. Trans.	Mech./Opt.	Algorithm	Routing
GPT-5	OpenAI	36.6%	42.8%	29.1%	41.7%	43.7%	29.1 %	28.9%
Claude-Sonnet-4.5	Anthropic	31.1%	37.2%	23.7%	38.3%	36.5%	19.7%	35.9%
Gemini-2.5-Pro	Google	30.4%	33.8%	26.2%	41.9%	27.6%	25.3%	29.1%
Kimi-K2-Instruct	Moonshot	30.4%	34.6%	25.1%	36.7%	33.1%	23.1%	31.4%
Doubao-Seed-1.6	ByteDance	26.9%	29.7%	23.4%	40.9%	21.0%	22.9%	25.2%

Table 2: The Performance of Representative LLMs on GeoCodeBench. Performance is assessed across two aspects: General 3D Capability (General) and Research Capability (Research), four dimensions: Geometric Transformations (Geo. Trans.), Mechanics/Optics Formulation (Mech./Opt.), Novel Algorithm Implementation (Algorithm), and Geometric Logic Routing (Routing).

Figure 3: Comparison of General and Research Capability.

Figure 4: Case Study: Consistent Failure Across LLMs on a Simple Function. The function forward_event approximates “event accumulation” using the logarithmic intensity difference derived from two event-camera frames. Despite its brevity and simplicity, all tested LLMs failed.

Figure 5: Case Study: Creativity Correctness. The function compute_epipolar_distance requires calculating the symmetric epipolar distance between corresponding image points $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$ given $T_{21}$ and $\mathbf{K}$ . GPT-5 uses the Fundamental Matrix ( $\mathbf{F}$ ) method ( $\mathbf{l}_{2}=\mathbf{F}\mathbf{p}_{1}$ ), operating directly on pixel coordinates. DeepSeek-R1, conversely, first transforms the inputs to normalized coordinates ( $\mathbf{x}_{1},\mathbf{x}_{2}$ ) and then applies the Essential Matrix ( $\mathbf{E}$ ) method ( $\mathbf{l}^{\prime}_{2}=\mathbf{E}\mathbf{x}_{1}$ ). Both approaches are mathematically equivalent ( $\mathbf{F}=\mathbf{K}^{-T}\mathbf{E}\mathbf{K}^{-1}$ ) and yield the correct final distance, thus demonstrating Creative Correctness : models select distinct, valid pathways to achieve the required $3\text{D}$ geometry constraint.

Figure 6: Case Study: Divergence between General and Research Capability. Left: DeepSeek-R1 correctly solves the basic geometric problem of mapping 2D image coordinates to 3D spherical coordinates using trigonometric functions, demonstrating strong foundational knowledge. Right: In contrast, the model fails to correctly implement the paper-specific function cross_warp_with_pose_depth_candidates . The error involves misinterpreting the required mutual projection logic (Yin $\leftrightarrow$ Yang) as separate projections, highlighting the difficulty LLMs face in procedural reasoning and fine-grained algorithmic synthesis based on paper content.

Figure 7: Impact of Paper Length on Detailed Dimension Performance.

Figure 8: Impact of Paper Length on Overall Performance.

Figure 9: Distribution of Common Failure Types across Models on GeoCodeBench.

Source	Repos Nums	Problems Nums
CVPR’25	28	55
ICCV’25	15	33
ICLR’25	4	12
Total	47	100

Table 3: Data Source Statistic of GeoCodeBench.

Component	Mean	Min	Max
Structured Paper Content	20,232	14,052	31,079
Code w/ Masked Function	3,802	68	14,036
Golden Implementation Function	753	222	2,917

Table 4: Tokens Statistic of Different Components in GeoCodeBench.

Figure 10: Problem Type and Representative Functions in GeoCodeBench.

Figure 12: Case Study: Incorrect Type of Distance Implemented.

Figure 13: Case Study: Errors of De Casteljau Split When Degrading to Lines.

Figure 14: Case Study: MGadapter Fidelity Differs with Text Length.

Figure 15: Case Study: Incorrect Implementation of the BRDF Formulas.

Figure 16: Case Study: Incorrect Normalization that Ignores Area-weighting Coefficients.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models
Sep 6, 2026Ling Liu, Bingchen Gong, Amal Dev Parakkat +13D Reconstruction Benchmarks 3D VLMs
Institut Polytechnique de Paris · École Polytechnique · Télécom Paris
VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation
Apr 2, 2026Yan Zheng, Florian BordesCode Generation Benchmarks 3D Spatial Reasoning
FAIR at Meta · The University of Texas at Austin
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
May 31, 2026Yipeng Gao, Lei Shu, Genzhi Ye +5VLM Evaluation 3D Asset Generation
Google DeepMind · University of Southern California · Google Research

Sep 6, 2026J/K move · Enter open · S save

3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models

Ling Liu, Bingchen Gong, Amal Dev Parakkat, Maks Ovsjanikov

Institut Polytechnique de Paris · École Polytechnique · Télécom Paris

3D Reconstruction Benchmarks 3D VLMs Tool Use in VLMs 3D Reconstruction

Open paper AskPDF arXiv

We introduce 3DHarnessBench, a benchmark that evaluates the agentic ability of frontier vision-language models (VLMs) to recover 3D geometry as Blender Python code from a variety of inputs. Unlike previous frameworks that prompt the VLMs with a fixed input (e.g., a single rendering or a text description), 3DHarnessBench evaluates four separate harness settings that progressively enable active agentic exploration, facilitated by recent Blender MCP functionality. Our hierarchy from Single-view, Multi-view, Active Visual (arbitrary viewpoint access), and Full 3D Interaction (complete access to the target object through Blender function calls) probes the models' abilities in both visual perception and active inference, tool calling, and self-correction. We observe that the ability of all frontier models to recover 3D geometry improves significantly with richer function call access, although the improvements are strongly model-dependent, revealing highly uneven agentic 3D-to-code capabilities. We will release the benchmark, code, outputs, and agent trajectories for reproducible 3D evaluation.

Abstract

Code Generation Benchmarks

Explore similar work

3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models

VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generation

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code