Learning to Code the World: Harness Evolution for Compositional 3D Reconstruction

Haoyu Yao; Chenguo Lin; Ruize Du; Yadong Mu

cs.CVOct 27, 2026

Learning to Code the World: Harness Evolution for Compositional 3D Reconstruction

Authors: Haoyu Yao, Chenguo Lin, Ruize Du, Yadong Mu

Organizations: 1Peking University

Abstract

Reconstructing editable and compositional 3D scenes from a real image requires jointly reasoning about geometry, object structure, and spatial relationships. We introduce SceneEvolve, a framework that represents 3D scenes as executable code and improves reconstruction by evolving the harness around a small open-source agent. SceneEvolve combines SceneAct, a compositional semantic action interface built on parameterized primitives, hierarchical composition, and semantic spatial relations, with SceneHarness-RSI, which leverages a stronger teacher agent to iteratively evolve the harness using feedback from development-set reconstructions. Throughout harness evolution, the model parameters, action semantics, and execution backend remain fixed. The resulting harness provides a reusable reconstruction procedure that enables the small open-source model to reconstruct unseen scenes without parameter updates or teacher assistance. Across two in-domain datasets and one out-of-domain dataset, SceneEvolve with the comparatively compact Qwen3.8-27B model improves all four evaluated reconstruction metrics while reducing mean inference time by 47.0% relative to the base model. More notably, despite relying on a substantially smaller model, SceneEvolve achieves 32.3% higher reconstruction quality than the large proprietary GPT-5.6-Sol model, while reducing mean equivalent deployment cost by 59.8%. Furthermore, the harness evolved using Qwen3.8-27B also generalizes to Kimi-K3, achieving comparable reconstruction performance to GPT-6-Astra. The project page is https://haoyu-agent.github.io/SceneEvolve.

Figures & tables

Figure 1: Our proposed SceneEvolve framework evolves the harness of a compact open-source code agent, enabling it to translate a single image into executable code that constructs an editable, compositional 3D scene. It improves reconstruction quality and reduces inference time compared with the base agent, while achieving higher reconstruction quality at lower cost than GPT-5.6-Sol.

Figure 2: Illustration of SceneEvolve for compositional image-to-3D reconstruction. The framework combines two components: (1) SceneAct translates scene intent into executable Blender operations through parameterized primitives, hierarchical composition, and semantic spatial relations. (2) SceneHarness-RSI couples an outer loop for teacher-guided harness evolution with an inner loop for candidate evaluation by a frozen student, retaining revisions that improve performance. The optimized harness is deployed on unseen scenes without parameter updates.

		In-domain								Out-of-domain
Model	Inference effort	ScanNet				CA-1M				ARKitScenes
		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DINOv2-S $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DINOv2-S $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DINOv2-S $\uparrow$
GPT-6-Astra	High	10.063	0.386	0.684	0.590	10.309	0.382	0.705	0.556	9.561	0.407	0.748	0.519
GPT-6-Astra	Medium	10.143	0.383	0.684	0.579	10.433	0.378	0.711	0.557	10.177	0.423	0.732	0.528
GPT-6-Astra	Low	9.888	0.385	0.701	0.596	10.108	0.372	0.691	0.552	10.245	0.403	0.712	0.475
Claude-Opus-5	High	9.174	0.352	0.735	0.438	8.781	0.368	0.749	0.441	8.551	0.389	0.751	0.417

Table 1: Performance comparison across different models on ScanNet and CA-1M (in-domain) and ARKitScenes (out-of-domain). We report PSNR, SSIM, LPIPS, and DINOv2 similarity. GPT-6-Astra serves as the reference. The best and second results are highlighted in bold and underlined .

Figure 3: Comparison of SceneEvolve with Qwen3.8-27B at xhigh , GPT-5.6-Sol at none , and GPT-6-Astra at none . The average DINOv2-S $\uparrow$ , inference time $\downarrow$ , and equivalent cost $\downarrow$ are reported.

Figure 4: Comparison of reconstruction quality against inference time and equivalent deployment cost across model configurations. The high , medium and low settings denote reasoning modes from deeper to shallower inference. DINOv2-S $\uparrow$ , inference time $\downarrow$ , and equivalent deployment cost $\downarrow$ are reported averaged over three datasets.

Figure 5: Comparison of SceneEvolve with Qwen3.8-27B at xhigh , GPT-5.6-Sol at none , and GPT-6-Astra at none on two in-domain datasets ScanNet and CA-1M and the out-of-domain dataset ARKitScenes. The DINOv2 similarity, inference time, and deployment cost are reported for each.

Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DINOv2-S $\uparrow$	Time $\downarrow$	Cost $\downarrow$
Base Qwen3.8-27B	9.049	0.397	0.788	0.198	1510.7	0.147
w/o SceneAct	9.102	0.405	0.771	0.305	1035.0	0.152
w/o SceneHarness-RSI	9.119	0.411	0.762	0.285	1424.0	0.155
SceneEvolve	9.166	0.424	0.739	0.392	800.2	0.117

Table 2: Component-ablation on Qwen3.8-27B, averaged across the three datasets. PSNR, SSIM, LPIPS, DINOv2 similarity, inference time, and equivalent cost are averaged across the three datasets.

Figure 6: Qualitative comparisons of indoor scene reconstruction with GPT-6-Astra, GPT-5.6-Sol, Qwen3.8-27B and Kimi-K3 on Scannet, CA-1M and ARKitScenes datasets. SceneEvolve reconstructs editable 3D scenes that capture object geometry, relative proportions, and spatial relationships in the reference images, demonstrating improved structural fidelity.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 957 papers

LEGO-Anything: Coding Agents for 3D Scene Reconstruction
Sep 28, 2026Xirui Li, Peng Shi, Mingwen Dong +73D Reconstruction Coding Agents
University of Maryland, College Park · AWS · Work done during an internship at AWS
$φ$ -Scene: Physically Grounded Image-to-3D Scene Reconstruction
Jun 19, 2026Haodong Li, Lulu Shao, Haolin Lu +43D Reconstruction Single-View 3D Reconstruction
University of California San Diego, USA
SceneReGen: Generative Reconstruction of 3D Scenes from a Single Image
Aug 25, 2026Zefan Tian, Yuteng Ye, Yiheng Zhang +53D Reconstruction Single-View 3D Reconstruction
Huawei · Northwestern Polytechnical University

Sep 28, 2026J/K move · Enter open · S save

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

Xirui Li, Peng Shi, Mingwen Dong, Sheng Zhang+6

University of Maryland, College Park · AWS · Work done during an internship at AWS

3D Reconstruction Coding Agents 3D Scene Representation Single-View 3D Reconstruction

Open paper AskPDF arXiv

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects scenes and renderings, and revises the program. To evaluate end-to-end scene recovery, we introduce LEGO-Bench, a simulator-grounded benchmark with 208 images from 104 diverse indoor and outdoor scenes. LEGO-Bench separately scores artifact validity, visible-surface geometry, and rendered appearance. Its simulator-grounded design enables extensibility and precise automatic evaluation. Among evaluated agents, GPT-6-astra achieves the strongest overall results, with 53.4% indoor and 39.6% outdoor scores, yet substantial gaps remain between delivering valid scene artifacts and faithfully recovering scene geometry and appearance. Analysis of agent construction trajectories reveals three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivate LEGO-Plugin, a training-free harness plugin for more controlled iterative scene construction, which improves all six evaluated models, with relative gains of up to 62.7% in overall score. Finally, we test whether reconstructed scenes can represent natural images and support vision tasks. In LEGO-World, we derive object detections, instance masks, and relative depth as deterministic queries on scenes reconstructed by GPT-6-astra. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

Abstract

3D Reconstruction

Explore similar work

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

φφφ-Scene: Physically Grounded Image-to-3D Scene Reconstruction

SceneReGen: Generative Reconstruction of 3D Scenes from a Single Image

$φ$ -Scene: Physically Grounded Image-to-3D Scene Reconstruction