BlenderFORGE: Framework for Optimizing Reactive 3D-Graphics Editing Ability of MLLMs
Organizations: The Hong Kong University of Science and Technology · SenseTime Research
Abstract
Multimodal Large Language Models (MLLMs) have shown promise as interactive agents, yet precise, programmatic 3D graphics editing remains difficult to train because existing systems largely rely on closed-source model APIs and lack deterministic, state-verifiable execution environments. We present BlenderFORGE, a trainable data-environment-optimization framework for open-weight MLLM agents that edit explicit Blender scenes through Blender Python (bpy) code generation. BlenderFORGE targets four fundamental state-verifiable editing task families: object placement, shape-key editing, lighting adjustment, and material modification. It integrates three components: (1) a scalable perturbation-based data generation pipeline built from curated assets and large-scale 3D scene repositories; (2) an isolated Blender Sandbox for deterministic code execution, structured state export, and closed-loop reward computation; and (3) a two-stage training pipeline that uses offline teacher trajectories for cold-start Supervised Fine-Tuning (SFT) and sandbox-computed rewards for multi-turn tool-agentic Reinforcement Learning (AgentLoop-RL). Experiments across the four task families show that the resulting open-weight Qwen3-VL-8B and Qwen3.5-9B BlenderFORGE agents substantially improve over their base models and achieve competitive performance with several zero-shot proprietary MLLM baselines on selected metrics. Additional cross-benchmark evaluations and camera-reframing experiments provide evidence of transfer beyond the original training setting.
Figures & tables
| Placement | Shape | Lighting | Material | |||||||||||||
| Perceptual | Normalized P-CD | Perceptual | Normalized ShapeKey L1 | Perceptual | Perceptual | |||||||||||
| Model | PL | N-CLIP | [email protected] | [email protected] | [email protected] | Med. | PL | N-CLIP | [email protected] | [email protected] | [email protected] | Med. | PL | N-CLIP | PL | N-CLIP |
| w/o change | 6.4 | 23.5 | 0.00 | 0.00 | 0.00 | 1.00 | 5.2 | 32.0 | 0.00 | 0.00 | 0.00 | 1.00 | 43.8 | 38.2 | 16.0 | 76.0 |
| Proprietary Models | ||||||||||||||||
| GPT-5.2 | 4.4 | 15.9 | 27.06 | 50.43 | 62.76 | 0.39 | 3.2 | 11.2 | 17.28 | 30.89 | 39.27 | 0.71 | 6.6 | 16.3 | 7.1 | 34.1 |
| Claude-Sonnet-4.6 | 5.9 | 21.8 | 10.00 | 25.00 | 32.50 | 0.91 | 3.6 | 19.4 | 13.50 | 26.00 | 35.00 | 0.87 | 6.9 | 18.3 | 4.6 | 35.2 |
| Training Stage | Perceptual | Normalized P-CD | Avg. turns | ||||||
| Traj. | Gen. | RL | PL | N-CLIP | [email protected] | [email protected] | [email protected] | Med. | |
| 18.8 | 57.1 | 0.00 | 1.50 | 1.50 | 21.31 | 9.29 | |||
| ✓ | 10.6 (8.2) | 33.3 (23.8) | 2.00 (2.00) | 5.00 (3.50) | 8.50 (7.00) | 1.64 (19.67) | 10.08 | ||
| ✓ | 6.5 (12.3) | 24.1 (33.0) | 4.50 (4.50) | 13.00 (11.50) | 24.00 (22.50) | 1.00 (20.31) | 8.01 | ||
| ✓ | ✓ | 6.9 (11.9) | 24.0 (33.1) | 6.50 (6.50) | 17.50 (16.00) | 25.50 (24.00) | 1.01 (20.30) | 5.76 | |
| ✓ | ✓ | 6.5 (12.3) | 24.9 (32.2) | 3.00 (3.00) | 14.00 (12.50) | 21.00 (19.50) | 1.03 (20.28) | 10.63 | |
| Task | Model | PL | N-CLIP | [email protected] | Med. |
| Placement | Base | 8.8 | 34.4 | 20.00 | 1.00 |
| BlenderFORGE | 8.4 | 30.3 | 36.00 | 0.91 | |
| Shape-Key | Base | 12.2 | 31.9 | 10.26 | 1.00 |
| BlenderFORGE | 8.9 | 26.6 | 17.95 | 0.88 | |
| Lighting | Base | 52.6 | 49.3 | – | – |
| BlenderFORGE | 19.0 | 25.3 | – | – |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
| Render engine | Cycles (GPU) |
| Samples / Max bounces | 128 / 8 |
| Adaptive sampling (threshold) | Enabled (0.2) |
| Caustics (reflective / refractive) | Disabled |
| Denoising | Enabled |
| Resolution |
| Endpoint | Method | Agent-Visible Behavior |
| /exec | POST | Executes bpy code on the Blender main thread; returns a JSON with success flag and captured stdout or traceback. |
| /preview | POST | Renders the current scene from a named camera via Cycles GPU; returns a base64-encoded PNG image. |
| /scene_info | GET | Exports the full scene descriptor as structured JSON (object names, types, poses, light attributes, shape-key values). |
| Model | Valid rate | Mean visibility | Mean angle |
| Qwen3-VL-8B Base | 13.3 | 9.0 | 167.2 |
| Qwen3-VL-8B BlenderFORGE | 35.0 | 31.6 | 142.8 |
| Qwen3.5-9B Base | 10.0 | 8.0 | 167.8 |
| Qwen3.5-9B BlenderFORGE | 35.0 | 23.7 | 143.0 |
| Placement | Shape-Key | |||
| Metric | Spec. | Gen. | Spec. | Gen. |
| PL | 5.6 | 5.7 | 5.8 | 5.1 |
| N-CLIP | 22.3 | 20.5 | 27.3 | 28.3 |
| [email protected] | 11.50 | 15.50 | 8.50 | 10.50 |
| [email protected] | 28.50 | 29.50 | 16.00 | 21.00 |
| [email protected] | 38.50 | 40.00 | 20.50 | 27.50 |
| Placement | Shape-Key | Lighting | Material | ||||
| Model | Thinking | [email protected] | PL | [email protected] | PL | PL | PL |
| Qwen3.5-9B | Yes | 3.00 | 7.8 | 6.00 | 8.4 | 36.1 | 16.9 |
| Qwen3.5-9B | No | 5.00 | 21.6 | 11.00 | 11.0 | 51.1 | 17.1 |
| Qwen3-VL-8B Thinking | Yes | 1.50 | 18.8 | 12.00 | 6.1 | 43.0 | 23.9 |
| Qwen3-VL-8B Instruct | No | 9.00 | 7.3 | 13.50 | 6.3 | 38.7 | 25.0 |
| Qwen3.5-9B BlenderFORGE | Yes | 40.00 | 5.7 | 27.50 | 5.1 | 7.3 | 7.8 |
| Group | Pairs | Base wins | SFT+RL wins | Ties |
| Overall | 160 | 18 | 108 | 34 |
| T | 80 | 10 | 48 | 22 |
| T+I | 80 | 8 | 60 | 12 |
| Placement / T+I | 20 | 0 | 17 | 3 |
| Shape-Key / T+I | 20 | 2 | 16 | 2 |
| Material / T+I | 20 | 2 | 15 | 3 |
| Task | Metric | 95% CI | -value | ||
| Placement | P-Chamfer | 200 | |||
| N-CLIP | 200 | ||||
| PL | 200 | ||||
| Shape-Key | ShapeKey-L1 | 200 | |||
| N-CLIP | 200 | ||||
| PL | 200 |
| Stage | GPU usage | CPU/RAM | Time | Total compute |
| Dataset generation | 2 RTX 5090 GPUs for scene generation and rendering | 16 cores / 256GB | 24 hours | 48 GPU-hours |
| SFT | 8 RTX 5090 GPUs for unified mixed-task training | 64 cores / 1TB | 8 hours | 64 GPU-hours |
| AgentLoop-RL | 8 RTX 5090 GPUs for training; up to 8 for sandbox rendering | Up to 128 cores / 2TB | 10 days | 1920 training; at most 1920 sandbox GPU-hours |
| Evaluation | 2 RTX 5090 GPUs per evaluation run | 16 cores / 256GB per run | 6 hours per run | 12 GPU-hours per run |
| Asset / software | Use in BlenderFORGE | License / terms treatment |
| 3D-FRONT / 3D-FUTURE | Base scenes and object assets for placement | Used under provider terms; restricted raw assets are not redistributed. |
| Infinigen | Lighting scenes and material definitions | Code: BSD 3-Clause ; preserve upstream notices and attribution. |
| Blender / bpy | Scene representation, editing API, rendering, and sandbox execution | Used under Blender GNU GPL terms; integration code retains notices. |
| Qwen3-14B / Qwen3-VL-8B / Qwen3.5-9B | Shape-key renaming and open-weight agent initialization | Apache 2.0 for these checkpoints; retain license and attribution notices. |
| GPT-5.2 / GPT-5.5 | Teacher trajectories and visual assessment | Provider API terms; no proprietary weights are redistributed. |
| Gemini | Imagine-OOD goal images and zero-shot baselines | Provider service terms; no proprietary weights are redistributed. |
3D Scene Editing
3D scene editing studies how to make controlled changes to objects, appearance, layout, and motion while keeping a scene coherent across views.
See all 94 papers