RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives

Chong Zeng; Yue Dong; Pieter Peers; Lvmin Zhang; Maneesh Agrawala

cs.CVSep 4, 2026

RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives

Authors: Chong Zeng, Yue Dong, Pieter Peers, Lvmin Zhang, Maneesh Agrawala

Organizations: Stanford University, USA · Microsoft Research, China · College of William & Mary, USA

Abstract

We present 'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.

Figures & tables

Panel of Figure 1 — Figure 1: RenderFormer-V2 can simulate global light transport in scenes that include transparent materials with caustics, environment lighting, volumetric scattering, textures, and scenes with over $100$ k primitives, without the need for per-scene training.

Figure 2: RenderFormer-V2 Neural Rendering Pipeline.

Panel of Figure 5 — Figure 5: Additional results demonstrating RenderFormer-V2’s ability to handle displacement mapping and volumetric scattering without the need for specialized code, and two complex scenes with a large number of primitives.

Panel of Figure 9 — Figure 9: Qualitative comparison of render quality of RenderFormer vs. RenderFormer-V2 for an increasing number of primitives. At $83.2$ K primitives, RenderFormer loses attentional focus, resulting in a darkened image.

Panel of Figure 10 — Figure 10: Rendering at higher resolutions also improves geometric details even for features larger than a pixel. E.g . , the claws on the dragon and Lucy’s face and hands are better resolved at $2048$ (right) than at $512$ (left) resolution.

Model Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	HDR-FLIP $\downarrow$
RenderFormer-V2	28.25	0.8982	0.0997	0.4200
w/o AS	27.36	0.8841	0.1247	0.4359
w/o SW	26.92	0.8715	0.1289	0.4517
AS w/o L	27.40	0.8813	0.1081	0.4250
AS w/o S	27.09	0.8754	0.1210	0.4485
AS w/o L, S	26.13	0.8474	0.1514	0.4959

Table 1: Sparse attention ablation with or without sliding windowed attention (SW) and attention sink (AS) with inclusion of light source tokens (L) and summary tokens (S), as well as varying number of summarization ratios. For each metric, the best , second best , and third best results are highlighted.

Panel of Figure 11 — Figure 11: A qualitative comparison of the ablation variants indicates that all components are essential to achieve the highest render quality.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	View-indep.	View-dep.
Layers	12	6
Model Dimension	768	768
Attention Heads	6	6
Attention Type	Sparse Self-Attn.	Cross-Attn. + Swin Self-Attn.
FFN Dimension	3072	3072
FFN Activation	SwiGLU

Appendix

Table 1: Architectural details of RenderFormer-V2’s two-stage transformer pipeline.

Panel of Figure 2 — Figure 2: Scenes from the main submission rendered with RenderFormer-V2 and compared to path-traced reference images.

Figure 3: RenderFormer-V2 and RenderFormer Model Architecture.

Panel of Figure 4 — Figure 4: Visual comparison of RenderFormer-V2 and RenderFormer against a path-traced reference. Difference maps (scaled by $\times 5$ ) are shown to highlight rendering errors.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FLIP $\downarrow$
RenderFormer	26.2423	0.9473	0.0886	0.1332
RenderFormer-V2	28.0810	0.9648	0.0503	0.1172
RenderFormer-V2 (finetuned)	30.7304	0.9742	0.0244	0.0952

Appendix

Table 2: Comparison of high-resolution render accuracy between RenderFormer and RenderFormer-V2 (with and without high-resolution fine-tuning). The metrics are computed on renderings at

2048\times 2048

resolution.

Tri Count	Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FLIP $\downarrow$
4K	RenderFormer-V2	33.84	0.9776	0.0198	0.1025
	RenderFormer	34.51	0.9823	0.0244	0.0838
8K	RenderFormer-V2	33.00	0.9682	0.0305	0.1230
	RenderFormer	32.89	0.9708	0.0358	0.1079
16K	RenderFormer-V2	29.96	0.9457	0.0502	0.1932
	RenderFormer	29.36	0.9418	0.0604	0.1555

Appendix

Table 3: Comparison between RenderFormer-V2 (ours) and RenderFormer across different triangle counts. For each triangle count and each metric, the better result is highlighted in bold.

Panel of Figure 6 — Figure 6: The $8$ test scenes from RenderFormer [ 43 ] rendered with RenderFormer-V2.

Panel of Figure 7 — Figure 7: Additional room scenes rendered with RenderFormer-V2.

Panel of Figure 8 — Figure 8: A variety of procedurally generated ablation scenes rendered with RenderFormer-V2 and compared to path-traced reference images. We also report PSNR, SSIM, LPIPS, and FLIP errors.

Neural Rendering

Neural rendering uses learned 3D scene and light representations to synthesize, edit, and relight images from new viewpoints.

See all 120 papers

RenderFormer++: Scalable and Physically Grounded Feed-Forward Neural Rendering
Jun 29, 2026Huangsheng Du, Haoran Zhu, Youcheng Cai +2Neural Rendering Efficient ViTs
University of Science and Technology of China, China
RADIANT: Object-Centric Feed-Forward Global Illumination
Oct 23, 2026Sarah Zhang, Zhiyang Dou, Minghao Guo +2Neural Rendering Physically Based Rendering
MIT CSAIL
Floating Radiance Networks
Aug 6, 2026Krzysztof Byrski, Rafał Tobiasz, Grzegorz Wilczyński +5Neural Rendering Novel View Synthesis
Jagiellonian University, Faculty of Mathematics and Computer Science · IDEAS Research Institute · Poznań University of Technology, Institute of Robotics and Machine Intelligence

Jun 29, 2026J/K move · Enter open · S save

RenderFormer++: Scalable and Physically Grounded Feed-Forward Neural Rendering

Huangsheng Du, Haoran Zhu, Youcheng Cai, Jinyang Meng+1

University of Science and Technology of China, China

Neural Rendering Efficient ViTs Physics-Informed ML Physically Based Rendering

Open paper AskPDF arXiv

We present RenderFormer++, a scalable and physically grounded feed-forward neural rendering framework for global illumination in mesh scenes. Existing Transformer-based neural rendering methods such as RenderFormer achieve promising cross-scene generalization, but suffer from limited physical consistency and poor scalability due to the quadratic attention complexity of triangle-level tokenization. To address these issues, we introduce Physics-Informed Transport Guidance (PITG), which embeds rendering-equation inductive biases into the attention mechanism and enforces transport consistency loss, enabling physically consistent light transport modeling. We further propose Hierarchical Object-Centric Tokenization (HOCT), which aggregates triangle-level features into compact object-level tokens via cross-attention with learnable queries, substantially reducing computational and memory costs while preserving geometric and radiometric information. Extensive experiments demonstrate that RenderFormer++ achieves scalable, stable, and generalizable feed-forward global illumination rendering across complex large-scale scenes with improved physical accuracy and efficiency over prior neural rendering methods.

Abstract

Neural Rendering

Explore similar work

RenderFormer++: Scalable and Physically Grounded Feed-Forward Neural Rendering

RADIANT: Object-Centric Feed-Forward Global Illumination

Floating Radiance Networks