Spectral rendering reproduces the wavelength-dependent appearance that RGB rendering cannot: metamerism, colour shifts under spectrally rich illumination, coloured glass, metallic reflection and subsurface scattering. It requires a spectrally aware renderer, but the rendering ecosystem is predominantly RGB: game engines, cloud render farms, GPU rasterisers and most offline renderers consume RGB alone, with no access to the internal wavelength propagation that spectral rendering relies on. We introduce learned spectral codes, a compact linear representation that enables general spectral rendering through unmodified RGB pipelines. A trained encoder compresses any visible-range spectral quantity into a small latent vector; the codes are packed as RGB triplets and processed by an off-the-shelf RGB renderer. The codec preserves the arithmetic of light transport, exactly for scaling and addition and approximately for element-wise products, so two standard RGB passes suffice to produce results closely matching a full spectral rendering. The same codec handles reflectance, illumination, coloured glass, conductors, and participating media: one pipeline across material classes, with no per-material modification to the renderer. Where spectral asset data is unavailable, lightweight neural upsamplers bridge legacy RGB inputs to the latent space. Across diverse materials and illumination, the method yields substantial quality improvements on the cases where RGB rendering fails most visibly, at rendering cost comparable to efficient spectral-sampling techniques and with no changes to the renderer.
Figures & tables
Figure 1. A spectral image rendered through an unmodified RGB path tracer. Our codec (a) approximates the spectral ground truth (b) in two standard RGB passes; the conventional RGB render (c) shows visible chromatic shifts. The insets (A)–(J) magnify the regions marked in (a)–(c), rendered at higher sample counts; the three rows of each inset column share one exposure, normalised per patch for legibility. The scene exercises every supported material class (spectral textures, measured Schott glass and clear glass, copper and gold conductors, dipole and homogeneous subsurface scattering, and participating-medium fog) under mixed warm/cool broadband illumination with red, green, and blue accents.
Figure 2. Learned spectral codec and multi-pass rendering pipeline. (a) Training: a non-negative encoder E ( Wenc∈R+k×n ) and decoder D ( Wdec∈R+n×k ) are fit jointly on reflectance and illumination spectra; the same matrices embed transmittance τ , absorption σa , scattering σs , and conductor IOR η,κ at inference. Four loss terms ( Le2e , Lrec , Lcode , Lcol ) are combined (Sec. 3.6 ). (b) Rendering: spectral assets encode once into k -dimensional codes ( k=3B ); legacy RGB assets bypass the encoder via per-pixel upsamplers f(R),f(L) (dashed). The code is split into B RGB-format triplets, processed by B unmodified RGB passes, then reassembled and decoded; B=2 at k=6 .
Figure 3. Cornell box under five illuminants. Top to bottom: daylight 3356 K, daylight 7150 K, narrowband Gaussian at 525 nm (FWHM 40 nm), the structured red-dominant source narrow 351 (main peak 670 nm), and CIE FL11. The middle column shows five magnified patches (marked in the renderings; top to bottom: red wall, blue sphere, shadowed sphere, yellow sphere, green wall) as adjacent ground-truth/ours pairs for direct colour comparison. Each ΔE2000 map embeds the illuminant SPD as an inset (top-left). Per-illuminant means and the RGB baseline are given in Tab. 1 . 1024 spp.
Figure 4. The five Cornell-box evaluation illuminants of Tab. 1 and Fig. 3 , normalised SPDs over 380–730 nm: a warm broadband (3356 K), a daylight broadband (7150 K), a 525 nm Gaussian (FWHM 40 nm), the structured red-dominant source narrow 351 (main peak 670 nm, secondary lobes near 540 and 410 nm), and the multi-peaked fluorescent FL11.
k=6 representation
broad 12
broad 63
narrow 525
narrow 351
FL11
avg
equal-width bins
2.02
2.18
1.73
10.47
5.36
4.35
bins at decoder transitions
1.86
1.98
2.92
9.23
5.00
4.20
adaptive-width bins
1.98
2.07
1.33
6.69
5.57
3.53
RGB baseline ( k=3 )
3.40
2.42
2.89
3.82
4.01
3.31
equal boxes, learned encoder
1.25
2.03
2.94
1.42
4.14
2.36
Gaussian basis, learned encoder
0.92
1.81
1.50
1.10
7.52
2.57
Table 1. Cornell-box mean ΔE2000 per illuminant for k=6 representations. Pure binning uses a bin-average encoder and box decoder with no training; the adaptive variant optimises the bin edges on the training illuminants; the “bins at decoder transitions” variant places bin edges at our learned decoder’s own transition wavelengths. All rows share the pipeline and ground truth of Fig. 3 , at 1024 spp.
Figure 5. Capacity baselines on the Cornell box under FL11. Top row: the spectral reference (a), our codec (b), the two encoder-trained bases of Tab. 1 (c, d), the same bases with untrained encoders (e, f; Supplemental Tab. 3), and the RGB baseline (g). Bottom row: each render’s ΔE2000 map against (a); all maps share the colour scale shown under (a), capped at ΔE2000=18 . The five magnified patches under each rendering (locations as in Fig. 3 ) allow direct colour comparison; per-variant means are in Tab. 1 and Supplemental Tab. 3, and the FL11 SPD is in Fig. 4 .
Figure 6. Material-class evaluation. Seven exemplars spanning four regimes: glass under Beer–Lambert (B–L) and surface-transmittance, dipole BSSRDF, and homogeneous medium. Columns: spectral GT (Mitsuba 0.6 spectral build, CMF → sRGB); ours at k=6 (two RGB passes); RGB baseline; per-pixel ΔE2000 for the latent and the RGB baseline. The patches column shows four magnified regions (marked in the renderings; top to bottom: head, foreleg, forefoot, tail) as adjacent ours | GT | RGB triplets. For the glass rows the ground-truth cell embeds the filter’s normalised transmittance spectrum (inset, top-left; 400–700 nm). Per-material means are given in Sec. 6.2 ; the remaining five materials are shown in Supplemental Sec. 6.4. 1024 spp for the glass and dipole rows, 4096 spp for the homogeneous rows.
regime
glass
RGB
latent
gain
surface
OG550
10.52
1.28
8.22×
GG495
3.28
1.08
3.02×
BG7
6.10
2.06
2.96×
RG610
3.92
1.37
2.85×
VG9
1.74
1.19
1.46×
Beer–Lambert
OG550
10.36
1.72
6.02×
Table 2. Coloured glass. Dragon-masked mean ΔE2000 at 1024 spp for five Schott filters in the surface-transmittance and Beer–Lambert regimes; RGB baseline, latent pipeline, and their ratio.
mode
material
RGB
latent
gain
dipole
yellow-green
3.49
1.09
3.20×
pale mint
1.90
1.52
1.25×
rose
3.97
2.66
1.49×
potato
2.76
1.22
2.27×
homogeneous
skin
5.61
2.19
2.56×
wine
14.39
2.19
6.57×
Table 3. Subsurface scattering. Dragon-masked mean ΔE2000 at 1024 spp (dipole) and 4096 spp (homogeneous); RGB baseline, latent pipeline, and their ratio.
Figure 7. Heterogeneous subsurface scattering. Per-voxel spectral hand volume rendered through Mitsuba’s heterogeneous-medium plugin; the albedo lifts five anatomical features to spectra via Gitlina et al.’s skin reflectance LUT ( 2020 ) ; construction details in Supplemental Sec. 6.3. Each rendering sits next to its own ΔE2000 map; the top-right cell shows three magnified regions (palm, knuckle ridge, fingertip; marked in the renderings) as adjacent ours | GT | RGB triplets above the shared colour bar. Codec gain is 4.29× (4096 spp).
conductor
RGB
latent
gain
Ag (silver)
8.43
2.08
4.06×
Au (gold)
8.98
0.65
13.9×
Cu (copper)
8.83
1.01
8.79×
Al (aluminium)
8.45
2.11
4.00×
Cr (chromium)
9.27
2.10
4.42×
Table 4. Conductors. Dragon-masked mean ΔE2000 at 1024 spp for five measured metals under D65; RGB baseline, latent pipeline, and their ratio.
Figure 8. Material-class evaluation: conductors. Three measured conductors (silver, gold, copper) rendered as a solid dragon under D65, k=6 codec, dragon-masked ΔE2000 versus spectral GT. Per-metal means are given in Sec. 6.2 . Spectral conductor IOR is encoded via the per-component η,κ scheme of Sec. 4 . The patches column shows four magnified regions (marked in the renderings) as adjacent ours | GT | RGB triplets.
Method
D65
LED-B1
LED-B3
FL11
Reflectance upsampling (ground-truth illuminant)
Rendered spectrally: requires a spectral renderer
Jakob & Hanika
0.65 / 1.33
2.00 / 8.49
1.39 / 4.31
2.01 / 6.96
Guarnera et al., LED
0.09 / 0.03
5.78 / 12.62
3.06 / 6.09
2.25 / 5.32
Guarnera et al., Gaussian
0.08 / 0.03
5.96 / 13.48
5.41 / 10.08
4.63 / 10.77
Rendered through the codec: two RGB passes
Table 5. Per-spectrum ΔE2000 on 218 held-out reflectances under four illuminants (mean / p95), grouped by what each route requires of the renderer. Bold marks the best mean and the best p95 per column within each group . Scaled uplift: the illuminant’s RGB uplifted with Jakob & Hanika and multiplied by D65, Mitsuba 3’s RGB environment-map convention. Guarnera et al. ( 2022 ) : per-spectrum basis fit by genetic-algorithm search, with their recommended 6-LED RGBW+ basis or their Gaussian basis.
48 measured lamps
Method
mean
median
worst
Reference: codec on the ground-truth lamp spectra (two RGB passes)
Codec, k=6
1.23
0.98
3.1
Lamp recovered from its RGB, then rendered through the codec (two RGB passes)
Scaled uplift (J&H × D65)
2.98
2.09
26.3
Guarnera et al., LED
4.28
3.45
19.8
Table 6. Illumination robustness on unseen lamps : mean ΔE2000 over the 218 held-out reflectances, per lamp, aggregated over the 48 measured lamps as mean, median, and worst lamp. Top row: reference, the codec applied to the ground-truth lamp spectra, so no recovery is involved. Middle block: methods recovering the lamp from its RGB alone, each encoded and rendered through the codec as deployed, so all four rows share one route and one renderer requirement; bold marks the best per column within this block. Their spectral behaviour, evaluated as their authors intended, is in Tab. 5 . Bottom: the RGB baseline with no spectral representation.
Figure 9. Latent upsampling on Cornell-box scenes lit by spectral environment maps. Outdoor (rows 1–2) and indoor (rows 3–4); odd rows render, even rows ΔE2000 with envmap thumbnail leftmost (sky pixels excluded from means). Coloured borders group each baseline with the corresponding route of ours: reflectance upsampling with GT envmap (Jakob & Hanika ( 2019 ) vs. ours, R up); illumination upsampling with GT materials (Guarnera et al. ( 2022 ) vs. ours, L up). The two baselines are rendered spectrally and carry uplift error only; every column of ours is the deployable two-pass RGB route and carries the codec’s error as well. “Ours, both up” upsamples both sides from RGB; “Ours, k=6 ” applies the codec to GT spectra. 512 spp; per-cell ΔE2000 above each error map. Hyperspectral environment maps from Morimoto et al. ( 2019 ) .
Figure 10. Cross-renderer evaluation: Blender Cycles (open-source path tracer), two lighting conditions. For each lighting condition, top row: the 16-pass spectral reference; the codec applied to the same spectral assets (two RGB passes); the full RGB-asset route through our upsamplers; and the renderer’s plain RGB baseline. Bottom row: four magnified patches (marked in the renderings) as adjacent GT | codec | ours | RGB swatches with the shared colour bar, then the per-pixel ΔE2000 of the three non-reference columns against the reference (means below; scale capped at 10 ). RGB-baseline renders are brightness-matched to the reference. The inset in the reference cell shows the illumination: one panel per light SPD, or the environment map where one is used.
Figure 11. Cross-renderer evaluation: Maya Arnold (offline production renderer), two lighting conditions: light 1 is a nearly flat broadband source, smoother than D65 and the case an RGB pipeline handles best; light 2 is a spectrally structured source. Layout as in Fig. 10 . Under light 1 all three routes lie within 2ΔE2000 of the reference and the RGB baseline is the most accurate ( 0.69 ); under light 2 the RGB baseline rises to 4.25 while both codec routes stay near 2.3 .
Figure 12. Cross-renderer evaluation: KeyShot (product visualisation), two lighting conditions. Layout as in Fig. 10 . For the environment-map condition the reference sky spectrum is the Jakob & Hanika uplift of the RGB map, which is an 8-bit image, so the reference sky is spectrally flat wherever the map is clipped.
Figure 13. Cross-renderer evaluation: Unreal Engine 5 (real-time rasteriser with Lumen), two lighting conditions. Layout as in Fig. 10 .
Figure 14. Timing scenes (spectral reference, 1920×1080 , 1024 spp). Left to right: Glass of Water (aXel, CC0) and Veach MIS (B. Bitterli, CC0), both shading-bound; the Grey & White Room (Wig42, CC BY 3.0), a textured interior whose frame time is dominated by traversal. The controlled-load variant adds three FL11 ceiling lamps outside the view.
scene
frame time bound by
latent / spectral
Cornell box, 15 configurations
shading
0.94 – 1.10
Glass of Water
shading
1.13 – 1.22
Veach MIS
shading
1.13 – 1.29
Grey & White Room
traversal
1.75 – 1.81
+ three FL11 lamps
traversal, heavier spectra
1.63 – 1.73
Table 7. Wall-clock ratio, latent (two RGB passes) over Mitsuba 3 hero wavelength sampling , across the spp and resolution sweep of each scene.
Cameras and the foundation models built on them recognize what a 3D scene contains, but not what it is made of: surfaces of the same colour can be different materials. Hyperspectral unmixing explains each spectrum as a mixture of material signatures, yet in 3D it is applied to rendered images after the views are composited, or tied to slow implicit fields, and no multi-view data with known composition exist to check the result. We present the Spectral World Renderer (SWR), which unifies world generation and unmixing. A coding agent builds executable 3D worlds that are rendered with measured spectra, so every hyperspectral image comes with exact labels and sub-pixel proportions; the resulting SWR-Bench provides eight indoor and outdoor worlds in which decompositions can be verified. SWR then unmixes inside rendering: each Gaussian of a compact 3D representation is decoded into material abundances before compositing, so one render yields spectra and material maps, and any surface can be queried by material name, a clicked pixel or a reference spectrum. Without material labels, SWR identifies materials best in all eight worlds, has the best mean PSNR on real captures, trains 3.7-12.9x faster than unmixing baselines, renders 34.9 labelled views/s and separates same-coloured materials.
Chia-Ming Lee, Yu-Jou Hsiao, Ming-Ching Chang +3
University at Albany, State University of New York · National Yang Ming Chiao Tung University · National Cheng Kung University
Neural rendering techniques allow for accurate reconstruction of the geometry and color appearance of 3D scenes. Some methods have extended their use to additional imaging modalities, such as multispectral, infrared, or polarimetric data. However, all of these approaches require expensive sensors and calibrated setups to capture new multimodal frames for each new scene. We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain multi-view consistent renderings of unconventional modalities for scenes where either only RGB frames or very few of the additional modalities are available. Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities. This step allows predicting accurate renderings of unconventional modalities during a fine-tuning phase supervised only by RGB images. Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.
We present a multispectral extension to 3D Gaussian Splatting (3DGS) for wavelength-aware view synthesis. Each Gaussian is augmented with spectral radiance, represented via per-band spherical harmonics, and optimized under a dual-loss supervision scheme combining RGB and multispectral signals. To improve rendering fidelity, we perform spectral-to-RGB conversion at the pixel level, allowing richer spectral cues to be retained during optimization. Our method is evaluated on both public and self-captured real-world datasets, demonstrating consistent improvements over the RGB-only 3DGS baseline in terms of image quality and spectral consistency. Notably, it excels in challenging scenes involving translucent materials and anisotropic reflections. The proposed approach maintains the compactness and real-time efficiency of 3DGS while laying the foundation for future integration with physically based shading models.