Spectral Rendering Without a Spectral Renderer: Learned Spectral Codes for RGB Pipelines

Jiaqi Yu; Dar'ya Guarnera; Giuseppe Claudio Guarnera

Figure 1. A spectral image rendered through an unmodified RGB path tracer. Our codec (a) approximates the spectral ground truth (b) in two standard RGB passes; the conventional RGB render (c) shows visible chromatic shifts. The insets (A)–(J) magnify the regions marked in (a)–(c), rendered at higher sample counts; the three rows of each inset column share one exposure, normalised per patch for legibility. The scene exercises every supported material class (spectral textures, measured Schott glass and clear glass, copper and gold conductors, dipole and homogeneous subsurface scattering, and participating-medium fog) under mixed warm/cool broadband illumination with red, green, and blue accents.

Figure 2. Learned spectral codec and multi-pass rendering pipeline. (a) Training: a non-negative encoder

\mathrm{E}

(

W_{\mathrm{enc}}\!\in\!\mathbb{R}_{+}^{k\times n}

) and decoder

\mathrm{D}

(

W_{\mathrm{dec}}\!\in\!\mathbb{R}_{+}^{n\times k}

) are fit jointly on reflectance and illumination spectra; the same matrices embed transmittance

\tau

, absorption

\sigma_{a}

, scattering

\sigma_{s}

, and conductor IOR

\eta,\kappa

at inference. Four loss terms (

\mathcal{L}_{\mathrm{e2e}}

,

\mathcal{L}_{\mathrm{rec}}

,

\mathcal{L}_{\mathrm{code}}

,

\mathcal{L}_{\mathrm{col}}

) are combined (Sec. 3.6 ). (b) Rendering: spectral assets encode once into

k

-dimensional codes (

k{=}3B

); legacy RGB assets bypass the encoder via per-pixel upsamplers

f^{(R)},f^{(L)}

(dashed). The code is split into

B

RGB-format triplets, processed by

B

unmodified RGB passes, then reassembled and decoded;

B{=}2

at

k{=}6

.

Figure 3. Cornell box under five illuminants. Top to bottom: daylight 3356 K, daylight 7150 K, narrowband Gaussian at 525 nm (FWHM 40 nm), the structured red-dominant source narrow 351 (main peak 670 nm), and CIE FL11. The middle column shows five magnified patches (marked in the renderings; top to bottom: red wall, blue sphere, shadowed sphere, yellow sphere, green wall) as adjacent ground-truth/ours pairs for direct colour comparison. Each

\Delta E_{2000}

map embeds the illuminant SPD as an inset (top-left). Per-illuminant means and the RGB baseline are given in Tab. 1 . 1024 spp.

Figure 4. The five Cornell-box evaluation illuminants of Tab. 1 and Fig. 3 , normalised SPDs over 380–730 nm: a warm broadband (3356 K), a daylight broadband (7150 K), a 525 nm Gaussian (FWHM 40 nm), the structured red-dominant source narrow 351 (main peak 670 nm, secondary lobes near 540 and 410 nm), and the multi-peaked fluorescent FL11.

$k{=}6$ representation	broad 12	broad 63	narrow 525	narrow 351	FL11	avg
equal-width bins	2.02	2.18	1.73	10.47	5.36	4.35
bins at decoder transitions	1.86	1.98	2.92	9.23	5.00	4.20
adaptive-width bins	1.98	2.07	1.33	6.69	5.57	3.53
RGB baseline ( $k{=}3$ )	3.40	2.42	2.89	3.82	4.01	3.31
equal boxes, learned encoder	1.25	2.03	2.94	1.42	4.14	2.36
Gaussian basis, learned encoder	0.92	1.81	1.50	1.10	7.52	2.57

Table 1. Cornell-box mean

\Delta E_{2000}

per illuminant for

k{=}6

representations. Pure binning uses a bin-average encoder and box decoder with no training; the adaptive variant optimises the bin edges on the training illuminants; the “bins at decoder transitions” variant places bin edges at our learned decoder’s own transition wavelengths. All rows share the pipeline and ground truth of Fig. 3 , at 1024 spp.

Figure 5. Capacity baselines on the Cornell box under FL11. Top row: the spectral reference (a), our codec (b), the two encoder-trained bases of Tab. 1 (c, d), the same bases with untrained encoders (e, f; Supplemental Tab. 3), and the RGB baseline (g). Bottom row: each render’s

\Delta E_{2000}

map against (a); all maps share the colour scale shown under (a), capped at

\Delta E_{2000}{=}18

. The five magnified patches under each rendering (locations as in Fig. 3 ) allow direct colour comparison; per-variant means are in Tab. 1 and Supplemental Tab. 3, and the FL11 SPD is in Fig. 4 .

Figure 6. Material-class evaluation. Seven exemplars spanning four regimes: glass under Beer–Lambert (B–L) and surface-transmittance, dipole BSSRDF, and homogeneous medium. Columns: spectral GT (Mitsuba 0.6 spectral build, CMF

\to

sRGB); ours at

k{=}6

(two RGB passes); RGB baseline; per-pixel

\Delta E_{2000}

for the latent and the RGB baseline. The patches column shows four magnified regions (marked in the renderings; top to bottom: head, foreleg, forefoot, tail) as adjacent ours | GT | RGB triplets. For the glass rows the ground-truth cell embeds the filter’s normalised transmittance spectrum (inset, top-left; 400–700 nm). Per-material means are given in Sec. 6.2 ; the remaining five materials are shown in Supplemental Sec. 6.4. 1024 spp for the glass and dipole rows, 4096 spp for the homogeneous rows.

regime	glass	RGB	latent	gain
surface	OG550	$10.52$	$1.28$	$\mathbf{8.22\times}$
	GG495	$3.28$	$1.08$	$3.02\times$
	BG7	$6.10$	$2.06$	$2.96\times$
	RG610	$3.92$	$1.37$	$2.85\times$
	VG9	$1.74$	$1.19$	$1.46\times$
Beer–Lambert	OG550	$10.36$	$1.72$	$\mathbf{6.02\times}$

Table 2. Coloured glass. Dragon-masked mean

\Delta E_{2000}

at 1024 spp for five Schott filters in the surface-transmittance and Beer–Lambert regimes; RGB baseline, latent pipeline, and their ratio.

mode	material	RGB	latent	gain
dipole	yellow-green	$3.49$	$1.09$	$\mathbf{3.20\times}$
	pale mint	$1.90$	$1.52$	$1.25\times$
	rose	$3.97$	$2.66$	$1.49\times$
	potato	$2.76$	$1.22$	$2.27\times$
homogeneous	skin	$5.61$	$2.19$	$2.56\times$
	wine	$14.39$	$2.19$	$\mathbf{6.57\times}$

Table 3. Subsurface scattering. Dragon-masked mean

\Delta E_{2000}

at 1024 spp (dipole) and 4096 spp (homogeneous); RGB baseline, latent pipeline, and their ratio.

Figure 7. Heterogeneous subsurface scattering. Per-voxel spectral hand volume rendered through Mitsuba’s heterogeneous-medium plugin; the albedo lifts five anatomical features to spectra via Gitlina et al.’s skin reflectance LUT ( 2020 ) ; construction details in Supplemental Sec. 6.3. Each rendering sits next to its own

\Delta E_{2000}

map; the top-right cell shows three magnified regions (palm, knuckle ridge, fingertip; marked in the renderings) as adjacent ours | GT | RGB triplets above the shared colour bar. Codec gain is

\mathbf{4.29\times}

(4096 spp).

conductor	RGB	latent	gain
Ag (silver)	$8.43$	$2.08$	$4.06\times$
Au (gold)	$8.98$	$0.65$	$\mathbf{13.9\times}$
Cu (copper)	$8.83$	$1.01$	$8.79\times$
Al (aluminium)	$8.45$	$2.11$	$4.00\times$
Cr (chromium)	$9.27$	$2.10$	$4.42\times$

Table 4. Conductors. Dragon-masked mean

\Delta E_{2000}

at 1024 spp for five measured metals under D65; RGB baseline, latent pipeline, and their ratio.

Figure 8. Material-class evaluation: conductors. Three measured conductors (silver, gold, copper) rendered as a solid dragon under D65,

k{=}6

codec, dragon-masked

\Delta E_{2000}

versus spectral GT. Per-metal means are given in Sec. 6.2 . Spectral conductor IOR is encoded via the per-component

\eta,\kappa

scheme of Sec. 4 . The patches column shows four magnified regions (marked in the renderings) as adjacent ours | GT | RGB triplets.

Reflectance upsampling (ground-truth illuminant)
Method	D65	LED-B1	LED-B3	FL11
Rendered spectrally: requires a spectral renderer
Jakob & Hanika	0.65 / 1.33	2.00 / 8.49	1.39 / 4.31	2.01 / 6.96
Guarnera et al., LED	0.09 / 0.03	5.78 / 12.62	3.06 / 6.09	2.25 / 5.32
Guarnera et al., Gaussian	0.08 / 0.03	5.96 / 13.48	5.41 / 10.08	4.63 / 10.77
Rendered through the codec: two RGB passes

Table 5. Per-spectrum

\Delta E_{2000}

on 218 held-out reflectances under four illuminants (mean / p95), grouped by what each route requires of the renderer. Bold marks the best mean and the best p95 per column within each group . Scaled uplift: the illuminant’s RGB uplifted with Jakob & Hanika and multiplied by D65, Mitsuba 3’s RGB environment-map convention. Guarnera et al. ( 2022 ) : per-spectrum basis fit by genetic-algorithm search, with their recommended 6-LED RGBW+ basis or their Gaussian basis.

Reference: codec on the ground-truth lamp spectra (two RGB passes)
	48 measured lamps
Method	mean	median	worst
Codec, $k{=}6$	1.23	0.98	3.1
Lamp recovered from its RGB, then rendered through the codec (two RGB passes)
Scaled uplift (J&H $\times$ D65)	2.98	2.09	26.3
Guarnera et al., LED	4.28	3.45	19.8

Table 6. Illumination robustness on unseen lamps : mean

\Delta E_{2000}

over the 218 held-out reflectances, per lamp, aggregated over the 48 measured lamps as mean, median, and worst lamp. Top row: reference, the codec applied to the ground-truth lamp spectra, so no recovery is involved. Middle block: methods recovering the lamp from its RGB alone, each encoded and rendered through the codec as deployed, so all four rows share one route and one renderer requirement; bold marks the best per column within this block. Their spectral behaviour, evaluated as their authors intended, is in Tab. 5 . Bottom: the RGB baseline with no spectral representation.

Figure 9. Latent upsampling on Cornell-box scenes lit by spectral environment maps. Outdoor (rows 1–2) and indoor (rows 3–4); odd rows render, even rows

\Delta E_{2000}

with envmap thumbnail leftmost (sky pixels excluded from means). Coloured borders group each baseline with the corresponding route of ours: reflectance upsampling with GT envmap (Jakob & Hanika ( 2019 ) vs. ours,

R

up); illumination upsampling with GT materials (Guarnera et al. ( 2022 ) vs. ours,

L

up). The two baselines are rendered spectrally and carry uplift error only; every column of ours is the deployable two-pass RGB route and carries the codec’s error as well. “Ours, both up” upsamples both sides from RGB; “Ours,

k{=}6

” applies the codec to GT spectra. 512 spp; per-cell

\overline{\Delta E}_{2000}

above each error map. Hyperspectral environment maps from Morimoto et al. ( 2019 ) .

Figure 10. Cross-renderer evaluation: Blender Cycles (open-source path tracer), two lighting conditions. For each lighting condition, top row: the 16-pass spectral reference; the codec applied to the same spectral assets (two RGB passes); the full RGB-asset route through our upsamplers; and the renderer’s plain RGB baseline. Bottom row: four magnified patches (marked in the renderings) as adjacent GT | codec | ours | RGB swatches with the shared colour bar, then the per-pixel

\Delta E_{2000}

of the three non-reference columns against the reference (means below; scale capped at

10

). RGB-baseline renders are brightness-matched to the reference. The inset in the reference cell shows the illumination: one panel per light SPD, or the environment map where one is used.

Figure 11. Cross-renderer evaluation: Maya Arnold (offline production renderer), two lighting conditions: light 1 is a nearly flat broadband source, smoother than D65 and the case an RGB pipeline handles best; light 2 is a spectrally structured source. Layout as in Fig. 10 . Under light 1 all three routes lie within

2\,\Delta E_{2000}

of the reference and the RGB baseline is the most accurate (

0.69

); under light 2 the RGB baseline rises to

4.25

while both codec routes stay near

2.3

.

Figure 12. Cross-renderer evaluation: KeyShot (product visualisation), two lighting conditions. Layout as in Fig. 10 . For the environment-map condition the reference sky spectrum is the Jakob & Hanika uplift of the RGB map, which is an 8-bit image, so the reference sky is spectrally flat wherever the map is clipped.

Figure 13. Cross-renderer evaluation: Unreal Engine 5 (real-time rasteriser with Lumen), two lighting conditions. Layout as in Fig. 10 .

Figure 14. Timing scenes (spectral reference,

1920{\times}1080

, 1024 spp). Left to right: Glass of Water (aXel, CC0) and Veach MIS (B. Bitterli, CC0), both shading-bound; the Grey & White Room (Wig42, CC BY 3.0), a textured interior whose frame time is dominated by traversal. The controlled-load variant adds three FL11 ceiling lamps outside the view.

scene	frame time bound by	latent / spectral
Cornell box, 15 configurations	shading	$0.94$ – $1.10$
Glass of Water	shading	$1.13$ – $1.22$
Veach MIS	shading	$1.13$ – $1.29$
Grey & White Room	traversal	$1.75$ – $1.81$
+ three FL11 lamps	traversal, heavier spectra	$1.63$ – $1.73$

Table 7. Wall-clock ratio, latent (two RGB passes) over Mitsuba 3 hero wavelength sampling , across the spp and resolution sweep of each scene.

Spectral Rendering Without a Spectral Renderer: Learned Spectral Codes for RGB Pipelines

Abstract

Figures & tables

Physically Based Rendering

Spectral World Renderer: Unified 3D Hyperspectral Unmixing

Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis

MSGS: Multispectral 3D Gaussian Splatting

Abstract

Physically Based Rendering

Explore similar work

Spectral World Renderer: Unified 3D Hyperspectral Unmixing

Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis

MSGS: Multispectral 3D Gaussian Splatting