Spectral World Renderer: Unified 3D Hyperspectral Unmixing

Chia-Ming Lee; Yu-Jou Hsiao; Ming-Ching Chang; Xin Li; Yu-Lun Liu; Chih-Chung Hsu

Figure 2: The SWR world generator gives every image its exact composition. From reference photographs and a short brief, a coding agent writes an executable world whose every surface carries a library-measured spectrum. One shared sub-pixel rendering produces the hyperspectral and RGB observations together with the material of every pixel, so the ground truth is exact for the rendered world. Checks on layout and rendering return flawed worlds for revision, yielding aligned, verifiable samples across all eight worlds (right).

Figure 3: No single spectrum describes a material. Grass specimens differ (a), and the same grass changes spectral shape between sunlight and shade (b). Both broadleaf spectra shown lie within independent leaf measurements (c).

Dataset	Bands	nm	Scenes $\times$ views	Capture	ID	Prop.	Gen.
Replica	3	RGB	18	indoor	✗	✗	✗
Hypersim	3	RGB	461 $\times$ 168	indoor	✗	✗	✗
Kubric	3	RGB	–	random	✗	✗	✓
NeSpoF	21	450–650	4 $\times$ 18–30	tabletop	✗	✗	✗
NeSpoF-Seg.	21	450–650	4 scenes	synthetic	✓	✗	✗
HS-NeRF	128/141	371–1087	8 $\times$ 48–433	turntable	✗	✗	✗

Table 1: Multi-view datasets with ground truth. ID: material labels; Prop.: geometric area proportions; Gen.: worlds on demand. NeSpoF-Seg.: synthetic segmentation truth from UnMix-NeRF [ 63 ] . Only SWR-Bench samples proportions with each image and models spectral variability.

Figure 4: Spectral World Renderer unmixes at every ray intersection, before compositing. An RGB anchor fixes the geometry, and each Gaussian carries harmonic features whose size does not depend on the band count. A shared decoder maps the feature at each intersection to abundances and an intensity, which mix scene-wide endmembers; the same visibility weights then render the spectrum and the abundances. Multi-view appearance clusters initialize the endmembers and regularize their assignment, so no material label enters training.

					UnMix-NeRF [ 63 ]				3DGS [ 37 ] + VCA / NNLS				Spectral World Renderer
Scene	Setting	Classes	Rare	Spectra	PSNR $\uparrow$	SAM $\downarrow$	mIoU $\uparrow$	EM $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	mIoU $\uparrow$	EM $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	mIoU $\uparrow$	EM $\downarrow$
Forbidden City	outdoor	9	3	47	26.13	7.37	0.19	27.4	25.03	5.05	0.17	30.7	28.37	4.45	0.55	4.4
Market	outdoor	6	0	29	31.87	3.61	0.13	30.9	30.02	1.85	0.24	26.9	33.63	1.95	0.78	2.4
Botanical	outdoor	6	0	29	30.24	5.99	0.10	38.8	29.57	3.12	0.22	27.1	32.88	2.97	0.71	2.4
NVIDIA HQ	outdoor	8	2	27	18.29	14.98	0.08	43.0	22.27	12.31	0.13	24.4	25.30	8.25	0.31	3.7
Study	indoor	5	2	5	33.78	2.81	0.20	22.9	32.53	1.52	0.30	14.9	33.14	2.01	0.52	0.9

Table 2: Without material labels, Spectral World Renderer identifies materials best on SWR-Bench. It has the highest mIoU in every world, the best or second-best PSNR and SAM, and learns the endmembers closest to the true spectra; a lower spectral angle (3DGS in Market and Study) does not imply better identification. Rare: classes under 3 % of test pixels; Spectra: specimens; EM in degrees.

Figure 5: Multi-view appearance clusters keep similar materials apart (Forbidden City). (a) White marble is four times brighter than grey masonry but only

3.8^{\circ}

apart in spectral shape; since intensity absorbs brightness, one endmember can explain both. (b) Clustering pixels of all training views by shape and brightness separates them, with clusters shared by every view. (c) Kept as a weak regularizer during training, these clusters give marble its own layer; without them it is absorbed into masonry.

Figure 6: Spectral World Renderer recovers the material layer closest to the ground truth. Across all eight worlds, each row tints one material by its abundance in a test view, for a material to which every method maps an endmember; IoU: summed abundance

>0.5

against the true mask. Unmixing rendered spectra afterwards (3DGS + VCA) or inside a radiance field (UnMix-NeRF) spreads the material over unrelated surfaces. All rows use the models of Table 2 .

	Market (SWR-Bench)			HSI-SC apple		City
	mIoU $\uparrow$	PSNR $\uparrow$	SAM $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	train $\downarrow$
w/o refinement	0.66	30.73	3.20	28.44	1.67	–
SWR	0.78	33.63	1.95	30.99	1.34	6 min 37 s

Table 3: RGB feature refinement improves both identification and spectral fidelity. Refining geometry and harmonic features on the RGB anchor gives the decoder spatially structured features; without it, the decoder reads random features.

Figure 7: One feature per Gaussian blends materials across edges and shadows. Within-primitive variation on a Market test view (inset: input; values for this view).

	City	FPS $\uparrow$ at $960\times 540\times 64$
Method	train $\downarrow$	Market	Bot.	Study	Karst	Mean
UnMix-NeRF	24 min 17 s	0.18	0.21	0.21	0.12	0.18
3DGS+VCA/NNLS	85 min 2 s	2.53	2.51	4.70	4.73	3.6
SWR	6 min 37 s	36.5	39.3	36.3	27.4	34.9

Table 4: SWR is fastest. FPS: views/s with spectra and label maps. Ours: our optimized fused fp16 decoder, skipping hits of weight

<10^{-4}

(99.97 % identical labels); baselines: reference code. City training spans all four stages, anchor to decoder.

		PSNR-leading baseline				SWR
Source	$L$	Method	PSNR $\uparrow$	SAM $\downarrow$	p95 $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	p95 $\downarrow$
NeSpoF	21	HyperGS	28.36	4.58	9.44	29.39	3.86	8.04
Apple	204	HyperGS	27.95	1.64	4.88	30.67	1.35	3.87
HS-NeRF	128 141	3DGS	26.32	5.71	16.75	26.51	5.97	17.31

Table 5: Best PSNR on every real source.

L

: bands; p95: 95th-percentile SAM; mean of 4 NeSpoF / 5 HS-NeRF scenes; baseline: best of 3DGS, HyperGS, DD-HGS.

Figure 8: SWR renders the most faithful spectra in this view. 204-band real capture (HSI-SC apple); inset: its abundances.

Figure 9: Per-Gaussian abundances find materials that RGB confuses. In City, foliage-coloured mineral leaves replace the boxed crowns. (a) The same query on RGB also retrieves ordinary trees. (b) The mineral lacks the NIR rise of foliage, so the abundance query retrieves only these trees. (c) For mineral and dry leaves, SWR separates the changed crowns from healthy foliage, whereas RGB clusters mix them. (d) A spectrum clicked on shadowed limestone (Karst) also retrieves the sunlit rock, which RGB misses (recall of this click over all test views).

World	Content	Cameras	Views	Transport	Noise (e - )	Obs. (dB)
Forbidden City	palace complex inspired by the Forbidden City, with golden roofs, red walls, courtyards, vegetation and water	3 orbits of 60 at 45, 170 and 430 m, pitched down $13^{\circ}$ , $32^{\circ}$ , $60^{\circ}$	135/21/24	two-term	5,000 / 2	45.4
Market	market square with stalls and pottery, enclosed by terrain, walls and buildings	3 rings of 60, inward and downward	135/21/24	two-term	5,000 / 2	45.6
Botanical	botanical garden with planters, trees and pergolas, enclosed by terrain, walls and buildings	3 rings of 60, inward and downward	135/21/24	two-term	5,000 / 2	46.4
NVIDIA HQ	twin triangular-roof offices, glazed facades, courtyard canopy and connected walkways, parked vehicles and 300 varied trees	descending spiral, facade, courtyard and low campus routes, 60 views each	180/28/32	extended	5,000 / 2	46.7
Study	scholar’s study with timber shelves, desk, paper, ink, inkstone and ceramic vessel, 5,000 K ceiling panel	eye-level and elevated orbits, desk details	64/16/16	path traced	3,000 / 3	44.8
Karst	river valley with limestone cliffs, vegetation, soil, wood and clear water	two river-bank routes, a cliff loop and a high loop	128/32/32	two-term + water	5,000 / 2	44.3

Table S1 : The eight worlds of SWR-Bench. Views: training / validation / test. Noise: electrons at the reference radiance and read noise. Obs.: PSNR of the noisy test views against the clean ones.

	PSNR $\uparrow$					GT dict. mIoU
World	Train	Test	p95 $\downarrow$	Acc. $\uparrow$	Clean	noisy	clean
Forbidden City	29.82	28.35	14.76	0.75	–	0.55	0.94
Market	34.11	33.72	8.13	0.66	0.42	0.70	0.94
Botanical	33.11	32.92	10.48	0.65	0.38	0.73	0.93
NVIDIA HQ	30.88	25.20	25.15	0.62	–	–	–
Study	33.09	33.06	7.00	0.89	0.52	–	–

Table S2 : Further metrics of reference Spectral World Renderer on SWR-Bench (test views,

K

equal to the number of classes). Train: PSNR on training views. p95: 95th-percentile angle. Acc.: pixel accuracy. Clean: mIoU when trained on clean radiance. GT dict.: NNLS with the true class spectra on noisy / clean radiance.

World	Classes	Spectra	E / U	Classes with their number of spectra	Illumination
Forbidden City	9 (11)	47	27 / 20	yellow glazed tile 5 U ; vermilion paint 3 U ; vegetation 8; white marble 4; grey masonry 4+5; painted eaves 2+2; soil and bark 7; water 1+2; distant city haze 1+3	G173 sun + sky
Market	6 (8)	29	29 / 0	travertine 1; stone group (white marble, pink marble, granite) 7; vegetation 8; fired clay 4; sandy soil 4; wood and bark 5	G173 sun + sky
Botanical	6 (8)	29	29 / 0	travertine 1; stone group (white marble, pink marble, granite) 7; vegetation 8; fired clay 4; sandy soil 4; wood and bark 5	G173 sun + sky
NVIDIA	8	27	21 / 6	white coating/marking 2+1; glazing 1; vegetation 8; dark panels/frames 2 U ; concrete 1+2; asphalt 1+1; soil and bark 7; wood 1	G173 sun + sky
Study	5	5	3 / 2	wood 1; paper 1 U ; ink 1 U ; dark stone 1; ceramic 1	5000 K panel
Karst	5	10	10 / 0	limestone 2; vegetation 5; soil 1; wood and bark 2; water (absorption model)	G173 sun + sky

Table S3 : Material classes and measured spectra of the eight SWR-Bench worlds. Benchmark class names describe assigned material roles, which may use measured proxies. Counts deduplicate native files;

n_{\mathrm{E}}{+}n_{\mathrm{U}}

denotes ECOSTRESS and USGS records, a single number ECOSTRESS, and a subscript U USGS. Parent-label counts appear in brackets. Forbidden City uses pigment proxies for yellow tiles and some paints, measured reflectance on opaque water surfaces, and bright solid-material proxies for haze. Karst water instead uses pure-water absorption and a Fresnel interface, with no fixed reflectance endmember. City variants share City geometry, cameras and sensor.

World	Spectra	in $[0,1]$	Ref.	$<3^{\circ}$	med. ( ∘ )	Veg.
Forbidden City	47	47/47	24	22	1.28	8/8
Market	29	29/29	21	17	2.01	8/8
Botanical	29	29/29	21	17	2.01	8/8
NVIDIA	27	27/27	19	17	1.35	8/8
Study	5	5/5	2	1	6.09	–
Karst	10	10/10	9	9	0.60	5/5

Table S4 : Reflectance checks per world, over 420–980 nm. Spectra: distinct measured spectra rendered. Ref.: spectra with at least three independent library measurements of the recorded source type;

<3^{\circ}

: how many are within

3^{\circ}

of one; med.: median nearest angle. Veg.: green-leaf feature checks. Missing independent references are unevaluated, including Forbidden City pigment-to-tile, opaque-water and distant-background proxies. Counts concern source measurements, not surveyed historical materials. A shared spectrum counts in each world; summaries count each (file,type) pair once (154 pairs of 125 distinct spectra). Natural: vegetation, bark, soil and rock.

Source	Scene	Spectral World Renderer			HyperGS			DD-HGS			3DGS			UnMix-NeRF
Source	Scene	PSNR	SAM	p95	PSNR	SAM	p95	PSNR	SAM	p95	PSNR	SAM	p95	PSNR	SAM	p95
MS	NeSpoF scene1	30.74	3.82	7.12	31.38	4.66	9.46	21.77	3.61	6.65	30.68	3.63	6.82	21.77	13.70	22.79
	NeSpoF scene2	28.38	3.57	8.00	26.56	4.30	9.15	21.40	3.47	7.64	27.57	3.32	7.44	21.69	14.33	22.41
	NeSpoF scene3	29.80	4.19	9.36	26.05	5.11	10.91	25.42	4.91	10.64	25.13	5.40	11.33	20.54	8.90	15.41
	NeSpoF scene4	28.64	3.85	7.66	29.44	4.27	8.24	23.09	3.44	6.82	29.07	3.68	7.41	19.36	11.56	20.40
	Mean	29.39	3.86	8.04	28.36	4.58	9.44	22.92	3.86	7.94	28.11	4.01	8.25	20.84	12.12	20.25

Table S5 : Spectral reconstruction on real captures, per scene (test views; the main paper reports the means). MS/HS: measured multispectral/hyperspectral; 3DGS stores one channel per band. PSNR is higher-is-better; SAM and p95 angle are lower-is-better. Darker shading marks best, second and third; OOM is retained.

		Regularized, test					$K=24$ , test
World	Classes	$K{=}$ classes	16	24	32	Val	w/o reg.	3DGS
Forbidden City	9	0.410	0.500	0.538	0.552	0.563	–	0.277
Market	6	0.340	0.611	0.784	0.754	0.783	0.505	0.323
Botanical	6	0.315	0.628	0.684	0.712	0.713	0.629	0.356
NVIDIA HQ	8	0.202	0.307	0.311	0.315	0.320	–	0.199
Study	5	0.402	0.516	0.521	0.517	0.517	0.588	0.514

Table S6 : Dictionary capacity (dominant-endmember mIoU, seed 42). Test mIoU of the regularized decoder for each candidate

K

; shaded: the

K

selected on validation views, whose validation mIoU is given in column Val. Last two columns: test mIoU at

K=24

without regularization and of 3DGS with VCA/NNLS. Karst includes water.

	NeSpoF scene3		Market
Variant	PSNR $\uparrow$	SAM $\downarrow$	PSNR $\uparrow$	SAM $\downarrow$	mIoU $\uparrow$
Per-hit unmixing	29.62	4.22	33.72	1.87	0.47
One feature/Gaussian	29.24	4.30	32.09	2.08	0.34
No harmonic encoding	29.61	4.21	33.65	1.89	0.30
Deferred (pixel)	29.16	4.17	34.37	1.57	0.37
No intensity ( $s{=}1$ )	29.54	4.37	33.70	2.14	0.22

Table S7 : Decoder controls. One change at a time, without appearance regularization; means of seeds 42 and 43. Free decoders output spectra.

Figure S1 : Harmonic features and their decoding. Columns show RGB, the per-hit feature (PCA), true and predicted materials, intensity, and the variation of mixture and intensity within each Gaussian, measured against its mean feature at the same viewing direction (fixed scales across scenes).

Figure S2 : Intensity carries shading; endmembers carry material and illumination mode. Test views of Market (

K{=}24

), City (

K{=}32

) and Office (

K{=}32

). The composited intensity

\hat{s}

follows the direct share of the illumination, and each material occupies distinct endmembers in sun and in shadow, so the over-complete dictionary represents illumination modes within a class. Columns: RGB; ground-truth direct share (Office: daylight share);

\hat{s}

; ground-truth material; dominant endmember, coloured by its class and shaded by the mean direct share of its pixels; total variation (TV) of

\hat{\mathbf{A}}

when the viewing direction changes by up to

45^{\circ}

.

Figure S3 : Bounding tetrahedra on test views. Each Gaussian

G_{i}

is enclosed by the regular tetrahedron of its canonical frame, scaled by

\bm{\sigma}_{i}

, rotated by

\mathbf{R}_{i}

and translated by

\bm{\mu}_{i}

; the barycentric coordinates of the point of maximal response weight its four vertex features (Eq. 3 of the main paper). Columns show the rendering, the tetrahedra weighted by opacity and visibility, the tetrahedra coloured by the dominant class decoded at each Gaussian centre, and the rendered material map. The tetrahedra follow the scene structure, and their colouring reproduces the class layout of the rendered map: the median visible Gaussian places more than 98 % of its centre abundance on one class. The bottom row isolates one Gaussian of Market (amber window).

Dataset	Scenes	Bands	Range (nm)	Resolution	Views
NeSpoF	4	21	450–650	$512\times 428$	15–26 / 3–4
HSI-SC apple	1	204	397–1004	$512\times 512$	56 / 8
HS-NeRF (SO)	4	128	371–1044	$348\times 260$	20–42 / 3–7
HS-NeRF (BS)	1	141	401–1087	$320\times 256$	50 / 8

Table S8 : Real captures. Views are training/test views per scene; SO/BS denote Surface Optics/BaySpec sensors.

Figure S4 : Qualitative comparison on HS-NeRF test views , the first test view of each of the five scenes. Rows show false colour (630, 550 and 460 nm) and per-pixel spectral angle (left scale). Bold marks the best scores.

Figure S5 : Qualitative comparison on NeSpoF test views , the first test view of each of the four scenes, as in Figure S4 .

Figure S6 : Qualitative comparison on the HSI-SC apple , first test view, as in Figure S4 . UnMix-NeRF runs out of memory.

Method	Stage	City (s)
UnMix-NeRF	VCA initialisation and training	1,457
3DGS + VCA/NNLS	64-band 3DGS	5,030.2
	rendering and VCA	71.3
	total	5,101.5
SWR	anchor Gaussians	72.8
	harmonic features	36.5

Table S9 : City training time by stage (one RTX 4090, seed 42, including start-up and loading; ours: mean of two runs).

Method	Market	Botanical	Study	Karst	Mean
UnMix-NeRF	0.18	0.21	0.21	0.12	0.18
3DGS + VCA/NNLS	2.53	2.51	4.70	4.73	3.6
incl. training renders	1.49	1.47	2.18	2.16	1.8
SWR, reference	3.38	3.79	3.52	2.64	3.3
SWR, fast	8.19	8.81	8.09	6.34	7.9
SWR, fused	36.5	39.3	36.3	27.4	34.9

Table S10 : Rendering speed (one RTX 4090): test views per second at

960\times 540\times 64

, including spectra and labels. For 3DGS + VCA/NNLS, the second row adds the training renders that VCA needs. Ours: full precision (reference), fp16 skipping intersections of weight

<10^{-4}

(fast), and the same in one fused kernel; dominant labels agree with the reference on 99.97 % of pixels.

Figure S7 : Rare classes are absorbed by larger ones. Per-class IoU against pixel share over the eight worlds, with

K

equal to the class count. NNLS with the true class spectra separates most rare classes that Spectral World Renderer merges into larger ones.

		Lost				SWR		RGB
Seed	Targets	typed	sparse	Found	Inst.	correct	wrong	correct	wrong
Metamer: foliage-coloured mineral leaves
42	61	13	43	5	57	5	0	5	11
43	61	16	43	2	54	2	0	2	2
Stress: dry leaves on 87 trees
42	87	6	62	19	63	19	1	19	1

Table S11 : Object queries in City. Every target is either lost before the spectral query or found; among the found trees, Spectral World Renderer retrieves no ordinary tree in the metamer variant, whereas RGB retrieves up to 11. Targets: substituted (metamer) or stressed trees. Lost: typed as another material, or too few Gaussians to form an instance. Instances: all tree instances returned by the type query. Bold: fewest wrong answers.

Figure S8 : A clicked spectrum retrieves its material across shading and separates metamers. (a) A pixel on shadowed limestone is clicked in a training view of Karst; its rendered spectrum is the query. (b) Selections on three test views at the Otsu threshold for the RGB colour angle, the raw spectral angle and the whitened spectral angle on per-Gaussian spectra (white outline: ground truth); AP and recall are computed over all test views for this click. The RGB query misses the sunlit rock, which the whitened query retrieves. (c) City metamer, shown on tree crowns (outline: mineral crowns): the living-leaf spectrum retrieves the mineral crowns in RGB but not with Spectral World Renderer, and the mineral spectrum retrieves only them.

Figure S9 : Illumination handling makes spectral queries robust to shading (mean over four worlds). (a) Mean AP of RGB queries and of spectral queries without illumination handling (raw), with the sun and sky cone and with whitening; dashed line: class-name query. (b) For shadowed clicks, the RGB query retrieves the shadowed part of the class but almost none of its sunlit part, whereas whitened spectral queries retrieve both.

	AP $\uparrow$			Recall $\uparrow$
Query	library	sunlit click	shadowed click	sunlit part
RGB colour angle	.50	.61	.45	.10
Spectral angle, raw	.55	.59	.48	.53
Ours	.61	.63	.61	.77
Class name (reference)	.82	–	–	–

Table S12 : Query by spectrum (mean over Market, City, Botanical and Karst). AP against the class masks on test views. Recall: proportion of the sunlit part of the class retrieved by shadowed clicks at the Otsu threshold. Ours: illumination cone for library spectra, whitening for clicks; click queries operate on rendered pixels, and per-Gaussian queries differ by at most .02 AP.

Figure S10 : Simulated hyperspectral observations: Forbidden City, Market, Botanical and NVIDIA HQ. Narrow bands share one exposure per world, so brightness differences between bands follow the radiance; the colour-infrared composite (R, G, B = 847, 651, 553 nm) renders vegetation red. Apparent reflectance divides the clean radiance by the white-reference radiance under each pixel’s illumination (median and 10–90 % range, over the library reflectance of the assigned specimen; shaded: red edge, 690–740 nm). Outdoor spectra reproduce the library reflectance, including chlorophyll absorption near 670 nm and the red edge; path-traced interiors deviate through interreflection and residual Monte Carlo variance. Bottom: clean and noisy crops of the 900 nm band with the signal-to-noise ratio of the noisy crop.

Figure S11 : Simulated hyperspectral observations: Study, Karst, City and Office. Layout as in Figure S10 . Indoor worlds are lit by white sources, so their bands differ less; the Office lights emit no near-infrared, which comes only from daylight through the blinds.

Figure S12 : Rendered views: Forbidden City, Market and Botanical. Fifteen cameras per world, spread along its trajectory (released RGB renders).

Figure S13 : Rendered views: NVIDIA HQ, Study and Karst. Fifteen cameras per world, spread along its trajectory (released RGB renders).

Figure S14 : Rendered views: City and Office. Fifteen cameras per world, spread along its trajectory (released RGB renders).

Figure S15 : Additional test views: Forbidden City and Market. Three separated test cameras per world, shown as full frames.

Figure S16 : Additional test views: Botanical and NVIDIA HQ. Three separated test cameras per world, shown as full frames.

Figure S17 : Additional test views: Study and Karst. Three separated test cameras per world, shown as full frames.

Figure S18 : Additional test views: City and Office. Three separated test cameras per world, shown as full frames.

Spectral World Renderer: Unified 3D Hyperspectral Unmixing

Abstract

Figures & tables

Hyperspectral Unmixing

Polyhedral Unmixing: Bridging Semantic Segmentation with Hyperspectral Unmixing via Polyhedral-Cone Partitioning

Spectral Rendering Without a Spectral Renderer: Learned Spectral Codes for RGB Pipelines

Agentic Multimodal Models for Environmental Hyperspectral Unmixing

Abstract

Hyperspectral Unmixing

Explore similar work

Polyhedral Unmixing: Bridging Semantic Segmentation with Hyperspectral Unmixing via Polyhedral-Cone Partitioning

Spectral Rendering Without a Spectral Renderer: Learned Spectral Codes for RGB Pipelines

Agentic Multimodal Models for Environmental Hyperspectral Unmixing