STAR-NT: Spatiotemporal Acceleration of Real-Time Neural Transparency Rendering

Time-varying implicit neural representations (INRs) provide a compact representation of scientific volumes and, for modalities such as dynamic X-ray computed tomography (CT), are often the only practical way to represent the data. However, interactive volume rendering of INRs is challenging, as cheap memory lookups are replaced by expensive neural inferences, hindering the performance. Therefore, conventional volume rendering methods such as ray marching with dense sampling are often impractical. While resampling, caching, and retraining can mitigate this cost, they compromise convenience and accuracy and become impractical for time-varying data. We tackle these challenges using a query-efficient stochastic volume rendering framework based on delta tracking. Our system employs a four-stage pipeline that exploits heterogeneous parallelism, using ray tracing cores for traversal and tensor cores for batched neural evaluation. Furthermore, we present strategies to reduce INR queries via ray budgeting and query pruning, thereby increasing per-frame performance. Using our renderer, many time-varying INRs can be rendered directly from their original representation. The system achieves ~30-40 FPS at 1024x1024 resolution on an RTX 4090 GPU and converges to high-fidelity images. Moreover, the system enables interactive temporal exploration of the continuous domain, with timestep updates taking approximately 1-2 ms.

Spiking Neural Networks (SNNs) provide an energy-efficient computing paradigm for neural rendering, but existing spike-based Neural Radiance Field (NeRF) models usually use a fixed inference time step for all scenes. This fixed temporal budget is inefficient because NeRF follows a scene-specific training paradigm, and different scenes require different temporal capacities to preserve rendering quality. This paper proposes Pretraining-based Adaptive Time-step Adjustment (PATA), a scene-wise adaptive time-step training framework for spike-based NeRF. PATA parameterizes the target inference time step as a trainable variable and optimizes it through a two-stage training process. A hybrid input mode strengthens early time-step outputs, while full-step soft supervision, smoothed rendering loss, and temporal-budget loss jointly maintain rendering fidelity and reduce temporal computation. The learned target time step is shared by all ray samples within a scene, preserving the parallel rendering structure of NeRF. Experiments on INGP-NeRF and TensoRF backbones across Synthetic-NeRF, Mip-NeRF 360, and LLFF show that PATA consistently reduces inference cost while maintaining competitive rendering quality. PATA reduces the estimated inference energy by up to 57.57% on INGP-NeRF and 68.90% on TensoRF, demonstrating its effectiveness across different neural rendering representations.

Achieving a desirable balance between rendering quality and real-time performance is a long-standing challenge in modern game and rendering engines, particularly on resource-constrained mobile devices such as laptops, tablets, and smartphones. Existing approaches to automatic rendering parameter optimization either depend on exhaustive per-scene pre-computation that spans several days, suffer from the prohibitive inference overhead of neural networks that prevents per-frame adaptation, or lack generalizability across heterogeneous hardware and diverse scenes. In this paper, we propose \textbf{LUT-Opt}, a lightweight, general-purpose framework for adaptive per-frame rendering parameter optimization. Our method decomposes the joint optimization of rendering time and image quality into a tractable two-stage pipeline. In the offline stage, we train a pair of XGBoost regressors to predict rendering time and image quality from rendering parameters, hardware state, and scene complexity descriptors. The trained ensemble models are then distilled into compact lookup tables (LUTs) through systematic discretization and a two-phase linear search that first constrains rendering time and subsequently maximizes structural similarity (SSIM). During runtime, the pre-computed LUT is queried every frame in sub-millisecond time, enabling truly adaptive parameter selection with negligible computational overhead. We validate LUT-Opt on two representative rendering techniques -- subsurface scattering (SSS) and hybrid-pipeline ambient occlusion (AO) -- implemented within Unreal Engine 5. Extensive experiments across multiple scenes and GPU configurations demonstrate that LUT-Opt reduces subsurface scattering rendering time by approximately 40% and ambient occlusion rendering time by roughly 70%, while incurring only about 2% increase in image quality error, with per-frame inference latency below 0.1\ ms.