OmniRen: Neural Rendering wih Heterogeneous Scene Primitives

We present ’RenderFormer-V2’, a unified learned transformer-based neural rendering model, complementary to modern physics-basedrendering systems, that can handle diverse light-transport effects suchas caustics, volumetric scattering, environment lighting, textured anddisplaced surfaces and out-of-distribution materials without per-scenetraining or specialized code. RenderFormer-V2 models global light trans-port as a sequence-to-sequence transformation. Following its predecessor,RenderFormer-V2 also employs a two stage process: a view-independentstage that resolves intra-scene primitive to primitive transport, and aview-dependent stage that transforms the internal neural scene repre-sentation into image pixels. Different from RenderFormer, our modelemploys a novel combined windowed-attention and rendering-informedattention sink in the view-independent stage to improve scalability whilemaintaining render accuracy. To further improve versatility, RenderFormer-V2 supports heterogeneous scene primitives, including environment mapsand participating media, and it employs a material encoding indepen-dent of the underlying surface reflectance model that encodes materialappearance via a novel neural embedding. We demonstrate the versatil-ity of RenderFormer-V2 on a variety of scenes and perform an extensiveablation of the improved attention mechanism.