Pixel Grimoire My learning blog

Summary of SIGGRAPH/HPG 2024

• •

Below is a summary of 2024 SIGGRAPH attendance in Denver, CO.

0. Terminology

NeRF

See Details [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://www.matthewtancik.com/nerf)

SLAM

See Details [Simultaneous localization and mapping](https://www.mathworks.com/discovery/slam.html)

3D Gaussian Splatting (3DGS)

See Details [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/)

1. Technical Papers

Best Papers

List **From Microfacets to Participating Media: A Unified Theory of Light Transport With Stochastic Geometry** Authors: Dario Seyb, Eugene d’Eon, Benedikt Bitterli, Wojciech Jarosz Category: Appearance models
Summary We derive a theory of light transport on stochastic implicit surfaces, a geometry model capable of expressing deterministic geometry, microfacet surfaces, participating media, and an exciting new continuum in between containing aggregate appearance, non-classical media, and more. Our model naturally supports spatial correlations, missing from most existing stochastic models.

**Walkin’ Robin: Walk on Stars With Robin Boundary Conditions** Authors: Bailey Miller, Rohan Sawhney, Keenan Crane, Ioannis Gkioulekas Category: Monte Carlo for PDEs
Summary We develop a grid-free Monte Carlo method for solving boundary value problems like the Poisson equation with Dirichlet, Neumann, and Robin boundary conditions. Unlike conventional PDE solvers, our method does not require volumetric meshing or global solves. It is robust, embarrassingly parallel, scalable to complex geometry, and allows view-dependent evaluation.

**Repulsive Shells** Authors: Josua Sassen, Henrik Schumacher, Martin Rumpf, Keenan Crane Category: Geometry - Editing and Deformation
Summary Shape spaces are a powerful tool for nonlinear interpolation, extrapolation, and averaging of geometric data, but previous shape spaces permit geometry to self-intersect in nonphysical ways. We introduce a shape space where geometry naturally avoids intersection, as well as an adaptive collision potential that prevents collision while converging under refinement.

**Lightning-Fast Method of Fundamental Solutions** Authors: Jiong Chen, Florian Schäfer, Mathieu Desbrun Category: Fluids
Summary This work introduces a variational preconditioner, based on the inverse Cholesky factorization, to improve the efficiency of solving dense systems discretized from boundary integral equations, effectively addressing the scalability issue commonly encountered in boundary-based approaches.

**Robust Containment Queries Over Collections of Rational Parametric Curves via Generalized Winding Numbers** Authors: Jacob Spainhour, David Gunderman, Kenneth Weiss Category: Vector Graphics
Summary We extend the theory of generalized winding numbers to unstructured collections of rational parametric curves with a numerically stable algorithm, thereby allowing for robust and accurate containment classifications at arbitrary locations for non-watertight and self-intersecting shapes.

Honorable Mentions

List **Solid Knitting** Authors: Yuichi Hirose, Mark Gillespie, Angelica M. Bonilla Fominaya, James McCann Category: 3D Fabrication **PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR Displays** Authors: Kenneth Chen, Thomas Wan, Nathan Matsuda, Ajit Ninan, Alexandre Chapiro, Qi Sun Category: VR, Eye Tracking, Perception **CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets** Authors: Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, Jingyi Yu Category: Generative 3D Geometry and Editing **DressCode: Autoregressively Sewing and Generating Garments From Text Guidance** Authors: Kai He, Kaixin Yao, Qixuan Zhang, Jingyi Yu, Lingjie Liu, Lan Xu Category: Clothing Geometry **Bilateral Guided Radiance Field Processing** Authors: Yuehao Wang, Chaoyi Wang, Bingchen Gong, Tianfan Xue Category: Radiance Field Processing **Fabric Tessellation: Realizing Freeform Surfaces by Smocking** Authors: ​​Aviv Segall, Jing Ren, Amir Vaxman, Olga Sorkine-Hornung Category: 3D Fabrication **Capacitive Touch Sensing on General 3D Surfaces** ​Authors: ​Gianpaolo Palma, Narges Pourjafarian, Jürgen Steimle, Paolo Category: Virtual Interaction and Real Devices **SMERF: Streamable Memory Efficient Radiance Fields for Real-time Large-scene Exploration** Authors: Daniel Duckworth, Peter Hedman, Christian Reiser, Peter Zhizhin, Jean-François Thibert, Mario Lučić, Richard Szeliski, Jonathan T. Barron Category: Fast Radiance Fields **Spin-It Faster: Quadrics Solve All Topology Optimization Problems That Depend Only on Mass Moments** Authors: Christian Hafner, Mickaël Ly, Chris Wojtan Category: Geometric Modeling & Analysis **Ray Tracing Harmonic Functions** Authors: Mark Gillespie, Denise Yang, Mario Botsch, Keenan Crane Category: Rendering, Sampling, and Tracing **Seamless Parametrization in Penner Coordinates** Authors: Ryan Capouellez, Denis Zorin Category: Geometry - Mapping and Fields **Theory of Human Tetrachromatic Color Experience and Printing** Authors: Jessica Lee, Nicholas Jennings, Varun Srivastava, Ren Ng Category: Perception, Image, Video


Category - Offline Rendering

Rendering, Sampling and Tracing

  • ReSTIR
    • Area ReSTIR, extending ReSTIR reservoirs to also integrate each pixel’s 4D ray space, including 2D areas on the film and lens. We design novel subpixel-tracking temporal reuse and shift mappings that maximize resampling quality in such regions. This robustifies ReSTIR against high-frequency content, letting us importance sample subpixel and lens coordinates and efficiently render antialiasing and depth of field.
      • e.g., subpixel location (𝑢, 𝑣) and aperture (𝑠, 𝑡).
    • Interleaving Markov Chain Monte Carlo (MCMC) mutations with reservoir resampling helps alleviate these issues, especially in scenes with glossy materials and difficult-to-sample lighting.
      • Problems with ReSTIR include such as spatiotemporal resampling often introduces noticeable correlation artifacts, while reservoirs holding more than one sample suffer from impoverishment in the form of duplicate samples.
      • Moreover, our approach does not introduce any bias, and in practice, we find considerable improvement in image quality with just a single mutation per reservoir sample in each frame.

Real-Time Rendering and Gaming (from Advances opening remarks)

Player Expectations

  • Narrative-driven experiences
  • Cross-platform plays
  • Live service models
  • Scale and immersion
  • Low Latency > Fidelity in multiplayer games
  • High demand for more & better in Gfx
    • 4K 120FPS in next
  • Getting high fidelity from less
  • Untethered devices are trending

Gaming Technology & Platform Trends

  • Increase HW fragmentation
    • Min-spec is not rising anywhere fast enough.
    • Gen8-level specs aren’t going anywhere.
  • Desire for fast runtime GI
    • Continuous effort to improve fast runtime GI and provide fast baking iteration.
    • Scalability for platform ecosystem is a must.
  • Atomic rendering primitives optionally abounds
    • Splats, triangles, rays have been converging and intermixing.
  • More interest in spatio-temporal upscaling and VRS
    • Extreme pressure on frame cost generates stronger interest in frame resue + larger resolutions drive cost up.
  • Dense large levels proliferate
    • GPU-driven has been adopted and proven a win.
    • Visibility buffer gaining ground.
    • CPU/GPU work graphs and work generation on GPUs gaining ground.
    • Example - Brainerd: Tessellation in CoD: Ghosts (SIGGRAPH Advances 2014)
    • Challenge: Disk footprint
  • Hybrid ray tracing has taken a foothold and is not going anywhere
    • Continuous effort to improve fast runtime GI and provide fast baking iteration.
    • Scalability for platform ecosystem is a must.
      • E.g. Denoiser
  • Continued evolution of compute based graphics is exciting
    • From GPU-driven pipeline workloads, spatio-temporal upscaling, to ML-based algorithms execution on device, we’re generalizing CS-based workflows more and more.

Runtime Machine Learning

  • ML techniques are well suited for compression of high-dimensional datasets for runtime use
    • Simulation: Animation/Deformation
    • Global illumination
    • VFX and etc.
  • ML upscalers proliferate but beware of blackbox systems
    • IHVs are excited about the value ML upscalers provide.
    • Developers are challenged by blackbox systems and need for custom training.
  • ML-based animation and deformation is the new SSAO
    • Tricky to get right, forever chasing minute artifacts;
    • Content-dependent;
    • Yet once there, you can’t go back
  • ML programming model still evolving
    • NPU programming models are not crisp yet.
    • Much ML innovation at runtime executes on cross-platform compute shaders.
  • Consumer-level GPU
  • Rendering tasks other than post-processing
  • Thread-scale problems
    • Rather than GPU-scale problems
  • Small neural networks, by design
    • For performance reasons

Examples from recent NV research

Neural Texture Compression (2023)

  • Small MLP to decompress stacks of material properties textures
    • 1 MLP network per material texture stack
    • 2 hidden layers
    • 4-16x compression improvement over standard BC (block compressed textures)

Real-Time Neural Apprearance Models (2024)

  • Small MLPs to represent layered material params and approximate scattering math
    • 2 MLP networks (BRDF eval & importance sampling) per material.
    • 2-3 hidden layers
    • 2-4x performance improvement

NeuralVDB (2024)

  • Represent both compressed volume data and sparse tree topology
    • 2-4 MLP networks per volume
    • 3-4 hidden layers
    • 10-100x compression ratio

Neural Radiance Cache (2021)

  • Neural network encodes sampled radiance information
    • 1 MLP network per probe
    • 2 hidden layers
    • Radiance probes updated (training) and queried (inference) dynamically

Simple MLPs are surprisingly powerful

  • Data compression
  • Approximation of complex math
  • Caching of complex signal data

Performance can be competitive with traditional rendering

  • Layer fusion
  • Leveraging reduced precision and sparsity

Challenges

  • Divergence
    • E.g. Each SM is querying on its own networks, or own texel values, etc.
  • Intermix with traditional shader code

Even MLP examples are leveraging enacode/decoder pairs and learned, latent space representations.

Will AI just render the whole image?

  • Questions to ask ourselves
    • What will be the source for high-quality training data?
    • How will we make it:
      • Stable
      • Reproducible
      • Interactive
      • Real-time
    • How will it interact with traditional rendering?
  • Example: OpenAI video created with Sora (2024)
    • Implicit understanding of physics, e.g. dust cloud
    • Generated from a text prompt
    • Probably not the solution to our question

Creation, Representation, Rendering

  • AI will disrupt all industries and fields of study
    • Content creation and complexity
    • Render performance and quality

4. Labs and Exibitions

Oppo booth demos

  • Real-time GI
    • Similar to UE5’s “Valley of the Ancients” demo with GI/day-night cycle/light shafts/etc
    • Hybrid ray march
  • Real-time volumetic cloud rendering
    • Raymarched volumetric approach
    • Used in their weather app as a proxy for cloud cover