What is Gaussian Splatting

Where it comes from

3D Gaussian Splatting (3DGS) was presented at SIGGRAPH 2023, in a paper by a team from Inria and the Max Planck Institute for Informatics. It came after neural radiance fields (NeRF, 2020), which already produced photorealistic images from photos but were slow to render.

The word splatting comes from computer graphics: a way of drawing in which each element is projected onto the screen as a blob. Here the blobs are shaped like a Gaussian, the bell curve from statistics. Hence the name.

How it works

1. 01Many photographs of the subject or space are taken from different angles.
2. 02A process called structure from motion works out where each photo was taken from and produces an initial point cloud.
3. 03Each point becomes a Gaussian. An optimization process draws the scene, compares it with the photos and adjusts the Gaussians over and over until they match.
4. 04During that fitting, Gaussians are added where detail is missing and removed where they are not needed. In the end a scene can hold millions.
5. 05To view it, the Gaussians are sorted by depth and the graphics card draws them blending their transparency, in real time.

What each Gaussian stores

Position

Where it sits in space.

Shape

A scale and a rotation that make it a more or less elongated ellipsoid.

Opacity

How much of what is behind it shows through.

Color

A color that changes with the viewing angle, stored with spherical harmonics. That is how highlights and reflections are reproduced.

In the original uncompressed format each Gaussian takes about 250 bytes. With millions of them a scene can weigh hundreds of megabytes, which is why compressed formats exist for the web and mobile devices.

Compared with other techniques

Photogrammetry

Produces a textured polygon mesh. It gives geometry useful for measuring or 3D printing, but simplifies hair, reflections and transparency.

NeRF

Represents the scene with a neural network. Quality is high but rendering is slow. Gaussian Splatting reaches comparable quality in real time.

Laser scanning

Measures geometry very precisely, but does not reproduce appearance with the same realism.

Limits to keep in mind

• It does not produce a closed surface. If a project needs a mesh for physics simulation or 3D printing, it is combined with other techniques.
• The lighting at capture time is stored in the color, so it should be planned around where the result will be integrated.
• Files are large and need compression for the web and mobile devices.
• Editing tools and standards are still evolving.

With motion: 4D Gaussian Splatting

Add time and the technique reconstructs people and scenes in motion from a rig of synchronized cameras. This is known as 4D Gaussian Splatting, and it is the basis of the volumetric video we record in our studio.

Where it can be viewed

In the browser, with web engines such as PlayCanvas, Babylon.js or three.js; in Unity and Unreal Engine, through plugins; and on virtual reality headsets. Each destination has its own format and optimization.

References

1. 3D Gaussian Splatting for Real-Time Radiance Field Rendering

   Kerbl, Kopanas, Leimkühler, Drettakis. ACM Transactions on Graphics 42(4), SIGGRAPH 2023.

2. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

   Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, Ng. ECCV 2020.

3. Splat File Formats

   PlayCanvas Developer Documentation.