From 3D to 4D Gaussian Splatting
The starting point is 3D Gaussian Splatting (3DGS), presented at SIGGRAPH 2023 by a team from Inria and the Max Planck Institute for Informatics. It represents a static scene as a cloud of 3D Gaussians: each one has a position, a shape (scale and rotation), an opacity and a color that changes with the viewing angle.
Those Gaussians are fitted from photographs taken from many viewpoints and drawn directly on screen, without polygon meshes. That is why the result looks photorealistic and can be explored in real time.
4DGS adds time. Since late 2023 several formulations have appeared: some deform a set of Gaussians over the take, others use Gaussians with a temporal dimension. Either way, the result is a dynamic scene that plays back like a video in which the viewer chooses the viewpoint.
What is Gaussian SplattingHow a 4D scene is captured
A moving scene cannot be photographed by walking a camera around it, because the subject changes at every instant. It takes a rig of many synchronized cameras recording at the same time from every angle.
At Dymensium's studio in Valencia we use a rig of 120 synchronized cameras. The takes are then processed to reconstruct every instant and generate the 4DGS scene.
How it is played back
A 4DGS scene is drawn in real time on the graphics card. Today it can be watched in the browser with WebGL or WebGPU viewers, in engines such as Unreal Engine or Unity through plugins, and on virtual reality headsets and augmented reality devices.
Data size is the main challenge. A dynamic scene takes far more space than a static model, so each destination needs its own compression and optimization.
4DGS compared with other techniques
These are the differences with the techniques closest to it:
- Mesh-based volumetric video
- Turns every frame into a textured polygon mesh. It fits classic 3D pipelines easily, but simplifies hair, fabric and transparency.
- Photogrammetry
- Reconstructs textured geometry from photos of a still subject. It is useful for measuring or 3D printing, but it does not capture motion and struggles with reflections and thin parts.
- NeRF
- Represents the scene with a neural network. Quality is high, but rendering is slow; Gaussian Splatting reaches comparable quality in real time.
- 360 video
- Records from a single point: the viewer can look around but cannot move. In 4DGS they can change position and walk around the performer.
What it is used for
Film and VFX, virtual production, video games, advertising and fashion, sports, education and performing arts. In every case, the value is showing real people in motion with a free camera, without animating them by hand.
Strengths and limitations
Strengths
- Photorealistic look, including hair, fabric and soft edges.
- Free camera: the scene can be seen from any angle.
- Real-time playback on the web, in virtual and augmented reality, and in 3D engines.
Limitations
- Lighting is recorded with the scene, so it has to be planned around where the capture will be integrated.
- Files are heavy and need optimizing for each platform.
- It is a recent technique: formats and tools are still evolving.
References
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering
Kerbl, Kopanas, Leimkühler, Drettakis. ACM Transactions on Graphics 42(4), SIGGRAPH 2023.
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
Wu et al. CVPR 2024.
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
Yang et al. ICLR 2024.