Overview
The DiffGI (Differentiable Geometry Image) framework proposes an end-to-end 3D-to-2D mapping method designed for surface representation and geometric optimization, particularly for thin-shell and non-manifold 3D geometries. It addresses challenges associated with existing 3D generative models, such as their reliance on implicit volumetric representations that often enforce watertight topology, making them less suitable for geometries like garments. DiffGI integrates a continuous 2D Truncated Signed Distance Function (TSDF) into a geometry image-based approach, aiming to encode boundary positions with subpixel precision. This framework incorporates a differentiable Marching Squares algorithm, facilitating the propagation of gradients from 3D surface losses back to the 2D latent space.
Research Context
Existing 3D generative models primarily use implicit volumetric representations. These representations typically enforce watertight topology, which hinders their effectiveness in modeling thin-shell and non-manifold structures such as garments. While geometry image-based approaches offer a surface-centric alternative, current methods often depend on discrete binary occupancy maps. A limitation of these binary maps is their resolution-dependent boundary encoding, which can lead to staircase artifacts. Additionally, downsampling discrete binary maps may result in information loss. Furthermore, in many existing geometry image methods, surface reconstruction is a non-differentiable post-processing step, separating it from the core learning pipeline.
Approach
DiffGI addresses the limitations of prior geometry image-based methods by replacing discrete binary occupancy maps with a continuous 2D Truncated Signed Distance Function (TSDF). This continuous field encodes boundary position at subpixel precision within a fixed grid resolution. This design choice aims to eliminate resolution-dependent staircase artifacts, even under aggressive downsampling. Building upon this continuous field, the framework introduces a differentiable Marching Squares algorithm. This algorithm is based on analytical linear interpolation, which enables gradients from 3D surface losses to propagate back to the 2D latent space. This differentiable pipeline subsequently allows for the training of a DiffGI-VAE, which is augmented with a geometry-aware normal rendering loss. The objective of this VAE is to compress complex 3D surfaces into an ultra-compact $32 \times 32$ latent space. On top of this latent space, a transformer-based latent diffusion model is instantiated, utilizing a flow-matching objective for conditional 3D generation.
Findings
- DiffGI replaces discrete binary occupancy maps with a continuous 2D Truncated Signed Distance Function (TSDF), which encodes boundary position at subpixel precision within a fixed grid resolution. This eliminates resolution-dependent staircase artifacts, even under aggressive downsampling.
- The introduced differentiable Marching Squares algorithm, based on analytical linear interpolation, enables gradients from 3D surface losses to propagate back to the 2D latent space.
- A DiffGI-VAE augmented with a geometry-aware normal rendering loss can compress complex 3D surfaces into an ultra-compact $32 \times 32$ latent space.
- A transformer-based latent diffusion model, instantiated on this latent space with a flow-matching objective, facilitates conditional 3D generation.
- Extensive experiments on garment and object datasets indicate that the DiffGI method achieves superior reconstruction fidelity compared to prior geometry-image and voxel-based approaches.
- The method demonstrates superior boundary precision when compared to prior geometry-image and voxel-based approaches.
- The DiffGI method requires significantly fewer computational resources compared to prior geometry-image and voxel-based approaches for complex 3D surface modeling.
Why This Matters
Existing 3D generative models often struggle with thin-shell and non-manifold geometries due to their reliance on implicit volumetric representations. DiffGI's continuous representation and differentiable reconstruction address artifacts and information loss issues inherent in discrete geometry image methods. This approach offers enhanced fidelity and computational efficiency in generating complex 3D surfaces.