Overview
RECAST (REconstructing Controllable Actors for Simulation and Testing) is a framework designed to facilitate closed-loop driving simulation. It leverages 3D Gaussian Splatting to construct view-complete actors from individual segmented vehicle observations within a driving log. The framework integrates these generated actors into a reconstructed scene. A key objective of RECAST is to enable planner-in-the-loop rendering, particularly under controlled interactions between the ego vehicle and other actors.
The development of RECAST addresses challenges inherent in closed-loop driving simulation, where rendered observations must maintain reliability even as the ego vehicle and surrounding actors deviate from their recorded trajectories. Such deviations expose viewpoints not present in the original source log, a scenario where existing data-driven simulators, which reconstruct dynamic actors from sparse observations, can produce rendering artifacts.
Research Context
Closed-loop driving simulation demands that rendered visual information remains consistent and accurate as simulated agents, specifically the ego vehicle and surrounding actors, maneuver beyond their pre-recorded paths. This movement often reveals angles and perspectives of actors that were not captured in the initial logged data. Conventional data-driven simulators typically reconstruct dynamic elements from limited observations, which can lead to visual inconsistencies or artifacts when these previously unobserved viewpoints are encountered.
Approach
RECAST utilizes a 3D Gaussian Splatting framework for actor generation. It takes a single segmented vehicle observation from a driving log and uses it to generate a view-complete actor. This actor is then registered within the reconstructed simulation scene. To enhance the adaptation of an image-to-3D prior for real vehicles, the researchers introduced RECAR, a dataset comprising approximately 20,000 real vehicles. This dataset includes 600,000 background-free RGBA images, encompassing a variety of vehicle colors and types.
The system employs a two-stage adaptation process to improve the generation of vehicles from real driving-log observations. This adaptation is critical for ensuring the realism and reliability of the reconstructed actors in simulation environments.
Findings
The evaluation of RECAST involved several metrics and comparisons:
- Actor-level performance: RECAST demonstrated an improvement in the Fréchet Inception Distance ($\text{FD}_{\text{incep}}$) metric at the actor level. It reduced $\text{FD}_{\text{incep}}$ from 9.788 to 7.992 when compared to unadapted TRELLIS.
- Scene-level performance: Under conditions where actor motion extended beyond logged trajectories, RECAST achieved a reduction in $\text{FD}_{\text{incep}}$ from 129.35 to 112.10 relative to Street Gaussians. Concurrently, it increased the $\text{CLIP}_{\text{margin}}$ ($\times1000$) from 0.14 to 3.47 compared to Street Gaussians.
- Planner-in-the-loop simulation: RECAST was used in a planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. In this context, RECAST increased the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) when compared with native Street Gaussians actors. The mean minimum predicted time-to-collision (TTC) also improved from 0.798 seconds to 2.150 seconds.
These experimental results collectively suggest that RECAST supports the evaluation of closed-loop planners, particularly in scenarios involving controlled ego-actor interactions that extend beyond simple log replay.
Why This Matters
The ability of RECAST to generate view-complete actors and integrate them into simulated environments under conditions of unrecorded motion is relevant for the development and testing of autonomous driving systems. Its demonstrated improvements in simulation metrics and collision rates indicate its utility for evaluating planner performance in dynamic, interactive scenarios beyond basic log replay.