Overview
Research introduces DriveDegrade, a benchmark developed to assess the image-degradation tolerance of camera-only end-to-end (E2E) driving models. These models are nearing deployment, operating in environments where camera streams are susceptible to various forms of degradation, including blur, noise, low light, adverse weather conditions, frame loss, and memory faults. The benchmark aims to quantify the resilience of driving policies to such corruptions before their driving capabilities are compromised. Traditional corruption-robustness benchmarks often focus on detection or bird's-eye-view perception, rather than the planning output directly responsible for vehicle control.
Research Context
The operational reliability of camera-only E2E driving models relies on their ability to maintain performance despite real-world visual input imperfections. Understanding the degradation tolerance of these models is crucial for their safe and effective deployment. The current landscape of robustness benchmarks does not specifically address the planning output of E2E driving systems under image degradation.
Approach
The DriveDegrade benchmark involves injecting 16 distinct corruption families, each at five different severity levels, directly into the image loader during evaluation. This on-the-fly injection process was applied to fifteen driving policies. The evaluation encompassed open-loop planning performance on established datasets, specifically nuScenes and NAVSIM. Additionally, a closed-loop anchor evaluation was performed using CARLA. This methodological setup allowed for a systematic assessment of how various forms and severities of image degradation impact the planning decisions of E2E driving models.
Findings
- Mild Degradation Impact: Mild image degradation was observed to have a negligible effect on the planning performance of the models tested.
- Corruption-Specific Fragility: The fragility of planning varied significantly depending on the type of corruption. Blur, JPEG compression, and simulated raindrops were identified as the corruption families that most significantly impaired planning. In contrast, adverse weather conditions and bit errors were tolerated much further into their respective severity ranges.
- Threshold for Failure: For the corruption families that caused planning failures, a clear threshold was identified at mid-severity levels, beyond which performance significantly declined.
- Distinguishing Degradation from Deletion: The benchmark separated corruptions that degrade image quality from those that effectively remove information. A flat performance curve can be ambiguous; therefore, this distinction clarifies whether a model is robust or simply insensitive to certain types of input. Planners relying on camera input are expected to lose accuracy when information is deleted, irrespective of their performance under quality degradation. This separation helped to sharply differentiate planners based on these two axes, enabling quantification of the 'ego-status shortcut' without misinterpreting indifference as robustness.
- Specific Planner Performance: A released vision-language-action planner exhibited a flat curve on both the degradation and information-removal axes. Furthermore, blinding all six of its cameras resulted in only an 11.5 percent performance reduction.
Why This Matters
The findings from the DriveDegrade benchmark offer specific insights into the resilience of camera-only E2E driving models to various types of image degradation. This characterization of degradation tolerance is directly relevant to understanding the operational limits and safety considerations for the deployment of such autonomous driving systems in real-world conditions where visual input imperfections are common.