Overview
Vision foundation models, despite their empirical success in generating high-fidelity estimates across 3-dimensional (3D) scenes, exhibit performance degradation when applied to wide field-of-view (FoV) images, such as those captured by fisheye cameras. This degradation stems from a covariate shift introduced by the radial distortion inherent in fisheye image pixels. A proposed methodology addresses this challenge by enabling vision foundation models to generalize effectively to fisheye camera inputs.
Research Context
Vision foundation models typically undergo training on large-scale datasets predominantly comprising perspective images. This training paradigm equips them with robust generalization capabilities for standard 3D scenes. However, the distinct radial distortion characteristic of fisheye lenses creates a fundamental difference in image geometry compared to perspective images. This discrepancy, identified as a covariate shift, results in erroneous outputs when pre-trained models are directly applied to fisheye imagery, highlighting a gap in their current applicability to diverse imaging modalities.
Approach
The core of the proposed method involves a set of learnable parameters designated as Distortion Extenders (DEX). DEX is engineered to model two key aspects: the fisheye distortion coefficients and the distributional shift observed between fisheye and perspective images within the latent space of the foundation models. The operational mechanism of DEX involves transforming the latent embeddings derived from fisheye images. This transformation aims to make these embeddings resemble those generated from perspective images. This alignment is achieved by minimizing a self-supervised alignment loss function.
The design of DEX emphasizes its flexibility. It is explicitly described as being architecture-agnostic, meaning it can integrate with different underlying model architectures. Furthermore, it is task-agnostic, indicating its applicability across various vision tasks without specific modifications.
Findings
- DEX successfully generalizes vision foundation models to fisheye cameras by mitigating the covariate shift caused by radial distortion.
- The method recovers high-fidelity estimates from fisheye images by transforming their latent embeddings to align with those of perspective images.
- DEX demonstrated consistent performance improvements over baseline methods.
- Empirical validation of DEX included its application to both monocular depth estimation and open-vocabulary segmentation tasks.
- The approach was tested on both convolution-based and Transformer-based architectures, indicating broad architectural compatibility.
- Performance enhancements were observed across indoor and outdoor fisheye datasets.
- A secondary finding indicated that the activations of DEX could be decoded to yield distortion coefficients, thereby supporting camera calibration processes.
Why This Matters
The ability to adapt vision foundation models to fisheye camera data is significant given the prevalent use of such cameras in applications requiring wide fields of view. By enabling these models to process distorted images without substantial re-training or architecture changes, the method expands their utility. The capacity to derive distortion coefficients from DEX activations further provides a practical tool for camera calibration, enhancing the accuracy of spatial measurements derived from fisheye imaging systems.