ICANEWS

Engineering & Technology

Evaluating Test Selection Metrics for Deep Learning Under Diverse Objectives and Distribution Shifts

Deep learning (DL)-based systems are increasingly integrated into critical applications, ranging from malware detection to autonomous driving. While these syste

Comprehensive Evaluation of Deep Learning Test Selection Metrics Underpinning System Reliability

Deep learning (DL)-based systems are increasingly integrated into critical applications, ranging from malware detection to autonomous driving. While these systems offer advanced capabilities, their behavior can become unpredictable when confronted with scenarios that differ from their training data, known as out-of-distribution (OOD) scenarios. This inherent susceptibility to OOD conditions poses significant risks, particularly in safety-critical domains where unexpected behaviors can have severe consequences. Consequently, a thorough and rigorous testing phase is paramount before such DL-based systems are deployed to ensure their reliability and robustness in real-world environments.

The Crucial Role of Test Selection Metrics

To facilitate effective testing of DL-based systems, researchers have developed a broad array of test selection metrics. These metrics are specifically designed to intelligently choose inputs that can effectively reveal potential weaknesses or provide valuable insights into system performance. The goal of these metrics is to optimize the testing process, ensuring that the selected test cases are representative, challenging, and capable of exposing faults or accurately estimating performance.

Despite the development of numerous test selection metrics, previous evaluations of their effectiveness have often been limited, exhibiting specific shortcomings that can hinder their practical application. These limitations have created a gap in understanding which metrics are most appropriate for different testing requirements and operational contexts.

Identifying Prior Evaluation Gaps and Research Objectives

A recent study, detailed in arXiv:2604.23342, has highlighted three primary limitations in previous evaluations of test selection metrics for deep learning systems:

  1. Narrow Testing Objectives: Many existing studies have focused on a restricted set of testing objectives. For example, a significant portion of research has evaluated metrics solely based on their ability to detect faults. This narrow focus leaves a critical question unanswered: how effective are these metrics when the objective is performance estimation, or when they are used to guide retraining processes? The effectiveness of a metric can vary substantially depending on the specific goal of the testing procedure.
  2. Limited Coverage of OOD Scenarios: The unpredictability of DL systems often manifests in OOD scenarios. However, prior evaluations have typically presented limited coverage of these diverse OOD conditions. Specifically, out-of-distribution scenarios such as natural shifts and label shifts have been rarely considered. Without evaluating metrics under a wide range of OOD conditions, their generalizability and resilience in real-world deployments, which are inherently dynamic and unpredictable, remain uncertain.
  3. Biased Dataset Selection: Another significant limitation identified is the biased selection of datasets used for evaluation. Most research in this area has predominantly focused on image data. While image data is important, this exclusive focus means that other data modalities, such as text and Android packages, remain largely underexplored. The performance of test selection metrics can be modality-dependent, and thus, evaluations that ignore non-image data provide an incomplete picture of their overall utility.

These identified limitations collectively indicate the absence of a unified benchmark. Such a benchmark would comprehensively examine the performance of test selection metrics across multiple testing objectives, diverse OOD scenarios, and various data modalities. This deficiency leaves practitioners in a state of uncertainty, making it difficult for them to determine which test selection metrics are most suitable for their specific objectives and application contexts.

Addressing the Gaps: A Comprehensive Empirical Study

To address the aforementioned gaps and provide clarity for practitioners, the research presented in arXiv:2604.23342 details an extensive empirical study. This study was meticulously designed to offer a unified and comprehensive evaluation of test selection metrics.

"To address this gap, we conduct an extensive empirical study of 15 existing metrics, evaluating them under three testing objectives (fault detection, performance estimation, and retraining guidance), five types of OOD scenarios (corrupted, adversarial, temporal, natural, and label shifts), three data modalities (image, text, and Android packages), and 13 DL models. In total, our study encompasses 1,640 experimental scenarios, offering a comprehensive evaluation and statistical analysis."

Scope of the Empirical Study

The study encompassed a substantial research scope, systematically investigating 15 existing test selection metrics. The comprehensiveness of the study can be understood through its multiple dimensions:

  • Number of Metrics Evaluated: The study quantitatively assessed the performance of $15$ distinct test selection metrics. This large number ensures a broad understanding of the current landscape of available metrics.
  • Testing Objectives: Unlike previous studies that often focused narrowly, this research evaluated metrics under three critical testing objectives:
    • Fault Detection: Assessing a metric's ability to identify errors or defects within the DL system.
    • Performance Estimation: Evaluating how well a metric can predict or estimate the overall performance characteristics of the system.
    • Retraining Guidance: Determining the utility of a metric in guiding the process of retraining a DL model, potentially by selecting data points most beneficial for improving the model.
  • Types of OOD Scenarios: To ensure robust evaluation under diverse real-world conditions, the study considered five distinct types of OOD scenarios:
    • Corrupted Shifts: Scenarios where input data is artificially corrupted.
    • Adversarial Shifts: Scenarios involving inputs specifically crafted to mislead the DL model.
    • Temporal Shifts: Scenarios where data distribution changes over time.
    • Natural Shifts: Scenarios reflecting naturally occurring variations in input data.
    • Label Shifts: Scenarios where the distribution of labels changes.
  • Data Modalities: To overcome the bias towards image data, the study incorporated three different data modalities:
    • Image: Traditional visual data, widely used in deep learning.
    • Text: 언어-based data, relevant for natural language processing applications.
    • Android Packages: A unique modality, crucial for applications like malware detection, expanding beyond typical data types.
  • Deep Learning Models: The investigation was further diversified by employing $13$ different Deep Learning models, ensuring that the findings are not specific to a single model architecture or type.

In summation, the study executed a total of $1,640$ experimental scenarios. This extensive undertaking allowed for a comprehensive evaluation of the test selection metrics across a wide array of conditions, followed by a thorough statistical analysis of the results. The scale and breadth of this empirical work provide a foundational step towards a more unified understanding of test selection metrics.

Implications for Practitioners

The explicit aim of this comprehensive evaluation is to resolve the uncertainty faced by practitioners regarding the most suitable test selection metrics. By providing empirical insights into the performance of 15 existing metrics under varied testing objectives, diverse OOD scenarios, and different data modalities, the study equips practitioners with the necessary information to make informed decisions. This allows them to align their metric selection with their specific testing objectives and the unique contexts of their DL system deployments.

The findings from this extensive study are expected to guide the selection of appropriate test selection metrics, thereby improving the rigorousness and efficiency of testing DL-based systems. This, in turn, contributes to enhancing the reliability and safety of deep learning applications, particularly in domains where unexpected behavior carries high stakes.

Key Findings