Overview
Research introduces EXE-Bench, a comprehensive benchmark designed to evaluate AI-based Windows malware detectors. The benchmark aims to provide a systematic assessment for determining the suitability of such detectors for production environments by considering performance, temporal robustness, adversarial resilience, and computational overhead. Through this framework, EXE-Bench aggregates these evaluations into a single score, facilitating direct comparison of models.
Research Context
Existing evaluations of AI-based Windows malware detectors exhibit several limitations. These include variability in the data used for both training and testing, a lack of temporal analysis to assess model endurance over time, and an absence of security evaluations against adversarial attacks, which could expose brittleness to content-injection attacks. Furthermore, current evaluations often neglect the computational requirements for deployment, potentially leading to slow inference on endpoints.
These identified gaps suggest that evaluations conducted solely post-deployment may not offer a complete picture of a detector's performance. The absence of systematic methodologies to compare detectors, particularly concerning their real-world applicability and resilience, motivated the development of EXE-Bench.
Approach
EXE-Bench was developed to systematically evaluate AI-based Windows malware detectors across four specific dimensions:
- Performance: Standard assessment of detection accuracy.
- Temporal Robustness: Evaluation of how well models maintain their performance over time.
- Adversarial Robustness: Assessment of model resilience against adversarial attacks, specifically those involving content injection.
- Computational Overhead: Measurement of the inference speed and computational resources required for deployment on endpoints.
The benchmark aggregates the results from these four dimensions into a single score. This aggregated score is intended to enable direct and fair comparisons between different AI-based malware detection models.
Findings
The analysis conducted using EXE-Bench highlighted that evaluations performed only after deployment are suboptimal and do not provide a full understanding of a detector's capabilities. A key observation was the enduring utility of domain knowledge, specifically when instantiated through feature engineering. Models incorporating this domain knowledge demonstrated significant resistance to both temporal shifts and adversarial attacks.
Conversely, many deep networks, despite potentially excelling immediately after deployment, showed less resilience to the passage of time and adversarial manipulations compared to those models benefiting from carefully engineered features.
Why This Matters
The systematic evaluation provided by EXE-Bench offers a structured method for assessing AI-based Windows malware detectors. By addressing gaps in existing evaluation practices—specifically regarding temporal analysis, adversarial robustness, and computational load—it provides clearer insights into which detectors are robust and practical for real-world production environments. The findings regarding the sustained value of feature engineering in resisting evolving threats and attacks are relevant for guiding future design and selection of malware detection systems.