Overview
ViPER (Vision-based Packing-Aware Encoder for Robust malware detection) is a system designed to improve malware detection by integrating packing awareness into a vision-based approach. It addresses the challenges posed by executable packing, which often obscures structural patterns in binary byte-to-image visualizations, impacting the efficacy of visual classifiers. The system utilizes a dual-head architecture built upon a LoRA-adapted ViT-B/14 backbone to simultaneously identify malware and detect packing. A packing-aware gating mechanism then conditions malware predictions based on the inferred packing state, allowing for distinct decision boundaries tailored to packed and unpacked inputs.
Research Context
Conventional malware detection methods often rely on analysis pipelines that can be evaded or require disassembly. Visualization-based malware detection offers an alternative by mapping raw binary bytes to grayscale images, then applying learned visual classifiers. This method is characterized as evasion-resistant and disassembly-free. However, executable packing presents a significant challenge to these visual classifiers. Packed binaries typically yield high-entropy images, which can obscure the underlying structural patterns crucial for detection. The prevalence of packing in both malicious and benign software (e.g., for compression or copy protection) means that packing state alone is not a definitive indicator of malicious intent. Existing visual detection approaches typically do not integrate a solution for this issue within a unified supervised framework.
Approach
The ViPER system is constructed on a LoRA-adapted ViT-B/14 backbone. Its architecture features two distinct heads: one for malware classification and another for packing detection. This dual-head design facilitates the concurrent learning of both tasks. To process inputs, ViPER employs a packing-aware gating mechanism. This mechanism dynamically adjusts malware predictions based on the system's determination of whether an input is packed or unpacked. This conditioning enables the establishment of separate decision boundaries, optimized for packed and unpacked inputs, respectively. To mitigate issues arising from packing label skew during the training phase, ViPER incorporates frequency-weighted losses. These losses are applied with stratified sampling across joint class-packing strata. The system was evaluated using a dataset comprising 200,000 Windows PE byteplot images.
Findings
Evaluation of ViPER on a dataset of 200,000 Windows PE byteplot images yielded several performance metrics. The system achieved a balanced accuracy of 0.8521. Its Receiver Operating Characteristic Area Under the Curve (ROC-AUC) was measured at 0.9260, and its Area Under the Precision-Recall Curve (AUPR) was 0.9279. Furthermore, ViPER demonstrated a packing detection AUC of 0.9949. These results indicate that ViPER outperformed representative state-of-the-art baselines across all primary metrics in the evaluated scenario.
Why This Matters
The ability of ViPER to effectively handle packed binaries within a unified framework suggests an advancement in the robustness of vision-based malware detection. This approach, by jointly considering packing state and malware classification, addresses a critical limitation of previous visualization-based methods, potentially enhancing their reliability in real-world scenarios where packing is prevalent in legitimate and malicious software.