Publication Date

Spring 2026

Degree Type

Master's Project

Degree Name

Master of Science in Computer Science (MSCS)

Department

Computer Science

First Advisor

Mark Stamp

Second Advisor

Fabio Di Troia

Third Advisor

Amith Kamath Belman

Keywords

Malware classification, Image-Based Analysis, Convolutional Neural Networks, Transfer learning, Ensemble Learning

Abstract

There are visible visual patterns in malware binary files. These can be converted into images for classification purposes. In this paper, we study three different tech- niques to extract features from images in order to classify malware binary families and to examine whether the combination of these techniques leads to an improved result compared to individual techniques. Each malware binary file is converted to image using eight image generation strategies proposed earlier in the literature. Then, three pipelines are applied to these images: the first one extracts handcrafted descriptors based on Histogram of Oriented Gradients, Haralick texture features, and 38 statistical features. The second pipeline extracts features by taking embeddings generated by pretrained neural network architectures. These include VGG16, ResNet50, and Vision Transformer. Finally, a custom Convolutional Neural Network is trained to process the images and generate 512-dimensional embedding vectors. Classical ML algorithms are trained on the generated representations. The highest accuracy for the handcrafted representation track, the custom CNN representation track, and the pretrained CNN representation track amounts to 0.7794, 0.7518, and 0.7394, respectively. The soft voting ensemble consisting of the top five configurations per track (15 voters total) achieved an accuracy of 0.9424 on the held-out dataset of 1,700 instances of 17 malware families, which is 0.1630 more accurate than the best-performing single model.

Download

Available for download on Saturday, May 22, 2027

Share

COinS