Publication Date
Fall 2023
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
Mark Stamp
Second Advisor
Thomas Austin
Third Advisor
Genya Ishigaki
Keywords
Malware, Hidden Markov Model, Random Forest, Convolutional Neural Network, hybrid models
Abstract
Many different machine learning and deep learning techniques have been successfully employed for malware detection and classification. Examples of popular learning techniques in the malware domain include Hidden Markov Models (HMM), Random Forests (RF), Convolutional Neural Networks (CNN), Support Vector Machines (SVM), and Recurrent Neural Networks (RNN) such as Long Short-Term Memory (LSTM) networks. In this research, we consider a hybrid architecture, where HMMs are trained on opcode sequences, and the resulting hidden states of these trained HMMs are used as feature vectors in various classifiers. In this context, extracting the HMM hidden state sequences can be viewed as a form of feature engineering that is somewhat analogous to techniques that are commonly employed in Natural Language Processing (NLP). We find that this NLP-based approach outperforms other popular techniques on a challenging malware dataset, with an HMM-Convolutional Neural Networks model yielding the best results.
Recommended Citation
Mehta, Ritik, "A Natural Language Processing Approach to Malware Classification" (2023). Master's Projects. 1302.
https://scholarworks.sjsu.edu/etd_projects/1302
