Publication Date

Fall 12-20-2020

Degree Type

Master's Project

Degree Name

Master of Science (MS)


Computer Science

First Advisor

Mark Stamp

Second Advisor

Teng Moh

Third Advisor

Samanvitha Basole


Malware, Gaussian mixture model-HMMs, opcode sequences, entropy-based sequence


Discrete hidden Markov models (HMM) are often applied to the malware detection and classification problems. However, the continuous analog of discrete HMMs, that is, Gaussian mixture model-HMMs (GMM-HMM), are rarely considered in the field of cybersecurity. In this study, we apply GMM-HMMs to the malware classification problem and we compare our results to those obtained using discrete HMMs. As features, we consider opcode sequences and entropy-based sequences. For our opcode features, GMM-HMMs produce results that are comparable to those obtained using discrete HMMs, whereas for our entropy-based features, GMM-HMMs generally improve on the classification results that we can attain with discrete HMMs.

Available for download on Monday, December 20, 2021