Publication Date

Spring 2016

Degree Type

Master's Project

Degree Name

Master of Science (MS)


Computer Science

First Advisor

Mark Stamp

Second Advisor

Chris Pollett

Third Advisor

Fabio Di Troia


Malware detection cryptanalysis Vigenère cipher


Previous research has applied classic cryptanalytic techniques to the malware detection problem. Speci cally, scores based on simple substitution cipher cryptanal- ysis and various generalizations have been considered. In this research, we analyze two new malware scoring techniques based on classic cryptanalysis. Our rst ap- proach relies on the Index of Coincidence, which is used, for example, to determine the length of the keyword in a Vigenère ciphertext. We also consider a score based on a more complete cryptanalysis of a Vigenère cipher. We nd that the Vigenère score is competitive with previous statistical-based malware scores.