Work on the use of hidden Markov models (HMM) to detect viruses has been carried out previously with good results , but metamorphic viruses like MetaPHOR  and metamorphic worms like MWOR  have proven to be able to evade detection techniques based on HMMs. The dueling HMM approach looks to detect such viruses by training an HMM model for each of the metamorphic virus / worm families. The tests and the results from these have shown that this approach has been able to detect the metamorphic MetaPHOR virus with reasonable accuracy but with significantly more overhead. This paper presents a tiered approach that improves on this by achieving the same results as the dueling approach but with significant performance improvement in terms of time. Essentially the idea is to eliminate most putative malware with the threshold approach, reserving the dueling HMM analysis for more difficult cases. We achieve accurate results with significantly less performance overhead than the dueling HMM strategy. Furthermore, our approach successfully detects MWOR worms with a high degree of accuracy.
Kalbhor, Ashwin, "A Tiered Approach to Detect Metamorphic Malware With Hidden Markov Models" (2014). Master's Projects. 360.