Using virtual machines for integrity checking

Abstract

Today’s arms race between the attackers and defenders of computer systems seems like a never ending story. Traditionally, the battle has been fought outside the computer’s operating system kernel, but in recent years the advent of kernel level malware has moved the battlefield inside the operating system, thus incapacitating many of the before trusted security mechanisms. When this happens the operating system can no longer be trusted, and new kinds of security tools must be developed.

This thesis looks at the potential of virtualization as a platform for performing integrity checking of a running operating system’s kernel. In theory, the use of virtualization should make it possible to establish a platform of trust in the system, even when the kernel of a virtualized guest kernel has been subverted.

The idea of monitoring an attacked system from a different protection domain than the attacked system is not new. The use of virtualization brings some extra benefits though: High visibility to the monitored system and good protection from outside attackers. Traditional computer surveillance systems have been forced to compromise between these two properties.

The reader is in this thesis introduced to the concept of kernel level malware, virtualization techniques and the internals of the Linux kernel. An architecture designed to address some of the problems surrounding the integrity checking of a running kernel, is presented. The details of this architecture is discussed, and a working prototype putting the architecture to the test against a suite of real attacks, is constructed.