Operating System Diversification

Department of Information Technology University of Turku Turku, Finland

Funding, duration

MATINE (Ministery of Defence) funded project for Jan-Nov 2014. Overall MATINE+UTU funding volume: 123.000e.

Background

Malware, or malicious software, is one of the main problems in Internet today. Malicious software uses prior knowledge about the identical interfaces of operating systems to accomplish its goals. To access resources on a computer, a malicious program has to know the interface that provides the resources. Because of the current operating system monoculture, an adversary can create a single malicious program that works on hundreds of millions of computers that use the same operating system.

As the number of malicious programs keeps growing and new variants keep popping up, traditional fingerprint-based antivirus software is becoming increasingly inefficient in the fight against malicious programs. Also, antivirus programs often only detect the threats they are already aware of. Therefore, new approaches to malware prevention are needed to complement them. Our approach adopts a proactive view by preventing malicious code from harmfully interacting with its environment even before it is executed.

Goals

Our goal was to provide protection to applications and software systems in a new way: we diversified the implementations of all software layers and their interfaces on the binary level. The system call interface of the operating system that can be used to access resources was diversified uniquely for each system and all the entry points to this interface were diversified accordingly.

In such system malware that uses prior knowledge about existing interfaces in an operating system is rendered useless because of diversification. It can no longer use the resources of the system. Only trusted applications that are diversified will work in the system. Also, even if the malware were to find out the secret diversification for one system, it cannot perform any large-scale attacks because the diversification is unique for each system.

Software

As a part of our project, we created a number of applications for system diversification. The applications operate on binaries and can be used to diversify existing software without access to the source code. This approach has it's limitations, but we concluded that it's upsides make it worthwhile for some applications.

The first piece of software in our diversification toolkit is meant to obfuscate the system call use in ELF-binaries. The tool can successfully rewrite binaries to use a modified set of system call numbers. The idea is, that without access to operating system's service, malware's capabilities are severely constrained. Of course, the modified binaries have to be run on a kernel that supports the same set of system calls.

Diversifying the system call interface is not enough to adequately secure a system. Applications seldom use system calls directly and instead rely on libraries to provide the necessary services. Because of this, we also developed a program to diversify shared libraries and applications using them. Our library diversifier renames the provided symbols, making it difficult for an attacker to find the provided services.

Publications

  1. S. Rauti, J. Holvitie, and V. Leppänen, 'Towards a diversification framework for operating system protection', in International Conference on Computer Systems and Technologies, ACM ICPS 883, p. 286-293, .
  2. S. Laurén, P. Mäki, S. Rauti, S. Hosseinzadeh, S. Hyrynsalmi and V. Leppänen, 'Symbol diversification of Linux binaries', in World Congress on Internet Security, .
  3. S. Rauti, S. Laurén, S. Hosseinzadeh, J. Mäkelä, S. Hyrynsalmi and V. Leppänen, 'Diversification of system calls in Linux binaries', in The 6th International Conference on Trustworthy Systems, LNCS, Springer, .