There has been much debate on whether Apple’s Mac OS X operating system is more secure by design or have taken advantage of security through obscurity, as Macs have traditionally had much lower penetration into the market than Windows based systems.

However, as Apple has enjoyed rapidly growing success over the past few years, not only with its iPhones and iPads but with also the growing popularity of Macs and Mac OS X, it comes as no surprise to see security companies predict that attacks on Apple’s systems and its users will also start increasing dramatically.

While ‘Security’ seems to be a competitive advantage, Apple has realised that some dedicated malware defences are needed via cloud web security, so it shipped an ‘Anti-Malware’ component with its Mac OS X updates. In June 2010, 3 (three) malware programs were on the list (link). When we checked in March 2011, there were still three!

Recently, Sophos made waves announcing a new “backdoor Trojan” for the Mac OS X platform, (http://nakedsecurity.sophos.com/2011/02/26/mac-os-x-backdoor-trojan-now-in-beta/). Although this Trojan (called Black Hole) is very new, it is not in a state that poses any threat, and doesn’t provide much for us to look at yet in an in-depth report.

Given that, we thought this a perfect opportunity to look at an existing piece of Mac malware known as HellRaiser 4.2, chosen because it is detected by Apple’s anti-malware defences, unlike ”Black Hole”. Using HellRaiser 4.2 lets us see how Apple’s defences work, so we can test their effectiveness and discover their limitations.

There is very little public information on the technical details of Mac OS X’s anti-malware feature, even in their animation tutorial videos, however, we will provide some technical details here. Furthermore after testing Apple’s anti-malware defences, you might have guessed that we found several ways around the built-in protection within just a few hours.

At ‘TrustDefender Labs’, we typically look at the most sophisticated Trojans available. While ongoing research on the latest malware Trojans has led us to have very high expectations on the levels of sophistication now seen in advanced malware, the Mac OS X HellRaiser 4.2 Trojan left us quite disappointed by being a relatively simple Trojan that, despite some “clever” features, doesn’t look like it was written by professionals.

Even though the Mac OS X malware industry is still in its infancy, the whole situation provides cause for concern going forward, especially as many of Apple’s end-users aren’t accustomed to treating malware as a serious security risk given Apple’s legacy as safe technology that’s easy to use and master, with historically miniscule and harmless malware threats compared with the barrage of malware, viruses and Trojans that Windows users have been subjected to for years. Introduction – Mac OS X and Malware

Apple and malware are two words that aren’t commonly spoken together. However, this is set to change in a big way in the near future. While Mac OS X only holds a relatively small percentage of the desktop market, its close smartphone and tablet cousin iOS is in the hands (literally!) of millions of people. This is a tempting target that  will see increasingly sophisticated and successful attempts at exploitation.

For all doubters, Apple has just released iOS 4.3, a massive update to iOS where Apple patched a stunning 49 (!) security vulnerabilities. Apple notes that these vulnerabilities “may lead to an unexpected application termination or arbitrary code execution”, which in plain English means having your device owned. The full list is available at Apple’s support site http://support.apple.com/kb/HT4564. It’s important to note that these vulnerabilities left iOS users unprotected until a new version of iOS was released, as opposed to having updates released on a much more timely cycle.

To give you an idea of the seriousness of unpatched vulnerabilities, we only need to look at one of the most targeted attacks on Windows operating systems and applications – Stuxnet.

The perpetrators behind Stuxnet exploited seven different zero-day attacks, developing highly successful and highly targeted malicious software to break into what would otherwise be highly secure installations. They went to incredible lengths to accomplish what they wanted. Do you think Stuxnet would have been less successful if the Siemens SCADA machines were controlled by Apple Macs?

This is the reality that we have to face in the future: there are inherent risks with what we do, regardless of the operating systems we use.

The trick is how you deal with the risk and whether you can manage the risks. This is the purpose of this TrustDefender Labs report, which looks at Apple’s built-in malware defences.

One of the existing pieces of malware on Mac OS X is HellRaiser 4.2. As we wanted to test Apple’s built-in protection, we needed to look at a Trojan that Apple already provides protection against, with HellRaiser 4.2 fitting the bill. We’ll look at it in this section a bit in more detail and will then look at Mac OS X’s defences.

Hellraiser is actually about 12 months old, with Virus Total reporting just over a 50% hit rate, so this Mac OS X Trojan isn’t new or unknown by any stretch of the imagination. However, we figured it’s still interesting to investigate, both in terms of what is capable of, and, more interestingly, what Apple is doing to protect against these kinds of things.

First up, let’s just see what happens if we run it:

Well, that’s a good start. It looks like Apple’s built in protection detects it!

However, let’s put that aside for a moment, and have a quick look at this malware. It’s not new, and seems pretty rudimentary, but offers a few clever features. HellRaiser is a Remote Administration Tool (RAT). The goal is to get the server running on a target system by whatever means necessary (i.e. social engineering), and then be able to connect to it from the RAT client.

Configuration is straightforward:

Once configured, installed and running on the target system, the server will email or FTP its local IP address and port to the specified host. From here, the cyber criminal can connect using the client software:

As you can see from the last image in the previous section, the HellRaiser client exposes quite a bit of functionality. The server and client communicate via a protocol that appears to be roughly plain text, albeit turned into hex and base64 encoded. For example, if we use the “tell” function to create a message on the target system, Wireshark reports the following message on the main control port:

5655784a56446f7848314e46546b51634d446b784d5455324e69353661584146486a4d344e6830354e7a6b3044516f3d

We can convert this into something a little more useful:

The information circled in red above shows that, over a data port, a zipped file is sent that contains a file named 0911566 which contains our message.

For the most part, HellRaiser works but has some quite serious limitations. Firstly, the software is somewhat buggy:

Secondly, given it waits for a client to attach to the server, any form of NAT or firewall will stop it.  Apple’s built in firewall stops the connection, as will most routers that perform some kind of Network Address Translation (NAT).

So, that’s what the cybercriminal is attempting. Now let’s look at what Apple’s OS X security experts are doing to protect us.  When a file is downloaded via Safari (or, indeed, any application that has “LSFileQuarantineEnabled” set to true), it is flagged with an extended attribute “com.apple.quaratine”. This contains some information about how the file was acquired, specifically the timestamp, application name, and the bundle identifier.

For example:

com.apple.quarantine: 0000;4d7468de;Googlex20Chrome;66B1A56C-4296-420D-A895-48DEA91D76F4|com.google.Chrome

When a file that has been downloaded is executed, “LaunchServices” will see this flag and prompt the user the first time the application is launched.

Only once a file is marked quarantined does Mac OS X’s built-in malware protection kick in. There is no official statement as to what this is called, so we and others in the security industry are calling it XProtect, after the name of the definition file. XProtect is configured and updated by a standard plist file, which is a human-readable XML file format, containing a blacklist signature database that all users of Internet security and anti-virus software on Windows systems are well aware of.

Looking in the malware definition list (XProtect.plist), we can see a few entries for HellRaiser:

This is a really useful protective feature out of the box, and as long as the OS is kept up-to- date, Mac OS X users get some level of protection. On the downside, this list is updated relatively infrequently, at least when compared to anti-virus engines. Some anti-virus vendors claim hourly or even more frequent updates, while Apple’s anti-malware definitions are fairly static. Currently only three malware types are part of the list.

Once the amount of malware targeting Mac OS X increases, Apple’s infrequent updating may prove to be insufficient, and it’s obviously no help against zero-day, click-jacking or session based trojans.

Please refer to the indepth report

Please refer to the indepth report

Please refer to the indepth report

Based on average systems, it appears all common web browsers and email clients do have this set, either in the applications plist or in the system-wide “Exception.plist”, however, IM and, worryingly, P2P programs tend not to. Apple’s own programs are usually well behaved. Both Mail and iChat will create quarantined files.

The trouble though is that Instant Messanger and especially P2P programs are designed to exchange files. That is actually their whole purpose!

You can have Mac OS X up to date, but if you download a Mac OS X targeting Trojan with your torrent network, XProtect will not protect you! And downloading files is the only reason why people use torrent clients obviously.

If some malware does manage to run on a system, it can freely download and execute whatever it likes with no interruption from the OS.

Please refer to the indepth report

The question of performance is also interesting.

“The number of new malicious programs detected during the year was approximately 13 million” – Kapersky, 2010

While the model of only running a scan on files flagged for quarantine means the performance impact should only hit on the first run of a downloaded application, the question of performance and scalability has to be asked. If the situation on Windows is anything to go by, Apple may have to deal with very large numbers.

Currently, “XProtect.plist” only has three different groups of malware. If we include variants we might be generous and say it detects 50 different strains. What happens if this number is increased?

I added 10,000 entries, each with a pattern match field to “XProtect.plist”, and it added about 7 seconds (!) to the time it took to launch a file flagged for quarantine that didn’t match any rule. This last clause is important, because “CoreServices” is smart enough to stop searching as soon as it finds a match. So the extra size didn’t make finding malware early in the list any slower.

This, in effect, means that non-malware faces an aggravating time penalty if the “XProtect.plist” gets too large.

It will be of no surprise to anyone reading this document that the idea of Mac OS X having perfect security is a complete myth. There is, however, a grain of truth in the idea that there is no malware on Mac OS X, albeit a small grain that is shrinking rapidly! As it stands today, there isn’t much malware that we can find.

That RATs (Remote Administration Tools) such as Sophos’s recently discovered Black Hole and HellRaiser get publicity says something about the current quantity and state of malware on Mac OS X. However, this shouldn’t be an excuse for Mac users to be lulled into a false sense of security. The malware is out there, and it’s growing both in terms of numbers and sophistication.

Apple is being proactive and introducing OS level security. It’s flawed security that is currently pretty flimsy, but nonetheless, it’s a (small) step in the right direction. It’s going to be interesting to see the direction Apple chooses from here.

iOS will only run signed binaries, so this type of protection is less important for that OS. Will XProtect get improved? Or will Mac OS X get locked down and only run signed binaries? And, how long will it be until there is a serious outbreak of malware on Apple’s platforms?