(No Ratings Yet)

Loading...

• Facebook
• Twitter
• LinkedIn
• Mail

January 29, 2021 4

All You Need to Know About the SolarWinds Attack

in ssl certificates

How the SolarWinds Hack Was Pulled Off—and What Software Developers Can Learn From It

What’s the safest way to buy something? To ensure you’re getting it brand new, fully functional, and not compromised in any way? It’s certainly not via that shady Craigslist ad with stolen photos and a price that’s too good to be true. Instead, the answer is directly from the manufacturer – off the assembly line and straight to you without any middlemen involved.

This doesn’t just go for physical products but software as well. Wouldn’t you feel more comfortable downloading a Windows update directly through Microsoft rather than from a third-party site? And speaking of software updates, they’ve become a major part of our lives in the 21^st century. All our “smart” devices are constantly connecting to the manufacturer’s website to get the latest and greatest versions of their software.

But what if there was something else lurking behind the scenes in those seemingly trustworthy updates? What if, instead of targeting an end-user’s machine directly, attackers instead targeted the software updates themselves? Compromise the update, and theoretically anyone that downloads the update is also compromised. It’s called a supply chain attack, and the recent SolarWinds hack is the most severe and wide-reaching instance we’ve ever seen. We’re talking 425 of the Fortune 500 being potentially affected, as well as numerous government agencies.

Does it mean you should stop updating your devices? Absolutely not. In fact, we still can’t stress enough how important it is to keep your devices up to date. Rather, the SolarWinds attack is an unsettling wake-up call for manufacturers and developers. It’s more critical than ever that companies create and maintain a culture of security and more specifically, one of effective and comprehensive code review and code signing processes.

So, who planned the SolarWinds attack and what were the pre-requisites? How was the attack itself carried out? How was the breach finally detected? And what can organizations do to protect themselves from similar attacks in the future?

Let’s hash it out.

The Initial Target – SolarWinds Orion

The attackers chose to initially target SolarWinds as the first step in their plan for a reason – the SolarWinds Orion platform is very popular amongst large enterprises and government agencies for managing infrastructure and assets. By first compromising SolarWinds, they would then have a direct route into the networks of their ultimate targets – the customers of SolarWinds.

This kind of strategy is called a “supply chain attack” because, as the name implies, the attack is made while the end product (either software or hardware) is still being created and/or manufactured. Basically, if the product is compromised before it reaches the end user, you’re looking at a supply chain attack. We’ve seen it before, and in the case of the SolarWinds Hack, the software updates were injected with malicious code during development before the update even went through the build process. No one caught the compromised code – it was signed with a code signing certificate by SolarWinds themselves and sent straight to their clients as part of the routine update process.

The advantages of a supply chain attack are clear – it gives hackers a secret backdoor into the network of every single end user that has the infected product. In the case of SolarWinds, it’s an even more powerful attack vector for hackers because the Orion Platform itself is designed to stretch across the user’s networks and have a significant degree of control.

Then, once the hackers get inside, they can go about deploying further pieces of malware to increase their capabilities, escalate the attack and remain undetected. For example, Malwarebytes discovered suspicious activity relating to their Office 365 tenant. It turns out that, once inside the Malwarebytes network, the attackers created a self-signed digital certificate containing credentials to the service principal account. It allowed them to make API calls and request emails through MSGraph. As we’ll go into more detail about shortly, the hackers used a variety of tools and techniques like these throughout the SolarWinds hack.

Who Was Impacted by the SolarWinds Attack?

The SolarWinds attack began with the perpetrators injecting malicious code into Orion software updates starting in March 2020, and over the next nine months used the backdoor to access government and business networks worldwide. Roughly 18,000 clients updated their software in that time period, and the full extent of the damage is still being determined.

The list of affected organizations keeps growing, but we know that technology giants such as Cisco, Microsoft, Intel, VMware, and Nvidia were impacted. Fortune 100 accounting firm Deloitte LLP and Belkin International, who you might know from their LinkSys brand of networking devices, also received compromised software updates from SolarWinds.

The public sector was hit hard by the SolarWinds attack, as well. The Department of Justice found that attackers breached their Office 365 system, including the email accounts of roughly 3,000 employees. They found the breach nine days after the SolarWinds hack was first discovered.

More alarmingly, the systems of the United States’ nuclear weapons arsenal were accessed. Fortunately, the attackers didn’t get into any of the critical defense systems. Other government groups that were hit by the SolarWinds attack include the Defense Department, the Health and Human Services Department, the Homeland Security Department, the State Department, and the Treasury Department.

Overall, it’s estimated that about 50 organizations were significantly impacted by the SolarWinds attack. The number that received the initial backdoor vulnerability, however, was much higher.

Who Was Responsible for the SolarWinds Hack?

Before we get into the technical details of the SolarWinds attack, it’s important to understand the type of party that could carry out such an operation and what their strategy would be.

All evidence points to an Advanced Persistent Threat (APT) actor being behind the SolarWinds attack. APTs are long-term attacks where the intruders maintain a presence on the victim’s network in order to steal high-value information. Large enterprises and government networks are typically targeted since the goals of APTs tend to be political or economic.

This kind of attack is often carried out by nation-states since they require significantly more sophistication and planning compared to your typical “in-and-out” style attack from smaller groups or individuals. APT attacks tend to be highly complex, carried out by teams of experienced cybercriminals, and backed by substantial resources – both technical and financial.

The specifics of the SolarWinds attack suggest that stealth and persistence were a top priority due to the fact that DLL injection techniques were employed. The hackers in question also put a great deal of effort into compartmentalizing their work. In other words, they chose their routes very carefully and did all they could to prevent the “connecting of the dots” from one action to another. They didn’t just immediately exploit every vulnerability they could find.

John Hultquist, Director of Intelligence Analysis at FireEye (which also happened to be the cybersecurity firm that first discovered the SolarWinds hack), explained how

This is about quality over quantity. Every organization they access endangers their access — which risks the entire operation.

Therefore, the attackers performed reconnaissance on every organization they were able to infiltrate with their initial backdoor. If they didn’t like what they saw, then they didn’t take any risks and moved on to another, more attractive target.

SolarWinds Attack Preparation & Test Run

While the attackers began injecting malicious code into SolarWinds Orion updates in March 2020, they gained access to SolarWinds’ systems much earlier than that. We know for sure that they were present in September 2019, but most experts agree that their planning began much earlier. A month later, in October, the attackers performed a test run to see if they could successfully insert their own code into the Orion Platform during the development process.

On October 10, files containing this “test” code were sent to customers as part of an update package and were downloaded just like any other update. These files were clean and did not contain a backdoor or any other malicious code, and they remained undetected until December 2020. An investigator who spoke anonymously with Yahoo News said

We’re thinking they wanted to test whether or not it was going to work and whether it would be detected. So it was more or less a dry run. They took their time. They decided to not go out with an actual backdoor right away. That signifies that they’re a little bit more disciplined and deliberate.

If the attackers were able to send out files in October, then they must have gained access to SolarWinds’ systems a few months earlier at the very least. As the investigator explains, 

This tells us the actor had access to SolarWinds’ environment much earlier than this year. We know at minimum they had access Oct. 10, 2019. But they would certainly have had to have access longer than that. So that intrusion [into SolarWinds] has to originate probably at least a couple of months before that — probably at least mid-2019 [if not earlier].

How the SolarWinds Hack was Executed

The attackers started adding malicious code to SolarWinds software updates on February 20, 2020. By March 26, 2020 the compromised updates were finally loaded on the SolarWinds software update servers, where they were subsequently distributed to customers and downloaded onto their machines. The updates installed a backdoor, known as Sunburst, on the user’s network that gave the attackers a direct means of access.

Once they were in, the SolarWinds software could be used to view the network structure and make configuration changes. The attackers could then use their newfound knowledge and abilities to attack other network locations and install more malware on the systems of their choosing.

Code Signing & the SolarWinds Attack

Let’s digress for a moment. At this point you may be wondering what role code signing played in the SolarWinds attack. Were code signing certificates being used by SolarWinds? Why didn’t they prevent this attack?

SolarWinds did indeed practice code signing with their software updates. However, the attackers inserted the malicious code during the development cycle, using stolen credentials to impersonate trusted accounts and remain undetected. The software updates were compromised before the code signing process was carried out, which would occur at the very end of the development process.

There is no evidence that any code signing certificates were stolen or compromised in the course of the SolarWinds attack. What was compromised, however, was SolarWinds’ code review and signing process. As we’ll discuss shortly, the malicious code was written to be inconspicuous and blend in with the rest of the actual code. This helped it slip through the human review process. SolarWinds then signed it with a valid code signing certificate, giving it the appearance of legitimate, trusted code to the outside observer.

Code signing is performed to ensure that software files remain identical to when they were originally signed. This prevents a third party from inserting malware somewhere in transit between the software vendor and the end user.

What code signing is not intended to do is compare the code being signed by the developer to what they actually intend to release. The code you’re signing can be non-functional and loaded with malware. The code signing certificate doesn’t care. It’s meant to ensure that the customer gets exactly what was released by the developer, period.

This is where the human element comes into play. Code signing certificates alone couldn’t have prevented the SolarWinds hack. However, had SolarWinds been following code signing best practices in conjunction with more effective code review, this fiasco could have indeed been avoided.

We’ll get into more detail on that shortly, but first let’s get back to examining the specific malware and backdoors used to execute the SolarWinds hack.

Sunspot Malware

Sunspot was the first piece of malware deployed after the attackers first gained access to the internal network of SolarWinds. It was installed on their build server in September of 2019. Sunspot only had one purpose, which was to insert the malicious backdoor into the SolarWinds Orion source code.

Sunspot did this by monitoring the build server for Orion build commands. When such a command was detected, Sunspot would covertly load the malicious code in place of the legitimate version. The compromised update package, which now contained backdoor malware, could then be distributed to SolarWinds’ customers just like any other update would be.

Sunburst Backdoor Malware

Sunburst is the backdoor malware that was inserted into the updates via Sunspot. Sunburst is located within the DLL file “SolarWinds.Orion.Core.Businesslayer.dll” and its main function is to communicate with the attacker’s servers through HTTP.

The update file that contains the backdoor is a trojanized version of a regular Windows Installer Patch file. The compromised SolarWinds.Orion.Core.BusinessLayer.dll file is included amongst legitimate, compressed update files. The malicious DLL is installed by either SolarWinds.BusinessLayerHost.exe or SolarWinds.BusinessLayerHostx64.exe (depending on the system architecture), which is a legitimate service that loads the real update files, as well.

Post-installation, the malware hibernates for two weeks to help further reduce suspicion. Upon waking up, it attempts to resolve a subdomain of a malicious domain controlled by the attackers (in this case, avsvmcloud.om). A CNAME record pointing to a Command and Control (C2) domain is returned by the DNS response.

The C2 traffic generated by the malware’s communications to the attacker’s domain is disguised as normal SolarWinds API communication, masquerading as the Orion Improvement Program (OIP) protocol. The data generated by the malware, including sensitive information about the victim’s network, users, servers, etc, is written to legitimate plugin configuration files and appears to be valid activity to anyone watching.

The backdoor malware employs hidden blocklists to detect security or anti-malware tools and temporarily ceases any suspicious actions when in their presence. These actions, called “jobs”, include tasks such as file transfers, the launching of executables, system profiling, and the disabling of system services.

The goal of Sunburst was to ultimately determine if the victim was a good fit for the deployment of more powerful malware on their system. The data that Sunburst collected was sent back to the attackers and analyzed. If the victim was either too high-risk or too insignificant, then Sunburst deleted itself. Otherwise, the attackers moved forward with deploying more malware.

The Solorigate DLL File

Now, about that DLL file that contained the Sunburst code. As we saw back in October 2019, the attackers began testing their code-insertion abilities with non-malicious code. This was done by adding empty classes to DLL files.

The next step was to add a non-empty class to an existing DLL file, one which would ultimately contain malicious code. They named the class OrionImprovementBusinessLayer to blend in with the rest of the legitimate code. The attackers wisely avoided obviously conspicuous terms like “backdoor” or “exploit” so as to not raise suspicions (and considering the malware went undetected for months, they clearly did a good job). The class contained the full functionality of the backdoor, with 13 subclasses and 16 methods used. The malware was designed to be lightweight and not interfere or impede the normal operation of the DLL.

The hackers also needed to put their code in the right place within the DLL. They chose a spot within a method that would get regularly invoked no matter what the user was doing. That way, it would be guaranteed to both initially execute and then keep running afterwards. They chose a method called RefreshInternal that met these requirements, and made sure to initialize the malware within a parallel thread so as to not disrupt the normal execution of RefreshInternal.

Teardrop & Raindrop

After the attackers finished collecting intel on potential victims via Sunburst, they would deploy another piece of malware to those that they determined were a good fit for escalation. Two different pieces of backdoor malware were used, called Teardrop and Raindrop. While there’s some differences between the two at the code level, they have almost identical functionality, which is to deploy the Cobalt Strike Beacon.

The Cobalt Strike Beacon is a tool for expanding and escalating access within the victim’s network. It can:

• Contact C2 servers
• Launch WMI to load malicious DLLs
• Steal credentials
• Query Active Directory
• Elevate current user privileges

Teardrop was deployed on machines that were originally infected with the Sunburst malware. Raindrop however, was deployed laterally to other systems on the same network. Raindrop was selectively used on just a few targets and was the last step in the SolarWinds attack as far as malware went. It does not appear that Sunburst was used to install Raindrop.

Discovery of the SolarWinds Hack

So, considering the sophistication of the attack and the lengths to which the attackers went to conceal their activity, how the heck did anyone figure out what was going on?

As we mentioned earlier, it started with FireEye, a security firm. They first determined that their system was compromised on December 8, although the link to SolarWinds was not immediately known. It ultimately took SolarWinds until December 18 to completely remove the infected update packages from their servers.

The discovery occurred when the attackers attempted to register a device with FireEye’s multifactor authentication system. An automated alert was generated and sent to their internal security team, which happens whenever an unknown device is being used. Charles Carmakal, senior vice president and chief technical officer at FireEye, explains how

They had to provide credentials to authenticate [their device] to the [multifactor authentication system] in order to authenticate to the FireEye VPN. It was the process the attacker followed to enroll in the MFA solution which is what generated the alert.

FireEye kept digging and soon found that it wasn’t just FireEye that was targeted. Carmakal said

We looked through 50,000 lines of source code, which we were able to determine there was a backdoor within SolarWinds.

Closing the SolarWinds Attack Backdoor

Soon after the initial discovery of the SolarWinds attack, it became apparent just how wide-ranging the impact was. Within a few days, Microsoft, FireEye, and GoDaddy responded. They created a kill switch for the backdoor that forces the malware to destroy itself.

This was accomplished by first seizing the C2 domain that was being used by the attackers, avsvmcloud.com. Instead of resolving to the hacker’s server, it instead resolves to a Microsoft-owned IP address of 20.140.0.1. This IP address is on the malware’s blocklist, and if the malware detects this address then it is instructed to delete itself. This is referred to as a “sinkhole” and is used as a way to trap and contain malware, as well as analyze the traffic to identify other victims. Per FireEye,

This killswitch will affect new and previous SUNBURST infections by disabling SUNBURST deployments that are still beaconing to avsvmcloud[.]com.

Protecting Against Future SolarWinds-Style Attacks

People & Culture

Since the discovery of the SolarWinds attack, a former security advisor at SolarWinds, Ian Thornton-Trump, has since come forward and revealed that he had previously voiced his own security concerns to management. Thornton-Trump made a presentation in 2017 to at least three SolarWinds executives regarding the need to tighter security measures and laid out a plan to improve their current infrastructure. He went so far as to say

The survival of the company depends on an internal commitment to security.

His suggestions were ultimately ignored. Thornton-Trump resigned a month later, feeling that the company didn’t invest enough in security technology and was severely lacking when it came to cultivating a culture of cybersecurity. He said

There was a lack of security at the technical product level, and there was minimal security leadership at the top. We knew in 2015 that hackers were looking for any route into a business. But SolarWinds did not adapt. That’s the tragedy. There were plenty of lessons to learn, but SolarWinds wasn’t paying attention to what was going on.

Unfortunately, Thornton-Trump can now say “I told you so.”

Proactive Not Reactive

Thorton-Trump wasn’t the first cybersecurity professional to voice concerns regarding their employer’s security culture and infrastructure. It happens in organizations all over the world, and all too often action isn’t taken until it’s too late. Companies will wait until they’ve suffered a security incident before making changes, and by then they’ve suffered the consequences – non-compliance fines, brand damage, pulling resources off other projects to fix the damage, etc.

Cybersecurity needs to be seen as a necessary expense, and companies must to be proactive when it comes to implementing a security plan. While cybersecurity itself doesn’t directly generate profits, it protects all your assets and should be treated with a proportional level of importance.

Proper Configurations

Having the right security tools at your disposal is critical and configuring them properly is equally important. Your door can have the best deadbolt in the world on it, but it’s of no use if you don’t lock it correctly.

You need to understand the specifics of your own network and the risks that go along with them. What works for another organization might not apply to your own.

In general though, for critical systems, you’ll want to be very selective of what traffic you allow in and out. In the case of the SolarWinds attack, for instance, a well-configured firewall could’ve stopped the attackers. A rule that only allowed the server to access the internet if it was using OIP protocol AND the FQDN api.SolarWinds.com would’ve kept the network safe.

Don’t just deploy protection, deploy it intelligently.

Code Signing Best Practices

As we touched on earlier, the SolarWinds code signing certificates themselves were NOT compromised or stolen. Rather, the code review and signing processes were compromised. To help minimize risk, always follow code signing best practices and be sure that your staff is trained on them:

• First and foremost, always review and verify all code before it’s signed
• Avoid distributing certificates without any controls in place and do not put them on storage mediums that can be accessed by anyone (like on a USB drive)
• Don’t use the same signing key on all files
• Don’t use the same signing key across product lines and business units
• Have controls in place so that only certain, known people can sign (and only grant access to those that absolutely require it)
• Keep records of who signed what and when the signing occurred

Certificate Management Best Practices

Not only does your organization need to have effective processes in place for the use of code signing certificates (or any kind of certificates, for that matter), but you also need to make sure that you’re properly managing all your various certificates. If not, you’re greatly increasing the risk of a certificate falling into the wrong hands and being used for a nefarious purpose. A few certificate management best practices that your team should follow include:

• Store private keys in a designated vault with strict access controls
• Set granular permissions and only give users access to functions that their job duties require
• Keep a list of all active certificates, including details like the location, owner, what domain they’re for, etc.
• Create certificate issuance and approval workflows to allow for the timely and trusted deployment of certificates
• Automate as much as possible

Going along with that last point, we also strongly recommend the use of an automated certificate management platform. Products like Sectigo Certificate Manager or DigiCert’s CertCentral take the burden of certificate management off your staff, and the automation capabilities minimize human error while allowing you to easily manage every certificate from a single location.

A Cybersecurity Silver Lining?

While the SolarWinds hack has been devastating in terms of its scope, there is the potential that some good will come from it in the end. Hopefully, organizations will use this as a learning experience, motivating them to review their own security infrastructure. The past can’t be undone, but we can learn from other’s mistakes and subsequently implement changes and improvements that will make our own systems safer than ever.