Although SpireTech uses Kaseya software for systems management, neither SpireTech or our customers were victims of the latest headline news breach. Our response, analysis of the hack, explanation of why we were not affected, and plans moving forward are below.
Background
Kaseya makes software for IT systems management. It is used by enterprises and managed service providers alike to streamline technician effectiveness, enabling a few technicians to manage thousands of systems at scale, including patch management, health monitoring, and providing helpdesk services. Kaseya, based in Miami Florida, is one of the largest vendors of this type of software. SpireTech has been using Kaseya software for over ten years.
Initial response
We sent the following notification to VIPsupport client key contacts on Friday afternoon:
On Friday 7/2/21 at 12:48pm PT we were notified by our Remote Monitoring & Management vendor, Kaseya, of an active security incident involving their software being used to deploy ransomware, and advising us to shutdown our management server until security experts can determine the cause.
We have shutdown our server under the presumption this will protect us (and you), and are actively monitoring our Sophos Intercept-X software for indicators of compromise – and at this point, there are none.
We will continue to monitor the situation closely, and our server will likely remain shutdown until after the holiday weekend. In the meantime, we have alternate software we can use to support customers when screensharing is needed.
Actions over the Independence Day holiday
We monitored the situation over the weekend, and watched as security experts dissected how the attack was launched, and monitored Kaseya’s response. Kaseya owned up to the situation, now making worldwide news, and was providing updates at least twice a day. Kaseya brought in security experts, engaged with government officials, cooperated with outside security researchers, and had their own teams working around the clock to solve the problem.
Tuesday July 6th, 2021:
Using the latest scan tools available from Kaseya and Sophos Endpoint software we determined that neither our server, nor any of our customers systems were affected by the attack. Kaseya is still recommending that all of their customers keep their Remote Management servers offline until they release a patch. Currently this is expected to happen July 8th. We intend to apply the patch and bring our system back online carefully as soon as possible. We will give Kaseya time to make sure they get the patch right, and proceed with caution when we believe the time is right to do so.
Huntress, a security vendor that sounded the early warning that likely thwarted the attack, provided a proof-of-concept demonstration of the attack and detailed postmortem that several of our employees attended. At this webinar we gained a better understanding of why we were not vulnerable to the attack. It also revealed that hackers had a persistent presence in the days leading up to the attack, and that the ransomware was timed to trigger at 4:30PM UTC Friday. The attack could have been far worse. It is unclear why they didn’t attack more companies or exfiltrate data – it appears to be a quick smash and grab.
Analysis of why we were not vulnerable:
Our Kaseya server resides behind a firewall that requires multi-factor authentication to access. However, certain elements of the Kaseya server need to be exposed to the internet for it to function correctly. In our implementation, we have been selective about what those elements are – eg, the most restrictive access possible, and this is part of what reduced the attack surface of our server.
In the postmortem provided by Huntress, we see the file being abused, dl.asp, is one that has to be exposed in a normal installation for Kaseya to function properly as it is needed for initial software installation. However at Spire, we do not use this method to download software installations – we’ve re-written it entirely, with different and enhanced functionality, so our Kaseya server is not vulnerable to this attack vector.
Wednesday July 7th, 2021
The timeline for service restoration has been repeatedly pushed back. Kaseya had difficulties with the patch, and has announced further delays. We are not in a hurry – quality is more important than speed.
The current downtime means that some of our standard processes have been affected. We are currently not able to monitor for some statuses, such as failing hard drives, and are not able to run our automatic daily procedures on all workstations, apply updates, and so on. However, we are still assisting our customers with their day-to-day technical needs using other tools.
Another reality check
Attacks like these are something that will continue to happen in our current connected world, more and more often large software vendors like Kaseya and Solarwinds and in turn customers using their software are being targeted. The importance of following all security best practices such as using MFA and maintaining a robust backup solution cannot be overstated as they are the best prevention against becoming the next victim.
Now, the supply chain can be the vector – your vendor’s vendors. Just like when the Solarwinds hack breached Microsoft, FireEye, and Palo Alto networks – if major security vendors themselves can be breached, the entire supply chain is vulnerable. Oftentimes we learn of breaches much later after hackers have had a persistent presence.
Luckily, our existing preventative measures – ones that go beyond Kaseya’s recommendations — protected us and our clients. No software is free of vulnerabilities, but we respond quickly when we learn of them to keep our clients secure to the best of our abilities.
As far as Kaseya’s response – it has been immediate, proactive, and communicative – instead of burying their head in the sand or deflecting, they’ve owned the issue and warned everyone via phone calls, emails and messages on Friday, likely preventing much greater damage. They kept us updated over the holiday weekend, and we’ve been monitoring their progress as they work through the issue. They are properly resourced to deal with the issue, and although we could seek another software vendor – similar hacks have happened to competing solutions.
Ultimately, we believe the day will come with Kaseya may no longer be necessary and we will have the ability to mostly replace its functionality with tools from the Microsoft stack, such as Intune/EMS. Currently, there is not feature parity and this would be a very inefficient and labor-intensive effort – so tools like Kaseya exist.
More Security tools are needed
Yesterday’s tools and solutions are not going to protect you from today’s threats. At SpireTech, we often review new security solutions, software, or tools to add to our stack. Clients often have a resistance to pay for these new tools, so sometimes we absorb the cost ourselves. This thought process needs to change. Security is a team sport – and good security costs money.
However, a good dose of reality is needed – we are not aware of any vendor saying their tool successfully stopped the attack before it happened. Signature detection was added after-the-fact. Because the ransomware was code-signed and distributed via a trusted process, it got past early detection.
Please feel free to contact us if you have any questions.