Critical infrastructure systems are becoming increasingly connected to traditional IT systems, and as a result, are increasingly targeted.
Critical infrastructure systems are becoming increasingly connected to traditional IT systems, and as a result, are being increasingly targeted. A Siemens study found that 56 percent of the world’s gas, wind, water and solar utilities experienced at least one shutdown or operational data loss per year. The potential repercussions of a critical infrastructure breach within an industrial setting go far beyond financial loss or reputational damage.
Attacks in this space have already resulted in large-scale societal consequences. A cyber attack on Colonial Pipeline, the largest pipeline system for refined oil products in the U.S., caused the company to suspend operations and left the East Coast with a temporary gas shortage. In this case, Colonial Pipeline shut down its fuel pipeline operations pre-emptively even though its’ operational technology (OT) systems were not directly impacted. Sources claimed that the shutdown was due to the invoicing and billing systems being encrypted and unavailable, leaving the business with no way to properly track or invoice clients for fuel.
Colonial Pipeline’s hack is an important case study about how critically interdependent IT and OT systems are, even if they’re segregated technically and air gapped appropriately. This incident highlighted the imminent need for cybersecurity and risk management programs for an organization’s operational/industrial control system (ICS) environments, as well as security being at the forefront for all OT engineers and plant managers.
Remote access vulnerabilities
Within the past few years, the convergence of IT and OT systems has expedited the adoption of remote access technologies in critical infrastructure settings. More recently, the COVID-19 pandemic forced organizations to limit the number of people who were physically located within a plant or OT site. This drastically increased the number of ICS management and monitoring systems that are directly connected to the internet, potentially leaving them accessible to remote attackers.
These unprotected remote access systems or solutions essentially act as bait to threat actors wanting to compromise critical infrastructure systems. For example, a security researcher from the University of Tulsa revealed the ability for hackers to control entire networks of U.S. wind farm turbines. The researchers broke into a facility and installed a Raspberry-Pi-based computer through which they were able to access the systems remotely. This experiment shone a light on the simplicity of covertly installing unauthorized remote access systems that provide easy access to OT systems that were through to be fully air gapped from the internet, and the potential damage that can be done.
Understanding OT vulnerabilities to mitigate the risks
Despite these glaring vulnerabilities, there is a widespread lack of knowledge about asset protection across the manufacturing, energy, and oil and gas industries. Many OT operators and engineers, unfortunately, are not yet aware of the severity of the potential risks that insecure remote access points can bring. There is a significant disconnect between risk perception and actively implementing processes and procedures to mitigate those risks, with little incentive to replace or improve equipment that still functions but does not meet security requirements. Industry professionals must be educated on the risks of vulnerable remote access points.
The reality of OT systems is that plants and manufacturing sites are often designed and built to last and to remain “stable” for decades to come. This means that there are often no plans to keep software updated by installing security patches, changing configurations to make the systems more secure, or reducing risk by turning off unnecessary features on Programmable Logic Controllers (PLCs). Consequently, the most common risks in critical infrastructure OT systems include the following: outdated operating systems, unencrypted passwords used to connect to systems over unsecured networks, remotely accessible devices, lack of passive network monitoring, weak access control systems, and the failure to keep antivirus signatures updated to track new malware strains.
It’s not uncommon for security providers to deploy passive network monitoring systems in OT environments and immediately see years old malware running rampant (such as slammer or conflicker) with the OT engineers being none the wiser. The attacks against these systems typically don’t require sophisticated zero-day vulnerabilities in the software used on engineering workstations or in PLCs but instead rely on abusing poor security hygiene and differing priorities.
Preventing future hacks
Poor cyber hygiene remains the top challenge in securing these systems. Thankfully, some steps can be taken to reduce risk. One key priority is to focus on protecting IT / OT network boundaries. More specifically, focus on remote access systems and software that may allow direct access to critical OT networks to enable engineers to monitor systems remotely.
Perhaps most importantly, proper education and increased awareness remains the most effective way to prevent incidents. If detection and risk mitigation is to be a priority, education must come first. Site managers and OT engineers must be aware of these risks and work closely and collaboratively with corporate information security teams to appropriately defend these environments in a non-disruptive way.
One of the simplest ways to deploy risk mitigation measures is deploying an approved set of remote access tools designed specifically for OT environments and ensuring these systems are appropriately configured to record a user’s activity and limit their access based on least privilege.
Implementing security practices and procedures is not a choice for most business-critical OT systems. Security must be at the forefront of all new critical infrastructure systems to protect against a growing number of vulnerabilities due to the convergence of IT and OT systems and the increasing number of access points. Companies must mitigate the risk of OT breaches and ensure their risk management and information security programs also help protect these environments.
This article was originally featured in Industry Today.
On July 2nd, a large-scale supply chain attack operation by the REvil ransomware group affected multiple I.T Managed Service Providers (MSPs) and leveraged the I.T MSP’s Kaseya VSA instances to infect the MSP’s clients. As of this writing the attack campaign has affected 60 I.T MSPs and over 1500 end clients.
The attack was operated by compromising self-hosted Kayseya VSA servers. The threat actors appear to have gained access by abusing authentication bypass and command injection bugs present on the management web UI. Once threat actors gained access to the VSA servers, they quickly locked legitimate users out of the systems and delivered a malicious payload to end user systems the compromised I.T management tool.
The Kudelski Security Cyber Fusion Center and Kudelski Group were not affected as this solution is not leveraged internally nor externally.
All self-hosted VSA servers. Unfortunately, there is currently no\ patch available, as such it is strongly recommended to keep the servers shutdown.
Once threat actors used their initial access to VSA servers they locked out administrators and leveraged VSA’s update mechanism to deploy their malware as a base64 encoded “.crt” file. The threat actors then used a powershell command to disable Windows Defender Antivirus, decode the file and save it in the c:\kworking directory of the Kaseya VSA software (which was typically excluded from AV scanning as recommended by Kaseya). Finally, the agent.exe malware dropper is started by the Kaseya agentmon.exe binary, gaining system level privileges.
The malware dropper extracted from the encoded agent.crt file was digitally signed with a valid digital signature using the following information:
• Name: PB03 TRANSPORT LTD.
• Email: [Brouilettebusiness@outlook[.]com]
• SUBJECT: CN=Sectigo RSA Code Signing, CAO=Sectigo Limited, L=Salford, S=Greater Manchester, C=GB
• Serial #: 119acead668bad57a48b4f42f294f8f0
• Issuer: https://sectigo[.]com/
Once executed, the dropper writes the following files to the c:\Windows path:
• MsMpEng.exe – a legitimate but very outdated Windows Defender executable
• Mpsvc.dll – the encryptor payload complied as a dynamic link library that is sideloaded by the vulnerable Defender executable
Known associated IOCs (SHA256):
• agent.exe (d55f983c994caa160ec63a59f6b4250fe67fb3e8c43a388aec60a4a6978e9f1e)
• mpsvc.dll (e2a24ab94f865caeacdf2c3ad015f31f23008ac6db8312c2cbfb32e4a5466ea2)
• mpsvc.dll (8dd620d9aeb35960bb766458c8890ede987c33d239cf730f93fe49d90ae759dd)
The threat actors appear to have performed initial exploitation activity from the following IP addresses:
• 18.223.199[.]234 (Amazon Web Services)
• 161.35.239[.]148 (Digital Ocean)
• 35.226.94[.]113 (Google Cloud)
• 162.253.124[.]162 (Sapioterra)
Cyber Fusion Center has been actively monitoring this attack campaign and continues to track the situation to keep our clients updated. The CFC will perform threat hunting on the IOCs listed in this advisory and any updated IOCs released in the future.
Additionally, the techniques leveraged by the threat actors in this attack campaign are not unique or novel, several threat actors have leveraged PowerShell cmdlets to disable security solutions in the past and often use the Certutil binary to decode or download malicious files. The CFC is able to actively monitor and response to these techniques leveraging Endpoint Detection and Response (EDR) tooling.
Kaseya’s R&D team was able to replicate the attack vector and is working on the process of remediating the malicious code and applying necessary patches.
All Kaseya hosted VSA servers as part of Kaseya’s SaaS solution were put into maintenance mode by Kaseya to prevent further exploitation.
Self-hosted VSA servers should remain shutdown until Kaseya provides a patch for the issue.
Colonial Pipeline, Oldsmar incidents highlight the challenge of securing older operational technology systems
Critical infrastructure is vital to the functioning of modern societies and economies, yet often these systems are not properly protected or are easily accessed and exploited, and thus remain a key target for threat actors. Although awareness around the severity of operational technology (OT) cyber risks is on the rise, the fact is, OT environments remain vulnerable.
In the first few months of the year, we’ve already seen news of several vulnerabilities in the sector exploited, such as the Florida water plant breach and most recently, the ransomware attack on Colonial Pipeline, one of the United States’ most critical fuel pipelines.
Given the longevity of the systems and technology implemented in industrial settings, security has historically been relegated to a second tier of priorities compared to uptime, reliability and stability. It comes as no surprise that 56 percent of the world’s gas, wind, water and solar utilities experience at least one shutdown or operational data loss per year, according to a Ponemon Institute report. That number has likely grown because of the pandemic, as many organizations weren’t prepared for remote management of critical systems. In fact, although leaders agree on the importance of remote access, Claroty reported last year that 26 percent of organizations struggled with the newly dispersed workforce and 22 percent did not have a pre-existing secure remote access solution that is secure enough for OT.
As OT environments continue to evolve in the face of new potential disruptions, it is time for leaders to prioritize security and understand implications so they can act to protect their organizations and nations’ critical infrastructure.
Learn more about the importance for OT cybersecurity in the Energy, Oil and Gas Industry by downloading our eBook
Understanding the New OT landscape
In the past few years, we have seen a convergence between OT and IT-based security infrastructures and processes. However, as we saw in the Colonial Pipeline attack, these integrated ecosystems have become considerably more difficult to secure, from misconfiguration, vulnerable hardware/software components and poor cybersecurity practices to the lack of visibility into connected assets and poor network segmentation.
Beyond the OT-IT environment convergence, the pandemic pushed many organizations to alter their cybersecurity processes to accommodate the new needs of remote work. However, adversaries quickly realized that targeting workers at home provided a viable path into OT networks, and turned to exploiting work from home, leveraging unpatched virtual private network (VPN) systems, interconnected IT and OT environments, and exploiting vulnerabilities in legacy Windows and OT systems.
OT has fast become a prime target for motivated and well-resourced threat actors who continue to redesign their tactics to penetrate new and enhanced security measures. In fact, 2020 saw a significant increase in exploitable vulnerabilities in OT. ICS-CERT advisories increased by more than 32 percent last year compared to 2019, and more than 75 percent of advisories were about “high” or “critical” severity vulnerabilities. Threat actors are also using ransomware campaigns to target OT environments because they understand how mission-critical these environments are. For example, if a pipeline carrying 45 percent of the United States’ East Coast’s fuel is shut down, it costs the pipeline operator millions of dollars per day.
The specialized and mission-critical nature of OT infrastructure technologies means that most security and threat intelligence solutions don’t have visibility into potential vulnerabilities, let alone the ability to defend against attacks.
Preventing and Mitigating Risks
So, what can be done to enhance security in today’s OT landscape? To protect, prevent and mitigate risks, there are several important steps organizations can take to improve their security posture.
- Implement a risk management program: OT is built around complex systems that oftentimes are not properly tracked in traditional asset management systems. Designing an effective OT security program requires a risk model that specifically maps the functional requirements of these systems while providing a holistic image of the potential real-world consequences of compromise. As part of the program, organizations that leverage the Purdue Model should ensure they’re documenting the number of traffic flows between levels, especially if the flow is across more than one Purdue level.
- Build a cyber incident response plan: If there was something we should have learned from the COVID-19 pandemic, it is that we need to be ready for anything. A comprehensive cyber incident response plan that includes both proactive and reactive measures is required to help prevent incidents and better allow the organization to respond if one does occur. Make sure to print the response plan and have it handy. What happens if the systems that store your incident response plan are encrypted or unavailable due to an attack?
- Protect third-party remote access: Organizations regularly rely on third-party vendors to complement their business; however, many do not have uniform cybersecurity policies and practices. Many OT sites even have third party vendors regularly conduct maintenance via remote access technology, which creates exploitable weaknesses in the operations chain. Establishing a supply chain management program that vets external vendors’ security standards and provides better control of third-party access is critical to reducing the risks third parties introduce.
- Enhance system monitoring procedures: It is no longer enough to simply build a network with a hardened perimeter. Securing OT systems against modern threats requires well-planned and well-implemented strategies that will allow defense teams to quickly and effectively detect, counter and respond to adversaries. At a minimum, corporate IT and OT domains should be physically and logically separated, networks must be segmented, and critical parts of the network isolated from untrusted networks, especially the internet. It is also important to deploy monitoring tools such as passive intrusion detection systems (IDS) specifically designed for OT environments. Passive systems are key because proactive systems may present false positive detection that could lead to downtime of critical systems.
- Develop informed security controls: To establish the required controls, we have to start with an asset inventory. Once the assets have been identified, organizations at a minimum need to implement the security features provided by device and system vendors. However, to deal with some critical vulnerabilities, we recommend turning on security features that apply Common Industrial Protocol (CIP) security controls, a fairly universal standard. Many PLC vendors also have physical switches on their appliances that prevent the changing of the PLC’ configurations, which should be used appropriately. We see many plants and OT sites with these switches always set to “config mode,” which allows for the PLC configuration to be changed (potentially by an attacker). These should be complemented with secure and hardened configurations (read/write protections, memory protection, etc.). Managing controls over time can be daunting and time intervals between OT system upgrades can be years long, so organizations need an effective change management program. The program should be able to identify compensatory controls that can be applied to remediate critical vulnerabilities that cannot be patched immediately. These controls can include a host monitoring system that detects and alerts when unauthorized changes are made to Human Machine Interfaces (HMIs), engineering workstations or to PLCs.
- Establish audits and security assessments: Finally, numerous factors affect the security of a system throughout its life cycle, so periodic testing and verification of the system are essential. Timely audits and assessments help eliminate the “path of least resistance” that an attacker could exploit.
This article was originally featured in Security Infowatch.
Tips for Breaking Through
In my last blog post, I looked at how challenges relating to SIEMs, default configurations, device-led strategies, and competing priorities can impede efficient threat detection and response. In this post, I’ll look at three things you can do to address them and how Kudelski Security MSS can help..
1. Develop your use cases (aka your common “lens”).
I mentioned use cases before, in the context of alert fatigue that arises when we let devices dictate our detection strategy and evaluate alerts across devices through a common lens. A use case is a high-level threat detection priority, not to be confused with detection rules.
For example, phishing might be a threat detection use case for your organization.
While it seems simple, to fully understand each use case, we have to understand your adversary, their motivations and the techniques they use to move through the kill chain.
We also have to understand how that attack will play out within the environment (scenarios) and the data sources required to detect the attack.
Finally, we have to know how each of the parties involved in containment and remediation should respond.
Kudelski Security has developed its own use case framework based on our years of threat detection and monitoring experience. This framework includes 16 common attack vectors with scenarios mapped to the enterprise MITRE ATT&CK matrix. One use case could have upwards of a hundred scenarios, and these scenarios are what you’ll tie your detection rules to. Even if you’re not a client of ours, it’s helpful to think of your threat detection strategy in this top-down way.
2. Prioritize detection based on your threat model.
Use cases provide the foundation for your threat detection strategy, but given the hundreds of potential scenarios and thousands of potential alerts, it’s still a lot to wade through. Threat modeling will help narrow this down to help you prioritize your detection strategy.
At Kudelski Security, when we onboard clients, we take them through a threat modeling exercise to identify the attacker groups targeting their geographic region and industry and the objectives of that group—e.g. ransomware, disruption of critical processes, political motivations, etc.
We also work to understand what we’re defending (from a technical and business perspective).
With their threat model defined, we can identify the Tactics, Techniques, and Procedures (TTPs) those threat groups use and map them to MITRE techniques. We may find that there are overlapping techniques between the groups.
Where we have the most overlap is where we’ll start with our detection strategy, then the second most, the third, and so on.
3. Collect the right data.
It’s only after completing the two steps above that you’re ready to actually start collecting data. You know the types of threats you’re most vulnerable to and the TTPs associated with those threats. Better yet, you know which tactics are shared across threats, helping you focus detection efforts where they’ll have the most immediate impact. This should give you a clearer picture of the sources and types of data you should be pulling into your SIEM.
As a rule of thumb, we always recommend collecting the following types of data. Nearly all the IR projects our team is brought into could have been detected with properly configured alerts from these sources.
|Microsoft Windows Logs (not just defaults)
||Intrusion Detection Systems
||IaaS Cloud Logs
|Mail Server / Gateways
||Web Application Firewalls
||Firewall Logs (accept & deny)
I started this series out with some grim facts about our ability to successfully detect threats, so it’s only fair that I end with some good news.
You likely already have everything you need to improve threat detection and response. What matters is collecting the right data from the right sources to detect the right threats for your environment.
It’s something my team and I are happy to help you out with. Just drop us a line.
This post summarizes content presented during a session at the 2021 European Cyber Summit session “Threat Monitoring and Detections as Code.”
For more information about Kudelski Security’s Managed Detection & Response capabilities, click here.
In the first of a two-part blog post on Managed Detection and Response, Fran Donoso, senior director of global security strategy, discusses four major issues that will be familiar to any security leader who has wrestled with making threat detection and response more efficient.
I hate to be the bearer of bad news, but your current detection strategies aren’t working. 91% of attacks aren’t even generating alerts and just over half of breaches are actually discovered by the security team. I wish I could say there was some larger issue at play here, but this is largely a problem we’ve brought upon ourselves. We set up our SIEMs, we categorize all of our alerts, and we put a response plan in place (hopefully). And yet, the problem persists. We’re doing all the right things, or at least we think we are. We and our technology are getting in our own way, creating roadblocks where we should be creating fast lanes that accelerate time to detection and response. Meanwhile, attackers are sitting inside our environments, sometimes for months or more, and we don’t even know it until the aftershocks are felt.
These roadblocks occur because for years our approach to threat detection has lulled us into a false sense of security where we think more data (which just leads to more alerts) is going to help us stop a breach. What if I told you that we as security professionals can be more effective with fewer alerts from fewer data sources. Sound crazy? I’m not the only one thinking this way. Here’s what Gartner has to say on the topic:
“Many customers fail with their threat monitoring, detection and response initiatives, because of the focus on monitoring a variety of log sources from whatever technologies they have deployed, instead of having the right sources generating telemetry and alerts, at the right time, in the right format, in the right locations.”
So, we know we’ve got issues, and we know we want to change. What should change look like? That’s what we’ll get into in this post. First, let’s dig a little deeper into what’s holding us back today.
The Four Roadblocks to Faster Threat Detection
Most organizations get SIEM wrong.
Let’s start with the low-hanging fruit, here. We’re bad at SIEM.
Too dramatic? Here’s what I mean. SIEM was initially positioned as the solution to all of our security monitoring problems. It would pull all your security telemetry into one place, allowing us to centralize triage and initiate response activities. Great! But it turns out not all that data coming into your SIEM are relevant plus there’s so many of them it all becomes noise, a queue for some poor SOC analyst to sift through.
To be fair, this really isn’t your SIEM’s fault. It’s only going to ever be as good as the data it’s collecting, and we’re feeding it some pretty bad data. If you have not set up your SIEM to collect the right (non-default) Microsoft Windows Event data or antivirus logs, then your SIEM is never going to tell you about the attack that’s using Cobalt Strike to move laterally throughout your network. This brings us to our next roadblock.
Default configurations kind of suck.
There. I said it. Default configurations are default. That means they’re designed with the lowest common denominator in mind. They don’t know your environment or the threats to it. They’re just there. And if you’re relying on default configurations for security alerting, you’re going to be missing critical pieces of information that could help you stop an attack earlier.
Take, for example, the default Windows logging configurations. If you plugged it in and turned it on, you’d be missing out on all these events.
No alerts for changes to policies or files or privileges or the security state. No logs related to Remote Prodcedure Call (RPC) events or removable storage or system integrity. It’s so limited that we created a CFC client guide that tells our customers what the defaults should be.
We let devices dictate our detection strategy.
You get a new IDS tool. You look at all the types of alerts you can get from it, you categorize them, and you figure out a response plan. Which leaves you with thousands of alerts from that one tool. Plus a thousand from your firewall, another thousand from your cloud service provider and so on. It’s a lot to sift through, and it may not even tell you all that much. An alert in your IDS may seem like no big deal in isolation. You need a common lens through which you can evaluate alerts coming in across all your relevant devices. At Kudelski Security, we call these lenses “use cases.” More on that in part two.
Everything’s a priority.
When everything’s a priority, nothing is a priority. And nowhere is that more true than threat detection and response. If you are set up to collect every log for every potential indicator of every potential threat, your team will quickly be overwhelmed with data, making it harder to identify and respond to the threats that actually do matter to you. You need to be able to see the forest through the trees.
Figuring out what those threats are requires a threat modeling exercise. Threat modeling may have once been considered a “nice to have” in the enterprise security space, but I’d argue, for the reasons above, that without it, you will end up running on a treadmill of alerts that never really gets you anywhere. The goal is continuous improvement. Start with the most important thing you can reasonably do today and then move on to the next. But you have to know where to start, and threat modeling can help you with that.
But it’s not all bad news. Look out for the second part of this series, which will look at practical advice on how to get round the roadblocks.
Learn about Kudelski Security’s Managed Detection and Response capabilities here.