The convergence of IT/OT means OT environments are no longer “walled off” from the rest of the organization or even the rest of the world. Exposure to cybersecurity threats in these systems is growing, and a successful attack could be extremely damaging to production, safety, and system availability.
Managing security and risk in OT environments isn’t as simple as porting over IT security best practices into the OT system. In IT, we’ve had decades to mature our security practices and minimize exposure. But the need to manage risk is universal, and we must adapt our strategies for the OT environments that we’re charged with securing.
The following article is based on a webinar with Mark Mattei, Director of Kudelski Security’s U.S. MSS Operations and Eric Johansen, Security Operations Practice Lead and guests from Claroty, Grant Geyer, Chief Product Officer, and Justin Woody, Director Alliances.
Common Challenges in OT Security
When thinking about OT security strategies, it’s important to understand some of the fundamental differences between IT and OT systems. There are three key areas that call for a more nuanced approach to OT security.
- Risk management should include security risk, but recognize safety and availability are usually top of mind for the OT side of organizations. This leads to information security oftentimes becoming an afterthought – many simply do not have cybersecurity expertise in-house. Indeed, risk to an OT organization typically refers to business risk — e.g. disruption of production, safety issues, inefficient resource utilization, loss of revenue, etc. In order for security strategies to have traction and widespread adoption therefore, they must include the extra step of connecting security risk to business risk factors. Speak OT when you discuss cybersecurity – how you can increase visibility in a non-disruptive way via passive monitoring, for instance – to help evangelize change.
- OT technology obsolescence periods are much longer than IT. Legacy systems that have sometimes been in place for 20-25 years proliferate in OT environments. Compare that to the IT world where equipment rarely lasts more than five years. This results in outdated, diverse endpoints where patches aren’t available, or updates can’t be made due to low compute power. This results in cybersecurity controls becoming that much more critical for OT.
- Production environments run 24x7x365 – In IT security, maintenance windows are frequent, and systems can be updated with regularity. However, the 24×7 nature of OT environments leaves a very small window available for patching and reboots. Even then, there is hesitancy around making changes to a system that is critical to production.
These factors do not constitute insurmountable problems. If you are responsible for security in OT environments, below are six strategies that you can employ to mitigate risk.
Strategies for Managing Security in OT Environments
Strategy #1 – End User Awareness
Frame end user training in terms of business risk, rather than cybersecurity risk.
The same end user security threats in IT environments exist in OT environments — phishing attacks, weak passwords, lack of physical device security. However, the primary focus for an OT engineer is to keep the system running, which means they are often unaware or possibly unconcerned about cybersecurity threats.
To adapt this strategy, it’s important to frame the conversation in terms of business and operational risk, rather than in terms of cybersecurity. It may also be helpful to give OT engineers and plant managers access to the security tools, so they can visualize all their assets and how a vulnerability in one could impact production of the whole.
Strategy #2 – Asset Discovery
Get visibility into processes, assets, sessions, and understand their associated risk.
Asset discovery is a critical security component for IT and OT environments, and yet it is one of the most difficult. OT systems notoriously lack visibility. Many organizations simply don’t know the assets that exist in their environment.
The first step, therefore, is quite simple: Get a detailed understanding of the assets that exist on the OT network. That means documenting the operating systems, the firmware levels, the software installed, the libraries that exist, how each asset communicates with another, and, perhaps most importantly, the criticality of the asset to the overall OT system.
Strategy #3 – Network Segmentation
IT/OT convergence will force OT environments to evolve beyond air-gapped networks.
As more IT elements are introduced into the OT environment, the air-gapped model, which so many OT networks have depended on as a primary security element, is eroding. For example, an OT engineer may want to check his or her email on an HMI on the plant floor, so they add a second NIC. Or, perhaps a vendor wants access to a device to do health and performance metric checks. In an OT environment, operations will trump security every time.
To enable the secure convergence of IT/OT, it’s important to think through network segmentation requirements well before access is requested. Don’t create new connections in an emergency, but rather, take the time to establish system-to-system connectivity through the Purdue Model and set up firewalls and firewall controls to create hierarchy in the network. The Purdue Model of Control of Hierarchy is a framework commonly used by manufacturers across industries and will be helpful to understand how data typically flows through these networks and, correspondingly, how to secure each of the network zones and their various respective elements.
Strategy #4 Threat Monitoring/Hunting and Incident Management
Clearly identify incident management roles and responsibilities throughout the OT organization. Threat monitoring and hunting is useless without it.
Take a crawl, walk, and run approach – knowing that there’s no “easy button” or “switch” you can use to get to stage. Recognize that visibility is the key first step – which leads to knowing what assets are in your environment, how assets connect to each other, how network segmentation is setup (or isn’t setup), and what vulnerabilities exist. Once you’ve established visibility – how will you monitor the network 24x7x365? What will you do when there is an alert? How will you validate it, triage it? What will you do when you have a security incident?
With the security challenges an OT environment presents, an incident can be extremely damaging in a short amount of time. IT security strategies such as threat monitoring, threat hunting, and incident management can help, but they require real-time collaboration and coordination between security and OT teams.
From the SOC or third-party MSSP to the plant manager to the OT engineer, roles and responsibilities must be clearly defined. Who will monitor for threats? Who will sift through the noise? What conditions are you looking for? Who do you notify when they are met?
Strategy #5 – Connectivity and Access Controls
For modern OT organizations, connectivity equals productivity. But many lack the proper access controls to securely connect.
Where well-established identity and access management practices are in place for IT environments, the same cannot be said for OT. Credentials are often shared internally and externally, and access is not limited to specific network devices or segments.
It’s important to assume and plan for “hyperconnectivity” in advance in order to securely enable productivity and operations. The same basic IT IAM principles apply here — identity management, password requirements, multi-factor authentication, syncing access to active directory. Having remote access capabilities can help as well (though avoid having the same remote access solution for both IT and OT in order to reduce attack surface and avoid downtime). In the event of an incident, you can see who had access to the impacted system and terminate connectivity if needed.
Strategy #6 – Vulnerability and Patch Management
Adapt vulnerability and patch management to the systems and maintenance windows of OT and leverage compensating controls in between.
The legacy systems, business criticality, and limited patch windows of OT environments complicate typical vulnerability and patch management strategies. Instead of patching your way through hundreds of vulnerabilities, you need to understand which vulnerable systems are most important to production. Ensure there is a plan in place to remediate during the next scheduled maintenance window – understanding that many OT vulnerabilities don’t have a patch or firmware update fix available at all. This is where leveraging compensating control mechanisms come into their own to limit the impact of the vulnerability of incident. Such mechanisms include the principle of least privilege, network segmentation and isolation (only allowing required traffic for control system operation), password management, and continuous threat monitoring with hunting (deep packet inspection).. Ultimately, it’s all about the balance of revenue and security.
For more information about how you can secure operational technology environments, visit https://www.kudelskisecurity.com/secure-ot-ics-networks/
Critical infrastructure sectors are vital to the functioning of modern societies and are vulnerable to damage from natural disasters, and physical incidents. However, ever since its consolidation with IT networks, Operational Technology (OT) threat landscape has increasingly evolved to accommodate cyber-attacks similar to that of IT networks as well; the same categories of malware that attack IT computers have become relevant to OT computers and systems and the isolation of OT networks can no longer seem to be considered an effective protective measure to OT networks.
Furthermore, international fragmentation regarding cybersecurity policies and procedures and misalignment of incentives for cybersecurity best practices act as formidable hindrances, placing OT practitioners in a difficult position of balancing the market pressures of rapid innovation and sustained investments.
Building upon our success in digital security, our study into the OT threat landscape has helped us summarize that misconfiguration, vulnerable hardware and software components, poor cybersecurity practices by subcontractors, outdated network components, and lack of cybersecurity awareness have been the predominant features easing the OT threat actors’ efforts over the years. Furthermore, IT-based security mechanisms in OT environments have been far less than optimal – firewalls causing excessive latency, undependable threat quarantining techniques, labor-intensive patch/update mechanisms, and unsuitable restore mechanisms. Hence, while we cannot assume that the current IT controls can transition to OT networks, defense-in-depth mechanisms can act as guiding points for securing OT environments – effective policies and procedures revolving around risk management, training and awareness, audits and assessments act as enablers to apply security controls from a standpoint of acceptable risk and prioritize safety and reliability. Physical security, network and host monitoring, and application management complement the efforts of OT security personnel.
Therefore, Kudelski Security believes in weaving together science, technology, and policy to develop sophisticated, yet practical, solutions that will help secure information, computer and network assets in various critical infrastructure sectors. Our Embedded Security Suite provides a three-pronged approach that ensures security is integrated throughout the OT product and system lifecycle and helps guarantee long-term confidentiality, integrity, availability, and safety.
Did this blog interest you or your organization? To better understand your OT risk posture and protection mechanisms that can be applied, click here to read our Operational Technology white paper.
Almost a decade has gone by since I performed my first risk analysis of a nuclear plant and discovered a completely new world. Since then, security professionals will have heard a lot more about the current state OT security (or lack thereof). Operational Technology designates systems specially designed to monitor or make changes to physical processes; these systems are often called Industrial Control Systems (ICS).
It doesn’t matter if we’re referring to Supervisory Control and Data Acquisition (SCADA) systems or Programmable Logic Controllers (PLCs), the fact is security was never considered during the design of OT or ICS systems and the protocols they implement. These systems were not built to be interconnected with traditional IT networks. Their security relies on physical “air gaps” and physical access control to the plants or locations where these systems are implemented.
It’s clear that the risks impacting OT systems have grown exponentially during the last 10-15 years. Additionally, we’ve seen an increase in the attack surface and potential impact of an outage or catastrophic system failure. Risks in this area continue to grow as businesses require interconnectivity between IT and OT networks to enable organizations to provide remote access for engineering, operation, support or monitoring activities.
OT networks often leverage standard commercial off the shelf (COTS) technologies such as Microsoft Windows, SQL Servers, and TCP/IP based networks along with customized ICS/OT hardware. Using these COTS solutions often makes the critical systems vulnerable to the same security risks and issues that IT systems face. In fact, the situation is arguably worse, because often patching is not possible due several operational constraints and availability requirements. These constraints often include the potential of losing vendor support if the underlying COTS software or systems are upgraded or the reality that many of these systems cannot be taken offline or rebooted in order to apply patches because they must keep running 24x7x365.
Another reason it’s not possible and often dangerous to run standard vulnerability scanning products is due to the inherent fragility of those systems and the problems that unexpected traffic can cause to them. To complicate matters further, the non-TCP/IP protocols used within these OT networks are often proprietary protocols where authentication or encryption are not present.
In short, these are technologies built with out-of-date operating systems with dozens (or hundreds) of well-known vulnerabilities, built using an insecure network and communications protocols. These technologies must now be interconnected to the corporate IT systems due to business requirements but the systems cannot be scanned, patched, or secured using traditional security solutions and methodologies. OT/SCADA systems are currently used to monitor and operate everything from factory production chains to the critical infrastructure required to deliver electricity to the masses. What could possibly go wrong here?
The risks highlighted above are not just theoretical, in the past few years we have seen a significant increase in the number of attacks specially designed to target ICS/SCADA systems such as:
- 2010 Stuxnet was uncovered. Stuxnet is worm-like malware that targets PLCs designed to enrich uranium. Stuxnet looked for specific Siemens PLCs connected to very specific hardware and if found modified the configuration causing centrifuges to spin too fast. Stuxnet was a targeted attack addressing the Iranian nuclear program that famously became the first nation-state backed cyberattack design to cause physical damage to industrial control systems.
- December 2015, a Ukrainian regional electricity distribution company reported service outages affecting 225,000 customers and lasted for several hours. The outages were discovered to be part of an attack on the power generation systems. Attackers were able to remotely access and control the ICS to cause the outage and delay the restoration efforts.
- June 2017 Crashoverride was uncovered. This malware specifically targets ICS electric grid components. When Crashoverride infects Windows machines, it automatically maps out the controls systems, records network logs (to later be replayed by operators). Crashoverride is an advanced modular malware framework that can adapt to many protocols and is designed to be stealthy, disruptive, and automatic.
- December 2017 Triton was discovered. Triton is a new malware strain designed to target ICS systems. Triton was discovered after causing a shutdown of critical infrastructure in Saudi Arabia. This malware targets Schneider Safety Instrumented Systems (SIS) controllers. By modifying these SIS controllers, the attackers are able to increase the likelihood of system failures resulting in physical damage to the ICS.
In addition to all these security challenges, we also need to be looking towards the future and prepare for the evolution of ICS and now “IOT” systems. I’m confident that, as we have seen in other industries like finance or telco in the past, ICS and SCADA vendors will move towards providing cloud-based offerings for some of their systems. I really think that in the near future we will be talking about Historian-, HMI-, PLC- or even Control-as-a-Service approaches.
With this risk landscape and the associated challenges, we can easily understand that CISOs are having a tough time being responsible for their organization’s ICS security programs. CISOs will face challenges not only because OT security is an entirely new world for most security professionals, but also because historically priorities and concerns for IT and OT teams have been quite different. The stringent operational and availability requirements placed on OT systems often create difficulties when traditional security teams need to work closely with OT engineers.
Furthermore, when we talk about risks and incidents in ICS we need to keep in mind that the potential damage is going beyond financial losses or reputational damage. Attacks in this space could very likely result in physical losses, severe damage to the environment or even the tragic cost of human lives.
Fortunately, it’s not all bad news since the industry is working diligently to design solutions to help mitigate these risks. New best practices and guidelines have been published such as the ISA/IEC-62443 (Formerly ISA-99), a series of standards and guides on how to implement secure ICS.
Additionally, vendors have recently built technologies to identify anomalies or potential intrusions through passively monitoring traffic that then monitors OT networks? It’s important to note that machine learning approaches will struggle to become operational and effective in traditional IT networks, though they work perfectly well on OT networks.
Machine learning works well in OT environments because the traffic and the communications are very consistent and predictable. These tools are not only useful for security professionals to receive easily understandable alerts on potential threats but are also helping OT teams to gain a new level of visibility within their operational technology network and assets that they’ve never had before. They have clear operational advantages. This allows organizations to both improve their detection capabilities while also providing the OT engineering staff tangible benefits. I believe that working closely with the OT teams to show them the operational capabilities of these OT security solutions will lead to better communication and cooperation between OT and IT teams.
All in all, while protecting and hardening ICS networks is an incredibly difficult challenge for any CISO, there are still paths for the success to be followed. I think the efforts should be put on identifying the potential risks, focus heavily on network segmentation including limiting the potential paths of connectivity between OT and IT networks using one-way data diodes. Finally, building a smart security monitoring approach that not only enables the identification of security threats but also provides visibility and added value to the operational team will be a key factor to success.
Do you want to learn more? Click here to read our new Operational Technology whitepaper.
Critical infrastructure is infused with proprietary protocols and software, air-gapped networks, and robust physical security systems—an amalgam that effectuated the notion of “security by obscurity” in the industrial control systems (ICS) community. Still, business needs eventually necessitated the convergence of information technology (IT) and ICS architectures. Although this seems like a match made in heaven, from an IT perspective, designing visibility and control into a system that inherently lacks them is a challenge that can be painful.
However, IT’s “defense-in-depth” security approach could be effective for ICS security. After all, this strategy employs a holistic approach to protect all assets—people, technology, operations, and adversarial awareness—while considering its interconnections and dependencies to provide effective layers of monitoring and protection based on exposure to cybersecurity risks (Figure 1).
1. A holistic approach. A defense-in-depth security approach employs a holistic methodology to protect all assets while considering dependencies to provide effective layers of monitoring and protection based on exposure to risks. Courtesy: Kudelski Security
The following highlights plausible best-practices for securing ICS environments using a defense-in-depth approach (Figure 2).
2. Defense-in-depth framework. Application and data security are at the center of all security efforts. Courtesy: Kudelski Security
Policy, Procedures, and Training
An effective ICS security program depends on the willingness of the operations staff and management to accept security as an enabler for all computer-oriented activities, as well as their ability to apply controls from a standpoint of acceptable risk.
With this in mind, organizational leadership must clearly define and communicate cybersecurity roles, responsibilities, expectations for performance, and authorities for managers, system administrators, and users through training programs and policies, while holding individuals accountable for their performance. This minimizes the likelihood of organizational personnel inadvertently disclosing sensitive information regarding supervisory control and data acquisition (SCADA) system design, operations, or security controls. Likewise, good management practices in handling delicate situations, recognizing and rewarding employees, and looking after their well-being can help diffuse potential insider threats.
Designing an effective ICS security architecture requires a risk model that maps functional requirements of these complex systems and provides a holistic image of potential real-world consequences. A thorough risk analysis procedure consists of identifying all assets (including software, network elements, and people) in the organization, as well as risk drivers or threats such as disgruntled employees, terrorists, hostile countries, and more.
Establishing a “Red Team” to identify potential attack scenarios and evaluate system vulnerabilities can help detect plausible intrusion methods, which should be evaluated as risks and categorized based on their likelihood of occurrence and impact to the organization. Note that actionable policies and procedures, along with monitoring and feedback, should be part of the risk management program. Periodic review is essential to stay current with evolving threat landscapes.
Vendor and Supply Chain Management
Organizations regularly employ contractors and third-party vendors who do not have uniform cybersecurity policies and practices. This creates exploitable weaknesses in the operations chain. Therefore, it is recommended that third-party requests be reviewed by IT—as well as legal and other relevant departments—with proper documentation. Documentation should be accompanied by regularly scheduled compliance reviews/revalidation, all based on assessed risks while confining intellectual property access to a need-to-know basis only. Likewise, rigid guidelines for evaluating the purchase of new SCADA devices must be established.
Incident Response Management
A comprehensive cyber incident response plan should include both proactive (to prevent incidents) and reactive measures (to detect and manage an incident). Therefore, it is recommended to establish a 24/7 incident monitoring program with the ability to detect threats to the ICS network. Having a comprehensive response plan (such as isolation strategies and disabling affected accounts) when adversarial activity is detected is also important. As critical is having a restoration plan—including establishing system backups (redundant hardware and fault-tolerant systems)—and disaster recovery plans (fallback mechanisms).
Audit and Assess
Auditing eliminates the “paths of least resistance” that an attacker could exploit. This involves technical audits of SCADA devices and networks, physical security surveys, and assessments of all remote sites connected to the SCADA network. This will identify security concerns while maintaining compliance with standards such as NIST-80053, NERC CIP, French ANSSI, CIDX/ACC, AGA 12, API, ISA/IEC 62443, CPNI, CPNI, ISO 27001, and others.
Compliance with standards/regulations does not guarantee continuous security, but it does provide a snapshot of required controls at a point-in-time. Considering numerous factors affect the security of a system throughout its life cycle, periodic testing and verification are important in achieving optimal security.
Physical considerations typically refer to a ringed architecture of layered security measures that restricts access to users to fulfill their duties only. Some measures include authentication for physical access such as key cards and biometrics, facility monitoring (cameras and motion detectors), perimeter defense (fences and anti-vehicle ditches), and visitor escort procedures.
Securing ICS against modern threats requires well-planned and implemented strategies to give network defense teams a chance to quickly and effectively detect, counter, and expel an adversary. Therefore, it is recommended to:
- Document network architecture and identify critical systems, connections to SCADA networks, and host-to-host communications paths. Evaluate the risks and disconnect items that aren’t required.
- Physically separate corporate and control domains. Ensure isolation of ICS networks from untrusted networks and allow real-time connectivity to external networks only if there is a defined business requirement or control function.
- Logically segment networks and isolate critical parts of systems. Demilitarized zones (DMZ) and data warehousing provide a secure buffer zone where services and data can be shared and secure transfer of data from the SCADA network to business networks can be ensured.
- Deploy network access control and manage authentication (preferably two-factor or more) by requiring separate credentials for corporate and control network zones, and store these in separate trust stores. Never share active directories, RSA ACE servers, or other trust stores between corporate and control networks.
- Require any remote access to be operator-controlled and time-limited. Firewalls, virtual private networks, callback (for dial-up), multi-factor authentication, user access control, and intrusion detection can provide “secure” remote access to computer networks.
- Engage network monitoring tools and complement them by enabling logging on all systems. Regularly audit system logs to detect suspicious activity as soon as possible.
- Take measures to avoid “watering hole” attacks. Use a web domain name (DNS) reputation system. Get updates from authenticated vendor sites. Validate the authenticity of downloads. Insist vendors digitally sign updates and publish hashes via an out-of-bound communications path, and require they use these to authenticate.
- Lockdown all unused ports, services on routers, switches, and network daemons. Change all default configurations and passwords.
- Deploy deception networks to boost the odds of finding an adversary early and mitigating overall damage.
Asset inventory is an accurate baseline for identifying necessary security controls. Having identified the assets, lock down all unused ports and services on the host, and restrict privileges to only those needed. Also, manage authentication (preferably multi-factor) with secure password policies—stressing length over complexity—which should be unique and changed at least every 90 days. Harden the host by methods that include application dynamic whitelisting, memory protection, write protection and read protection.
Implement change management policies and procedures for protection against improper modifications prior to, during, and after commissioning. Have a configuration/patch management program centered on the safe importation and implementation of trusted patches. Monitor host activity and alert unauthorized changes.
Application and Data Management
Applications and data are critical elements of ICS environments. Avoid embedding hard-coded passwords in ICS applications. Also, demand that vendors disclose any backdoors or vendor interfaces to your SCADA systems and expect them to provide systems that are capable of being secured.
Conduct an initial assessment (static and dynamic analysis) and ensure compatibility of the application with the host operating system before deploying it. Restrict access to the application and data only to intended users. Finally, it is recommended to use cryptographic controls and data sanitation techniques to maintain the integrity and authenticity of the data collected.
3. ICS threat spectrum. State-sponsored actors have the motivation, capabilities, and means to be especially disruptive, but defense-in-depth security solutions are particularly effective against those threats. Courtesy: Kudelski Security
No environment is 100% secure. A threat-actor, through intent, capability, and opportunity, will always pose a threat to an ICS network by trying to compromise an organization’s systems through its operations, personnel, technology, and other vulnerabilities. Implementing the strategies and controls presented in this article can greatly improve the security posture of ICS.
This said, the determination of a security control is context-based, and there might arise a situation where ICSs have functional or operational properties that disallow application of a security control. In such cases, it is recommended to identify, assess, and implement necessary compensatory controls and ensure the SCADA security policies and standards complement the organization. IT security policies should also evolve to meet changing threat profiles and be scalable to accommodate different standards and regulations.
It needs to be foremost in everyone’s mind that in the SCADA world, availability, reliability, and stability are the most important criteria to be considered.
—Vishruta Rudresh is senior cybersecurity researcher at Kudelski Security www.kudelskisecurity.com.
Courtesy of Power Magazine. Read the original article here.