by Mark Mattei | Oct 1, 2020 | Operational Technology
The convergence of IT/OT means OT environments are no longer “walled off” from the rest of the organization or even the rest of the world. Exposure to cybersecurity threats in these systems is growing, and a successful attack could be extremely damaging to production, safety, and system availability.
Managing security and risk in OT environments isn’t as simple as porting over IT security best practices into the OT system. In IT, we’ve had decades to mature our security practices and minimize exposure. But the need to manage risk is universal, and we must adapt our strategies for the OT environments that we’re charged with securing.
The following article is based on a webinar with Mark Mattei, Director of Kudelski Security’s U.S. MSS Operations and Eric Johansen, Security Operations Practice Lead and guests from Claroty, Grant Geyer, Chief Product Officer, and Justin Woody, Director Alliances.
Common Challenges in OT Security
When thinking about OT security strategies, it’s important to understand some of the fundamental differences between IT and OT systems. There are three key areas that call for a more nuanced approach to OT security.
- Risk management should include security risk, but recognize safety and availability are usually top of mind for the OT side of organizations. This leads to information security oftentimes becoming an afterthought – many simply do not have cybersecurity expertise in-house. Indeed, risk to an OT organization typically refers to business risk — e.g. disruption of production, safety issues, inefficient resource utilization, loss of revenue, etc. In order for security strategies to have traction and widespread adoption therefore, they must include the extra step of connecting security risk to business risk factors. Speak OT when you discuss cybersecurity – how you can increase visibility in a non-disruptive way via passive monitoring, for instance – to help evangelize change.
- OT technology obsolescence periods are much longer than IT. Legacy systems that have sometimes been in place for 20-25 years proliferate in OT environments. Compare that to the IT world where equipment rarely lasts more than five years. This results in outdated, diverse endpoints where patches aren’t available, or updates can’t be made due to low compute power. This results in cybersecurity controls becoming that much more critical for OT.
- Production environments run 24x7x365 – In IT security, maintenance windows are frequent, and systems can be updated with regularity. However, the 24×7 nature of OT environments leaves a very small window available for patching and reboots. Even then, there is hesitancy around making changes to a system that is critical to production.
These factors do not constitute insurmountable problems. If you are responsible for security in OT environments, below are six strategies that you can employ to mitigate risk.
Strategies for Managing Security in OT Environments
Strategy #1 – End User Awareness
Frame end user training in terms of business risk, rather than cybersecurity risk.
The same end user security threats in IT environments exist in OT environments — phishing attacks, weak passwords, lack of physical device security. However, the primary focus for an OT engineer is to keep the system running, which means they are often unaware or possibly unconcerned about cybersecurity threats.
To adapt this strategy, it’s important to frame the conversation in terms of business and operational risk, rather than in terms of cybersecurity. It may also be helpful to give OT engineers and plant managers access to the security tools, so they can visualize all their assets and how a vulnerability in one could impact production of the whole.
Strategy #2 – Asset Discovery
Get visibility into processes, assets, sessions, and understand their associated risk.
Asset discovery is a critical security component for IT and OT environments, and yet it is one of the most difficult. OT systems notoriously lack visibility. Many organizations simply don’t know the assets that exist in their environment.
The first step, therefore, is quite simple: Get a detailed understanding of the assets that exist on the OT network. That means documenting the operating systems, the firmware levels, the software installed, the libraries that exist, how each asset communicates with another, and, perhaps most importantly, the criticality of the asset to the overall OT system.
Strategy #3 – Network Segmentation
IT/OT convergence will force OT environments to evolve beyond air-gapped networks.
As more IT elements are introduced into the OT environment, the air-gapped model, which so many OT networks have depended on as a primary security element, is eroding. For example, an OT engineer may want to check his or her email on an HMI on the plant floor, so they add a second NIC. Or, perhaps a vendor wants access to a device to do health and performance metric checks. In an OT environment, operations will trump security every time.
To enable the secure convergence of IT/OT, it’s important to think through network segmentation requirements well before access is requested. Don’t create new connections in an emergency, but rather, take the time to establish system-to-system connectivity through the Purdue Model and set up firewalls and firewall controls to create hierarchy in the network. The Purdue Model of Control of Hierarchy is a framework commonly used by manufacturers across industries and will be helpful to understand how data typically flows through these networks and, correspondingly, how to secure each of the network zones and their various respective elements.
Strategy #4 Threat Monitoring/Hunting and Incident Management
Clearly identify incident management roles and responsibilities throughout the OT organization. Threat monitoring and hunting is useless without it.
Take a crawl, walk, and run approach – knowing that there’s no “easy button” or “switch” you can use to get to stage. Recognize that visibility is the key first step – which leads to knowing what assets are in your environment, how assets connect to each other, how network segmentation is setup (or isn’t setup), and what vulnerabilities exist. Once you’ve established visibility – how will you monitor the network 24x7x365? What will you do when there is an alert? How will you validate it, triage it? What will you do when you have a security incident?
With the security challenges an OT environment presents, an incident can be extremely damaging in a short amount of time. IT security strategies such as threat monitoring, threat hunting, and incident management can help, but they require real-time collaboration and coordination between security and OT teams.
From the SOC or third-party MSSP to the plant manager to the OT engineer, roles and responsibilities must be clearly defined. Who will monitor for threats? Who will sift through the noise? What conditions are you looking for? Who do you notify when they are met?
Strategy #5 – Connectivity and Access Controls
For modern OT organizations, connectivity equals productivity. But many lack the proper access controls to securely connect.
Where well-established identity and access management practices are in place for IT environments, the same cannot be said for OT. Credentials are often shared internally and externally, and access is not limited to specific network devices or segments.
It’s important to assume and plan for “hyperconnectivity” in advance in order to securely enable productivity and operations. The same basic IT IAM principles apply here — identity management, password requirements, multi-factor authentication, syncing access to active directory. Having remote access capabilities can help as well (though avoid having the same remote access solution for both IT and OT in order to reduce attack surface and avoid downtime). In the event of an incident, you can see who had access to the impacted system and terminate connectivity if needed.
Strategy #6 – Vulnerability and Patch Management
Adapt vulnerability and patch management to the systems and maintenance windows of OT and leverage compensating controls in between.
The legacy systems, business criticality, and limited patch windows of OT environments complicate typical vulnerability and patch management strategies. Instead of patching your way through hundreds of vulnerabilities, you need to understand which vulnerable systems are most important to production. Ensure there is a plan in place to remediate during the next scheduled maintenance window – understanding that many OT vulnerabilities don’t have a patch or firmware update fix available at all. This is where leveraging compensating control mechanisms come into their own to limit the impact of the vulnerability of incident. Such mechanisms include the principle of least privilege, network segmentation and isolation (only allowing required traffic for control system operation), password management, and continuous threat monitoring with hunting (deep packet inspection).. Ultimately, it’s all about the balance of revenue and security.
For more information about how you can secure operational technology environments, click here.
by Mark Mattei | Oct 22, 2019 | Managed Security Services
Threat actors, advanced persistent threats, and simple cybercriminals are always looking for the latest way to get in or take advantage of potential victims. An avenue of approach is defined as a route of an attacking force leading to its objective. The latest and easiest avenue of approach is Office 365. Since this capability is relatively new and IT organizations have not put as much thought and expertise around defending this critical communications capability in the same ways they did with their on-premise Exchange infrastructure, the threat has been able to take advantage of this lack of attention.
Office 365 is complex and has many caveats without a lot of security guidance or documentation available. The initial vector remains to be primarily phishing. Although two-factor authentication has helped reduce phishing, there are many cases in the past several years where attackers phished the 2-factor code as easily as the normal credentials. There are even several open-source two-factor bypass frameworks that are being leveraged daily to compromise users. With all the available ways to continue to steal user credentials, the attackers continue to go after the O365 as a way to manipulate or execute social engineering to steal money.
In one scenario, an attacker gained access to an account on O365 and enabled send on behalf privileges, created administrator accounts, created inbox rules for certain individuals – all in attempts to hide malicious communication activity and spoofed emails. The attacker sent an email from a self-created email string with the response and forwards of legitimate company executives with instructions to wire funds.
In another scenario, attackers sent requests for payment with a PDF invoice that contained new, attacker-controlled, account information. All of these actions leveraged unauthorized access to the email environment. The activity went undetected for many weeks. The new norm for defenders must be to monitor and review activity, configuration changes, inbox rules, and account delegations. Hunting in real-time and watching not just for security events, but also suspicious or abnormal IT activity is a must for reducing the dwell time. Fraud and security teams must develop processes and playbooks for working together to combat this attacker technique.
So how do we impede or block the O365 avenue of approach? The playbooks must include what alerting is available by security teams and what use cases or non-security related activity a security and or fraud team may need to identify malicious activity. Ensuring there is a monitoring and hunting capability while doing configuration verification is simply a must. This includes a thorough review of current licensing and logging so that when an incident happens, administrators are not blind to attacker activity because logging was insufficient. An important report to review is the malware detections report. The ability to detect a security control failure and limit the impact of account compromises is paramount. Just like other systems, using multi-factor authentication for O365 helps protect the data and devices accessible by each individual, but is not the silver bullet. Limiting the number of global administrators and monitoring the activity of those administrator accounts identifies when the most valuable accounts are being used. Another good practice is turning on, consuming and eventing on mailbox auditing for all users allows for the visibility of unauthorized access of exchange online activity. Email is, of course, a normal phishing avenue of approach, so, understanding how your users within your O365 environment are being targeted by malware to then determine further mitigations or more aggressive malware defense actions is key.
Some other security actions are reviewing mailbox access by non-owners which identifies possible malicious activity and turning on Spam notifications. This allows you to see which accounts are blocked for sending spam, which is also an indication of an attacker using that account. Whatever actions you take, make sure there is a continuous periodic review. Never use a set it and forget it approach. Additionally, Microsoft has been rolling out more advanced security options for O365 within its Automated Investigations and Response (AIR) framework to include some Security playbooks for automation of opening investigations. The initial set of playbooks include User-reported Phish Message, URL Click verdict change, Malware ZAP, Phish ZAP, and email investigations.
O365 is complex and moving from on-premise exchange to O365 does not reduce your need for security activities and actions required to defend your environment. Attackers will continue to use this Avenue of Approach until we as security professionals force them to move to a different avenue in order to gain ground. Making this lucrative objective a hardened target should be on everyone’s to-do list.
by Mark Mattei | Aug 23, 2018 | Managed Security Services
“Military intelligence” is no oxymoron. I’m not a career intelligence professional, but I have worked with some of the best intel organizations and operations in the world, including cyber operations and U.S. military intelligence. So, when I need to assess cyber intelligence, I revert to the framework used in a military environment.
The essential basics of any intelligence operation, whatever the sector, cover requirements definition, collection, processing and exploitation, analysis and production and dissemination. So, what particular insights do you examine within this framework used by the best cyber intelligence organizations?
A critical part of any intelligence operation is determining the need. Just saying ‘I need cyber intelligence’ or ‘I am going to create cyber intelligence’ will get you nowhere. A consumer or producer of intelligence needs to understand what is required in order to not only build a collection platform which meets the needs but executes the required collection. If you’re a cyber intelligence organization, the value of your production not only depends on your analysis but is just as dependent, if not more, on your collection.
Another aspect of your needs may be strategic and not just tactical. Strategic intelligence can help when building a network or security architectures or detection capabilities and hunting operations. There are knowledge bases for threat techniques, such as the MITRE Adversarial Tactics, Techniques & Common Knowledge (ATT&CKTM), which can be used to evaluate your defenses or detection capabilities. Some of the best organizations use and build their security operations and detection frameworks from these threat techniques. These organizations use strategic intelligence to protect against threats to things in their vertical, infrastructure or their architecture.
Another part of strategic intelligence is actor and intent. Although intent may be evident in some situations, APTs have a very different intent from a simple ransomware attack. Intent and attribution can be a specific requirement for government and law enforcement to meet their needs, but intent also can be useful in other sectors like critical infrastructure. Understanding the long-term goal or intent of intellectual property theft, denial of service or physical destruction within your sector can go a long way toward understanding your risks, your specific strategic intelligence requirements and the real-time tactical intelligence you require to mitigate those risks.
The size and/or scope of your collection platform capability will determine the size of your output. Single intelligence sources or implementing single-function processes like scraping the web for malicious content or links are valuable but deliver limited intelligence with specific applications. If you only collect, process and analyze malware, it stands to reason that you will only produce malware intelligence. Collection capabilities really come from the ability to acquire unique data. Companies execute collection with various techniques, media and locations. Incident response collects data. Security products collect data. Web and darknet scraping collect data. Intrusion and Network analysis collects data. Hunting collects data. The best intelligence organizations are multi-faceted, so they can fuse together all the intelligence collected from different platforms.
Size and scope of collection are analogous to your own internal network collection and processing. Think about your network Security Information and Event Management System (SIEM). Your SIEM scales in value with more data sources (collection platform) and better correlation (processing) within the platform. If you have one data source, firewalls, for instance, you get collection and correlation from only firewalls. But if you have servers, endpoint detection capabilities, email gateway logs as well as firewalls providing data that you can correlate the information you receive from these multiple sources. When it comes to intelligence collection, companies who have a large platform or multiple platforms provide different intelligence than a provider who scrapes the dark web for specific attributes. Both can be valuable but again this goes back to your need and requirements. The main point to remember: not all intelligence providers are created equal and one big differentiator is the quality of their collection platforms.
The ability to process raw data plays a significant role in an intelligence provider’s ability to produce real-time intelligence. The best intelligence organizations have developed two important capabilities: vast collection and big data analytics. Using, storing and executing complex analytics on large amounts of data is challenging. The future is now when it comes to using artificial intelligence such as machine learning to support operations. The key to success is figuring out which providers are just using “AI” as a buzzword. Data, without good analytics, only yields piles of data with no actionable outcome. The larger and more diverse the data types and structures, the better your data storage and your ability to perform analytics must be. If you understand your provider’s ability to conduct analytics on their collection, you are another step closer to ROI on intelligence.
The goal of intelligence analysis is to figure out what will happen next. Great providers understand they must assess what is happening now and why it’s happening. Intelligence activities include trying to determine the attacker tactics, techniques and procedures. Some attackers use botnets, malware, ransomware. Others use phishing, metasploit or file-less attacks. All these techniques and the tactics of code writing, timing, sequence, targeting, and infrastructure used, need to be collected to find and attribute the most sophisticated threats.
The best nation-state actors develop techniques to look like other nation states. Finding advanced persistent threats (APT) take an enormous amount of data combed through by the best analytics fast enough to find the needle in a field on haystacks. Understanding your provider’s analysis capabilities is very different from knowing their collection methods, analytics and production capabilities. Good analysis comes from years of experience working to get in the mind of the threat actors, to understand their motivation and the goals of those threats. When assessing analysis, look for experience and historic achievements as well as a good methodology for using what they collect to reach conclusions on your requirements.
In some ways, understanding how you will consume threat intelligence or how it will be provided determines your requirements. Understanding how intel is disseminated is key: Are there automated feeds? Do I get an email? Do I read it on a portal? Are indicators of compromise provided? Is it a list of exploits being used against the newest vulnerabilities? How is it structured to be used by my security tools like direct SIEM ingestion?
In its simplest form, the intelligence needs to be actionable by security staff or security tools. In other words, have an actual effect on your defenses. Knowing the Chinese hacked the Office of Personnel Management (OPM), the Russians hacked the DNC, or the latest botnet is spreading across America may be good to know, but how does that help your security staff change your security posture?
What of that is actionable? Does your security team or provider get actionable intelligence and how do they make it useful? Do they have a way to translate data, information and intelligence into a useful defense scheme or execute real-time targeted hunting in your unique environment based on your atmospherics, architectures, vulnerabilities and priorities? How many times have you seen the intel provider send you an email with links to other web articles? Having an intelligence feed because its required by regulation, maybe checking the box, but you must figure out how to use that feed to the max extent possible. How does crawling the web help my situation? Situational awareness about threats is one thing, but actionable intelligence is what reduces risk, finds threats and stops breaches.
Even the best intelligence-producing organizations are producing for a specific need. Know what your needs are, so you can make sure you choose one that gives you actionable intelligence for your particular needs – tactical or strategic. The current landscape for cyber intelligence is vast and confusing. Providers will give you the intelligence they gain based on their own collection, processing, analysis and production capabilities.
Article originally appeared in SC Magazine. Read it here.
by Mark Mattei | Apr 3, 2018 | Managed Security Services
The newest buzz word around cybersecurity and managed services is managed hunt operations; the main nuance which might be lost is simple enough, hunting is not new! From platforms to people, everyone is touting the need to find the threats in your network, but security professionals have been looking for and finding threats in networks for 20 years. This “new” concept or theory of hunting has been executed by the best network defenders with the help of sensors, logs, AV, tools, and various scanners for a very long time.
The real trick is going from hunting to search and destroy. While finding historical evidence that attackers have been stealing your Intellectual property for the last four months and remediating may seem to be a success for most threat hunting capabilities. The truth is, discovering threat actors executing commands and watching the techniques is the goal for any modern hunt team. Crushing your advisory in real time as they move laterally, looking to steal intellectual property (IP), Personally Identifiable Information (PII) or Payment Card Industry (PCI) is the dream scenario for any member of your enterprise hunt team.
How many times has your security analyst said, “I can see at this time, this process ran which is an indication of possible blah, blah, blah.” The goal needs to be, “I see the attacker dumping hashes from memory using Mimicatz… I see the active RDP session and the attackers attempt to move laterally from Host 10.X.X.X. I see PowerShell activity on X host not associated with our internal SCCM.”
Active real-time hunting reduces the “find” time from the most recent estimate of about 99 days down to near real time. This real-time hunting takes talent, training, and humans actively executing structured activities to find threat activity. In military terms, some would say it’s a movement to contact. Movement to contact defined by FM 3-0 Operations is a type of offensive operation designed to develop the situation and establish or regain contact. A cyber movement to contact requires not only some of the best behavior-based detection capabilities and best internal collection capabilities but real-time interactive operations within the networks, systems, and hosts.
Other types of hunts we can take from military tactics, techniques and procedures are:
Area Defense: A defensive task that concentrates on denying enemy forces access to designated terrain for a specific time rather than destroying the enemy outright. This type of hunting operation allows us to conserve or use resources to focus on the “crown jewels.” These tactics may include blocking, canalization into the engagement area of the defenders choosing. Some newer deception technologies allow for a more advanced defense as opposed to the honeypot scenario.
Attack: An offensive task that destroys or defeats enemy forces, seizes and secures terrain, or both. Hunting operations within one own’s network which can be categorized as an attack must focus on the threat tools or capabilities, ensure the threat does not own, hold or control infrastructure which is too valuable to be simply wiped and baselined.
Pursuit: An offensive task designed to catch or cut off a hostile force attempting to escape, with the aim of destroying it. Or in other words, making sure the threat knows they were caught and has no way back into the network. Shut the preverbal “backdoor.”
All that being said, hunting needs planning, real-time humans executing operations. Using a military framework may help organize the plan, but either way, get eyes on the threat actions in real time.
As opposed to attacking someone in their network, hunters can find and render any threat attempt useless through understanding tactics and techniques an attacker would use. Once in contact, the hunters must clearly understand what actions to take. If your analysts see real-time activity, have you developed a real-time response to each of the interactive scenarios? Understanding the requirements of not just finding and blocking bad stuff but knowing what tools and actions to take if your hunter sees the active RDP session, finds PowerShell running, sees certain processes running or sees the recon scanning activity is critical.
Thoroughly thought out plans, hunts, hunter actions, responses and activities upon finding the threat is sometimes referred to as hunting maturity level. What level is your organization? Start by developing a plan for real interactive hunting, build hunting goals, train hunters, understand the needed tools so we create a contested environment.
by Mark Mattei | Mar 19, 2018 | Managed Security Services
A cursory glance at any MSSP listing shows that the focus of most mainstream network and security operations centers (SOCs) is generally health monitoring, configuration, accounting, performance, security (FCAPS), mean time to repair (MTTR), and the security events as they arise.
It’s not a focus that is enjoying enormous success. According to Gartner, breach activity in 2017 was up by 43.8% year-over-year and the scale and severity of attacks as well as reporting requirements are increasing.
Speed of response is at the heart of the issue. Some of the recent largest-scale breaches, such as OPM, Equifax, Target, etc., may have had a slow decision cycle. And this is where the idea of ‘fusion’ provides an interesting answer. Fusion seeks to make better decisions based on the best available information possible and gain the advantage of having a faster decision cycle than your enemy or threat.
Clearly, the decision maker who has the fastest process to gather the best, most up-to-date information possible is going to have the advantage. This is not a new concept. As retired general Stan McChrystal said “The answer is for leaders to have a process in place that helps them gather relevant information, adequately consider dissenting views from a mix of trusted sources, make a decision, communicate the decision, and act on it. Such a system does not eliminate risk entirely, as real decisions always involve uncertainty and risks, but it does help to ensure that the decision made is well-informed, timely, and the best course of action in an evolving and complex environment.”
The military has evolved in some part due to Gen. McChrystal’s vision for fusion. Put simply, fusing who has the information with who needs the information is critical for timely decision making and action.
In cyber, this is even faster and more important than in any other domain. Before the Internet, the telephone, the telegraph, radio, and carrier pigeon, information traveled at the speed of humans. Think Paul Revere or Pheidippides. Now information travels at the speed of light, so decision cycles are faster. The need for fusion is even more important because of technology, not less important because we have technology. Traditional fusion is intelligence with operations. The critical piece to figure out in any “fusioning” is what needs to be fused. In some organizations fusing Cyber Intelligence and threat activity has led to an evolution on cyber defense, but this still falls short for two reasons.
First, using contextual information not only from IT operations but from business operations adds huge value to the speed of understanding cyber events. The old false positive problem is significantly reduced by knowing up front or in real time the cause of an event in context to operations. Think PowerShell – PowerShell may be legit if done by an Admin yet may be bad if being done by an external RDP connection.
Knowing if SCCM is being used at the same time PowerShell launches is a huge win for fusing IT operations information with security event information. With understanding IT and Business context, event fatigue then becomes minimal and the one event which is almost the same but is missing the business contextual information does not get missed because your only analyst is drowning in useless events.
Second, get rid of the notion that intelligence feeds will solve all problems in real time. “If I could only automate those feeds I’d catch the crook in the act!” If you don’t know and understand your threat through intelligence way before they break the window, you won’t see them or catch them until it’s too late. CrowdStrike estimates the average attacker takes 1 hour and 58 minutes to move laterally in your network. This means you need to have a decision cycle faster than two hours to stop that initial compromise from becoming much worse. Cyber intelligence is knowing the threat, building detection for those threats, and then spending your time hunting for those threats not relying on some automated detection with real-time cyber intelligence.
For cyber decision making, attackers fuse the latest vulnerabilities with techniques and capabilities to exploit those vulnerabilities. For the defender, the fusion comes from having the intelligence information, the network contextual information and the activities that are occurring in real time on the infrastructure. Only then can the defender reduce the decision cycle to an actionable timeframe, block the attacker decisively, contain the damage to critical assets – and hopefully – avoid becoming the next big cyber attack headline.