by Nathan Hamiel | Jun 1, 2023 | Artificial Intelligence
Much has been made about the privacy and confidentiality issues with ChatGPT. Just take a look at the press for a list of companies prohibiting ChatGPT usage by their employees, but I’d argue there’s an equally if not far more concerning issue impacting privacy and confidentiality beyond ChatGPT, and that’s individual experimenters distributed across organizations.
There’s a fair bet that people inside your organization are experimenting with LLMs, and they may not be the people you expect. These aren’t developers, they are the everyday staff spread across your organization. Focusing prohibitions and policies on ChatGPT can be a distraction from the more significant issue, far more data can leave your organization over API calls than through the web interface of ChatGPT, and it’s happening right now without your knowledge.
Accessibility and Ease of Use
Back in the day, security teams struggled with cloud computing because anyone with a credit card could have a cloud. Security professionals would claim they weren’t using cloud resources, but when you talked with developers, they said they used cloud resources all of the time. This disconnect has come to the AI space because anyone with a credit card has access to a hosted large language model.
To take this a step further, there’s no need to have a GPU or even know anything about machine learning to leverage this technology. Anyone capable of writing sentences and copying/pasting some Python code can make API calls to a hosted LLM. API calls are simple to understand, here’s the page on OpenAI’s website for the ChatCompletion API. Not to mention the countless tutorials on YouTube (both good and bad.)
Below I’ve put an entire working example that if placed into a Jupyter notebook, would print the answer to the question, “What was the first production vehicle made by Ford Motor Company?”
This accessibility means that it may not be your developers experimenting with LLMs to optimize tasks and solve problems, it could be anyone at your company (and probably is). Regardless of what industry you are in, this affects you.
It’s also important to realize that it’s not just OpenAI you need to think about. Many different providers, from startups to big tech companies, have stood up solutions in this area and are hosting access to LLMs, all with different business models and usage and privacy policies.
People experimenting may not have clear goals or even know what they expect in return. Quite often, they may feed data into an LLM to merely see what happens or, due to hype, expect some kind of magic to happen. Regardless, the impact on organizations is the same.
Negative Impacts
The positive impacts of successful experiments should be obvious, and I won’t highlight them here. There are several real-world negative impacts that result from these activities. I’ll highlight a couple below.
Privacy and Confidentiality
Privacy and confidentiality are the obvious impacts that everyone talks about. Your sensitive business data could leave your organization and end up in the hands of a 3rd party. This data could then be used and viewed by humans to train and improve future models. Even if you trust the 3rd party, you are still at the mercy of their data storage and data protection requirements.
Keep in mind data stored for training and improvement of models isn’t stored with the same requirements as it would be in your regular cloud computing account. It needs to be optimized and available for retraining as well as human evaluation.
Violation of Regulatory Requirements
The various staff at your organization may not be familiar with all of the regulatory requirements your organization falls under. Given this, it could be possible that staff uses the simple applications they build to violate regulatory requirements. Think about taking something like patient data and using an LLM to summarize the content or a simple application they build to make a sensitive decision about coverage.
Inexperience With Development
People outside of your development team aren’t accustomed to following development standards and associated security guidelines. As such, they may make critical mistakes in their code. For example, they may grab data from the Internet and through it into a Python eval() or, even worse, a fully autonomous experiment that spins away trying to optimize a task that also grabs untrusted data and throws it into an eval().
This is less of a concern than an application being exposed externally because, as an experiment, it shouldn’t be deployed or exposed more widely, but depending on what the code does and the permissions of the user running it, it can still have negative consequences that affect the company.
Inexperience With Evaluation
Staff members outside your development teams may not be familiar with how to evaluate and benchmark the software that they build. The system could be giving the right answer but for the wrong reasons or give the wrong answer entirely, but since it’s coming from an AI, it’s more trusted. This could lead to a business impact where incorrect decisions are made because of a faulty design. There are many cases were an LLM is the wrong solution to a problem, but it may be chosen anyway because of ease of use or hype around the technology.
Sunken Cost-Driven Adoption
It may be the case that since people spent time building something, they use it anyway, regardless of how effective it is or whether it actually optimizes a process. Anyone who’s ever worked at a company where this has happened has felt this pain. These are experiments and not formal development projects. As such, proper requirements haven’t been gathered, and critical pieces are often missed that are necessary for generalizing to a larger team or problem.
I’ve had this happen multiple times in my career, where someone hacks together a partially functional, almost non-usable experiment and tries to drive the adoption of it to a larger group, selling it as the new way to do things. Don’t do this to your coworkers.
Addressing The Problem
The first thing to realize is that this isn’t just a problem for the developers inside your organization. It’s a larger problem that potentially affects everyone in your organization. Think about approaching the issue with a larger scope more like how you do awareness for and issue like phishing. Give people and overview of the issue and a solution.
Avoid Strict Prohibitions
It’s tempting, in this case, to just give a blanket “No” and move on with life. This would be a mistake for many reasons, but mostly for the fact that this approach doesn’t work. This is one of the lessons we should have learned by now. Just like the cloud, people will do what they are going to do and just won’t tell the security team. Instead of saying no, you should be harnessing people’s excitement and pointing them in the right direction. There’s a much better chance for conversation when the people engaging you feel like they won’t get a no in response.
Guidelines and Policy
Create some clear guidelines and policy choices around usage and ensure they are communicated to the entire company. An entire list of items that should go in here is specific to each company and industry, as well as outside the scope of this blog post.
Any prohibitions for certain types of data or use in specific processes should be clearly outlined along with an associated “why.” Many times, if people understand the why they are much more likely to follow the policy.
If you have an approved provider for LLM services, make sure it’s specified, and any process you have internally for communicating and approving experiments should also be documented.
By the way, now would be a good time to work on that data classification project that you’ve been putting off for the past decade. This will help in ensuring specific data is protected from exposure.
Workshops and Awareness Activities
Set up a series of workshops for discussions and encourage people to reach out. Use these activities to communicate the guidelines and policies and give people a path to share and explore their ideas in a productive way. Highlight what could go wrong and how to avoid it. Communicate specific processes you put in place around how people can reach out and what they need to do to experiment.
Experiment Inventory and Registry
It’s good to keep a running list of these experiments at your organization, so you can follow up if necessary. It would be best to have a process where people can register an experiment on their own and mark it as decommissioned when completed. Without adding too much overhead, you’ll want to capture, at a minimum, who owns the experiment, what business process it supports, what data it uses, and the experiment’s status.
This is good for collaboration, but from a security perspective, there are some high-level things you want to be sure of.
- Sensitive data isn’t leaving the company
- Regulatory and Policy Requirements are being followed
- Fully autonomous experiments are kept in check
- The impact of a security issue is kept to a minimum
I realize this can be an ask, and a lot can go wrong here, but it’s a start. It’s better than not knowing. Perform some experiments yourself around this topic and see what works best for your organization.
Feedback
This will be a new process at your organization. Ensure feedback is flowing in both directions. Collect feedback on the process to ensure that issues are being addressed. Ensure that feedback is also flowing back to the participants of the process, where they may be opening up unnecessary risk.
Conclusion
Due to the hype and accessibility, there is a whole lot of sensitive data leaving organizations without their knowledge. With the right approach, you can put a series of steps in place to ensure you maintain an innovative spirit while ensuring your data is protected. With the right balance you can leverage a productivity boost while also ensuring your data is protected.
If you are concerned about attacks such as prompt injection, have a look at our previous blog about reducing the impact of prompt injection through application design.
by Nathan Hamiel | Mar 29, 2023 | Artificial Intelligence
Headlines about ChatGPT and the updated GPT-4 are everywhere. Even with new updates, these models still hallucinate, and unfortunately, so do people writing articles about this technology. There is quite a bit of circular reporting on this topic especially related to software development and security. I’ve seen outrageous claims about these tools that are more unbelievable than an episode of CSI Cyber. Rather than talk about why we are good at fooling ourselves when making predictions, let’s move past the hype and focus on reality.
Reality Check
Let’s have a bit of a reality check about LLMs and their capabilities in general. As a security professional, I’m tech agnostic. I don’t love or hate any technology. I feel this is my job to help people innovate at scale using the technology they prefer. Overall, I’m optimistic about the use of machine learning and deep learning as approaches to solve security challenges and help security professionals accomplish their goals. We are seeing some of this today. Given the scale of the problems and the amount of data, it’s just not possible for a human to dig through all of this data. We need better tools to automate activities and increase our effectiveness. So, I’m on board.
Unfortunately, when it comes to transformer-based LLMs, many have been spouting opinions of a world-changing future, with these tools being touted as impactful on humanity as the printing press. The reality is much more mundane. We continue to see article after article filled with opinions from people who’ve never used these tools, all parroting talking points from something they’ve previously read. When you dig beneath the surface and look at the examples given, they often include research and lab experiments that don’t reflect real-world scenarios. This parroting creates a strange false consensus. Max Fischer, in his book The Chaos Machine, said, “When people think something has become a matter of consensus, psychologists have found, they tend not only to go along, but to internalize that sentiment as their own.” Nobody gets in trouble or loses repetitional points for saying the same thing as everyone else, and so it continues.
When you dig beneath the surface and look at the examples given, they often include research and lab experiments that don’t reflect real-world scenarios.
Time for a step back. Pundits making these claims are purely guessing because these systems aren’t capable of doing what they claim today. They are filling in the gaps about very massive and real problems which haven’t been solved today with transformer-based LLMs. Arguably, there are some of these problems which may not be solvable at all, and this is supported by AI experts. Hence the guessing. If you believe everything is just a matter of scale, then you are probably on the overly optimistic side because why wouldn’t we be able to just make things bigger? For example, would a trillion-parameter model solve all of the current problems? I believe these problems run far deeper than scale. That doesn’t mean these systems can’t be useful, but it depends on the problem and the situation.
Memorization is impressive when a human does it, but not so much when a computer does it. We know that machine learning and deep learning systems have a tendency to memorize their training data. The impact of memorization is a problem with generalizing to new problems or situations. There’s some evidence that GPT-4 also memorizes its training data. This memorization means that if you rephrase a question on one of those impressive tests, GPT-4 will get the answer wrong. Tell this to all of the folks that think LLMs are actually performing reasoning. Do you really want ChatGPT to be your lawyer now?
Besides, the reality of being a security professional isn’t sitting around answering CISSP questions all day. Our job is to apply knowledge to specific situations where it makes sense. But one thing is for sure, humans continue to be bad at constructing tests, and we continue to be fooled by Stochastic Parrots.
Before we start digging into the claims of supercharged attackers or criminal usage, keep in mind It’s not that ChatGPT is incapable of some of these tasks. It’s that it’s not particularly good at some tasks and far from the best tool for the job for others. Now, let’s look at some of the security-focused claims and issues.
It’s not particularly good at some tasks and far from the best tool for the job for others.
Supercharged Attacker Claims
Countless articles have been published about ChatGPT supercharging attackers, making them more efficient and allowing them to operate above their skills and capabilities. The reality just doesn’t line up. Anyone who’s used these systems knows they make mistakes, and confidently. You often need to rephrase the prompt and make adjustments, piecing together a more complex piece of software. You need to have the skills and experience to understand when the system isn’t giving you the correct output. So, in summary, as of today, this isn’t allowing attackers to operate above their skill level. The NCSC has a similar assessment. Besides, if these tools supercharged attackers, well, they’d kind of be supercharged now, and we just aren’t seeing any evidence of that.
Writing Malware
Yes, there have been people who’ve written malware and bypassed EDR with ChatGPT. This sounds impressive and potentially scary on the surface, but digging into the details, it becomes less so. You typically see the code output is in something like Python, a language not typically used for malware. There was also a bunch of trial and error to get the system to properly generate the output. So, these experiments were done for research purposes, specifically to use ChatGPT for the novelty factor. Certainly interesting and great research, but not the most real-world examples of an attacker usage in the wild. Marcus Hutchins has a good breakdown of this.
Finding Vulnerabilities
ChatGPT can read and understand code, and it’s possible to feed a code snippet into the tool and have it find a vulnerability, or maybe not. This functionality is incredibly hit or miss. It seems possible to find some common coding flaws in common languages some of the time and miss things other times. This certainly isn’t as effective or accurate as a dedicated security tool, so it begs the question of why people talk so much about this fact in the first place.
Enhanced Phishing Claims
Another example I see is that attackers can use tools like ChatGPT to conduct better phishing attacks. In reality, this doesn’t seem to pan out. The whole point of creating a convincing phishing email is the “convincing” part. You want to take an official email communication with all of its formatting and make it look as indistinguishable from an official communication as possible. This isn’t something that ChatGPT does.
There are also guardrails in place that prevent you from creating these emails and creating them at scale. Sure, with some work, the guardrails can be bypassed, but all of this means that ChatGPT is a less-than-ideal tool for phishing tasks. With a copy of an official email and a translation tool, you can get much further. Beyond intuition, there’s also some data to support that humans still write better phishing emails.
Real Risks
Just because a system like ChatGPT doesn’t supercharge less experienced attackers doesn’t mean that there aren’t real risks from these systems, just maybe not the ones you are thinking about.
The Real Risk to Security: Application Integration
Application integration of LLMs is the largest risk facing security. Today, tools like ChatGPT don’t really “do” much. They can’t reach out into the world and make things happen like scheduling an appointment, ordering groceries, or changing the channel on your television, but that’s about to change. OpenAI announced ChatGPT plugins. This opens up many more possibilities and I can’t wait to see all of the new attacks that result from this increased attack surface.
With the integration of these tools into your applications and search engines, you open up a whole new world of attack surface, turning a previously robust application into one with a wide-open door.
With LLMs, the surface of all possible issues isn’t known at deployment time. It’s not obvious all of the ways the system can fail, failure, in this case, can now affect your application. These unknowns can manifest into security issues in strange ways that you may not be able to fix because these calls are made over an API to a 3rd party provider. I’ve described this as having a single interface with an unlimited number of undocumented protocols, all waiting to be exploited. Wait until someone tricks a previously robust application into ordering a pizza on a celebrities credit card.
A single interface with an unlimited number of undocumented protocols, all waiting to be exploited.
We are already getting a glimpse at this future. Where people are planting prompts in the wild, waiting for these tools to encounter them. A kind of reverse prompt injection. You never know where these prompts may end up. Wait until attackers inject prompts into log files hoping that security tools parsing the data will run into it.
Be extremely cautious when considering the integration of these tools into your applications. Ask yourself what features you are getting and if they are worth the risk for the application. Err on the side of caution and try to use API calls with more restricted functionality, especially if you are just using text summarization or some other specific feature and don’t need the full chat experience.
Privacy and the Unavoidable Chatification of Everything
One thing that we can all agree on is that current LLM technology is terrible for privacy. The data you provide to these systems is verbosely logged, sent to a 3rd party, and potentially viewed by humans as part of the additional training process. There’s nothing stopping your input from being used for further training, being evaluated to improve the system, or ending up on Facebook. Your data is handled with fewer security and privacy protections than it normally would under these conditions. To be used in this context means it needs to be unencrypted, have less access control, and potentially have more copies because people, teams, and systems need access.
Your data is handled with fewer security and privacy protections than it normally would under these conditions.
Today, these features are an opt-in inside of products, but in the inevitable chatification of everything, they may be on by default, making it much harder to control from a security perspective. Think about just how much sensitive data about an organization could be gleaned from having access to data from everyone’s Microsoft Word documents. You are giving a 3rd party visibility into your organization’s workings and trusting them completely with it.
Misinformation
Misinformation risks fall into a strange area. I look at these from two different perspectives, one is the industrialization of misinformation by malicious actors, and the other one is small-scale one-off misinformation, either accidentally by the model itself or on purpose by an individual.
First off, I don’t see the OpenAI product ChatGPT industrializing misinformation at scale for bad actors. The guardrails in place as well as the As-a-Service hosting method, make it unlikely and more of an annoyance for malicious use in this context. But there are other LLMs without such guardrails in place that could be self-hosted and potentially used for this purpose. For example, Meta’s LLaMA model was leaked online and could be repurposed for this activity.
On the second point, we are entering an era where there are no guarantees that these models won’t be self-referential. Meaning there’s no meaningful way to tell if a new model trained on text content isn’t learning based on its previous output or another LLM’s output. This is concerning, especially for use cases where people are trying to use these systems as truth machines. Even small pieces of misinformation can become impactful when re-learned back into a model.
We are entering an era where there are no guarantees that these models won’t be self-referential.
Steve Bannon had a famous quote that is applicable here. He said the goal was to “Flood the zone with…” Well, you know. There is a real risk that LLMs without guardrails could very well be the perfect tool for this task. Time will tell.
Realities
Okay, so ChatGPT doesn’t pose a weapons-grade update to attackers, and it won’t supercharge inexperienced attackers to operate above their skill level, but that doesn’t mean that it’s useless in the context of information security. There are many different tasks and efficiency gains for security and development teams.
I believe we are at the beginning of what people will use these tools for. For the most part, ChatGPT and other LLMs are useful for things like text summarization tasks as well as text generation tasks. If you think about it, it makes sense, they are trained with examples of written language. This is why they can imitate writing style so well.
We’ve also seen that these tools have the ability to deal with programming languages in both writing and understanding code. People have used them to help understand obfuscated code and create small programs quickly. Advancements in this area are happening fast.
We often have to deal with large amounts of data, so tasks like parsing texts, summarizing content, writing documentation, and many other tasks are ones that you may find will assist you and your team be much more productive. It may surprise you to learn that we didn’t need ChatGPT for these. These were tasks that LLMs were pretty good at long before ChatGPT.
Just like on the attacker side, your staff needs to have the experience to know when the tool isn’t outputting the right data. Keep in mind that small-scale experiments and successes rarely have applicability in the real world. So, do your own experiments applied to your own use cases and evaluate the results accordingly.
If you have development teams using these tools, it’s important to ensure you have security built into your development processes to catch the potential issues from these tools. There’s no guarantee that the output is safe. For more information, download the Kudelski Security Research white paper Addressing Risks from AI Coding Assistants here: https://resources.kudelskisecurity.com/en/kudelski-security-ai-coding-assistants
Conclusion
Overall, I think we should prepare to be surprised, for better or worse. Even though these tools don’t seem to supercharge attackers, we should be mindful of the opportunities and pitfalls they present for defenders. They can open our applications and organizations up to additional attacks and privacy issues, but they can also lend themselves to being a productivity boost and help us streamline activities. With the right balance, we can have the best of both worlds. We just need to ignore the hype and focus on the realities.
by Nathan Hamiel | Oct 14, 2021 | Artificial Intelligence
If you are reading this post, then there’s a good chance you understand the need for security surrounding AI systems. As more and more development teams look at ways to increase the intelligence of their applications, the surrounding teams, including security teams, are struggling to keep up with this new paradigm and this trend is only accelerating.
Security leaders need to start bridging the gap and engaging with teams that are developing applications within the spectrum of AI. AI is transforming the world around us, not only for enterprises but also for society, which means failures have a much more significant impact.
Why you should care about AI Security
Everything about AI development and implementation is risky. We as a society tend to overestimate the capabilities of AI, and these misconceptions fuel modern AI product development, turning something understandable into something complex, unexplainable, and, worst of all, unpredictable.
AI is not magic. To take this a step further, what we have isn’t all that intelligent either. We have a brute force approach to development that we run many times until we get a result that we deem acceptable. It’s the kind of thing that if a human developer iterated that many times, you’d find unacceptable.
Failures with these systems happen every day. Imagine being denied acceptance to a university without explanation or being arrested for a crime you didn’t commit. These are real impacts felt by real people. The science fiction perspective of AI failures is a distraction to the practical applications deployed and used daily. Still, most AI projects never make it into production.
Governments are taking notice as well. A slew of new regulations, both proposed and signed into law, are coming out for regions all over the world. These regulations mandate controls and explainability, as well as an assessment of risk for these published models. Some of those, like the EU’s proposed legislation on harmonized AI, even have prohibitions built-in as well.
From a security perspective, ML/DL applications can’t solely be treated as traditional applications. AI systems are a mixture of traditional platforms and new paradigms requiring security teams to adapt when evaluating these systems. This adaptation requires more access to systems than they have had traditionally.
There are requirements for safety, security, privacy, and many others in the development space. Still, there seems to be confusion about who is responsible for what when it comes to AI development. In many cases, the security team, typically the best equipped to provide security feedback, aren’t part of the conversation.
What makes AI Risky?
When it comes to risk in AI, it’s a combination of factors coming together to create risk. It’s important to realize that developers aren’t purposefully trying to create systems that cause harm. So, if this is the case, let’s look at a few issues creating risk.
Poorly defined problems and goals
Problems can crop up at the very beginning of a project. Often, there is a push for differentiators in application development, and this push for more “intelligence” may come from upper management. This puts additional pressure on application developers to use the technology whether it is a good fit for the problem or not.
Not all problems are good candidates for technologies like machine learning and deep learning, but still, you hear comments such as “We have some data, let’s do something cool” or “let’s sprinkle some ML on it.” These statements are not legitimate goals for an ML project and may cause more harm than good.
Requests for this technology can lead to unintended, negative consequences and build up unnecessary technical debt. ML implementations are rarely set it and forget it type of systems. The data may change and shift over time as well as the problem the system was meant to solve.
Velocity
The velocity of development isn’t a problem constrained to the development of machine learning systems. The speed at which systems are developed has been a problem for safety and security for quite some time. When you add the potential unpredictability of machine learning and deep learning applications, it adds an additional layer of risk.
Developers may be reluctant to follow additional risk and security guidelines due to the perception that they get in the way of innovation or throw up unnecessary roadblocks. This reluctance is regardless of any legal requirements to perform certain activities. This perception needs to be managed by risk management and security teams.
The mantra, “move fast and break things,” is a luxury you have when the cost of failure is low. Unfortunately, the cost of failure for many systems is increasing. Even if the risk seems low at first, these systems can become a dependency for a larger system.
Increased attack surface
Machine learning code is just a small part at the center of a larger ecosystem of traditional and new platforms. A model does nothing on its own and requires supporting architecture and processes, including sensors, IoT devices, APIs, data streams, backends, and even other machine learning models chained together, to name a few. These components connected and working together create the larger system, and attacks can happen at any exposure point along the way.
Lack of Understanding Around AI Risks
In general, there is a lack of understanding surrounding AI risks. This lack of understanding extends from the stakeholders down to the developers. A recent survey from FICO showed there was confusion about who was responsible for risk and security steps. In addition, these leaders also ranked a decision tree as a higher risk than an artificial neural network, even though a decision tree is inherently explainable.
If you’ve attended an AI developer conference or webcast, if risk is mentioned, they are talking about risks involved in developing AI applications, not risks to or from the AI applications. Governance is also discussed in terms of maximizing ROI and not in ensuring that critical steps are adhered to during the development of the system.
Supply Chain Issues
In the world of AI development, model reuse is encouraged. This reuse means that developers don’t need to start from scratch in their development processes. Pre-trained models are available on many different platforms, including Github, model zoos, and even from cloud providers. The issue to keep in mind is that you inherit all of the issues with the previous model. So, if that model has issues with bias, by reusing the model, you are amplifying the issue.
It’s also possible to have backdoors in models where a particular pattern could be shown to a model to have it take a different action when that pattern is shown. Previously, I wrote a blog post on creating a simple neural network backdoor.
A common theme running across issues relating to risk is lack of visibility. In the previous survey, 65% of Data and AI leaders responded that they couldn’t explain how a model makes decisions or predictions. That’s not good, considering regulations like GDPR have a right to explanation.
Other Characteristics Inherent to AI Applications
There are other characteristics specific to AI systems that can lead to increased exposure and risk.
Fragility. Not every attack on AI has to be cutting edge. Machine learning systems can be fragile and break under the best of conditions.
Lack of explicit programming logic. In a traditional application, you specify from start to finish the application’s behavior. With machine learning systems, you give it some data, and it learns based on what it sees in the data. It then uses what it learned to apply to future decisions.
Model uncertainty. Many of the world’s machine learning and deep learning applications only know what they’ve been shown. For example, ImageNet candidates only know the world by the thousand labeled categories it’s been shown. If you show one of the candidates something outside of those thousand things, it goes with the closest thing it knows.
AI Threat Landscape
Threats to AI systems are a combination of traditional and new attacks. Attacks against the infrastructure surrounding these systems are well known. But, with intelligent systems, there are also new vectors of attack that need to be accounted for during testing. These attacks would have inputs specifically constructed and focused on the model being attacked.
Types of Machine Learning Attacks & Their Impacts
In this section, I’ll spell out a couple of the most impactful attacks against machine learning systems. These attacks are not an exhaustive list but they cover a couple of the most interesting attacks relating to AI risk.
Model Evasion Attacks are a type of machine learning attack where an attacker feeds the model an adversarial input, purposefully perturbed for the given model. Based on this perturbed input, the model makes a different decision. You can think of this type of attack as having the system purposefully misclassify one image as another or classify a message as not being spam when it actually is.
Impact: The model logic is bypassed by an attacker.
Model Poisoning Attacks feed “poisoned” data to the system to change the decision boundary to apply to future predictions. In these attacks, attackers are intentionally provided bad data to the model to re-train it.
Impact: Attackers degrade a model by poisoning it, reducing accuracy and confidence.
Membership Inference Attacks are an attack on privacy rather than an attempt to exploit the model. Supervised learning systems tend to overfit to their training data. This overfitting can be harmful when you’re training on sensitive data because the model will be more confident about things it’s seen before than things it hasn’t seen before. This attack means an attacker could determine if someone was part of a particular training set which could reveal sensitive information about the individual.
Impact: Sensitive data is recovered from your published model.
Model Theft / Functional Extraction happens when an attacker creates an offline copy of your model and uses that to create attacks against the model. Then they can use what they’ve learned back against your production system.
Impact: Your model may be stolen or used to create more accurate attacks against it.
Model inaccuracies. Inaccuracies inherent to the model could cause harm to people or groups. Your model may have biases that cause people to be declined for a loan, denied entry to school, or even arrested for a crime they didn’t commit.
Impact: Your inaccurate model causes people harm.
An inaccurate model becomes the truth. To take the previous point to an extreme, imagine a model with inaccuracies is now used as a source of truth. This condition can be much harder to identify and may exist for a long time before being identified. Usage of this model would not allow people any recourse, because the source of the decision was the model.
Impact: Your inaccurate model causes systemic and possibly societal harm.
Where to go from here
Now that we’ve covered some of the issues, it’s time to start building up your defenses. I’ll cover that in another post. In the meantime, check out some of these AI security resources below.
Additional reading
by Nathan Hamiel | Jul 26, 2021 | Artificial Intelligence
Until recently, the regulation of AI was left up to the organizations developing the technology, allowing these organizations to apply their own judgment and ethical guidelines to the products and services they create. Although this is still widely true, it may be about to change. New regulations are on the horizon, and some already signed into law, mandating requirements that could mean costly fines for non-compliance. In this post, we look at some themes across this legislation and give you some tips to begin your preparation.
Landscape
When you think of regulations surrounding AI, your mind probably wanders to the use of the technology in things like weapons systems or public safety. The fact of the matter is, harms from these systems extend far beyond these narrow categories.
Many developers are just using the tools available to them. Developers create experiments and evaluate the final result on a simple set of metrics and shipping to production if it meets a threshold. They aren’t thinking specifically about issues related to risk, safety, and security.
AI systems can be unpredictable, which is ironic since often you are using them to predict something. Why unpredictability surfaces is beyond the scope of this post, but it has to do with both technical and human factors.
We’ve had laws indirectly relating to regulations of AI for quite some time and probably haven’t realized it. Not all these regulations specifically spell out AI. They may be part of other consumer safety legislation. For example, the Fair Credit Reporting Act may come into play when making automated decisions about creditworthiness and dictate the data used. In the context of machine learning, this applies to the data used to train a system. So, current regulation that prohibits specific pieces of information such as protected classifications (race, gender, religion, etc.) from being used or prohibits specific practices, then that also applies to AI, whether it spells it out or not.
Governments and elected officials are waking up to the dangers posed by the harm resulting from the unpredictability of AI systems. One early indicator of this is in GDPR Recital 71. In summary, this is the right to explanation. If there is an automated process for determining whether someone gets a loan or not, a person denied has a right to be told why they were rejected. Hint, telling someone one of the neurons in your neural network found them unworthy isn’t an acceptable explanation.
Recently, the EU released a proposal specifically targeting AI systems and system development. This proposed legislation outlines requirements for high-risk systems as well as prohibitions on specific technologies, such as those meant to influence mood as well as ones that create grades like a social score.
Although the US tried to pass similar legislation called the Algorithmic Accountability Act, it did not pass. The US Government did, however, release a draft memo on the regulation of AI. This document covers the evaluation of risks as well as issues specific to safety and security.
The US legislation not passing doesn’t mean the individual US States aren’t taking action on this issue. One example of this is Virginia’s Consumer Data Protection Act.
This is far from an exhaustive list and one thing is for sure, more pieces of regulation are coming, and organizations need to prepare. In the short term, these regulations will continue to lack cohesion and focus and will be hard to navigate.
Themes
Even though the specifics of these regulations vary across the geographic regions, some high-level themes tie them together.
Responsibility
The overarching goal of regulation is to inform and hold accountable. These new regulations push the responsibility for these systems onto the creators. Acting irresponsibly or unethically will cost you.
Scope
Each regulation has a scope and doesn’t apply universally to all applications across the board. Some have a broad scope, and some are very narrow. They can also lack common definitions making it hard to determine if your application is in scope or not. Regulations may specifically call out a use case or may imply it through a definition of data protection. Most lawmakers aren’t technologists so expect differences across the various legislation you encounter and determine common themes.
Risk Assessments and Mitigations
A major theme of all the proposed legislation is understanding risk and providing mitigations. This assessment should evaluate both risks to and from the system. None of the regulations dictate a specific approach or methodology, but you’ll have to show that you evaluated risks and what steps you took to mitigate those risks. So, in simple terms, how would your system cause harm if it is compromised or fails, and what did you do about it?
Validation
Rules aren’t much good without validation. You’ll have to provide proof of the steps you took to protect your systems. In some cases, this may mean algorithmic verification by providing ongoing testing. The output of the testing could be proof you show to the auditor.
Explainability
Simply put, why did your system make the decision it did? What factors lead to the decision? Explainability also plays a role outside of regulation. Coming up with the right decision isn’t good enough. When your systems lack explainability, they may make the right decision but for the wrong reason. Based on issues with data, the system may “learn” a feature that has high importance but, in reality, isn’t relevant.
How Can Companies Prepare?
The time to start preparing is now, and you can use the themes of current and proposed regulation as a starting point. It will take some time, depending on your organization and the processes and culture currently in place.
AI Strategy and Governance
A key foundation in compliance is the implementation of a strategy and governance program tailored to AI. An AI strategy and governance program allows organizations to implement specific processes and controls and audit compliance.
This program will affect multiple stakeholders, so it shouldn’t be any single person’s sole responsibility. Assemble a collection of stakeholders into an AI governance working group and, at a minimum, include members from the business, development, and security team.
Inventory
You can’t prepare or protect what you don’t know. Taking and maintaining a proper inventory of AI projects and their criticality levels to the business is a vital first step. Business criticality levels can feed into other phases, such as risk assessments. A byproduct of the inventory is that you communicate with the teams developing these systems and gain feedback for your AI strategy.
Implement Threat and Risk Assessments
A central theme across all of the new regulations is the specific calling out of risk assessments. Implementing an approach where you evaluate both threats and risks will give you a better picture of the protection mechanisms necessary to protect the system and mitigate potential risks and abuses.
At Kudelski Security we have a simple approach for evaluating threats and risks to AI systems consisting of five phases. This approach provides tactical feedback to stakeholders for quick mitigation.
KS AI Threat and Risk Assessment
If you are looking for a quick gut check on the risk of the system, ask a couple of questions.
- What does the system do?
- Does it support a critical business process?
- Was it trained on sensitive data?
- How exposed is the system going to be?
- What would happen if the system failed?
- Could the system be misused?
- Does it fall under any regulatory compliance?
If you would like to dive deeper, check out a webcast I did for Black Hat called Preventing Random Forest Fires: AI Risk and Security First Steps.
Develop AI Specific Testing
Testing and validation of systems implementing machine learning and deep learning technology require different approaches and tooling. An AI system combines traditional and non-traditional platforms, meaning that standard security tooling won’t be effective across the board. However, depending on your current tooling and environment, standard tooling could be a solid foundation.
Security testing for these systems should be more cooperative than some of the more traditional adversarial approaches. Testing should include working with developers to get more visibility and creating a security pipeline to test attacks and potential mitigations.
It may be better to think of security testing in the context of AI more as a series of experiments than as a one-off testing activity. Experiments from both testing and proposed protection mechanisms can be done alongside the regular development pipeline and integrated later. AI attack and defense is a rapidly evolving space, so having a separate area to experiment apart from the production pipeline ensures that experimentation can happen freely without affecting production.
Documentation
Models aren’t useful on their own. They require supporting infrastructure and may be distributed across many devices. This distribution is why documentation is critical. Understanding data usage and how all of the components work together allows for a better determination of the threats and risks to your systems.
Focus on explainability
Explainability, although not always called out in the legislation, is implied. After all, you can’t tell someone why they were denied a loan if you don’t have an explanation from the system. Explainability is important in a governance context as well. Ensuring you are making the right decision for the right reasons is vital for the normal operation of a system.
Some models are more explainable than others. When performing benchmarking for model performance, it’s a good idea to benchmark the model against a simpler, more explainable model. The performance may not be that different and what you get in return is something more predictable and explainable.
Conclusion
Move fast and break things is a luxury you can afford when the cost of failure is low. More and more machine learning is making its way into high-risk systems. By implementing a strategy and governance program and AI-specific controls, you can reduce your risk and attack surface and comply with regulations. Win-win.
by Kudelski Security Team | Jun 15, 2018 | Artificial Intelligence
Artificial intelligence (AI) is being discussed quite a bit, in fact, maybe the term is used too much. After all, it means different things in different situations, and vendors are using the term particularly loosely to tie into a hot market and hopefully sell more product. When it comes to information and cyber security – a realm so vital to a company’s reputation – there’s a need to see through the hype and ask the questions that really matter.
For starters, to me, the main question is not whether AI will find its way in our daily life, but what will it mean to us as cybersecurity professional? What are the security risks when adopted by various business lines for different purposes? What are the benefits to our profession? And what are the risks of not considering the opportunity of AI to help us do our job better – or of failing to monitor closely what the business will use it for?
Like so many technology disruptions, AI will change part of the business landscape and it will also shape our own cybersecurity backyard. The logic is implacable, when there is a business opportunity, there are investments to be made and AI presents potential across many aspects of our modern life.
In its simplest essence, AI perceives and processes information from its environment and can use incomplete data to maximize the success rate of completing certain tasks. As you can guess, I just described tons of activities that we as human do every day in the medical, financial, industrial, psychological, analytical and military sectors.
At the moment, I don’t think we should overly focus on its potential to replace cognitive beings. Instead, we should appreciate that AI can leverage broader data input, discover patterns we can’t easily distinguish and is capable of sorting and matching data much faster than we can. Moreover, it never gets tired and can work 24×7. Ultimately, this will result in potentially better and faster decision making, where emotions or artistic sense might not be the primary quality by which we measure output.
That said, all of AI is not “rosy,” and when matched with robotics, it can be the stuff of nightmares. AI comes with challenges, and while it can autonomously trigger actions based on an algorithmic logic, the logic must be flawless. If not, it will create a lot of “mistakes” and very fast. The necessary algorithms rely on data, hence input quality must be tightly controlled, otherwise, garbage-in, garbage-out, right? So, it’s imperative organizations decide what should and shouldn’t be automated, and it’s an approach that needs to be validated by humans first. AI strategy done well can effectively address a skill shortage, but done wrong and with a “set and forget” mentality, it’ll backfire.
Still, keep in mind that AI can also reflect some of the flaws of its creator. Because humans come with their fair share of challenges, let’s focus on two examples.
Trust, either the lack of or too much of it, can make AI react in a way we did not foresee. Sometimes, emotionless decision-making might be best, sometimes not. The more AI we create, the more we will need to deploy a transposable trust model for this community to interact with each other. After all, in the AI world, there is often little-to-no space for human interference if you want to capture its full benefit.
Transparency is another issue. As a society, we are not ready to entrust to machines many of the things we currently make decisions on – and security is a particularly sensitive area. Without transparency and accountability of the AI, how can we start tackling the notion of responsibility when something goes wrong? And, we must consider at what point will use of AI be mandatory under certain conditions? Will tomorrow’s doctor be personally liable for not using an AI and misdiagnosing potentially cancerous cells construction? As happens with humans, what if the physician simply failed to update a codebase?
It’s no secret that there’s is a lack of qualified security personnel today. That said, I feel it is our responsibility to explore ways to use AI as soon as possible in order to remove any item from our task list that can be automated. As a rule of thumb, I think of the Pareto Principal – AI should do the 80 percent of the job so we can focus on the 20 percent where human interaction and decision-making is a must.
Pareto likely never saw this coming, yet, the formula applies to our profession. AI could allow us to free up time and deliver more value with the same salary cost structure.
And believe me, we will need time because part of that 20 percent will be required to analyze the new business risks of using AI in the real world, one fraught with real and increasing security challenges.
