Secrets Management for Security and Speed
Are you trying to decide whether your business engineering teams need to sacrifice speed for security, or security for speed? Chances are you’ve implicitly already made that decision to keep your promises to the business.
In a world where high entropy strings can be discovered in a matter of seconds on public repositories, a secret that leaks is a secret compromised. Yet, time alone is your enemy. As time goes by, chances your secrets fall into the wrong hands increase, and damage could be irreversible for your projects, your teams or your company. Data exfiltration, unauthorized access to your infrastructure or applications, auditors getting back at you, losing your customers, reputation damage forcing you to close down your business are all dangers that await you at the corner. Maybe nothing happens, but are you willing to take that risk?
Time has come for proper secrets management to enable and empower our engineering teams to keep delivering business value at scale, supported by an engineering-oriented governance for security. Are you up to keep your secrets secret in a world evolving at DevOps velocity?
What is a secret and what is the problem?
A secret is anything that you don’t want to share. It is something private that, if exposed, would reveal information that is not supposed to be put in the spotlight and put you and/or your data at risk. As such, it needs to be considered as a threat in its own right.
A secret could be one of the following:
- PII (Personally Identifiable Information)
- Private certificates or private keys
- AWS or Openstack private keys/tokens
- Any sensitive information that, if leaked, increases your risk level
The common problem with these secrets is their usual long lifetime, maybe unlimited for some, most likely due to lack of governance around them. They remain unchecked, become forgotten, sometimes passed around and ultimately, they get leaked out either by mistake (e.g. debug logs) or intentionally (e.g. internal leak). Either way, they get out of control and the pain that comes when trying to rotate or revoke the exposed secrets is huge.
Secret sprawl
The cozy “security perimeter” we once knew in the 20th century is long gone. We live in a world of borderless infrastructures, where the perimeter is no longer defined. Hybrid infrastructures define the new rules. Your applications now run partly on your on-premise KVM or VMware infrastructure, your private Openstack cloud and your AWS tenant all at once to satisfy your global customer base while also meeting possible regulatory compliance needs. It is a complex world and it is not about to get any simpler. Risks increase as complexity increases.
We can find secrets in a lot of places, for instance:
- Hardcoded in plain text in your application (base64 or symmetric encryption is just as bad)
- In your git repos (e.g. find those high entropy strings with truffleHog)
- Environment variables (very handy with containers, and also for people watching over your processes)
- Configuration management systems
Secrets distribution
Secrets distribution concerns both persons and applications. You need to be able to securely distribute your secrets right from the start. This is also part of the secure introduction goals.
Managing countless secrets for your containers being torn down and built up repeatedly is something that needs to be baked in, and while containers orchestration platform such as Kubernetes also have their secrets handling mechanisms, it can be enhanced with stronger encryption mechanisms and a unified automated governance framework. Your CI/CD pipelines also require secrets as they run in a fully automated, yet fully secure fashion — these orchestration platforms usually have an awful amount of privileges and may need to receive even more security love than any other piece of your infrastructure.
Applied governance
People who work with me know that I am not fond of paperwork procedures of any kind. This also applies to security. I am a fervent advocate of bringing everything back to code, and security is no different.
We need to aim for applied security for our security governance through policies as code that cannot be circumvented, the type of governance that put you back in control.
Ephemeral secrets: Lease, revoke, expire
The one goal we need to achieve is to reduce the potential attack vectors that could be introduced by unmanaged secrets, in order to ultimately reduce our attack surface.
Ultimately, the best access is one that does not remain any longer than necessary. Do you have an AWS Lamba that needs to take a database backup? How about leveraging access keys generated on-the-fly for your Lamba as well as a dynamic user/password on your database to get the job done? At the end of it, have them revoked through your automation.
You could apply the very same principle to grant administrative access to your engineers. Have them request access when needed, grant them short-lived credentials – e.g. 1 day — that could be renewed. Do you need to suddenly remove access? Revoke. Furthermore, you no longer need to be afraid to forget to revoke access either, as the lease will eventually expire quickly anyways.
All your auth are belong to us
In these times of decentralization, keeping secrets may be the odd one as it is deemed better to keep everything centralized to maintain a tight grip on everything secrets related.
You need to be in full control of who/what has access/seen what secrets, in what capacity and when and be sure that you have full audit trails (not just for your yearly audit, but to also to detect unauthorized attempts for example). Storage needs to be secured, just as much as the transit of the secret information.
Moreover, authentication and authorization can be tightly controlled in one secure place, through fine-grained policies you can enforce and manage as code, such as with an in-band git workflow, enforcing your mandatory 4-eye review.
Break-glass
Many of us have faced the departure of an employee, possibly an employee knowing a lot about everything that could have dumped the password databases on his USB stick before walking out of the door. Maybe she would still have access to some obscure, unmanaged cloud tenant remotely connected to your on-premise infrastructure, opening an unforeseen vector of attack.
Or maybe you just got breached.
If you read carefully, then you should now feel like there is some relief you could gain with an automated secret management solution. The immediate effect of implementing such a solution at scale is to reduce your exposure and lower your risks to much more acceptable levels.
Conclusion
This was a short high-level introduction that hopefully makes you realize there is a way in keeping engineering teams velocity high while maintaining proper security governance to provide a truly secure environment for your business. There is no doubt it is quite a bit of work, yet the heavy lifting done by products like Vault (Hashicorp), KeyWhiz (Square Engineering) or Knox (Pinterest) just makes it all the easier. Couple that with good configuration management practices, and you’ve got yourself the foundation for a winning solution.
The list of supported operations on a variety of infrastructure components just keep on growing, so that you now can secure with relative ease your AWS, various databases and containers, provision a very easy to use PKI or even leverage Encryption-as-a-Service capabilities to put in the hands of your developers. Don’t go reinvent the wheel, as home-grown solutions will be hard to maintain, and hard to actually design/implement the proper way. I would rather advise to leverage solution made by dedicated and focused teams, that will let you focus on your core business.
It is very important in my eyes that security is no longer seen as an impediment by engineering teams, just as it is important that it is shift left as much as possible, engrained into the infrastructure or products themselves. Security is everyone’s responsibility.
Sources
Mitchell, Jeffrey. “Think like a Vault Developer: Secure Introduction & Containers”, https://fr.slideshare.net/DynamicInfraDays/containerdays-nyc-2016-the-secure-introduction-problem-getting-secrets-into-containers-jeff-mitchell, 2016-11-09.
Vargo, Seth. “Introduction to Vault”, https://speakerdeck.com/sethvargo/introduction-to-vault, 2015-06-20.
Stevens, Chris. “Using Hashicorp Vault for Secrets Management”, https://speakerdeck.com/stevenscg/using-hashicorp-vault-for-secrets-management, 2017-06-25.
Langlois, Tyler. “Pods, Tokens, and a Little Glue: Integrating Kubernetes and Vault in Elastic Infrastructure”, https://www.elastic.co/blog/kubernetes-vault-integration-devops-team, 2017-09-19.
Multiple contributors, “Kubernetes Vault Integration” README.md, https://github.com/Boostport/kubernetes-vault, 2017-10-11.
Tierno, Peter. “Hashicorp Vault Plugin”, https://wiki.jenkins.io/display/JENKINS/HashiCorp+Vault+Plugin, 2017-06-22.
Vault documentation, https://www.vaultproject.io/docs