DevSecOps, Containers, & Unified Secrets Management

DevOps secrets and sensitive data are a given in the development world. You can’t write a secure app without taking into account how to validate passwords, prevent unauthorized access, retrieve cryptographic keys, and access API tokens. DevOps has streamlined release cycles, which helps development efficiency, but can sometimes undermine security as lack of standardization and/or expertise around secrets management results in unprotected API access tokens, credentials embedded in source code, cryptographic keys stored insecurely on the filesystem, or other sensitive values stored in configuration files.

Enter secrets management and PAM (privileged access management). When used as part of a holistic and disciplined development and release process, these tools can help ensure that these secrets and sensitive values so necessary to the operation of our software (but so often protected less rigorously than we'd like) are stored securely, used only in authorized ways, and that access to them is controlled, monitored, and auditable.

This post covers a few of the most common secrets-related mistakes present in deployed DevOps applications and explains how to prevent them with security hygiene in the DevSecOps lifecycle, proper management of secrets, and implementation of privileged access.

The Golden Login Problem is Back

In the 90s and early aughties we often struggled with the “golden login.” Very often, applications employed static, application-level accounts to authenticate themselves to backend stores of sensitive information, like relational databases, to middleware components, such as application frameworks, or to remote web services. Consider for example an n-tier application with a web UI tier, an application (business logic) tier, and a database tier. The UI tier might authenticate to the application tier using one “golden” login (i.e. application account), and the application tier might authenticate to the database using another.

There are several problems with this approach. If an attacker finds a vulnerability in the web app, they can then access the protected components with the same privileges the application itself has. Likewise, attempting to audit access to the backend components is challenging. After all, how do you tie activity to a particular user when all actions occur under one account?

Limiting the access available to the web app can help arrest some of this problem—as can building in “guard rails” (for example, using stored procedures to limit what queries an application account can make to a backend database). However, today’s complex apps make this problem even tougher to address.

Rather than a single login to contend with, today’s devs are often using multiple logins between services tied together by scripting, and, often, embedded credentials are stored for access. Likewise, API tokens can often provide inappropriate levels of privilege should they become exposed. Cryptographic keys—either application keys or even certificate private keys—can also threaten the application if they are lost, stolen, or should the application become compromised.

To mitigate risks around accounts and access, I recommend that organizations:

• Inventory what’s connected to what and how/where embedded credentials are stored
• Inventory and document any non-public keys, credentials, tokens, or other secrets.
• Map out applications using data flow diagrams (also useful for threat modeling) and highlight places where critical secrets values are stored
• Don’t forget to remove access when/if the app is no longer in use

Addressing security risks around containers

Another aspect of the DevOps security problem is the rise of containers for ease of deployment and mobility for apps and workloads. Containers are like toys that ship with batteries included or Ikea furniture that comes with all the hex wrenches needed to assemble the furniture in the box. This is fantastic for ease of use and portability, but not so good for security when developers look for ways to ship secrets like credentials and keys in the container. Additionally, the tools used to manage containerized app deployment, like Kubernetes, need access to those secrets to function.

Remember that one of the main advantages of using containers is portability. This is a huge boon to development and release processes, but it can also make it easier to mishandle secrets. For example, we probably all know of at least one instance where keys or access tokens intended only for development use made their way into staging or, worse yet, production environments. Ideally, we have automated workflows in place to prevent that from happening, but these are only useful to the extent that developers know (and remember) to use them.

Three best practices I recommend to reduce risk of exposure of container secrets are:

• Don’t store secrets in the clear in the container
• Look to secrets management and PAM solutions to help safeguard key storage
• Automate secrets generation/issuance (and protection) as part of the deployment process

Lack of interoperability - a common challenge that impacts security

Yes, there are a number of secrets management solutions available, including for all the major cloud DevOps platforms, Google Cloud, AWS, and Azure. Unfortunately, these systems don’t always play well together. The result? Silos of secrets rather than a holistic secrets management program across all the platforms; for example, different secrets management approaches depending on whether the application is hosted in house or in the cloud (or depending on which cloud provider is in use).

You might employ Azure Key Vault to store secrets in some apps and AWS HSM for others. You might use Kubernetes Secrets containers managed in some environments and AWS Secrets Manager in AWS Fargate. The point? Where the secrets are impacts decisions about how and where you store them. Be alert to interoperability issues from the earliest design phases.

These are key considerations I recommend you take into account to avoid or mitigate interoperability issues and enforce strong, consistent secrets management:

• Look for a multi-cloud solution to help deliver a normalized experience across the entire lifecycle
• Use of centralized vaults for secure storage of secrets and credentials
• Rotate credentials via a centralized solution so they don’t get stale, Long active times increases the likelihood of leakage and misuse. Standing credentials should be removed entirely.
• Implement MFA (multi-factor authentication) or JIT (just-in-time) access - so plain old UID/PW credential attacks (e.g., credential stuffing) are blocked
• Add application-level abstractions to “wrap” calls to underlying secrets storage mechanisms.

For a more in-depth exploration of DevSecOps considerations and best practices, check out my on-demand webinar: Putting the SECrets in DevSecOps.

Related Reading

DevOps Secrets Safe (web page)

DevOps Secrets Safe (datasheet)

How to Manage your CI/CD Toolset Secrets with One Solution: 6 Benefits of DevOps Secrets Safe (blog)