Free Privileged Account Discovery Tool: Identify & secure credentials to stop lateral movement. Download Free

BeyondTrust
  • Products
    Privileged Password Management
    Discover, manage, audit, and monitor privileged accounts
    Password Safe DevOps Secrets Safe
    Endpoint Privilege Management
    Manage privileges on Windows, Mac, Linux, and Unix endpoints
    Windows and Mac Unix and Linux Active Directory Bridge
    Secure Remote Access
    Centrally manage and secure remote access for service desks and vendors
    Remote Support Privileged Remote Access
    BeyondInsight Analytics
    See All Solutions
  • Resources

    Universal Privilege Management

    Our innovative Universal Privilege Management approach secures every user, asset, and session across your entire enterprise.

    Watch Video

    Learn

    Case Studies
    Competitor Comparisons
    Datasheets
    Glossary
    Product Demos
    Whitepapers

    Attend

    Events
    Go Beyond
    Training
    Webinars

    Support

    Changelog
    Professional Services
    Technical Documentation
  • Blog
  • Partners
  • Contact
  • Support
  • Services
  • Training
  • Events
  • Company

DevSecOps, Containers, & Unified Secrets Management

January 26, 2021

  • Blog
  • Archive

DevOps secrets and sensitive data are a given in the development world. You can’t write a secure app without taking into account how to validate passwords, prevent unauthorized access, retrieve cryptographic keys, and access API tokens. DevOps has streamlined release cycles, which helps development efficiency, but can sometimes undermine security as lack of standardization and/or expertise around secrets management results in unprotected API access tokens, credentials embedded in source code, cryptographic keys stored insecurely on the filesystem, or other sensitive values stored in configuration files.

Enter secrets management and PAM (privileged access management). When used as part of a holistic and disciplined development and release process, these tools can help ensure that these secrets and sensitive values so necessary to the operation of our software (but so often protected less rigorously than we'd like) are stored securely, used only in authorized ways, and that access to them is controlled, monitored, and auditable.

This post covers a few of the most common secrets-related mistakes present in deployed DevOps applications and explains how to prevent them with security hygiene in the DevSecOps lifecycle, proper management of secrets, and implementation of privileged access.

The Golden Login Problem is Back

In the 90s and early aughties we often struggled with the “golden login.” Very often, applications employed static, application-level accounts to authenticate themselves to backend stores of sensitive information, like relational databases, to middleware components, such as application frameworks, or to remote web services. Consider for example an n-tier application with a web UI tier, an application (business logic) tier, and a database tier. The UI tier might authenticate to the application tier using one “golden” login (i.e. application account), and the application tier might authenticate to the database using another.

There are several problems with this approach. If an attacker finds a vulnerability in the web app, they can then access the protected components with the same privileges the application itself has. Likewise, attempting to audit access to the backend components is challenging. After all, how do you tie activity to a particular user when all actions occur under one account?

Limiting the access available to the web app can help arrest some of this problem—as can building in “guard rails” (for example, using stored procedures to limit what queries an application account can make to a backend database). However, today’s complex apps make this problem even tougher to address.

Rather than a single login to contend with, today’s devs are often using multiple logins between services tied together by scripting, and, often, embedded credentials are stored for access. Likewise, API tokens can often provide inappropriate levels of privilege should they become exposed. Cryptographic keys—either application keys or even certificate private keys—can also threaten the application if they are lost, stolen, or should the application become compromised.

To mitigate risks around accounts and access, I recommend that organizations:

  • Inventory what’s connected to what and how/where embedded credentials are stored
  • Inventory and document any non-public keys, credentials, tokens, or other secrets.
  • Map out applications using data flow diagrams (also useful for threat modeling) and highlight places where critical secrets values are stored
  • Don’t forget to remove access when/if the app is no longer in use

Addressing security risks around containers

Another aspect of the DevOps security problem is the rise of containers for ease of deployment and mobility for apps and workloads. Containers are like toys that ship with batteries included or Ikea furniture that comes with all the hex wrenches needed to assemble the furniture in the box. This is fantastic for ease of use and portability, but not so good for security when developers look for ways to ship secrets like credentials and keys in the container. Additionally, the tools used to manage containerized app deployment, like Kubernetes, need access to those secrets to function.

Remember that one of the main advantages of using containers is portability. This is a huge boon to development and release processes, but it can also make it easier to mishandle secrets. For example, we probably all know of at least one instance where keys or access tokens intended only for development use made their way into staging or, worse yet, production environments. Ideally, we have automated workflows in place to prevent that from happening, but these are only useful to the extent that developers know (and remember) to use them.

Three best practices I recommend to reduce risk of exposure of container secrets are:

  • Don’t store secrets in the clear in the container
  • Look to secrets management and PAM solutions to help safeguard key storage
  • Automate secrets generation/issuance (and protection) as part of the deployment process

Lack of interoperability - a common challenge that impacts security

Yes, there are a number of secrets management solutions available, including for all the major cloud DevOps platforms, Google Cloud, AWS, and Azure. Unfortunately, these systems don’t always play well together. The result? Silos of secrets rather than a holistic secrets management program across all the platforms; for example, different secrets management approaches depending on whether the application is hosted in house or in the cloud (or depending on which cloud provider is in use).

You might employ Azure Key Vault to store secrets in some apps and AWS HSM for others. You might use Kubernetes Secrets containers managed in some environments and AWS Secrets Manager in AWS Fargate. The point? Where the secrets are impacts decisions about how and where you store them. Be alert to interoperability issues from the earliest design phases.

These are key considerations I recommend you take into account to avoid or mitigate interoperability issues and enforce strong, consistent secrets management:

  • Look for a multi-cloud solution to help deliver a normalized experience across the entire lifecycle
  • Use of centralized vaults for secure storage of secrets and credentials
  • Rotate credentials via a centralized solution so they don’t get stale, Long active times increases the likelihood of leakage and misuse. Standing credentials should be removed entirely.
  • Implement MFA (multi-factor authentication) or JIT (just-in-time) access - so plain old UID/PW credential attacks (e.g., credential stuffing) are blocked
  • Add application-level abstractions to “wrap” calls to underlying secrets storage mechanisms.

For a more in-depth exploration of DevSecOps considerations and best practices, check out my on-demand webinar: Putting the SECrets in DevSecOps.

Related Reading

DevOps Secrets Safe (web page)

DevOps Secrets Safe (datasheet)

How to Manage your CI/CD Toolset Secrets with One Solution: 6 Benefits of DevOps Secrets Safe (blog)

Photograph of Diana Kelley

Diana Kelley, CTO, Executive Mentor, Research Analyst, Security Keynote Speaker

Diana Kelley’s security career spans over 30 years. She is Co-Founder and CTO of SecurityCurve and donates much of her time to volunteer work in the cybersecurity community, including serving on the ACM Ethics & Plagiarism Committee, as CTO and Board member at Sightline Security, Board member and Inclusion Working Group champion at WiCyS, Cybersecurity Committee Advisor at CompTIA, and RSAC US Program Committee. Diana produces the #MyCyberWhy series, hosts BrightTALK’s The Security Balancing Act, and is a Principal Consulting Analyst with TechVision Research and a member of The Analyst Syndicate. She was the Cybersecurity Field CTO for Microsoft, Global Executive Security Advisor at IBM Security, GM at Symantec, VP at Burton Group (now Gartner), and a Manager at KPMG. She is a sought after keynote speaker, the co-author of the book Cryptographic Libraries for Developers, has been a lecturer at Boston College's Masters program in cybersecurity, the EWF 2020 Executive of the Year, and one of Cybersecurity Ventures 100 Fascinating Females Fighting Cybercrime.

Stay Up To Date

Get the latest news, ideas, and tactics from BeyondTrust. You may unsubscribe at any time.

I agree to receive product related communications from BeyondTrust as detailed in the Privacy Policy, and I may manage my preferences or withdraw my consent at any time.

Up next

From January 21, 2021:
Privilege Management for Unix & Linux Continues Rapid Growth by Securing Cloud Infrastructure
From January 28, 2021:
Understanding Sudo Vulnerability CVE-2021-3156 and How Privilege Management for Unix & Linux Can Protect Your Enterprise

You May Also Be Interested In:

Whitepapers

The Guide to Multicloud Privilege Management

Whitepapers

Evolving Privileged Identity Management (PIM) In The 'Next Normal'

Webcasts

Putting the SECrets in DevSecOps

BeyondTrust Logo
  • Facebook
  • Twitter
  • LinkedIn

Keep up with BeyondTrust

I agree to receive product related communications from BeyondTrust as detailed in the Privacy Policy, and I may manage my preferences or withdraw my consent at any time.

Customer Support
Contact Sales

Products

  • Endpoint Privilege Management
  • Password Management
  • Privileged Remote Access
  • DevOps Secrets Safe
  • Remote Support

Resources

  • Blog
  • Case Studies
  • Competitor Comparisons
  • Datasheets
  • Glossary
  • Videos
  • Webcasts
  • Whitepapers

About

  • Company
  • Careers
  • Contact
  • Events
  • Leadership Team
  • Partner Program
  • Press

Languages

  • English
  • German
  • French
  • Spanish
  • Korean
  • Portuguese
  • Japanese
  • Privacy
  • Security
  • Manage Cookies
  • WEEE Compliance

Copyright © 1999 — 2020 BeyondTrust Corporation. All rights reserved. Other trademarks identified on this page are owned by their respective owners. BeyondTrust Corporation is not a chartered bank or trust company, or depository institution. It is not authorized to accept deposits or trust accounts and is not licensed or regulated by any state or federal banking authority.