DevOps Security Best Practices

What is DevOps Security?

DevOps security refers to the discipline and practice of safeguarding the entire DevOps environment through strategies, policies, processes, and technology. Security should be built into every part of the DevOps lifecycle, including inception, design, build, test, release, support, maintenance, and beyond. Today, this type of "baked-in" DevOps security is often called DevSecOps, which aims to improve security through improved collaboration and shared responsibility that overlays the entire DevOps workflow.

This article explores the fundamental considerations for applying security to DevOps environments and provides an overview of DevOps security definitions, challenges, and best practices.

DevOps Security Challenges and Considerations

The DevOps ethos has ushered in a transformation of how organizations develop, operate, and maintain applications and IT infrastructure, both onsite and in cloud environments. By blending two traditionally separate IT worlds, IT development and IT operations, a DevOps model aggregates many functions — specifications and requirements, coding, testing, operational readiness, implementation, and more.

DevOps is generally complemented by agile software development processes, which promotes cross-team alignment and collaboration, as well as bespoke development. A relentless pursuit of velocity, automation, and monitoring characterizes all steps of DevOps software development, from integration, testing, releasing, to deployment and infrastructure management. These practices facilitate condensed development cycles and product release timeframes, while keeping product features and capabilities responsive to customer feedback and evolving business objectives.

DevOps Security Best Practices

What are the implications of DevOps for security? Let’s delve into how DevOps practices and commonly used tools create particular security issues/considerations, then discuss best practices and technologies for addressing these issues.

DevOps’ focus on speed often leaves security teams flat-footed and reactive:

DevOps teams are frequently mal-aligned with Infosec teams. DevOps pushes and modifies batches of code over very short time frames (hours or days), which may far outpace the speed at which security teams can keep up with code review. If security (configuration checks, code analysis, vulnerability scanning, etc.) is not adequately automated, the DevOps output will either be dramatically slowed, or lack proper security hygiene.

In practice, the fallout from this misalignment includes insecure code, inadvertent vulnerabilities, misconfigurations, hardcoded passwords, and other weakness in application security that can be exploited by attackers, or contribute to operational dysfunction, including downtime.

Cultural resistance to security:

There’s a widespread perception that introducing security will slow or derail the development process. However, the time and effort cost of catching a security flaw early in the design and development process is much lower than having to retroactively fix problematic code and weaknesses later in the development cycle.

DevOps and cloud environments:

The typical DevOps environment relies on cloud deployments, thereby sharing many cloud security considerations. DevOps teams often leverage new, open-source or immature tools to manage hundreds of security groups and thousands of server instances. In these fast-moving environments, which operate at tremendous scale, a simple misconfiguration error or security malpractice, such as sharing of secrets (APIs, privileged credentials, SSH keys, etc.) can be broadly propagated, causing widespread operational dysfunction, or numerous exploitable security and/or compliance issues.

Containers and other tools carry their own risks:

The ascension of containers and the tools to manage them (Docker, Kubernetes, CoreOs etc.) across DevOps environments confers exceptional productivity and innovation potential for users, while at the same time spawning new security headaches. First, consider the security implications of the containers themselves. As an ultra-lightweight and portable packaging platform for applications, containers can be spun up and down almost instantly—and run across almost any kind of computer and cloud. However, without proper controls in place, containers can pose security risks due to lack of visibility into the containers themselves, which is complicated because they share an OS with other containers.

Frequently, containers are not adequately scanned for vulnerabilities. A study by ThreatStack underscored this, with an alarming 94% of respondents indicating that containers pose negative security implications for their organizations.

Unmanaged secrets and poor privileged access controls open dangerous backdoors:

Most facets of DevOps are highly interconnected, rapidly changing, and utilize secrets. DevOps secrets may include privileged account credentials, SSH Keys, APIs tokens, etc., and may be used by humans or non-humans (e.g., applications, containers, micro-services and cloud instances). Inadequate secrets management is a common shortcoming of DevOps environments, providing a tantalizing avenue for attackers to tamper with security and other controls, disrupt operations, steal information, and basically own an organization’s IT infrastructure. A typical DevOps environment may leverage several dozen tools (Chef, Puppet, Ansible, Salt, etc.) that all require secrets management.

Additionally, to help expedite workflows, DevOps teams may allow almost unrestricted access to privileged accounts (root, admin, etc.), by multiple individuals, who may share credentials—a practice that virtually eliminates the possibility of a clean audit trail. Various orchestration, configuration management, and other DevOps tools may also be granted vast privileges. With privileged access rights in hand, a hacker or piece of malware can gain full control of the systems and data, so it’s essential for organizations to rein in excessive privilege rights and access.

Uber Delivers a Cautionary Lesson for the DevOps Culture

Most facets of DevOps are highly interconnected, rapidly changing, and utilize secrets. DevOps secrets may include privileged account credentials, SSH Keys, APIs tokens, etc., and may be used by humans or non-humans (e.g., applications, containers, micro-services and cloud instances). Inadequate secrets management is a common shortcoming of DevOps environments, providing a tantalizing avenue for attackers to tamper with security and other controls, disrupt operations, steal information, and basically own an organization’s IT infrastructure. A typical DevOps environment may leverage several dozen tools (Chef, Puppet, Ansible, Salt, etc.) that all require secrets management.

Additionally, to help expedite workflows, DevOps teams may allow almost unrestricted access to privileged accounts (root, admin, etc.), by multiple individuals, who may share credentials—a practice that virtually eliminates the possibility of a clean audit trail. Various orchestration, configuration management, and other DevOps tools may also be granted vast privileges. With privileged access rights in hand, a hacker or piece of malware can gain full control of the systems and data, so it’s essential for organizations to rein in excessive privilege rights and access.

9 DevOps Security Best Practices

While it’s clear that security should be ingrained throughout the entire DevOps lifecycle, how do you accomplish this without hampering speed, agility, and other essential DevOps tenets? DevOps teams not only need to partner with security teams to overlay proper controls, but also must take ownership of baking security into their own processes and culture. When this is culturally imbued throughout the organization, it’s referred to as “DevSecOps.”

To tighten DevOps security, while balancing the need for agility, consider implementing the following initiatives and technologies:

1. Embrace a DevSecOps model

Effective DevOps security demands cross-functional collaboration and buy-in to ensure security considerations are integrated into the entire product development lifecycle (product design, development, delivery, operations, support, etc.). DevSecOps will entail embedding governance and cybersecurity functions such as identity and access management (IAM), privilege management, firewalling / unified threat management, code review, configuration management, and vulnerability management throughout the DevOps workflow. When done right, you have aligned security with DevOps and enable efficient product releases, while avoiding costly recalls or fixes after code/products are released. For this to succeed, everyone needs to take ownership of adhering to security best practices within their roles.

2. Enforce policy & governance

Communication and governance are vital to holistic security for DevOps environments—or any environment. Create transparent cybersecurity policies and procedures that are easy for developers and other team members to understand and agree to. This will help teams to develop code that meets security requirements.

3. Automate your DevOps security processes and tools

Without automated security tools for code analysis, configuration management, patching and vulnerability management, and privileged credential / secrets management, you stand no chance of scaling security to DevOps processes. Automation also minimizes risk arising from human error, and the associated downtime or vulnerabilities. Prioritize the deployment of automated tools to identify potential threats, problematic or vulnerable code, and issues with process and infrastructure. The closer you can match the speed of security to the DevOps process, the less likely you are to face culture resistance to embedding security practices.

4. Perform comprehensive discovery

Ensure that all approved and unapproved devices, tools, and accounts are continuously discovered, validated, and brought under security management in accordance to your policy.

5. Conduct vulnerability management

Vulnerabilities should be appropriately scanned, assessed, and remediated across development and integration environments before they are deployed to production. Rely on penetration testing and other attack mechanisms to identify weaknesses in pre-production code and to indicate areas for improvement. When products are launched into an operational environment, DevOps security can run tests and tools against the production software and infrastructure to identify and patch exploits and issues.

6. Adopt configuration management

Scan to identify and remediate misconfigurations and potential errors. Harden all configurations using industry best practices. Provide continuous configuration and hardening baseline scanning across servers and code/builds for physical, virtual, and cloud assets.

7. Secure access with DevOps secrets management

Eliminate embedded credentials tucked away in code, scripts, files, service accounts, in various tools, cloud platforms, etc. This involves separating the password from the code, so that when it’s not in use, it’s securely stored in a centralized password safe. Privileged password management solutions can force applications and scripts to call (or request) use of the password from a centralized password safe. By implementing API calls, you gain control over scripts, files, code, and embedded keys. Then, you can automate rotation of the passwords as often as policy dictates. [Learn the 8 best practices for privileged password management -infographic]

8. Control, monitor, and audit access with privileged access management

Enforce least privilege access rights to reduce opportunities for internal or external attackers to escalate privileged user rights or exploit bad code. In practical terms, this means eliminating administrator privileges on end-user machines, securely storing privileged account credentials and requiring a simple workflow process for check-out. Monitor all privileged sessions to ensure privileged activities are legitimate and adhere to compliance mandates. Enforcing a least privilege model should also entail restricting access for developers and testers to specific development, management and production systems, while still allowing them the appropriate permissions to build machines and images, and deploy, configure and remediate production issues on machines and images.

Additionally, consider restricting developer access to specific container templates and runtime access to containers on systems, while still allowing the appropriate permissions necessary to code, build, test, verify, and manage application components. The leading-edge privileged access management solutions can automate the control, monitoring, and auditing of privileged access as well as the full lifecycle of secrets/privileged credential management. By implementing a PAM solution, you can rapidly provision and audit privileged access for all your users, machines, and applications based on a least privilege model.

9. Segment networks

Segmenting the network reduces an attacker’s “line of sight” access. Group assets, including application and resource servers, into logical units that do not trust one another. In the case of access that needs to traverse the trust zones, deploy a secured jump server with multi-factor authentication, adaptive access authorization, and use session monitoring to provide oversight. Further segment access-based context, including user, role, application, and data being requested.

DevOps security can enable a productive DevOps ecosystem, while helping to identify and remediate code vulnerabilities and operational weaknesses long before they become an issue. Introducing DevOps security early in the product lifecycle ensures that security underpins every part of application and systems development. This, in turn, enhances availability, reduces the possibility of data breaches, and ensures the development and provisioning of powerful technology to meet business needs.