Kubernetes Integration

The Kubernetes integration for DevOps Secrets Safe (DSS) enables injection of secrets from DSS into Kubernetes pods.

You can configure pods to retrieve secrets from DSS before starting their primary application. The application consuming the secrets does not need to have any knowledge of DSS to use the secret at runtime.

Prerequisites

This integration uses the Kubernetes service account construct as the basis for identity in DSS. Permissions for access to secrets in DSS can be granted to specific service accounts that are represented by user principal identities within DSS.

Overview

Applications can opt in to DSS secret retrieval by adding the DSS secrets-agent to their Kubernetes manifests as an init container or a sidecar. The secrets-agent retrieves secrets and makes them available to the target consumer without requiring that consumer to be aware of DSS.

To support this feature, the DSS instance must have a Kubernetes-type identity provider configured, and the Kubernetes ServiceAccount assigned to the DSS application pods must be granted access to the TokenReview API for the cluster.

At runtime, secret retrieval from DSS by Kubernetes pods follows the order of operations pictured in the figure below:

1. The InitContainer requests a DSS authentication token using the ServiceAccount JWT for the pod it is running inside of.
2. DSS validates the ServiceAccount JWT using the TokenReview API on the cluster's API server. In order to authenticate to the cluster's API server, DSS assumes the identity of its own pod's ServiceAccount when communicating with the Kubernetes API. The DSS ServiceAccount requires additional role-based access control (RBAC) permissions on the cluster for it to access the cluster's TokenReview API.
3. The secret-retrieval container requests the secret contents from DSS and writes them to a volume that is shared with the application container. In this case, and init-container pattern is being used; secrets-agent exits, and the application container begins execution with its target secret contents from DSS available on its file system at the shared volume path.

This guide provides a walkthrough of the steps required to configure and demonstrate this integration. The walkthrough regularly assumes the following default installation parameters for the DSS instance and cluster being used:

• DSS is installed in the secrets-safe namespace of the Kubernetes cluster.
• The TLS certificate presented by the DSS instance is trusted (not self-signed).
• The Kubernetes identity provider for DSS is configured with the default name kubernetes.

Deviations from any of these assumptions necessitate extra configuration described in the Detailed Manifest Example.

If DSS is not installed into the secrets-safe namepsace, then replace the namespace identifier string secrets-safe in configurations and account names with the alternative namespace used for DSS.

DSS Configuration - Kubernetes Identity Provider

The Kubernetes identity provider enables authentication to DSS using Kubernetes ServiceAccount tokens. Kubernetes ServiceAccounts become user principals in DSS that use the ServiceAccount JWT as their login credential.

The Kubernetes identity provider requires only minimal configuration within the DSS instance for this in-cluster use case, because DSS services are running inside pods and can obtain the cluster's parameters from their pod's environment.

{
      "Name": "Kubernetes",
      "Type": "Kubernetes",
      "Enabled": true
}

Cluster Configuration - Add RBAC Permissions for the DSS ServiceAccount

In order for DSS to validate ServiceAccount tokens, DSS itself assumes the identity of its pod's ServiceAccount. This DSS ServiceAccount requires additional RBAC permissions on the cluster for it to access the cluster's TokenReview API and to list the cluster's ServiceAccounts.

This example assumes that DSS is installed in the namespace secrets-safe and that the DSS pods are using the default ServiceAccount within that namespace. If DSS is installed in a different namespace, replace secret-safe in the examples with the namespace used. If a service account different from default is assigned to DSS, use that service account's name instead.

The RBAC permissions required for DSS secret-retrieval integration are defined in the following manifest:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dss-kubernetes-identity
rules:
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: dss-kubernetes-identity-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dss-kubernetes-identity
subjects:
  - kind: ServiceAccount
    name: default
    namespace: secrets-safe

kubectl apply -f secret-retrieval-privileges.yaml

The Kubernetes API server must be running with --service-account-lookup to ensure that deleted ServiceAccount tokens are properly revoked. If this setting is not enabled, JWTs for deleted ServiceAccounts remain as valid credentials. This setting is enabled by default as of Kubernetes 1.7.

Configuring an Application for Secret Retrieval

In order for an application to opt in to DSS secret retrieval:

• DSS authorization for the application pod's identity must be configured.
• The application's manifest must have the DSS secrets-agent container and Shared Volume added.

This section first presents a guide to setting up the required authorization in DSS to allow a pod's ServiceAccount to retrieve secrets, then shows the pod manifest modifications necessary to add the secret-retrieval behavior to the pod at runtime.

DSS Permissions - Authorize Access to the Target Secret

The ServiceAccount for the application must be authorized to access its target secrets in DSS. This is accomplished by the following steps:

1. Create the ServiceAccount principal in DSS.

   You can do this by using the ServiceAccount name, or creation at first login to DSS using the ServiceAccount's JWT.

   • Use the ServiceAccount Name

     DSS principals for ServiceAccounts can be created using the ServiceAccount name.

     The ServiceAccount name is formatted according to Kubernetes RBAC subject rules, using a period (.) as the separator rather than a colon (:).

     Format the ServiceAccount for DSS as:

     system.serviceaccount.<namespace>.<account-name>

     To print a list of all ServiceAccounts for a cluster in this format, at the bash console for the Kubernetes cluster administrator, run the command:

     OLD_IFS=$IFS; IFS=$'\n'; for i in `kubectl get serviceaccount --all-namespaces`; do echo -n "system.serviceaccount."; echo -n $i | awk '{printf $1}'; echo -n "."; echo -n $i | awk '{print $2}'; done; IFS=$OLD_IFS

     The list of service accounts is returned in the format:

     system.serviceaccount.NAMESPACE.NAME

     Knowing the name of the target service account, the DSS principal can be created using the ssrun user create CLI command.

     For example, to create a DSS principal for the default ServiceAccount in the secrets-safe namespace, use the command:

     ssrun user create -i kubernetes -n system.serviceaccount.secrets-safe.default

   • Use ServiceAccount JWT

     Logging in to DSS using the ServiceAccount JWT requires obtaining the ServiceAccount JWT from the cluster.

     The JWT for the ServiceAccount can be obtained from the cluster using kubectl as follows:

     kubectl get secret <name-of-token-secret> -n <application-namespace> -o jsonpath="{.data.token}"| base64 --decode

     In the above command, <name-of-token-secret> is the name of the Kubernetes secret for the target ServiceAccount credential, and <application-namespace> is the cluster namespace where the target application exists. If the secret-retrieval application is installed in the default namespace, the -n parameter can be omitted.

     Use the obtained JWT as the password credential at the DSS connect/token endpoint, targeting the external identity provider named Kubernetes, if following this example. When the login to DSS succeeds, the ServiceAccount principal identity is created in DSSand is eligible for access to secrets.

2. Grant secret access permissions to the ServiceAccount principal.

3. Log into DSS as a user with sufficient permissions to manage authorization for principals in the Kubernetes identity provider.

4. Create sample secrets for the demonstration app to retrieve. The secrets secret/hello:world1 and secret/hello:world2 are created with arbitrary content for the purpose of this example. Create those secrets before proceeding to the next step.

{
  "principalUri": "/principal/Kubernetes/user/system.serviceaccount.secrets-safe.default",
  "resourceUri": "/secret/hello",
  "operations": [
    "read"
  ],
  "access": "allow"
}

The trailing part of the principal URI for the service account is system.serviceaccount.secrets-safe.default. Principal URIs for Kubernetes ServiceAccounts always take this form in DSS and adhere to the template system.serviceaccount.<namespace>.<account-name>.

For more information, please see Referring to subjects.

Application Manifest Additions

The final step in demonstrating DSS secret injection into pods is to modify the manifest of the target application to include the resources that retrieve secrets and write them to the pod file system.

apiVersion: v1
kind: Pod
metadata:
  name: secret-injection-sample
spec:
  initContainers:
    # DSS Secret Retrieval Client
  - name: secret-retrieval
    image: beyondtrust/secrets-agent:unstable
    env:
    - name: SECRETSSAFE_TARGET_SECRETS
      value: hello:world1,hello:world2
    - name: SECRETSSAFE_OUTPUT_PATH
      value: /run/secretssafe
    volumeMounts:
    - name: secrets-output
      mountPath: /run/secretssafe
  containers:
  - name: target-application
    image: busybox:1
    command: [ "sh", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
    - name: secrets-output
      mountPath: /run/secretssafe
  volumes:
  - name: secrets-output
    emptyDir:
      medium: Memory

After saving the above pod manifest to a YAML file named secret-retrieval-example.yaml, apply it to the cluster using kubectl apply -f secret-retrieval-example.yaml. This creates the pod on the cluster.

Observe the pulling of the InitContainer image using kubectl describe for the pod. Verify the target secret contents are injected to the directory at /run/secretssafe using kubectl exec -it <pod-name> sh to start an interactive shell session inside the running pod. From there, navigate to the /run/secretssafe directory and inspect the contents of the files hello_world1 and hello_world2.

The important configuration values from the manifest above are the following environment variables set on the secret-retrieval InitContainer:

• SECRETSSAFE_TARGET_SECRETS: Comma-separated list of secrets to retrieve from DSS. Secrets are saved such that scope names become directories, and colon (:) delimiters are changed to underscores (_) in the output file name.
• SECRETSSAFE_OUTPUT_PATH: Path to the base directory where secret contents are placed on the pod file system. In the above example, this is defined as the path to an initially empty volume that is shared between the InitContainer and target application by volumeMount configurations.

Detailed Manifest Example

There are more options available for configuring the secret-retrieval InitContainer that are required in scenarios in which any of the following are true:

• DSS is not installed in the secrets-safe namespace.
• The identity provider is not named kubernetes.
• The DSSHTTPS certificate is self-signed.

Available options are:

• SECRETSSAFE_HOST: Host portion of the base URL for the DSS instance being contacted. For the in-cluster use case, this defaults to standardgateway.secrets-safe.svc.cluster.local and need only be changed if DSSis running in a namespace other than secrets-safe. Replace the secrets-safe segment in the URL with the DSS namespace:

  standardgateway.<DSS-namespace>.svc.cluster.local

• SECRETSSAFE_PORT: Port portion of the base URL for the DSS instance being contacted. For the in-cluster use case, this defaults to 8443 and need only be changed if the Gateway service of DSS is configured to use a different port. Set this to the port number where the DSS API is exposed.
• SECRETSSAFE_ID_PROVIDER: The name of the Kubernetes-type identity provider configured in the DSS instance being contacted. Defaults to kubernetes. Change this only if the Kubernetes identity provider in the DSS instance being contacted has a name other than kubernetes.
• SECRETSSAFE_VERIFY_CA: The file path to the PEM-encoded CA certificate that can be used to verify the self-signed DSS HTTPS certificate, or the string false to skip certificate verification when contacting DSS.

Disabling CA verification by setting this value to false is for testing or debugging purposes only and is not secure.

The InitContainer environment variable list from the manifest example above can be expanded to include these options:

initContainers:
    # DSS Secret Retrieval Client
  - name: secrets-agent
    image: beyondtrust/secrets-agent:unstable
    env:
    - name: SECRETSSAFE_TARGET_SECRETS
      value: hello:world1,hello:world2
    - name: SECRETSSAFE_OUTPUT_PATH
      value: /run/secretssafe
    - name: SECRETSSAFE_HOST
      value: standardgateway.<DSS-namespace>.svc.cluster.local
    - name: SECRETSSAFE_PORT
      value: 8443
    - name: SECRETSSAFE_ID_PROVIDER
      value: kubernetes
    - name: SECRETSSAFE_VERIFY_CA
      value: <path-to-CA-cert-file.pem>