What are Kubernetes Secrets?

The concept of Kubernetes Secrets refers to any type of confidential credential that requires privileged access in order to interact with it. These objects often act as keys or methods of authentication with protected computing resources in secure applications, tools, or computing environments. In this article, we’re going to discuss how Kubernetes handles Secrets and what makes a Kubernetes Secret unique.

Why are Kubernetes Secrets Important?

In a distributed computing environment, it is important that containerized applications remain ephemeral and not share their resources with other pods. This is especially true in relation to Public Key Infrastructure (PKI) authentication keys and other confidential resources that pods need to access external resources. For this reason, applications need a way to authenticate without keys and other secrets being held in the application itself.

Kubernetes offers a solution to this that follows the path of least privilege. Essentially, Kubernetes Secrets are placeholders for sensitive information like credentials, tokens, and certificates, a mechanism for abstracting them. Kubernetes Secrets act as separate objects which can be queried by the application Pod to provide credentials to the application for access to external resources. Kubernetes Secrets can only be accessed by Pods if they are explicitly part of a mounted volume or at the time when the Kubelet is pulling the image to be used for the Pod.

How Does Kubernetes Leverage Secrets?

The Kubernetes API provides various built-in Secret types for a variety of use cases found in the wild. When you create a Secret, you can declare its type by leveraging the `type` field of the Secret resource, or an equivalent `kubectl` command line flag. The Secret type is used for programmatic interaction with the Secret data.

What are the different ways to create K8s Secrets?

There are multiple ways to create a Kubernetes Secret.

How is a Kubernetes Secret created using kubectl?

To create a Secret via kubectl, you’re going to want to first create a text file to store the contents of your Secret, in this case a username.txt and password.txt:

echo -n 'admin' > ./username.txt
echo -n '1f2d1e2e67df' > ./password.txt

Then you’ll want to leverage the kubectl create secret to package these files into a Secret, with the final command looking like this:

kubectl create secret generic db-user-pass \
  --from-file=./username.txt \
  --from-file=./password.txt

The output should look like this:

secret/db-user-pass created

How is a Kubernetes Secret created from a config file?

You can also store your secure data in a JSON or YAML file and create a Secret object from that. The Secret resource contains two distinct maps:

• data: used to store arbitrary data, encoded using base64

• stringData: allows you to provide Secret data as unencoded strings

The keys of data and stringData must consist of alphanumeric characters, ‘-’ (dash), ‘_’ (underscore), or ‘.’ (period).

For example, to store two strings in a Secret using the data field, you can convert the strings to base64 as follows:

echo -n 'admin' | base64

The output should look like this:

YWRtaW4=

And for the next one:

echo -n '1f2d1e2e67df' | base64

The output should look similar to:

MWYyZDFlMmU2N2Rm

You can then write a secret config that looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

How is a Kubernetes Secret created using kustomize?

You can also generate a Secret object by defining a secretGenerator in a kustomization.yaml file that references other existing files. For example, the following kustomization file references the ./username.txt and the ./password.txt files, for example:

secretGenerator:
- name: db-user-pass
  files:
  - username.txt
  - password.txt

Then apply the directory containing the kustomization.yaml to create the Secret:

kubectl apply -k .

The output should look similar to this:

secret/db-user-pass-96mffmfh4k created

How is a Kubernetes Secret edited?

You can edit an existing Secret with the following command:

kubectl edit secrets mysecret

This command opens the default configured editor and allows for updating the base64 encoded Secret values in the data field:

  # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: { ... }
  creationTimestamp: 2016-01-22T18:41:56Z
  name: mysecret
  namespace: default
  resourceVersion: "164619"
  uid: cfee02d6-c137-11e5-8d73-42010af00002
type: Opaque

How are Kubernetes Secrets used?

Secrets can be used in a variety of ways, such as being mounted as data volumes or exposed as environment variables to be used by a container in a Pod. Secrets can also be used by other parts of the system, without being directly exposed to the Pod. For example, Secrets can hold credentials that other parts of the system should use to interact with external systems on your behalf.

How can Kubernetes Secrets be used as environment variables in a Pod?

To use a secret in an environment variable in a Pod, you’ll want to:

1. Create a secret or use an existing one. Multiple Pods can reference the same secret.

   1. Modify your Pod definition in each container that you wish to consume the value of a Secret key to add an environment variable for each Secret key you wish to consume. The environment variable that consumes the Secret key should populate the Secret’s name and key in  env[].valueFrom.secretKeyRef.

2. Modify your image and/or command line so that the program looks for values in the specified environment variables.

This is an example of a Pod that uses secrets from environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
  restartPolicy: Never

What are immutable Kubernetes Secrets, and what are the benefits?

Kubernetes provides an option to set individual Secrets as immutable, which can be implemented to enhance the security and performance of your cluster. For clusters that extensively use Secrets (at least tens of thousands of unique Secret to Pod mounts), preventing changes to their data has the following advantages:

• Protects you from accidental (or unwanted) updates that could cause application outages

• Improves performance of your cluster by significantly reducing load on kube-apiserver, by closing watches for Secrets marked as immutable

How are immutable Kubernetes Secrets created?

This feature is controlled by the ImmutableEphemeralVolumes feature gate, which is enabled by default since v1.19. You can create an immutable Secret by setting the immutable field to true. For example:

apiVersion: v1
kind: Secret
metadata:
  ...
data:
  ...
immutable: true

What are the built-in types of Kubernetes Secrets?

• Opaque Secrets – The default Secret type if omitted from a Secret configuration file.

• Service account token Secrets – Used to store a token that identifies a service account. When using this Secret type, you need to ensure that the `kubernetes.io/service-account.name` annotation is set to an existing service account name.

• Docker config Secrets – Stores the credentials for accessing a Docker registry for images.

• Basic authentication Secret – Used for storing credentials needed for basic authentication. When using this Secret type, the `data` field of the Secret must contain the `username` and `password` keys.

• SSH authentication secrets – Used for storing data used in SSH authentication. When using this Secret type, you will have to specify a `ssh-privatekey` key-value pair in the `data` (or `stringData`) field as the SSH credential to use.

• TLS secrets – For storing a certificate and its associated key that are typically used for TLS . This data is primarily used with TLS termination of the Ingress resource, but may be used with other resources or directly by a workload. When using this type of Secret, the `tls.key` and the `tls.crt` key must be provided in the data (or `stringData`) field of the Secret configuration.

• Bootstrap token Secrets – Used for tokens used during the node bootstrap process. It stores tokens used to sign well known ConfigMaps.

What are the limitations of Kubernetes Secrets?

Kubernetes Secrets are just one mechanism you should consider for your overall Kubernetes security strategy. Once a container has consumed a secret via an environment variable, the value is freely available within the container. The value is also stored without encryption in etcd. This doesn’t mean you shouldn’t use Kubernetes Secrets, but that you should understand their limitations, follow best practices, and know that you will need to combine it with other security mechanisms to adequately protect your Kubernetes deployments.

What are some best practices for maintaining and increasing the security of Kubernetes secrets?

Encrypt Secret data at rest

By default, etcd stores Kubernetes Secrets unencrypted. Enable at-rest encryption for your Secret objects, including the key-value data they contain.

Implement least-privilege access to Secrets

Follow RBAC best practices and only assign minimal rights to users and service accounts. Only allow the most privileged, system-level components to access watch or list, and only the components that require get access for Secrets to execute their normal functions should have it.

As for humans, only cluster admins should have access to etcd, and get, watch, and list access (including read-only access) to Secrets should be restricted. Third-party authorization solutions can provide more robust access control, such as restricting access to Secrets that have certain annotations.

Use annotations to enforce specific rules for Secret management

By annotating a ServiceAccount with kubernetes.io/enforce-mountable-secrets set to true, Kubernetes enforces specific rules for Pods running as this ServiceAccount, that require the ServiceAccount’s secrets field to include Secrets mounted as volumes as well as Secrets referenced in envFrom for containers. Additionally, the ServiceAccount’s imagePullSecrets field must include Secrets referenced in a Pod’s imagePullSecrets. These rules are enforced whenever a Pod is created or updated.

Strengthen etcd management policies

Any durable storage used by etcd that is no longer needed should be wiped or shredded. For deployments with multiple etc instances, protect Secret data in transit by encrypting SSL/TLS communication between the instances.

Use 3rd-party Secrets store providers

Keep confidential credentials and other sensitive data outside your cluster by using external Secrets store providers that Pods can access. These solutions leverage the Kubernetes Secrets Store CSI Driver.

How are Kubernetes Secrets managed using Lens Desktop?

Lens Desktop makes it easy to create Kubernetes Secrets for a cluster using a dialog menu. Simply enter the parameters, and click Create.

After creating Kubernetes Secrets, you can view, filter, and search through a table of Secrets for each cluster and click on each item to view further details, edit, or delete it.

Conclusion

So now that you’ve had a brief introduction to what a Kubernetes Secret is, you’re ready to learn how to use them in practice. Download our free eBook, Kubernetes 5 Minutes at a Time, and check out Chapter 9,  “Secrets with Environment Variables and Volume Mounts,” to learn more, with practice exercises you can try to inject Kubernetes Secrets as environment variables or by using files in a volume.