In the two previous posts in this series, I have discussed security in GKE - starting your Kubernetes journey with one of the most important things - and then GitOps and CICD in GKE - showing how you can use Git, along with some GCP services, to make sure you are deploying Kubernetes in an auditable and reproducible way.
Next on the list is how you handle secret data in GKE while enabling the GitOps pattern discussed in my last blog. This blog will go over the native secrets in Kubernetes and will show you how to make this data even more secure.
A Secret in Kubernetes is a resource which is used for storing small amounts of private data to be consumed by pods on the cluster. A basic secret YAML definition looks like:
This would create a secret resource named “test” with the data shown. These data values are base64 encoded and are decoded when read by the pod.
These resources live only in the clusters “etcd” database when not in use and then when they are called by a pod, are only pulled onto the node which hosts the pod that requires it. This serves to reduce the exposure of the secret on the cluster.
The biggest issue with these resources, especially when Git is being used as heavily as it is in a GitOps workflow, is that these values are only encoded using base64. This means that these values can be easily turned into plain text by anyone who has access to the encoded string, in turn making it a risk to add the manifest of the secret to your source control repository.
This is, of course, a huge roadblock for a GitOps workflow as, if the resource isn’t in Git, it cannot be properly watched and deployed by your continuous delivery tool. So we need to find a way of using an encrypted version of this secret in Git which can be converted into a normal Kubernetes secret post-deployment.
This is where Bitnami’s Sealed Secrets comes in.
Bitnami Sealed Secrets
“Sealed Secrets” is a tool that uses RSA key pairs to encrypt and decrypt secret manifests to allow them to be uploaded to Git. It utilises a controller, deployed to the cluster, to decrypt sealed secrets on deployment to the cluster.
At CTS, we have deployed this tool many times in production and found it to be a good one. It is easy to deploy and does the job it is intended to do well.
However, one major drawback is in the management of the key and certificate required which must be managed and distributed outside of the cluster without being uploaded anywhere to ensure they don’t get exposed. This means that another tool must be used for this adding to the overhead of using it.
Another issue is that a secret must be created containing the key pair and deployed to the cluster upon cluster creation to ensure that it can be used to decrypt the secrets you have created. This is a manual step which intervenes with the automatic deployment of your manifests using your deployment tool and presents a bootstrapping problem.
Another option is to utilise Hashicorp’s Vault for secret storage. This tooling is intended to be used as a place to store secret data in a secure way and can be utilised for Kubernetes using a webhook to inject secret data from the Vault into pods that require it.
In my experience, Vault is not a simple tool to deploy and manage and requires its own infrastructure to be run on. However, Vault is a very well-defined tool for secret management and may need to be used when very specific security requirements are in place. This means it is recommended if you have the time and resources to manage Vault as a separate tool.
If self-managing infrastructure for your secrets is not for you, Google has a Key Management Service available to users which provides and manages keys for asymmetric encryption and decryption which can be accessed by anyone who has the correct IAM permissions. This addresses many of the concerns about using Bitnami’s Sealed Secrets discussed above. Next, I will discuss some ways you can utilise this service in GKE.
GCP offers a couple of options to manage secrets while utilizing the managed services it has available. Berglas is a command-line tool for doing just that - utilizing Google’s Key Management Service (KMS) and bucket storage (GCS).
This tool can be used in Kubernetes for applications which utilise environment variables for the secret values. In the manifest for the pod, when you define the container, you would use a specific syntax for referencing a Berglas secret object in the environment variables section which is then mutated by a webhook.
Where secrets are required as files or simply not as environment variables, this tool can be a little more complicated to use and either requires the application to be developed with Berglas in mind or Berglas to be installed in the container to modify the entrypoint used.
A recent contender in this field, GCP’s Secret Manager is a managed service which encrypts and stores secret data. This is very similar to Berglas but is managed and reduces further the overhead required for management of the solution.
In order to use the tool in Kubernetes, Berglas can be utilised as described above for environment variables or by modifying the entrypoint and referencing a Secret Manager secret rather than a Berglas one. If you want to use Secrets Manager directly, it requires the application to use the available client libraries to access secrets.
This may be an issue for modernising applications that have already been written or for when specific requirements don’t allow it but Secret Manager is an excellent managed solution for the problem of secret management in Kubernetes and completely removes the need for the secret to be stored in Git.
Write your own!
One last option is that you could write your own tooling to manage secret encryption and decryption like Sealed Secrets using a managed service like KMS for managing the keys for you. This is an excellent way to learn the ins and outs of secrets in Kubernetes but does have a pretty huge amount of initial overhead - not for the faint of heart!
Your applications secrets are a very important part of your deployment and it is most important that these are kept safe. However, this can cause a few problems when you want to use Git to manage your manifests.
This was a quick comparison of some tools for dealing with secret encryption that I’ve had experience with when deploying Kubernetes workloads using a GitOps workflow. Bitnami Sealed Secrets is a great option with no extra integrations required by your application, Vault can be hard work to manage but is a well-established tool for secret management, Berglas and Google’s Secrets Manager allow you to use Google’s managed service for managing your keys but also may require some extra work to integrate into your application.
It’s up to you to make a decision about what tool you implement for this job but it is important that you think about your secrets carefully when starting to deploy to Kubernetes.