Using system environment variables with Kustomize
UPDATE (2022/07/28): The trick used here was previously documented officially, but the documentation has been reverted (because it is supposed to be a bug). Instead, a warning message now shows up: This Kustomization is relying on a bug that loads values from the environment when they are omitted from an env file. This behaviour will be removed in the next major release of Kustomize. See https://github.com/kubernetes/website/issues/35669 and https://github.com/kubernetes-sigs/kustomize/issues/4731
UPDATE(2022/01/07): The trick used here to set values from local environment variables has finally been documented in the official documentation: kubernetes/website#30348
UPDATE(2020/07/01): This uses an eschewed (but undocumented) feature of Kustomize. This trick may change, break, or inexplicably disappear at any time. So use this at your own risk!
On my journey with Kubernetes, I played a little bit with Kustomize, which is a great tool for adjusting Kubernetes YAML resources to various deployment environments. I was actually surprised to see how Kustomize enforces the use of files (versioned) to build the Kubernetes manifests. But I also stumbled upon an undocumented feature in its code, allowing to use runtime environment variables.
After a general overview of what Kustomize allows to do, we will walk through leveraging runtime system environment variables to parameterize Kustomize files.
A typical use case for this might be in Continuous Deployment (CD) contexts, where, rather than generating configuration files, we could easily leverage existing environment variables for deployment.
For reference, the whole Kustomize project used throughout this blog post is available at GitHub://rm3l/kustomize_envvar
Overview
In a nutshell, using Kustomize, we would be able to:
- start from a set of base (and general-purpose) YAML files, which we do not want to alter
- apply patches to such base YAML files, resulting in customized YAML files which will get submitted to a given Kubernetes cluster
Since version 1.14, the kubectl command comes bundled with Kustomize, allowing us to use commands such as:
|
|
or
|
|
Anatomy of a Kustomize project
Let’s walk through a simple scenario to better understand what we want to do.
Say we have the following project structure for Kustomize:
|
|
Base
base is the folder containing the set of raw Kubernetes files which we do not want to alter at all.
- base/deployment.yaml is a typical YAML Deployment descriptor for Kubernetes:. For example, it deploys 3 replicas of a Pod configured via a ConfigMap that needs to be provided. Here we will configure the ENABLE_RISKY environment variable flag.
|
|
- And base/kustomization.yaml is a descriptor needed for Kustomize. In this case, it simply declares all resources that should be included by Kustomize using the resources field:
|
|
Overlays
Overlays contain a set of variants off of a same base Kustomization configuration, and allows to apply patches to cater to various environment needs.
As the name suggests here, overlays/staging contains the variant for a staging environment. It will allow us to provide the the-map ConfigMap with the ENABLE_RISKY key configurable dynamically at runtime via a system environment variable.
- overlays/staging/kustomization.yaml looks like what follows. It declares the base directory, along with the way to generate the the-map ConfigMap from a given key-value properties file, along with a patch to apply to :
|
|
- overlays/staging/deployment.yaml file is a simple deployment which once merged with the base one, will change the number of replicas from 3 to 2 for the staging variant. As you can see below, we do not need to provide a complete valid Kubernetes Deployment resource. But using the patchesStrategicMerge strategy, Kustomize is able to find the resource using its name (the-deployment here) and merge the two files.
|
|
- overlays/staging/config.properties is a simple key-value properties file used by Kustomize to generate the the-map ConfigMap:
|
|
Building the Kustomize overlay
Now that all the structure of our Kustomize project is defined, we can test it by generating Kubernetes YAML descriptors for our staging overlay:
|
|
The command above outputs the resulting Kubernetes resources which we can then pipe and apply against out Kubernetes cluster:
|
|
Using system environment variables
As we can see in the sample project below, the ConfigMap keys and values are pretty static, and cannot be overridden easily. For this, we could simply copy the overlays/staging folder and change the new overlay config.properties accordingly, but this is cumbersome to me just for changing values in the ConfigMap.
The kustomize command exposes an edit command, which edits the kustomization.yaml file, and can be called with environment variables if needed.
What I wanted to do instead is use the same overlays/staging variant, but alter the ENABLE_RISKY property at runtime from environment variables, without editing any kustomization.yaml files.
To do so, the trick is basically to change 2 things:
- Declare the alter-able key in the overlays/staging/config.properties file, i.e., with no value, e.g.:
|
|
- Export the environment variable prior to calling Kustomize:
|
|
As we can see in the resulting output, the ENABLE_RISKY variable has been successfully changed to “true”.
Explanations
After digging a little bit into Kustomize source code, I found out that, when keys are set but without values, the key-value loaders in Kustomize generators default to the context environment when parsing env files. See this code block:
|
|
This is not documented at all, but good to keep in mind, until this behavior eventually changes in the future. I don’t know whether this is intentional, but for now, I thought it was worth sharing to people with similar needs.
EDIT
- I just stumbled upon this open issue in GitHub describing this, along with other possible alternatives (like using the envsubst command).