Using system environment variables with Kustomize

Using system environment variables with Kustomize

On my journey with Kubernetes, I played a little bit with Kustomize, which is a great tool for adjusting Kubernetes YAML files to various deployment constraints.

After a general overview of what Kustomize allows to do, this blog post is more about giving few tips about how we can leverage system environment variables to parameterize Kustomize files.

A typical use case for this might be in Continuous Deployment (CD) contexts, where, rather than generating configuration files, we could easily leverage existing environment variables for deployment.

Overview

In a nutshell, using Kustomize, we would be able to:

  • start from a set of base (and general-purpose) YAML files, which we do not want to alter
  • apply patches to such base YAML files, resulting in customized YAML files which will get submitted to a given Kubernetes cluster

Since version 1.14, the kubectl command comes bundled with Kustomize, allowing us to use commands such as:

kubectl kustomize /path/to/kustomize/overlay | kubectl apply -f -

or

kubectl apply -k /path/to/kustomize/overlay

Anatomy of a Kustomize project

Let's walk through a simple scenario to better understand what we want to do.

Say we have the following project structure for Kustomize:

❯ tree     
.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays
    └── staging
        ├── config.properties
        ├── deployment.yaml
        └── kustomization.yaml

Base

base is the folder containing the set of raw Kubernetes files which we do not want to alter at all.

  • base/deployment.yaml is a typical YAML Deployment descriptor for Kubernetes:. For example, it deploys 3 replicas of a Pod configured via a ConfigMap that needs to be provided. Here we will configure the ENABLE_RISKY environment variable flag.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
    spec:
      containers:
      - name: the-container
        image: monopole/hello:1
        command: ["/hello",
                  "--port=8080",
                  "--enableRiskyFeature=$(ENABLE_RISKY)"]
        ports:
        - containerPort: 8080
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              name: the-map
              key: ALT_GREETING
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              name: the-map
              key: ENABLE_RISKY
  • And base/kustomization.yaml is a descriptor needed for Kustomize.  In this case, it simply declares all resources that should be included by Kustomize using the resources field:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

Overlays

Overlays contain a set of variants off of a same base Kustomization configuration, and allows to apply patches to cater to various environment needs.

As the name suggests here, overlays/staging  contains the variant for a staging environment. It will allow us to provide the the-map ConfigMap with the ENABLE_RISKY key configurable dynamically at runtime via a system environment variable.

  • overlays/staging/kustomization.yaml looks like what follows. It declares the base directory, along with the way to generate the the-map ConfigMap from a given key-value properties file, along with a patch to apply to :
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: staging-

bases:
  - ../../base

configMapGenerator:
  - name: the-map
    env: config.properties

patchesStrategicMerge:
  - deployment.yaml

  • overlays/staging/deployment.yaml file is a simple deployment which once merged with the base one, will change the number of replicas from 3 to 2 for the staging variant. As you can see below, we do not need to provide a complete valid Kubernetes Deployment resource. But using the patchesStrategicMerge strategy, Kustomize is able to find the resource using its name (the-deployment here) and merge the two files.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
spec:
  replicas: 2
  • overlays/staging/config.properties is a simple key-value properties file used by Kustomize to generate the the-map ConfigMap:
ALT_GREETING=Hiya
ENABLE_RISKY=false

Building the Kustomize overlay

Now that all the structure of our Kustomize project is defined, we can test it by generating Kubernetes YAML descriptors for our staging overlay:

❯ kubectl kustomize overlays/staging


apiVersion: v1
data:
  ALT_GREETING: Hiya
  ENABLE_RISKY: "false"
kind: ConfigMap
metadata:
  name: staging-the-map-7c88gg7h68
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: staging-the-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
    spec:
      containers:
      - command:
        - /hello
        - --port=8080
        - --enableRiskyFeature=$(ENABLE_RISKY)
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              key: ALT_GREETING
              name: staging-the-map-7c88gg7h68
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              key: ENABLE_RISKY
              name: staging-the-map-7c88gg7h68
        image: monopole/hello:2
        name: the-container
        ports:
        - containerPort: 8080

The command above outputs the resulting Kubernetes resources which we can then pipe and apply against out Kubernetes cluster:

❯ kubectl kustomize overlays/staging  | kubectl apply -f -

configmap/staging-the-map-5mfm8kmm8t created
deployment.apps/staging-the-deployment created

Using system environment variables

As we can see in the sample project below, the ConfigMap keys and values are pretty static, and cannot be overridden easily. For this, we could simply copy the overlays/staging folder and change the new overlay config.properties accordingly, but this is cumbersome to me just for changing values in the ConfigMap.

What I wanted to do instead is use the same overlays/staging variant, but alter the ENABLE_RISKY property at runtime from environment variables.

To do so, we need to change 2 things:

  • Declare the alter-able key in the overlays/staging/config.properties file, i.e., with no value, e.g.:
ALT_GREETING=Hiya
ENABLE_RISKY
  • Export the environment variable prior to calling Kustomize:
❯ ENABLE_RISKY="true" kubectl kustomize overlays/staging

apiVersion: v1
data:
  ALT_GREETING: Hiya
  ENABLE_RISKY: "true"
kind: ConfigMap
metadata:
  name: staging-the-map-5mfm8kmm8t
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: staging-the-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
##### The rest of the output has been omitted for brevity ####

As we can see in the resulting output, the ENABLE_RISKY variable has been successfully changed to "true".

Explanations

After digging a little bit into Kustomize source code, I found out that, when keys are set but without values, the key-value loaders in Kustomize generators default to the context environment when parsing env files. See this code block:

    if len(data) == 2 {
        kv.Value = data[1]
    } else {
        // No value (no `=` in the line) is a signal to obtain the value
        // from the environment.
        kv.Value = os.Getenv(key)
    }
    kv.Key = key
    return kv, nil

This is not documented at all, but good to keep in mind, until this behavior eventually changes in the future. But for now, I thought it was worth sharing for people with similar needs.

EDIT

  • I just stumbled upon this open issue in GitHub describing this, along with other possible alternatives.