How to Manage Common Labels and Annotations in Kustomize for Base and Overlays

In this article, we are going to cover How to Manage Common Labels and Annotations in Kustomize for Base and Overlays.

Managing metadata in Kubernetes configurations can be tricky, especially when working with tools like Kustomize. Labels and annotations play a crucial role in organizing and identifying resources, but improper use of common labels in Kustomize can lead to challenges when using overlays. This article explains the nuances of managing common labels and annotations effectively in Kustomize and demonstrates best practices with examples.

Prerequisites

Before starting, ensure you have the following:

1. Kustomize installed: Download and install Kustomize from kustomize.io.
2. Basic understanding of Kubernetes: Familiarity with YAML and Kubernetes objects like Deployments, Services, etc.
3. Directory structure setup: Create a base and overlays directory structure as shown below.
4. kubectl installed: Required for applying the configurations.

What are Labels and Annotations?

Labels

Labels are key-value pairs used to categorize and organize Kubernetes resources. They are critical for resource selection, as they enable controllers to match resources based on label selectors. For example:

labels:
  app: my-app
  environment: production

Annotations

Annotations are also key-value pairs, but unlike labels, they are not used for selection or organization. Instead, annotations store metadata that helps describe a resource. For example:

annotations:
  managed-by: kustomize
  description: "Production deployment"

Key Difference

• Labels influence the behavior of resources (e.g., by selectors).
• Annotations are purely informational and do not affect resource behavior.

The Problem with Common Labels in Kustomize

When you define common labels in the base of a Kustomize setup, these labels propagate to all resources, including those in overlays. This can cause:

1. Unintended conflicts: Overlays may need environment-specific labels that differ from the base.
2. Redundancy: Overlays may end up with duplicate or unnecessary labels.

Example Scenario

Imagine a base configuration with a deployment resource:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app

If you set a common label environment: base in the base’s kustomization.yaml, it will also apply to overlays, potentially overriding overlay-specific labels.

The Solution: Use Annotations and Overlay-Specific Labels

To avoid conflicts, follow these practices:

1. Use common annotations in the base for global metadata.
2. Define labels specific to each overlay to customize resources for different environments.
3. Avoid setting common labels in the base.

Directory Structure

Organize your Kustomize configuration files as follows:

.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    └── prod
        └── kustomization.yaml

Step-by-Step Guide

1. Base Configuration

Define your base resources with annotations but avoid adding common labels.

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: nginx

base/kustomization.yaml

resources:
- deployment.yaml

commonAnnotations:
  description: base-configuration
  managed-by: kustomize
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

2. Overlay Configuration

Customize overlays with specific labels and avoid inheriting conflicting labels from the base.

overlays/prod/kustomization.yaml

resources:
- ../../base

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: true
  pairs:
    environment: production

overlays/dev/kustomization.yaml

resources:
- ../../base

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- includeSelectors: true
  pairs:
    environment: development

Commands and Outputs

Base Build

Run the following command to build the base resources:

kustomize build base/

Output:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    description: base-configuration
    managed-by: kustomize
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      annotations:
        description: base-configuration
        managed-by: kustomize
      labels:
        app: my-app
    spec:
      containers:
      - image: nginx
        name: my-app

Overlay Build (Production)

Run the following command to build the production overlay:

kustomize build overlays/prod/

Output:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    description: base-configuration
    managed-by: kustomize
  labels:
    environment: production
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      environment: production
  template:
    metadata:
      annotations:
        description: base-configuration
        managed-by: kustomize
      labels:
        app: my-app
        environment: production
    spec:
      containers:
      - image: nginx
        name: my-app

Overlay Build (Development)

Run the following command to build the development overlay:

kustomize build overlays/dev/

Output:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    description: base-configuration
    managed-by: kustomize
  labels:
    environment: development
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      environment: development
  template:
    metadata:
      annotations:
        description: base-configuration
        managed-by: kustomize
      labels:
        app: my-app
        environment: development
    spec:
      containers:
      - image: nginx
        name: my-app

Best Practices

1. Separate metadata responsibilities:
   • Use annotations for base-level information.
   • Apply specific labels in overlays.
2. Keep the base generic: Avoid environment-specific configurations in the base.
3. Use overlays effectively: Customize overlays to add environment-specific metadata and configurations.
4. Validate your configuration: Use kustomize build to ensure configurations are as expected before applying them to the cluster.

Conclusion:

Managing labels and annotations in Kustomize requires careful planning to avoid conflicts and redundancy. By using annotations in the base and applying labels in overlays, you can create a clean and flexible configuration setup. This approach ensures seamless integration of overlays while keeping the base reusable and conflict-free.

By following these practices, you can streamline the management of Kubernetes resources and avoid common pitfalls when working with Kustomize.

Related Articles:

Managing Namespaces in Kubernetes with Kustomize

Reference:

kustomize official page