Streamlining Local Development on Kubernetes

こんにちは、次世代システム研究室のN.M.です。

Introduction

Firstly, you probably shouldn’t be developing your app on Kubernetes locally.
If possible, use a more lightweight solution, such as Telepresence for K8s to develop locally and interact with a remote cluster.
But, sometimes you need to test a Kubernetes configuration, and rather than use a remote Kubernetes provider, it’s faster, cheaper, and safer to do locally.

Here I test a non-trivial Kubernetes configuration locally.
One aspect of the configuration sets up Nginx and a rust Actix server connected using a Unix Domain Socket (UDS). This is usually faster than connecting through a TCP socket.

Another aspect of the configuration shows how to use Persistent Volumes to support a DB running within K8s.

Our K8s cluster should be easy to create and easy to delete when we are finished with it.

Initialization techniques and tools are discussed that make the process easier.

k3d

K3d is made by Rancher and runs k3s within Docker, it’s lightweight and easy to use.

After starting k3d, check k3d’s Docker containers with docker ps

To start a k3d cluster

k3d cluster create multichoice-dev -p "8888:80@loadbalancer" --registry-create k3d-registry

This creates a local single-node cluster called multichoice-dev. It sets up an LB for ingress that is reachable on the host through port 8888.

An image registry called k3d-registry is created. It runs on the host docker and is reachable from the host and from k3d. You can build and push images from the host and pull images from inside k3d when used in deployment manifests.

As described here, in order to have services reachable from the load balancer you need to set up a K8s ingress resource.

An example is given below

# apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multichoice
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: actix-multichoice-svc
                port:
                  number: 9999
          - path: /db(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: db-admin-svc
                port:
                  number: 7777

To use HTTP for testing:
ingress.kubernetes.io/ssl-redirect
– disables the default behavior which redirects to the HTTPS when given an HTTP URL.

To map paths to different backend services:
nginx.ingress.kubernetes.io/rewrite-target: /$2
– will use the second regular expression match in a path, when invoking the backend service.
The example above uses the regular expression /db(/|$)(.*). When calling http://localhost:8888/db/pgAdmin, the ingress controller would call a backend such as http://db-admin-svc:7777/pgAdmin.

To delete our k8s cluster and all its Docker containers, run the command below

❯ k3d cluster delete multichoice-dev

Handling Container Images

K8s is, of course, a container orchestration tool. So we are going to be building, tagging, pushing images from our host, and pulling images from our manifests inside k3d.

When we started the cluster we told k3d to set up a container registry, k3d-registry, for this purpose.

❯ docker ps -f name=k3d-registry
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e5d8fb5dd9af registry:2 "/entrypoint.sh /etc…" 2 hours ago Up 2 hours 0.0.0.0:57505->5000/tcp k3d-registry

k3d-registry maps a randomly generated host port, 57505 to the default port, 5000 on k3d.

When accessing the registry from our host we will use 57505.
When accessing from within K8s we will use 5000.

To create an image and make it available to K8s we need to perform the following three steps on our host

Build the image

❯ docker build -t dieselcli -f Dockerfile_diesel .

Tag the image

❯ docker tag dieselcli localhost:57505/dieselcli

Push the image

❯ docker push localhost:57505/dieselcli

To refer to this image from a manifest we would use the following inside a yaml resource manifest

image: k3d-registry:5000/dieselcli:latest

Automating container deployment

The 3 steps to create a new container image in the k3d-registry, as well as updating K8s manifests image reference can get tiring quickly. Luckily this can be automated. A tool that is specifically designed for this is Skaffold, developed by Google.

Skaffold uses a configuration file to tell it how to build and deploy images to K8s.

You can generate this file using the init command.

❯ skaffold init -f skaffold.yaml

This command will look for Dockerfiles (and other application builders), image references, and K8s manifest files, Kustomize files, or Helm files. It will then prompt the user to confirm how the images should be built and deployed. After receiving the answers it needs, skaffold generates the config file.

Skaffold is quite smart about looking at the files in your project and analyzing their contents.

You may also make any necessary manual changes to skaffold.yaml.

Skaffold discovers and uses the container registry (here k3d-registry), generates new image tags and uses those tags to deploy the images to K8s.

Skaffold can be run in two modes watch mode or run mode. In watch mode, it watches for any changes to files referenced by your images. In run mode it runs once, builds and deploys, then exits.

Run skaffold in watch mode

skaffold dev

It is smart enough to know what files are needed by the images. For example, with Docker builds, it looks inside Dockerfiles, to see which files it should watch. Change any of these files and skaffold will rebuild, re-tag, re-push, and re-deploy your images.

Type ctl-C to quit skaffold in watch mode. Skaffold will clean up by returning the K8s cluster to the state it was in before.

Run skaffold once to build and deploy images.

skaffold run --tail

The --tail option outputs the container logs.

Initializing the Application

Sometimes you need to perform application initialization. We will be restarting K8s and redeploying our application multiple times, so it’s best to automate this.

The manifest segment below uses a initContainer to perform such an initialization step. In this case, it creates a DB schema.

initContainers:
  - name: init-db
    image: k3d-registry:5000/dieselcli:latest
    command: ["diesel", "migration", "run"]

The image, k3d-registry:5000/dieselcli:latest has already been built, pushed and deployed by skaffold.

Here is the Dockerfile

FROM rustlang/rust:nightly AS diesel_cli

WORKDIR /app
RUN apt-get update -qq && \
    rm -rf /var/lib/apt/lists/* && \
    cargo install diesel_cli

COPY --chown=1000:3000 migrations migrations

USER 1000:3000

ENV DATABASE_URL=postgres://multichoice:postgres@multichoice-db/multix
# RUN diesel setup
CMD [ "sleep","100000" ]

The image provides a container with the diesel CLI (a CLI for the rust diesel ORM). initContainer invokes the command diesel migration run. This creates the application schema as we have defined it in our source directory. The command runs migration files (create table SQL statements) under the migrations directory.

Now, whenever we restart k3d, k8s will handle creating the DB schema.

Creating the DB and using a Persistent Volume for Data

Creating a DB inside K8s has specific challenges. The data should be persistent across deployments, for this we use a Persistent Volume and Persistent Volume Claim.

To initialize the DB, we use a configMap to an executable script. This automates DB initialization.

Here is the deployment manifest for my postgres DB

apiVersion: apps/v1
kind: Deployment
metadata:
  name: multichoice-db
spec:
  selector:
    matchLabels:
      app: multichoice-db
  template:
    metadata:
      labels:
        app: multichoice-db
    spec:
      volumes:
        - name: postgres-pv-storage
          persistentVolumeClaim:
            claimName: postgres-pvc
        - name: db-init
          configMap:
            name: db-init-conf
            defaultMode: 0777
            items:
              - key: init-user-db.sh
                path: init-user-db.sh

      containers:
        - name: multichoice-db
          image: postgres:latest
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              value: multichoice
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-password
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
            - name: POSTGRES_DB
              value: multichoice
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-pv-storage
            - mountPath: /docker-entrypoint-initdb.d/init-user-db.sh
              subPath: init-user-db.sh
              name: db-init

It uses a persistent volume mount for DB data files, postgres-pv-storage.

It also uses a configMap volume mount for DB initialization, db-init-conf. This gets mounted to the /docker-entrypoint-initdb.d/init-user-db.sh path. The postgres image runs any scripts or SQL it finds in /docker-entrypoint-initdb.d/ as part of DB initialization.

Here is the init-user-db.sh script

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE USER multix;
    CREATE DATABASE multix;
    GRANT ALL PRIVILEGES ON DATABASE multix TO multix;
EOSQL

It’s a simple bash shell to create a database user and database.

The key point to note here is that I specify subPath: init-user-db.sh. Without the subPath, /docker-entrypoint-initdb.d/init-user-db.sh becomes a symbolic link to a separate file created by K8s. Then, when the postgres container tries to run the link, a permission denied error occurs. Using subPath fixes this problem.

Using a Unix Domain Socket for fast connection between Nginx and the App

There are times when you may want Nginx (or another webserver) to be in front of your application. You may need special features of Nginx, for example, if you have an API that is accessed directly from client browsers, you may need to handle CORS requests.

But, if possible you don’t want to have a TCP connection between Nginx and the app, rather you want a faster Unix Domain Socket (UDS) connection. To allow this on K8s, the Nginx and App containers must both be running on the same Pod, so they may share the socket file.

An example manifest configuration is shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: actix-multichoice
spec:
  selector:
    matchLabels:
      app: actix-multichoice
  template:
    metadata:
      labels:
        app: actix-multichoice
    spec:
      containers:
        - name: actix-multichoice
          image: k3d-registry:5000/actixmultichoice:latest
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          volumeMounts:
            - mountPath: /var/run/actix
              readOnly: false
              name: actix-uds

        - name: nginx
          image: k3d-registry:5000/nginx
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
            limits:
              cpu: 500m
              memory: 500Mi
          ports:
            - containerPort: 9000
          volumeMounts:
            - mountPath: /etc/nginx
              readOnly: true
              name: nginx-conf
            - mountPath: /var/run/actix
              readOnly: false
              name: actix-uds
          livenessProbe:
            exec:
              command:
                - sh
                - -c
                - curl --fail http://localhost:9000/tests
            initialDelaySeconds: 10
            periodSeconds: 3
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - curl --fail http://localhost:9000/tests
            initialDelaySeconds: 6
            periodSeconds: 3

      initContainers:
        - name: init-db
          image: k3d-registry:5000/dieselcli:latest
          command: ["diesel", "migration", "run"]

      volumes:
        - name: nginx-conf
          configMap:
            name: actix-nginx-conf
            items:
              - key: nginx.conf
                path: nginx.conf
        - name: actix-uds
          emptyDir: {}
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000

This defines two containers, nginx and actix-multichoice. These two containers talk to each other via a UDS, /var/run/actix/actix.sock. This file is accessed by a shared emptyDir volume, actix-uds.

Using securityContext ensures Nginx and actix are both run as the same user and group. This ensures, the UDS file /var/run/actix/actix.sock is readable and writable by both processes.

The nginx container also uses a configMap to initialize its configuration, nginx.conf. This is mounted as readOnly. This is a standard Nginx configuration file.

We’ve already talked about the initContainers section.

Skaffold output for our App

Here is the output produced by skaffold run --tail on the deployment described above:

❯ skaffold run --tail
Generating tags...
 - k3d-registry:5000/actixmultichoice -> k3d-registry:5000/actixmultichoice:ae728e8-dirty
 - k3d-registry:5000/dieselcli -> k3d-registry:5000/dieselcli:ae728e8-dirty
 - k3d-registry:5000/nginx -> k3d-registry:5000/nginx:ae728e8-dirty
Checking cache...
 - k3d-registry:5000/actixmultichoice: Found Locally
 - k3d-registry:5000/dieselcli: Found Locally
 - k3d-registry:5000/nginx: Found Locally
Starting test...
Tags used in deployment:
 - k3d-registry:5000/actixmultichoice -> k3d-registry:5000/actixmultichoice:dcb40d96af9503c15bb18328311f7e5bac11a204f02cec95a07fb05e46d81690
 - k3d-registry:5000/dieselcli -> k3d-registry:5000/dieselcli:2cf408c54e30f93d88542feb2041e21eea4de5deeee2f8f730585f4c8adc4e6c
 - k3d-registry:5000/nginx -> k3d-registry:5000/nginx:a82fb99d455cd8fe10ca6ecda5d72f78c081763b7524986fe4ae9a4ce997aa7b
Starting deploy...
Loading images into k3d cluster nodes...
 - k3d-registry:5000/actixmultichoice:dcb40d96af9503c15bb18328311f7e5bac11a204f02cec95a07fb05e46d81690 -> Loaded
 - k3d-registry:5000/dieselcli:2cf408c54e30f93d88542feb2041e21eea4de5deeee2f8f730585f4c8adc4e6c -> Loaded
 - k3d-registry:5000/nginx:a82fb99d455cd8fe10ca6ecda5d72f78c081763b7524986fe4ae9a4ce997aa7b -> Loaded
Images loaded in 2 minutes 53.749 seconds
 - configmap/actix-nginx-conf-mg4b44m6k5 created
 - service/actix-multichoice-svc created
 - deployment.apps/actix-multichoice created
Waiting for deployments to stabilize...
 - deployment/actix-multichoice is ready.
Deployments stabilized in 3.316 seconds
Press Ctrl+C to exit
[nginx] /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
[nginx] /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
[nginx] /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
[nginx] 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
[nginx] /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
[nginx] /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
[nginx] /docker-entrypoint.sh: Configuration complete; ready for start up
[actix-multichoice] [2022-01-11T06:23:29Z INFO  actix_server::builder] Starting 1 workers
[actix-multichoice] [2022-01-11T06:23:29Z INFO  actix_server::builder] Starting "actix-web-service-"/var/run/actix/actix.sock"" service on "/var/run/actix/actix.sock" (pathname)
[init-db] Running migration 2021-11-06-082934_add_tests
[init-db] Running migration 2021-11-07-070724_add_answers
[actix-multichoice] [2022-01-11T06:52:56Z INFO actix_web::middleware::logger] - "GET /api/tests HTTP/1.0" 200 2 "-" "curl/7.64.0" 0.001773
[actix-multichoice] [2022-01-11T06:52:57Z INFO actix_web::middleware::logger] - "GET /api/tests HTTP/1.0" 200 2 "-" "curl/7.64.0" 0.001341
[nginx] 127.0.0.1 - - [11/Jan/2022:06:52:57 +0000] "GET /api/tests HTTP/1.1" 200 2 "-" "curl/7.64.0"
[nginx] 127.0.0.1 - - [11/Jan/2022:06:53:06 +0000] "GET /api/tests HTTP/1.1" 200 2 "-" "curl/7.64.0"

image tags for actixmultichoice, dieselcli, nginx have been created and loaded into the k3d node
nginx is initialized
actix-multichoice is listening to the /var/run/actix/actix.sock UDS
init-db has set up our toy DB schema
actix-multichoice is processing the readiness and liveness probes
nginx is also processing the readiness and liveness probes
- Liveness and readiness probe configuration are provided above. These allow K8s to do zero-downtime deploys. Also, as you can see, they provide quick confirmation that the app is working properly.

Kustomize

Kustomize provides a lot of advantages over vanilla K8s manifests such as configurations easily customizable for your target environments. It can easily do things such as injecting custom namespace per environment.

I generally like to use it because it adds little complexity and allows easy customizations. Currently, I don’t have a production environment, but by using kustomize, with structured configuration directories, I know I can more easily create a production-ready environment.

As already stated, skaffold can understand and use kustomize files, but if I want to deploy containers that do not use my k3d-registry, I can simply use kustomize directly:

kubectl apply -k deploy/local/db

kubectl also understands kustomize files.

deploy/local/db is a path to these files:

deploy/local/db
├── db-admin-deployment.yaml
├── db-admin-service.yaml
├── db-deployment.yaml
├── db-service.yaml
├── init-user-db.sh
├── kustomization.yaml
├── local-path-storage.yaml
├── pv.yaml
├── pvc.yaml
└── secret.yaml

kustomization.yaml is the kustomize configuration file. The syntax is powerful but here, it just tells kustomize what manifests to use and creates a configMap for DB initialization:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

configMapGenerator:
  - name: db-init-conf
    files:
      - init-user-db.sh
      
resources:
  - local-path-storage.yaml
  - secret.yaml
  - pv.yaml
  - pvc.yaml
  - db-deployment.yaml
  - db-service.yaml
  - db-admin-deployment.yaml
  - db-admin-service.yaml

Kustomize and ConfigMaps

ConfigMaps allow you to define configurations used in K8s. In this post, Nginx is configured using a configMap, and DB initialization also uses a configMap.

Changing configMap contents does not automatically trigger a reload of containers referencing that configMap.

When testing an Nginx configuration, you probably will want to try multiple versions of a config until you get one that works properly. Automatic reloading of containers referencing configMaps is provided by using a Kustomize configMaps generator

Kustomize configMaps generator renames any new configMap with a unique name and updates any references. So, any containers referencing the configMap will be reloaded automatically by K8s.

Visual Studio Code Kubernetes Tools

Microsoft’s Kubernetes extension provides some time-saving tools when working with K8s.

Install the extension, then view Kubernetes extension commands in the Command Palette: ⇧⌘P Kubernetes:

Here are the 3 commands I use most frequently

Set namespace to use for subsequent commands

Kubernetes: Use Namespace

View container logs

Kubernetes: Logs

Open container terminal

Kubernetes: Terminal

I personally find the Kubernetes extension UI much easier to use than kubectl.

In addition, Kubernetes Tools provides useful snippets to create much of the skeleton code required for K8s manifest yaml files. For example to easily create a deployment resource, simply create a new file in vs code, type deployment, and select the Kubernetes Deployment option as prompted. A basic configuration template for a deployment resource will be created.

Docker Multi-stage builds

Multi-stage builds allow rebuilding only what needs to be rebuilt, thereby making image creation faster.

Docker multi-stage builds allow the separation of the image build into separate stages. Each stage is only rebuilt as needed thereby saving time on the overall build. For example, your app image may be separated into stages to build a base image, download and install library dependencies, and compile source.

If you only change your application source code, you don’t want to download all your dependencies again.

An example multi-stage build Dockerfile is below:

FROM lukemathwalker/cargo-chef:latest-rust-1.56.0 AS chef

WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json


FROM chef AS builder 
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --recipe-path recipe.json
# Build application
COPY . .
RUN cargo build

# We do not need the Rust toolchain to run the binary!
FROM debian:bullseye-slim AS runtime
RUN apt-get update
RUN apt-get install -y libpq-dev
ENV DATABASE_URL=postgres://multichoice:postgres@multichoice-db/multix
COPY --chown=1000:3000 --from=builder /app/target/debug/actix-multichoice /usr/local/bin

USER 1000:3000

CMD [ "/usr/local/bin/actix-multichoice" ]

There are 3 container images being created here: chef, builder, and runtime.

Each image only gets rebuilt if the files it depends on are changed. Here I am using chef for rust docker builds. Chef is a tool to speed up docker multi-stage builds.

chef computes the recipe file, builder caches our dependencies and builds the binary, runtime is our runtime environment.

Summary

Ideas discussed to streamline developing locally with K8s:

k3d because it’s lightweight and can easily create and delete clusters
skaffold to handle the development lifecycle for your containers
Know your containers, they may provide ways to do initialization, such as the postgres container initialization script used here
initContainers to do application initialization
liveness probes and readiness probes to quickly verify your application is functioning correctly
VS Code Kubernetes extension commands to view logs, run bash on your containers etc
kustomize configMap generator to automate dependent container reloading
docker multi-stage builds to speed up build times

次世代システム研究室では、グループ全体のインテグレーションを支援してくれるアーキテクトを募集しています。インフラ設計、構築経験者の方、次世代システム研究室にご興味を持って頂ける方がいらっしゃいましたら、ぜひ募集職種一覧からご応募をお願いします。