Lima with k3s and nerdctl

こんにちは、次世代システム研究室のN.M.です。

 

Introduction

In modern software development, having a reliable local environment that can replicate production scenarios is essential. This ensures consistency and reduces unexpected deployment issues. In this blog, I’ll share my experience and steps in setting up a local development environment using three tools: Lima, k3s, and nerdctl.

Lima provides a straightforward way to run Linux virtual machines on macOS. When paired with k3s, a lightweight Kubernetes distribution, it allows developers to have a manageable Kubernetes cluster on their local machines. Adding nerdctl, a container tool compatible with Docker CLI, completes the setup, providing a comprehensive container management solution.

The aim of this article is to outline the process of integrating these tools, highlighting the benefits, challenges encountered, and solutions derived. This guide serves as a resource for developers looking to establish a robust and efficient local development setup using Lima, k3s, and nerdctl.

Lima

https://github.com/lima-vm/lima

– Is a CNCF Sandbox project: https://www.cncf.io/projects/lima/
– Seamless Integration: Easily run Linux virtual machines on macOS, essentially offering “Linux-on-Mac”.
– Lightweight: Consumes fewer resources than traditional virtualization solutions.
– Unified Environment: Promotes consistency between macOS and Linux deployments.

k3s

https://k3s.io/

– Simplicity: Offers a streamlined installation process and operation, removing many of the external dependencies of standard Kubernetes.
– Lightweight: Stripped-down version of Kubernetes, designed for edge and IoT use cases, making it resource-efficient.
– Versatile: Suitable for development, testing, and even production for specific use cases. Its lightweight nature makes it particularly useful for CI/CD scenarios.

nerdctl

https://github.com/containerd/nerdctl

– Docker Compatibility: Provides a command-line interface that’s compatible with Docker CLI, easing the transition for developers.
– Containerd Backend: Directly leverages containerd, a high-performance container runtime and the de-facto container standard.
– Feature-rich: Supports various functionalities like mounting volumes, port mappings, in addition to encrypted containers and p2p container distribution using IPFS.
In combination, Lima, k3s, and nerdctl offer developers a streamlined and resource-efficient environment to deploy, test, and manage containerized applications on their local machines, ensuring consistency with larger, production-ready environments.

Background Story

Lima is configured via yaml files, the lima repository contains many example configurations. Browsing the k3s.yaml configuration, I found the section below:

# containerd is managed by k3s, not by Lima, so the values are set to false here.
containerd:
system: false
user: false

So, rather than install k3s and nerdctl (which runs on top of containerd), and risk conflicting installations, I decided to be safe and utilized the existing lima configurations for 2 separate VMs, the default(https://github.com/lima-vm/lima/blob/master/examples/default.yaml) and the k3s.yaml mentioned.
These VMs would need to ability to connect to each other so images built and hosted on a private registry in the default VM can be pulled to the `k3s` VM.

 

Configuring a Lima Shared Network

Like many VMs Lima VM networks can be of 3 types:

• host – Direct Access: The VM directly uses the host’s network. There’s no network isolation between the VM and the host.
• bridge – The default network type: This network type provides a bridge that connects the VM’s network interface to a private virtual network on the host.
• shared – This network type is similar to the bridge network type, but it also allows VMs to communicate with each other.

All, lima VMs are configured with a yaml file, there are many ready-built examples here.

To enable shared networks the following snippet must be added to the VM yaml:

networks:
- lima: shared

Also, you may need to alter the mount location to something more suitable to you own environment. Note that the default.yaml has 2 mount points, a read only mount point to home “~” and a read-write mount point to “/tmp/lima”.

Install vmnet_socket

https://github.com/lima-vm/socket_vmnet

– Necessary for shared networking

Description from the github repository:

socket_vmnet provides vmnet.framework support for QEMU.

socket_vmnet does not require QEMU to run as the root user.

(But socket_vmnet itself has to run as the root, in most cases.)

$ brew install socket_vmnet

Configure lima networking

– configure vmnet path
– edit ~/.lima/_config/networks.yaml and add the path to the socket_vmnet binary file. Note that this must be a file, not a symbolic link otherwise Lima will produce an error.

paths:
socketVMNet: "/opt/homebrew/Cellar/socket_vmnet/1.1.2/bin/socket_vmnet"

– copy ~/projects/lima/examples/default.yaml to ~/projects/lima/examples/sharedcontainerd.yaml, this will be the basis for our nerdctl VM

• configure networks to shared as shown above

Now that Lima is configured, start the VMs:
– start the VMs

$ limactl start ~/projects/lima/examples/sharedcontainerd.yaml
$ limactl start ~/projects/lima/examples/k3s.yaml

 

Add avahi

– https://www.avahi.org/
– avahi adds mDNS support for machines on a local network and works for Lima shared networks.

$ sudo apt install avahi-daemon
$ sudo systemctl start avahi-daemon
$ sudo systemctl enable avahi-daemon

Now we can refer to lima-sharedcontainerd as lima-sharedcontainerd.local and lima-k3s as lima-k3s.local, instead of by IP and mDNS will resolve these for us.
The relevant part of /etc/nsswitch.conf is below:

hosts: files mdns4_minimal [NOTFOUND=return] dns

This means our $(hostname).local names will be resolved by mDNS.

Test connectivity

– at the sharedcontainerd VM

$ nc -l lima-sharedcontainerd.local 5000

– at the k3s VM

$ nc lima-sharedcontainerd.local 5000
hello

– then if successful we see the output at sharedcontainerd

hello

 

Configure the VMs /etc/hosts

Even after using avahi, nslookup and more importantly name resolution from inside k3s was not working, I was able to solve this by entering the VM names and addresses into `/etc/hosts`, not ideal, but it works for the time being.
So I created entries in each host to the other’s IP address and name. For example in lima-k3s /etc/hosts:

192.168.105.4 lima-sharedcontainerd.local

With this change I was able to access the local container registry running on lima-sharedcontainerd.local from lima-k3s.local the Kubernetes.
I found some notes that looked relevant in the [documentation](https://github.com/lima-vm/lima/blob/v0.12.0/docs/network.md)

DNS (192.168.5.3)

The DNS.

If useHostResolver in lima.yaml is true, then the hostagent is going to run a DNS server over tcp and udp – each on a separate randomly selected free port. This server does a local lookup using the native host resolver, so it will deal correctly with VPN configurations and split-DNS setups, as well as mDNS, local /etc/hosts etc. For this the hostagent has to be compiled with CGO_ENABLED=1 as default Go resolver is broken.

Setting up a k3s private local image registry

For k3s to use the local registry we need to add the following configuration on *lima-k3s*:

sudo cat /etc/rancher/k3s/registries.yaml
mirrors:
"lima-sharedcontainerd.local:5000":
endpoint:
- "http://lima-sharedcontainerd.local:5000"

 

Note the endpoint is plain HTTP, there are other options to configure for TSL or authentication: https://docs.k3s.io/installation/private-registry

Test k3s Sample Application

I used the rust sample server and container setup at https://qiita.com/fits/items/2596fd944025984bf881 for my test, which creates a directory structure like:

├── Cargo.toml
├── Dockerfile
├── sampleapp.yaml
└── src
     └── main.rs

– Cargo.toml rust project build spec
– Dockerfile container docker build file
– sampleapp.yaml the kubernetes manifest for the deployment
– src/main.rs the rust code

– build the container

$ nerdctl build -t lima-sharedcontainerd.local:5000/sampleapp .

– run a local docker registry

$ nerdctl run -d -p 5000:5000 --name registry registry:latest

– push the container to the registry

nerdctl push lima-sharedcontainerd.local:5000/sampleapp

– test the registry

$ nerdctl pull --insecure-registry lima-sharedcontainerd.local:5000/sampleapp:latest

Note that –insecure-registry flag which needs to be used for a plain HTTP registry

Pro-tip:

To run nerdctl on the host you can install it with brew install nerdctl and then

export LIMA_INSTANCE=sharedcontainerd

To run kubectl on the host you can use the k3s.yaml created when you started the k3s VM

export KUBECONFIG /Users/<USER>/.lima/k3s/copied-from-guest/kubeconfig.yaml

 

Testing the registry

$ curl -X GET http://lima-sharedcontainerd.local:5000/v2/_catalog

{"repositories":["sampleapp"]}

Test the service

$ curl http://localhost:8080/
ok:/%

Summary

We have seen how to configure lima with 2 VMs in shared network mode to enable mutual access. This provides a lightweight and promising local environment for Kubernetes development. Some enhancements could be made to Lima provisioning to further automate setup and eliminate the need for the /etc/hosts configuration, but the use of these tools provides a good platform for further enhancements going forward.

 

References

– lima presentation: https://www.youtube.com/watch?v=g5GCsbjkzRM&t=7s
– lima: https://github.com/lima-vm/lima
– k3s: https://k3s.io/
– nerdctl: https://github.com/containerd/nerdctl
– avahi: https://www.avahi.org/
– sampleapp: https://qiita.com/fits/items/2596fd944025984bf881

 

次世代システム研究室では、グループ全体のインテグレーションを支援してくれるアーキテクトを募集しています。インフラ設計、構築経験者の方、次世代システム研究室にご興味を持って頂ける方がいらっしゃいましたら、ぜひ募集職種一覧からご応募をお願いします。