Running K3s in LXC on Proxmox

Posted on 4 mins read

It has been a while since I’ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I’ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it’d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine’s resources.

What is K3s?#

K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.

Configure Proxmox#

This gist contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that bridge-nf-call-iptables should be loaded, but I did not understand the benefit of doing this.

Disable swap#

There is an issue on Kubernetes regarding swap here. There claims to be support for swap in 1.22, but for now let’s disable it:

sysctl vm.swappiness=0
swapoff -a

It might be worth experimenting with swap enabled in the future to see how that might affect performance.

Enable IP Forwarding#

To avoid IP Forwarding issues with Traefik, run the following on the host:

sudo sysctl net.ipv4.ip_forward=1
sudo sysctl net.ipv6.conf.all.forwarding=1
sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf
sudo sed -i 's/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g' /etc/sysctl.conf

Create LXC container#

Create an LXC container in the Proxmox interface as you normally would. Remember to:

  • Uncheck unprivileged container
  • Use a LXC template (I chose a debian 11 template downloaded with pveam)
  • In memory, set swap to 0
  • Create and start the container

Modify container config#

Now back on the host run pct list to determine what VMID it was given.

Open /etc/pve/lxc/$VMID.conf and append:

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"
lxc.cgroup2.devices.allow: c 10:200 rwm

All of the above configurations are described in the manpages. Notice that cgroup2 is used since Proxmox VE 7.0 has switched to a pure cgroupv2 environment.

Thankfully cgroup v2 support has been supported in k3s with these contributions:

Enable shared host mounts#

From within the container, run:

echo '#!/bin/sh -e
ln -s /dev/console /dev/kmsg
mount --make-rshared /' > /etc/rc.local
chmod +x /etc/rc.local
reboot

Install K3s#

One of the simplest ways to install K3s on a remote host is to use k3sup. Ensure that you supply a valid CONTAINER_IP and choose the k3s-version you prefer. As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases here.

ssh-copy-id root@$CONTAINER_IP
k3sup install --ip $CONTAINER_IP --user root --k3s-version v1.22.3+k3s1

If all goes well, you should see a path to the kubeconfig generated. I moved this into ~/.kube/config so that kubectl would read this by default.

Wrapping up#

Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton’s Getting Started guide and was able to deploy it in a few commands.

$ kubectl get all --namespace tekton-pipelines
NAME                                               READY   STATUS    RESTARTS      AGE
pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8      1/1     Running   1 (50m ago)   12h
pod/tekton-dashboard-6bf858f977-qt4hr              1/1     Running   1 (50m ago)   11h
pod/tekton-pipelines-controller-69fd7498d8-f57m4   1/1     Running   1 (50m ago)   12h

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                              AGE
service/tekton-pipelines-controller   ClusterIP   10.43.44.245    <none>        9090/TCP,8080/TCP                    12h
service/tekton-pipelines-webhook      ClusterIP   10.43.183.242   <none>        9090/TCP,8008/TCP,443/TCP,8080/TCP   12h
service/tekton-dashboard              ClusterIP   10.43.87.97     <none>        9097/TCP                             11h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tekton-pipelines-webhook      1/1     1            1           12h
deployment.apps/tekton-dashboard              1/1     1            1           11h
deployment.apps/tekton-pipelines-controller   1/1     1            1           12h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/tekton-pipelines-webhook-8566ff9b6b      1         1         1       12h
replicaset.apps/tekton-dashboard-6bf858f977              1         1         1       11h
replicaset.apps/tekton-pipelines-controller-69fd7498d8   1         1         1       12h

NAME                                                           REFERENCE                             TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   9%/100%   1         5         1          12h

I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.

If I’m feeling adventurous, I might experiment with K3s rootless.