Kubernetes - Install Longhorn (Persistent In-Cluster Storage)#
392 words | 5 min read
In the previous post, we completed a working K3S cluster configuration by setting up networking for our cluster, enabling Hubble obervability, enabling the Gateway API and also verifying that everything works by connecting to the Hubble UI from outside the cluster.
Now, we will install and configure Longhorn in our K3S Cluster.
We should already have the longhorn partition mounted at /var/lib/longhorn.
1. Configure the master node to be a Longhorn storage node#
Note
You only need to do these steps if you want to use the master node as a storage node.
1.1. Check the node taints on the master node#
When we installed k3s, we tainted the master node to prevent workloads from running on it - which is the recommended practice.
We can check the node taint with the following command.
k get nodes -o json | jq '.items[].spec.taints'
It should show the following taints on the master node:
1.2. Remove the node taints on the master node#
In my case, I want to be able to use the disk space on all nodes for storage. So, we will remove the taint on the master node.
k taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-
k taint nodes $(hostname) node-role.kubernetes.io/control-plane:NoSchedule-
2. Install Longhorn prerequisites#
2.1. Install the prerequisite packages for Longhorn on all nodes#
dnf install iscsi-initiator-utils jq which nfs-utils cryptsetup
2.2. Load required kernel modules#
modprobe iscsi_tcp dm_crypt
2.3. Configure the kernel modules to be loaded at boot#
echo "iscsi_tcp" > /etc/modules-load.d/k3s_longhorn.conf
echo "dm_crypt" >> /etc/modules-load.d/k3s_longhorn.conf
3. Verify that the environment is ready for installing Longhorn#
Note
To be safe, you should download the script and examine it before running it (instead of piping it directly to bash)
curl https://raw.githubusercontent.com/longhorn/longhorn/v1.6.x/scripts/environment_check.sh | bash
Wait for the script to finish and make sure there are no errors.
4. Install Longhorn#
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.x/deploy/longhorn.yaml
Wait for the pods in the “longhorn-system” namespace to be in running state.
Note
You can use “kubectl get pods -A -o wide” to check the status of the pods.
5. Configure the gateway for access to the Longhorn UI#
5.1. Create the Gateway API configuration YAML (longhorn-gateway.yaml)#
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: longhorn-gateway
namespace: longhorn-system
spec:
gatewayClassName: cilium
listeners:
- protocol: HTTP
port: 80
name: web-lh
allowedRoutes:
namespaces:
from: Same
# hostname: longhorn.iyers.org
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: longhorn-route
namespace: longhorn-system
spec:
parentRefs:
- name: longhorn-gateway
namespace: longhorn-system
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: longhorn-frontend
port: 80
5.2. Apply the Gateway configuration#
k apply -f longhorn-gateway.yaml
6. Connect to the Longhorn UI#
6.1. Get the IP address of the Longhorn Gateway#
k get svc -A -o wide
The output should resemble what is shown below
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 3h38m <none>
kube-system cilium-gateway-hubble-gateway LoadBalancer 10.43.249.181 10.44.0.1 80:31626/TCP 3h27m <none>
kube-system hubble-peer ClusterIP 10.43.116.230 <none> 443/TCP 3h33m k8s-app=cilium
kube-system hubble-relay ClusterIP 10.43.187.77 <none> 80/TCP 3h28m k8s-app=hubble-relay
kube-system hubble-ui ClusterIP 10.43.128.178 <none> 80/TCP 3h28m k8s-app=hubble-ui
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h38m k8s-app=kube-dns
longhorn-system cilium-gateway-longhorn-gateway LoadBalancer 10.43.206.88 10.44.0.2 80:31729/TCP 11s <none>
longhorn-system longhorn-admission-webhook ClusterIP 10.43.147.233 <none> 9502/TCP 8m57s app=longhorn-manager
longhorn-system longhorn-backend ClusterIP 10.43.218.137 <none> 9500/TCP 8m58s app=longhorn-manager
longhorn-system longhorn-conversion-webhook ClusterIP 10.43.105.179 <none> 9501/TCP 8m58s app=longhorn-manager
longhorn-system longhorn-frontend ClusterIP 10.43.24.95 <none> 80/TCP 8m58s app=longhorn-ui
longhorn-system longhorn-recovery-backend ClusterIP 10.43.215.212 <none> 9503/TCP 8m57s app=longhorn-manager
Note the External-IP of the Longhorn Gateway (10.44.0.2 in the example shown above).
6.2. Connect to the Longhorn Gateway#
Open a browser and connect to http://10.44.0.2. This should open the Longhorn UI where you can check the disk space allocations, as well as view and make changes to the settings.
At this point, you have a working Kubernetes Cluster on Bare metal, with BGP networking, Load balancing and in-cluster storage.
Next, we will install some services that utilize the Longhorn storage.
Comments
Comments powered by giscus, use a GitHub account to comment.