Kubernetes - Deploy Services that use Longhorn

Kubernetes - Deploy Services that use Longhorn#

625 words | 8 min read

In the previous post, we installed Longhorn, and configured external access to the Longhorn UI.

Now, we will utilize the Longhorn in-cluster storage.

1. Define a Storage Class#

We need to define a storage class before we can define volumes for storage.

1.1. Define a Storage Class#

Create a configuration file (longhorn-rwxclass.yaml) for the storage class, as shown below.

Note

If you want to define this storage class as the default, set the annotation “storageclass.kubernetes.io/is-default-class” to “true”. There should only be one default storage class in a cluster.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-rwx
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  fsType: "ext4"
  nfsOptions: "vers=4.2,noresvport,softerr,timeo=600,retrans=5,rw,hard"
reclaimPolicy: Retain

1.2. Apply the configuration#

k apply -f longhorn-rwx.yaml

1.3. Check the Storage Classes in your cluster#

k get storageclass -A

2. Create a Persistent Volume Claim (PVC)#

2.1. Define a Persistent Volume Claim#

Create a configuration file (nginx-pvc.yaml) for the Persistent Volume Claim (PVC), as shown below:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-rwx
spec:
  accessModes:
    # - ReadWriteOnce
    - ReadWriteMany
  storageClassName: longhorn-rwx
  resources:
    requests:
      storage: 1Gi

2.2. Apply the configuration#

k apply -f nginx-pvc.yaml

2.3. Check the PVCs in your cluster#

k get pvc -A

Note

Longhorn will automatically create a PersistentVolume (PV) when you create a PVC based on a Storage Class

2.4. Check the PVs in your cluster#

k get pv -A

3. Deploy NGinx in the cluster#

3.1. Define an NGinX pod#

Create a configuration file (nginx-pod.yaml) for the NGinX Webserver Pod.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
    app: nginx
  name: nginx-webserver
  namespace: default
spec:
  containers:
  - name: nginx-server
    image: nginx:latest
    volumeMounts:
    - name: rwx-volume
      mountPath: /usr/share/nginx/html
      # mountPath: /data
  volumes:
  - name: rwx-volume
    persistentVolumeClaim:
      claimName: nginx-rwx

3.2. Apply The Configuration#

k apply -f nginx-pod.yaml

3.3 Define a Service for the NginX Pod#

Create a configuration file (nginx-svc.yaml) for the NGinX Service.

apiVersion: v1
kind: Service
metadata:
  labels:
    run: nginx
  name: nginx-svc
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
    app: nginx
  sessionAffinity: None
  type: ClusterIP

3.4. Apply The Configuration#

k apply -f nginx-svc.yaml

3.5. Define The Gateway for the NGinX Webserver#

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    port: 80
    name: nginx-web
    allowedRoutes:
      kinds:
        - kind: HTTPRoute
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: nginx-route
spec:
  parentRefs:
  - name: nginx-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    # - backendRefs:
      # - kind: Service
      - name: nginx-svc
        port: 80

3.6. Apply The Gateway Configuration#

k apply -f nginx-gateway.yaml

3.7. Obtain the IP address of the NGinX Gateway#

k get svc -A -o wide

3.8. Connect to the NGinX Gateway#

Open a browser and connect to the IP address of the NGinX Gateway (in my case, it is http://10.44.0.3).

You will get an error (403 - Forbidden) because there are no files for the Webserver to serve.

4. Add a file into the persistent storage of the Webserver pod#

4.1. Connect to the Nginx pod#

k exec -it nginx-webserver -- /bin/bash

This should take you to a prompt that looks like this:

root@nginx-webserver:/#

4.2. Create the index.html file for testing#

Type the following at the command prompt

cat - > /usr/share/nginx/html/index.html

Now, paste the following HTML (or another one of your choice) and press CTRL+D to save the file.

<!DOCTYPE html>
<html>

<head>
	<meta http-equiv="content-type" content="text/html; charset=utf-8" />
	<title>Test By Lytn1ng</title>
</head>

<body lang="en-US">
	<p>This is a test of NginX using Longhorn persistent in-cluster storage.</p>
</body>

</html>

4.3. Test the NGinX Webserver#

Open a browser and connect to http://10.44.0.3/. You should see the test HTML that we just created.

5. Test the persistence in Longhorn#

So far, so good. But what about the “persistent” part that we have been talking about for so long? How do we know that our data (i.e. the HTML file) will persist?

5.1. Destroy our deployed artifacts#

Destroy the Gateway, Service and pod that we deployed.

Important

Do NOT destroy the PV or the PVC that we created for NGinX in Longhorn. That’s where our data will be persisted even after the pod is destroyed.

k delete -f nginx-gateway.yaml

k delete -f nginx-svc.yaml

k delete -f nginx-pod.yaml

Now, our NGinX pod, Service and gateway have all been destroyed. You can check and verify with kubectl commands that our pod, Service and Gateway (Loadbalancer) are no longer running.

But has our data been persisted? Let’s find out.

5.2. Re-deploy the artifacts#

k apply -f nginx-pod.yaml

k apply -f nginx-svc.yaml

k apply -f nginx-gateway.yaml

5.3. Check the IP address of the re-deployed Gateway#

k get svc -A -o wide

5.4. Connect to the Gateway#

Open a browser and connect to the Gateway External-IP (http://10.44.0.3/).

You should see the HTML file that you created in Step 4.2 above.

So, although the pod running NGinX was destroyed and recreated, the index.html that was stored in the Longhorn PV survived, and was available to our newly created NGinX pod.

Conclusion#

This concludes the series of posts on

  • Installing Kubernetes on bare metal

  • Configuring BGP networking for Kubernetes

  • Configuring Cilium as the CNI and the Load Balancer using the Gateway API

  • Enabling Hubble Observability

  • Installing Longhorn

  • Enabling Access using the Gateway API

  • Creating a PersistentVolumeClaim (PVC)

  • Deploying an NginX pod, Service and Gateway

  • Deploying custom HTML in the NGinX pod

  • Destroying and Re-creating the NginX pod, Service and Gateway

  • Verifying that our custom data persisted the destruction/recreation.

I have tested these steps multiple times (mostly because something didn’t work, and I wanted to start over after fixing whatever mistake).

I hope you will benefit from this series on Kubernetes to set up your own cluster, and further your own knowledge and interest in this topic.

Kubernetes - Install Longhorn (Persistent In-Cluster Storage)  

Comments

Comments powered by giscus, use a GitHub account to comment.