Terminology Confusion in Kubernetes: StorageClass, PersistentVolume, PersistentVolumeClaim

There are several terms associated with storing persistent data in Kubernetes: StorageClass, PersistentVolume, PersistentVolumeClaim…

Photo by Patrick Lindenberg on Unsplash

StorageClass

Imagine that as a system administrator, you want to offer multiple storage types to the developers: AWS EBS, Ceph, Cinder, etc.

You must install the driver for every storage type you want to use in your Kubernetes cluster.

After that, you must inform your cluster, “Hey, you may utilize this storage with this driver.”. To be able to inform your cluster, you must use Kubernetes’ StorageClass component.

The StorageClass component may have been called “Add a Storage Type to the Cluster”

Here’s an example of how to add AWS EBS storage to a Kubernetes cluster:

01-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-storageclass

provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

Before adding the EBS storage to the cluster, you must install the EBS Storage’s drivers on the cluster:

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

More details are here

Amazon EBS storage is now recognized by your cluster. The cluster will match the provisioner in the YAML file and the driver you installed.

There are available two binding modes: Immediate (default binding mode) and WaitForFirstConsumer

WaitForFirstConsumer binding mode is available for cloud-based storage (AWS, GCP, Azure).

WaitForFirstConsumer mode: A Persistent Volume will be provisioned when a Pod attempts to use the Persistent Volume Claim.

Immediate mode: A Persistent Volume will be provisioned when a Persistent Volume Claim is created.

Persistent Volume? Persistent Volume Claim? What are those?

Imagine that you have storage with 15 TB of space. You will need to run 3 different projects on the storage. You can split your storage into 3 different persistent volumes, and each project can have 5 TB of disk space at most.

The developer of each project can use 5 TB of disk space for their DTAP environments. They will have to use the Persistent Volume Claim component to ask for the disk space they need. This is static provisioning.

A lot of clusters on the cloud tend to use dynamic provisioning, as it is a flexible way to provide sufficient disk space.

For dynamic provisioning, you don’t need to create a Persistent Volume manually. The developer of each project can request disk space via the Persistent Volume Claim component. Kubernetes will create a persistent volume for you automatically. They must define the storage class they want to use.

(Note: Before creating this persistent volume claim, I created the storage class with Immediate binding mode.)

02-persistent-volume-claim.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-database-pv-claim
spec:
  storageClassName: ebs-storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 33Gi

Let’s test it:

➜  ~ kubectl create -f 02-persistent-volume-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created

➜  ~ kubectl get pvc
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
ebs-database-pv-claim   Bound    pvc-73fb8137-6b9d-432a-8db4-5a907981314a   33Gi       RWO            ebs-storageclass   5s
➜  ~ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS       REASON   AGE
pvc-73fb8137-6b9d-432a-8db4-5a907981314a   33Gi       RWO            Delete           Bound    default/ebs-database-pv-claim   ebs-storageclass            28s

EBS is created by Kubernetes:

Kubernetes created an EBS disk and a persistent volume for us. After that, the persistent volume claim component was created and linked to the persistent volume.

How to use Persistent Volume and Persistent Volume Claim components together?

Since the persistent volume needs static provisioning, you will have to create an EBS disk by hand before using the persistent volume:

aws ec2 create-volume --size 15 --availability-zone eu-west-1b

(Note: The EBS disk must be in the same availability zone as your Kubernetes node)

Take note of the VolumeId in the output.

03-persistent-volume.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: ebs-database-pv
spec:
  storageClassName: ebs-storageclass
  capacity:
    storage: 41Gi
  accessModes:
    - ReadWriteOnce
  csi:
    driver: ebs.csi.aws.com
    fsType: ext4
    volumeHandle: "EBS_VOLUME_ID"

Create a persistent volume:

➜  ~ kubectl create -f 03-persistent-volume.yml
persistentvolume/ebs-database-pv created
➜  ~ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS       REASON   AGE
ebs-database-pv   41Gi       RWO            Retain           Available           ebs-storageclass            7s

You can create a persistent volume claim in this persistent volume:

04-persistent-volume-claim.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-database-pv-claim
spec:
  storageClassName: ebs-storageclass
  volumeName: ebs-database-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 33Gi

Create a persistent volume claim and check it:

➜  ~ kubectl create -f 04-persistent-volume-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created
➜  ~ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS       REASON   AGE
ebs-database-pv   41Gi       RWO            Retain           Bound    default/ebs-database-pv-claim   ebs-storageclass            110s
➜  ~ kubectl get pvc
NAME                    STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS       AGE
ebs-database-pv-claim   Bound    ebs-database-pv   41Gi       RWO            ebs-storageclass   22s

Please notice that we requested a smaller size than what was available in the persistent volume and received the same amount of available size in the persistent volume.

Try to attach multiple volume claims to one persistent volume

Let’s try to create another volume claim on the same persistent volume:

05-persistent-volume-claim-another.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-database-pv-claim-another
spec:
  storageClassName: ebs-storageclass
  volumeName: ebs-database-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

➜  ~ kubectl create -f 05-persistent-volume-claim-another.yml
persistentvolumeclaim/ebs-database-pv-claim-another created
➜  ~ kubectl get pvc
NAME                            STATUS    VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS       AGE
ebs-database-pv-claim           Bound     ebs-database-pv   41Gi       RWO            ebs-storageclass   4m15s
ebs-database-pv-claim-another   Pending   ebs-database-pv   0                         ebs-storageclass   4s

ebs-database-pv-claim-another is in the “Pending” status. Let’s see why:

➜  blog kubectl describe pvc ebs-database-pv-claim-another
Name:          ebs-database-pv-claim-another
Namespace:     default
StorageClass:  ebs-storageclass
...
Events:
  Type     Reason         Age               From                         Message
  ----     ------         ----              ----                         -------
  Warning  FailedBinding  3s (x7 over 82s)  persistentvolume-controller  volume "ebs-database-pv" already bound to a different claim.

A persistent volume and a persistent volume claim have a one-to-one relationship. If you need another persistent volume claim, you will require another persistent volume.

Request a larger disk space than what is available

Let’s try to create a persistent volume claim (100 Gib) that has a larger size than what is available in the persistent volume (41 Gib).

06-persistent-storage-claim.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-database-pv-claim
spec:
  storageClassName: ebs-storageclass
  volumeName: ebs-database-pv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

(Note: Before executing this yaml file, I deleted the persistent volume.)

Create it:

➜  ~ kubectl create -f 03-persistent-volume.yml
persistentvolume/ebs-database-pv created
➜  ~ kubectl create -f 06-persistent-storage-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created
➜  ~ kubectl describe pvc
Name:          ebs-database-pv-claim
Namespace:     default
StorageClass:  ebs-storageclass
...
Events:
  Type     Reason          Age               From                         Message
  ----     ------          ----              ----                         -------
  Warning  VolumeMismatch  0s (x2 over 11s)  persistentvolume-controller  Cannot bind to requested volume "ebs-database-pv": requested PV is too smallKubernetes didn’t allow me to use a larger size than what is available in the persistent volume.