Data Volume IO Throttling¶

In HwameiStor, it allows users to specify the maximum IOPS and throughput for volumes on a Kubernetes cluster.

Please follow the steps below to create a volume with maximum IOPS and throughput and create a workload to use it.

Create a New StorageClass with Maximum IOPS and Throughput Parameters¶

By default, HwameiStor does not automatically create such a StorageClass during installation, so you need to create the StorageClass manually.

An example StorageClass is as follows:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hwameistor-storage-lvm-hdd-sample
parameters:
  convertible: "false"
  csi.storage.k8s.io/fstype: xfs
  poolClass: HDD
  poolType: REGULAR
  provision-iops-on-creation: "100"
  provision-throughput-on-creation: 1Mi
  replicaNumber: "1"
  striped: "true"
  volumeKind: LVM
provisioner: lvm.hwameistor.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Compared to the regular StorageClass created by the HwameiStor installer, the following parameters have been added:

provision-iops-on-creation: Specifies the maximum IOPS for the volume at creation time.
provision-throughput-on-creation: Specifies the maximum throughput for the volume at creation time.

After creating the StorageClass, you can use it to create PVC (PersistentVolumeClaim).

Create PVC using StorageClass¶

An example PVC is as follows:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-sample
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: hwameistor-storage-lvm-hdd-sample

After creating the PVC, you can create a Deployment to use the PVC.

Create a Deployment with PVC¶

An example Deployment is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: pod-sample
  name: pod-sample
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-sample
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: pod-sample
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: pvc-sample
      containers:
      - command:
        - sleep
        - "100000"
        image: busybox
        name: busybox
        resources: {}
        volumeMounts:
        - name: data
          mountPath: /data
status: {}

After creating the Deployment, you can use the following command to test the IOPS and throughput of the volume:

kubectl exec -it pod-sample-5f5f8f6f6f-5q4q5 -- /bin/sh
dd if=/dev/zero of=/data/test bs=4k count=1000000 oflag=direct

Note: Due to cgroupv1 limitations, the settings for maximum IOPS and throughput may not take effect on non-direct IO.

How to Change the Maximum IOPS and Throughput of a Data Volume¶

The maximum IOPS and throughput are specified in the parameters of the StorageClass, and you cannot directly change them as they are now immutable.

Unlike other storage vendors, HwameiStor is a Kubernetes-based storage solution that defines a set of operational primitives based on Kubernetes CRDs. This means you can modify the relevant CRD to change the actual maximum IOPS and throughput of a volume.

The following steps show how to change the maximum IOPS and throughput of a data volume.

Find the LocalVolume CR for the Specified PVC¶

$ kubectl get pvc pvc-sample

NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        AGE
demo             Bound     pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10   10Gi       RWO            hwameistor-storage-lvm-hdd          5d23h
pvc-sample       Bound     pvc-cac82087-6f6c-493a-afcd-09480de712ed   10Gi       RWO            hwameistor-storage-lvm-hdd-sample   5d23h

$ kubectl get localvolume

NAME                                       POOL                   REPLICAS   CAPACITY      USED       STATE   RESOURCE   PUBLISHED   FSTYPE   AGE
pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10   LocalStorage_PoolHDD   1          10737418240   33783808   Ready   -1         master      xfs      5d23h
pvc-cac82087-6f6c-493a-afcd-09480de712ed   LocalStorage_PoolHDD   1          10737418240   33783808   Ready   -1         master      xfs      5d23h

According to the printed output, the LocalVolume CR for the PVC is pvc-cac82087-6f6c-493a-afcd-09480de712ed.

Modify LocalVolume CR¶

kubectl edit localvolume pvc-cac82087-6f6c-493a-afcd-09480de712ed

In the editor, locate the spec.volumeQoS section and modify the iops and throughput fields. Note that a blank value indicates no limit.

Finally, save the changes and exit the editor. The settings will take effect within a few seconds.

Note: In the future, once Kubernetes supports it (link), we will allow users to directly modify the maximum IOPS and throughput of volumes.

How to Check the Actual IOPS and Throughput of a Data Volume¶

HwameiStor uses cgroupv1 to limit the IOPS and throughput of data volumes. Therefore, you can use the following command to check the actual IOPS and throughput of a data volume.

$ lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0   160G  0 disk
├─sda1            8:1    0     1G  0 part /boot
└─sda2            8:2    0   159G  0 part
  ├─centos-root 253:0    0   300G  0 lvm  /
  ├─centos-swap 253:1    0   7.9G  0 lvm
  └─centos-home 253:2    0 101.1G  0 lvm  /home
sdb               8:16   0   100G  0 disk
├─LocalStorage_PoolHDD-pvc--cac82087--6f6c--493a--afcd--09480de712ed
                253:3    0    10G  0 lvm  /var/lib/kubelet/pods/3d6bc980-68ae-4a65-a1c8-8b410b7d240f/v
└─LocalStorage_PoolHDD-pvc--c354a56a--5cf4--4ff6--9472--4e24c7371e10
                253:4    0    10G  0 lvm  /var/lib/kubelet/pods/521fd7b4-3bef-415b-8720-09225f93f231/v
sdc               8:32   0   300G  0 disk
└─sdc1            8:33   0   300G  0 part
  └─centos-root 253:0    0   300G  0 lvm  /
sr0              11:0    1   973M  0 rom

$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_iops_device
253:3 100

$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_iops_device
253:3 100

$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
253:3 1048576

$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
253:3 1048576