跳转至

Rook-ceph 存储方案

DCE 5.0 支持众多第三方存储方案,我们针对 Rook-ceph 进行了详细的测试,并最终将其作为 Addon 集成了应用商店中。 以下是对 Rook-ceph 的详细调研和测评。

有关应用商店 Addon 的图形化界面安装、部署、卸载等操作说明,将于稍后提供。

简介

Rook 是一个开源云原生存储编排器,为各种存储解决方案(存储提供商)提供平台、框架和支持并与云原生环境原生集成。

Rook 支持多个存储提供商。它将分布式存储软件转变为自我管理、自我扩展和自我修复的存储服务。 它通过自动执行存储提供商的部署、引导、配置、预配、扩展、升级、迁移、灾难恢复、监视和资源管理来实现此目的。 当与 Kubernetes 一起使用时,Rook 使用 Kubernetes 调度和编排平台提供的功能,为调度、生命周期管理、资源管理、安全性、监控和用户体验提供无缝体验。

  • Rook-ceph 架构

    rook-ceph 架构

  • Rook 管理

    rook 管理

  • CSI 存储资源调配

    rook 资源调配

  • Ceph 数据路径

    rook 数据路径

  • 分布式架构

    Ceph 是一种分布式、可扩展的开源存储解决方案,适用于块、对象和共享文件系统存储。 Ceph 在过去几年中已经发展成为开源分布式存储解决方案的标准,在大中型企业中进行了多年的生产部署。

    rook 分布式架构

  • 组件:Ceph 存储池、归置组(PG)、对象、OSD

    rook 组件

  • Ceph RBD 工作原理

    rook 原理

测试验证

准备环境

本次测试使用 4 个虚拟机节点来部署一个 Kubernetes 集群:

  • 1 个 master 节点
  • 3 个 worker 节点

其中 kubelet 版本为 1.23.6

[root@k8s-10-6-162-21 ~]# kubectl get no
NAME                            STATUS           ROLES                              AGE         VERSION
k8s-10-6-162-21                 Ready            control-plane,master               19d         v1.23.6
k8s-10-6-162-22                 Ready            <none>                             19d         v1.23.6
k8s-10-6-162-23                 Ready            <none>                             19d         v1.23.6
k8s-10-6-162-24                 Ready            <none>                             13d         v1.23.6

部署 Rook-Ceph

克隆 GitHub 源码

[root@k8s-10-6-162-21 ~]# git clone https://github.com/rook/rook.git

[root@k8s-10-6-162-21 ~]# ls
anaconda-ks.cfg calico.yaml rook rook-ceph-image rook-ceph-image.zip

[root@k8s-10-6-162-21 rook]# git branch
master
* remotes/origin/release-1.7

[root@k8s-10-6-162-21 ceph]# cd /root/rook/cluster/examples/kubernetes/ceph

[root@k8s-10-6-162-21 ceph]# ls
ceph-client.yaml create-external-cluster-resources.py flex object-openshift.yaml rgw-external.yaml
cluster-external-management.yaml create-external-cluster-resources.sh flex-migrator.yaml object-test.yaml scc.yaml
cluster-external.yaml csi images.txt object-user.yaml storageclass-bucket-delete.yaml
cluster-on-local-pvc.yaml dashboard-external-https.yaml import-external-cluster.sh object.yaml storageclass-bucket-retain.yaml
cluster-on-pvc.yaml dashboard-external-http.yaml monitoring operator-openshift.yaml test-data
cluster-stretched-aws.yaml dashboard-ingress-https.yaml nfs-test.yaml operator.yaml toolbox-job.yaml
cluster-stretched.yaml dashboard-loadbalancer.yaml nfs.yaml osd-purge.yaml toolbox.yaml
cluster-test.yaml direct-mount.yaml object-bucket-claim-delete.yaml pool-ec.yaml volume-replication-class.yaml
cluster.yaml filesystembak.yaml object-bucket-claim-retain.yaml pool-mirrored.yaml volume-replication.yaml
common-external.yaml filesystem-ec.yaml object-ec.yaml pool-test.yaml
common-second-cluster.yaml filesystem-mirror.yaml object-external.yaml pool.yaml
common.yaml filesystem-test.yaml object-multisite-pull-realm.yaml pre-k8s-1.16
crds.yaml filesystem.yaml object-multisite.yaml rbdmirror.yaml

部署 Operator、CRD 和参数

[root@k8s-10-6-162-21 ceph]# kubectl create -f crds.yaml -f common.yaml -f operator.yaml

[root@k8s-10-6-162-21 ceph]# kubectl get crd | grep ceph
cephblockpools.ceph.rook.io                           2022-10-14T02:22:34Z
cephclients.ceph.rook.io                              2022-10-14T02:22:34Z
cephclusters.ceph.rook.io                             2022-10-14T02:22:35Z
cephfilesystemmirrors.ceph.rook.io                    2022-10-14T02:22:35Z
cephfilesystems.ceph.rook.io                          2022-10-14T02:22:35Z
cephnfses.ceph.rook.io                                2022-10-14T02:22:35Z
cephobjectrealms.ceph.rook.io                         2022-10-14T02:22:35Z
cephobjectstores.ceph.rook.io                         2022-10-14T02:22:35Z
cephobjectstoreusers.ceph.rook.io                     2022-10-14T02:22:35Z
cephobjectzonegroups.ceph.rook.io                     2022-10-14T02:22:35Z
cephobjectzones.ceph.rook.io                          2022-10-14T02:22:35Z
cephrbdmirrors.ceph.rook.io                           2022-10-14T02:22:35Z
objectbucketclaims.objectbucket.io                    2022-10-14T02:22:35Z
objectbuckets.objectbucket.io                         2022-10-14T02:22:35Z
volumes.rook.io                                       2022-10-14T02:22:35Z

部署 cluster 和 toolbox

[root@k8s-10-6-162-21 ceph]# kubectl create -f cluster.yaml

[root@k8s-10-6-162-21 ceph]# kubectl create -f toolbox.yaml

[root@k8s-10-6-162-21 ceph]# kubectl get po -n rook-ceph -owide
NAME                  READY       STATUS      RESTARTS        AGE IP         NODE              NOMINATED NODE      READINESS       GATES
csi-cephfsplugin-bmbtc   3/3      Running     6 (4d1h ago)    7d7h           10.6.162.24       k8s-10-6-162-24     <none>           <none>
csi-cephfsplugin-provisioner-5c8b6d6f4-hvfg6  6/6   Running   678 (38m ago)  7d7h              10.244.56.106       k8s-10-6-162-23  <none> <none>
csi-cephfsplugin-provisioner-5c8b6d6f4-w6snr  6/6   Running   545 (38m ago)  4d1h              10.244.169.249      k8s-10-6-162-22  <none> <none>
csi-cephfsplugin-r87qt   3/3      Running     6 (4d1h ago)    7d7h           10.6.162.23       k8s-10-6-162-23          <none>      <none>
csi-cephfsplugin-xqrmr   3/3      Running     6 (4d1h ago)    7d7h           10.6.162.22       k8s-10-6-162-22          <none>      <none>
csi-rbdplugin-mcpm6      3/3      Running     6 (4d1h ago)    7d7h           10.6.162.22       k8s-10-6-162-22          <none>      <none>
csi-rbdplugin-pj4cc      3/3      Running     6 (4d2h ago)    7d7h           10.6.162.23       k8s-10-6-162-23          <none>      <none>
csi-rbdplugin-provisioner-8564cfd44-mnjr8 6/6 Running 665 (38m ago) 4d8h     10.244.56.102     8s-10-6-162-23           <none>      <none>
csi-rbdplugin-provisioner-8564cfd44-tgzc8 6/6 Running 545 (37m ago) 4d1h     10.244.169.248    k8s-10-6-162-22          <none>      <none>
csi-rbdplugin-tnpbc      3/3      Running     6 (4d1h ago)     7d7h          10.6.162.24       k8s-10-6-162-24          <none>      <none>
rook-ceph-crashcollector-k8s-10-6-162-22-5c4444c7bb-pd6tf   1/1 Running 0 27h 10.244.169.208   k8s-10-6-162-22          <none>      <none>
rook-ceph-crashcollector-k8s-10-6-162-23-775d89c75-kgnzq    1/1 Running 2 (4d2h ago) 7d7h 10.244.56.105k8s-10-6-162-23  <none>      <none>
rook-ceph-crashcollector-k8s-10-6-162-24-6cc87876c6-tk6q2   1/1 Running 0 27h 10.244.49.18     k8s-10-6-162-24          <none>      <none>
rook-ceph-mds-myfs-a-94f74bd56-k2gt7 1/1 Running 0             27h           10.244.49.22      k8s-10-6-162-24          <none>      <none>
rook-ceph-mds-myfs-b-5679797bbd-jr8gt 1/1 Running 69 (38m ago) 27h           10.244.169.209    k8s-10-6-162-22          <none>      <none>
rook-ceph-mgr-a-5b69d56b4d-6z87x     1/1 Running 0             2d7h          10.244.49.9       k8s-10-6-162-24          <none>      <none>
rook-ceph-mon-a-7589d48f7d-lgps2     1/1 Running  25 (3h44m ago) 21h         10.244.56.122     k8s-10-6-162-23          <none>      <none>
rook-ceph-mon-b-6577cdc487-fk564     1/1 Running 161 (3h14m ago) 7d7h        10.244.169.244    k8s-10-6-162-22          <none>      <none>
rook-ceph-mon-d-66bdbd9dc-pn6wc      1/1 Running 83 (38m ago)   4d           10.244.49.61      k8s-10-6-162-24          <none>      <none>
rook-ceph-operator-866c96498c-mm2xd  1/1 Running 4 (4d1h ago)   7d7h         10.244.169.242    k8s-10-6-162-22          <none>      <none>
rook-ceph-osd-0-6d774dcf99-6qggj     1/1 Running 167 (38m ago)  7d3h         10.244.169.245    k8s-10-6-162-22          <none>      <none>
rook-ceph-osd-1-7f7c599677-qx4sh     1/1 Running 26 (4d2h ago)  7d3h         10.244.56.101     k8s-10-6-162-23          <none>      <none>
rook-ceph-osd-2-7564c58d96-bknjf     1/1 Running 125 (35m ago)  7d3h         10.244.169.243    k8s-10-6-162-22          <none>      <none>
rook-ceph-osd-3-5599b7bdcb-wmdkx     1/1 Running 0              4d1h         10.244.49.51      k8s-10-6-162-24          <none>      <none>
rook-ceph-osd-4-7c6d79cdff-4fzdc     1/1 Running 0              4d1h         10.244.49.55      k8s-10-6-162-24          <none>      <none>
rook-ceph-osd-5-ffb7b5ff7-phssl      1/1 Running 26 (4d2h ago)  7d3h         10.244.56.103     k8s-10-6-162-23          <none>      <none>
rook-ceph-osd-prepare-k8s-10-6-162-22-kjwzt 0/1 Completed 0     5h34m        10.244.169.222    k8s-10-6-162-22          <none>      <none>
rook-ceph-osd-prepare-k8s-10-6-162-23-fngbr 0/1 Completed 0     5h34m        10.244.56.67      k8s-10-6-162-23          <none>      <none>
rook-ceph-osd-prepare-k8s-10-6-162-24-p7c28 0/1 Completed 0     5h34m        10.244.49.27      k8s-10-6-162-24          <none>      <none>
rook-ceph-tools-55f548895f-8pdhm     1/1 Running          0     4d1h         10.244.56.107     k8s-10-6-162-23          <none>      <none>

部署 dashboard

[root@k8s-10-6-162-21 ~]# kubectl apply -f dashboard-external-https.yaml

[root@k8s-10-6-162-21 ~]# kubectl get svc -n rook-ceph
NAME                                           TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics                       ClusterIP 10.111.92.191 <none> 8080/TCP,8081/TCP 8d
csi-rbdplugin-metrics                          ClusterIP 10.106.136.163 <none> 8080/TCP,8081/TCP 8d
rook-ceph-mgr                                  ClusterIP 10.97.170.29 <none> 9283/TCP 8d
rook-ceph-mgr-dashboard                        ClusterIP 10.103.18.81 <none> 8443/TCP 8d
rook-ceph-mgr-dashboard-external-https         NodePort 10.111.121.36 <none> 8443:30738/TCP 8d
rook-ceph-mon-a                                ClusterIP 10.107.216.105 <none> 6789/TCP,3300/TCP 8d
rook-ceph-mon-b                                ClusterIP 10.100.168.72 <none> 6789/TCP,3300/TCP 8d
rook-ceph-mon-d                                ClusterIP 10.101.56.41 <none> 6789/TCP,3300/TCP 5d16h

dashboard-1

dashboard-2

dashboard-3

操作 Ceph 存储管理所用的 Rook 工具

[root@k8s-10-6-162-21 ceph]# kubectl exec -it rook-ceph-tools-55f548895f-fzbq2 -n rook-ceph -- bash

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph -s
cluster:
id: d65486e5-a8c3-4ca5-8ff6-9daddfc0f6c4
health: HEALTH_OK

services:
mon: 3 daemons, quorum a,b,d (age 71m)
mgr: a(active, since 30h)
mds: 1/1 daemons up, 1 hot standby
osd: 6 osds: 6 up (since 68m), 6 in (since 7d)

data:
volumes: 1/1 healthy
pools: 4 pools, 480 pgs
objects: 464 objects, 1009 MiB
usage: 3.8 GiB used, 596 GiB / 600 GiB avail
pgs: 480 active+clean

io:
client: 1.5 KiB/s rd, 2 op/s rd, 0 op/s wr

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph mgr services
{
"dashboard": "https://10.244.49.9:8443/",
"prometheus": "http://10.244.49.9:9283/"
}

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 600 GiB 596 GiB 3.8 GiB 3.8 GiB 0.63
TOTAL 600 GiB 596 GiB 3.8 GiB 3.8 GiB 0.63

--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 1 64 0 B 0 0 B 0 189 GiB
replicapool 4 128 880 MiB 290 2.6 GiB 0.45 189 GiB
myfs-metadata 7 256 1.5 MiB 25 3.0 MiB 0 283 GiB
myfs-data0 8 32 115 MiB 149 231 MiB 0.04 283 GiB

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 k8s-10-6-162-22 524M 99.4G 0 0 0 89 exists,up
1 k8s-10-6-162-23 677M 99.3G 0 0 0 0 exists,up
2 k8s-10-6-162-22 501M 99.5G 0 0 0 0 exists,up
3 k8s-10-6-162-24 694M 99.3G 0 0 1 15 exists,up
4 k8s-10-6-162-24 744M 99.2G 0 0 0 0 exists,up
5 k8s-10-6-162-23 747M 99.2G 0 0 0 0 exists,up

Ceph 存储配置

创建 RBD 池和归置组 (PG):

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph osd pool create replicapool 128

[root@k8s-10-6-162-21 kubernetes]# ceph osd pool application enable replicapool rbd

Creating storage class:

[root@k8s-10-6-162-21 rbd]# pwd
/root/rook/deploy/examples/csi/rbd
[root@k8s-10-6-162-21 rbd]# kubectl apply -f storageclass.yaml

[root@k8s-10-6-162-21 ~]# kubectl get sc
NAME                       PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block            rook-ceph.rbd.csi.ceph.com Delete Immediate true 5d20h
rook-cephfs                rook-ceph.cephfs.csi.ceph.com Delete Immediate true 2d17h

应用部署验证

在 RBD 上部署 Mysql 和 WordPress

[root@k8s-10-6-162-21 examples]# cd kubernetes/
ceph mysql2-cephfs.yaml mysql.yaml README.md wordpress.yaml
[root@k8s-10-6-162-21 kubernetes]# pwd
/root/rook/cluster/examples/kubernetes
[root@k8s-10-6-162-21 kubernetes]# kubectl apply -f mysql.yaml
[root@k8s-10-6-162-21 kubernetes]# kubectl apply -f wordpress.yaml

在 Cephfs 上部署 Mysql

[root@k8s-10-6-162-21 kubernetes]# kubectl apply -f mysql2-cephfs.yaml

[root@k8s-10-6-162-21 kubernetes]# kubectl get po
NAME                                                                   READY STATUS RESTARTS AGE
nginx-85b98978db-gbjtb                             1/1 Running 0 6d18h
nginx-85b98978db-rbprt                             1/1 Running 0 6d18h
nginx-85b98978db-w5m24                             1/1 Running 0 6d18h
wordpress-b98c66fff-2bh8n                          1/1 Running 5 (6d17h ago) 6d17h
wordpress-mysql-79966d6c5b-wqv9b                   1/1 Running 0 6d17h
wordpress-mysql2-6857bbcf7d-2rl9w                  1/1 Running 0 3d19h

[root@k8s-10-6-162-21 kubernetes]# kubectl get svc
NAME                    TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes              ClusterIP 10.96.0.1 <none> 443/TCP 22d
nginx                   NodePort 10.107.222.243 <none> 80:31090/TCP 21d
wordpress               LoadBalancer 10.109.11.61 <pending> 80:32249/TCP 6d23h
wordpress-mysql         ClusterIP None <none> 3306/TCP 6d23h
wordpress-mysql2        ClusterIP None <none> 3306/TCP 3d19h

[root@rook-ceph-tools-55f548895f-8pdhm /]# ceph osd pool ls
device_health_metrics
replicapool
myfs-metadata
myfs-data0

dashboard-4

dashboard-5

dashboard-6

[root@k8s-10-6-162-21 kubernetes]# kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-19515505-cd8f-41d1-ae91-bb42c3eb64f3   20Gi       RWO Delete Bound default/mysql-pv-claim2 rook-cephfs 3d20h
persistentvolume/pvc-7647bc80-febc-4299-a62a-8446d2c364c6   20Gi       RWO Delete Bound default/wp-pv-claim rook-ceph-block 6d23h
persistentvolume/pvc-b07feec8-adc8-4e22-abd6-177b5db4fdb1   20Gi       RWO Delete Bound default/mysql-pv-claim rook-ceph-block 7d

NAME                                     STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-pv-claim     Bound pvc-b07feec8-adc8-4e22-abd6-177b5db4fdb1 20Gi RWO rook-ceph-block 7d
persistentvolumeclaim/mysql-pv-claim2    Bound pvc-19515505-cd8f-41d1-ae91-bb42c3eb64f3 20Gi RWO rook-cephfs 3d20h
persistentvolumeclaim/wp-pv-claim        Bound pvc-7647bc80-febc-4299-a62a-8446d2c364c6 20Gi RWO rook-ceph-block 6d23h

成功部署后,查看 wordpress/mysql 上的博客:

blog-1

还在博客上做了评论:

blog-2

数据持久性验证

删除并重启 mysql / wordpress Pod:

[root@k8s-10-6-162-21 kubernetes]# kubectl delete po wordpress-mysql-79966d6c5b-wc6fs

blog-3

评论