您的位置 首页 golang

Kubernetes 基于GlusterFS+heketi的高可用动态存储管理StorageClass

说明:

Kubernetes中使用GlusterFS作为持久化存储,要提供storageClass使用需要依赖Heketi工具。Heketi是一个具有resetful接口的glusterfs管理程序,作为kubernetes的Storage存储的external provisioner。 “Heketi提供了一个RESTful管理界面,可用于管理GlusterFS卷的生命周期。借助Heketi,像OpenStack Manila,Kubernetes和OpenShift这样的云服务可以动态地配置GlusterFS卷和任何支持的持久性类型。Heketi将自动确定整个集群的brick位置,确保将brick及其副本放置在不同的故障域中。Heketi还支持任意数量的GlusterFS集群,允许云服务提供网络文件存储,而不受限于单个GlusterFS集群。
heketi:提供基于RESTful接口管理glusterfs的功能,可以方便的创建集群管理glusterfs的node,device,volume;与k8s结合可以创建动态的PV,扩展glusterfs存储的动态管理功能。主要用来管理glusterFS volume的生命周期,初始化时候就要分配好裸磁盘(未格式化)设备.

注意事项:

安装Glusterfs客户端:每个kubernetes集群的节点需要安装gulsterfs的客户端,如glusterfs-cli,glusterfs-fuse.主要用于在每个node节点挂载volume。
加载内核模块:每个kubernetes集群的节点运行modprobe dm_thin_pool,加载内核模块。
高可用(至少三个节点):至少需要节点用来部署glusterfs集群,并且这3个节点每个节点需要至少一个空余的磁盘。

基础设施要求:

正在运行的glusterfs集群,至少有三个node节点,每个节点至少有一个可用的裸块设备(如EBS卷或本地磁盘,就是没有格式化的).
用于运行GlusterFS节点必须为GlusterFS通信打开相应的端口(如果开启了防火墙的情况下,没开防火墙就不需要这些操作)。

安装GlusterFS

安装依赖及常用工具包:
yum -y install flex bison openssl openssl-devel libxml2-devel gcc lrzsz vim*
查找gluster的最新软件仓库:
yum search centos-release-gluster
安装最新版本的gluster软件仓库:
yum install centos-release-gluster7.noarch -y
安装gluster源,并安装glusterfs及相关软件包
yum install glusterfs glusterfs-server glusterfs-cli glusterfs-geo-replication glusterfs-rdma -y
客户端安装GlusterFS客户端软件
yum install glusterfs-fuse glusterfs-cli
启动Glusterd服务
systemctl start glusterd
systemctl enable glusterd –now //设开机自启,并立即启动服务

在任意一个节点上添加信任节点
gluster peer probe node99
gluster peer probe node110
gluster peer probe node145
gluster peer probe node108

查看节点状态:
gluster peer status //查看节点状态

Number of Peers: 3Hostname: node110Uuid: 5f13e231-25cf-475b-81ca-22122b1bfe55State: Peer in Cluster (Connected)Other names:10.14.151.110Hostname: node99Uuid: 7988f095-db67-47ee-913b-c232d1d4e954State: Peer in Cluster (Connected)Hostname: node145Uuid: 36e84b55-dab7-4551-9b44-490039d1bdf6State: Peer in Cluster (Connected)Other names:10.14.151.145

创建复制卷

mkdir /glusterfs/storage1/rep_vol1
gluster volume create rep_vol1 replica 2 node99:/glusterfs/storage1/rep_vol1 node108:/glusterfs/storage1/rep_vol1

创建分布式卷

gluster volume create vdisk1 node108:/brick1 node110:/brick1 brick node145:/brick1 force

创建分布式复制卷

gluster volume create fbfz replica 2 transport tcp node108:/gluster/fbfz1 node110:/gluster/fbfz1 node145:/gluster/fbfz1 node108:/gluster/fbfz2 node110:/gluster/fbfz2 node145:/gluster/fbfz2 force

启动卷
gluster volume start rep_vol1
查看卷状态
gluster volume status
gluster volume info
客户端测试挂载卷
mount -t glusterfs node108:rep_vol /tmp/aaa
mount -t glusterfs node145:fbfz /mnt/fbfz
客户端测试卷数据存储
for i in `seq -w 1 3`;do cp -rp /var/log/messages /tmp/aaa/test-$i;done

ls /tmp/aaa111  1.txt  2.txt  anaconda-ks.cfg  test-1  test-2  test-3

其他备用操作

停止卷:
gluster volume stop vdisk2
删除卷:
gluster volume delete vdisk2
将某个存储节点主机从信任池中删除:
gluster peer detach node2

安装Heketi

Heketi是由golang编写,直接静态编译运行二进制即可,也可以通过yum安装以及docker部署,主要会产生db文件存储cluster、node、device、volume等信息。

安装二进制包
wget -c https://github.com/heketi/heketi/releases/download/v8.0.0/heketi-v8.0.0.linux.amd64.tar.gz tar zxvf heketi-v9.0.0.linux.amd64.tar.gz
mkdir -pv /opt/heketi/{bin,conf,data}
cp heketi/heketi.json /opt/heketi/conf/
cp heketi/{heketi,heketi-cli} /opt/heketi/bin/

创建ssh-key

我们glusterFS部署在k8s集群外,所以heketi通过ssh管理glusterFS。需要创建免秘钥登陆到所有glusterFS节点。
ssh-keygen -f /opt/heketi/conf/heketi_key -t rsa -N ''
ssh-copy-id -i /opt/heketi/conf/heketi_key.pub root@node108 -p 2222
ssh-copy-id -i /opt/heketi/conf/heketi_key.pub root@node110 -p 2222
ssh-copy-id -i /opt/heketi/conf/heketi_key.pub root@node145 -p 2222

配置文件修改

{  "_port_comment": "Heketi Server Port Number",  "port": "18080",  "_use_auth": "Enable JWT authorization. Please enable for deployment",  "use_auth": true,  "_jwt": "Private keys for access",  "jwt": {    "_admin": "Admin has access to all APIs",    "admin": {      "key": "adminkey"    },    "_user": "User only has access to /volumes endpoint",    "user": {      "key": "userkey"    }  },  "_glusterfs_comment": "GlusterFS Configuration",  "glusterfs": {    "_executor_comment": [      "Execute plugin. Possible choices: mock, ssh",      "mock: This setting is used for testing and development.",      "      It will not send commands to any node.",      "ssh:  This setting will notify Heketi to ssh to the nodes.",      "      It will need the values in sshexec to be configured.",      "kubernetes: Communicate with GlusterFS containers over",     "            Kubernetes exec api."    ],    "executor": "ssh",    "_sshexec_comment": "SSH username and private key file information",    "sshexec": {      "keyfile": "/opt/heketi/conf/heketi_key",      "user": "root",      "port": "2222",      "fstab": "/etc/fstab"    },    "_kubeexec_comment": "Kubernetes configuration",    "kubeexec": {      "host" :"https://kubernetes.host:8443",      "cert" : "/path/to/crt.file",      "insecure": false,      "user": "kubernetes username",      "password": "password for kubernetes user",      "namespace": "OpenShift project or Kubernetes namespace",      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"    },    "_db_comment": "Database file name",    "db": "/opt/heketi/data/heketi.db",    "_loglevel_comment": [      "Set log level. Choices are:",      "  none, critical, error, warning, info, debug",      "Default is warning"    ],    "loglevel" : "debug"  }}

注意:
需要说明的是,heketi有三种executor,分别为mock、ssh、kubernetes,建议在测试环境使用mock,生产环境使用ssh,当glusterfs以容器的方式部署在kubernetes上时,才使用kubernetes。我们这里将glusterfs和heketi独立部署,使用ssh的方式。
使用docker部署的时候,还需将/var/lib/heketi/mounts 挂载至容器里面, heketi 会将此目录作为 gluster volume的挂载点。

systemd配置
cat /usr/lib/systemd/system/heketi.service

[Unit]Description=RESTful based volume management framework for GlusterFSBefore=network-online.targetAfter=network-online.targetDocumentation=https://github.com/heketi/heketi[Service]Type=simpleLimitNOFILE=65536ExecStart=/opt/heketi/bin/heketi --config=/opt/heketi/conf/heketi.jsonKillMode=processRestart=on-failureRestartSec=5SuccessExitStatus=15StandardOutput=syslogStandardError=syslog[Install]WantedBy=multi-user.target

启动heketi服务
systemctl start heketi
systemctl enable heketi
systemctl status heketi

Heketi管理GlusterFS

添加cluster
/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json cluster create

{"id":"bb1362c3360d80419c822b9994381608","nodes":[],"volumes":[],"block":true,"file":true,"blockvolumes":[]}

将3个glusterfs节点作为node添加到cluster

添加节点

opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json node add --cluster "bb1362c3360d80419c822b9994381608" --management-host-name node108 --storage-host-name 10.14.151.108 --zone 1

{"zone":1,"hostnames":{"manage":["node108"],"storage":["10.14.151.108"]},"cluster":"bb1362c3360d80419c822b9994381608","id":"227fd34c519f0a2c9d5a5b7f3d048745","state":"online","devices":[]}

/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json node add --cluster "bb1362c3360d80419c822b9994381608" --management-host-name node110 --storage-host-name 10.14.151.110 --zone 1

{"zone":1,"hostnames":{"manage":["node110"],"storage":["10.14.151.110"]},"cluster":"bb1362c3360d80419c822b9994381608","id":"5f2d7412f0c874634aa8ee18865533bf","state":"online","devices":[]}

/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json node add --cluster "bb1362c3360d80419c822b9994381608" --management-host-name node145 --storage-host-name 10.14.151.145 --zone 1

{"zone":1,"hostnames":{"manage":["node145"],"storage":["10.14.151.145"]},"cluster":"bb1362c3360d80419c822b9994381608","id":"e20c47a9d9a31a300ed85ccc37441608","state":"online","devices":[]}

查看节点:
/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey node list

Id:227fd34c519f0a2c9d5a5b7f3d048745     Cluster:bb1362c3360d80419c822b9994381608Id:5f2d7412f0c874634aa8ee18865533bf     Cluster:bb1362c3360d80419c822b9994381608Id:e20c47a9d9a31a300ed85ccc37441608     Cluster:bb1362c3360d80419c822b9994381608

添加device

机器只是作为gluster的运行单元,volume是基于device创建的。同时需要特别说明的是,目前heketi仅支持使用裸分区或裸磁盘(未格式化)添加为device,不支持文件系统。

/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json device add --name="/dev/sdb" --node "227fd34c519f0a2c9d5a5b7f3d048745"
/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json device add --name="/dev/sdb" --node "5f2d7412f0c874634aa8ee18865533bf"
/opt/heketi/bin/heketi-cli --user admin --server http://10.143.143.111:18080 --secret adminkey --json device add --name="/dev/sdb" --node "e20c47a9d9a31a300ed85ccc37441608"

注:–node参数给出的id是上一步创建node时生成的,实际配置中,要添加每一个节点的每一块用于存储的硬盘。

生产实际配置

以上ad-hoc命令均可通过配置文件创建然后导入:

$ sudo cat /data/heketi/conf/topology.json

{    "clusters": [        {            "nodes": [                {                    "node": {                        "hostnames": {                            "manage": [                                "k8s1"                            ],                            "storage": [                                "10.111.209.188"                            ]                        },                        "zone": 1                    },                    "devices": [                        "/dev/vdc1"                    ]                },                {                    "node": {                        "hostnames": {                            "manage": [                                "k8s2"                            ],                            "storage": [                                "10.111.209.189"                            ]                        },                        "zone": 1                    },                    "devices": [                        "/dev/vdc1"                    ]                },                {                    "node": {                        "hostnames": {                            "manage": [                                "k8s3"                            ],                            "storage": [                                "10.111.209.190"                            ]                        },                        "zone": 1                    },                    "devices": [                        "/dev/vdc1"                    ]                }                         ]        }    ]}

创建:

$ sudo heketi-cli topology load –json topology.json

添加volume

这里仅仅是做一个测试,实际使用中,会由kubernetes自动创建pv.

创建一个大小为3G,副本为2的volume

/opt/heketi/bin/heketi-cli –user admin –server http://10.143.143.111:18080 –secret adminkey volume create –size 3 –replica 2

Name: vol_61055753aa935407b80e1137647733f6

Size: 3

Volume Id: 61055753aa935407b80e1137647733f6

Cluster Id: bb1362c3360d80419c822b9994381608

Mount: 10.14.151.108:vol_61055753aa935407b80e1137647733f6

Mount Options: backup-volfile-servers=10.14.151.110,10.14.151.145

Block: false

Free Size: 0

Reserved Size: 0

Block Hosting Restriction: (none)

Block Volumes: []

Durability Type: replicate

Distribute Count: 1

Replica Count: 2

kubernetes storageclass 配置

创建storageclass

添加storageclass-glusterfs.yaml文件,内容如下:

cat storageclass-glusterfs.yaml

apiVersion: v1

kind: Secret

metadata:

name: heketi-secret

namespace: default

data:

base64 encoded password. E.g.: echo -n "mypassword" | base64

key: YWRtaW5rZXk=

type: kubernetes.io/glusterfs


apiVersion: storage.k8s.io/v1beta1

kind: StorageClass

metadata:

name: glusterfs

provisioner: kubernetes.io/glusterfs

allowVolumeExpansion: true

parameters:

resturl: "http://10.143.143.111:18080"

clusterid: "bb1362c3360d80419c822b9994381608"

restauthenabled: "true"

restuser: "admin"

secretNamespace: "default"

secretName: "heketi-secret"

restuserkey: "adminkey"

gidMin: "40000"

gidMax: "50000"

volumetype: "replicate:2"

kubectl apply -f storageclass-glusterfs.yaml

secret/heketi-secret created

storageclass.storage.k8s.io/glusterfs created

kubectl get sc

NAME PROVISIONER AGE

csi-cephfs cephfs.csi.ceph.com 260d

glusterfs kubernetes.io/glusterfs 1m

注意:

storageclass.beta.kubernetes.io/is-default-class: "true" #表示此storageClass作为default sc,创建pvc不指定sc时,默认使用此sc.reclaimPolicy: Retain #表示pv回收策略为保留,删除pvc时将不删除pv。

更详细的用法参考:<u>https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfs</u>

创建pvc

kubectl create -f glusterfs-pvc.yaml

cat glusterfs-pvc.yaml

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

name: glusterfs-mysql1

namespace: default

annotations:

volume.beta.kubernetes.io/storage-class: "glusterfs"

spec:

accessModes:

  • ReadWriteMany

resources:

requests:  storage: 1Gi

kubectl create -f glusterfs-pvc.yaml

persistentvolumeclaim/glusterfs-mysql1 created

kubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 1y

pvc-5ae5026d-91d2-11ea-bd25-fa157e638e00 1Gi RWX Delete Bound default/glusterfs-mysql1 glusterfs 5m

kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

glusterfs-mysql1 Bound pvc-5ae5026d-91d2-11ea-bd25-fa157e638e00 1Gi RWX glusterfs 4m

创建pod,使用pvc

cat mysql-deployment.yaml


kind: Deployment

apiVersion: extensions/v1beta1

metadata:

name: mysql

namespace: default

spec:

replicas: 1

template:

metadata:  labels:    name: mysqlspec:  containers:  - name: mysql    image: mysql:5.7    imagePullPolicy: IfNotPresent    env:    - name: MYSQL_ROOT_PASSWORD      value: root123456    ports:      - containerPort: 3306    volumeMounts:    - name: gluster-mysql-data      mountPath: "/var/lib/mysql"  volumes:    - name: gluster-mysql-data      persistentVolumeClaim:        claimName: glusterfs-mysql1

kubectl create -f mysql-deployment.yaml

deployment.extensions/mysql created

[# kubectl get deploy

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

mysql 1 1 1 1 4m

kubectl get pods

NAME READY STATUS RESTARTS AGE

mysql-b75b5dcfb-cb7qm 1/1 Running 0 5m

]# kubectl exec -ti mysql-b75b5dcfb-cb7qm sh

df -Th

Filesystem Type Size Used Avail Use% Mounted on

overlay overlay 120G 22G 99G 18% /

tmpfs tmpfs 64M 0 64M 0% /dev

tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup

/dev/mapper/centos-root xfs 120G 22G 99G 18% /etc/hosts

shm tmpfs 64M 0 64M 0% /dev/shm

10.14.151.108:vol_00de560fab819d81ade9aae98fcdd4d1 fuse.glusterfs 1016M 254M 763M 25% /var/lib/mysql

tmpfs tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount

tmpfs tmpfs 16G 0 16G 0% /proc/scsi

tmpfs tmpfs 16G 0 16G 0% /sys/firmware

创建statefulset


apiVersion: v1

kind: Service

metadata:

name: nginx

labels:

app: nginx

spec:

ports:

  • port: 80

    name: web

clusterIP: None

selector:

app: nginx

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: nginx

spec:

selector:

matchLabels:  app: nginx # has to match .spec.template.metadata.labels

serviceName: "nginx"

replicas: 3 # by default is 1

template:

metadata:  labels:    app: nginx # has to match .spec.selector.matchLabelsspec:  terminationGracePeriodSeconds: 10  containers:  - name: nginx    image: nginx    ports:    - containerPort: 80      name: web    volumeMounts:    - name: www      mountPath: /usr/share/nginx/html

volumeClaimTemplates:

  • metadata:

    name: www

    spec:

    accessModes: [ "ReadWriteOnce" ]

    storageClassName: "glusterfs"

    resources:

    requests:  storage: 1Gi

kubectl apply -f nginx-statefulset.yml

service/nginx created

statefulset.apps/nginx created

kubectl get pod,pv,pvc

NAME READY STATUS RESTARTS AGE

pod/nginx-0 1/1 Running 0 116s

pod/nginx-1 1/1 Running 0 98s

pod/nginx-2 1/1 Running 0 91s

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

persistentvolume/pvc-5ac3eba9-fe12-11e8-a19e-00163e14b18f 1Gi RWO Retain Bound default/www-nginx-0 glusterfs 99s

persistentvolume/pvc-65f27519-fe12-11e8-a19e-00163e14b18f 1Gi RWO Retain Bound default/www-nginx-1 glusterfs 93s

persistentvolume/pvc-69b31512-fe12-11e8-a19e-00163e14b18f 1Gi RWO Retain Bound default/www-nginx-2 glusterfs 86s

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

persistentvolumeclaim/www-nginx-0 Bound pvc-5ac3eba9-fe12-11e8-a19e-00163e14b18f 1Gi RWO glusterfs 116s

persistentvolumeclaim/www-nginx-1 Bound pvc-65f27519-fe12-11e8-a19e-00163e14b18f 1Gi RWO glusterfs 98s

persistentvolumeclaim/www-nginx-2 Bound pvc-69b31512-fe12-11e8-a19e-00163e14b18f 1Gi RWO glusterfs 91s

我们可以看到RECLAIM POLICY: Retain ,经过测试

删除pvc,pv status会变成Released状态,且不会被删除删除pv, 通过heketi-cli volume list查看,volume不会被删除

kubernetes pv和gluster volume不一致时,可使用heketi来统一管理volume.此文档heketi和glusterfs都在kubernetes集群外部署。对于支持AWS EBS的磁盘,可通过EBS storageClass方式将glusterFS heketi部署在容器中管理.参考https://github.com/gluster/gluster-kubernetes

参考文档

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfshttps://github.com/heketi/heketi/blob/master/docs/admin/readme.md

<u>https://www.cnblogs.com/breezey/p/8849466.html</u>

验证测试:

kubectl exec -ti mysql-b75b5dcfb-cb7qm sh

[图片上传失败…(image-c1efde-1589539985617)]

进容器看挂载的是10.14.151.108的volume,

[图片上传失败…(image-842e18-1589539985617)]

关闭10.14.151.108的gluster服务,模拟故障

[图片上传失败…(image-5e7f0e-1589539985617)]

进容器后写数据:

[图片上传失败…(image-338d7e-1589539985617)]

[图片上传失败…(image-a82f69-1589539985617)]

说明挂掉一个节点是不影响用户使用的。


文章来源:智云一二三科技

文章标题:Kubernetes 基于GlusterFS+heketi的高可用动态存储管理StorageClass

文章地址:https://www.zhihuclub.com/315.shtml

关于作者: 智云科技

热门文章

发表评论

您的电子邮箱地址不会被公开。

网站地图