Clever Geek Handbook

A practical example of connecting a Ceph-based storage to a Kubernetes cluster

The Container Storage Interface (CSI) is a unified interface for communication between Kubernetes and storage systems. We have already talked about it briefly , and today we will take a closer look at the combination of CSI and Ceph: we will show how to connect the Ceph storage to the Kubernetes cluster.

The article contains real examples, albeit slightly simplified for ease of perception. We will not consider installing and configuring Ceph and Kubernetes clusters.



Are you wondering how it works?





, Kubernetes, , , kubespray. Ceph — , , . , , 10 /.



, !



Ceph , :



ceph health
ceph -s


RBD :



ceph osd pool create kube 32
ceph osd pool application enable kube rbd


Kubernetes. Ceph CSI RBD. , , Helm.

, ceph-csi-rbd:



helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml


cephrbd.yml. ID IP- Ceph:



ceph fsid  #    clusterID
ceph mon dump  #    IP-


cephrbd.yml. PSP (Pod Security Policies). nodeplugin provisioner , , :



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true


— Kubernetes.



helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace


, RBD !

Kubernetes StorageClass. Ceph.



Ceph kube:



ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'


:



ceph auth get-key client.rbdkube


:



AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==


Secret Kubernetes — , userKey:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  #        ,   
  #  Ceph. ID      ,
  #   storage class
  userID: rbdkube
  userKey: <user-key>


: 



kubectl apply -f secret.yaml


StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   #       
   #   .
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard


clusterID, ceph fsid, Kubernetes: 



kubectl apply -f storageclass.yaml


, PVC (Persistent Volume Claim):



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc


, Kubernetes Ceph :



kubectl get pvc
kubectl get pv


! Ceph?

:



rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # ,  ,   ID ,


, RBD.

pvc.yaml 2Gi :



kubectl apply -f pvc.yaml


, , .



rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc


, PVC . , Kubernetes PVC YAML: 



kubectl get pvc rbd-pvc -o yaml


:



message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending



, — .

, . PVC/PV .



Pod, :



---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false


PVC:



kubectl get pvc


, .



RBD ( – Rados Block Device), , . , , CephFS.

Ceph Kubernetes CSI CephFS.



Helm-:



helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml


cephfs.yml. , Ceph:



ceph fsid
ceph mon dump


:



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true


, address:port. cephfs , v2.

httpMetrics ( Prometheus ) , nginx-proxy, Kubespray’. , , .



Helm- Kubernetes:



helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace


Ceph, . , CephFS . fs :



ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'


, :



ceph auth get-key client.fs


Secret StorageClass.

, RBD:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  #     
  adminID: fs
  adminKey: <  >


:



kubectl apply -f secret.yaml


– StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  #    CephFS,     
  fsName: cephfs

  # ()  Ceph,      
  # pool: cephfs_data

  # ()      Ceph-fuse
  # :
  # fuseMountOptions: debug

  # ()     CephFS  
  # . man mount.ceph     . :
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  #       /  Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # ()     ceph-fuse (fuse), 
  #  ceph kernelclient (kernel).
  #   ,      ,
  #    ceph-fuse  mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug


clusterID Kubernetes:



kubectl apply -f storageclass.yaml




, , PVC:



---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc


PVC/PV:



kubectl get pvc
kubectl get pv


CephFS, -. , .



Ceph :



#  
mkdir -p /mnt/cephfs

#     
ceph auth get-key client.admin >/etc/ceph/secret.key

#    /etc/fstab
# !!  ip     
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs


, FS Ceph , . , - , .



, CephFS . Kubernetes PVC — , , 7Gi.



: 



kubectl apply -f pvc.yaml


, :



getfattr -n ceph.quota.max_bytes <-->


, , attr.



,



YAML , .

— . Ceph Kubernetes, :



Kubernetes c

RBD

RBD Kubernetes Ceph

RBD Kubernetes CSI

CephFS

CephFS Kubernetes CSI



Kubernetes Kubernetes , CephFS . GET/POST Ceph.



, Ceph. -, .



: , Southbridge, Certified Kubernetes Administrator, .



More articles:


All Articles