Skip to main content

OpenEBS

  • openebs/openebs
    • openebs.io
    • Jiva 源于 longhorn fork，在一些实现上有分歧
      • 目前区别已经非常大
      • 分歧点: Control Plane functionality, iSCSI Target implementation, Quorum, Backup / Restore, UI
tip
  • DB 类型应用最好使用 Local PV
  • 无集群能力 DB 可使用 NVMe, SPDK, cStor + ZFS
  • 流应用（Kafka）存储推荐 NVMe Mayastor

存储类型

VolumeStorageRequirement
OpenEBS MayastorSSDs/Cloud Volumes低延时, HA, 同步副本, 快照, 克隆, Thin provisioning
OpenEBS cStorDisks/SSDs/Cloud Volumes保护节点异常, 同步副本, 快照, 克隆, Thin provisioning
OpenEBS Jivahostpath, 外部挂载保护节点异常, 同步副本, Thin provisioning
Local PV - Hostpathhostpath, 外部挂载低延时, 本地持久卷
Local PV - DeviceDisks/SSDs/Cloud Volumes低延时, 本地持久卷
Local PV - ZFSDisks/SSDs/Cloud Volumes低延时, 本地持久卷, 快照, 克隆
Local PV - Rawfile低延时, 本地持久卷

目录结构​

  • /var/openebs/
    • local/ - Local PV - Hostpath
    • ndm/
    • sparse/

Helm​

  • https://github.com/openebs/charts
  • helm 子 chart
    • cstor
    • jiva
    • localpv-provisioner
      • StorageClass: openebs-hostpath
      • /var/openebs/local
    • lvm-localpv
    • openebs-ndm - node-disk-manager
    • zfs-localpv
tip

helm 功能区分 子 chart 模式和 bundle 的 templates

# 需要先安装 iscsi
# https://docs.openebs.io/docs/next/prerequisites.html

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install --namespace openebs --name openebs openebs/openebs

Operator​

  • 是 Helm Operator，更建议直接安装 HELM
# 直接安装
kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

zfs pv​

kubectl get csinodes -A
openebs-zfspv.sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfspv
allowVolumeExpansion: true
parameters:
  poolname: 'zfspv-pool'
  # zfs,ext2,ext3,ext4,xfs,btrfs
  # 非 zfs 则用 zvol
  # 有些应用在指定的 fs 上运行的更好 - https://github.com/openebs/zfs-localpv/issues/169
  # 不设置默认 ext4
  fstype: 'zfs'
  # 允许多次挂载
  # Deployment 可以不用 Recreate 策略
  shared: 'yes'
  # yes - 即便 zpool 空间不够也会提供 pv
  # no - 会确保保留空间足够
  thinprovision: 'no'
  dedup: 'no'
  # on,off,lzjb,zstd,gzip,gzip-1,gzip-2,gzip-3,gzip-4,gzip-5,gzip-6,gzip-7,gzip-8,gzip-9,zle,lz4
  # on - 默认加密算法
  compression: 'on'
  # zfs 配置
  # recordsize:
  # zvol 配置
  # volblocksize:
provisioner: zfs.csi.openebs.io
allowedTopologies:
  - matchLabelExpressions:
      - key: openebs.io/nodeid
        values:
          - zfspv-node1
          - zfspv-node2

快照

zfspv-snapclass.yaml
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1beta1
metadata:
  name: zfspv-snapclass
  annotations:
    # snapshot.storage.kubernetes.io/is-default-class: "true"
driver: zfs.csi.openebs.io
deletionPolicy: Delete

对 PVC 进行快照

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: zfspv-snap
spec:
  volumeSnapshotClassName: zfspv-snapclass
  source:
    persistentVolumeClaimName: csi-zfspv
kubectl get volumesnapshot.snapshot
# 查看快照
kubectl get zfssnap -n openebs
zfs list -t all

Data Engine​

  • cStor
    • Apache-2.0, C
    • IOs on ZVOL/uZFS over network
    • synchronous replication, snapshots and clones
  • Jiva
  • Mayastor
    • Apache-2.0, Rust
    • cloud native declarative data plane
      • abstract storage resources - Storage unification
      • Low latency
      • support Micro-VM based containers
      • Programmatic based storage access
    • NVMe SSD
    • SPDK, NVMe-oF
    • Linux kernel 5.4+ - nvme-tcp, ext4, optionally xfs
    • x86-64, SSE4.2
  • Maya
    • non-csi - 3.0 废弃
    • 使用 CStore CSI 或 Jiva CSI

CSI​

abbr.stand for
CASContainer Attached Storage
CSPCCStorPoolCluster
CSIContainer Storage Interface
Storage Pool ClaimSPC
CSPcStor Storage Pool
CVcStor Volume
CVRcStor Volume Replica
SPDKIntel Storage Performance Development Kit
MSNMayastor Node

LocalPV ZFS​

  • openebs/zfs-localpv
    • 节点上创建 ZPOOL - SC
    • 支持 FS、Block
    • 支持 快照、克隆、Resize
    • 支持 Velero 备份恢复
caution
zfs-localpv.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfspv
parameters:
  # zfs 参数
  recordsize: '4k'
  # volblocksize: "4k" # zvol 使用该参数
  compression: 'off'
  dedup: 'off'
  fstype: 'zfs' # 其他 fs 会创建为 zvol
  poolname: 'zfspv-pool' # 匹配创建的名字
provisioner: zfs.csi.openebs.io
allowedTopologies:
  - matchLabelExpressions:
      # 限定节点
      - key: kubernetes.io/hostname
        values:
          - zfspv-node1
          - zfspv-node2
      # 自定义 key
      - key: openebs.io/rack
        values:
          - rack1
# 默认使用 zfs scheduler - VolumeWeighted, CapacityWeighted
# 可使用 kubernetes scheduler - pod 节点亲和
volumeBindingMode: WaitForFirstConsumer
csi-zfspv.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: csi-zfspv
spec:
  storageClassName: openebs-zfspv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

snapshot​

zfspv-snapclass.yaml
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1beta1
metadata:
  name: zfspv-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: 'true'
driver: zfs.csi.openebs.io
deletionPolicy: Delete
zfspv-snap.yaml
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: zfspv-snap
spec:
  volumeSnapshotClassName: zfspv-snapclass
  source:
    persistentVolumeClaimName: csi-zfspv
kubectl get volumesnapshot.snapshot
# 节点上也可以查看快照
zfs list -t all

clone​

zfspv-clone.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: zfspv-clone
spec:
  storageClassName: openebs-zfspv
  dataSource:
    # clone 自 snapshot
    name: zfspv-snap
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
    # clone 自 volume
    name: zfspv-pvc
    kind: PersistentVolumeClaim
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

Backup/Restore​

  • 需要 Velero
  • 需要 openebs/velero-plugin 插件
zfspv-full.yaml
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: zfspv-full
  namespace: velero
spec:
  provider: openebs.io/zfspv-blockstore
  config:
    bucket: velero
    prefix: zfs
    incrBackupCount: '3' # number of incremental backups we want to have
    namespace: openebs # this is the namespace where ZFS-LocalPV creates all the CRs, passed as OPENEBS_NAMESPACE env in the ZFS-LocalPV deployment
    provider: aws
    region: minio
    s3ForcePathStyle: 'true'
    s3Url: http://minio.velero.svc:9000

FAQ

driver name zfs.csi.openebs.io not found in the list of registered CSI drivers​

# 确保 driver 存在
kubectl get csinodes -oyaml

driver null 可能是注册出问题了

# 可尝试删除 openebs-zfs-localpv-node 的 Pod
kubectl delete csinodes NODEID
kubectl get pods -A -l role=openebs-zfs -owide

还有问题，应该是 kubelet 目录配置不对，默认是 /var/lib/kubelet，但 k0s、rke、k3s 需要修改。