使用Velero Restic快速完成云原生应用及PV数据从GKE到至ACK的迁移

本文记录使用Velero Restic快速完成云原生应用及PV数据从GKE到至ACK的迁移的实践过程。 此过程也同样适用于自建Kubernetes集群内的应用及PV数据迁移至ACK。

实践步骤概览

（1）创建GKE集群（或自建Kubernetes集群）
（2）在GKE集群上部署示例应用Jenkins Application并执行一个构建任务
（3）创建ACK集群
（4）在ACK集群中部署Minio Service用于应用迁移时数据中转服务
（5）在GKE上部署Velero并备份整个Jenkins Application（或其他任意应用）
（6）把Jenkins Application使用的容器镜像同步到阿里云容器镜像仓库
（7）在ACK上创建Jenkins Application所使用的StorageClass
（8）在ACK上部署Velero并恢复整个Jenkins Application
（9）在ACK上替换Jenkins Application所使用的容器镜像
（10）验证Jenkins Application能否正常提供服务及其构建任务是否存在

环境物料清单

（1）GKE集群（或自建Kubernetes集群）
（2）ACK集群

创建GKE集群并部署示例应用Jenkins Application

(1) 在MarketPlace中找到Jenkins应用并配置部署到GKE集群的jenkins命名空间下：

(2)应用正在完成部署：

(3)查看Ingress并访问jenkins服务：

(4)Jenkins应用的初始化配置：

(5)Jenkin应用上创建一个名为gke-to-ack的任务并执行构建：
这个步骤相当于往PV存储卷中写数据。

（6）Jenkins应用的k8s资源列表如下

$ kubectl -n jenkins get deploy
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jenkins-jenkins-deployment   1         1         1            1           177m
$ kubectl -n jenkins get svc
NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
jenkins-jenkins-agents-connector   ClusterIP   10.39.241.75    <none>        50000/TCP        177m
jenkins-jenkins-ui                 NodePort    10.39.253.141   <none>        8080:31759/TCP   177m
$ kubectl -n jenkins get ing
NAME                 HOSTS   ADDRESS        PORTS     AGE
jenkins-jenkins-ui   *       xx.xx.xx.xx   80, 443   177m
$ kubectl -n jenkins get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-jenkins-pvc   Bound    pvc-e64eacc4-a482-11e9-bfa9-42010a8000da   8Gi        RWO            standard       177m
$ kubectl -n jenkins get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                       STORAGECLASS   REASON   AGE
pvc-e64eacc4-a482-11e9-bfa9-42010a8000da   8Gi        RWO            Delete           Bound    jenkins/jenkins-jenkins-pvc                                 standard                177m

创建ACK集群并部署Minio对象存储应用

Minio对象存储服务用于存储PV持久化数据， 我们即将支持阿里云OSS对象存储服务。

(1) 自定义替换minio-deploy.yaml中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY值并部署：

kubectl apply -f minio-deploy.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: minio

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: minio
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        emptyDir: {}
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        env:
        - name: MINIO_ACCESS_KEY
          value: "<your MINIO_ACCESS_KEY>"
        - name: MINIO_SECRET_KEY
          value: "your MINIO_SECRET_KEY"
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: minio
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run Minio in a test/trial environment, for example with Minikube.
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    component: minio

(2) 查看minio LoadBalancer信息并登陆应用：

$ kubectl -n minio get svc
NAME    TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
minio   LoadBalancer   172.21.2.216   xxx.xxx.xxx.xx   9000:30912/TCP   2d4h

(3) 替换minio-job.yaml中的minio_server_url minino_access_key minio_secret_key并运行job在minio中创建名为velero的bucket：

kubectl apply -f minio-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  namespace: minio
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add velero <your minio_server_url> <your minino_access_key> <your minio_secret_key> && mc --config-dir=/config mb -p velero/velero"
        volumeMounts:
        - name: config
          mountPath: "/config"

(4) 查看bucket是否创建成功：

在GKE和ACK上部署velero

(1) 安装velero client：
请从official release下载最新版本的velero客户端：

(2) 安装velero server：
创建credentials-velero文件并设置aws_access_key_id 与 aws_secret_access_key的值

[default]
aws_access_key_id = <your minio_access_key>
aws_secret_access_key = <your minio_secret_key>

ACK集群中请替换minio_server_url的值并指定image参数部署velero server

velero install     --provider aws    --image registry.cn-hangzhou.aliyuncs.com/acs/velero:latest  --bucket velero     --secret-file ./credentials-velero     --use-volume-snapshots=false     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic     --wait

GKE中请替换minio_server_url的值并部署velero server

velero install     --provider aws    --bucket velero     --secret-file ./credentials-velero     --use-volume-snapshots=false     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic     --wait

在GKE集群中备份Jenkins Application

(1)在备份带volume信息的pod之前，我们要通过给pod加annotation来告诉velero哪些pod需要包含volume数据， 查看jenkins应用的Deployment资源：

$ kubectl -n jenkins get deploy jenkins-jenkins-deployment -oyaml

(2)为pod加annotation:

$ kubectl -n jenkins get po
NAME                                          READY   STATUS      RESTARTS   AGE
jenkins-deployer-zh5p4                        0/1     Completed   0          109m
jenkins-jenkins-deployment-7df86c64d4-tqqlr   1/1     Running     0          109m
$ kubectl -n jenkins annotate pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr backup.velero.io/backup-volumes=jenkins-jenkins-pvc
pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr annotated

(3)创建备份

$ velero backup create gcloud-jenkins-backup-restic --include-namespaces jenkins --wait
Backup request "gcloud-jenkins-backup-restic" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe gcloud-jenkins-backup-restic` and `velero backup logs gcloud-jenkins-backup-restic`.

(4) 查看备份

$ ./velero backup get
NAME                           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
gcloud-jenkins-backup-restic   Completed   2019-07-12 18:48:48 +0800 +08   29d       default            <none>

（5）登录Minio Server查看

同步（或批量）迁移容器镜像

把Jenkins应用使用的容器镜像导入到阿里云容器镜像仓库，导入到的地址为：

registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3

批量迁移可参考：https://github.com/AliyunContainerService/sync-repo.git

在ACK集群中创建Jenkins应用所使用的StorageClass standard

我们知道不同的云厂商，后端的存储基础设施是不一样的， 我们需要先创建一个相同名称的StorageClass来屏蔽对底层存储基础设施差异的感知。

$ kubectl apply -f storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: standard
provisioner: alicloud/disk
parameters:
  type: cloud
reclaimPolicy: Delete

在ACK集群中恢复Jenkins Application

查看备份信息：

$ velero  backup get
NAME                           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
gcloud-jenkins-backup-restic   Completed   2019-07-12 18:48:48 +0800 CST   29d       default            <none>

恢复应用：

$ velero restore create --from-backup gcloud-jenkins-backup-restic

此处需要编辑deployment修改image为registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3

查看restore的状态：

$ velero restore describe gcloud-jenkins-backup-restic-20190712190536
$ velero restore logs gcloud-jenkins-backup-restic-20190712190536

restore完成后的状态为：

$ velero restore get
NAME                                          BACKUP                         STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
gcloud-jenkins-backup-restic-20190712190536   gcloud-jenkins-backup-restic   Completed   0          0        2019-07-12 19:05:36 +0800 CST   <none>

查看ACK集群上jenkins应用的ingress并访问服务进行验证

$ kubectl -n jenkins get ing
NAME                 HOSTS   ADDRESS          PORTS     AGE
jenkins-jenkins-ui   *       xx.xx.xx.xx   80, 443   56m

至此，一个带pv存储的jenkins应用被完整地从GKE迁移到ACK上。

参考链接：
https://velero.io/docs/v1.0.0/get-started/
https://velero.io/docs/v1.0.0/restic/#troubleshooting
https://github.com/heptio/velero
https://github.com/AliyunContainerService/velero-plugin