LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
はじめに
https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/
5/3にkubernetesの新バージョンStargazerがリリースされました。
なので手元にあるv1.23のKubernetesクラスターを、v1.24にアップグレードします。
アップグレード方法は、以下の公式ドキュメントに記載されています。
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
kuberneteesクラスターの確認
kubernetesクラスターのバージョンは、1.23.0です。
kubeuser@kubenewmaster1:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kubenewmaster1 Ready control-plane,master 80d v1.23.0 192.168.1.251 <none> Ubuntu 20.04.3 LTS 5.4.0-105-generic containerd://1.4.12 kubenewworker1 Ready <none> 34d v1.23.0 192.168.1.252 <none> Ubuntu 20.04.3 LTS 5.4.0-105-generic containerd://1.4.12 kubenewworker2 Ready <none> 34d v1.23.0 192.168.1.249 <none> Ubuntu 20.04.3 LTS 5.4.0-104-generic containerd://1.4.12 kubeuser@kubenewmaster1:~$ kubectl get all --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default pod/beauty-deploy-7d59f6c75f-hbk8b 1/1 Running 0 34d default pod/beauty-deploy-7d59f6c75f-xp9fd 1/1 Running 0 34d default pod/food-deploy-54544fc85d-5k5h9 1/1 Running 0 34d default pod/food-deploy-54544fc85d-hhn4r 1/1 Running 0 34d default pod/food-deploy-54544fc85d-smn2l 1/1 Running 0 34d default pod/nfs-subdir-external-provisioner-9c9c5866f-2n2l6 1/1 Running 0 34d default pod/patch-test-6bd9658956-npz4m 1/1 Running 0 34d default pod/patch-test-6bd9658956-qcjhj 1/1 Running 0 34d default pod/pvc-deploy-77f7d456bf-jnp57 1/1 Running 0 34d ingress-nginx pod/ingress-nginx-controller-54d8b558d4-nhzd6 1/1 Running 0 34d kube-system pod/calico-kube-controllers-958545d87-gp64m 1/1 Running 0 34d kube-system pod/calico-node-clp2d 1/1 Running 0 34d kube-system pod/calico-node-mk6d2 1/1 Running 1 (34d ago) 80d kube-system pod/calico-node-w6znf 1/1 Running 0 34d kube-system pod/coredns-64897985d-8cmk8 1/1 Running 0 34d kube-system pod/coredns-64897985d-hg2mc 1/1 Running 0 34d kube-system pod/etcd-kubenewmaster1 1/1 Running 1 (34d ago) 47d kube-system pod/kube-apiserver-kubenewmaster1 1/1 Running 1 (34d ago) 47d kube-system pod/kube-controller-manager-kubenewmaster1 1/1 Running 1 (34d ago) 47d kube-system pod/kube-proxy-h69k5 1/1 Running 0 34d kube-system pod/kube-proxy-r2lls 1/1 Running 0 34d kube-system pod/kube-proxy-tchz7 1/1 Running 1 (34d ago) 47d kube-system pod/kube-scheduler-kubenewmaster1 1/1 Running 1 (34d ago) 47d kube-system pod/metrics-server-5549448c75-9ln74 1/1 Running 1 (34d ago) 34d kube-system pod/metrics-server-5549448c75-j4s9n 1/1 Running 1 (34d ago) 34d metallb-system pod/controller-8487b756b9-fqqgs 1/1 Running 0 34d metallb-system pod/speaker-2cgdg 1/1 Running 1 (34d ago) 34d metallb-system pod/speaker-hg9pc 1/1 Running 1 (34d ago) 34d metallb-system pod/speaker-sdppp 1/1 Running 2 (34d ago) 78d newrelic pod/newrelic-bundle-kube-state-metrics-857949d85-8hnlq 1/1 Running 1 (34d ago) 34d newrelic pod/newrelic-bundle-newrelic-infrastructure-g5rf5 1/1 Running 1 (34d ago) 34d newrelic pod/newrelic-bundle-newrelic-infrastructure-l5sc2 1/1 Running 5 (34d ago) 73d newrelic pod/newrelic-bundle-newrelic-infrastructure-lv97l 1/1 Running 1 (34d ago) 34d newrelic pod/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx 2/2 Running 0 34d newrelic pod/newrelic-bundle-nri-metadata-injection-55865fd795-zgmwt 1/1 Running 0 34d NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/beauty-service ClusterIP 10.110.24.120 <none> 80/TCP 52d default service/default-backend-svc NodePort 10.98.72.7 <none> 80:32325/TCP 78d default service/food-service ClusterIP 10.98.132.161 <none> 80/TCP 52d default service/food-svc NodePort 10.98.251.13 <none> 80:30516/TCP 78d default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 80d default service/test-configmap-pod-svc NodePort 10.109.89.91 <none> 80:32091/TCP 74d default service/wear-svc NodePort 10.103.57.82 <none> 80:32519/TCP 78d ingress-nginx service/ingress-nginx-controller LoadBalancer 10.109.224.109 192.168.1.253 80:30985/TCP,443:32160/TCP 79d ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.101.176.203 <none> 443/TCP 79d kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 80d kube-system service/metrics-server ClusterIP 10.109.152.9 <none> 443/TCP 73d newrelic service/newrelic-bundle-kube-state-metrics ClusterIP 10.105.244.136 <none> 8080/TCP 73d newrelic service/newrelic-bundle-nri-metadata-injection ClusterIP 10.104.75.83 <none> 443/TCP 73d NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/calico-node 3 3 3 3 3 kubernetes.io/os=linux 80d kube-system daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 80d metallb-system daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 78d newrelic daemonset.apps/newrelic-bundle-newrelic-infrastructure 3 3 3 3 3 <none> 73d NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/beauty-deploy 2/2 2 2 52d default deployment.apps/food-deploy 3/3 3 3 52d default deployment.apps/nfs-subdir-external-provisioner 1/1 1 1 75d default deployment.apps/patch-test 2/2 2 2 75d default deployment.apps/pvc-deploy 1/1 1 1 51d ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 79d kube-system deployment.apps/calico-kube-controllers 1/1 1 1 80d kube-system deployment.apps/coredns 2/2 2 2 80d kube-system deployment.apps/metrics-server 2/2 2 2 73d metallb-system deployment.apps/controller 1/1 1 1 78d newrelic deployment.apps/newrelic-bundle-kube-state-metrics 1/1 1 1 73d newrelic deployment.apps/newrelic-bundle-nri-kube-events 1/1 1 1 73d newrelic deployment.apps/newrelic-bundle-nri-metadata-injection 1/1 1 1 73d NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/beauty-deploy-7d59f6c75f 2 2 2 52d default replicaset.apps/food-deploy-54544fc85d 3 3 3 52d default replicaset.apps/food-deploy-5f46fbf467 0 0 0 52d default replicaset.apps/food-deploy-6856f7c899 0 0 0 52d default replicaset.apps/food-deploy-6f75b7b466 0 0 0 52d default replicaset.apps/nfs-subdir-external-provisioner-9c9c5866f 1 1 1 75d default replicaset.apps/patch-test-6bd9658956 2 2 2 75d default replicaset.apps/pvc-deploy-77f7d456bf 1 1 1 51d ingress-nginx replicaset.apps/ingress-nginx-controller-54d8b558d4 1 1 1 79d kube-system replicaset.apps/calico-kube-controllers-958545d87 1 1 1 80d kube-system replicaset.apps/coredns-64897985d 2 2 2 47d kube-system replicaset.apps/coredns-78fcd69978 0 0 0 80d kube-system replicaset.apps/metrics-server-5549448c75 2 2 2 73d kube-system replicaset.apps/metrics-server-7dc4d5d9cf 0 0 0 73d metallb-system replicaset.apps/controller-8487b756b9 1 1 1 78d newrelic replicaset.apps/newrelic-bundle-kube-state-metrics-857949d85 1 1 1 73d newrelic replicaset.apps/newrelic-bundle-nri-kube-events-6d57ffcb4f 1 1 1 73d newrelic replicaset.apps/newrelic-bundle-nri-metadata-injection-55865fd795 1 1 1 73d NAMESPACE NAME COMPLETIONS DURATION AGE ingress-nginx job.batch/ingress-nginx-admission-create 1/1 15s 79d ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 15s 79d
利用可能バージョンの確認
kubeuser@kubenewmaster1:~$ sudo apt update [sudo] password for kubeuser: Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB] Get:4 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB] Hit:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [16.3 kB] Get:8 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1750 kB] Get:9 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB] Get:10 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [921 kB] Get:11 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.7 kB] Get:12 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1422 kB] Get:13 http://jp.archive.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB] Get:14 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB] Get:15 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [702 kB] Get:16 http://jp.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [125 kB] Get:17 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB] Hit:18 https://baltocdn.com/helm/stable/debian all InRelease Fetched 5637 kB in 6s (981 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 68 packages can be upgraded. Run 'apt list --upgradable' to see them. kubeuser@kubenewmaster1:~$ sudo apt-cache madison kubeadm kubeadm | 1.24.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.20-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.19-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.18-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.4-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.7-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.1-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.0-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.3-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.5.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
コントロールプレーンのアップグレード
コントロールプレーンをv1.24へアップグレードします。
まずはkubeadm。
kubeuser@kubenewmaster1:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.24.0-00 && sudo apt-mark hold kubeadm Canceled hold on kubeadm. Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease Hit:3 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease Hit:4 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease Hit:5 https://download.docker.com/linux/ubuntu focal InRelease Get:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B] Hit:7 https://baltocdn.com/helm/stable/debian all InRelease Fetched 9383 B in 2s (5616 B/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: cri-tools The following packages will be upgraded: cri-tools kubeadm 2 upgraded, 0 newly installed, 0 to remove and 66 not upgraded. Need to get 24.3 MB of archives. After this operation, 13.2 MB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.0-00 [9003 kB] Fetched 24.3 MB in 3s (8474 kB/s) (Reading database ... 144846 files and directories currently installed.) Preparing to unpack .../cri-tools_1.23.0-00_amd64.deb ... Unpacking cri-tools (1.23.0-00) over (1.19.0-00) ... Preparing to unpack .../kubeadm_1.24.0-00_amd64.deb ... Unpacking kubeadm (1.24.0-00) over (1.23.0-00) ... Setting up cri-tools (1.23.0-00) ... Setting up kubeadm (1.24.0-00) ... kubeadm set on hold. kubeuser@kubenewmaster1:~$ kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"} kubeuser@kubenewmaster1:~$ sudo kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' W0505 23:54:46.598205 1201192 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration! [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.23.0 [upgrade/versions] kubeadm version: v1.24.0 [upgrade/versions] Target version: v1.24.0 [upgrade/versions] Latest version in the v1.23 series: v1.23.6 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 3 x v1.23.0 v1.23.6 Upgrade to the latest version in the v1.23 series: COMPONENT CURRENT TARGET kube-apiserver v1.23.0 v1.23.6 kube-controller-manager v1.23.0 v1.23.6 kube-scheduler v1.23.0 v1.23.6 kube-proxy v1.23.0 v1.23.6 CoreDNS v1.8.6 v1.8.6 etcd 3.5.1-0 3.5.3-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.23.6 _____________________________________________________________________ Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 3 x v1.23.0 v1.24.0 Upgrade to the latest stable version: COMPONENT CURRENT TARGET kube-apiserver v1.23.0 v1.24.0 kube-controller-manager v1.23.0 v1.24.0 kube-scheduler v1.23.0 v1.24.0 kube-proxy v1.23.0 v1.24.0 CoreDNS v1.8.6 v1.8.6 etcd 3.5.1-0 3.5.3-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.24.0 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________ kubeuser@kubenewmaster1:~$ sudo kubeadm upgrade apply v1.24.0 [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' W0505 23:55:54.415618 1202068 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration! [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.24.0" [upgrade/versions] Cluster version: v1.23.0 [upgrade/versions] kubeadm version: v1.24.0 [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull' [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.0" (timeout: 5m0s)... [upgrade/etcd] Upgrading to TLS for etcd [upgrade/staticpods] Preparing for "etcd" upgrade [upgrade/staticpods] Renewing etcd-server certificate [upgrade/staticpods] Renewing etcd-peer certificate [upgrade/staticpods] Renewing etcd-healthcheck-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/etcd.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) [apiclient] Found 1 Pods for label selector component=etcd [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4262380187" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Renewing apiserver-etcd-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) [apiclient] Found 1 Pods for label selector component=kube-apiserver [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade [upgrade/staticpods] Renewing controller-manager.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-controller-manager.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) [apiclient] Found 1 Pods for label selector component=kube-controller-manager [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Preparing for "kube-scheduler" upgrade [upgrade/staticpods] Renewing scheduler.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-scheduler.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) [apiclient] Found 1 Pods for label selector component=kube-scheduler [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes. [upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes. [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.0". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.23.0 kubenewworker1 Ready <none> 34d v1.23.0 kubenewworker2 Ready <none> 34d v1.23.0
そして、コントロールプレーン(Master Node)で稼働しているpodなどの退避。
kubeuser@kubenewmaster1:~$ kubectl drain kubenewmaster1 --ignore-daemonsets node/kubenewmaster1 cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-mk6d2, kube-system/kube-proxy-9sq4j, metallb-system/speaker-sdppp, newrelic/newrelic-bundle-newrelic-infrastructure-l5sc2 evicting pod kube-system/coredns-64897985d-hg2mc evicting pod kube-system/calico-kube-controllers-958545d87-gp64m evicting pod kube-system/coredns-64897985d-8cmk8 pod/coredns-64897985d-hg2mc evicted pod/calico-kube-controllers-958545d87-gp64m evicted pod/coredns-64897985d-8cmk8 evicted node/kubenewmaster1 drained
続いて、kubeletとkubectlの更新。
kubeuser@kubenewmaster1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.24.0-00 kubectl=1.24.0-00 && sudo apt-mark hold kubelet kubectl Canceled hold on kubelet. Canceled hold on kubectl. Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease Hit:3 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease Hit:4 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease Hit:5 https://download.docker.com/linux/ubuntu focal InRelease Hit:7 https://baltocdn.com/helm/stable/debian all InRelease Hit:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 64 not upgraded. Need to get 28.6 MB of archives. After this operation, 9032 kB disk space will be freed. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.0-00 [9316 kB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.0-00 [19.2 MB] Fetched 28.6 MB in 3s (8598 kB/s) (Reading database ... 144846 files and directories currently installed.) Preparing to unpack .../kubectl_1.24.0-00_amd64.deb ... Unpacking kubectl (1.24.0-00) over (1.23.0-00) ... Preparing to unpack .../kubelet_1.24.0-00_amd64.deb ... Unpacking kubelet (1.24.0-00) over (1.23.0-00) ... Setting up kubectl (1.24.0-00) ... Setting up kubelet (1.24.0-00) ... kubelet set on hold. kubectl set on hold. kubeuser@kubenewmaster1:~$ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"} kubeuser@kubenewmaster1:~$ kubelet --version Kubernetes v1.24.0 kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready,SchedulingDisabled control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.23.0 kubenewworker2 Ready <none> 34d v1.23.0
systemdのユニットファイルの変更をsystemdに反映します。
kubeuser@kubenewmaster1:~$ sudo systemctl daemon-reload kubeuser@kubenewmaster1:~$ sudo systemctl restart kubelet kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready,SchedulingDisabled control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.23.0 kubenewworker2 Ready <none> 34d v1.23.0
仕上げに、コントロールプレーンをuncordon(podなどの配置可能)にします。
kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewmaster1 node/kubenewmaster1 uncordoned kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.23.0 kubenewworker2 Ready <none> 34d v1.23.0
これで、コントロールプレーンのアップグレードは完了。
なお、複数台のコントロールプレーンがある場合は、
基本的には1台目と同じですが、
"kubeadm upgrade apply"の代わりに "kubectl upgrade node"を使います。
ワーカーノードのアップグレード
続いて、ワーカーノードをアップグレードします。
今回のクラスターでは2台あるので、1台ずつ実施ます。
まず、kubeadmの更新。
kubeuser@kubenewworker1:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.24.0-00 && sudo apt-mark hold kubeadm [sudo] password for kubeuser: Canceled hold on kubeadm. Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB] Get:5 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B] Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB] Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [16.3 kB] Get:8 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1750 kB] Get:9 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB] Get:10 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [921 kB] Get:11 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1422 kB] Get:12 http://jp.archive.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB] Get:13 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB] Get:14 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [702 kB] Get:15 http://jp.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [125 kB] Fetched 5611 kB in 4s (1399 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: cri-tools The following packages will be upgraded: cri-tools kubeadm 2 upgraded, 0 newly installed, 0 to remove and 68 not upgraded. Need to get 24.3 MB of archives. After this operation, 13.2 MB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.0-00 [9003 kB] Fetched 24.3 MB in 3s (8872 kB/s) (Reading database ... 144772 files and directories currently installed.) Preparing to unpack .../cri-tools_1.23.0-00_amd64.deb ... Unpacking cri-tools (1.23.0-00) over (1.19.0-00) ... Preparing to unpack .../kubeadm_1.24.0-00_amd64.deb ... Unpacking kubeadm (1.24.0-00) over (1.23.0-00) ... Setting up cri-tools (1.23.0-00) ... Setting up kubeadm (1.24.0-00) ... kubeadm set on hold. kubeuser@kubenewworker1:~$ sudo kubeadm upgrade node [upgrade] Reading configuration from the cluster... [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks [preflight] Skipping prepull. Not a control plane node. [upgrade] Skipping phase. Not a control plane node. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
続いて、ワーカーノードで稼働中のpodなどの退避。
この操作は、コントロールプレーン(Master Node)で実施するので、間違えないように注意します。
kubeuser@kubenewmaster1:~$ kubectl drain kubenewworker1 --ignore-daemonsets node/kubenewworker1 cordoned error: unable to drain node "kubenewworker1" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-5549448c75-j4s9n, newrelic/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx, continuing command... There are pending nodes to be drained: kubenewworker1 cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-5549448c75-j4s9n, newrelic/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx
再び、ワーカーノードでの操作に戻り、kubeletとkubectlを更新。
kubeuser@kubenewworker1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.24.0-00 kubectl=1.24.0-00 && sudo apt-mark hold kubelet kubectl Canceled hold on kubelet. Canceled hold on kubectl. Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Hit:3 https://download.docker.com/linux/ubuntu focal InRelease Get:4 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB] Hit:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Fetched 336 kB in 2s (216 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 66 not upgraded. Need to get 28.6 MB of archives. After this operation, 9032 kB disk space will be freed. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.0-00 [9316 kB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.0-00 [19.2 MB] Fetched 28.6 MB in 3s (9111 kB/s) (Reading database ... 144772 files and directories currently installed.) Preparing to unpack .../kubectl_1.24.0-00_amd64.deb ... Unpacking kubectl (1.24.0-00) over (1.23.0-00) ... Preparing to unpack .../kubelet_1.24.0-00_amd64.deb ... Unpacking kubelet (1.24.0-00) over (1.23.0-00) ... Setting up kubectl (1.24.0-00) ... Setting up kubelet (1.24.0-00) ... kubelet set on hold. kubectl set on hold. kubeuser@kubenewworker1:~$ sudo systemctl daemon-reload kubeuser@kubenewworker1:~$ sudo systemctl restart kubelet kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.24.0 kubenewworker1 Ready,SchedulingDisabled <none> 34d v1.24.0 kubenewworker2 Ready <none> 34d v1.23.0
再びコントロールプレーン(Master Node)での操作に切替、
ワーカーノードでpod配置を可能(uncordon)にします。
kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewworker1 node/kubenewworker1 uncordoned kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.24.0 kubenewworker2 Ready <none> 34d v1.23.0
これで1台目のワーカーノードのアップグレードは完了。
2台目以降も同様の方法でアップグレードすれば、OKです。
kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.24.0 kubenewworker2 Ready,SchedulingDisabled <none> 34d v1.24.0 kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewworker2 node/kubenewworker2 uncordoned kubeuser@kubenewmaster1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION kubenewmaster1 Ready control-plane 80d v1.24.0 kubenewworker1 Ready <none> 34d v1.24.0 kubenewworker2 Ready <none> 34d v1.24.0
アップグレード後の確認
アップグレード後のリソース稼働状況を確認します。
ワーカーノードの退避の際に、一部errorが出てましたが大丈夫そうです。
kubeuser@kubenewmaster1:~$ kubectl get all --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default pod/beauty-deploy-7d59f6c75f-hbk8b 1/1 Running 0 36d default pod/beauty-deploy-7d59f6c75f-xp9fd 1/1 Running 0 36d default pod/food-deploy-54544fc85d-5k5h9 1/1 Running 0 36d default pod/food-deploy-54544fc85d-hhn4r 1/1 Running 0 36d default pod/food-deploy-54544fc85d-smn2l 1/1 Running 0 36d default pod/nfs-subdir-external-provisioner-9c9c5866f-2n2l6 1/1 Running 5 (30h ago) 36d default pod/patch-test-6bd9658956-npz4m 1/1 Running 0 36d default pod/patch-test-6bd9658956-qcjhj 1/1 Running 0 36d default pod/pvc-deploy-77f7d456bf-jnp57 1/1 Running 0 36d ingress-nginx pod/ingress-nginx-controller-54d8b558d4-nhzd6 1/1 Running 0 36d kube-system pod/calico-kube-controllers-958545d87-2wzxh 1/1 Running 0 30h kube-system pod/calico-node-clp2d 1/1 Running 0 36d kube-system pod/calico-node-mk6d2 1/1 Running 1 (36d ago) 81d kube-system pod/calico-node-w6znf 1/1 Running 0 36d kube-system pod/coredns-64897985d-6cscg 1/1 Running 0 30h kube-system pod/coredns-64897985d-wjpxv 1/1 Running 0 30h kube-system pod/etcd-kubenewmaster1 1/1 Running 0 30h kube-system pod/kube-apiserver-kubenewmaster1 1/1 Running 0 30h kube-system pod/kube-controller-manager-kubenewmaster1 1/1 Running 0 30h kube-system pod/kube-proxy-9sq4j 1/1 Running 0 30h kube-system pod/kube-proxy-n97dz 1/1 Running 0 30h kube-system pod/kube-proxy-pfdgz 1/1 Running 0 30h kube-system pod/kube-scheduler-kubenewmaster1 1/1 Running 0 30h kube-system pod/metrics-server-5549448c75-9ln74 1/1 Running 1 (36d ago) 36d kube-system pod/metrics-server-5549448c75-j4s9n 1/1 Running 1 (36d ago) 36d metallb-system pod/controller-8487b756b9-fqqgs 1/1 Running 0 36d metallb-system pod/speaker-2cgdg 1/1 Running 1 (36d ago) 36d metallb-system pod/speaker-hg9pc 1/1 Running 1 (36d ago) 36d metallb-system pod/speaker-sdppp 1/1 Running 2 (36d ago) 80d newrelic pod/newrelic-bundle-kube-state-metrics-857949d85-8hnlq 1/1 Running 1 (36d ago) 36d newrelic pod/newrelic-bundle-newrelic-infrastructure-g5rf5 1/1 Running 1 (36d ago) 36d newrelic pod/newrelic-bundle-newrelic-infrastructure-l5sc2 1/1 Running 5 (36d ago) 74d newrelic pod/newrelic-bundle-newrelic-infrastructure-lv97l 1/1 Running 1 (36d ago) 36d newrelic pod/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx 2/2 Running 0 36d newrelic pod/newrelic-bundle-nri-metadata-injection-55865fd795-zgmwt 1/1 Running 0 36d NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/beauty-service ClusterIP 10.110.24.120 <none> 80/TCP 53d default service/default-backend-svc NodePort 10.98.72.7 <none> 80:32325/TCP 79d default service/food-service ClusterIP 10.98.132.161 <none> 80/TCP 53d default service/food-svc NodePort 10.98.251.13 <none> 80:30516/TCP 79d default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 81d default service/test-configmap-pod-svc NodePort 10.109.89.91 <none> 80:32091/TCP 75d default service/wear-svc NodePort 10.103.57.82 <none> 80:32519/TCP 79d ingress-nginx service/ingress-nginx-controller LoadBalancer 10.109.224.109 192.168.1.253 80:30985/TCP,443:32160/TCP 80d ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.101.176.203 <none> 443/TCP 80d kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 81d kube-system service/metrics-server ClusterIP 10.109.152.9 <none> 443/TCP 74d newrelic service/newrelic-bundle-kube-state-metrics ClusterIP 10.105.244.136 <none> 8080/TCP 74d newrelic service/newrelic-bundle-nri-metadata-injection ClusterIP 10.104.75.83 <none> 443/TCP 74d NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/calico-node 3 3 3 3 3 kubernetes.io/os=linux 81d kube-system daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 81d metallb-system daemonset.apps/speaker 2 2 2 2 2 kubernetes.io/os=linux 80d newrelic daemonset.apps/newrelic-bundle-newrelic-infrastructure 3 3 3 3 3 <none> 74d NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/beauty-deploy 2/2 2 2 53d default deployment.apps/food-deploy 3/3 3 3 53d default deployment.apps/nfs-subdir-external-provisioner 1/1 1 1 76d default deployment.apps/patch-test 2/2 2 2 76d default deployment.apps/pvc-deploy 1/1 1 1 52d ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 80d kube-system deployment.apps/calico-kube-controllers 1/1 1 1 81d kube-system deployment.apps/coredns 2/2 2 2 81d kube-system deployment.apps/metrics-server 2/2 2 2 74d metallb-system deployment.apps/controller 1/1 1 1 80d newrelic deployment.apps/newrelic-bundle-kube-state-metrics 1/1 1 1 74d newrelic deployment.apps/newrelic-bundle-nri-kube-events 1/1 1 1 74d newrelic deployment.apps/newrelic-bundle-nri-metadata-injection 1/1 1 1 74d NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/beauty-deploy-7d59f6c75f 2 2 2 53d default replicaset.apps/food-deploy-54544fc85d 3 3 3 53d default replicaset.apps/food-deploy-5f46fbf467 0 0 0 53d default replicaset.apps/food-deploy-6856f7c899 0 0 0 53d default replicaset.apps/food-deploy-6f75b7b466 0 0 0 53d default replicaset.apps/nfs-subdir-external-provisioner-9c9c5866f 1 1 1 76d default replicaset.apps/patch-test-6bd9658956 2 2 2 76d default replicaset.apps/pvc-deploy-77f7d456bf 1 1 1 52d ingress-nginx replicaset.apps/ingress-nginx-controller-54d8b558d4 1 1 1 80d kube-system replicaset.apps/calico-kube-controllers-958545d87 1 1 1 81d kube-system replicaset.apps/coredns-64897985d 2 2 2 48d kube-system replicaset.apps/coredns-78fcd69978 0 0 0 81d kube-system replicaset.apps/metrics-server-5549448c75 2 2 2 74d kube-system replicaset.apps/metrics-server-7dc4d5d9cf 0 0 0 74d metallb-system replicaset.apps/controller-8487b756b9 1 1 1 80d newrelic replicaset.apps/newrelic-bundle-kube-state-metrics-857949d85 1 1 1 74d newrelic replicaset.apps/newrelic-bundle-nri-kube-events-6d57ffcb4f 1 1 1 74d newrelic replicaset.apps/newrelic-bundle-nri-metadata-injection-55865fd795 1 1 1 74d NAMESPACE NAME COMPLETIONS DURATION AGE ingress-nginx job.batch/ingress-nginx-admission-create 1/1 15s 80d ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 15s 80d
おわりに
kubernetesを触り始めた初期はクラスターのアップグレードに対して非常に大変だなぁと思っていたのですが、
少しkubernetesと親しくなれたのかあまり抵抗がなくなってきましたw
kubernetesは、コミニティにおいて最新の3バージョン(現在なら、v1.24・v1.23・v1.22)がサポート対象となり、各バージョンも約1年間のパッチ(マイナーバージョン)リリースされるので比較的早いサイクルでのアップグレード作業が必要になります。
https://kubernetes.io/releases/release/
https://kubernetes.io/releases/
そのため、
・開発状況の確認
・不具合状況の確認
・アップグレード検証
・本番環境でのアップグレード作業
なども早いサイクルで実施することになります。