Opensourcetechブログ

OpensourcetechによるNGINX/Kubernetes/Zabbix/Neo4j/Linuxなどオープンソース技術に関するブログです。

kubeadmで構築したkubernetesクラスターのアップグレード(v1.23 to v1.24)


LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。


はじめに


https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/

5/3にkubernetesの新バージョンStargazerがリリースされました。
なので手元にあるv1.23のKubernetesクラスターを、v1.24にアップグレードします。

アップグレード方法は、以下の公式ドキュメントに記載されています。
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/



kuberneteesクラスターの確認
kubernetesクラスターのバージョンは、1.23.0です。

kubeuser@kubenewmaster1:~$ kubectl get nodes -o wide
NAME             STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kubenewmaster1   Ready    control-plane,master   80d   v1.23.0   192.168.1.251   <none>        Ubuntu 20.04.3 LTS   5.4.0-105-generic   containerd://1.4.12
kubenewworker1   Ready    <none>                 34d   v1.23.0   192.168.1.252   <none>        Ubuntu 20.04.3 LTS   5.4.0-105-generic   containerd://1.4.12
kubenewworker2   Ready    <none>                 34d   v1.23.0   192.168.1.249   <none>        Ubuntu 20.04.3 LTS   5.4.0-104-generic   containerd://1.4.12

kubeuser@kubenewmaster1:~$ kubectl get all --all-namespaces
NAMESPACE        NAME                                                          READY   STATUS    RESTARTS      AGE
default          pod/beauty-deploy-7d59f6c75f-hbk8b                            1/1     Running   0             34d
default          pod/beauty-deploy-7d59f6c75f-xp9fd                            1/1     Running   0             34d
default          pod/food-deploy-54544fc85d-5k5h9                              1/1     Running   0             34d
default          pod/food-deploy-54544fc85d-hhn4r                              1/1     Running   0             34d
default          pod/food-deploy-54544fc85d-smn2l                              1/1     Running   0             34d
default          pod/nfs-subdir-external-provisioner-9c9c5866f-2n2l6           1/1     Running   0             34d
default          pod/patch-test-6bd9658956-npz4m                               1/1     Running   0             34d
default          pod/patch-test-6bd9658956-qcjhj                               1/1     Running   0             34d
default          pod/pvc-deploy-77f7d456bf-jnp57                               1/1     Running   0             34d
ingress-nginx    pod/ingress-nginx-controller-54d8b558d4-nhzd6                 1/1     Running   0             34d
kube-system      pod/calico-kube-controllers-958545d87-gp64m                   1/1     Running   0             34d
kube-system      pod/calico-node-clp2d                                         1/1     Running   0             34d
kube-system      pod/calico-node-mk6d2                                         1/1     Running   1 (34d ago)   80d
kube-system      pod/calico-node-w6znf                                         1/1     Running   0             34d
kube-system      pod/coredns-64897985d-8cmk8                                   1/1     Running   0             34d
kube-system      pod/coredns-64897985d-hg2mc                                   1/1     Running   0             34d
kube-system      pod/etcd-kubenewmaster1                                       1/1     Running   1 (34d ago)   47d
kube-system      pod/kube-apiserver-kubenewmaster1                             1/1     Running   1 (34d ago)   47d
kube-system      pod/kube-controller-manager-kubenewmaster1                    1/1     Running   1 (34d ago)   47d
kube-system      pod/kube-proxy-h69k5                                          1/1     Running   0             34d
kube-system      pod/kube-proxy-r2lls                                          1/1     Running   0             34d
kube-system      pod/kube-proxy-tchz7                                          1/1     Running   1 (34d ago)   47d
kube-system      pod/kube-scheduler-kubenewmaster1                             1/1     Running   1 (34d ago)   47d
kube-system      pod/metrics-server-5549448c75-9ln74                           1/1     Running   1 (34d ago)   34d
kube-system      pod/metrics-server-5549448c75-j4s9n                           1/1     Running   1 (34d ago)   34d
metallb-system   pod/controller-8487b756b9-fqqgs                               1/1     Running   0             34d
metallb-system   pod/speaker-2cgdg                                             1/1     Running   1 (34d ago)   34d
metallb-system   pod/speaker-hg9pc                                             1/1     Running   1 (34d ago)   34d
metallb-system   pod/speaker-sdppp                                             1/1     Running   2 (34d ago)   78d
newrelic         pod/newrelic-bundle-kube-state-metrics-857949d85-8hnlq        1/1     Running   1 (34d ago)   34d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-g5rf5             1/1     Running   1 (34d ago)   34d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-l5sc2             1/1     Running   5 (34d ago)   73d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-lv97l             1/1     Running   1 (34d ago)   34d
newrelic         pod/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx          2/2     Running   0             34d
newrelic         pod/newrelic-bundle-nri-metadata-injection-55865fd795-zgmwt   1/1     Running   0             34d

NAMESPACE       NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
default         service/beauty-service                           ClusterIP      10.110.24.120    <none>          80/TCP                       52d
default         service/default-backend-svc                      NodePort       10.98.72.7       <none>          80:32325/TCP                 78d
default         service/food-service                             ClusterIP      10.98.132.161    <none>          80/TCP                       52d
default         service/food-svc                                 NodePort       10.98.251.13     <none>          80:30516/TCP                 78d
default         service/kubernetes                               ClusterIP      10.96.0.1        <none>          443/TCP                      80d
default         service/test-configmap-pod-svc                   NodePort       10.109.89.91     <none>          80:32091/TCP                 74d
default         service/wear-svc                                 NodePort       10.103.57.82     <none>          80:32519/TCP                 78d
ingress-nginx   service/ingress-nginx-controller                 LoadBalancer   10.109.224.109   192.168.1.253   80:30985/TCP,443:32160/TCP   79d
ingress-nginx   service/ingress-nginx-controller-admission       ClusterIP      10.101.176.203   <none>          443/TCP                      79d
kube-system     service/kube-dns                                 ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP       80d
kube-system     service/metrics-server                           ClusterIP      10.109.152.9     <none>          443/TCP                      73d
newrelic        service/newrelic-bundle-kube-state-metrics       ClusterIP      10.105.244.136   <none>          8080/TCP                     73d
newrelic        service/newrelic-bundle-nri-metadata-injection   ClusterIP      10.104.75.83     <none>          443/TCP                      73d

NAMESPACE        NAME                                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system      daemonset.apps/calico-node                               3         3         3       3            3           kubernetes.io/os=linux   80d
kube-system      daemonset.apps/kube-proxy                                3         3         3       3            3           kubernetes.io/os=linux   80d
metallb-system   daemonset.apps/speaker                                   3         3         3       3            3           kubernetes.io/os=linux   78d
newrelic         daemonset.apps/newrelic-bundle-newrelic-infrastructure   3         3         3       3            3           <none>                   73d

NAMESPACE        NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/beauty-deploy                            2/2     2            2           52d
default          deployment.apps/food-deploy                              3/3     3            3           52d
default          deployment.apps/nfs-subdir-external-provisioner          1/1     1            1           75d
default          deployment.apps/patch-test                               2/2     2            2           75d
default          deployment.apps/pvc-deploy                               1/1     1            1           51d
ingress-nginx    deployment.apps/ingress-nginx-controller                 1/1     1            1           79d
kube-system      deployment.apps/calico-kube-controllers                  1/1     1            1           80d
kube-system      deployment.apps/coredns                                  2/2     2            2           80d
kube-system      deployment.apps/metrics-server                           2/2     2            2           73d
metallb-system   deployment.apps/controller                               1/1     1            1           78d
newrelic         deployment.apps/newrelic-bundle-kube-state-metrics       1/1     1            1           73d
newrelic         deployment.apps/newrelic-bundle-nri-kube-events          1/1     1            1           73d
newrelic         deployment.apps/newrelic-bundle-nri-metadata-injection   1/1     1            1           73d

NAMESPACE        NAME                                                                DESIRED   CURRENT   READY   AGE
default          replicaset.apps/beauty-deploy-7d59f6c75f                            2         2         2       52d
default          replicaset.apps/food-deploy-54544fc85d                              3         3         3       52d
default          replicaset.apps/food-deploy-5f46fbf467                              0         0         0       52d
default          replicaset.apps/food-deploy-6856f7c899                              0         0         0       52d
default          replicaset.apps/food-deploy-6f75b7b466                              0         0         0       52d
default          replicaset.apps/nfs-subdir-external-provisioner-9c9c5866f           1         1         1       75d
default          replicaset.apps/patch-test-6bd9658956                               2         2         2       75d
default          replicaset.apps/pvc-deploy-77f7d456bf                               1         1         1       51d
ingress-nginx    replicaset.apps/ingress-nginx-controller-54d8b558d4                 1         1         1       79d
kube-system      replicaset.apps/calico-kube-controllers-958545d87                   1         1         1       80d
kube-system      replicaset.apps/coredns-64897985d                                   2         2         2       47d
kube-system      replicaset.apps/coredns-78fcd69978                                  0         0         0       80d
kube-system      replicaset.apps/metrics-server-5549448c75                           2         2         2       73d
kube-system      replicaset.apps/metrics-server-7dc4d5d9cf                           0         0         0       73d
metallb-system   replicaset.apps/controller-8487b756b9                               1         1         1       78d
newrelic         replicaset.apps/newrelic-bundle-kube-state-metrics-857949d85        1         1         1       73d
newrelic         replicaset.apps/newrelic-bundle-nri-kube-events-6d57ffcb4f          1         1         1       73d
newrelic         replicaset.apps/newrelic-bundle-nri-metadata-injection-55865fd795   1         1         1       73d

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           15s        79d
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           15s        79d




利用可能バージョンの確認

kubeuser@kubenewmaster1:~$ sudo apt update
[sudo] password for kubeuser:
Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Get:4 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [16.3 kB]
Get:8 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1750 kB]
Get:9 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB]
Get:10 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [921 kB]
Get:11 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.7 kB]
Get:12 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1422 kB]
Get:13 http://jp.archive.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB]
Get:14 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB]
Get:15 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [702 kB]
Get:16 http://jp.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [125 kB]
Get:17 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.4 kB]
Hit:18 https://baltocdn.com/helm/stable/debian all InRelease
Fetched 5637 kB in 6s (981 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
68 packages can be upgraded. Run 'apt list --upgradable' to see them.

kubeuser@kubenewmaster1:~$ sudo apt-cache madison kubeadm
   kubeadm |  1.24.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.21.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.21.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.21.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.21.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.20.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.20.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.19.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.19.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.20-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.19-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.18-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.18.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.4-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.18.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.17.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.7-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.17.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.16.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.16.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.15.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.15.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.15.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.15.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.14.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.14.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.13.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.13.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.13.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.13.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.12.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.12.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.11.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.11.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.10.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.10.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.10.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.10.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.10.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.9.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.9.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.9.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.8.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.1-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.0-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.8.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.7.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.7.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.7.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.7.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.7.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.3-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.7.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.6.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.6.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.6.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.6.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.6.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |   1.5.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages




コントロールプレーンのアップグレード
コントロールプレーンをv1.24へアップグレードします。
まずはkubeadm。

kubeuser@kubenewmaster1:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.24.0-00 && sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
Get:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Hit:7 https://baltocdn.com/helm/stable/debian all InRelease
Fetched 9383 B in 2s (5616 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 66 not upgraded.
Need to get 24.3 MB of archives.
After this operation, 13.2 MB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.0-00 [9003 kB]
Fetched 24.3 MB in 3s (8474 kB/s)
(Reading database ... 144846 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.23.0-00_amd64.deb ...
Unpacking cri-tools (1.23.0-00) over (1.19.0-00) ...
Preparing to unpack .../kubeadm_1.24.0-00_amd64.deb ...
Unpacking kubeadm (1.24.0-00) over (1.23.0-00) ...
Setting up cri-tools (1.23.0-00) ...
Setting up kubeadm (1.24.0-00) ...
kubeadm set on hold.

kubeuser@kubenewmaster1:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

kubeuser@kubenewmaster1:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0505 23:54:46.598205 1201192 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.0
[upgrade/versions] kubeadm version: v1.24.0
[upgrade/versions] Target version: v1.24.0
[upgrade/versions] Latest version in the v1.23 series: v1.23.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     3 x v1.23.0   v1.23.6

Upgrade to the latest version in the v1.23 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.23.0   v1.23.6
kube-controller-manager   v1.23.0   v1.23.6
kube-scheduler            v1.23.0   v1.23.6
kube-proxy                v1.23.0   v1.23.6
CoreDNS                   v1.8.6    v1.8.6
etcd                      3.5.1-0   3.5.3-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.23.6

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     3 x v1.23.0   v1.24.0

Upgrade to the latest stable version:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.23.0   v1.24.0
kube-controller-manager   v1.23.0   v1.24.0
kube-scheduler            v1.23.0   v1.24.0
kube-proxy                v1.23.0   v1.24.0
CoreDNS                   v1.8.6    v1.8.6
etcd                      3.5.1-0   3.5.3-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.24.0

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________


kubeuser@kubenewmaster1:~$ sudo kubeadm upgrade apply v1.24.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0505 23:55:54.415618 1202068 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.24.0"
[upgrade/versions] Cluster version: v1.23.0
[upgrade/versions] kubeadm version: v1.24.0
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.0" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4262380187"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-05-05-23-57-30/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes.
[upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes.
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.


kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
kubenewmaster1   Ready    control-plane   80d   v1.23.0
kubenewworker1   Ready    <none>          34d   v1.23.0
kubenewworker2   Ready    <none>          34d   v1.23.0



そして、コントロールプレーン(Master Node)で稼働しているpodなどの退避。

kubeuser@kubenewmaster1:~$ kubectl drain kubenewmaster1 --ignore-daemonsets
node/kubenewmaster1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-mk6d2, kube-system/kube-proxy-9sq4j, metallb-system/speaker-sdppp, newrelic/newrelic-bundle-newrelic-infrastructure-l5sc2
evicting pod kube-system/coredns-64897985d-hg2mc
evicting pod kube-system/calico-kube-controllers-958545d87-gp64m
evicting pod kube-system/coredns-64897985d-8cmk8
pod/coredns-64897985d-hg2mc evicted
pod/calico-kube-controllers-958545d87-gp64m evicted
pod/coredns-64897985d-8cmk8 evicted
node/kubenewmaster1 drained



続いて、kubeletとkubectlの更新。

kubeuser@kubenewmaster1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.24.0-00 kubectl=1.24.0-00 && sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
Hit:7 https://baltocdn.com/helm/stable/debian all InRelease
Hit:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 64 not upgraded.
Need to get 28.6 MB of archives.
After this operation, 9032 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.0-00 [9316 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.0-00 [19.2 MB]
Fetched 28.6 MB in 3s (8598 kB/s)
(Reading database ... 144846 files and directories currently installed.)
Preparing to unpack .../kubectl_1.24.0-00_amd64.deb ...
Unpacking kubectl (1.24.0-00) over (1.23.0-00) ...
Preparing to unpack .../kubelet_1.24.0-00_amd64.deb ...
Unpacking kubelet (1.24.0-00) over (1.23.0-00) ...
Setting up kubectl (1.24.0-00) ...
Setting up kubelet (1.24.0-00) ...
kubelet set on hold.
kubectl set on hold.


kubeuser@kubenewmaster1:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
kubeuser@kubenewmaster1:~$ kubelet --version
Kubernetes v1.24.0

kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS                     ROLES           AGE   VERSION
kubenewmaster1   Ready,SchedulingDisabled   control-plane   80d   v1.24.0
kubenewworker1   Ready                      <none>          34d   v1.23.0
kubenewworker2   Ready                      <none>          34d   v1.23.0



systemdのユニットファイルの変更をsystemdに反映します。

kubeuser@kubenewmaster1:~$ sudo systemctl daemon-reload
kubeuser@kubenewmaster1:~$ sudo systemctl restart kubelet

kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS                     ROLES           AGE   VERSION
kubenewmaster1   Ready,SchedulingDisabled   control-plane   80d   v1.24.0
kubenewworker1   Ready                      <none>          34d   v1.23.0
kubenewworker2   Ready                      <none>          34d   v1.23.0



仕上げに、コントロールプレーンをuncordon(podなどの配置可能)にします。

kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewmaster1
node/kubenewmaster1 uncordoned
kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
kubenewmaster1   Ready    control-plane   80d   v1.24.0
kubenewworker1   Ready    <none>          34d   v1.23.0
kubenewworker2   Ready    <none>          34d   v1.23.0


これで、コントロールプレーンのアップグレードは完了。
なお、複数台のコントロールプレーンがある場合は、
基本的には1台目と同じですが、
"kubeadm upgrade apply"の代わりに "kubectl upgrade node"を使います。



ワーカーノードのアップグレード
続いて、ワーカーノードをアップグレードします。
今回のクラスターでは2台あるので、1台ずつ実施ます。

まず、kubeadmの更新。

kubeuser@kubenewworker1:~$ sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.24.0-00 && sudo apt-mark hold kubeadm
[sudo] password for kubeuser:
Canceled hold on kubeadm.
Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Get:5 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [16.3 kB]
Get:8 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1750 kB]
Get:9 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.0 kB]
Get:10 http://jp.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [921 kB]
Get:11 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1422 kB]
Get:12 http://jp.archive.ubuntu.com/ubuntu focal-security/main Translation-en [246 kB]
Get:13 http://jp.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.1 kB]
Get:14 http://jp.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [702 kB]
Get:15 http://jp.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [125 kB]
Fetched 5611 kB in 4s (1399 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 68 not upgraded.
Need to get 24.3 MB of archives.
After this operation, 13.2 MB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.0-00 [9003 kB]
Fetched 24.3 MB in 3s (8872 kB/s)
(Reading database ... 144772 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.23.0-00_amd64.deb ...
Unpacking cri-tools (1.23.0-00) over (1.19.0-00) ...
Preparing to unpack .../kubeadm_1.24.0-00_amd64.deb ...
Unpacking kubeadm (1.24.0-00) over (1.23.0-00) ...
Setting up cri-tools (1.23.0-00) ...
Setting up kubeadm (1.24.0-00) ...
kubeadm set on hold.

kubeuser@kubenewworker1:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.


続いて、ワーカーノードで稼働中のpodなどの退避。
この操作は、コントロールプレーン(Master Node)で実施するので、間違えないように注意します。

kubeuser@kubenewmaster1:~$ kubectl drain kubenewworker1 --ignore-daemonsets
node/kubenewworker1 cordoned
error: unable to drain node "kubenewworker1" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-5549448c75-j4s9n, newrelic/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx, continuing command...
There are pending nodes to be drained:
 kubenewworker1
cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-5549448c75-j4s9n, newrelic/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx


再び、ワーカーノードでの操作に戻り、kubeletとkubectlを更新。

kubeuser@kubenewworker1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.24.0-00 kubectl=1.24.0-00 && sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 http://jp.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:3 https://download.docker.com/linux/ubuntu focal InRelease
Get:4 http://jp.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://jp.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Fetched 336 kB in 2s (216 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 66 not upgraded.
Need to get 28.6 MB of archives.
After this operation, 9032 kB disk space will be freed.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.0-00 [9316 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.0-00 [19.2 MB]
Fetched 28.6 MB in 3s (9111 kB/s)
(Reading database ... 144772 files and directories currently installed.)
Preparing to unpack .../kubectl_1.24.0-00_amd64.deb ...
Unpacking kubectl (1.24.0-00) over (1.23.0-00) ...
Preparing to unpack .../kubelet_1.24.0-00_amd64.deb ...
Unpacking kubelet (1.24.0-00) over (1.23.0-00) ...
Setting up kubectl (1.24.0-00) ...
Setting up kubelet (1.24.0-00) ...
kubelet set on hold.
kubectl set on hold.

kubeuser@kubenewworker1:~$ sudo systemctl daemon-reload

kubeuser@kubenewworker1:~$ sudo systemctl restart kubelet

kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS                     ROLES           AGE   VERSION
kubenewmaster1   Ready                      control-plane   80d   v1.24.0
kubenewworker1   Ready,SchedulingDisabled   <none>          34d   v1.24.0
kubenewworker2   Ready                      <none>          34d   v1.23.0


再びコントロールプレーン(Master Node)での操作に切替、
ワーカーノードでpod配置を可能(uncordon)にします。

kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewworker1
node/kubenewworker1 uncordoned
kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
kubenewmaster1   Ready    control-plane   80d   v1.24.0
kubenewworker1   Ready    <none>          34d   v1.24.0
kubenewworker2   Ready    <none>          34d   v1.23.0


これで1台目のワーカーノードのアップグレードは完了。
2台目以降も同様の方法でアップグレードすれば、OKです。

kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS                     ROLES           AGE   VERSION
kubenewmaster1   Ready                      control-plane   80d   v1.24.0
kubenewworker1   Ready                      <none>          34d   v1.24.0
kubenewworker2   Ready,SchedulingDisabled   <none>          34d   v1.24.0
kubeuser@kubenewmaster1:~$ kubectl uncordon kubenewworker2
node/kubenewworker2 uncordoned
kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
kubenewmaster1   Ready    control-plane   80d   v1.24.0
kubenewworker1   Ready    <none>          34d   v1.24.0
kubenewworker2   Ready    <none>          34d   v1.24.0



アップグレード後の確認
アップグレード後のリソース稼働状況を確認します。
ワーカーノードの退避の際に、一部errorが出てましたが大丈夫そうです。

kubeuser@kubenewmaster1:~$ kubectl get all --all-namespaces
NAMESPACE        NAME                                                          READY   STATUS    RESTARTS      AGE
default          pod/beauty-deploy-7d59f6c75f-hbk8b                            1/1     Running   0             36d
default          pod/beauty-deploy-7d59f6c75f-xp9fd                            1/1     Running   0             36d
default          pod/food-deploy-54544fc85d-5k5h9                              1/1     Running   0             36d
default          pod/food-deploy-54544fc85d-hhn4r                              1/1     Running   0             36d
default          pod/food-deploy-54544fc85d-smn2l                              1/1     Running   0             36d
default          pod/nfs-subdir-external-provisioner-9c9c5866f-2n2l6           1/1     Running   5 (30h ago)   36d
default          pod/patch-test-6bd9658956-npz4m                               1/1     Running   0             36d
default          pod/patch-test-6bd9658956-qcjhj                               1/1     Running   0             36d
default          pod/pvc-deploy-77f7d456bf-jnp57                               1/1     Running   0             36d
ingress-nginx    pod/ingress-nginx-controller-54d8b558d4-nhzd6                 1/1     Running   0             36d
kube-system      pod/calico-kube-controllers-958545d87-2wzxh                   1/1     Running   0             30h
kube-system      pod/calico-node-clp2d                                         1/1     Running   0             36d
kube-system      pod/calico-node-mk6d2                                         1/1     Running   1 (36d ago)   81d
kube-system      pod/calico-node-w6znf                                         1/1     Running   0             36d
kube-system      pod/coredns-64897985d-6cscg                                   1/1     Running   0             30h
kube-system      pod/coredns-64897985d-wjpxv                                   1/1     Running   0             30h
kube-system      pod/etcd-kubenewmaster1                                       1/1     Running   0             30h
kube-system      pod/kube-apiserver-kubenewmaster1                             1/1     Running   0             30h
kube-system      pod/kube-controller-manager-kubenewmaster1                    1/1     Running   0             30h
kube-system      pod/kube-proxy-9sq4j                                          1/1     Running   0             30h
kube-system      pod/kube-proxy-n97dz                                          1/1     Running   0             30h
kube-system      pod/kube-proxy-pfdgz                                          1/1     Running   0             30h
kube-system      pod/kube-scheduler-kubenewmaster1                             1/1     Running   0             30h
kube-system      pod/metrics-server-5549448c75-9ln74                           1/1     Running   1 (36d ago)   36d
kube-system      pod/metrics-server-5549448c75-j4s9n                           1/1     Running   1 (36d ago)   36d
metallb-system   pod/controller-8487b756b9-fqqgs                               1/1     Running   0             36d
metallb-system   pod/speaker-2cgdg                                             1/1     Running   1 (36d ago)   36d
metallb-system   pod/speaker-hg9pc                                             1/1     Running   1 (36d ago)   36d
metallb-system   pod/speaker-sdppp                                             1/1     Running   2 (36d ago)   80d
newrelic         pod/newrelic-bundle-kube-state-metrics-857949d85-8hnlq        1/1     Running   1 (36d ago)   36d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-g5rf5             1/1     Running   1 (36d ago)   36d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-l5sc2             1/1     Running   5 (36d ago)   74d
newrelic         pod/newrelic-bundle-newrelic-infrastructure-lv97l             1/1     Running   1 (36d ago)   36d
newrelic         pod/newrelic-bundle-nri-kube-events-6d57ffcb4f-5jcnx          2/2     Running   0             36d
newrelic         pod/newrelic-bundle-nri-metadata-injection-55865fd795-zgmwt   1/1     Running   0             36d

NAMESPACE       NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
default         service/beauty-service                           ClusterIP      10.110.24.120    <none>          80/TCP                       53d
default         service/default-backend-svc                      NodePort       10.98.72.7       <none>          80:32325/TCP                 79d
default         service/food-service                             ClusterIP      10.98.132.161    <none>          80/TCP                       53d
default         service/food-svc                                 NodePort       10.98.251.13     <none>          80:30516/TCP                 79d
default         service/kubernetes                               ClusterIP      10.96.0.1        <none>          443/TCP                      81d
default         service/test-configmap-pod-svc                   NodePort       10.109.89.91     <none>          80:32091/TCP                 75d
default         service/wear-svc                                 NodePort       10.103.57.82     <none>          80:32519/TCP                 79d
ingress-nginx   service/ingress-nginx-controller                 LoadBalancer   10.109.224.109   192.168.1.253   80:30985/TCP,443:32160/TCP   80d
ingress-nginx   service/ingress-nginx-controller-admission       ClusterIP      10.101.176.203   <none>          443/TCP                      80d
kube-system     service/kube-dns                                 ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP       81d
kube-system     service/metrics-server                           ClusterIP      10.109.152.9     <none>          443/TCP                      74d
newrelic        service/newrelic-bundle-kube-state-metrics       ClusterIP      10.105.244.136   <none>          8080/TCP                     74d
newrelic        service/newrelic-bundle-nri-metadata-injection   ClusterIP      10.104.75.83     <none>          443/TCP                      74d

NAMESPACE        NAME                                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system      daemonset.apps/calico-node                               3         3         3       3            3           kubernetes.io/os=linux   81d
kube-system      daemonset.apps/kube-proxy                                3         3         3       3            3           kubernetes.io/os=linux   81d
metallb-system   daemonset.apps/speaker                                   2         2         2       2            2           kubernetes.io/os=linux   80d
newrelic         daemonset.apps/newrelic-bundle-newrelic-infrastructure   3         3         3       3            3           <none>                   74d

NAMESPACE        NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/beauty-deploy                            2/2     2            2           53d
default          deployment.apps/food-deploy                              3/3     3            3           53d
default          deployment.apps/nfs-subdir-external-provisioner          1/1     1            1           76d
default          deployment.apps/patch-test                               2/2     2            2           76d
default          deployment.apps/pvc-deploy                               1/1     1            1           52d
ingress-nginx    deployment.apps/ingress-nginx-controller                 1/1     1            1           80d
kube-system      deployment.apps/calico-kube-controllers                  1/1     1            1           81d
kube-system      deployment.apps/coredns                                  2/2     2            2           81d
kube-system      deployment.apps/metrics-server                           2/2     2            2           74d
metallb-system   deployment.apps/controller                               1/1     1            1           80d
newrelic         deployment.apps/newrelic-bundle-kube-state-metrics       1/1     1            1           74d
newrelic         deployment.apps/newrelic-bundle-nri-kube-events          1/1     1            1           74d
newrelic         deployment.apps/newrelic-bundle-nri-metadata-injection   1/1     1            1           74d

NAMESPACE        NAME                                                                DESIRED   CURRENT   READY   AGE
default          replicaset.apps/beauty-deploy-7d59f6c75f                            2         2         2       53d
default          replicaset.apps/food-deploy-54544fc85d                              3         3         3       53d
default          replicaset.apps/food-deploy-5f46fbf467                              0         0         0       53d
default          replicaset.apps/food-deploy-6856f7c899                              0         0         0       53d
default          replicaset.apps/food-deploy-6f75b7b466                              0         0         0       53d
default          replicaset.apps/nfs-subdir-external-provisioner-9c9c5866f           1         1         1       76d
default          replicaset.apps/patch-test-6bd9658956                               2         2         2       76d
default          replicaset.apps/pvc-deploy-77f7d456bf                               1         1         1       52d
ingress-nginx    replicaset.apps/ingress-nginx-controller-54d8b558d4                 1         1         1       80d
kube-system      replicaset.apps/calico-kube-controllers-958545d87                   1         1         1       81d
kube-system      replicaset.apps/coredns-64897985d                                   2         2         2       48d
kube-system      replicaset.apps/coredns-78fcd69978                                  0         0         0       81d
kube-system      replicaset.apps/metrics-server-5549448c75                           2         2         2       74d
kube-system      replicaset.apps/metrics-server-7dc4d5d9cf                           0         0         0       74d
metallb-system   replicaset.apps/controller-8487b756b9                               1         1         1       80d
newrelic         replicaset.apps/newrelic-bundle-kube-state-metrics-857949d85        1         1         1       74d
newrelic         replicaset.apps/newrelic-bundle-nri-kube-events-6d57ffcb4f          1         1         1       74d
newrelic         replicaset.apps/newrelic-bundle-nri-metadata-injection-55865fd795   1         1         1       74d

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           15s        80d
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           15s        80d




おわりに
kubernetesを触り始めた初期はクラスターのアップグレードに対して非常に大変だなぁと思っていたのですが、
少しkubernetesと親しくなれたのかあまり抵抗がなくなってきましたw
kubernetesは、コミニティにおいて最新の3バージョン(現在なら、v1.24・v1.23・v1.22)がサポート対象となり、各バージョンも約1年間のパッチ(マイナーバージョン)リリースされるので比較的早いサイクルでのアップグレード作業が必要になります。
https://kubernetes.io/releases/release/
https://kubernetes.io/releases/

そのため、
・開発状況の確認
・不具合状況の確認
・アップグレード検証
・本番環境でのアップグレード作業
なども早いサイクルで実施することになります。

Opensourcetech by Takahiro Kujirai