LinuCエヴァンジェリスト・Open Source Summit Japan 2022ボランティアリーダーの鯨井貴博@opensourcetechです。
はじめに
今回は、以前作成したDualStackのkubernetesクラスターを1.26から1.27にアップグレードします。
アップグレード方法
対象となるkubernetesクラスターは、以下の3台構成(master1、worker2)です。
kubeuser@master01:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master01 Ready control-plane 58d v1.26.0 192.168.1.41 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker01 Ready <none> 58d v1.26.0 192.168.1.45 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker02 Ready <none> 58d v1.26.0 192.168.1.46 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18
アップグレードは、以下の方法に沿って行いました。
Upgrading kubeadm clusters
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
Upgrading Linux nodes
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/
なお、v1.26.00からv1.27.00へのアップグレードで、apt-getはaptに変更したり、sudoを付けたり少しカスタマイズしています。
パッケージマネージャ管理情報の最新化 ※全ノード
全ノードでパッケージマネージャ管理情報の最新化をします。
kubeuser@master01:~$ sudo apt update [sudo] password for kubeuser: Get:1 https://download.docker.com/linux/ubuntu jammy InRelease [48.9 kB] Hit:3 https://baltocdn.com/helm/stable/debian all InRelease Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B] Hit:4 http://jp.archive.ubuntu.com/ubuntu jammy InRelease Get:5 http://jp.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] Get:6 http://jp.archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] Get:7 http://jp.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB] Get:8 http://jp.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [578 kB] Get:9 http://jp.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [166 kB] Get:10 http://jp.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [14.4 kB] Get:11 http://jp.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [887 kB] Get:12 http://jp.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [35.3 kB] Get:13 http://jp.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [8452 B] Get:14 http://jp.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [363 kB] Get:15 http://jp.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [108 kB] Get:16 http://jp.archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [9732 B] Get:17 http://jp.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [709 kB] Fetched 3273 kB in 6s (537 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done 52 packages can be upgraded. Run 'apt list --upgradable' to see them. W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. W: https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
アップグレード可能なバージョンの確認をします。
kubeuser@master01:~$ sudo apt-cache madison kubeadm kubeadm | 1.27.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.27.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.26.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.26.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.26.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.26.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.26.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.25.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.24.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.23.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.22.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.21.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.20.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.19.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.20-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.19-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.18-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.4-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.18.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.17-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.7-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.17.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.16.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.15.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.14.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.13.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.12.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.11.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.10.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.9.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.1-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.0-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.8.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.16-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.15-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.3-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.7.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.13-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.12-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.11-01 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.10-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.9-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.8-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.4-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.6.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages kubeadm | 1.5.7-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
masterノードの更新
では、masterノードで更新をはじめます。
v1.27.0のkubeadmをインストールします。
kubeuser@master01:~$ sudo apt-mark unhold kubeadm && sudo apt update && sudo apt install -y kubeadm=1.27.0-00 && sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:3 https://baltocdn.com/helm/stable/debian all InRelease
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
Hit:4 http://jp.archive.ubuntu.com/ubuntu jammy InRelease
Hit:5 http://jp.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:6 http://jp.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:7 http://jp.archive.ubuntu.com/ubuntu jammy-security InRelease
Fetched 8993 B in 1s (6543 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
52 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.
Need to get 9931 kB of archives.
After this operation, 1393 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.27.0-00 [9931 kB]
Fetched 9931 kB in 2s (5454 kB/s)
(Reading database ... 117701 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.27.0-00_amd64.deb ...
Unpacking kubeadm (1.27.0-00) over (1.26.0-00) ...
Setting up kubeadm (1.27.0-00) ...
Scanning processes...
Scanning candidates...
Scanning linux images...
Restarting services...
Service restarts being deferred:
systemctl restart networkd-dispatcher.service
systemctl restart systemd-logind.service
systemctl restart unattended-upgrades.service
systemctl restart user@1000.service
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
kubeadm set on hold.
kubeuser@master01:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:09:06Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
kubeadmによるアップグレードの確認。
kubeuser@master01:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.26.0
[upgrade/versions] kubeadm version: v1.27.0
[upgrade/versions] Target version: v1.27.1
[upgrade/versions] Latest version in the v1.26 series: v1.26.4
W0512 09:07:53.612438 1210314 compute.go:307] [upgrade/versions] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.26.0 v1.26.4
Upgrade to the latest version in the v1.26 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.26.0 v1.26.4
kube-controller-manager v1.26.0 v1.26.4
kube-scheduler v1.26.0 v1.26.4
kube-proxy v1.26.0 v1.26.4
CoreDNS v1.9.3 v1.10.1
etcd 3.5.6-0 3.5.7-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.26.4
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.26.0 v1.27.1
Upgrade to the latest stable version:
COMPONENT CURRENT TARGET
kube-apiserver v1.26.0 v1.27.1
kube-controller-manager v1.26.0 v1.27.1
kube-scheduler v1.26.0 v1.27.1
kube-proxy v1.26.0 v1.27.1
CoreDNS v1.9.3 v1.10.1
etcd 3.5.6-0 3.5.7-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.27.1
Note: Before you can perform this upgrade, you have to update kubeadm to v1.27.1.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
kubeadmでv1.27.0を適用します。
kubeuser@master01:~$ sudo kubeadm upgrade apply v1.27.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.27.0"
[upgrade/versions] Cluster version: v1.26.0
[upgrade/versions] kubeadm version: v1.27.0
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W0512 09:09:40.107121 1211700 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0512 09:11:33.407263 1211700 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.27.0" (timeout: 5m0s)...
{"level":"warn","ts":"2023-05-12T09:12:45.193Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007ec1c0/192.168.1.41:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upgrade/etcd] Upgrading to TLS for etcd
W0512 09:15:13.389028 1211700 staticpods.go:305] [upgrade/etcd] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0512 09:15:16.230359 1211700 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-05-12-09-12-43/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3766369176"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-05-12-09-12-43/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-05-12-09-12-43/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-05-12-09-12-43/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1024203952/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.27.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
masterノードで稼働中のPodなど(deamonset以外)を多ノードへ退避します。
kubeuser@master01:~$ kubectl drain master01 --ignore-daemonsets node/master01 cordoned Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-xdkfv, kube-system/kube-proxy-xff4d, metallb-system/speaker-wgxdr evicting pod kube-system/calico-kube-controllers-57b57c56f-p6xds pod/calico-kube-controllers-57b57c56f-p6xds evicted node/master01 drained
kubeletとkubectlを1.27.0に更新します。
kubeuser@master01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt update && sudo apt install -y kubelet=1.27.0-00 kubectl=1.27.0-00 && sudo apt-mark hold kubelet kubectl [sudo] password for kubeuser: Canceled hold on kubelet. Canceled hold on kubectl. Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease Hit:3 https://baltocdn.com/helm/stable/debian all InRelease Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Hit:4 http://jp.archive.ubuntu.com/ubuntu jammy InRelease Hit:5 http://jp.archive.ubuntu.com/ubuntu jammy-updates InRelease Hit:6 http://jp.archive.ubuntu.com/ubuntu jammy-backports InRelease Hit:7 http://jp.archive.ubuntu.com/ubuntu jammy-security InRelease Reading package lists... Done Building dependency tree... Done Reading state information... Done 52 packages can be upgraded. Run 'apt list --upgradable' to see them. W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. W: https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 50 not upgraded. Need to get 29.0 MB of archives. After this operation, 13.9 MB disk space will be freed. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.27.0-00 [10.2 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.27.0-00 [18.8 MB] Fetched 29.0 MB in 5s (5621 kB/s) (Reading database ... 117701 files and directories currently installed.) Preparing to unpack .../kubectl_1.27.0-00_amd64.deb ... Unpacking kubectl (1.27.0-00) over (1.26.0-00) ... Preparing to unpack .../kubelet_1.27.0-00_amd64.deb ... Unpacking kubelet (1.27.0-00) over (1.26.0-00) ... Setting up kubectl (1.27.0-00) ... Setting up kubelet (1.27.0-00) ... Scanning processes... Scanning candidates... Scanning linux images... Restarting services... Service restarts being deferred: systemctl restart networkd-dispatcher.service systemctl restart systemd-logind.service systemctl restart unattended-upgrades.service systemctl restart user@1000.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. kubelet set on hold. kubectl set on hold.
kubeletの再起動をします。
kubeuser@master01:~$ sudo systemctl daemon-reload kubeuser@master01:~$ sudo systemctl restart kubelet
最後に退避したポッドなどをmasterノードでも稼働するようにします。
kubeuser@master01:~$ kubectl uncordon master01 node/master01 uncordoned
これでmasterノードは完了。
kubeuser@master01:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master01 Ready control-plane 58d v1.27.0 192.168.1.41 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker01 Ready <none> 58d v1.26.0 192.168.1.45 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker02 Ready <none> 58d v1.26.0 192.168.1.46 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18
複数台のmasterノードで構成されるkubernetesクラスターの場合、
以下の部分以外は同じように更新をします。
workerノードの更新
続いて、workerノードの更新です。
※複数台ある場合は、同様の手順を繰り返します。
kubeadmの更新。
kubeuser@worker01:~$ sudo apt-mark unhold kubeadm && sudo apt update && sudo apt install -y kubeadm=1.27.0-00 && sudo apt-mark hold kubeadm [sudo] password for kubeuser: Canceled hold on kubeadm. Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Hit:3 http://jp.archive.ubuntu.com/ubuntu jammy InRelease Hit:4 http://jp.archive.ubuntu.com/ubuntu jammy-updates InRelease Hit:5 http://jp.archive.ubuntu.com/ubuntu jammy-backports InRelease Hit:6 http://jp.archive.ubuntu.com/ubuntu jammy-security InRelease Reading package lists... Done Building dependency tree... Done Reading state information... Done 51 packages can be upgraded. Run 'apt list --upgradable' to see them. W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. W: https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be upgraded: kubeadm 1 upgraded, 0 newly installed, 0 to remove and 50 not upgraded. Need to get 9931 kB of archives. After this operation, 1393 kB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.27.0-00 [9931 kB] Fetched 9931 kB in 3s (2872 kB/s) (Reading database ... 109593 files and directories currently installed.) Preparing to unpack .../kubeadm_1.27.0-00_amd64.deb ... Unpacking kubeadm (1.27.0-00) over (1.26.0-00) ... Setting up kubeadm (1.27.0-00) ... Scanning processes... Scanning candidates... Scanning linux images... Restarting services... /etc/needrestart/restart.d/systemd-manager systemctl restart packagekit.service ssh.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-timesyncd.service systemd-udevd.service udisks2.service upower.service Service restarts being deferred: systemctl restart networkd-dispatcher.service systemctl restart systemd-logind.service systemctl restart unattended-upgrades.service systemctl restart user@1000.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host.
ノードのアップグレード。
kubeuser@worker01:~$ sudo kubeadm upgrade node [upgrade] Reading configuration from the cluster... [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks [preflight] Skipping prepull. Not a control plane node. [upgrade] Skipping phase. Not a control plane node. [upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1053288049/config.yaml [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
ノード上で稼働するPodなどを退避。
kubeuser@master01:~$ kubectl drain worker01 --ignore-daemonsets node/worker01 cordoned error: unable to drain node "worker01" due to error:[cannot delete Pods declare no controller (use --force to override): blue/blue2, default/test-webserver, orange/orange1, cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-7bc864c59-pscgp], continuing command... There are pending nodes to be drained: worker01 cannot delete Pods declare no controller (use --force to override): blue/blue2, default/test-webserver, orange/orange1 cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-7bc864c59-pscgp
kubeletとkubectlの更新。
kubeuser@worker01:~$ sudo apt-mark unhold kubelet kubectl && sudo apt update && sudo apt install -y kubelet=1.27.0-00 kubectl=1.27.0-00 && sudo apt-mark hold kubelet kubectl Canceled hold on kubelet. Canceled hold on kubectl. Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Hit:3 http://jp.archive.ubuntu.com/ubuntu jammy InRelease Get:4 http://jp.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] Get:5 http://jp.archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] Get:6 http://jp.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB] Fetched 337 kB in 3s (133 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done 51 packages can be upgraded. Run 'apt list --upgradable' to see them. W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. W: https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. Need to get 29.0 MB of archives. After this operation, 13.9 MB disk space will be freed. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.27.0-00 [10.2 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.27.0-00 [18.8 MB] Fetched 29.0 MB in 7s (4135 kB/s) (Reading database ... 109593 files and directories currently installed.) Preparing to unpack .../kubectl_1.27.0-00_amd64.deb ... Unpacking kubectl (1.27.0-00) over (1.26.0-00) ... Preparing to unpack .../kubelet_1.27.0-00_amd64.deb ... Unpacking kubelet (1.27.0-00) over (1.26.0-00) ... Setting up kubectl (1.27.0-00) ... Setting up kubelet (1.27.0-00) ... Scanning processes... Scanning candidates... Scanning linux images... Restarting services... Service restarts being deferred: systemctl restart networkd-dispatcher.service systemctl restart systemd-logind.service systemctl restart unattended-upgrades.service systemctl restart user@1000.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. kubelet set on hold. kubectl set on hold.
kubeletの再起動。
kubeuser@worker01:~$ sudo systemctl daemon-reload kubeuser@worker01:~$ sudo systemctl restart kubelet
workerノードでPodが稼働できるようにします。
kubeuser@master01:~$ kubectl uncordon worker01 node/worker01 uncordoned
workerノードのアップグレード完了。
kubeuser@master01:~$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master01 Ready control-plane 58d v1.27.0 192.168.1.41 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker01 Ready <none> 58d v1.27.0 192.168.1.45 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker02 Ready <none> 58d v1.26.0 192.168.1.46 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18
複数台ある場合、全てのworkerノードをアップグレードしたら終了となります。
kubeuser@master01:~$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master01 Ready control-plane 58d v1.27.0 192.168.1.41 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker01 Ready <none> 58d v1.27.0 192.168.1.45 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18 worker02 Ready <none> 58d v1.27.0 192.168.1.46 <none> Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.18
