LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
kubernetesでerror: You must be logged in to the server (Unauthorized)となって認証が通らなくなった際の復旧に関するメモです。
はじめに
具体的には、以下のようになった場合です。
そのままですが、Unauthorized(認証出来ん！)っていうてますね。
kubeuser@kubenewmaster1:~$ k get nodes error: You must be logged in to the server (Unauthorized) kubeuser@kubenewmaster1:~$ k get all error: You must be logged in to the server (Unauthorized)
復旧方法
kubernetesの認証では、以下のようなconfigファイルが使われます。
kubeuser@kubenewmaster1:~$ cat .kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: SL0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1ESXhOREV6TURVeU5Wb1hEVE15TURJeE1qRXpNRFV5TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS0VGCk5HaWtvK2xNcTBadE9zQk1TNFRWZUh6VGFiNjdxYkhKVGMwYktROVRaNnZlZE1jV29vcGNvOFJYSjdzbk5TaksKN0g4WEdKQm1lV1ZnckRYZHprSU9raE50WmZkVnoyaENYWU5LczNCV0RHWXZ3T0ozSDF3ajRmTmlRakkyYXo2Ygozcnd1T1RmaTdQRnZvRlBDQjNqdEFXOHVuWGZ3eVJxaS9xVEpZcDltMUVzRGRCa0dvQzd6MXpTUzBOWERiNEdHCkRKY2ZYK3UvL0d2bCtuVXdRdURiQUtOQ2lRbEllSlJtZHlBWlVRaXYyTDJMYy9rRkdUMTVNbVVvMW8wVzFaTG8KVXFWZW96QVRsZk4xbXpEM2FzV1VpS29WK05waGwzMTNBSWpZYWRkU0wrREkxNVd5YWJxdHBHY2YweUhJcEZmZQp0YmpvUjU2SHIyNS90OWN5QzVVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZLYzVuL0pKSXBzdy9zQnJzSWR3N0JKZFRScmFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRU9HajZTa256R21FNTZ2LzBWcAo2eDlva08yZDM5eEgzODEzM1RZeU5kU0d4cU0xSFdNVWlBaHV3ZG5KSU1PQlZtczNHS2g4WFFiSjdlSTlqdjZUCktMbnp0dFhxWVcwOERrYlZtOEM5SmQyRUoxVnlQVEZZdXRHSitXSVlKdHk2QlBEV2g3cU1VeTZVMm5VKzZQbEYKNVpmSDcwNEF1R0YwS1JIME5sVTJ3SVIvUHI0NnljMFo4WktlRUdOdzBPcHRPbysxMmtiVmkzYVBxQkVLbkVrTwp0N3cvL2k0RFVxbzJ5RGU4NkRHS2xIOSttMWJ3ZWpVUEJZM2xDUnRrMW5LRWExY0YvaXdUUW51cjVkZmxvZzhFCm4zUEw5OXNEMnVUSmhJMUR3L3hwdldKRG9mb1hJeHpiejd0VHBEejNrV1NUOFU4VjNpRmNPa2oreCtuem5XMFEKN0ZrPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://kubenewmaster1:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: SL0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJYkZnWFZDVzd4azR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TWpBeU1UUXhNekExTWpWYUZ3MHlNekF5TVRReE16QTFNekZhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXgxejIvTTI1OGpSRWpkZ2sKejFScitpMU15M2NSZTRGZW1IbGJ4QlhNRUNjNzVzSWRmQ0VUdlI1ZmlFc3pOdkNHeXVnL0kxaUVVakFXd3JtVwpzdTJ5cUNkWkc4OUpRelI3bWk2OTdNc01KVGNHRDNFK0xDZjNNVzhKelBFRnJtbnJHeDNVOFNnM3RrbWEreVVyCmEzMDd6UXBwSTBYTHIrMXVOemI2MFdRSHVraGNnZFY1bFlvMGVvd1p5d3N2dFV6MnBQT2hScTFsVmxrUytKQXcKR0NMeFN5a1I4dnl6cStnQXB6SFpmR0l5RXJmV0lIWE9LY2hlQTRwS1NjTGZOVUhVWHBQMlMwcUN2L2QyWlArOQpPMWE3NDErcWtKSW9WaWdQK2ZORUpXdUF5NGQrMTFRZktETlNzdHlmeGpmVm1nMEJUNXRoZjZabFNCdTJBUnFRCnEyYi94d0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JTbk9aL3lTU0tiTVA3QWE3Q0hjT3dTWFUwYQoyakFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBTko3dmE5NDlxYmlseGtHa3pIUzdkZEZPTHR5c0dBMHZaSUUyCmxOMzEvQUlNOEMrWXVpQ2tndFdBcGZPSnRXejYzblNNZlh2amZPdlBKQW1GeDFvQXY0cHpWR2M2bEtRTVBFK0wKZEZKZWpJT1BRa1JRN1BHTzloSk4ramppZmhhanpkZ3RIWnExOURwM0xtTjVGcnJQT3RwM3N4OE1BTUxDU1FVRwpXajd3ek82KzNlZTVhemlmYi9CRUUveFB5dWR1aFQ4ODJsMnpiMTNMdldmczRYZVpUT1ZMdFdIU3hvL1diZE9oCjZ2TVROYUxzRFNveCtpWWdHSmtGYWl6NDdlRzVEVWpFa3F4VzdiSXEwTDNHdWJtRWhNOW5la3JsYlBseDZlODAKd0dGWnZaKy9zak51T3NQVWZjVFJYTWtKUUUvZzNVd3pTSUZGWlIydXVtUGFhRFBrVWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client-key-data: SL0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBeDF6Mi9NMjU4alJFamRna3oxUnIraTFNeTNjUmU0RmVtSGxieEJYTUVDYzc1c0lkCmZDRVR2UjVmaUVzek52Q0d5dWcvSTFpRVVqQVd3cm1Xc3UyeXFDZFpHODlKUXpSN21pNjk3TXNNSlRjR0QzRSsKTENmM01XOEp6UEVGcm1uckd4M1U4U2czdGttYSt5VXJhMzA3elFwcEkwWExyKzF1TnpiNjBXUUh1a2hjZ2RWNQpsWW8wZW93Wnl3c3Z0VXoycFBPaFJxMWxWbGtTK0pBd0dDTHhTeWtSOHZ5enErZ0FwekhaZkdJeUVyZldJSFhPCktjaGVBNHBLU2NMZk5VSFVYcFAyUzBxQ3YvZDJaUCs5TzFhNzQxK3FrSklvVmlnUCtmTkVKV3VBeTRkKzExUWYKS0ROU3N0eWZ4amZWbWcwQlQ1dGhmNlpsU0J1MkFScVFxMmIveHdJREFRQUJBb0lCQUNoaXNMWHRodW1GcFEyRwo1NDRJY0FjeC9naUppa1VXbys4SFJvdW1UcnhHOWw5OG16UjJEdVdVclkyU2prRm00Q2RpZk1mUU9wM2JtQURDClQ4RFhYZ1dxVXViTFN2QU9SYXVxSkZjL21xby9SejhCbGJLa05mTVJwMDZZMUtuTVV4QWZMdS9iVWMzZmcwRzAKK2VMQWI4ak5meGJpSUt6MjBBam5Yay9rajV3d2lPUGhjeVNreEhUMmtDVDVTVURsNCswd3l2Z1prN0VkWm1PYQppdlVjU2pseWNFcTBsMnBmM2tWK2hYVk5ZSWxoY0kwM3dGWDFIMEdBREcvT3lnNDNQVWxmb3ZCQWU1TDJLVkcvCmFVUXFqazkvZytPaXY2anRwdWU1b2dEbGthN0hjQ0lDMkdOT0k3S2hCNVhUbFV6eWY1MlNMTlNZeVplVnRWb3YKUm9FbjNBRUNnWUVBMzJXN2RQby9ZYTA3MjJNNU5QK3dCeFRzbHlBamQwblFESHgxcExpbkRzMWNBUmpWRnZxcgpJRHhna2wrQTJDRGJpWkg2dW9SZHRsRm5CYnhZRlVocDNMMjFaaE1BR2wzU092K0libmIzK0daVTJKdTljWURqCnZLRklvQ0RRMnJOQVVveWE4djF2WHRhR1krL245T2pGQlJlVnptenZuU1F6a24rTUo4N29tVWNDZ1lFQTVIVk0KUCtGb0hMem5Qd2dTZnQ3NDNGL2lLdW9PbWJFd3VCbUM3bXF1dWpIM3cyUFJWb1cvMVhBemtGQnRPNVF2UVdYNwpGZG1iVFQ4NW1UN3BldXhIV3VLUldUb25LWFJrWmI1T3dXU1RlVk5IWVZ4S2JrSW15WkFKWnZWRFhQZWhOT2NXCnc1eTNvWGdnK0Y1M0o1RDNocXM0MGgwRXVENTBIVlp0ZGdyaHBZRUNnWUFlLzdqaFpKQkM5NHpreG9IN3JyYzQKWkZqb0o1ZUVTQVBNbDhDaldOUWxvNjF1b1ltQUpNeDJMcXFmNVF5MThPbEZ6N0hoQzlrTklZS1FNekJ0MDV5TQordTRlK2VmN3dLVVpkcmZ4ekNSZ25hS01aQ0FIamdFTC9iMWNLdkdRUjJ0WGlSYy9QSmVscTFMK3J4MmF5R24rCmFPVnF2WWNLWVNtZTNJQVFUZy9NcFFLQmdIWGQzcDBHbWtSWllhVXZjUHRyNWxFc1Z1OTFHbHRKQTYyMzI4bE4KMlIvUEw5anE0dElVNTBnalB6Y3hoMm01cGpmRGVhdG9QYXU0OXVxTmZzQWdyeC9Bek9TUUVDeGZGSDA1bGtCSQp0NTFjemZMNVBwMXNHNzdhUlQrTlFsZndtb2RFd29YaGtRd0pnbGtodzYveUp3S2Z6QXo3VTdnSzRMVlNKZDlFCjllNEJBb0dBRUViZTRzTkhnUDFVS3NBdlJOOFN3MUZZeHdPWFFsWmxIMDFKQXQ1d290UXV2WUo0elQwRkJSdTYKR292VDF5RU9xRzdqbndQczZwZVRlVU56c09nblBRR1FtUitmSUl1QURXekFsUTNxSk55T2VwWEFsSDhJVDlGagp1QzcrMjZUSHJNNUJQTTN4dFVlenlLV2s5T1VvbVh5UWlEMHBWWXRJNTdWNHBabDdYREk9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
なので、この元データからコピーして復旧します。
kubeuser@kubenewmaster1:~$ cp -p ./.kube/config ./.kube/config_old kubeuser@kubenewmaster1:~$ sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
無事に復旧出来ました。
kubeuser@kubenewmaster1:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
kubeuser@kubenewmaster1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubenewmaster1 Ready control-plane 369d v1.24.0
kubenewworker1 Ready <none> 324d v1.24.0
kubenewworker2 Ready <none> 324d v1.24.0
おまけ
kubernetesで使われる証明書の有効性のチェックは、以下のように行います。
kubeuser@kubenewmaster1:~$ sudo kubeadm certs check-expiration [sudo] password for kubeuser: [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf May 06, 2023 00:02 UTC 72d ca no apiserver May 05, 2023 23:59 UTC 72d ca no apiserver-etcd-client May 05, 2023 23:59 UTC 72d etcd-ca no apiserver-kubelet-client May 05, 2023 23:59 UTC 72d ca no controller-manager.conf May 06, 2023 00:01 UTC 72d ca no etcd-healthcheck-client May 05, 2023 23:58 UTC 72d etcd-ca no etcd-peer May 05, 2023 23:58 UTC 72d etcd-ca no etcd-server May 05, 2023 23:58 UTC 72d etcd-ca no front-proxy-client May 05, 2023 23:59 UTC 72d front-proxy-ca no scheduler.conf May 06, 2023 00:01 UTC 72d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Feb 12, 2032 13:05 UTC 8y no etcd-ca Feb 12, 2032 13:05 UTC 8y no front-proxy-ca Feb 12, 2032 13:05 UTC 8y no
また、証明書の期限切れた際の更新方法は
以下に詳細があります。
https://kubernetes.io/ja/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
