Opensourcetechブログ

OpensourcetechによるNGINX/Kubernetes/Zabbix/Neo4j/Linuxなどオープンソース技術に関するブログです。

Kubernetes 1.27の新機能(Feature Gate)を有効化し、InPlacePodVerticalScalingを試してみる

kubernetes Feature Gate InPlacePodVerticalScaling 1.27


LinuCエヴァンジェリスト・Open Source Summit Japan 2022ボランティアリーダーの鯨井貴博@opensourcetechです。


はじめに
今回は、kubernetes 1.27の新機能(Feature Gate)として実装されたInPlacePodVerticalScalingを有効化して使ってみようと思います。

https://kubernetes.io/blog/2023/05/12/in-place-pod-resize-alpha/


Feature Gateとは
KubernetesのFeature Gateは、今後正式機能となる候補の位置づけになります。
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/




機能は、Alpha → Beta → GA → 非推奨(deprecated) → 廃止 というふうにステータスが変更されていきます。

今回使おうとしているInPlacePodVerticalScalingは1.27ではAlphaとなっており、デフォルトでdisable(無効)となっているため手動で有効化する必要があります。



InPlacePodVerticalScalingとは
InPlacePodVerticalScalingは、Podに定義されているリソース(CPU・メモリー)を変更する際 通常はPodの再起動が行われますが、再起動をせずに値の変更を可能にする機能です。
想定されている用途は、当初Podに割り当てたリソースが過不足するときに変更しますが、そのタイミングでPodの再起動が発生しサービス影響が出ることを防ぐためとのことです。

https://kubernetes.io/blog/2023/05/12/in-place-pod-resize-alpha/
https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/



1.26までのPodの挙動
まずはこれまでどういう挙動だったか、確認してみましょう。
normal-nginxというPodのリソースを更新して、再起動されることを確認します。

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ vi normalpod.yaml 
kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ cat normalpod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: normal-nginx
  namespace: default
spec:
  containers:
  - name: normal-nginx
    image: nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: RestartContainer
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      limits:
        memory: "100Mi"
        cpu: "100m"
      requests:
        memory: "50Mi"
        cpu: "50m"

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl apply -f normalpod.yaml 
pod/normal-nginx created

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod -o wide
NAME                         READY   STATUS    RESTARTS      AGE    IP            NODE       NOMINATED NODE   READINESS GATES
normal-nginx                 1/1     Running   0             2m3s   10.0.5.50     worker01   <none>           <none>

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod normal-nginx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: e844c13672de3dcf6a09563d7aa73473eefd9a323b8ac6a6856f234b2d51e5f3
    cni.projectcalico.org/podIP: 10.0.5.50/32
    cni.projectcalico.org/podIPs: 10.0.5.50/32,fd12:b5e0:383e:0:c9e5:fc91:8eec:52d/128
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"normal-nginx","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"normal-nginx","resizePolicy":[{"resourceName":"cpu","restartPolicy":"RestartContainer"},{"resourceName":"memory","restartPolicy":"RestartContainer"}],"resources":{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"50m","memory":"50Mi"}}}]}}
  creationTimestamp: "2023-05-14T11:32:36Z"
  name: normal-nginx
  namespace: default
  resourceVersion: "8519282"
  uid: 944c7251-6430-4aa9-9c5a-80befbdb2e97
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: normal-nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: RestartContainer
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qnr74
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: worker01
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-qnr74
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:32:36Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:33:05Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:33:05Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:32:36Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - allocatedResources:
      cpu: 50m
      memory: 50Mi
    containerID: containerd://2bbd8f6c61bfec6dab7008ad5106817e5456bba4794dd27ec0842cdd028f7f1f
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:480868e8c8c797794257e2abd88d0f9a8809b2fe956cbfbc05dcc0bca1f7cd43
    lastState: {}
    name: normal-nginx
    ready: true
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-05-14T11:33:03Z"
  hostIP: 192.168.1.45
  phase: Running
  podIP: 10.0.5.50
  podIPs:
  - ip: 10.0.5.50
  - ip: fd12:b5e0:383e:0:c9e5:fc91:8eec:52d
  qosClass: Burstable
  startTime: "2023-05-14T11:32:36Z"


kubectl patchでcpuのrequestsとlimitsを更新します。

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl patch pod normal-nginx --patch '{"spec":{"containers":[{"name":"normal-nginx", "resources":{"requests":{"cpu":"40m"}, "limits":{"cpu":"90m"}}}]}}'
pod/normal-nginx patched


restartCount: 1となっており、コンテナが再起動されたことが分かります。

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod normal-nginx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: e844c13672de3dcf6a09563d7aa73473eefd9a323b8ac6a6856f234b2d51e5f3
    cni.projectcalico.org/podIP: 10.0.5.50/32
    cni.projectcalico.org/podIPs: 10.0.5.50/32,fd12:b5e0:383e:0:c9e5:fc91:8eec:52d/128
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"normal-nginx","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"normal-nginx","resizePolicy":[{"resourceName":"cpu","restartPolicy":"RestartContainer"},{"resourceName":"memory","restartPolicy":"RestartContainer"}],"resources":{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"50m","memory":"50Mi"}}}]}}
  creationTimestamp: "2023-05-14T11:32:36Z"
  name: normal-nginx
  namespace: default
  resourceVersion: "8519583"
  uid: 944c7251-6430-4aa9-9c5a-80befbdb2e97
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: normal-nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: RestartContainer
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      limits:
        cpu: 90m
        memory: 100Mi
      requests:
        cpu: 40m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qnr74
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: worker01
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-qnr74
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:32:36Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:33:05Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:33:05Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:32:36Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - allocatedResources:
      cpu: 40m
      memory: 50Mi
    containerID: containerd://d72e981e6595cbbedd5a6f6a4b70a9dad2fe88445141e7a84c2fc28df00c7540
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:480868e8c8c797794257e2abd88d0f9a8809b2fe956cbfbc05dcc0bca1f7cd43
    lastState:
      terminated:
        containerID: containerd://2bbd8f6c61bfec6dab7008ad5106817e5456bba4794dd27ec0842cdd028f7f1f
        exitCode: 0
        finishedAt: "2023-05-14T11:36:02Z"
        reason: Completed
        startedAt: "2023-05-14T11:33:03Z"
    name: normal-nginx
    ready: true
    resources:
      limits:
        cpu: 90m
        memory: 100Mi
      requests:
        cpu: 40m
        memory: 50Mi
    restartCount: 1
    started: true
    state:
      running:
        startedAt: "2023-05-14T11:36:05Z"
  hostIP: 192.168.1.45
  phase: Running
  podIP: 10.0.5.50
  podIPs:
  - ip: 10.0.5.50
  - ip: fd12:b5e0:383e:0:c9e5:fc91:8eec:52d
  qosClass: Burstable
  startTime: "2023-05-14T11:32:36Z"



Feature Gateの有効化
Feature Gateの有効化は、kubernetesクラスターを構成する各ノードで機能を有効にすればOKです。

まず、masterノード。
kubeadm・kube-apiserver・kube-schedulerで有効にしました。

kubeadmの設定変更。

kubeuser@master01:~$ sudo cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

kubeuser@master01:~$ sudo cat /var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9 --feature-gates=InPlacePodVerticalScaling=true"


kube-apiserverの設定変更。

kubeuser@master01:~$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.41:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.41
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.1.0.0/16,fd12:b5e0:383f::/112
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.27.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.41
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.41
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.41
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

kubeuser@master01:~$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

kubeuser@master01:~$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.41:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.41
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.1.0.0/16,fd12:b5e0:383f::/112
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=InPlacePodVerticalScaling=true
    image: registry.k8s.io/kube-apiserver:v1.27.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.41
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.41
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.41
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}


kube-schedulerの設定変更。

kubeuser@master01:~$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    image: registry.k8s.io/kube-scheduler:v1.27.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

kubeuser@master01:~$ sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml

kubeuser@master01:~$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --feature-gates=InPlacePodVerticalScaling=true
    image: registry.k8s.io/kube-scheduler:v1.27.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}


設定変更の反映。

kubeuser@master01:~$ sudo systemctl daemon-reload
kubeuser@master01:~$ sudo systemctl restart kubelet



続いて、workerノードでの設定変更。
kubeadmのみです。

kubeuser@worker01:~$ sudo cat /var/lib/kubelet/kubeadm-flags.env
[sudo] password for kubeuser: 
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

kubeuser@worker01:~$ sudo vi /var/lib/kubelet/kubeadm-flags.env

kubeuser@worker01:~$ sudo cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9 --feature-gates=InPlacePodVerticalScaling=true"

kubeuser@worker01:~$ sudo systemctl daemon-reload
kubeuser@worker01:~$ sudo systemctl restart kubelet



1.27でのPodの挙動
では、最後に新機能の挙動を確認しましょう。
resize-nginxというPodを使っています。
InPlacePodVerticalScalingはデフォルトでrestartPolicy: NotRequiredという設定なので、特にyamlファイルに記入しなくてもOKです。
※kubectl getのyaml形式で出力したものを見ると、そうなっているのが分かります。

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ vi testpod.yaml

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ cat testpod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: resize-nginx
  namespace: default
spec:
  containers:
  - name: resize-nginx
    image: nginx
    resources:
      limits:
        memory: "100Mi"
        cpu: "100m"
      requests:
        memory: "50Mi"
        cpu: "50m"
kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl apply -f testpod.yaml 
pod/resize-nginx created

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod -o wide
NAME                         READY   STATUS    RESTARTS      AGE    IP            NODE       NOMINATED NODE   READINESS GATES
resize-nginx                 1/1     Running   0             66s    10.0.30.75    worker02   <none>           <none>

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod resize-nginx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 6cc5509cb764550bf71555fb4ee2958048f24666f9c81a02a6bb45a624995845
    cni.projectcalico.org/podIP: 10.0.30.75/32
    cni.projectcalico.org/podIPs: 10.0.30.75/32,fd12:b5e0:383e:0:7bf:50a7:b256:1e68/128
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"resize-nginx","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"resize-nginx","resources":{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"50m","memory":"50Mi"}}}]}}
  creationTimestamp: "2023-05-14T11:10:02Z"
  name: resize-nginx
  namespace: default
  resourceVersion: "8517128"
  uid: fa3f0810-3c51-4fc4-99a0-cf934860ea11
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: resize-nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: NotRequired
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xgbl8
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: worker02
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-xgbl8
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:10:04Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:11:01Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:11:01Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:10:03Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - allocatedResources:
      cpu: 50m
      memory: 50Mi
    containerID: containerd://2096ed2178a1e9b1e03a3c4c429d1e4390b5fd3d5538600f168d1d219659cf3a
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:480868e8c8c797794257e2abd88d0f9a8809b2fe956cbfbc05dcc0bca1f7cd43
    lastState: {}
    name: resize-nginx
    ready: true
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-05-14T11:11:00Z"
  hostIP: 192.168.1.46
  phase: Running
  podIP: 10.0.30.75
  podIPs:
  - ip: 10.0.30.75
  - ip: fd12:b5e0:383e:0:7bf:50a7:b256:1e68
  qosClass: Burstable
  startTime: "2023-05-14T11:10:04Z"


では、リソースの変更をして再起動されるかどうかの確認です。

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl patch pod resize-nginx --patch '{"spec":{"containers":[{"name":"resize-nginx", "resources":{"requests":{"cpu":"40m"}, "limits":{"cpu":"90m"}}}]}}'
pod/resize-nginx patched

kubeuser@master01:~/InPlacePodVerticalScaling_1.27featuregate$ kubectl get pod resize-nginx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 6cc5509cb764550bf71555fb4ee2958048f24666f9c81a02a6bb45a624995845
    cni.projectcalico.org/podIP: 10.0.30.75/32
    cni.projectcalico.org/podIPs: 10.0.30.75/32,fd12:b5e0:383e:0:7bf:50a7:b256:1e68/128
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"resize-nginx","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"resize-nginx","resources":{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"50m","memory":"50Mi"}}}]}}
  creationTimestamp: "2023-05-14T11:10:02Z"
  name: resize-nginx
  namespace: default
  resourceVersion: "8517812"
  uid: fa3f0810-3c51-4fc4-99a0-cf934860ea11
spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: resize-nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: NotRequired
    resources:
      limits:
        cpu: 90m
        memory: 100Mi
      requests:
        cpu: 40m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xgbl8
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: worker02
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-xgbl8
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:10:04Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:11:01Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:11:01Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-05-14T11:10:03Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - allocatedResources:
      cpu: 40m
      memory: 50Mi
    containerID: containerd://2096ed2178a1e9b1e03a3c4c429d1e4390b5fd3d5538600f168d1d219659cf3a
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:480868e8c8c797794257e2abd88d0f9a8809b2fe956cbfbc05dcc0bca1f7cd43
    lastState: {}
    name: resize-nginx
    ready: true
    resources:
      limits:
        cpu: 90m
        memory: 100Mi
      requests:
        cpu: 40m
        memory: 50Mi
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-05-14T11:11:00Z"
  hostIP: 192.168.1.46
  phase: Running
  podIP: 10.0.30.75
  podIPs:
  - ip: 10.0.30.75
  - ip: fd12:b5e0:383e:0:7bf:50a7:b256:1e68
  qosClass: Burstable
  startTime: "2023-05-14T11:10:04Z"


設定変更がされても、restartCount: 0のままであることが確認出来ます。


おわりに
今回は1.27の新機能(Feature Gate)であるInPlacePodVerticalScalingを使ってみましたが、以下の気づきがありました。
・kubernetesの新機能(Feature Gate)について ・新機能(Feature Gate)を有効にする方法
・InPlacePodVerticalScalingのパラメータによる挙動の違いの理解

また、ステータスがAlphaなので今後どのようになっていくかわかりませんが、
1.28、1.29、、、とバージョンアップされていく過程やこの機能に関するコミニティでのディスカッションなど色々と着目するべきポイントもわかり、多くの収穫のある検証でした♪

Opensourcetech by Takahiro Kujirai