Opensourcetechブログ

OpensourcetechによるNGINX/Kubernetes/Zabbix/Neo4j/Linuxなどオープンソース技術に関するブログです。

kubernetesのラベルにbool型を指定する際には準拠するYAMLの仕様(version1.1)に気を付けた方がいい

kubernetes YAML YAML1.1 YAML1.2 nodeSelector


LinuCエヴァンジェリスト・Open Source Summit Japan 2022ボランティアリーダーの鯨井貴博@opensourcetechです。


はじめに
kubernetesのラベルでbool型を使ってちょっとはまったので、それに関するメモです。


やってみたこと
kubernetesのノードに gpu: onというラベルを付与して、nodeSelectorを使うとPodを起動できなかったというものになります。

ラベルの付与。

kubeuser@master01:~/nodeSelector$ kubectl get nodes --show-labels
NAME       STATUS   ROLES           AGE    VERSION   LABELS
master01   Ready    control-plane   153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master01,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
worker01   Ready    <none>          153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker01,kubernetes.io/os=linux
worker02   Ready    <none>          153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker02,kubernetes.io/os=linux


kubeuser@master01:~/nodeSelector$ kubectl label nodes worker01 gpu=on
node/worker01 labeled

kubeuser@master01:~/nodeSelector$ kubectl label nodes worker02 gpu=off
node/worker02 labeled

kubeuser@master01:~/nodeSelector$ kubectl get nodes --show-labels
NAME       STATUS   ROLES           AGE    VERSION   LABELS
master01   Ready    control-plane   153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master01,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
worker01   Ready    <none>          153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=on,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker01,kubernetes.io/os=linux
worker02   Ready    <none>          153d   v1.27.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=off,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker02,kubernetes.io/os=linux


nodeSelector(ノード指定)を使ったPodの起動。
※失敗します。

kubeuser@master01:~/nodeSelector$ vi testpod.yaml 

kubeuser@master01:~/nodeSelector$ cat testpod.yaml 
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: testpod
  name: testpod
spec:
  containers:
  - image: nginx
    name: testpod
  nodeSelector:
    gpu: on
  dnsPolicy: ClusterFirst
  restartPolicy: Always

kubeuser@master01:~/nodeSelector$ kubectl apply -f testpod.yaml 
Error from server (BadRequest): error when creating "testpod.yaml": Pod in version "v1" cannot be handled as a Pod: json: cannot unmarshal bool into Go struct field PodSpec.spec.nodeSelector of type string



原因
kubernetesが対応するYAML仕様(version1.1)の影響。
https://yaml.org/type/bool.html


これによって、on ⇒ trueと判断されてエラーとなっていた。

具体的には、kubectl applyで適用したyamlファイルには以下のようにnodeSelectorを設定。

  nodeSelector:
    gpu: on


この場合、実際の処理では{"gpu":true}というように解釈され、ラベルが不一致のためエラーとなる。
※Error from server (BadRequest): error when creating "testpod.yaml": Pod in version "v1" cannot be handled as a Pod: json: cannot unmarshal bool into Go struct field PodSpec.spec.nodeSelector of type string

kubeuser@master01:~/nodeSelector$ kubectl apply -f testpod.yaml --v=8
I0815 02:04:24.774645 1742867 loader.go:373] Config loaded from file:  /home/kubeuser/.kube/config
I0815 02:04:24.775607 1742867 round_trippers.go:463] GET https://master01:6443/openapi/v2?timeout=32s
I0815 02:04:24.775659 1742867 round_trippers.go:469] Request Headers:
I0815 02:04:24.775680 1742867 round_trippers.go:473]     Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0815 02:04:24.775696 1742867 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0815 02:04:24.835154 1742867 round_trippers.go:574] Response Status: 200 OK in 59 milliseconds
I0815 02:04:24.835186 1742867 round_trippers.go:577] Response Headers:
I0815 02:04:24.835358 1742867 round_trippers.go:580]     Etag: "6DD572D6B65B40387414DEC34DB6ED25FB343DC811B484B15F252FB9CF1136C15768DD248AAF8920CC2EE9156EFCB699E0C67BE484E9A3F4A9C13691BBEF7734"
I0815 02:04:24.835370 1742867 round_trippers.go:580]     X-From-Cache: 1
I0815 02:04:24.835492 1742867 round_trippers.go:580]     Content-Type: application/com.github.proto-openapi.spec.v2.v1.0+protobuf
I0815 02:04:24.835508 1742867 round_trippers.go:580]     X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0815 02:04:24.835677 1742867 round_trippers.go:580]     Vary: Accept-Encoding
I0815 02:04:24.835755 1742867 round_trippers.go:580]     Vary: Accept
I0815 02:04:24.835921 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 49e87840-df8b-4ec9-a45d-91cc40a97e6b
I0815 02:04:24.836035 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 0bd22c41-fc71-4844-bd81-85e6526adc42
I0815 02:04:24.836116 1742867 round_trippers.go:580]     Accept-Ranges: bytes
I0815 02:04:24.836199 1742867 round_trippers.go:580]     Audit-Id: b1e7c105-5357-4964-8fa4-8fed005ae6bc
I0815 02:04:24.836297 1742867 round_trippers.go:580]     Cache-Control: no-cache, private
I0815 02:04:24.836389 1742867 round_trippers.go:580]     Date: Tue, 15 Aug 2023 02:04:24 GMT
I0815 02:04:24.836450 1742867 round_trippers.go:580]     Last-Modified: Sat, 13 May 2023 11:50:17 GMT
I0815 02:04:24.984149 1742867 request.go:1186] Response Body:
00000000  0a 03 32 2e 30 12 15 0a  0a 4b 75 62 65 72 6e 65  |..2.0....Kuberne|
00000010  74 65 73 12 07 76 31 2e  32 37 2e 30 42 ec af c3  |tes..v1.27.0B...|
00000020  01 12 8c 02 0a 22 2f 2e  77 65 6c 6c 2d 6b 6e 6f  |....."/.well-kno|
00000030  77 6e 2f 6f 70 65 6e 69  64 2d 63 6f 6e 66 69 67  |wn/openid-config|
00000040  75 72 61 74 69 6f 6e 2f  12 e5 01 12 e2 01 0a 09  |uration/........|
00000050  57 65 6c 6c 4b 6e 6f 77  6e 1a 57 67 65 74 20 73  |WellKnown.Wget s|
00000060  65 72 76 69 63 65 20 61  63 63 6f 75 6e 74 20 69  |ervice account i|
00000070  73 73 75 65 72 20 4f 70  65 6e 49 44 20 63 6f 6e  |ssuer OpenID con|
00000080  66 69 67 75 72 61 74 69  6f 6e 2c 20 61 6c 73 6f  |figuration, also|
00000090  20 6b 6e 6f 77 6e 20 61  73 20 74 68 65 20 27 4f  | known as the 'O|
000000a0  49 44 43 20 64 69 73 63  6f 76 65 72 79 20 64 6f  |IDC discovery do|
000000b0  63 27 2a 2a 67 65 74 53  65 72 76 69 63 65 41 63  |c'**getServiceAc|
000000c0  63 6f 75 6e 74 49 73 73  75 65 72 4f 70 65 6e 49  |countIssuerOpen [truncated 20255200 chars]
I0815 02:04:25.051181 1742867 round_trippers.go:463] GET https://master01:6443/openapi/v3?timeout=32s
I0815 02:04:25.051209 1742867 round_trippers.go:469] Request Headers:
I0815 02:04:25.051221 1742867 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0815 02:04:25.051233 1742867 round_trippers.go:473]     Accept: application/json, */*
I0815 02:04:25.052918 1742867 round_trippers.go:574] Response Status: 200 OK in 1 milliseconds
I0815 02:04:25.052947 1742867 round_trippers.go:577] Response Headers:
I0815 02:04:25.052966 1742867 round_trippers.go:580]     Cache-Control: no-cache, private
I0815 02:04:25.052978 1742867 round_trippers.go:580]     Audit-Id: 616e906f-ed2d-4b26-a977-956b9ce6554a
I0815 02:04:25.052992 1742867 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0815 02:04:25.053004 1742867 round_trippers.go:580]     Last-Modified: Tue, 15 Aug 2023 02:04:25 GMT
I0815 02:04:25.053016 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 49e87840-df8b-4ec9-a45d-91cc40a97e6b
I0815 02:04:25.053028 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 0bd22c41-fc71-4844-bd81-85e6526adc42
I0815 02:04:25.053041 1742867 round_trippers.go:580]     Content-Length: 10877
I0815 02:04:25.053053 1742867 round_trippers.go:580]     Date: Tue, 15 Aug 2023 02:04:25 GMT
I0815 02:04:25.053073 1742867 round_trippers.go:580]     Accept-Ranges: bytes
I0815 02:04:25.054019 1742867 request.go:1188] Response Body: {"paths":{".well-known/openid-configuration":{"serverRelativeURL":"/openapi/v3/.well-known/openid-configuration?hash=860EC895A499D419847FB5C2819FB2CB2942ACA11E0E468103C631522DF46618C3935CD7D94FDE778740B9CDF9B757E8CA996097764C75F16B43C5B100528124"},"api":{"serverRelativeURL":"/openapi/v3/api?hash=0C2DE5E6F774C61AC67092C7159CF34F6D9F0A64CED7D3EF00A3D014FA7DD223F86A758BCCA62A6273C9FC4F49B396A31C4DE145C9B5758B4F5E33D8E28E9095"},"api/v1":{"serverRelativeURL":"/openapi/v3/api/v1?hash=04A5CF45A4D23D1626ABD01A4AA04C41D368439019776A97119705FEAD377CD290DD17C580C25DEDB8818F9EF3DEAD91F7C870F39EE2DED05628EDF380DAD047"},"apis":{"serverRelativeURL":"/openapi/v3/apis?hash=6F03364EADA6E43BBE65FA0B7313BDF9FFEC026315C35598E219D3D57F3DC51AE8A6FBD4C0D77BEF4F447E737E5E5D2BB987D7624B27578B24C9219DCD6B698E"},"apis/admissionregistration.k8s.io":{"serverRelativeURL":"/openapi/v3/apis/admissionregistration.k8s.io?hash=1559FF3C7D137EEA7D089463EB5577036CCFD9F0D6C7B9C9B613D7A9C6041582791B7F5B1F781DDA37522F5668773F67D3DD2BCCE4721FE952F794C [truncated 9853 chars]
I0815 02:04:25.054396 1742867 round_trippers.go:463] GET https://master01:6443/openapi/v3/api/v1?hash=04A5CF45A4D23D1626ABD01A4AA04C41D368439019776A97119705FEAD377CD290DD17C580C25DEDB8818F9EF3DEAD91F7C870F39EE2DED05628EDF380DAD047&timeout=32s
I0815 02:04:25.054408 1742867 round_trippers.go:469] Request Headers:
I0815 02:04:25.054418 1742867 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0815 02:04:25.054427 1742867 round_trippers.go:473]     Accept: application/json
I0815 02:04:25.069601 1742867 round_trippers.go:574] Response Status: 200 OK in 15 milliseconds
I0815 02:04:25.069639 1742867 round_trippers.go:577] Response Headers:
I0815 02:04:25.069996 1742867 round_trippers.go:580]     Date: Sat, 13 May 2023 11:50:18 GMT
I0815 02:04:25.070085 1742867 round_trippers.go:580]     X-From-Cache: 1
I0815 02:04:25.070176 1742867 round_trippers.go:580]     Cache-Control: public, immutable
I0815 02:04:25.070359 1742867 round_trippers.go:580]     Vary: Accept
I0815 02:04:25.070499 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 49e87840-df8b-4ec9-a45d-91cc40a97e6b
I0815 02:04:25.070624 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 0bd22c41-fc71-4844-bd81-85e6526adc42
I0815 02:04:25.070760 1742867 round_trippers.go:580]     X-Varied-Accept: application/json
I0815 02:04:25.070932 1742867 round_trippers.go:580]     Accept-Ranges: bytes
I0815 02:04:25.071060 1742867 round_trippers.go:580]     Audit-Id: 932f4f05-1546-40fb-9de7-1814a8f333d6
I0815 02:04:25.071194 1742867 round_trippers.go:580]     Etag: "04A5CF45A4D23D1626ABD01A4AA04C41D368439019776A97119705FEAD377CD290DD17C580C25DEDB8818F9EF3DEAD91F7C870F39EE2DED05628EDF380DAD047"
I0815 02:04:25.071329 1742867 round_trippers.go:580]     Expires: Mon, 13 May 2024 11:50:18 UTC
I0815 02:04:25.071441 1742867 round_trippers.go:580]     Last-Modified: Fri, 12 May 2023 09:47:24 GMT
I0815 02:04:25.071518 1742867 round_trippers.go:580]     Content-Length: 1345444
I0815 02:04:25.071590 1742867 round_trippers.go:580]     Content-Type: application/json
I0815 02:04:25.079158 1742867 request.go:1188] Response Body: {"openapi":"3.0.0","info":{"title":"Kubernetes","version":"v1.27.0"},"paths":{"/api/v1/":{"get":{"tags":["core_v1"],"description":"get available resources","operationId":"getCoreV1APIResources","responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"$ref":"#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList"}},"application/vnd.kubernetes.protobuf":{"schema":{"$ref":"#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList"}},"application/yaml":{"schema":{"$ref":"#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList"}}}},"401":{"description":"Unauthorized"}}}},"/api/v1/componentstatuses":{"get":{"tags":["core_v1"],"description":"list objects of kind ComponentStatus","operationId":"listCoreV1ComponentStatus","responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"$ref":"#/components/schemas/io.k8s.api.core.v1.ComponentStatusList"}},"application/json;stream=watch":{"schema":{"$ref":"#/components/schemas/i [truncated 1344420 chars]
I0815 02:04:25.133681 1742867 round_trippers.go:463] GET https://master01:6443/api/v1/namespaces/default/pods/testpod
I0815 02:04:25.133997 1742867 round_trippers.go:469] Request Headers:
I0815 02:04:25.134136 1742867 round_trippers.go:473]     Accept: application/json
I0815 02:04:25.134272 1742867 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0815 02:04:25.137084 1742867 round_trippers.go:574] Response Status: 404 Not Found in 2 milliseconds
I0815 02:04:25.137115 1742867 round_trippers.go:577] Response Headers:
I0815 02:04:25.137132 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 49e87840-df8b-4ec9-a45d-91cc40a97e6b
I0815 02:04:25.137292 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 0bd22c41-fc71-4844-bd81-85e6526adc42
I0815 02:04:25.137319 1742867 round_trippers.go:580]     Content-Length: 182
I0815 02:04:25.137339 1742867 round_trippers.go:580]     Date: Tue, 15 Aug 2023 02:04:25 GMT
I0815 02:04:25.137362 1742867 round_trippers.go:580]     Audit-Id: 1d9325fa-ea03-4cf9-9cca-547b2f0bb611
I0815 02:04:25.137382 1742867 round_trippers.go:580]     Cache-Control: no-cache, private
I0815 02:04:25.137516 1742867 round_trippers.go:580]     Content-Type: application/json
I0815 02:04:25.137563 1742867 request.go:1188] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"testpod\" not found","reason":"NotFound","details":{"name":"testpod","kind":"pods"},"code":404}
I0815 02:04:25.137918 1742867 request.go:1188] Request Body: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"testpod\"},\"name\":\"testpod\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"testpod\"}],\"dnsPolicy\":\"ClusterFirst\",\"nodeSelector\":{\"gpu\":true},\"restartPolicy\":\"Always\"}}\n"},"labels":{"run":"testpod"},"name":"testpod","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"testpod"}],"dnsPolicy":"ClusterFirst","nodeSelector":{"gpu":true},"restartPolicy":"Always"}}
I0815 02:04:25.138244 1742867 round_trippers.go:463] POST https://master01:6443/api/v1/namespaces/default/pods?fieldManager=kubectl-client-side-apply&fieldValidation=Strict
I0815 02:04:25.138436 1742867 round_trippers.go:469] Request Headers:
I0815 02:04:25.138625 1742867 round_trippers.go:473]     Content-Type: application/json
I0815 02:04:25.138744 1742867 round_trippers.go:473]     Accept: application/json
I0815 02:04:25.138856 1742867 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0815 02:04:25.140559 1742867 round_trippers.go:574] Response Status: 400 Bad Request in 1 milliseconds
I0815 02:04:25.140811 1742867 round_trippers.go:577] Response Headers:
apiVersion: v1
I0815 02:04:25.141080 1742867 round_trippers.go:580]     Cache-Control: no-cache, private
I0815 02:04:25.141204 1742867 round_trippers.go:580]     Content-Type: application/json
I0815 02:04:25.141231 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 49e87840-df8b-4ec9-a45d-91cc40a97e6b
I0815 02:04:25.141344 1742867 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 0bd22c41-fc71-4844-bd81-85e6526adc42
I0815 02:04:25.141369 1742867 round_trippers.go:580]     Content-Length: 254
I0815 02:04:25.141511 1742867 round_trippers.go:580]     Date: Tue, 15 Aug 2023 02:04:25 GMT
I0815 02:04:25.141537 1742867 round_trippers.go:580]     Audit-Id: 6f4b74f5-139c-4d52-bbe5-f1f6811bbaa9
I0815 02:04:25.141581 1742867 request.go:1188] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: json: cannot unmarshal bool into Go struct field PodSpec.spec.nodeSelector of type string","reason":"BadRequest","code":400}
I0815 02:04:25.141913 1742867 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"testpod.yaml\": Pod in version \"v1\" cannot be handled as a Pod: json: cannot unmarshal bool into Go struct field PodSpec.spec.nodeSelector of type string",
  "reason": "BadRequest",
  "code": 400
}]
Error from server (BadRequest): error when creating "testpod.yaml": Pod in version "v1" cannot be handled as a Pod: json: cannot unmarshal bool into Go struct field PodSpec.spec.nodeSelector of type string



解決方法
on/offを" "(ダブルクォーテーション)や' '(シングルクォーテーション)で囲う

具体的には、Podのyamlファイルで以下のようにしてやる。

  nodeSelector:
    gpu: "on"
  nodeSelector:
    gpu: 'on'


これで "nodeSelector":{"gpu":"on"}というように処理される。

kubeuser@master01:~/nodeSelector$ kubectl apply -f testpod.yaml --v=8
.
.
.
省略
.
.
.
I0815 02:42:44.891207 1762996 request.go:1188] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"testpod\" not found","reason":"NotFound","details":{"name":"testpod","kind":"pods"},"code":404}
I0815 02:42:44.892273 1762996 request.go:1188] Request Body: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"testpod\"},\"name\":\"testpod\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"testpod\"}],\"dnsPolicy\":\"ClusterFirst\",\"nodeSelector\":{\"gpu\":\"on\"},\"restartPolicy\":\"Always\"}}\n"},"labels":{"run":"testpod"},"name":"testpod","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"testpod"}],"dnsPolicy":"ClusterFirst","nodeSelector":{"gpu":"on"},"restartPolicy":"Always"}}



参考情報
YAML1.1
https://yaml.org/type/bool.html
YAML1.2 ※true/falseのみに変更されている
https://yaml.org/spec/1.2.2/
kubectl / kubernetes not 100% compatible with YAML 1.2 #34146
　※kubernetesってYAML1.2に準拠しねぇのかっていう議論
https://github.com/kubernetes/kubernetes/issues/34146
　　⇒影響が大きく簡単には準拠させるのは難しそうですね。。。
本家ラベルに関するページ
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set


おわりに
当初、kubernetesでnodeSelectorやNode Affinityをやってみようと思っていたところからまさかの副産物。。。
実際に手を動かしてみてみる、やっぱり大事ですねw

Opensourcetech by Takahiro Kujirai