LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
はじめに
今回は、kubernetesのPodログ出力ツール、sternを使ってみます。
sternとは
sternとは、kubernetesの複数Podのログを出力できるソフトウェアです。
https://github.com/stern/stern
よくあるやり方だと、以下のようにkubectl logsを使って指定したPodのログを確認します。
kubeuser@master01:~$ kubectl logs nginx-6cbc9bb4f5-pfhfz /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2023/03/23 12:45:59 [notice] 1#1: using the "epoll" event method 2023/03/23 12:45:59 [notice] 1#1: nginx/1.23.0 2023/03/23 12:45:59 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 2023/03/23 12:45:59 [notice] 1#1: OS: Linux 5.15.0-67-generic 2023/03/23 12:45:59 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2023/03/23 12:45:59 [notice] 1#1: start worker processes 2023/03/23 12:45:59 [notice] 1#1: start worker process 31 2023/03/23 12:45:59 [notice] 1#1: start worker process 32 10.0.241.64 - - [23/Mar/2023:12:48:18 +0000] "GET / HTTP/1.1" 200 256 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0" "-" . . .
しかし、上記では例えば同じDeploymentに含まれる複数のPodのログを同時に確認したいなどの用途には不向きです。
そこでsternの出番となります。
sternのダウンロード
では、sternを使っていきましょう。
なお、kubernetesクラスターはこちらの記事で作ったものを使っています。
まず、以下から利用する環境に合ったファイルをダウンロードします。
https://github.com/stern/stern/releases
kubeuser@master01:~$ cd /tmp kubeuser@master01:/tmp$ wget https://github.com/stern/stern/releases/download/v1.25.0/stern_1.25.0_linux_amd64.tar.gz https://github.com/stern/stern/releases/download/v1.25.0/stern_1.25.0_linux_amd64.tar.gz --2023-04-23 11:03:10-- https://github.com/stern/stern/releases/download/v1.25.0/stern_1.25.0_linux_amd64.tar.gz Resolving github.com (github.com)... 20.27.177.113 Connecting to github.com (github.com)|20.27.177.113|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/306013800/7fbd05dd-084e-4375-9e36-41763de70a10?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230423T110310Z&X-Amz-Expires=300&X-Amz-Signature=025c17d3a7adc3dc91b31e1049a644a3cd519691436b61f4645b7dfef9cde0eb&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=306013800&response-content-disposition=attachment%3B%20filename%3Dstern_1.25.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2023-04-23 11:03:10-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/306013800/7fbd05dd-084e-4375-9e36-41763de70a10?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230423T110310Z&X-Amz-Expires=300&X-Amz-Signature=025c17d3a7adc3dc91b31e1049a644a3cd519691436b61f4645b7dfef9cde0eb&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=306013800&response-content-disposition=attachment%3B%20filename%3Dstern_1.25.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ... Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 11913988 (11M) [application/octet-stream] Saving to: ‘stern_1.25.0_linux_amd64.tar.gz’ stern_1.25.0_linux_amd64.tar.gz 100%[===========================================================>] 11.36M 11.1MB/s in 1.0s 2023-04-23 11:03:12 (11.1 MB/s) - ‘stern_1.25.0_linux_amd64.tar.gz’ saved [11913988/11913988] kubeuser@master01:/tmp$ ls -lh total 48M -rw-r--r-- 1 kubeuser kubeuser 12K Apr 13 23:04 LICENSE drwx------ 3 root root 4.0K Mar 10 07:53 snap-private-tmp -rwxr-xr-x 1 kubeuser kubeuser 37M Apr 13 23:08 stern -rw-rw-r-- 1 kubeuser kubeuser 12M Apr 13 23:16 stern_1.25.0_linux_amd64.tar.gz drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-ModemManager.service-Ds1zyj drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-logind.service-sHCFky drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-resolved.service-ty0UFz drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-timesyncd.service-M6kZ5e drwx------ 3 root root 4.0K Mar 10 22:17 systemd-private-12876a8f657245b6bc203004c2e26ba5-upower.service-BkdU2n
tar.gzを解凍します。
kubeuser@master01:/tmp$ tar zxvf stern_1.25.0_linux_amd64.tar.gz LICENSE stern kubeuser@master01:/tmp$ ls -lh total 12M -rw-r--r-- 1 kubeuser kubeuser 12K Apr 13 23:04 LICENSE drwx------ 3 root root 4.0K Mar 10 07:53 snap-private-tmp -rwxr-xr-x 1 kubeuser kubeuser 37M Apr 13 23:08 stern -rw-rw-r-- 1 kubeuser kubeuser 12M Apr 13 23:16 stern_1.25.0_linux_amd64.tar.gz drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-ModemManager.service-Ds1zyj drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-logind.service-sHCFky drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-resolved.service-ty0UFz drwx------ 3 root root 4.0K Mar 10 07:52 systemd-private-12876a8f657245b6bc203004c2e26ba5-systemd-timesyncd.service-M6kZ5e drwx------ 3 root root 4.0K Mar 10 22:17 systemd-private-12876a8f657245b6bc203004c2e26ba5-upower.service-BkdU2n
実行ファイルsternのコピー
続いて、sternの実行ファイルをPATHにあるディレクトリにコピー(もしくは、実行ファイルがあるディレクトリをPATHに追加する)します。
今回は、/usr/binにコピーしました。
kubeuser@master01:/tmp$ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin kubeuser@master01:/tmp$ cp -p stern /usr/bin/ [sudo] password for kubeuser: kubeuser@master01:/tmp$ which stern /usr/bin/stern
sternによるログ出力
まず、使い方の確認から。
ざっくりですが、stern Pod名に含まれる文字列 (-n namespace指定)とすればOKです。
kubeuser@master01:/tmp$ stern --help
Tail multiple pods and containers from Kubernetes
Usage:
stern pod-query [flags]
Flags:
-A, --all-namespaces If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
--color string Force set color output. 'auto': colorize if tty attached, 'always': always colorize, 'never': never colorize. (default "auto")
--completion string Output stern command-line completion code for the specified shell. Can be 'bash', 'zsh' or 'fish'.
--config string Path to the stern config file (default "~/.config/stern/config.yaml")
-c, --container string Container name when multiple containers in pod. (regular expression) (default ".*")
--container-state strings Tail containers with state in running, waiting, terminated, or all. 'all' matches all container states. To specify multiple states, repeat this or set comma-separated value. (default [all])
--context string Kubernetes context to use. Default to current context configured in kubeconfig.
--ephemeral-containers Include or exclude ephemeral containers. (default true)
-e, --exclude stringArray Log lines to exclude. (regular expression)
-E, --exclude-container stringArray Container name to exclude when multiple containers in pod. (regular expression)
--exclude-pod stringArray Pod name to exclude. (regular expression)
--field-selector string Selector (field query) to filter on. If present, default to ".*" for the pod-query.
-h, --help help for stern
-i, --include stringArray Log lines to include. (regular expression)
--init-containers Include or exclude init containers. (default true)
--kubeconfig string Path to kubeconfig file to use. Default to KUBECONFIG variable then ~/.kube/config path.
--max-log-requests int Maximum number of concurrent logs to request. Defaults to 50, but 5 when specifying --no-follow (default -1)
-n, --namespace strings Kubernetes namespace to use. Default to namespace configured in kubernetes context. To specify multiple namespaces, repeat this or set comma-separated value.
--no-follow Exit when all logs have been shown.
--node string Node name to filter on.
--only-log-lines Print only log lines
-o, --output string Specify predefined template. Currently support: [default, raw, json, extjson, ppextjson] (default "default")
-p, --prompt Toggle interactive prompt for selecting 'app.kubernetes.io/instance' label values.
-l, --selector string Selector (label query) to filter on. If present, default to ".*" for the pod-query.
-s, --since duration Return logs newer than a relative duration like 5s, 2m, or 3h. (default 48h0m0s)
--tail int The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
--template string Template to use for log lines, leave empty to use --output flag.
-T, --template-file string Path to template to use for log lines, leave empty to use --output flag. It overrides --template option.
-t, --timestamps string[="default"] Print timestamps with the specified format. One of 'default' or 'short'. If specified but without value, 'default' is used.
--timezone string Set timestamps to specific timezone. (default "Local")
--verbosity int Number of the log level verbosity
-v, --version Print the version and exit.
kubeuser@master01:/tmp$ stern --version
version: 1.25.0
commit: f13bde422e977c7a69ec0827c0337b2bc8e44444
built at: 2023-04-13T23:06:41Z
namespace "default"で起動しているnginx-6cbc9bb4f5とつくPodをターゲットにします。
kubeuser@master01:~$ kubectl get pods NAME READY STATUS RESTARTS AGE hpa-nginx-668b69fd79-k7d2q 1/1 Running 0 20d nginx-6cbc9bb4f5-jm8mr 1/1 Running 0 32d nginx-6cbc9bb4f5-pfhfz 1/1 Running 0 32d nginx2-f96cfc57b-vnsfm 1/1 Running 0 32d test-webserver 1/1 Running 0 16d webserver-master01 1/1 Running 0 29d
kubeuser@master01:~$ stern nginx-6cbc9bb4f5 + nginx-6cbc9bb4f5-jm8mr › nginx + nginx-6cbc9bb4f5-pfhfz › nginx nginx-6cbc9bb4f5-jm8mr nginx 10.0.241.64 - - [24/Apr/2023:13:03:36 +0000] "GET / HTTP/1.1" 200 256 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0" "-" nginx-6cbc9bb4f5-jm8mr nginx 2023/04/24 13:03:36 [error] 31#31: *4333 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.0.241.64, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.51", referrer: "http://192.168.1.51/" nginx-6cbc9bb4f5-jm8mr nginx 10.0.241.64 - - [24/Apr/2023:13:03:36 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://192.168.1.51/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0" "-"
上記ログだけだと分からないですが、
sternのいいところは以下の画像のようにPodごとに色分けされている点です。
しかも、tail -fのようにリアルタイムログ確認も出来ます。
Serviceで提供されているIPアドレス(192.168.1.51)経由でPodへのアクセスを見てみましょう。
kubeuser@master01:~$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hpa-nginx LoadBalancer 10.1.114.112 192.168.1.53 80:31842/TCP 20d kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 41d nginx LoadBalancer 10.1.55.116 192.168.1.51 80:30267/TCP 33d test-webserver LoadBalancer 10.1.80.108 192.168.1.54 80:31241/TCP 16d
みやすくていいですね♪
おわりに
今回はsternを使ってみましたが、非常に便利ですね。
いい点としては繰り返しになりますが、以下が挙げられます。
・複数Podのログを同時出力
・リアルタイムログ出力
・Podの色分けによる可視化のしやすさ
・Pod名の指定が楽
これはぜひともkubernetes環境にはインストールしておきたいですね！
