LinuCエヴァンジェリスト・Open Source Summit Japan 2022ボランティアリーダーの鯨井貴博@opensourcetechです。
はじめに
今回はkubernetesのPod(コンテナ)のパケットキャプチャを取得してみようと思います。
※kubectlのプラグインksniffを使います。
ksniffとは
ksniffは、kubernetesクラスター内の指定Podにおけるパケットキャプチャを可能にするkubectlのプラグインです。
ksniff
Capture packets in Kubernetes with this open source tool
当然ですが、手元にkuberneteクラスターがあるという前提となります。
以下のクラスターを使ってます。
KubernetesクラスターのUpgrade(1.26 to 1.27)
ksniffの導入
①gitのインストール
今回の環境だと既にインストール済みでした。
未インストールの場合は、aptなどを使ってインストールしましょう。
kubeuser@master01:~$ git --version git version 2.34.1
②kubectlプラグインマネージャ"krew"のインストール
以下を参考にインストールします。
krew
krewのインストール方法
krewの使い方
kubeuser@master01:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:10:18Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:04:24Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
kubeuser@master01:~$ (
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
++ mktemp -d
+ cd /tmp/tmp.MVOAgxbryj
++ uname
++ tr '[:upper:]' '[:lower:]'
+ OS=linux
++ uname -m
++ sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'
+ ARCH=amd64
+ KREW=krew-linux_amd64
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
# append to the history file, don't overwrite it
shopt -s histappend
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000
# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize
# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar
# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
xterm-color|*-256color) color_prompt=yes;;
esac
# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes
if [ -n "$force_color_prompt" ]; then
if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
# We have color support; assume it's compliant with Ecma-48
# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
# a case would tend to support setf rather than setaf.)
color_prompt=yes
else
color_prompt=
fi
fi
if [ "$color_prompt" = yes ]; then
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt
# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
;;
*)
;;
esac
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
+ curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz
+ tar zxvf krew-linux_amd64.tar.gz
./LICENSE
./krew-linux_amd64
+ ./krew-linux_amd64 install krew
Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.
Updated the local copy of plugin index.
Installing plugin: krew
Installed plugin: krew
\
| Use this plugin:
| kubectl krew
| Documentation:
| https://krew.sigs.k8s.io/
| Caveats:
| \
| | krew is now installed! To start using kubectl plugins, you need to add
| | krew's installation directory to your PATH:
| |
| | * macOS/Linux:
| | - Add the following to your ~/.bashrc or ~/.zshrc:
| | export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
| | - Restart your shell.
| |
| | * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable
| |
| | To list krew commands and to get help, run:
| | $ kubectl krew
| | For a full list of available plugins, run:
| | $ kubectl krew search
| |
| | You can find documentation at
| | https://krew.sigs.k8s.io/docs/user-guide/quickstart/.
| /
/
kubeuser@master01:~$ vi .bashrc
kubeuser@master01:~$ cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
# append to the history file, don't overwrite it
shopt -s histappend
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000
# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize
# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar
# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
xterm-color|*-256color) color_prompt=yes;;
esac
# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes
if [ -n "$force_color_prompt" ]; then
if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
# We have color support; assume it's compliant with Ecma-48
# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
# a case would tend to support setf rather than setaf.)
color_prompt=yes
else
color_prompt=
fi
fi
if [ "$color_prompt" = yes ]; then
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt
# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
;;
*)
;;
esac
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'
# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'
# Add an "alert" alias for long running commands. Use like so:
# sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'
# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
#add for kubectl krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
その後、shellの再起動(今回は一回ログアウトして再ログインした)してbashに.bashrcを読み込ませます。
③plugin(sniff)のインストール
kubectl krewを使ってsniffをインストールします。
le custom resources with Giant Swarm no hlf Deploy and manage Hyperledger Fabric components no hns Manage hierarchical namespaces (part of HNC) no htpasswd Create nginx-ingress compatible basic-auth secrets no ice View configuration settings of containers insid... no iexec Interactive selection tool for `kubectl exec` no images Show container images used in the cluster. no ingress-nginx Interact with ingress-nginx no ingress-rule Update Ingress rules via command line no ipick A kubectl wrapper for interactive resource sele... no istiolog Manipulate istio-proxy logging level without is... no janitor Lists objects in a problematic state no kadalu Manage Kadalu Operator, CSI and Storage pods no karbon Connect to Nutanix Karbon cluster no karmada Manage clusters with Karmada federation. no kc Interactive CRUD operations to manage kubeconfig no klock Watches resources no kluster-capacity Scheduler simulation for capacity analysis. no konfig Merge, split or import kubeconfig files no kopilot Diagnose/audit resources with AI no krew Package manager for kubectl plugins. yes kruise Easily handle OpenKruise workloads no ks Simple management of KubeSphere components no ktop A top tool to display workload metrics no kubescape Scan resources and cluster configs against secu... no kubesec-scan Scan Kubernetes resources with kubesec.io. no kudo Declaratively build, install, and run operators... no kuota-calc Calculate needed quota to perform rolling updates. no kurt Find what's restarting and why no kuttl Declaratively run and test operators no kyverno Kyverno is a policy engine for kubernetes no lineage Display all dependent resources or resource dep... no linstor View and manage LINSTOR storage resources no liqo Install and manage Liqo on your clusters no log2rbac Fine-tune your RBAC using log2rbac operator no marvin Scan clusters with your own checks written in CEL. no match-name Match names of pods and other API objects no mc Run kubectl commands against multiple clusters ... no minio Deploy and manage MinIO Operator and Tenant(s) no moco Interact with MySQL operator MOCO. no modify-secret modify secret with implicit base64 translations no mtail Tail logs from multiple pods matching label sel... no multiforward Port Forward to multiple Kubernetes Services no multinet Shows pods' network-status of multi-net-spec no neat Remove clutter from Kubernetes manifests to mak... no net-forward Proxy to arbitrary TCP services on a cluster ne... no node-admin List nodes and run privileged pod with chroot no node-restart Restart cluster nodes sequentially and gracefully no node-shell Spawn a root shell on a node via kubectl no node-ssm start aws ssm session to SSM managed EKS node no nodepools List node pools/groups no np-viewer Network Policies rules viewer no ns Switch between Kubernetes namespaces no nsenter Run shell command in Pod's namespace on the nod... no oidc-login Log in to the OpenID Connect provider no oomd Show recently OOMKilled pods no open-svc Open the Kubernetes URL(s) for the specified se... no openebs View and debug OpenEBS storage resources no operator Manage operators with Operator Lifecycle Manager no oulogin Login to a cluster via OpenUnison no outdated Finds outdated container images running in a cl... no passman Store kubeconfig credentials in keychains or pa... no permissions Displays and traces service account permissions no pexec Execute process with privileges in a pod no pod-dive Shows a pod's workload tree and info inside a node no pod-inspect Get all of a pod's details at a glance no pod-lens Show pod-related resources no pod-logs Display a list of pods to get logs from no pod-shell Display a list of pods to execute a shell in no podevents Show events for pods no popeye Scans your clusters for potential resource issues no preflight Executes application preflight tests in a cluster no print-env Build config files from k8s environments. no profefe Gather and manage pprof profiles from running pods no promdump Dumps the head and persistent blocks of Prometh... no prompt Prompts for user confirmation when executing co... no prune-unused Prune unused resources no psp-util Manage Pod Security Policy(PSP) and the related... no pv-migrate Migrate data across persistent volumes no pvmigrate Migrates PVs between StorageClasses no rabbitmq Manage RabbitMQ clusters no rbac-lookup Reverse lookup for RBAC no rbac-tool Plugin to analyze RBAC permissions and generate... no rbac-view A tool to visualize your RBAC permissions. no realname-diff Diffs live and local resources ignoring Kustomi... no reap Delete unused Kubernetes resources. no relay Drop-in "port-forward" replacement with UDP and... no reliably Surfaces reliability issues in Kubernetes no rename-pvc Rename a PersistentVolumeClaim (PVC) no resource-capacity Provides an overview of resource requests, limi... no resource-snapshot Prints a snapshot of nodes, pods and HPAs resou... no resource-versions Print supported API resource versions no restart Restarts a pod with the given name no rm-standalone-pods Remove all pods without owner references no rolesum Summarize RBAC roles for subjects no roll Rolling restart of all persistent pods in a nam... no rook-ceph Rook plugin for Ceph management no safe Prompts before running edit commands no schemahero Declarative database schema migrations via YAML no score Kubernetes static code analysis. no secretdata Viewing decoded Secret data with search flags no service-tree Status for ingresses, services, and their backends no shovel Gather diagnostics for .NET Core applications no sick-pods Find and debug Pods that are "Not Ready" no skew Find if your cluster/kubectl version is skewed no slice Split a multi-YAML file into individual files. no snap Delete half of the pods in a namespace or cluster no sniff Start a remote packet capture on pods using tcp... no socks5-proxy SOCKS5 proxy to Services or Pods in the cluster no sort-manifests Sort manifest files in a proper order by Kind no split-yaml Split YAML output into one file per resource. no spy pod debugging tool for kubernetes clusters with... no sql Query the cluster via pseudo-SQL no ssh-jump Access nodes or services using SSH jump Pod no sshd Run SSH server in a Pod no ssm-secret Import/export secrets from/to AWS SSM param store no starboard Toolkit for finding risks in kubernetes resources no status Show status details of a given resource. no stern Multi pod and container log tailing no strace Capture strace logs from a running workload no sudo Run Kubernetes commands impersonated as group s... no support-bundle Creates support bundles for off-cluster analysis no switch-config Switches between kubeconfig files no tail Stream logs from multiple pods and containers u... no tap Interactively proxy Kubernetes Services with ease no tmux-exec An exec multiplexer using Tmux no topology Explore region topology for nodes or pods no trace Trace Kubernetes pods and nodes with system tools no tree Show a tree of object hierarchies through owner... no ttsum Visualize taints and tolerations no tunnel Reverse tunneling between cluster and your machine no unlimited Show running containers with no limits set no unused-volumes List unused PVCs no vela Easily interact with KubeVela no view-allocations List allocations per resources, nodes, pods. no view-cert View certificate information stored in secrets no view-secret Decode Kubernetes secrets no view-serviceaccount-kubeconfig Show a kubeconfig setting to access the apiserv... no view-utilization Shows cluster cpu and memory utilization no view-webhook Visualize your webhook configurations no viewnode Displays nodes with their pods and containers a... no virt Control KubeVirt virtual machines using virtctl no volsync Manage replication with the VolSync operator no vpa-recommendation Compare VPA recommendations to actual resources... no warp Sync and execute local files in Pod no whisper-secret Create secrets with improved privacy no who-can Shows who has RBAC permissions to access Kubern... no whoami Show the subject that's currently authenticated... no windows-debug Windows node access via kubectl no kubeuser@master01:~$ kubectl krew install sniff Updated the local copy of plugin index. Installing plugin: sniff Installed plugin: sniff \ | Use this plugin: | kubectl sniff | Documentation: | https://github.com/eldadru/ksniff | Caveats: | \ | | This plugin needs the following programs: | | * wireshark (optional, used for live capture) | / / WARNING: You installed plugin "sniff" from the krew-index plugin repository. These plugins are not audited for security by the Krew maintainers. Run them at your own risk.
パケットキャプチャしてみる
導入できたので、早速キャプチャしてみます。
使い方ですが、以下の通りです。
Pod(コンテナ)にwiresharkなどパケットキャプチャツールが含まれていればいいのですが、そういうケースはあまりないのかなと思うので今回は[-o output-file]を使ってファイルに格納します。
kubeuser@master01:~$ kubectl sniff
Usage:
sniff pod [-n namespace] [-c container] [-f filter] [-o output-file] [-l local-tcpdump-path] [-r remote-tcpdump-path] [flags]
Examples:
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube
Flags:
-c, --container string container (optional)
-x, --context string kubectl context to work on (optional)
-f, --filter string tcpdump filter (optional)
-h, --help help for sniff
--image string the privileged container image (optional)
-i, --interface string pod interface to packet capture (optional) (default "any")
-l, --local-tcpdump-path string local static tcpdump binary path (optional)
-n, --namespace string namespace (optional)
-o, --output-file string output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
--pod-creation-timeout duration the length of time to wait for privileged pod to be created (e.g. 20s, 2m, 1h). A value of zero means the creation never times out. (default 1m0s)
-p, --privileged if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
-r, --remote-tcpdump-path string remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
--socket string the container runtime socket path (optional)
--tcpdump-image string the tcpdump container image (optional)
-v, --verbose if specified, ksniff output will include debug information (optional)
Error: not enough arguments
パケットキャプチャの対象とするPodの確認。
nginx2-f96cfc57b-vnsfmにします。
kubeuser@master01:~$ kubectl get pods NAME READY STATUS RESTARTS AGE hpa-nginx-668b69fd79-k7d2q 1/1 Running 1 (23d ago) 62d nginx-jm8mr 1/1 Running 0 74d nginx-6cbc9bb4f5-pfhfz 1/1 Running 0 75d nginx2-f96cfc57b-vnsfm 1/1 Running 0 75d test-webserver 1/1 Running 2 (23d ago) 59d webserver-master01 1/1 Running 0 25d
Pod名していでsniffを実行します。
kubeuser@master01:~$ kubectl sniff nginx2-f96cfc57b-vnsfm -o ./test2.pcap INFO[0000] using tcpdump path at: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: 'nginx2' INFO[0000] sniffing method: upload static tcpdump INFO[0000] sniffing on pod: 'nginx2-f96cfc57b-vnsfm' [namespace: 'default', container: 'nginx2', filter: '', interface: 'any'] INFO[0000] uploading static tcpdump binary from: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump' INFO[0000] uploading file: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx2' INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx2', pod: 'nginx2-f96cfc57b-vnsfm', namespace: 'default' INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' INFO[0000] file found: '' INFO[0000] file was already found on remote pod INFO[0000] tcpdump uploaded successfully INFO[0000] output file option specified, storing output in: './test2.pcap' INFO[0000] start sniffing on remote container INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx2', pod: 'nginx2-f96cfc57b-vnsfm', namespace: 'default'
パケットキャプチャを開始したら、そのPodへ通信が割り振られるServiceへアクセスします。
kubeuser@master01:~$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hpa-nginx LoadBalancer 10.1.114.112 192.168.1.53 80:31842/TCP 62d kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 84d nginx LoadBalancer 10.1.55.116 192.168.1.51 80:30267/TCP 75d test-webserver LoadBalancer 10.1.80.108 192.168.1.54 80:31241/TCP 59d
何回かアクセスしたら、ctrl+cでパケットキャプチャを停止します。
kubeuser@master01:~$ ll test2.pcap -rw-rw-r-- 1 kubeuser kubeuser 16355 Jun 7 02:12 test2.pcap
あとは、scpなどでWiresharkなどが導入済みのPCへコピーしてファイルを開けばOKです。
おわりに
Podへのヘルスチェックが失敗した・Podで障害が発生したなどの場合にはkubernetesの機能として自動的にPod再起動が行われるので利用する場面(開発環境？)は限定的かもしれませんが、かゆいところに届くいいツールですね！
トレーシングで俯瞰的に処理の問題点を見つけて、sniffで深堀とかするとよさそうだなぁ。
