Opensourcetechブログ

OpensourcetechによるNGINX/Kubernetes/Zabbix/Neo4j/Linuxなどオープンソース技術に関するブログです。

kubernetesのPod(コンテナ)のパケットキャプチャを取得する方法


LinuCエヴァンジェリスト・Open Source Summit Japan 2022ボランティアリーダー鯨井貴博@opensourcetechです。


はじめに
今回はkubernetesのPod(コンテナ)のパケットキャプチャを取得してみようと思います。
※kubectlのプラグインksniffを使います。


ksniffとは
ksniffは、kubernetesクラスター内の指定Podにおけるパケットキャプチャを可能にするkubectlのプラグインです。
ksniff
Capture packets in Kubernetes with this open source tool

当然ですが、手元にkuberneteクラスターがあるという前提となります。
以下のクラスターを使ってます。
KubernetesクラスターのUpgrade(1.26 to 1.27)

ksniffの導入
①gitのインストール
今回の環境だと既にインストール済みでした。
未インストールの場合は、aptなどを使ってインストールしましょう。

kubeuser@master01:~$ git --version
git version 2.34.1


②kubectlプラグインマネージャ"krew"のインストール
以下を参考にインストールします。
krew
krewのインストール方法
krewの使い方

kubeuser@master01:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:10:18Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:04:24Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}


kubeuser@master01:~$ (
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)
++ mktemp -d
+ cd /tmp/tmp.MVOAgxbryj
++ uname
++ tr '[:upper:]' '[:lower:]'
+ OS=linux
++ uname -m
++ sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'
+ ARCH=amd64
+ KREW=krew-linux_amd64
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi
+ curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz
+ tar zxvf krew-linux_amd64.tar.gz
./LICENSE
./krew-linux_amd64
+ ./krew-linux_amd64 install krew
Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.
Updated the local copy of plugin index.
Installing plugin: krew
Installed plugin: krew
\
 | Use this plugin:
 |      kubectl krew
 | Documentation:
 |      https://krew.sigs.k8s.io/
 | Caveats:
 | \
 |  | krew is now installed! To start using kubectl plugins, you need to add
 |  | krew's installation directory to your PATH:
 |  | 
 |  |   * macOS/Linux:
 |  |     - Add the following to your ~/.bashrc or ~/.zshrc:
 |  |         export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
 |  |     - Restart your shell.
 |  | 
 |  |   * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable
 |  | 
 |  | To list krew commands and to get help, run:
 |  |   $ kubectl krew
 |  | For a full list of available plugins, run:
 |  |   $ kubectl krew search
 |  | 
 |  | You can find documentation at
 |  |   https://krew.sigs.k8s.io/docs/user-guide/quickstart/.
 | /
/


kubeuser@master01:~$ vi .bashrc 

kubeuser@master01:~$ cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'

# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k

#add for kubectl krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"


その後、shellの再起動(今回は一回ログアウトして再ログインした)してbashに.bashrcを読み込ませます。
③plugin(sniff)のインストール
kubectl krewを使ってsniffをインストールします。

le custom resources with Giant Swarm            no
hlf                             Deploy and manage Hyperledger Fabric components     no
hns                             Manage hierarchical namespaces (part of HNC)        no
htpasswd                        Create nginx-ingress compatible basic-auth secrets  no
ice                             View configuration settings of containers insid...  no
iexec                           Interactive selection tool for `kubectl exec`       no
images                          Show container images used in the cluster.          no
ingress-nginx                   Interact with ingress-nginx                         no
ingress-rule                    Update Ingress rules via command line               no
ipick                           A kubectl wrapper for interactive resource sele...  no
istiolog                        Manipulate istio-proxy logging level without is...  no
janitor                         Lists objects in a problematic state                no
kadalu                          Manage Kadalu Operator, CSI and Storage pods        no
karbon                          Connect to Nutanix Karbon cluster                   no
karmada                         Manage clusters with Karmada federation.            no
kc                              Interactive CRUD operations to manage kubeconfig    no
klock                           Watches resources                                   no
kluster-capacity                Scheduler simulation for capacity analysis.         no
konfig                          Merge, split or import kubeconfig files             no
kopilot                         Diagnose/audit resources with AI                    no
krew                            Package manager for kubectl plugins.                yes
kruise                          Easily handle OpenKruise workloads                  no
ks                              Simple management of KubeSphere components          no
ktop                            A top tool to display workload metrics              no
kubescape                       Scan resources and cluster configs against secu...  no
kubesec-scan                    Scan Kubernetes resources with kubesec.io.          no
kudo                            Declaratively build, install, and run operators...  no
kuota-calc                      Calculate needed quota to perform rolling updates.  no
kurt                            Find what's restarting and why                      no
kuttl                           Declaratively run and test operators                no
kyverno                         Kyverno is a policy engine for kubernetes           no
lineage                         Display all dependent resources or resource dep...  no
linstor                         View and manage LINSTOR storage resources           no
liqo                            Install and manage Liqo on your clusters            no
log2rbac                        Fine-tune your RBAC using log2rbac operator         no
marvin                          Scan clusters with your own checks written in CEL.  no
match-name                      Match names of pods and other API objects           no
mc                              Run kubectl commands against multiple clusters ...  no
minio                           Deploy and manage MinIO Operator and Tenant(s)      no
moco                            Interact with MySQL operator MOCO.                  no
modify-secret                   modify secret with implicit base64 translations     no
mtail                           Tail logs from multiple pods matching label sel...  no
multiforward                    Port Forward to multiple Kubernetes Services        no
multinet                        Shows pods' network-status of multi-net-spec        no
neat                            Remove clutter from Kubernetes manifests to mak...  no
net-forward                     Proxy to arbitrary TCP services on a cluster ne...  no
node-admin                      List nodes and run privileged pod with chroot       no
node-restart                    Restart cluster nodes sequentially and gracefully   no
node-shell                      Spawn a root shell on a node via kubectl            no
node-ssm                        start aws ssm session to SSM managed EKS node       no
nodepools                       List node pools/groups                              no
np-viewer                       Network Policies rules viewer                       no
ns                              Switch between Kubernetes namespaces                no
nsenter                         Run shell command in Pod's namespace on the nod...  no
oidc-login                      Log in to the OpenID Connect provider               no
oomd                            Show recently OOMKilled pods                        no
open-svc                        Open the Kubernetes URL(s) for the specified se...  no
openebs                         View and debug OpenEBS storage resources            no
operator                        Manage operators with Operator Lifecycle Manager    no
oulogin                         Login to a cluster via OpenUnison                   no
outdated                        Finds outdated container images running in a cl...  no
passman                         Store kubeconfig credentials in keychains or pa...  no
permissions                     Displays and traces service account permissions     no
pexec                           Execute process with privileges in a pod            no
pod-dive                        Shows a pod's workload tree and info inside a node  no
pod-inspect                     Get all of a pod's details at a glance              no
pod-lens                        Show pod-related resources                          no
pod-logs                        Display a list of pods to get logs from             no
pod-shell                       Display a list of pods to execute a shell in        no
podevents                       Show events for pods                                no
popeye                          Scans your clusters for potential resource issues   no
preflight                       Executes application preflight tests in a cluster   no
print-env                       Build config files from k8s environments.           no
profefe                         Gather and manage pprof profiles from running pods  no
promdump                        Dumps the head and persistent blocks of Prometh...  no
prompt                          Prompts for user confirmation when executing co...  no
prune-unused                    Prune unused resources                              no
psp-util                        Manage Pod Security Policy(PSP) and the related...  no
pv-migrate                      Migrate data across persistent volumes              no
pvmigrate                       Migrates PVs between StorageClasses                 no
rabbitmq                        Manage RabbitMQ clusters                            no
rbac-lookup                     Reverse lookup for RBAC                             no
rbac-tool                       Plugin to analyze RBAC permissions and generate...  no
rbac-view                       A tool to visualize your RBAC permissions.          no
realname-diff                   Diffs live and local resources ignoring Kustomi...  no
reap                            Delete unused Kubernetes resources.                 no
relay                           Drop-in "port-forward" replacement with UDP and...  no
reliably                        Surfaces reliability issues in Kubernetes           no
rename-pvc                      Rename a PersistentVolumeClaim (PVC)                no
resource-capacity               Provides an overview of resource requests, limi...  no
resource-snapshot               Prints a snapshot of nodes, pods and HPAs resou...  no
resource-versions               Print supported API resource versions               no
restart                         Restarts a pod with the given name                  no
rm-standalone-pods              Remove all pods without owner references            no
rolesum                         Summarize RBAC roles for subjects                   no
roll                            Rolling restart of all persistent pods in a nam...  no
rook-ceph                       Rook plugin for Ceph management                     no
safe                            Prompts before running edit commands                no
schemahero                      Declarative database schema migrations via YAML     no
score                           Kubernetes static code analysis.                    no
secretdata                      Viewing decoded Secret data with search flags       no
service-tree                    Status for ingresses, services, and their backends  no
shovel                          Gather diagnostics for .NET Core applications       no
sick-pods                       Find and debug Pods that are "Not Ready"            no
skew                            Find if your cluster/kubectl version is skewed      no
slice                           Split a multi-YAML file into individual files.      no
snap                            Delete half of the pods in a namespace or cluster   no
sniff                           Start a remote packet capture on pods using tcp...  no
socks5-proxy                    SOCKS5 proxy to Services or Pods in the cluster     no
sort-manifests                  Sort manifest files in a proper order by Kind       no
split-yaml                      Split YAML output into one file per resource.       no
spy                             pod debugging tool for kubernetes clusters with...  no
sql                             Query the cluster via pseudo-SQL                    no
ssh-jump                        Access nodes or services using SSH jump Pod         no
sshd                            Run SSH server in a Pod                             no
ssm-secret                      Import/export secrets from/to AWS SSM param store   no
starboard                       Toolkit for finding risks in kubernetes resources   no
status                          Show status details of a given resource.            no
stern                           Multi pod and container log tailing                 no
strace                          Capture strace logs from a running workload         no
sudo                            Run Kubernetes commands impersonated as group s...  no
support-bundle                  Creates support bundles for off-cluster analysis    no
switch-config                   Switches between kubeconfig files                   no
tail                            Stream logs from multiple pods and containers u...  no
tap                             Interactively proxy Kubernetes Services with ease   no
tmux-exec                       An exec multiplexer using Tmux                      no
topology                        Explore region topology for nodes or pods           no
trace                           Trace Kubernetes pods and nodes with system tools   no
tree                            Show a tree of object hierarchies through owner...  no
ttsum                           Visualize taints and tolerations                    no
tunnel                          Reverse tunneling between cluster and your machine  no
unlimited                       Show running containers with no limits set          no
unused-volumes                  List unused PVCs                                    no
vela                            Easily interact with KubeVela                       no
view-allocations                List allocations per resources, nodes, pods.        no
view-cert                       View certificate information stored in secrets      no
view-secret                     Decode Kubernetes secrets                           no
view-serviceaccount-kubeconfig  Show a kubeconfig setting to access the apiserv...  no
view-utilization                Shows cluster cpu and memory utilization            no
view-webhook                    Visualize your webhook configurations               no
viewnode                        Displays nodes with their pods and containers a...  no
virt                            Control KubeVirt virtual machines using virtctl     no
volsync                         Manage replication with the VolSync operator        no
vpa-recommendation              Compare VPA recommendations to actual resources...  no
warp                            Sync and execute local files in Pod                 no
whisper-secret                  Create secrets with improved privacy                no
who-can                         Shows who has RBAC permissions to access Kubern...  no
whoami                          Show the subject that's currently authenticated...  no
windows-debug                   Windows node access via kubectl                     no


kubeuser@master01:~$ kubectl krew install sniff
Updated the local copy of plugin index.
Installing plugin: sniff
Installed plugin: sniff
\
 | Use this plugin:
 |      kubectl sniff
 | Documentation:
 |      https://github.com/eldadru/ksniff
 | Caveats:
 | \
 |  | This plugin needs the following programs:
 |  | * wireshark (optional, used for live capture)
 | /
/
WARNING: You installed plugin "sniff" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.



パケットキャプチャしてみる
導入できたので、早速キャプチャしてみます。

使い方ですが、以下の通りです。
Pod(コンテナ)にwiresharkなどパケットキャプチャツールが含まれていればいいのですが、そういうケースはあまりないのかなと思うので今回は[-o output-file]を使ってファイルに格納します。

kubeuser@master01:~$ kubectl sniff
Usage:
  sniff pod [-n namespace] [-c container] [-f filter] [-o output-file] [-l local-tcpdump-path] [-r remote-tcpdump-path] [flags]

Examples:
kubectl sniff hello-minikube-7c77b68cff-qbvsd -c hello-minikube

Flags:
  -c, --container string                container (optional)
  -x, --context string                  kubectl context to work on (optional)
  -f, --filter string                   tcpdump filter (optional)
  -h, --help                            help for sniff
      --image string                    the privileged container image (optional)
  -i, --interface string                pod interface to packet capture (optional) (default "any")
  -l, --local-tcpdump-path string       local static tcpdump binary path (optional)
  -n, --namespace string                namespace (optional)
  -o, --output-file string              output file path, tcpdump output will be redirect to this file instead of wireshark (optional) ('-' stdout)
      --pod-creation-timeout duration   the length of time to wait for privileged pod to be created (e.g. 20s, 2m, 1h). A value of zero means the creation never times out. (default 1m0s)
  -p, --privileged                      if specified, ksniff will deploy another pod that have privileges to attach target pod network namespace
  -r, --remote-tcpdump-path string      remote static tcpdump binary path (optional) (default "/tmp/static-tcpdump")
      --socket string                   the container runtime socket path (optional)
      --tcpdump-image string            the tcpdump container image (optional)
  -v, --verbose                         if specified, ksniff output will include debug information (optional)
Error: not enough arguments


パケットキャプチャの対象とするPodの確認。
nginx2-f96cfc57b-vnsfmにします。

kubeuser@master01:~$ kubectl get pods
NAME                         READY   STATUS    RESTARTS      AGE
hpa-nginx-668b69fd79-k7d2q   1/1     Running   1 (23d ago)   62d
nginx-jm8mr       1/1     Running   0             74d
nginx-6cbc9bb4f5-pfhfz       1/1     Running   0             75d
nginx2-f96cfc57b-vnsfm       1/1     Running   0             75d
test-webserver               1/1     Running   2 (23d ago)   59d
webserver-master01           1/1     Running   0             25d


Pod名していでsniffを実行します。

kubeuser@master01:~$ kubectl sniff nginx2-f96cfc57b-vnsfm  -o ./test2.pcap
INFO[0000] using tcpdump path at: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' 
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'nginx2'                 
INFO[0000] sniffing method: upload static tcpdump       
INFO[0000] sniffing on pod: 'nginx2-f96cfc57b-vnsfm' [namespace: 'default', container: 'nginx2', filter: '', interface: 'any'] 
INFO[0000] uploading static tcpdump binary from: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump' 
INFO[0000] uploading file: '/home/kubeuser/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx2' 
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx2', pod: 'nginx2-f96cfc57b-vnsfm', namespace: 'default' 
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' 
INFO[0000] file found: ''                               
INFO[0000] file was already found on remote pod         
INFO[0000] tcpdump uploaded successfully                
INFO[0000] output file option specified, storing output in: './test2.pcap' 
INFO[0000] start sniffing on remote container           
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx2', pod: 'nginx2-f96cfc57b-vnsfm', namespace: 'default'


パケットキャプチャを開始したら、そのPodへ通信が割り振られるServiceへアクセスします。

kubeuser@master01:~$ kubectl get svc
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
hpa-nginx        LoadBalancer   10.1.114.112   192.168.1.53   80:31842/TCP   62d
kubernetes       ClusterIP      10.1.0.1       <none>         443/TCP        84d
nginx            LoadBalancer   10.1.55.116    192.168.1.51   80:30267/TCP   75d
test-webserver   LoadBalancer   10.1.80.108    192.168.1.54   80:31241/TCP   59d


何回かアクセスしたら、ctrl+cでパケットキャプチャを停止します。

kubeuser@master01:~$ ll test2.pcap 
-rw-rw-r-- 1 kubeuser kubeuser 16355 Jun  7 02:12 test2.pcap


あとは、scpなどでWiresharkなどが導入済みのPCへコピーしてファイルを開けばOKです。



おわりに
Podへのヘルスチェックが失敗した・Podで障害が発生したなどの場合にはkubernetesの機能として自動的にPod再起動が行われるので利用する場面(開発環境?)は限定的かもしれませんが、かゆいところに届くいいツールですね!

トレーシングで俯瞰的に処理の問題点を見つけて、sniffで深堀とかするとよさそうだなぁ。

Opensourcetech by Takahiro Kujirai