Kubernetes: debug Running Pods

Fleeting

• External reference: https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/

kubernetes, Debug Running Pods

to test how a container restart

to dig into the filesystem of some distroless container (/proc/1/root)

kubectl debug with the --target option¹.

k debug -ti pod/clk-collector-6cd64d897b-dxdk9 --image busybox --target otc-container

get access to the filesystem

in the debug container, /proc/1/root is the filesystem of the main container.

But you may have a permission denied. In that case, look at its group and user ls -n /proc/1/root lrwxrwxrwx 1 10001 0 0 Nov 3 16:55 /proc/1/root

Then, create a user with those exact user and group id.

adduser -u 10001 -G root test

Then su to that user and you will have access to the container filesystem.

looking at the listening ports

The debug container share the same network namespace, so simply run netstat -tupln

to start another pod like the one you want to debug with slightly modified settings

changing the image and the command of the container

k debug -ti pod/clk-collector-6cd64d897b-dxdk9 --image busybox --container otc-container -- sh²

creating a copy of the pod with a debug container inside³

k debug -ti pod/clk-collector-6cd64d897b-dxdk9 --image busybox --share-process -- sh

this is slightly less practical than to dig into the filesystem of some distroless container as /proc/1 is not necessarily the pod to debug here. .

run the same pod with a different command⁴

k debug -ti pod/clk-collector-6cd64d897b-dxdk9 --copy-to test --container otc-container -- sh

This won’t help for distroless ones.

It may be useful in combination with --set-image to provide a devel version of the image, not distroless⁵.

debug the node itself

kubectl debug node/mynode -it –image=ubuntu

— https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/

This will mount the node filesystem under /host

Permalink