Hello there! 👋

Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible.

Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers.

If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is probably the most obvious choice when things go sideways:

However, good security practices demand that our containers don't include debugging tools (or even shells) by default. Here is what you can do if the misbehaving container is based on a slim or a distroless image:

• Install debugging tools on the fly (slim).
• Temporarily switch to a "fat" base image (slim & distroless).
• Run an improvised debugger "sidecar" container using docker run and sharing target's namespaces.
• Run a proper debugger "sidecar" container with kubectl debug.

Quite a few options, huh? Well, from my experience, none of them is super user-friendly. Installing debugging tools on demand is tedious, getting the docker run flags by heart from the first attempt is hard, and kubectl debug is not as flexible as I'd like it to be (you cannot run the debugger as the given user, make the sidecar privileged, etc.).

Third-party tools to the rescue!

​cdebug is a my favorite tool that I wrote specifically to streamline the container debugging UX. It allows executing commands in scratch, distroless, and slim containers, using the debugger image of choice and making the debugger sidecar see the target's filesystem as-is. And it works the same for Docker, container, and, since recently, Kubernetes 🚀

Do recommend giving cdebug a try while solving today's challenges:

• ​Edit a File in a Running Container: a Simple Case​
• ​Edit a File in a Running Container: a Slim Base Image Case​
• ​Edit a File in a Running Kubernetes Pod: a Tricky Case​

Good luck!

P.S. Traditional reminder - with iximiuz Labs Premium, you can get 2-4x faster VMs, unlimited daily playtime, and full content access. Monthly, yearly, and lifetime plans are available, with proper invoices, so that you can include this expense in your employer's dev education budget 😎