DevOps Challenge of the Week - Issue #6


Hello there! 👋

Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible.

Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers.

If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is probably the most obvious choice when things go sideways:

However, good security practices demand that our containers don't include debugging tools (or even shells) by default. Here is what you can do if the misbehaving container is based on a slim or a distroless image:

  • Install debugging tools on the fly (slim).
  • Temporarily switch to a "fat" base image (slim & distroless).
  • Run an improvised debugger "sidecar" container using docker run and sharing target's namespaces.
  • Run a proper debugger "sidecar" container with kubectl debug.

Quite a few options, huh? Well, from my experience, none of them is super user-friendly. Installing debugging tools on demand is tedious, getting the docker run flags by heart from the first attempt is hard, and kubectl debug is not as flexible as I'd like it to be (you cannot run the debugger as the given user, make the sidecar privileged, etc.).

Third-party tools to the rescue!

​cdebug is a my favorite tool that I wrote specifically to streamline the container debugging UX. It allows executing commands in scratch, distroless, and slim containers, using the debugger image of choice and making the debugger sidecar see the target's filesystem as-is. And it works the same for Docker, container, and, since recently, Kubernetes 🚀

Do recommend giving cdebug a try while solving today's challenges:

Good luck!

P.S. Traditional reminder - with iximiuz Labs Premium, you can get 2-4x faster VMs, unlimited daily playtime, and full content access. Monthly, yearly, and lifetime plans are available, with proper invoices, so that you can include this expense in your employer's dev education budget 😎

Ivan Velichko

Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way 🚀

Read more from Ivan Velichko

Hello 👋 Ivan's here with a slightly delayed September roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month, I worked on an assorted set of topics. Skill Paths First off, the skill paths! I finally finished the underlying machinery, and now iximiuz Labs supports a new type of content - short roadmaps that you can use to develop or improve a specific skill: how to debug distroless containers, how to copy images from one repository to another,...

Hello friends! Ivan's here with another monthly roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 The issue's main topic is iximiuz Labs' largest-ever upgrade: Fresher and more streamlined look of the frontend UI 💙 A new 5.10 Linux kernel built with nftables support (finally, we can try out kube-proxy's nftables mode). New default playground user - laborant (yep, rootless containers learning for). New playgrounds: Ubuntu 24.04, Debian Trixie, Fedora, and Incus (yay! more...

Hello friends! Ivan's here with a slightly delayed July roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month, I got nerd-sniped by cgroups. It all started when I ran into a pretty significant difference in how Docker and Kubernetes handle the OOM events. When you limit the memory usage of a multi-process Docker container, the OOM killer often terminates only one of the processes if the container runs out of memory. If this process is not the...