DevOps Challenge of the Week - Issue #6


Hello there! 👋

Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible.

Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers.

If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is probably the most obvious choice when things go sideways:

However, good security practices demand that our containers don't include debugging tools (or even shells) by default. Here is what you can do if the misbehaving container is based on a slim or a distroless image:

  • Install debugging tools on the fly (slim).
  • Temporarily switch to a "fat" base image (slim & distroless).
  • Run an improvised debugger "sidecar" container using docker run and sharing target's namespaces.
  • Run a proper debugger "sidecar" container with kubectl debug.

Quite a few options, huh? Well, from my experience, none of them is super user-friendly. Installing debugging tools on demand is tedious, getting the docker run flags by heart from the first attempt is hard, and kubectl debug is not as flexible as I'd like it to be (you cannot run the debugger as the given user, make the sidecar privileged, etc.).

Third-party tools to the rescue!

​cdebug is a my favorite tool that I wrote specifically to streamline the container debugging UX. It allows executing commands in scratch, distroless, and slim containers, using the debugger image of choice and making the debugger sidecar see the target's filesystem as-is. And it works the same for Docker, container, and, since recently, Kubernetes 🚀

Do recommend giving cdebug a try while solving today's challenges:

Good luck!

P.S. Traditional reminder - with iximiuz Labs Premium, you can get 2-4x faster VMs, unlimited daily playtime, and full content access. Monthly, yearly, and lifetime plans are available, with proper invoices, so that you can include this expense in your employer's dev education budget 😎

Ivan Velichko

Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way 🚀

Read more from Ivan Velichko
Diagram showing desired network policy configuration between frontend and backend pods

Hey, fellow server dweller 👋 Ivan here with an exciting iximiuz Labs update! The month isn't over yet, so it's not quite time for the traditional monthly roundup. However, there have been so many updates on the platform in the past couple of weeks that they couldn't possibly fit into a single email. So, let's dive in 🚀 Backend Revamp: Faster, Smarter, Stronger Over the past few weeks, I rolled out a significant backend rewrite at iximiuz Labs, and I couldn't be more excited to share the...

Hello 👋 Ivan's here with November's roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month was (extremely) development-heavy. Two-thirds of it went into the implementation of custom playground machinery and a new Kubernetes "Omni" playground, and in the last part, I was unexpectedly busy with expanding the platform's capacity and launching a new server in India 🎉 The latter became possible thanks to the support of all of you who got the premium...

Hello, fellow server dweller 👋 I've got two exciting announcements to make. Starting with the shorter one, this year, I decided to give Black Friday a try. This is an experiment - iximiuz Labs hasn't done sales before and won't have any in the foreseeable future, at least not until next November. So, if you wanted to get a premium membership but the price felt too high, this is your rare chance to get it with a 50% discount. The offer is limited to exactly one week. Now, to the second, much...