Ivan Velichko

DevOps Challenge of the Week - Issue #6

Published 4 months ago • 1 min read

Hello there! 👋

Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible.

Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers.

If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is probably the most obvious choice when things go sideways:

However, good security practices demand that our containers don't include debugging tools (or even shells) by default. Here is what you can do if the misbehaving container is based on a slim or a distroless image:

  • Install debugging tools on the fly (slim).
  • Temporarily switch to a "fat" base image (slim & distroless).
  • Run an improvised debugger "sidecar" container using docker run and sharing target's namespaces.
  • Run a proper debugger "sidecar" container with kubectl debug.

Quite a few options, huh? Well, from my experience, none of them is super user-friendly. Installing debugging tools on demand is tedious, getting the docker run flags by heart from the first attempt is hard, and kubectl debug is not as flexible as I'd like it to be (you cannot run the debugger as the given user, make the sidecar privileged, etc.).

Third-party tools to the rescue!

cdebug is a my favorite tool that I wrote specifically to streamline the container debugging UX. It allows executing commands in scratch, distroless, and slim containers, using the debugger image of choice and making the debugger sidecar see the target's filesystem as-is. And it works the same for Docker, container, and, since recently, Kubernetes 🚀

Do recommend giving cdebug a try while solving today's challenges:

Good luck!

P.S. Traditional reminder - with iximiuz Labs Premium, you can get 2-4x faster VMs, unlimited daily playtime, and full content access. Monthly, yearly, and lifetime plans are available, with proper invoices, so that you can include this expense in your employer's dev education budget 😎

Ivan Velichko

Software Engineer at day. Tech Storyteller at night.

Building - a place to help you learn Containers and Kubernetes the fun way 🚀

Read more from Ivan Velichko

Hello friends! It's time for my traditional monthly roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙 Before we get started, I want you to know that this newsletter's previous issue (dispatched mid-May) was delivered to only about 1/5th of my usual email audience due to an unfortunate DNS misconfiguration. The good news is that you can still find it and all previous issues on Also, if you reply to this email, it'd help to restore the domain's...

18 days ago • 4 min read

Hello friends! Ivan's here - with another well overdue roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙 This time, I have a great excuse - in April, we became parents of a lovely little boy. Expectedly, he immediately made us overly busy for a few weeks, but no complaints! I wouldn't trade this business for anything else in my life ❤️ Now, to more technical news. What I was working on My main focus remains on iximiuz Labs 🚀 Content authoring redesign In March &...

about 1 month ago • 10 min read

Hello friends! Ivan's here - with a well overdue February roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙 What I was working on A lot of stuff on the dev side - not so much on the content side. But things are soon to reverse 🤞 Announcing labCTL - the long-awaited iximiuz Labs CLI A dozen people have asked me over the past year-ish if there'll be access to the playgrounds from the local terminal and not only from the browser. And while I myself wanted this feature...

3 months ago • 7 min read
Share this post