These days I'm spending a lot of time dealing with containers. All kinds of containers. Bloated and slim, containing web-servers and CLIs, written in compiled and interpreted languages, running atop Docker and Kubernetes... And these containers often misbehave.
My typical container debugging routine involves:
I can solve most (if not all) of the above tasks by juggling standard tools and sub-commands (1, 2, 3, 4), but how nice it would be if there existed one ring tool to rule them all.
I started researching the available tools, and here is what I found.
debug-ctr - "a command-line tool for interactive troubleshooting when a container has crashed or a container image doesn't include debugging utilities, such as distroless images." A very nice CLI that either 1) mounts a volume with debugging tools into a running container (using some black magic of modern Linux kernels) or 2) "clones" the target container with a new mount containing the debugging tools (handy when the target has crashed). But it's Docker- and probably Linux-only, and no image- and network-related functionality (yet?).
docker-opener - "shell-in to any Docker container easily." This tool runs a shell in the target container. If there is no shell, it brings its own (busybox) to the target container and runs it. Despite the name, there is also a whole bunch of other helper commands, including (rather primitive) port forwarding. Almost meets my goals, but again seems to be limited to Docker (and Docker Compose).
cntr - "a container debugging tool based on FUSE." This is probably the most impressive one, at least judging by the amount of effort needed to implement something like that. It allows you to mount the host filesystem into a running container (or rather to mount the container's filesystem to the host and then launch a host's shell re-using the container's namespaces, IIUC). Since the implementation is based on FUSE and namespace manipulation, it supports an extremely wide range of runtimes (Docker, Podman, LXC/LXD, rkt, systemd-nspawn, containerd, etc). But it has only two (and very similar) commands - "exec" and "attach" and assumes access to the runtime's machine (i.e., no Kubernetes ephemeral containers "by design").
Two other honorable mentions are the docker-slim debug command and the docker-debug tool, that are essentially the subsets of the debug-ctr and docker-opener offerings.
As you can see, none of the above projects was even close to fully satisfying my needs. So, as usually happens with programmers, I decided to write my own tool 🙈
Behold github.com/iximiuz/cdebug!
The goals that I'm chasing with cdebug are:
You can think of "cdebug" as a marketplace for various container debugging commands. And the way it's designed should make it work for the majority of the most popular contemporary runtimes, including but not limited to Docker, containerd, and Kubernetes.
The tool is WiP, and the following commands are currently implemented.
The "cdebug exec" command is a crossbreeding of the "docker exec" and "kubectl debug" commands. You point the tool at a running container, say what toolkit image to use, and it starts a debugging "sidecar" container that feels like a "docker exec" session to the target container:
Here is how it works under the hood:
By default, the "busybox:latest" image is used for the debugger sidecar, but you can override it with the "--image" flag. Combining this with the superpower of Nix and Nixery, you can get all your favorite debugging tools by simply listing them in the image name:
cdebug exec -it --image nixery.dev/shell/ps/vim/tshark <target-container>
This command is another crossbreeding. This time it's "kubectl port-forward" and "ssh -L".
Currently, only local port forwarding is supported, but remote port forwarding ("ssh -R") is under active development.
Local port forwarding use cases:
Here is how the port forwarding command is implemented in the simple ("direct") case:
So, all the heavy lifting is done by socat (forwarding) and the container runtime (port publishing). Accessing the container's localhost is also possible, but with a trick:
The tool is very early, but I already rely on it in my daily work. Looking forward to the calm(er) Christmas season to add support for more commands and runtimes. And, of course, I'm really curious to hear back from you!
Have a productive week ahead!
Ivan
Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way 🚀
Hey, fellow server dweller 👋 Ivan here with an exciting iximiuz Labs update! The month isn't over yet, so it's not quite time for the traditional monthly roundup. However, there have been so many updates on the platform in the past couple of weeks that they couldn't possibly fit into a single email. So, let's dive in 🚀 Backend Revamp: Faster, Smarter, Stronger Over the past few weeks, I rolled out a significant backend rewrite at iximiuz Labs, and I couldn't be more excited to share the...
Hello 👋 Ivan's here with November's roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month was (extremely) development-heavy. Two-thirds of it went into the implementation of custom playground machinery and a new Kubernetes "Omni" playground, and in the last part, I was unexpectedly busy with expanding the platform's capacity and launching a new server in India 🎉 The latter became possible thanks to the support of all of you who got the premium...
Hello, fellow server dweller 👋 I've got two exciting announcements to make. Starting with the shorter one, this year, I decided to give Black Friday a try. This is an experiment - iximiuz Labs hasn't done sales before and won't have any in the foreseeable future, at least not until next November. So, if you wanted to get a premium membership but the price felt too high, this is your rare chance to get it with a 50% discount. The offer is limited to exactly one week. Now, to the second, much...