These days I'm spending a lot of time dealing with containers. All kinds of containers. Bloated and slim, containing web-servers and CLIs, written in compiled and interpreted languages, running atop Docker and Kubernetes... And these containers often misbehave.
My typical container debugging routine involves:
I can solve most (if not all) of the above tasks by juggling standard tools and sub-commands (1, 2, 3, 4), but how nice it would be if there existed one ring tool to rule them all.
I started researching the available tools, and here is what I found.
debug-ctr - "a command-line tool for interactive troubleshooting when a container has crashed or a container image doesn't include debugging utilities, such as distroless images." A very nice CLI that either 1) mounts a volume with debugging tools into a running container (using some black magic of modern Linux kernels) or 2) "clones" the target container with a new mount containing the debugging tools (handy when the target has crashed). But it's Docker- and probably Linux-only, and no image- and network-related functionality (yet?).
docker-opener - "shell-in to any Docker container easily." This tool runs a shell in the target container. If there is no shell, it brings its own (busybox) to the target container and runs it. Despite the name, there is also a whole bunch of other helper commands, including (rather primitive) port forwarding. Almost meets my goals, but again seems to be limited to Docker (and Docker Compose).
cntr - "a container debugging tool based on FUSE." This is probably the most impressive one, at least judging by the amount of effort needed to implement something like that. It allows you to mount the host filesystem into a running container (or rather to mount the container's filesystem to the host and then launch a host's shell re-using the container's namespaces, IIUC). Since the implementation is based on FUSE and namespace manipulation, it supports an extremely wide range of runtimes (Docker, Podman, LXC/LXD, rkt, systemd-nspawn, containerd, etc). But it has only two (and very similar) commands - "exec" and "attach" and assumes access to the runtime's machine (i.e., no Kubernetes ephemeral containers "by design").
Two other honorable mentions are the docker-slim debug command and the docker-debug tool, that are essentially the subsets of the debug-ctr and docker-opener offerings.
As you can see, none of the above projects was even close to fully satisfying my needs. So, as usually happens with programmers, I decided to write my own tool 🙈
Behold github.com/iximiuz/cdebug!
The goals that I'm chasing with cdebug are:
You can think of "cdebug" as a marketplace for various container debugging commands. And the way it's designed should make it work for the majority of the most popular contemporary runtimes, including but not limited to Docker, containerd, and Kubernetes.
The tool is WiP, and the following commands are currently implemented.
The "cdebug exec" command is a crossbreeding of the "docker exec" and "kubectl debug" commands. You point the tool at a running container, say what toolkit image to use, and it starts a debugging "sidecar" container that feels like a "docker exec" session to the target container:
Here is how it works under the hood:
By default, the "busybox:latest" image is used for the debugger sidecar, but you can override it with the "--image" flag. Combining this with the superpower of Nix and Nixery, you can get all your favorite debugging tools by simply listing them in the image name:
cdebug exec -it --image nixery.dev/shell/ps/vim/tshark <target-container>
This command is another crossbreeding. This time it's "kubectl port-forward" and "ssh -L".
Currently, only local port forwarding is supported, but remote port forwarding ("ssh -R") is under active development.
Local port forwarding use cases:
Here is how the port forwarding command is implemented in the simple ("direct") case:
So, all the heavy lifting is done by socat (forwarding) and the container runtime (port publishing). Accessing the container's localhost is also possible, but with a trick:
The tool is very early, but I already rely on it in my daily work. Looking forward to the calm(er) Christmas season to add support for more commands and runtimes. And, of course, I'm really curious to hear back from you!
Have a productive week ahead!
Ivan
Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way 🚀
Hello 👋 Ivan's here with a slightly delayed September roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month, I worked on an assorted set of topics. Skill Paths First off, the skill paths! I finally finished the underlying machinery, and now iximiuz Labs supports a new type of content - short roadmaps that you can use to develop or improve a specific skill: how to debug distroless containers, how to copy images from one repository to another,...
Hello friends! Ivan's here with another monthly roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 The issue's main topic is iximiuz Labs' largest-ever upgrade: Fresher and more streamlined look of the frontend UI 💙 A new 5.10 Linux kernel built with nftables support (finally, we can try out kube-proxy's nftables mode). New default playground user - laborant (yep, rootless containers learning for). New playgrounds: Ubuntu 24.04, Debian Trixie, Fedora, and Incus (yay! more...
Hello friends! Ivan's here with a slightly delayed July roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙 What I was working on This month, I got nerd-sniped by cgroups. It all started when I ran into a pretty significant difference in how Docker and Kubernetes handle the OOM events. When you limit the memory usage of a multi-process Docker container, the OOM killer often terminates only one of the processes if the container runs out of memory. If this process is not the...