Hello 👋

Ivan's here with a slightly delayed September roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙

What I was working on

This month, I worked on an assorted set of topics.

Skill Paths

First off, the skill paths! I finally finished the underlying machinery, and now iximiuz Labs supports a new type of content - short roadmaps that you can use to develop or improve a specific skill: how to debug distroless containers, how to copy images from one repository to another, how to forward TCP ports, how to control system resource consumption, etc.

The cumulative number of tutorials, course lessons, and challenges on iximiuz Labs has crossed 70, and I'm getting more and more questions about the "right" order to consume the available materials. While it's rather unrealistic to develop a comprehensive "DevOps learning path" that would work for everyone, it's definitely possible to prepare curated lists of practical challenges (sprinkled with a little bit of theory and colorful diagrams) to master more scoped topics, one skill at a time.

Here are the first two examples that I prepared in September:

🛠️ Get Started with Linux Control Groups (cgroup v2)​

🛠️ Copy Container Images Like a Pro​

...and many more to come!

Kamal 2 Playground

​Kamal by DHH and 37signals folks (rather unexpectedly for me) got a major version upgrade, and it's a good one!

Kamal is an imperative alternative to declarative orchestrators like Docker Swarm or Kubernetes. The idea is that you have a single "deploy" machine with the kamal CLI installed. This machine "broadcasts" (over SSH) docker run, docker logs, and the like commands to a set of identical "worker" servers to start and operate N replicas of your app container.

The "deploy" machine can be your laptop, and the "worker" hosts are just vanilla Linux VMs. Workers don't run any kubelet-like agents - only the Docker daemon and a node-local HTTP reverse proxy are required on a worker server. Pretty much how it was done in the good old days but handier, thanks to Kamal’s opinionated way of cooking Docker.

When Kamal showed up on my radar ~a year ago, it immediately caught my attention because I helped to implement something like Kamal twice in the (relatively) recent past for some of my clients. From my experience, many businesses running on Kubernetes would be better off using a simpler infra - if you have just a few servers to manage, there is nothing wrong with an Ansible playbook executing Docker commands on them to roll out your app. But Kamal made it even easier.

I liked Kamal so much that I even created a Kamal playground on iximiuz Labs. Kamal 2 was released this month (replacing declarative Traefik with its own imperative kamal-proxy, adding Let’s Encrypt support and streamlining a few other rough edges), and I rushed to update iximiuz Labs Kamal playground so that anyone can go and explore Kamal 2 in a multi-node environment.

The playground consists of three hosts:

• A deploy machine
• Two worker servers

The deploy machine has a simple Kamal-ized Python app at ~/svc-a. Check out how simple it is to roll this app out to two worker servers with a single kamal setup command:

Better labctl experience

I'm a frequent user of labctl, but it's much more efficient to rely on the help of the community to catch bugs and spot missing features. Thanks to Jędrzej Domański, Alex Ellis, and Richard Gee, the iximiuz Labs CLI got a few QoL improvements this month:

• SSH sessions aren't terminated anymore after 5 minutes of inactivity (see this fix in wsmux - it was a tricky one).
• A new labctl cp command was added to copy files to and from remote playgrounds (using scp under the hood).
• For arkade users, there is now a handier way to install labctl: arkade get labctl.

Just a reminder, here is how easy it is to start a remote playground, copy a file to it, and then SSH into it right from your local terminal:

Some work-in-progress Docker content

When I'm not adding features or rebuilding playgrounds, I'm working on my "panoramic" Docker course. Here is a by-product that you might find useful on its own:

iximiuz Labs update

One of the traditionally less appreciated parts of my work finally starts getting more attention. I personally use iximiuz Labs playgrounds a lot - in my daily work, research, and learning. It's super handy when you can get a fresh Docker host or a Kubernetes cluster in just one click and ~10 seconds of waiting, launch some less trusted stuff on it, experiment, and then tier it all down without leaving the slightest trace on your precious host system.

iximiuz Labs playgrounds are also always up-to-date - I keep a close eye on a bunch of Linux distros, Kubernetes, Docker, Podman, Dagger, and a number of other tools and upgrade playgrounds within a day or two after a new version is released.

However, my understanding was that the majority of the iximiuz Labs users valued the platform primarily for its content part...

However, in September, I got a fresh influx of feedback, and it seems that more and more people are buying premium memberships to get faster playgrounds with unlimited Internet access, which they use for their "free-form" study and even work!

This is heartwarming and reassuring 🙏

September was also the best ever month financially - the cumulative "revenue" from Gumroad and Patreon crossed $3,000. I am not claiming a $3k MRR yet because the recurrence is not guaranteed, but fingers crossed for October 🤞

Last but not least, thanks to Gumroad making it really easy to do, I launched an affiliate program. If you're a frequent iximiuz Lab user and want to help me spread the word about the platform and make some money on referred sales, go click that link and register.

What I was reading

​alpine, distroless or scratch? - While the article tries to promote the use of scratch images, I recommend everyone to think twice before migrating to them. FROM scratch == FROM <empty-folder> == missing stuff. For instance, scratch container images lack important system folders such as /tmp or /home, proper user management via /etc/passwd file, CA certificates, the time zone database, and probably some other things. A statically linked Go demo app working fine in a scratch image doesn’t guarantee your production app's system dependencies will be fully satisfied. In many cases, distroless or just slim variants of mainstream Linux distros (Debian, Ubuntu) are a much safer bet for your production containers. Alpine may or may not be a good option - it depends on how well you know your app's dependency on C stdlib implementation details.

​Choosing the best Node.js Docker image by Snyk (continuing the above topic) - Articles like that tend to age quickly, but this particular example seems to have been well maintained for two years already. Picking the right base image for your app often feels like rocket science (or black magic or art, depending on how you see it). The whole post is definitely worth reading, but the conclusion (which I fully agree with) is - unless you know what you’re doing, go with node:lts-bookworm-slim, and you'll be good. Funnily, it contradicts the Docker Hub Node.js page, which recommends using the "fat" variants like node:lts-bookworm. IMO, it’s a very dated piece of advice from the time when people were more concerned with disk usage than the number of vulnerabilities in their images. Think twice before writing FROM node or FROM node:bookworm in your Dockerfile. Slim (or distroless if you know how to cook them) is the way!

​Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode - Progress is great, so I’m all for the wider adoption of cgroup v2. Just beware that this move might not be as harmless as it tries to sound. Funnily, one of the links in this blog post is a merged GitHub PR that enabled cgroup-aware OOM killer in all Kubernetes clusters that use cgroup v2. And apparently, it broke even Kubernetes’ own CI jobs (running atop Kubernetes, I presume) 🙈

​Kubernetes 1.31: Custom Profiling in Kubectl Debug Graduates to Beta - A very weak shot at fixing the kubectl debug user experience. With the new --custom flag, you can partially override the target Pod spec, but what about the spec of the ephemeral container? Does the new flag make solving this and this debugging challenges easier? Seems like cdebug will remain my debugger choice for the time being. Overall, it starts to feel that the original idea to use a single kubectl debug command for executing commands in running Pods, copying Pods for further debugging, and troubleshooting Node issues wasn’t such a good design after all. But, of course, I'm biased.

​What every SRE should know about GNU/Linux resolvers and Dual-Stack applications - Yet another masterpiece by Viacheslav Biriukov, this time on the Linux networking subsystem. Make sure to check out two other deep dives - on file descriptors, pipes & terminals and on Linux page cache.

​I Like Makefiles - This post mixes together the concepts of a build tool and a task runner, but apart from that, I do share the author's sentiment. Every project I started in the past ~7 years has a top-level Makefile, even if it invokes some trivial commands like npm run dev and docker build -t <img> .. My more advanced projects may even have Makefiles in subfolders. So, it's Makefiles all the way down because make is ridiculously ubiquitous. Ranted about it in this lesson of my (still unfinished 🙈) Dagger course.

​Go structs are copied on assignment (and other things about Go I’d missed) - I admire Julia's courage in admitting that she didn't know these (rather basic) parts of Go. Every senior engineer should foster this honest quality in themselves and set the right example to other, potentially more junior team members. It’s absolutely fine not to know some things about your language or immediate stack. We all learn something new at the job daily, regardless of our level of maturity.

​Good programmers worry about data structures and their relationships - Oh, yes! It’s not the first time I have run into this quote by Linus Torvalds, and every time, I have the urge to share it with as many people as I can. Linus goes even further and, in his traditional tough manner, states that "…the difference between a bad programmer and a good one is whether he considers his code or his data structures more important". I'm not sure if it’s a real differentiator for good and bad programmers, but it definitely is for readable & maintainable vs. "flow of mind that only the author can understand" kinds of code.

​Monolith First by Martin Fowler. You actually wouldn't believe how far a company can get with a single stateless HTTP backend, a Postgres database, and a bunch of async workers crunching non-OLTP workloads. I personally "scaled" such architecture to millions of users, and the truth is that not much of "scaling" was needed. Just add a couple more backend servers and maybe shard your DB a little. No Kubernetes, no even cloud. And, of course, no microservices - "even experienced architects working in familiar domains have great difficulty getting boundaries right at the beginning."

Wrapping up

That is it for September. Hope there was something in this email that you personally found useful.

Have a productive month! ☕

Ivan

P.S. My traditional reminder - if you want to learn the Server Side craft faster and support my work, consider getting the premium membership. There is a very good chance you will be able to expense it using your learning and development budget.