Hello friends!

Ivan's here with a slightly delayed July roundup of all things Linux, Containers, Kubernetes, and Server Side 🧙

What I was working on

This month, I got nerd-sniped by cgroups. It all started when I ran into a pretty significant difference in how Docker and Kubernetes handle the OOM events.

When you limit the memory usage of a multi-process Docker container, the OOM killer often terminates only one of the processes if the container runs out of memory. If this process is not the topmost process of the container (PID 1), the container will keep running, which may or may not be desirable.

Pods consist of containers, and originally, this OOM killer behavior was present in Kubernetes, too. When a Pod's container with limited resources and multiple processes inside would run out of memory, the OOM killer would terminate only some of the processes, often leaving the container (hence, the Pod) running.

This made some of us start tracking down "invisible" OOM kills, hunting zombie Pods, and dodging silent Pod killers, so in Kubernetes 1.28, the OOM behavior was changed to terminate the entire container if any of its processes ran out of memory.

However, while improving things for some workloads, in perfect agreement with Hyrum's Law, this change broke some other workloads, and outages followed. So, naturally, a requirement arose to tweak the OOM killer behavior depending on the workload's needs.

The problem is that the change in Kubernetes 1.28 was unconditional, and neither Kubernetes API nor kubelet config can be used to restore the pre-1.28 OOM killer behavior. At least not yet.

So it all sounded like a perfect candidate for an iximiuz Labs Kubernetes challenge, and the one followed: Can you make a Pod Survive an OOM Event Without Restarting?

As it usually happens, while working on this rather hard problem, I accidentally created half a dozen beginner-friendly challenges and a step-by-step tutorial on cgroups 🙈

In the second half of the month, I got back to my work on the Byte-Sized Docker course. Not revealing the course itself yet, but here is another portion of Containers 101 problems that I've prepared for it:

• ​Build and Publish a Container Image With Docker​
• ​Copy a Container Image from One Repository to Another with Docker​
• ​Copy a Container Image from One Repository to Another - The Pro Way​

Last but not least, there were two improvements to the platform itself - the next challenge recommendation on completed (or failed) submissions and... wait for it... the About page! Right, twenty months into the project's existence. And I rather like how it came out - in particular, the page touches upon the meaning and pronunciation of the word iximiuz. Spoiler: it's [ eg-zim-ee-uhs ] and it's derived from a real (but obsolete) English word with a Latin origin.

Ivan Velichko @iximiuz Just shipped an iximiuz Labs UX upgrade 🚀 Now, after successfully completing or failing a challenge, you'll get an encouraging message and the next content recommendation. Nothing huge, but I wanted this feature for so long 🙈 https://labs.iximiuz.com/challenges 11:23 PM • Jul 28, 2024 8 Retweets 61 Likes Read 1 replies

​

What I was reading

​How I reduced the size of my very first published docker image by 40% - Back to the roots! Read this masterpiece to learn how to reduce the size of a container image the hard[est possible] way. While I admire the author’s dedication, it’s unlikely practical to follow their approach for en masse container image optimization. And that’s why we need tools like DockerSlim - identification of the app’s direct and transitive dependencies and adding only them (instead of the entirety of the base image files) to the final container image is exactly what DockerSlim does, but it does it in an automated (or semi-automated) fashion, so it’s much more practical to use. Ah, and by the way, the fact that Docker Scout reported 0 CVEs in the minified container is likely because it couldn’t scan it properly due to the lack of the package manager in the image and not because the added dependencies actually had no CVEs in them.

​Docker and BuildKit: how building images really works - a deep read on the architecture of the Docker “image building subsystem.” It touches briefly on the now-gone “old way” and then sheds a great deal of light on the new (by now default) BuildKit-based architecture.

​Making containers bootable for fun and profit - an LWN post about bootc, a tool to leverage “standard” container images to create bootable OS images. This idea has kept haunting me since I initially played with it back in 2019 (see my docker-to-linux project). It’s also how iximiuz Labs playgrounds are baked (touched upon it briefly in this post), although instead of bootc I wrote my own bake ~~tool~~ scripts (simple but quite powerful).

​Go 1.23: Interactive release notes - A masterpiece on new language features added in the Go 1.23 release. I truly admire the author’s dedication to producing such interactive learning experiences. At the same time, I’m concerned about the future of Go. Once simple language starts repeating Python’s history. Do we really need all these new features offering N different ways of solving the same problem? Now, I’m certain that adding generics to the language two years ago opened Pandora’s box.

A Reddit comment on what Wiz is and is not. IMO, this explanation is also applicable to the entire “cyber security” domain, which increasingly becomes much more about audit and compliance than actual protection. “Wiz agentless solution is easy to deploy, maintain and helps reduce the application attack surface. As an agentless solution, though, it ultimately isn’t protection, its just auditing. … Any agent based solution has enforcement capabilities is true security, they can stop and block attacks.” We need more Edera’s and Tetragon’s and fewer Wiz and Snyke’s. Thank you for coming to my TED talk.

​Using S3 as a container registry - A curious read on how to speed up 10x pushing of container images by using an S3 bucket instead of a “proper” container registry. In a nutshell, you can accelerate things a lot using concurrent chunk uploads while pushing a blob to S3 (an image layer, in this case). However, the OCI Distribution spec mandates sequential uploads even within a single layer, so S3 is expectedly faster, but such a registry is “noncompliant,” and the article has just about the right number of disclaimers.

​TIL: 8 versions of UUID and when to use them - A sweet short read on different kinds of UUIDs. As someone (often) using UUIDv4 for my primary keys, I’m truly looking forward to UUIDv7 becoming natively supported by major databases like Postgres. Time-sortable and, at the time, fully random IDs are awesome.

​zx - A neat scripting language that allows you to mix bash (or PowerShell) and JavaScript in one file with almost no extra syntactic noise. I’d use bash expressions for shelling out to whatever system commands I need, and JavaScript - for the conditional logic and data manipulation. Super handy. If only it was a statically linked binary with no dependency on Node.js.

Wrapping up

Usually, things slow down a little during the summer, but July was another great month for iximiuz Labs. For the past three months, it's been hitting $2,000 MRR, and I made a decision to officially reduce my working hours and start spending a full day a week working solely on the labs (that's in addition to my weekends and evenings, of course). Since the platform's release mid-last year, almost 12K people have launched a grand total of 55K microVMs and submitted 7,000 solutions. The numbers are absolutely mindblowing and also promising, so I'm doubling down on this project. Onward and upward!

If you're learning Linux, networking, containers, or Kubernetes and haven't tried the Premium tier yet, give it a try. Trust me, you'll notice the difference - with larger VMs and full internet access from the playgrounds, everything starting with solving challenges and down to the free-form experimentation becomes so much smoother and efficient.

Also, if you're a manager and want/need to help your team skill up in the server-side and Cloud Native topics that I teach, drop me a message - I'm always happy to discuss group subscriptions with a generous discount.

Happy learning!

Ivan

​