Ivan on Containers, Kubernetes, and Backend Development


Hi friends!

It's Ivan again, with my traditional roundup of all things around iximiuz.com.

This month was mostly about container images and container runtimes. Here is what I've got for you:

  • Some random container thoughts and explanatory drawings
  • A bunch of handy tricks I wish I had learned sooner on my Cloud Native journey
  • A couple of streams on containerization tech (DockerSlim and containerd)
  • A new GitHub repository with yet another awesome list of resources
  • One big and one small(er) blog posts based on my recent research
  • And, of course, a whole lot of great reads, podcasts, and videos πŸ”₯

Let's get started!

SPONSORED Packages and software supply chains get compromised more and more often. So, everyone these days should be a security expert. I already confined every my project to a separate VM, but how can I know if there is some malicious activity happening inside? Learn how to use the Linux audit system in this practical post by Teleport (no marketing fluff).


​

Container Runtime Visualised

While preparing for one of the workshops, I came up with the following visual explanation of the low-level container runtime:

If this topic sounds interesting, you can find a few useful links in this Twitter thread.


​

Random Evening Thoughts

Might be not fully accurate, but this explanation could help get through a popular misconception:

twitter profile avatar
Ivan Velichko
Twitter Logo
@iximiuz
September 24th 2022
34
Retweets
294
Likes

​

Docker Tips & Tricks

To improve your container game 😎

Dealing With Containers Faster

Since (a very recent) discovery, this trick has saved me hours! Turns out, you can exec, stop, kill, remove, etc docker containers using just a short prefix of their IDs! Instead of typing (or rather copy-pasting):

docker rm b5758e341c2a

...you can just go with:

docker rm b5

Credit goes to Michael Irwin!

​

Store Build Results In Local Folders

Another Docker trick that I learned on Twitter (thanks to Chris Guest). Turns out, the buildx plugin supports this for quite a while:

docker buildx build -o <some-dir> .

This is pretty powerful since it allows you to use Docker as a build system. You can find more details here (bake is the keyword to look for).

Non-Docker Tips & Tricks

Not so much about containers, but I must share them:


​

What I Was... Talking About

This is a new (and quite challenging) category for me, but I'm glad I gave it a try:

  • ​Hands-on Introduction to DockerSlim | Rawkode Live - Learn how to create extremely small container images using DockerSlim (and what magic is hidden behind the tool). As always on Rawkode Academy, it was a highly practical session.
  • ​containerd - a secret hero of the Cloud Native world - My first ever workshop where you can learn about the most widespread container runtime and how you can use it to dig deeper into containers, troubleshoot Kubernetes issues, or even as a building block for your own software.

​

What I Was Writing

On the blog, Twitter, ...and GitHub!

Awesome Container Tinkering

A yet another awesome list of resources? Nope, it's a list of awesome resources!

I started this GitHub project to collect links to (and thoughts about):

  • Lower-level building blocks (runtimes, libraries, CLIs)
  • Solutions for running containers locally and remotely
  • Helper tools and libraries to inspect, edit, or move images
  • In-depth write-ups and video resources on the above.

And I'm hoping for your contribution!

Distroless Container Images

Over the past few months, I spent quite some time researching the topic of how to create decent container images. Distroless (1.0) is one of the ways. You can learn more about the limitations of the scratch base images and the pros and cons of the distroless (aka "scratch for everyone else") base images in the below Twitter thread or in a long-form write-up on my blog - What's Inside Of a Distroless Container Image: Taking a Deeper Look:

twitter profile avatar
Ivan Velichko
Twitter Logo
@iximiuz
September 5th 2022
242
Retweets
1,006
Likes

​

How To Extract Image Filesystem Without Running Any Containers

Collected a bunch of receipts in this blog article.

Containers Are Just Linux Processes

...is another popular misconception. Intrigued? Then read on!

Docker History And Evolution

For the digital archeologists (like myself), I called out on Twitter asking folks if the Docker's history is documented somewhere. The thread received lots of great replies (with some of them coming from people who are/were directly involved), shedding light on Docker's original architecture (monolith written in Go, essentially just a UX layer on top of LXC), its evolution (LXC's functionality got re-implemented in Go giving birth to, first, libcontainer, and then runc) and reshaping (containerd being factored out). Go check it out πŸ˜‰


​

What I Was Reading

  • ​LXC and LXD: a different container story - when a superior UX beats the superior tech. This article will make you feel that LXC and LXD are a viable alternative to Docker (in reality, Docker-like containers and the corresponding UX & ecosystem became the de facto standard). As an engineer, I'm on the LXC side, but as a pragmatic developer, I'm siding with Docker.
  • ​Yet Another Brief History of container(d) - a related one. From the old-day VMs, through Linux namespaces & cgroups, to LXC, then Docker, libcontainer, runc, and containerd. Well worth a read if you feel like a history lesson.
  • ​Extending Docker’s Integration with containerd - containerd was factored out back in 2015-2016, but the work continues even in 2022. The article is not about history, though. It's about the practical implications of this move. Docker starts using even more features of containerd. And this is good (at least) for two reasons: 1) more code reuse means better compatibility; 2) Docker's image dealing capabilities will start being enriched faster.
  • ​Where are containerd’s graph drivers? - if you want to learn more about the above move - the story behind storage drivers.
  • ​Vulnerability Management for Go - this is just great! A much-needed addition to any modern programming language.
  • ​In Defense of Package Managers - (in the context of deplorable software security situation) Package Managers isn't the problem. The problem is how we write (with lots of bugs & vulnerabilities) and release (developers are quicker than distro maintainers) our software. LTS Linux distros can't keep up with the industry's pace. Is stream distros with frequent patches the future?
  • ​What your scanner doesn't know **can** hurt you - continuing the above theme...
  • ​HackTricks - a worthwhile resource on container security with some practical examples, including Docker containers breakout.
  • ​Securing Containers with Seccomp: Part 1 & Part 2 - a highly-practical mini-series on how to use seccomp profiles to fine-tune your containers.
  • ​Istio: Introducing Ambient Mesh - this does sound like a reasonable redesign to me! Migrating legacy workloads to Istio is not easy (I know it first-hand), but with the new basic L4 layer implemented on the ambient infra side, this initial step might become much easier. And since there is no need for sidecar injection anymore, the final steps will also likely be simplified. The single-tenant, scalable, and optional L7 proxy layer implemented as a bunch of Envoy Pods does sound like a good idea too. If, from the app's code standpoint, mesh remains fully transparent, these changes are a very strong reason to start liking Istio again!
    • 🎬 Istio Ambient Mesh Launch Demo - nice demo (only 10m), but you can skip the hands-on part and instead look at three diagrams that Christian Posta shows there. They might shed more light than the above Istio.io blog post.
    • ​[Kubernetes Podcast] Ambient Mesh, with Justin Pettit and Ethan Jackson - a lot of technical details, historical reasons for the redesign, insights on how meshes are actually used (mostly for mTLS), and how this new design was actually done by [virtual] networking people (vs. the sidecar-based design that was influenced by a background in application development).
  • ​Why fix Kubernetes and Systemd? - Kubernetes vs systemd. Wait, what? But on second thought, it makes total sense. Kubelet's and systemd's responsibilities do intersect substantially. Kris NΓ³va continues the "war on sidecars" started by service meshes some time ago.

And that's all the links for today!


​

Stay Tuned

Well, I thought the previous one was big... I should probably start doing this newsletter twice a month, what do you think? Hit reply and let me know :)

​

Cheers

Ivan Velichko

​

Ivan Velichko

Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way πŸš€

Read more from Ivan Velichko

Hello πŸ‘‹ Ivan's here with a slightly delayed September roundup of all things Linux, Containers, Kubernetes, and Server Side πŸ§™ What I was working on This month, I worked on an assorted set of topics. Skill Paths First off, the skill paths! I finally finished the underlying machinery, and now iximiuz Labs supports a new type of content - short roadmaps that you can use to develop or improve a specific skill: how to debug distroless containers, how to copy images from one repository to another,...

Hello friends! Ivan's here with another monthly roundup of all things Linux, Containers, Kubernetes, and Server Side πŸ§™ The issue's main topic is iximiuz Labs' largest-ever upgrade: Fresher and more streamlined look of the frontend UI πŸ’™ A new 5.10 Linux kernel built with nftables support (finally, we can try out kube-proxy's nftables mode). New default playground user - laborant (yep, rootless containers learning for). New playgrounds: Ubuntu 24.04, Debian Trixie, Fedora, and Incus (yay! more...

Hello friends! Ivan's here with a slightly delayed July roundup of all things Linux, Containers, Kubernetes, and Server Side πŸ§™ What I was working on This month, I got nerd-sniped by cgroups. It all started when I ran into a pretty significant difference in how Docker and Kubernetes handle the OOM events. When you limit the memory usage of a multi-process Docker container, the OOM killer often terminates only one of the processes if the container runs out of memory. If this process is not the...