Hello friends!

It's time for the traditional round-up of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙

I've got a lot of stuff to share with you this month, so without any further ado, let's get started!

What I was writing (and talking about)

I started October with preparations for my first Kubernetes 101 workshop. It was supposed to be about Pods, Services, EndpointSlices, and the like, but, as often happens, I found myself pulled in a slightly different direction...

How Kubernetes Automated Age-Old Infra Patterns

What if you don't start from Pods, Deployments, and Service and instead take a step back and look at how things were done in the past with good old VMs? Can it help understand Kubernetes faster and deeper? In the end, I got so excited about this idea that the workshop video got a supporting long-form blog post:

​[not a] Kubernetes 101 - Pods, Deployments, and Services As an Attempt To Automate Age-Old Infra Patterns 📚

In the blog post, I start by showing how to run a single instance of the (containerized) app on just one server, then show how to replicate and scale the app instances, organize the load balancing, and even touch upon zero-downtime deployments, failure recovery, and service discovery. The main idea is to show how the level of automation complexity increases at every stage of the service architecture evolution. After the problem set and the "traditional" solutions become clear, I show how Kubernetes takes infrastructure management to the next level by packaging all of the above (and then some more) infra patterns into single-line commands.

Kinda cool, especially when you already know how to use Kubernetes, don't have to manage the clusters, and the infrastructure bills are paid by your employer.

How Kamal Uses the 80/20 Principle To Achieve The Same Goal

What about those of us who truly love servers and/or care about the infra bills? Did you know that auctioned servers on Hetzner are x10 cheaper than metal EC2 instances, and the maintenance costs of the low-scale fleet of servers are usually exaggerated? Speaking from experience 😉

This LB+Docker+Traefik+Ansible (or Capistrano) setup that I was trying to demo in the first part of the above blog post is actually pretty typical. I've set up variations of it several times in my career, so I was pleasantly surprised to discover that DHH and 37signals folks created a tool called Kamal that does pretty much the same thing.

Kamal, formerly known as MRSK, is a relatively new piece of infra software with a simple but powerful idea - if you know Linux (a bit) and Docker (slightly more than a bit), then running a bunch of containers by broadcasting vanilla SSH commands to a handful of VMs might be everything you need.

I spent some time playing with Kamal, and here is what I found:

• You can launch one (easy) or more (requires you to be a bit of a Traefik expert) services with simple `kamal setup` and `kamal deploy`.
• Kamal is fully imperative, and you can see all SSH commands that it executes on the remote servers.
• Kamal is agentless - the tool doesn't require installation on the remote servers, so only Docker and sshd should be there.
• Replication and zero-downtime deployments are out of the box.
• But because Kamal is imperative and agentless, it can only run N replicas of your app when you ask it to - ensuring there is always N replicas (even if some of the servers crashed) is out of scope.
• No load balancing out of the box - you'll have to either use a managed balancer by your infra provider or set up an (HA) load balancer yourself.
• No service discovery out of the box - if you have more than one service, you'll either need to route traffic through that shared LB or configure client-side service discovery (probably overkill).
• No hardening of the servers - Kamal does only basic service provisioning of the remote servers (installs Docker and curl), so you'll need to set up a firewall and the like yourself.

Overall, I liked the tool, but I'd like the load balancing and service discovery automation to be eventually covered, too. Here is a new multi-node iximiuz Labs playground where you can experiment with Kamal 🧑‍🔬

What I was building

The Kubernetes workshop and my Kamal experiments made me finally find time (and excuse) to finish the multi-node playground support. The labs were designed to be multi-node from day one, but a few things were still missing.

Since October, the following multi-VM playgrounds have become available:

• ​A multi-node K3s cluster​
• ​Docker Swarm cluster​
• ​Kamal playground​
• ​Mini-LAN playground​

Another (significant) quality of life improvement that I'm particularly proud of is that every playground now gets an accompanying short-lived container registry that is accessible on the "fake" registry.iximiuz.com address (which resolves to 192.168.1.2). This makes sharing images between playground VMs so much easier ❤️‍🔥

One more playground that made it to iximiuz Labs in October is a preconfigured Zig (programming language) dev environment - it's a VM with a loaded VS Code, Docker, and other batteries included. This language keeps popping up on my radar, and the code snippets that I've seen so far look pretty sleek.

Last but not least, kexp - the visual Kubernetes Explorer is closing in on 500 stars on GitHub (in just one month of its existence 🚀), and it also got a few features shipped this month:

• A visual explainer for the graph objects color coding.
• Pod and Node status visualization.

​A kind reminder that you can support my OSS work and the development of iximiuz Labs by becoming a patron. In addition to my gratitude, you'll also get:

• Unlimited daily playground minutes.
• More CPU, RAM, and faster disk and network.
• ​Additional practice sections (more coming soon).
• Insights into my creative process (in the form of Patreon posts).

What I was reading

​A Comprehensive Guide to API Gateways, Kubernetes Gateways, and Service Meshes - This is a very good read on the mentioned tech. If you’ve been confusing API Gateways, Ingress Gateways, Kubernetes Gateway API, and Kubernetes Ingress - it’ll definitely help sort things out. I particularly liked the narrative - start from the basic concepts (what is an API Gateway) and then connect the dots with the corresponding Kubernetes abstractions (Gateway API and Ingress). Also, it’s heavily illustrated. Do recommend!

​Sticky sessions and canary releases in Kubernetes - A brief, colorful post by Daniele Polencic on how Nginx Ingress Controller implements sticky sessions. There is also an interesting plot twist - turns out you can leverage sticky sessions for doing canary releases. The only bit that’s missing in this wonderful blog post is the practical part. Luckily, you can compensate for it with my K3s playground 😉

​User Namespaces: Now Supports Running Stateful Pods in Alpha! - Kubernetes’ support of User Namespaces keeps improving. The article also has a handy ~6m video demoing how simple it is to break out of a container. One more time - containers are security means.

​Why do Kubernetes Control Planes have an odd number of members? - A fun fact you may learn from the article: a cluster with two control plane nodes is actually less reliable than with one. A good first principle thinking explanation.

​What happens when … Kubernetes edition! - A solid read but could definitely use some illustrations 🎨

​Structured Logging with slog - The biggest Go addition since generics, probably. Finally, a standard structured logger implementation. I took a brief look, and the design seems to satisfy all my needs from a logger. Glad to see it shipped!

​Redis Explained - One piece of high-quality content with a ton of visuals.

​Stacked Diffs (and why you should know about them) - I’m surprised by how low awareness of this technique is among developers outside of FAANG. The idea is that you do smaller PRs and don’t wait until the previous PR is merged, even if the next code change depends on the change in the previous PR - you just branch out of the previous branch and, when ready - publish another PR, diffing it with its parent branch, and not the trunk. Stacked MRs/PRs/Diffs unblock you and also simplify lives for reviewers. The downside is that a stacked sequence of PRs might be a pain to maintain (rebases have a ripple effect), but the idea is to simplify the reviewer’s life to get the PRs merged faster. Plus, the tooling matters.

​The Inner-Platform Effect by Matt Rickard - If you’re building an internal platform, this short blog post combined with this sentiment by Justin Garrison will luckily make you think. But don't think too much - there are always folks on the other end of the spectrum, like DHH shipping Kamal in 2023 🙈

​Anticipate the Cheap - A short, enjoyable story and a business lesson. Extreme achievements require an extreme amount of confidence and boldness.

The space calms down a bit, and some good LLM resources start emerging in my feed - Simon Willison’s Weblog and One Useful Thing substack newsletter.

Wrapping up

Is it a new record? Probably yes :) Hope you enjoy this issue, and have a creative week ahead!

Cheers

Ivan