Hello 👋

It's this time of the month again! My traditional roundup of all things Linux, Containers, Kubernetes, and Server Side, delivered straight into your inbox 📬

What I was working on

October was very productive for me - I shipped no major iximiuz Labs features (it's always hard to resist the temptation!) and instead dedicated all my available time to content work.

The main focus was on Container Images. It's the subject of the first module of my "panoramic" Docker course, and it is almost done (the module, not the course 🙈). The content is pretty self-sufficient, so I'll likely release the first five-ish lessons (and a bunch of accompanying challenges) in November without waiting for the rest of the course materials.

While working on the first module, I wrote two complementary posts. One is about picking the right good-enough base image for your Node.js application. Turns out, it's not so straightforward, but understanding what's inside of different variants of the Node.js base images can help:

The second post (that I published just a couple of days ago) is on how to build small(er) and secure(r) container images using multi-stage builds. This is no new feature, and there are plenty of posts on Docker multi-stage builds on the Internet, yet I keep running into single-stage Dockerfiles, often started with a FROM <fat-sdk-image> instruction. So, I tried writing a post that covers not only the How's but also the Why's:

Another reason there aren't enough multi-stage builds may be that Dockerfiles fall into a no man's land. Application developers are usually too overloaded to prioritize proper image composition, while DevOps folks may be more attentive to it, but often lack detailed knowledge of specific language stacks. After all, there are countless ways to build a Node.js application or choose a runtime base image for a Go service. To address this, I’ve included a section with several practical examples on structuring multi-stage Dockerfiles for Node.js, Java, PHP, Rust, and Go applications (and hopefully, Python examples will be added eventually as well).

Last but not least, I had the chance to join a podcast! Huge thanks to the ​mkdev.me​ team for the invitation and for making it such a great experience 💙

Here’s the transcript, along with YouTube and Spotify links: ​The Art of Tech Blogging with Ivan Velichko from Iximiuz Labs​

What I was reading

​Advanced Dockerfiles: Faster Builds and Smaller Images Using BuildKit and Multistage Builds - Did you know that you can have conditional RUN instructions in your Dockerfile? Something like if MY_BUILD_ARG == foo RUN <cmd1> else RUN <cmd2>. Of course, there is no such syntax, but you can emulate this behavior with multi-stage builds 🤯 This is just one of the mind-blowing examples from the post.

A good question on r/node: Why node-gyp native Node.js addons like bcrypt or sqlite cause so many problems and even a better answer explaining the machinery behind it. How did it pop up on my radar? Native dependencies in your package.json are probably the only reason to use the node:lts or any other "fat" variant, including node:latest, of the Node.js container image. Compilation of native addons is the reason why the node:lts image has the entire GNU Complier Collection, Python 3, and ~800 other packages inside. Do you compile them in production (I truly hope not)? Then don’t use node:lts or node:latest for your production workloads.

​Do Docker containers share RAM for files memory mapped from the same layer but a different image? - Another great question! It’s actually slightly more specific than the title - assuming you have two images based on the same Linux base image, would the identical shared objects (.so files) from different images be "deduplicated" in memory (as it normally happens when multiple non-containerized processes load the same library)? And the right answer is, of course, it depends. But in most cases, containers will indeed share that piece of memory!

​AWS data center latencies - A fun visualization of DC to DC latencies right on a 3D globe. Don’t know how trustworthy the data is, but at first sight, the numbers look plausible.

​Understanding Round Robin DNS - An entertaining read about the author's experience with round-robin DNS. The original assumption was that round-robin DNS could be used as a free load-balancing solution. The author set up 3 servers (US, EU, Asia) and created 3 DNS A records with the same hostname but different IP addresses. Accessing this host from Chrome, Firefox, Safari, and curl - directly and via Cloudflare - yielded interesting results. It turned out that browsers don’t always pick the closest server (but curl does), and when they pick a server, they rarely change their mind. At the same time, Cloudflare picks a fixed server based on the client IP and wouldn’t change the choice even if this server goes down permanently (but the A record remains). In other words, you probably don’t want to use round-robin DNS for anything but redundancy… which was kinda a well-known fact since at least 2012 🙈

​Understanding DNS resolution on Linux and Kubernetes by Jérôme Petazzoni - The beginning and the end of the post describes the author’s wild goose chase attempt to get rid of the DNSConfigForming ... Nameserver limits were exceeded, some nameservers have been omitted warning in Kubernetes. However, the middle of the post is pure gold. It explains how DNS resolution works in Linux, how glibc differs from musl when it comes to DNS, and how systemd (as usual) has its own way of cooking DNS. Then, equipped with this knowledge, the author moves on to explaining how Kubernetes DNS works… and why the warning that triggered this deep dive is actually purely harmless.

​Service Discovery in Kubernetes: Combining the Best of Two Worlds by… yours truly - There was a phrase in Jérôme’s post that I don’t fully agree with - "Kubernetes provides DNS-based service discovery." In this (older) post, I argue that Kubernetes has rather a hybrid service discovery implementation that I call "Network-Side Service Discovery".

​Kubernetes networking: service, kube-proxy, load balancing - A massive read by Gulcan Topcu from Learnk8s! It’s packed with so many illustrations, probably more than most blog posts have words. Highly recommend it if you’re just starting out with Kubernetes networking.

​https://github.com/f1ko/demystifying-cni - finishing the topic of Kubernetes networking for today. The repository offers an explanation of what the Container Network Interface (CNI) actually is and how to write one from scratch. Because what could be a better way to learn?

​Good Retry, Bad Retry: An Incident Story - Another massive read. This time on the role of retries in a large-scale micro-service architecture. If you already tried exponential backoff, and it didn’t help, go read that post. Otherwise, just add the backoff - from my experience, in 90% of the cases, it’s everything you need.

​How do HTTP servers figure out Content-Length? - A short and sweet post that may make you start thinking about the hidden complexities of the simple on-the-surface HTTP protocol. If the headers are sent before the response body, and there is a Content-Length header, should the server buffer the whole response in memory before sending it to the client? What if it’s a few gigabytes of data? What about streaming responses like server-sent events?

​Deployment process with Hashicorp Nomad by Maksym Prokopov - Kamal keeps popping up on my radar as a simpler (compared to Kubernetes) deployment option. But we should not forget about Nomad’s existence - it has a bunch of nice qualities, including “native” integration with Consul and Vault.

​Thinking Like an AI - An enjoyable read on how LLMs actually generate their output. It's tremendously oversimplified, but nevertheless, I find it helpful for people like me who don’t have time to read a proper explanation - understanding this kind of basics makes my daily interactions with GenAI more efficient.

​Writes and Write-Nots by Paul Graham - What an irony! While one of the main AI promises today is to make us all smarter, PG predicts that GenAI will actually make humanity dumber by “freeing” us from the need to write [for work or pleasure]. Writing is thinking.

Wrapping up

How is iximiuz Labs doing? I'm glad you asked!

While it didn't break the September record, the combined revenue from Patreon contributions and Premium membership sales in October again crossed the $3,000 mark. And the platform's usage also kept steadily growing.

Here are some interesting facts:

• ~60% of sandbox environments launched last month were free-play (i.e., not part of any tutorial, challenge, or course).
• 60% of the platform usage came from paid accounts (while > 90% of registered accounts are free).
• The most popular playgrounds (traditionally) were: Ubuntu, Docker, and K3s.

It looks like more and more people return to iximiuz Labs for its playgrounds - not a particular piece of content I may have published. Cases I know of:

• DevOps/Platform/SRE engineers use iximiuz Labs to prepare for CKA/CKAD certification.
• A few companies conducted internal workshops on containers and Kubernetes using iximiuz Labs playgrounds (and a group discount).
• University professors keep reaching out to me, asking if they can use the platform in their lectures (but I've yet to hear back about an actual attempt).
• I use playgrounds for all sorts of quick hacking and experiments, so I assume other folks also do that, but it'd be nice to hear about some real-world examples.

I also keep getting requests for purchasing power parity discounts (PPP), and I'm gladly offering them to people outside of Western Europe and the US. However, too few discount codes I offered converted into actual sales, and it made me reconsider my discount policy. Now, I'll be trying to match Netflix's pricing model - thanks to a few people who suggested it on Twitter. This basically means that the discounts I offer will be more generous.

If you wanted to buy the premium membership but the price was too high, or the discount I gave you was too small, please reach out - we'll find a solution! And, of course, always double-check - chances are, you can expense iximiuz Labs Premium membership using your learning and development budget.

Have a productive November!

Cheers

Ivan