Hello friends!

Ivan's here - with another well overdue roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙

This time, I have a great excuse - in April, we became parents of a lovely little boy. Expectedly, he immediately made us overly busy for a few weeks, but no complaints! I wouldn't trade this business for anything else in my life ❤️

Now, to more technical news.

What I was working on

My main focus remains on iximiuz Labs 🚀

Content authoring redesign

In March & April, I had to put a lot of effort into separating the content from the code. Historically, I used the Nuxt Content plugin and kept my content markdown files and the actual platform's code in the same repository. This allowed me to cut quite a few corners and start shipping the content faster. In particular, I could rely on the dev version of the platform for content editing and benefit from Nuxt's default hot-reloading mechanism.

Over time, though, the initial quick win turned into an annoying limitation. Spinning up the dev version of the platform (a non-trivial process, by the way) just to work on a piece of content, re-deploying the whole app just to fix a typo in a blog post - things like that started negatively impacting my motivation to work on content. As we all know, DevEx matters - the longer your inner dev loop, the more flaky your CI, the less your willingness to write good, well-tested code. And apparently, the same is true for the AuthorEx!

The last straw was a few really bright individuals offering me a hand with content production. Obviously, I couldn't expect them to spin up their own dev versions of iximiuz Labs to work on content. So, refactoring it was.

Long story short, I had to find a way to host the raw content materials on some remote storage and allow people to use the production version of labs.iximiuz.com to create and edit content, hot-reloading included. As a result:

• The Nuxt Content plugin was replaced with a much lower-level Nuxt MDC component. Apparently, the former doesn't support dynamic content editing (it caches all markdown during the build step) while the latter is a much more flexible library to parse (rich) markdown blobs on-demand.
• The markdown and static assets were moved from the main repository (and Nuxt Content's build-time cache) to an S3-compatible file storage (I use Tigris at the moment, mainly because they have direct integration with Fly.io).
• The serverless MongoDB Atlas instance got upgraded to a dedicated cluster (increasing my infra bills by another $60/mo 🤦‍♂️). Turns out, the serverless variant doesn't support watching collections for changes, and that's what I wanted to use for the proper hot reloading implementation.
• Some cool server-sent events magic was added to the frontend app (foreman).
• ​The CLI got a new family of commands: labctl content create|list|push.

Did it change the "Author Experience"? Hell, yes! The inner loop shortened drastically, and the requirement to run a dev version of the platform is gone.

But an interesting by-product of the above rework is that every premium tier user of the platform can now create tutorials, challenges, and even courses!

It's not a well-documented feature (yet), but technically, it's already available. Beware, though, that by default, this content is for personal use and won't be accessible to other platform users. I still have to figure out the copy-right concerns, content sharing, and review workflows, but I'm already happy to work with individuals if they see fit for this functionality in their work (trainers, course instructors, dev rel folks working on product demos?)

​

Other iximiuz Labs improvements

• 🎉 8h-long playgrounds come to the premium tier. A bunch of people asked for longer playgrounds, and we delivered!
• 🚀 Free tier egress became x2 faster (~30MBit/s -> 60MBit/s).
• 🔥 Playgrounds access from your local VS Code with labctl playground start --ide or labctl ssh-proxy --ide. In particular, it means you can use all your fav extensions, including Copilot (and the like), while messing in ephemeral playgrounds!

Ivan Velichko @iximiuz ❤️‍🔥 A single CLI command to provision DevOps playgrounds in under 10 seconds - Multi-node Kubernetes clusters - Docker, Podman, containerd - Linux, networking, and more! ...with direct access from your IDE (including Copilot) and full-fledged local and remote port forwarding👇 10:19 PM • Mar 27, 2024 46 Retweets 234 Likes Read 7 replies

​

What I was writing (and not only)

​Building a Tool to Debug Minimal Container Images is a recent KubeCon talk by Kyle Quest and Saiyam Pathak on how to build a kubectl debug/docker debug replacement tool that would work the same for all major container runtimes (Kubernetes, Docker, containerd, Podman) and provide a more user-friendly UX. I did contribute to the talk a little, too - you may recognize the style of the drawings 😉

The below Twitter thread is a good starting point:

Ivan Velichko @iximiuz Debugging Distroless Containers Like a Pro 🧵 When a containerized app starts acting up, "docker exec" is the most obvious choice for troubleshooting. But what if there is no shell in the container image: - Google's distroless - Slim(med) - Chainguard - FROM scratch - etc? 3:33 PM • Mar 30, 2024 57 Retweets 259 Likes Read 6 replies

​

If you check the above materials and decide to put the new knowledge to work, I've carefully crafted a few container debugging challenges - do check them out.

Last but not least, I finally fell in love with Dagger! ❤️‍🔥 The tool has always been on my radar, but mainly because of its exceptional team and the noble end goal - to fix our CI, once and for all. Despite my interest, I couldn't really find a good way to use the solution itself. However, the recent release of Dagger Functions changed everything! Things have finally clicked for me.

I spent way more time than I'm willing to admit digging into the internals, reading all available materials, and migrating some of my personal projects to Dagger, and while the beginner's perspective is fresh, I decided to use it to my advantage.

​What Is Dagger and Why You Might Want to Use It is the first lesson in the mini-course I'm working on (and two more lessons are already in the works).

I also prepared:

• ​a Dagger playground where you can play with the tool in different setups (Docker, Podman, nerdctl, and Kubernetes).
• ​a few Dagger challenges (and many more are coming soon).

Hope you'll enjoy it!

What I was reading

Catching up on my reading list, finally!

🎬 A 10-year Detour: The Future of Application Delivery in a Containerized World (a.k.a “Dude, Where’s My Platform?”) - someone finally said it! Well, not someone - Solomon Hykes himself. The idea that you can offload all infra concerns to a cloud provider or a fancy PaaS and focus only on the app business logic is a utopia. Much like with physical goods, where building a factory is often an integral part of the development and production, working on a software project should include building and owning the “factory” - i.e., the platform. Cloud providers and PaaS can help with it, but it’s you who owns the factory, so you better embrace it. This will impose constraints on the project’s design, but there is absolutely nothing wrong with it - it’s just the reality.

​Don’t get hit with the pendulum: DevOps shifted too far left - I absolutely love the title. The idea that developers should fully own Ops as part of their responsibilities (look, we built all these great tools to make Ops easier for you) always sounded unrealistic to me. The DevOps movement shouldn’t have been about the full shift to the left. As with any other domain, the balanced approach is vital, and the optimal state lays somewhere in the middle.

​BuildKit in depth: Docker’s build engine explained - a not too deep/not too shallow read on BuildKit. Since Docker 23.0 (~March 2023), BuildKit is the default image build engine, so it’s a good idea to understand a little how it builds images under the hood - it may help speeding the builds up.

​What is Buildkit? - another great post on BuildKit, this time by Adam Gordon Bell of Earthly. I’d recommend it as a complementary (and much more hands-on) material to the above blog post.

​How ima.ge.cx works - look ma’, a real-world example of a fully serverless architecture! But the actual reason why I’m sharing this post is the ima.ge.cx service itself! It gives you the ability to browse container images’ contents right in your browser, including viewing the textual files. Just add an image name to the link like this https://ima.ge.cx/nginx:latest. The implementation is quite ingenious: images are never pulled from the registries fully, and instead, some clever HTTP range requests, S3 SELECTs, and step functions are used to retrieve this data on the fly. I’ve no idea how come I didn’t come across it sooner 🤦

​Onboarding roulette: deleting our employee accounts daily - “I’m a strong believer in automated tests,” but “…it can take 30 minutes to create [a feature], but might take hours or days to establish automated tests for it.” That’s why I keep saying that balance is important, especially for smaller companies. “Product creation requires the art of strategically taking on debt. The “loan” allows teams to validate quickly, and discovered value can later be spent paying down the tech debt (with interest). This is true for startups raising venture, and it’s true for teams building MVPs. Spend too much time building expensive tests up front, and you may run out of time to ship, learn, and pivot.” The alternative testing approaches are as valid as automated testing, can be cheaper to implement, and they can provide real verification of the correctness and not just assertions of some conditions in code. “Keep in mind that “no automated tests” doesn’t mean “no testing at all.” […] Production traffic and thorough alerting can serve as a weak replacement for automated testing. […] Better than production traffic is a subset of production traffic (canary rollouts). […] Better than canary rollouts is dogfooding.”

​Benchmark results of Kubernetes network plugins (CNI) over 40Gbit/s network 2024 - a solid piece on Kubernetes CNI performance. For a while, I’ve had a gut feeling that unless you’re in a highly specialized domain, the performance of the CNI shouldn’t really matter, and this article kinda confirms it. You should probably focus on the CNI’s UX and feature completeness while pondering the options.

​Kubernetes 1.30: Beta Support For Pods With User Namespaces - the state of User Namespaces support in Kubernetes. The TL;DR is they’re likely coming relatively soon, but containerd and runc, the two most widely used container runtimes, still don’t support them fully (CRI-O and crun already do). Bonus: the blog post also has a video nicely demonstrating the recent container breakout vulnerability, making it relatively easy to understand.

​Istio Ambient is not a “Node Proxy” - a nice illustrated summary of the (at the time) new Istio Ambient architecture. I particularly liked how the architectural differences are described from a historical standpoint, adding an evolutionary aspect to the picture.

​Spinning YARN - A New Linux Malware Campaign Targets Docker, Apache Hadoop, Redis and Confluence - yet another campaign, nothing really new, but the post itself is pretty good. It demonstrates a bunch of anti-forensics and general system-weakening techniques used by the involved malware after it made its way to the host. Clearly, checking your shell history file or the ps output (1, 2) on the compromised host won’t really help much. But at the same time, it messes up the host so badly (adds an SSH key and systemd services, opens ports, and starts a crypto miner, etc.), so it must be really hard not to notice one of your servers has been compromised…

​List Of Top 8 Service Catalog Tools - not every company is big enough to use Kubernetes to need a proper service catalog. But with developers' headcount approaching a hundred and the number of microservices being in dozens, having a centralized place where all services, their owners, SLOs, dependencies, runbooks, etc. are described becomes rather a necessity, and this article gives a surprisingly good overview of the available solutions (spoiler alert - it’s not only Backstage).

​I, Cyborg: Using Co-Intelligence by Ethan Mollick (again) - a bunch of practical advice on how AI can be currently used to produce good writing (no, it’s not about generating articles with AI) and, maybe even more important, facilitate reading. The latter is well-aligned with my vision for iximiuz Labs to be a tool for thought and not just a “blog with playgrounds.” I’m already thinking of the possible way to incorporate AI into the student experience.

​How LLMs Work, Explained Without Math - an engineer explaining LLMs to engineers. Enjoyed the larger part of the post (the ending gets a little bit too intense with too many fancy terms being thrown at the reader).

​What’s worth learning if we have AGI? - a Patreon post by Andy Matuschak [an edu researcher whose work motivated me to build iximiuz Labs], which I cannot recommend reading highly enough. It’s about picking your learning strategy in the world of powerful enough AI (I’m not certain about AGI - this one can put everything upside down, perhaps more like ChatGPT on steroids). The author argues that in many areas, including software development, to solve a problem means to improvise while composing solutions for subproblems and “reflecting in action”. While AI may excel in solving every individual subtask (e.g., generating a piece of code or refactoring a code base), the end solution is often not known at the beginning of the work and may not be possible without the original and idiosyncratic vision of the engineer. The AI helps realize this vision much faster, so the engineering job will include more and more “steering” of the AI [agents]. But how can you steer the AI if you haven’t gained your own technical expertise and formed your own repertoire of moves? Thus, AI or not, being fluent in your domain and seeing the conceptual patterns behind implementation details remain the vital qualities of a successful engineer. But to acquire them, you have to spend enough time building systems without over-relying on the “magic box” of AI. More hands-on, more focusing on fundamentals, more composing bigger blocks, and much less about memorizing particular commands, flags, and syntax details.

Wrapping up

Working on iximiuz Labs is a fun challenge for me personally, but I also keep hearing how the platform helped someone master a new skill or learn a new technology ~daily. With 8,500 registered users and 40,000 virtual machines fired up, the usage of the platform keeps growing, and I'm happy to put more love into it. However, the sustainability of this effort remains my biggest concern. The free tier is truly generous, but I do encourage you to try the premium tier. It comes with a 7-day free trial, so you'll get the taste of the much faster and more durable VMs, unlimited egress, and other yummy premium features. And if you find it useful, keep the subscription going and help me make the platform even better 🙌

Cheers

Ivan