Hello there!

It's Ivan, with a slightly overdue June's roundup of all things server-side craft 🧙

What I was working on

The better part of the month went into finalizing and then releasing Playgrounds 2.0. It was the topic of the mid-month issue, so you likely already saw it, but I also decided to publish a proper write-up (on my good old blog) featuring the new iximiuz Labs capabilities and the most interesting implementation details. Check out Server-Side Playgrounds Reimagined: Build, Boot, and Network Your Own Virtual Labs, if you want to learn more about:

• The essence of iximiuz Labs playgrounds
• How VMs are constructed by mixing the rootfs and the kernel
• How to run custom playgrounds with simple YAML manifests
• How to bake and use custom VM rootfs images
• How to create multi-network playgrounds
• Why you should start creating Linux playgrounds today

I didn't see it coming, but the work on Playgrounds 2.0 resulted in an almost complete rewrite of the "core engine" and a good part of the orchestration logic, so I had to put the further development of persistent playgrounds on pause - simply to let the new code roast in production and stabilize the system.

Even though iximiuz Labs currently has over 400 automated tests (with more than half of them being e2e), and more than 300 playgrounds are started every night by CI "solution bots", some failure modes and emerging patterns can only be spotted in production, given enough time and usage. And luckily, the usage numbers are through the roof!

I'm hoping to return to persistent playgrounds later this month (and I still believe we're just a couple of weeks away from the MVP), but for now, I'm temporarily shifting the focus back to the content work, where the key theme will be the Docker Roadmap 🩵

Here are the challenges that I prepared for you in June:

Networking:

• ​Configure Routes to Connect Two Private Networks​
• ​Enable Internet Access for a Private Network with a NAT Gateway​

Linux Storage​

• ​Mount a Drive with Existing Data and Read Its Contents​
• ​Create an Ext4 Filesystem on an Unformatted Drive​
• ​Create a GUID Partition Table (GPT) on a Blank Drive​
• ​Split a Drive into Multiple Partitions And Format Them as Ext4 and Btrfs​

Kubernetes

• ​Deploy a Pod with a Container Image from a Private Registry​

Have fun!

Community update

People continue to publish great content on iximiuz Labs, and I cannot be happier about it. These are just some of the playgrounds that Márk Sági-Kazár prepared in June:

• ​HA Kubernetes Cluster Playground​
• ​Nix (on Ubuntu) Playground​
• ​Kubeadm Playground​
• ​etcd Cluster Playground​
• ​OpenSearch Playground​

All Márk's playgrounds are created with true love for the craft and should be extremely helpful for people studying Kubernetes and its components.

Another gem that showed up in June - 🎬 Remote Homelab Access with Tailscale (and an accompanying blog post) by Adam Leskis is a proof of concept, framed as a realistic "remote homelab" setup, that iximiuz Labs playgrounds can be part of your Tailscale WireGuard network and even SSH-ed to from the phone. Kudos to the author for figuring it all out and thoroughly documenting their experience 💪 And if you want to "borrow" the provisioning scripts, Adam kindly published the playground as well.

What I was reading

Quite a lot, actually! Catching up on my reading list.

​Image Compatibility In Cloud Native Environments (Kubernetes Blog) - Tell me "OCI runtime" is a poor execution environment abstraction without telling me. We've been told that container images we build and run locally will work exactly the same in a remote Kubernetes cluster, a managed service like AWS App Runner or Google Cloud Run, or any other OCI-compatible runtime. Well, they lied to us. Turns out, sometimes, the kernel version, the presence of kernel modules, or versions of some host OS tools (e.g., iptables, GPU drivers, etc.) matter. So, now we've got an extension of the OCI spec trying to address this problem, and Kubernetes' Node Feature Discovery (NFD) gets to implement it first.

​Introducing WizOS: Securing Wiz from the ground up with hardened, near-zero-CVE container base images - It's always great to see another "hardened" image provider, but this post and the approach outlined in it give a rather weak impression of the new WizOS. The choice of the docker.io/library/golang base image on the screenshot of "a typical vulnerable image" Wiz now has an alternative for is particularly bizarre - the golang image should never be used as a runtime base image anyway, and very few companies actually scan build-time container images for vulnerabilities.

​Secure, Minimal, Production-Ready Images - Looks like everyone is into building better images! But Docker's offer sounds more reasonable and useful - wouldn't it be great to have a more secure less noisy version of Nginx, Redis, or any other popular open-source container image that you can simply pull from Docker Hub. And on top of that, it seems now possible to assemble a perfect Debian or Alpine-based image, adding only the packages you need and taking them from Docker-maintained hardened package repositories. I mean, Chainguard has had this for ages, but the more providers, the merrier the lower the price!

​Introducing SecureBuild - What did I tell you? The third announcement in a ~month. Replicated's images are Wolfi-based (unmodified), but unlike Chainguard's images, Replicated wants to try to make them behave the same way as their Docker Hub counterparts, so you run them the exact same way you would run/use the original images. Definitely worth a shot (for Replicated) and may or may not grow into something really good. Time will show.

​Kubernetes security diagram (cheatsheet) - Next time someone tells you Kubernetes is easy to protect, send them this diagram. You may not need to have all of the parts highlighted there bullet-proof hardened, but it should be a conscious decision what to secure and to what extent.

​Kubernetes v1.33: Updates to Container Lifecycle (Kubernetes Blog) - Since Kubernetes 1.33, you can finally change the stop signal sent to the pod by the container runtime without rebuilding the whole image. Now you can set spec.containers[].lifecycle.stopSignal: to override the default STOPSIGNAL instruction - but don't forget to also specify spec.os.name (the latter is needed because Kubernetes allegedly supports not only Linux containers). For now, it'll only work if the ContainerStopSignals feature gate is enabled and the container runtime supports this feature.

​Kubernetes v1.33: Image Pull Policy the way you always thought it worked! (Kubernetes Blog) - Speaking of Kubernetes security gotchas. Did you know that a Pod with an imagePullPolicy: IfNotPresent by default will be able to access any images present on the Node even if it wouldn’t be able to pull them from the remote registry due to insufficient permissions? This is a fun "leak" that iximiuz Labs started suffering from, too, since the introduction of "external" rootfs sources. I even had to put a warning in the playground constructor UI, making it clear that any rootfs image can be accessed by any playground, regardless of the playground's owner and/or access control settings (so use random names or better don't put anything sensitive in the playground rootfs images). Luckily, in Kubernetes 1.33, work has started to address this issue.

​Kubernetes v1.33: User Namespaces enabled by default! (Kubernetes Blog) - Good news! But the full support requires a relatively fresh Linux kernel (6.3+), so I'm afraid even with the recent upgrade to 6.1, you still cannot play with user namespaces in Kubernetes on iximiuz Labs. At least not fully (because, to some extent, they work in Kubernetes since the 5.12 kernel). So I should probably ship another kernel upgrade - and better sooner than later 🙈

​What You Need To Know About Apple's New Container Framework - There was a lot of buzz about the just-released Apple container CLI (well, at least in my feeds), but very few posts managed to highlight the essential difference between Apple's and Docker's implementation of containers on macOS - unlike Docker Desktop, Lima and similar desktop container environments that launch a single Linux virtual machine to run containers on macOS, Apple's Containerization framework executes each Linux container inside of its own lightweight virtual machine (VM), providing hardware-level isolation for each container. But don't get it wrong - we didn't get "native" macOS containers. In both Docker Desktop's and Apple container's cases you're still running good old OCI-compatible Linux containers (implemented with namespaces and cgroups) - the difference is only in the number of (rather transparent for the end user) virtual machines spawned to get Linux namespaces and cgroups available on macOS, and the plumbing to connect these containers with the host system. Quoting the Edera folks: "Apple's entry into container runtimes isn't just about providing an alternative to Docker Desktop on macOS. It's a validation of hypervisor-level isolation as the security model containers should have had from the beginning." And I tend to agree with this take.

​Building effective agents by Anthropic (more practical) and Agents by Chip Huyen (more academic), plus this helpful summary - Interesting and closely aligned reads on how "agentic" systems are built. Assuming they were written independently, it starts to look like "design patterns" of agentic systems are starting to crystallise… and they are not very different from higher-level approaches that have been used in "traditional" software engineering for ages: decompose the problem on sub-problems, use specialized solutions (sub-agents) for well-defined tasks, use an orchestrator and synthesizer to scatter/gather the tasks, use "worker" agents to solve sub-tasks in parallel, and evaluator to analyze the final result (a.k.a., Planner / Executor(s) / Evaluator pattern), etc. And whatever you do, keep it simple, iterate, and verify the results as often as you can. These should, in fact, be approaches and techniques inherent to any good engineer. So, building systems that leverage LLMs might not be that different from building systems that don't!

​How we built our multi-agent research system by Anthropic and its digested version by Simon Willison - A fresh and experiential read that only confirms the validity of the "design patterns" from the above 6-month-old papers. Which is great, because if something in AI didn't age like milk a week after it was published, it's a real gem.

​The lethal trifecta for AI agents: private data, untrusted content, and external communication by Simon Willison - the CAP theorem of agentic systems. If all three match, the system is insecure regardless of how many times your MCP server’s description says "secure & authenticated access" and/or if you run it in a safe and isolated environment (but still with access to sensitive data). AI agents are similar to the eval construct in your favorite programming language (but higher-level, of course) - you don't eval untrusted input unless you limit the impact by extreme sandboxing. But agents eval data they process, and extreme sandboxing reduces agency 🤦

​GitHub MCP Exploited: Accessing private repositories via MCP - A must-read for all MCP enthusiasts out there. None of the MCP servers that 1) have access to your data, 2) read untrusted content, and 3) can communicate externally are secure, even if they were created with good intent (likely most of them) and by trusted vendors.

​How to Stop Your Human From Hallucinating - As an engineer, I learned to value determinism and precision in my systems, likely because heisenbugs are the worst kind of issue one may get to troubleshoot. But expecting the same level of determinism, punctuality, and correctness from people would be unrealistic. And, nevertheless, we don't dismiss other people as helpers in our work. Yes, my colleague may not be correct 100% of the time (and so am I), but still, as a team, we can ship more stuff and faster - if we find an effective way of communication, sharing context, and validating the results. And this is exactly how you should treat the AI agents - not precise, not always producing the right results, but performing much better when the problem is clear and the context is complete and unambiguous. So, that same set of "soft skills" that allow you to function better as a team is applicable for augmenting your capabilities with (coding or not) agents.

Wrapping up

I ranted on Twitter the other day about how iximiuz Labs is not yet a sustainable business. Even though $90K on Gumroad (plus $20K on Patreon) looks like a great figure, subtracting $12K of Gumroad fees (13%) and about $30K of bills paid for infrastructure and collateral services so far, makes the final profit just about $70K. Definitely not enough money to make a living in Western Europe, especially given that it took me 2.5 years to earn this amount.

And nevertheless, I'm hopeful! The trend on the above chart looks promising, and I'm truly grateful for the amount of support I received from you all in June. The playground usage numbers also align with the upward trend in premium membership sales, so we're likely on the right track here 🚀

Happy hacking!

Ivan

P.S. There is a good chance that you can expense iximiuz Labs Premium using your employer's learning and development budget.

P.P.S. If you're offering Linux, DevOps, or Kubernetes training and struggle with providing identical learning environments for your students, check out iximiuz Labs for Trainers.

​