Hello there 👋

It’s Ivan, with a traditional roundup of all things Linux, Containers, Kubernetes, and Server Side delivered to you at a non-traditional time of the month.

It’s not the end of the month, and I also somehow managed to miss the previous month’s issue, but here it is: an assorted collection of iximiuz Labs updates, new community learning materials, and a few good reads that have been sitting in my stash for a little too long.

The main topic of this issue is… documentation and search. Yep, not the most clickbaity announcement ever, but I’m actually excited about it.

What I was working on

Custom Playgrounds finally have proper docs!

If you’ve been following iximiuz Labs for a while, you probably know that Playgrounds started as a supporting piece for Tutorials, Challenges, and Courses.

Every learning material needed a safe Linux environment attached to it, so I built a Firecracker-powered playground engine that could spin up one or more remote VMs in a few seconds, connect them into a private LAN, and expose them to the learner through the browser or good old SSH.

But over time, Playgrounds have become much more than “the terminal on the side.”

People use them as disposable Linux machines, “remote” homelabs, workshop environments, demo stages, bug repros, ephemeral Kubernetes clusters, Docker hosts, coding-agent sandboxes, and probably a few other things I still haven’t discovered.

And the more powerful Playgrounds became, the more obvious the missing piece was - there was no single, complete, up-to-date place explaining how to build your own.

Well, now there is! 🥳

Custom Playgrounds docs

The new docs cover it all - from the smallest possible single-VM playground to much more advanced setups:

• Playground building blocks: machines, drives, networks, tabs, and init tasks
• Your first custom playground with labctl
• Multi-drive VMs
• Multi-machine playgrounds
• Multi-network topologies
• Startup provisioning with init tasks
• Custom UI tabs
• Using custom rootfs images
• Sharing and access control
• The full playground manifest reference
• ...and even a few practical recipes​

Why does it matter? Because iximiuz Labs' custom playground is probably the closest thing to “CodePen for infrastructure” that I can currently imagine.

If you write a blog post about Kubernetes, you can attach a runnable cluster to it.

If you want to explain a networking problem, you can create three VMs, connect them through two subnets, and let people actually route packets instead of staring at a diagram.

If you’re preparing a workshop, you can give every student the same starting point.

If you’re debugging an issue with a teammate, you can create a minimal repro environment and share it as a URL.

And if you’re looking for a new DevOps/SRE/Platform role, a polished custom playground may be a much better proof of skill than a pile of generated YAML in a GitHub repo. A hiring team can start it, inspect it, break it, fix it, and immediately see how you think about real systems.

So, if you’ve been waiting for a good moment to try building one, this is it: Create a Custom Playground ⚒️

Search is coming to the command line

A week ago, I added site-wide search that works across all learning materials, playgrounds, and documentation. Another long-awaited feature many of you have asked about.

And now, you can search on iximiuz Labs from the terminal, too 🚀

Why would you want to do it? Because now you can ask your favorite coding agent to compose a personalized learning path (from existing human-crafted materials, of course) and store it as a plain Markdown page or an Obsidian note. Some examples:

labctl search "kubernetes networking"

labctl search --kind challenge --category linux "namespaces"

Community learning materials

One of the nicest things about the past few months has been watching people publish more and more of their own materials on iximiuz Labs.

​Automating AI Out of a Job by Adam Leskis - A thoughtful post on using AI not as a replacement for learning, but as a way to build small tools that help you learn better. Adam wanted to get more hands-on with Linux performance troubleshooting, so instead of repeatedly asking an LLM what vmstat, iostat, sar, or top mean, he vibe-coded a pair of CLI tools based on Brendan Gregg’s USE methodology that can run on any Linux server. And to make getting started even easier, Adam also prepared a playground with these tools pre-installed: USE Practice Playground. And a video walkthrough is also available:

​

​Cloud Tracing Lab Playground, also by Adam, lets users diagnose realistic distributed tracing scenarios in a preloaded K3s VM to get more familiar with traces in general and jaeger-ui specifically. There’s guided investigation through distributed traces, along with identifying slow spans and comparing two traces to determine what changed that made it slow down.

​Combining 2FA and Public Key Authentication for better Linux SSH security by Med Unes - the tutorial shows how to harden SSH access for a stock Debian server the way you would do it in production: replace passwords with public-key auth, disable root login, add a TOTP second factor with PAM, and move off port 22. This is sysadmin 101 stuff, which must not be forgotten, no matter how much of our work we offload to agents.

​Kubernetes Storage: From Ephemeral to Persistent - a skill path​ by Kaloyan Preslavski. It’s a four-challenge sequence that walks through the Kubernetes storage stack from the simplest ephemeral volume to manually managed persistent storage. The path covers:

• emptyDir volumes: sharing data between containers, lifecycle, memory-backed storage, and pod eviction
• dynamic provisioning with StorageClass
• default storage class injection
• static PV binding patterns
• ..and many other storage-related topics

Silver Stack by Muhammad Ibtisam - a series of playgrounds starting with an Ubuntu-based Dev Machine carrying the common DevOps tools, then adding Jenkins, Nexus, and SonarQube servers, and finally combining all 4 VMs into a complete CI/CD solution. And the best part, all the machines can be accessed on their own domains, like jenkins.ibtisam-iq.com or sonar.ibtisam-iq.com - thanks to the Cloudflare Tunnel magic.

Ibtisam thoroughly documented his setup in a blog post, Building a Self-Hosted CI/CD Stack Behind NAT: From EC2 to iximiuz Labs with Cloudflare Tunnel, and the custom playground manifests are available on GitHub. I cannot think of a better way to demonstrate your infrastructure skills than this combo 👏

​Podman and Linux Namespaces by Shan Desai - a deep and practical post on how Podman works with Linux user namespaces and how to configure or update UID/GID mappings for the current user.

Shan has also prepared a Podman with Docker Compose Playground that shows how to use Docker's Compose implementation with Podman (instead of Docker itself) for a better user experience (because Podman's Compose doesn't always behave).

Omkar Shelke published another batch of Kubernetes challenges, and they are really fun!

• ​Inject an Ephemeral Container into a Distroless Pod and Test Cross-Namespace HTTP​
• ​Run a Short Task as a Standalone Pod Without a Job Controller​
• ​Puzzle Block Game Application Is Gone, Recover It from etcd​
• ​Take and Restore an etcd Snapshot on a Kubernetes Cluster​

​Native SSH Reverse Tunneling with Pomerium by Nick Taylor. You can expose a local service (e.g., a dev server) to the public Internet with a simple reverse SSH tunnel (ssh -R). However, if you want to control who can open such a tunnel and who can access the published service, you can use Pomerium (open-source) to terminate the ssh -R sessions. Pomerium places OIDC authentication in front of the resulting HTTPS route and enforces access policies. Handy for dev boxes, CI runners, on-call laptops, and homelabs sitting behind NAT, and Nick shows how to configure it (of course, you'll be able to try it right in the playground).

Good reads

​You Need to Rewrite Your CLI for AI Agents - A good practical read on how to enhance (not fully rewrite - the title is clickbait) your CLI to make it more agent-friendly. The key idea is to make your CLIs support both modes: human-friendly (individual flags for each parameter, table or text output, or even a TUI) and agent-friendly (raw --json for input params and --output json for output). The author also recommends allowing introspection of the API schema (behind the CLI) with the helper command, making the CLI embed the documentation (so that agents don’t Google it, wasting a ton of tokens), and baking in the skills (which can be installed for each agent session). Very concrete and actionable advice.

​The Coming Loop by Armin Ronacher - On when to use and not to use loops while building software and potential future implications of letting the harness decide when a task is done. “…Present-day models tend to produce code that is too defensive, too complex, too local in its reasoning. They avoid strong invariants. They add fallbacks instead of making bad states impossible. They duplicate code, invent bad abstractions, and paper over unclear design with more machinery. Worse, though, I so far see very little progress in this improvement.” Armin’s summary is just perfect. Nothing to add, nothing to remove.

​The HTTP QUERY Method - RFC 10008 - if you, like me, periodically find yourself in a situation where semantically you need a GET request, but the parameters are too long to fit in the query string, I’ve got some good news for you - the ugly POST-based workarounds are no more. The new QUERY method is similar to POST, but QUERY requests can be automatically repeated or restarted without concern for partial state changes, just like normal GETs. Now we just need to wait for the new method to become widely (enough) supported by the libraries, proxies, and networking firmware (which can take ten more years).

Wrapping up

That’s it for this not-quite-monthly roundup. If you’ve built an interesting setup recently - a cluster, a debugging lab, a CI/CD pipeline, a weird networking topology, or a reproducible failure scenario - consider turning it into a playground. And come to our weekly community call to show and tell about it!

Happy hacking!

Ivan

P.S. The Summer Skill-Up sale is coming to an end - don't miss the chance to upgrade your membership and unlock access to the entire iximiuz Labs learning collection and more powerful playgrounds.