Hello 👋

After preparing over 100 practical exercises on Linux, Docker, Kubernetes, and networking, which have been solved more than 50K times in total, I can confidently say that the topic that attracts the least interest is building container images.

No wonder most images end up bloated and full of CVEs.

One reason for the lack of interest may be that container images are no-man's land. Developers may not feel the pain that bloated & vulnerable images cause in production, and the ops people may not be sufficiently versed in the application stacks to feel confident suggesting Dockerfile changes.

And this creates a unique opportunity for you.

If you're a developer, learning just a tiny bit of Linux & Docker can quickly make you a go-to person for all things building images. And if you're an SRE/DevOps engineer, getting your hands dirty with Node.js, Python, or Go ecosystems & application stacks will help you fix image issues right in Dockerfiles - not when a bloated image congests your release pipelines or production gets hacked. And this is how you become instrumental for the project's success 😉

If you liked the sound of it, I've got some good news: the second module of my hands-on Docker roadmap is finally complete, and it focuses on all things building images. The module consists of:

• 28 hands-on Challenges
• 6 Tutorials
• 2 Skill Paths

You'll start by building images from existing Dockerfiles, then gradually progress to writing simple Dockerfiles for Node.js, Go, and Python, installing system and application dependencies, compiling apps during the docker build process, optimizing images with multi-stage builds, supporting foreign CPU architectures, and finally producing production-ready images.

The roadmap is heavily focused on learning by doing, but it also includes deep dives into image internals (the OCI Image Spec, visually explained) and the fundamental Linux capabilities behind container images, so the skills you'll get won't be superficial.

And I left the best part to the end - to celebrate this important milestone, I'm making all materials of the second module free until May 🎉 But don't put it on the back burner - the materials will be free only for one month!

New iximiuz Labs features

My main focus in April was on preparing new learning materials and supporting a couple of events during KubeCon (what a crazy week), but I also managed to ship a couple of features.

Daily Practice

The main dashboard got a new default tab that recommends up to 4 daily challenges - one per key category: Linux, Networking, Docker, and Kubernetes.

I've already heard from several people that they really liked this addition, as it reduces the friction of picking your next exercise and also tricks you (in a good way) to play the Inbox Zero game and make all the doughnut charts green.

And just a friendly reminder that solving a challenge permanently adds +5 minutes to your max daily usage time, and the first 10 solved challenges will reward you with a month of free Premium access 🚀

New Labctl IDE command

No more bizarre labctl ssh-proxy --ide workarounds.

A quick example: start a new Go development sandbox, clone a repo into it, and open your local VS Code pointed to that repo:

PLAY_ID=$(labctl playground start golang)
​
labctl ide code $PLAY_ID --repo https://github.com/iximiuz/kexp

Check the docs for a few more examples & use cases.

Automatic HTTP(S) Port Scanning

Exposing HTTP/HTTPS ports has become much handier. The VM's resident agent (examiner) gained port-scanning capability to detect open HTTP/HTTPS ports and recommend them in the Expose Port UI dialog. The feature is also available in the CLI: you can use the new labctl expose port --scanned flag to expose all detected ports at once.

Thanks to Arjun Dandagi for nerd-sniping me!

Materials by the community

One more fruitful month for the iximiuz Labs community!

​Uncloud: How to Deploy a Django Web Application - a new tutorial by Anton Ovchinnikov and Pasha Sviderski on how to deploy a Django app to a bunch of VMs, the pragmatic & lightweight way. If your setup has outgrown a single-host Docker Compose but Kubernetes' complexity still feels like unnecessary overkill, Uncloud might be a solid middle ground.

​An iximiuz Cluster of Clusters with Tailscale and Cilium by Adam Leskis. A fascinating experiment in which Adam connected a few iximiuz Labs' playgrounds, each running several VMs, into a single overlay network, then provisioned a Kubernetes cluster on top, deployed a test microservice application, and visualized its traffic. This is exactly how learning happens! Surely it'll make a great story for a technical interview, and the fact that the results are documented as a blog post and a reusable playground makes it twice as valuable. Be like Adam!

​Log Parser Playground by Adam again. The playground generates one log parsing activity at a time, and regenerates a new one when you complete the parsing check. It uses authentic log formats such as Nginx access logs and Syslog entries to simulate realistic parsing scenarios. Perfect for training Linux command-line skills.

​Arjun Dandagi continued his experiments with running GUI applications on iximiuz Labs. This time, we've got Omakub (an opinionated Ubuntu setup) as a remote desktop playground. You can play with its beautiful interfaces right from the browser (via Apache Guacamole) or any compatible local RDP client. Latency can definitely be improved, but one use case where it doesn’t matter much is running a coding agent in the same playground and connecting it directly to a remote Chrome instance.

A new batch of hands-on Kubernetes challenges by Omkar Shelke. If you're preparing for CKA/CKAD/CKS exams, solving them is a good way to practice:

• ​Rolling Update and Rollback a Deployment​
• ​Manually Schedule Deployment Pods on a Specific Node​
• ​Manage Static Pod Lifecycle on a Cluster Node​
• ​Create a Pod with Guaranteed QoS Under ResourceQuota Limits​
• ​Set Up Horizontal Pod Autoscaler with CPU and Memory Targets​
• ​Resolve OOMKilled Errors by Reading VPA Memory Recommendations​
• ​Bind a PersistentVolume to a Specific Node Using Node Affinity​
• ​Restrict Namespace Resources Using ResourceQuota​
• ​Enforce Resource Policies Using LimitRange​
• ​Debug Why a ConfigMap Could Not Be Created​
• ​Enforce Immutable Container​
• ​Disable ServiceAccount Token Automounting in a Pod​
• ​Grant a ServiceAccount Access to a Specific Secret Using RBAC​
• ​Troubleshoot RBAC Permissions for a Failing Deployment​
• ​Debug and Resolve RBAC Failures for Two Deployments​
• ​Use kubectl-convert to Fix Deprecated APIs in a Kubernetes Manifest​

Good technical reads & links

​A Decade of Docker Containers - Despite the lengthy and a little too broad foreword, the article eventually turns into a great technical write-up on which Linux capabilities made Docker (and later on - OCI) containers possible, and how the tech "emulates" the user experience on non-Linux platforms. A bunch of helpful diagrams included. Definitely a good recap, even if you already "know it all".

​Your Container Is Not a Sandbox by Emir Beganović of Booking.com - An impressive, interactive, and most importantly, comprehensive overview of the microVM ecosystem (Firecracker, Cloud Hypervisor, QEMU, libkrun, crosvm, and more). Why would you want to use a microVM instead of a container? Because containers are not a security boundary. Docker solved application packaging and resource sharing problems, but the default isolation Linux containers provide is too weak. It’s usually good enough for trusted multi-tenancy, but 8 escape CVEs in 18 months make it a poor choice for potentially rogue workloads like AI agents or unvetted AI-generated code. And by the way, you can and often should still use containers - but inside microVMs.

​Running Agents on Kubernetes with Agent Sandbox - Kubernetes Blog - A Sandbox CRD is coming to Kubernetes (in the form of an API extension for now). The idea is that while agents can run in pods, their composition, scaling, and identity requirements differ from the traditional Kubernetes workloads - stateless services. IIUC, the Sandbox will be a new "deployment primitive", similar to pods but optimized for higher isolation (via gVisor or Kata Containers), faster startup time (via warm pooling), and stable identities (hostnames and service accounts, I guess). The project is early on, but you can already give it a try in a test cluster (the blog post has an example).

​Clinejection - Compromising Cline's Production Releases just by Prompting an Issue Triager - Simon Willison's take on yet another exploit of GitHub actions. The novelty in this one is that it was done via a prompt injection targeting a seemingly harmless AI automation workflow (doing issue triaging). But IMO, the scariest part here is how the lateral movement happened. The targeted workflow didn’t contain any critical tokens to steal, so the attackers spoiled the GitHub actions cache (of the npm install step), which was then reused by a much more critical release workflow. As a result, 4,000 developer machines installed a malicious VS Code extension. Geneious and devious!

​Trivy Under Attack Again: Widespread GitHub Actions Tag Compromise Exposes CI/CD Secrets - Twice in one month. It's a bit ironic that a vulnerability scanner that you installed to make sure your container images are secure may end up being the thing that actually steals your data. Software development becomes increasingly insecure, and GitHub and other software supply chain links aren't keeping up. One more reason to do more work in isolated sandboxes like iximiuz Labs playgrounds.

​Package Managers Need to Cool Down - A "dependency cooldown" is something I've been exercising since way before the original post about it was written by William Woodruff. The idea is simple - if npm outdated (or the like command) shows a bunch of fresh versions, don't rush to upgrade to all of them just yet. Instead, go vet them - not just for release notes and compatibility warnings, but also to simply see if it looks like a legitimate release (and not as a developer account compromise). Don't have time for it? Then just wait until someone else discovers such issues by letting the new versions "cool down."

​Awesome npm Security Best Practices by Liran Tal, my Node.js security go-to person - a complementary (to the above post) resource with a bunch of "best practices" which, in particular, shows how to configure the package installation cooldown period for all popular package managers (npm, pnpm, yarn, etc.).

Wrapping up

That is it for March. Lots of things to explore and learn, but it'll definitely pay off. Focusing on fundamentals now matters more than ever, and the best way to get them straight is by doing! 💪

Happy hacking!

Ivan