Hello, fellow server dweller 👋

Ivan's here with the last Server Side roundup of the year!

What I was working on

Since my previous update about two weeks ago, when I announced twice bigger playgrounds and a declarative way to create custom playgrounds via labctl, I managed to ship one more (larger) feature and prepare a new batch of DevOps challenges, thanks to the GenAI holiday season 🙈

Tasks Dev Tools

If you have tried authoring a challenge or tutorial or creating a custom playground on iximiuz Labs, you know first-hand how tedious debugging init (and regular) tasks can be.

But not anymore!

With the new Tasks Dev Tools, accessible by the content & playground authors from the bottom right corner of a running playground, all tasks can be handily tracked in the UI. This includes their current status, the stdout and stderr, and the history of previous runs.

Check out this short demo to see the Tasks Dev Tools in action:

Tasks Dev Tools have been available for only a few days, but several people have already reported successfully using the new capability to track down issues with their playground initialization logic, and I myself cannot imagine content authoring without it anymore.

New DevOps challenges

[Easy] Ensure a Graceful Termination for a Containerized Node.js Application - Can you resolve issues with a Node.js container's termination process to ensure graceful shutdown on 'docker stop'?

[Medium] Ensure a Graceful Termination for a Container With an Entrypoint Script - A follow-up. Can you make a container exit gracefully after the 'docker stop' command while preserving its extensible entrypoint mechanism?

[Medium] Diagnose and Resolve Stability Issues in a Go Application Deployment - This challenge focuses on debugging memory usage issues in a Go application deployed in a Kubernetes cluster. The goal is to ensure the application can handle moderate traffic without crashing, even when it runs with significantly constrained memory resources.

The above three challenges are based on real-life stories - don't miss your chance to learn from others' mistakes! 🤓

Additionally, two "classic" sysadmin tasks were added - probably too easy, but they were also motivated by my daily routine:

• [Easy] Which Port Is a Service Listening On?​
• [Easy] Which Process Is Listening on a Port?​

Last but not least, the platform got its very first CKA Practice challenge, which was contributed by Adam Leskis: CKA Practice: Configure Network Policies To Restrict Traffic Between Pods.

Give them a try, you'll definitely learn a thing or two!

iximiuz Labs in 2024 🚀

It's hard for me to comprehend, but I wrote the first line of code for iximiuz Labs exactly two years ago - on New Year's Eve, 2023.

After a few months of hacking in solitude, followed by a short period of closed alpha and a much longer open beta, I announced the platform's general availability in January 2024. But the active development of the platform hasn't stopped!

Here is what we've gotten in 2024:

• ​labctl - to start playgrounds, forward ports, access playgrounds from a local IDE, and, of course, content creation! What started as David Flanagan's feature request ended up being one of the most frequently used parts of the system.
• Content Author Experience went from "all content is stored in a subfolder of the labs' repo, and only I can edit it" to "anyone can author a challenge or tutorial from their local text editor, with a proper page hot-reload and content syncing to a remote S3 bucket."
• ​12 new playgrounds were released, with the recently added Kubernetes Omni playground already displacing the previous Top-3 habitue K3s.
• ​Custom playgrounds for demos, collaboration, workshops, and your own hands-on blog posts became available for everyone.
• ​The new "Asia" playground region was added to improve the local experience for one of the largest groups of users.

As a cross-cutting theme of the year, all playgrounds were gradually getting longer (up to 8 hours on the premium tier) and more powerful (2-4 cores, 8 GB RAM, 40 GiB disk space). A default non-root user was added, and the kernel was upgraded with full nftables support for the perfect learning and hacking experience.

​The platform's UI has been almost fully redesigned, adding a lot of UX improvements (tips & explainers here and there, visual coding of similar elements, font size change in the web terminal, playground tabs surviving page reload, tab renaming, and more), and the backend has been largely rewritten to decrease the operational burden and improve the performance and reliability of the platform.

On the content side of things:

• ​60 DevOps challenges were released (with two contributed by guest authors Adam Leskis and Felipe Cruz).
• Two courses were created: Computer Networking Fundamentals For Developers, DevOps, and Platform Engineers and An Alternative Introduction to Dagger (the latter is only half-baked due to the apparent lack of interest 🤷‍♂️).
• ​A new content format - Skill Paths - was introduced, and the two very first Skill Paths were released: Copy Container Images Like a Pro and Get Started with Linux Control Groups.
• Three new tutorials were published: on Linux cgroups, on how to pick a Node.js base image, and how and why to use multi-stage Docker builds.

The year also brought 12,000 new users (300% growth) and more than 70,000 VM launches, with a mind-blowing number of hours spent on the platform 🚀

And the most surprising but heart-warming and reassuring achievement was the huge Black Friday success when hundreds of people decided to support iximiuz Labs (and get all the benefits of the premium tier in return):

This all makes iximiuz Labs well-positioned for an even more fruitful 2025!

What I was reading

​Why Code Security Matters - Even in Hardened Environments - An "Arbitrary File Write" vulnerability occurs when unsanitized user input is used as a filesystem path. This vulnerability often serves as a gateway for more severe exploits, such as "Remote Code Execution." A common example involves writing a malicious shell script to the root crontab file. But if your application runs in a hardened environment (e.g., under a non-root user and on a read-only filesystem), this shouldn’t be a concern, right? Not necessarily. Check out this masterpiece blog post for an example of how a Node.js container, even in a hardened setup, can be exploited by writing a carefully crafted data structure to an anonymous pipe opened by the Node.js runtime. Stay vigilant when writing code - hardening is never bulletproof!

​Nightmares on npm: How Two Malicious Packages Facilitate Data Theft and Destruction - The spooky season is never over. A simple npm install command can bring a lot of havoc into your dev or CI environment and/or steal sensitive data. And you can only do so much because vetting all transitive dependencies isn’t realistic. That’s why it’s important to exercise general development hygiene: vet your vendors, isolate projects from each other using separate dev machines to limit the blast radius, and never mint "god" tokens with rw access to everything.

​🎬 The dumpster fire of software supply chain security by Kyle Kelly - Finally, a sane voice in the software supply chain security space. Vulnerability management is overrated. One of the points Kyle makes in the video (and I fully agree with him) is that simply running npm install on a dev machine might be a much riskier action than shipping code with pretty much all the time false positive high and critical vulnerabilities to production.

​The Hater’s Guide to Kubernetes - I largely agree with the author's perspective. Some parts of Kubernetes are handier than others, and adopting a pragmatic approach - embracing Kubernetes while significantly limiting the features you depend on - makes sense. It reminded me of how Doom 3 was developed in C++ while all its predecessors were written in C. Despite C++ being a more feature-rich language, the team defined a strict set of "do's and don'ts," intentionally limiting the language features they used to keep complexity in check. This approach effectively "killed the complexity" before it could arise. But even with this "rule of thumb," you should probably adopt Kubernetes only when you find yourself reimplementing its poorer alternative. Anything earlier than that might be premature (IMO, of course).

​AvitalTamir/cyphernetes - A Kubernetes Query Language. I haven't tried using it yet, so I cannot share my experience, but writing MATCH (p:Pod) WHERE p.status.phase != "Running" DELETE p instead of a nasty-looking bash pipeline with a few kubectl commands glued together by xargs does sound appealing. Might come in handy while writing solution checks for iximiuz Labs Kubernetes challenges.

​When Kubernetes and Go Don’t Work Well Together - Despite the title, the issue isn't exclusive to Kubernetes. It's a fairly common scenario where a language runtime (Go, in this case), particularly its garbage collection component, is confused by stricter resource limits set for the application container. When running inside a container, an application often perceives all the host system's available CPU cores and memory. However, the configured cgroup may impose much stricter resource limits. Setting GOMAXPROCS to the number of allocated CPU cores and GOMEMLIMIT to around 90% of the cgroup's memory limit is generally a good practice. That said, you should probably avoid restricting a modern backend application to just 400 MB of RAM 🙈.

​GOMEMLIMIT is a game changer for high-memory applications - This post is linked at the end of the above post, but it's so good that I wanted to highlight it here, too.

​Exploring A Basic Kubernetes Network Plugin by Rory McCune - A mini-dive into (one layer of) Kubernetes networking. Kubernetes doesn't mandate how Pods get their IP addresses and how these IP addresses are routed between nodes. These decisions are offloaded to the "networking plugin" (often called a CNI, but it's not very accurate). The post sheds light on the mechanics of this layer of Kubernetes networking using kind and kindnetd as an example.

​kubespec.dev - kinda sorta kubectl explain in the browser. A neat alternative to the CLI experience, plus some resources have a decent example section.

​Automating Infrastructure as Code with Vertex AI - A rather long write-up on how one company automated generation of Kubernetes YAML manifests for their CRDs with Gemini 1.5. Personally, I find the topic unexciting, but it's a useful data point. For instance, from the post, you'll learn that you don't always need RAG, especially if your context window is large, that writing lengthy and unambiguous prompts is likely the key, and that testing of LLM "programs" remains an unsolved issue.

​Why pipes sometimes get "stuck": buffering by Julia Evans - A good research on stdio buffering in different command-line tools and programming languages with a bunch of examples of how it can get in the way while scripting things and what workarounds you can apply.

​Running Lima on GitHub Actions - A short and handy one. Why might you want to do it? GitHub Actions de facto force you to run your Linux build scripts on Ubuntu. With Lima and just a few lines of extra scripting (see the post), it becomes possible to run commands on non-Ubuntu distros (e.g., Debian, Fedora, etc.) without any significant performance penalty. And another reason is when you want to emulate a multi-host setup during a CI run.

​Keyset Cursors, Not Offsets, for Postgres Pagination - A must-read for any backend developer on how to organize "proper" pagination in the API response (and why you likely want to avoid explicit page numbers backed by LIMIT + OFFSET queries). The article mentions Postgres, but the keyset approach is valid for any relational and non-relational databases. In the latter case, when you can often rely only on a certain key ordering to iterate the records, knowing how to use the keyset pagination becomes not just nice vital but vital.

Wrapping up

There is a lot of FUD and predictions about the nearing extinction of software engineers (especially from people far from hands-on positions). GenAI is undoubtedly making code cheaper and faster to write, but this will likely induce an increase in demand in perfect accord with the Jevons paradox, which will mean an exponential growth of software. And more software means more and larger engineering problems to solve, especially in operations and security domains, where LLMs are much less helpful. So, instead of switching to ML, it might be the best time to double down on an ops, system architect, or security engineer career, and iximiuz Labs can help you with that 😎

Happy New Year!

Ivan