Hello friends!
It's Ivan again, with a monthly roundup from iximiuz.com.
Busy times! A lot of learning and a lot of work. Spent the better part of the month wearing my SRE hat - serverless (as in FaaS) observability is still hard. The last time I tried my luck with AWS Lambda was in 2017-ish. We ended up with some ten(s of) rather ad-hoc and small functions, and it quickly became painful to monitor them. In 2022 this problem seems to be solved - there is AWS X-Ray and also plenty of 3rd-party offerings allowing you to auto-instrument your functions and generate nicely-looking dashboards... But it all breaks apart as soon as you get a slightly less trivial routing logic. Observing something more granular than a single function invocation is still hard. OpenTelemetry in general and ADOT in particular give some hope, but integrating them with AWS Lambda remains tricky. Curious to hear your thoughts. How do you collect metrics on the level of a route if your lambda function serves a full-blown REST API?
Switching gears, there's some exciting news on the educational front! I've been talking about converting my blog into some sort of a learning platform for quite a while... But I haven't been just talking! And after almost a year of (sporadic) work, I'm ready to present a thing that will eventually become the foundation of that platform. Intrigued? Then read on!
SPONSORED SSH Certificates: How Do OpenSSH Certificates Compare to X.509? I knew they are not entirely the same even though they share the main principles, but the actual difference remained a mystery for me. Until I stumbled upon this condensed read by Teleport. Good stuff for infra folks, go check it out!
โ
As you might have noticed, I'm a visual learner. Otherwise, why there would be so many diagrams on my blog? ๐ But I also believe that the hands-on experience matters. It's not enough to just read about a tool, or a trick, or a language, even if the text is well-sprinkled with drawings. You have to get your hands dirty and try it out - it'll deepen the understanding and make the knowledge more durable.
So, how to combine text, visuals, and interactivity in one place?
Right, build a Web-based Kubernetes dashboard! But fear not, I'm not trying to rewrite Tanzu's Octant ๐ And it's not a replacement for the explanatory text or static diagrams either. Instead, the dashboard is meant to be a complementary mechanism augmenting the existing and future blog materials.
Here is a (very early) example - an attempt to show what happens to a Deployment when one of the Pods gets deleted:
Having such a tool embedded in a blog page should facilitate the learning experience. Instead of just trusting me that a Kubernetes controller consists of a Deployment with a single leader Pod running on behalf of a Service Account and watching instances of a Custom Resource, the reader should be able to see it with their own eyes by just deploying the Kubebuilder's example project!
Here is another example - an attempt to visualize a Deployment receiving a Pod Template spec update:
For those of us dealing with and/or writing Kubernetes controllers, a solid understanding of their behavior is a must. The above example illustrates the behavior of some of the built-in controllers, but similarly, a custom controller can be visualized. Now, imagine how breathtakingly it should be to watch a chaos monkey controller harvesting Pods in the wild ๐
Another important thing that can be studied with such a tool is how to work with the Kubernetes API. There are many ways to interact with Kubernetes resources - they can be updated, patched (using different strategies), or a change can be applied to them. The mighty kubectl tool often shields you from these details, and it's good and bad at the same time. One of my design goals while working on the Kubernetes UI is to keep all the interactions with the API server transparent. So, it should be possible to use it like a Postman's REST client but for Kubernetes API.
Another pet peeve of mine while developing applications (or controllers) for Kubernetes is the lack of the tools to quickly create a persistent view of the cluster (or multiple clusters) showing me updates to the relevant resources in real-time. For instance, when I'm working on a controller, I'd typically want to periodically check the states of the manager's deployment and pod(s), refer to the corresponding service account, and review roles and role bindings. So far, I've been doing it with ad-hoc kubectl queries or (if the project is long enough) by hacking together some nasty scripts harnessing kubectl with fancy selectors and the Linux watch tool. But having a page showing me (potentially interconnected and periodically refreshed) objects corresponding to a number of selectors would save me from a lot of pain. And probably for a multi-cluster/multi-version application leveraging a service mesh to glue things together it'd be even more useful.
The tool I've been working on is in a pre-alpha state, so many of the above things still exist only as rough ideas. But I can already use the dashboard to record some small demos. Hopefully, next month I'll be able to apply it for something more serious - like going through the Kubebuilder book and visualizing the progress.
The tool is written in Go (surprise surprise) with a Vue.js frontend, and I have all the plans to make it open-source one day. It's designed to work as a standalone desktop application and as a client-server deployment (with the server part having access to the Kubernetes clusters). And I'm really curious to hear your feedback!
โ
Didn't have much time for writing this month, but I hope to catch up in the coming months - reinforced with the new tool ๐ Here is a short (but apparently well-received) article I published at the beginning of June - How To Start Programming In Go: Advice For Fellow DevOps Engineers. As always, based on personal experience, so YMMV.
โ
โ
Well, this is it for June. Hope you enjoyed the month as much as I have :)
Oh, totally forgot! That Kubernetes UI I've been working on, I don't have a good name for it yet. If someone has an idea, I'd love to hear it!
Stay safe and healthy!
Cheers,
Ivan Velichko
โ
Building labs.iximiuz.com - a place to help you learn Containers and Kubernetes the fun way ๐
Hello ๐ Ivan's here with a slightly delayed September roundup of all things Linux, Containers, Kubernetes, and Server Side ๐ง What I was working on This month, I worked on an assorted set of topics. Skill Paths First off, the skill paths! I finally finished the underlying machinery, and now iximiuz Labs supports a new type of content - short roadmaps that you can use to develop or improve a specific skill: how to debug distroless containers, how to copy images from one repository to another,...
Hello friends! Ivan's here with another monthly roundup of all things Linux, Containers, Kubernetes, and Server Side ๐ง The issue's main topic is iximiuz Labs' largest-ever upgrade: Fresher and more streamlined look of the frontend UI ๐ A new 5.10 Linux kernel built with nftables support (finally, we can try out kube-proxy's nftables mode). New default playground user - laborant (yep, rootless containers learning for). New playgrounds: Ubuntu 24.04, Debian Trixie, Fedora, and Incus (yay! more...
Hello friends! Ivan's here with a slightly delayed July roundup of all things Linux, Containers, Kubernetes, and Server Side ๐ง What I was working on This month, I got nerd-sniped by cgroups. It all started when I ran into a pretty significant difference in how Docker and Kubernetes handle the OOM events. When you limit the memory usage of a multi-process Docker container, the OOM killer often terminates only one of the processes if the container runs out of memory. If this process is not the...