Hello friends!

It's Ivan again, with a monthly roundup from iximiuz.com.

Busy times! A lot of learning and a lot of work. Spent the better part of the month wearing my SRE hat - serverless (as in FaaS) observability is still hard. The last time I tried my luck with AWS Lambda was in 2017-ish. We ended up with some ten(s of) rather ad-hoc and small functions, and it quickly became painful to monitor them. In 2022 this problem seems to be solved - there is AWS X-Ray and also plenty of 3rd-party offerings allowing you to auto-instrument your functions and generate nicely-looking dashboards... But it all breaks apart as soon as you get a slightly less trivial routing logic. Observing something more granular than a single function invocation is still hard. OpenTelemetry in general and ADOT in particular give some hope, but integrating them with AWS Lambda remains tricky. Curious to hear your thoughts. How do you collect metrics on the level of a route if your lambda function serves a full-blown REST API?

Switching gears, there's some exciting news on the educational front! I've been talking about converting my blog into some sort of a learning platform for quite a while... But I haven't been just talking! And after almost a year of (sporadic) work, I'm ready to present a thing that will eventually become the foundation of that platform. Intrigued? Then read on!

SPONSORED SSH Certificates: How Do OpenSSH Certificates Compare to X.509? I knew they are not entirely the same even though they share the main principles, but the actual difference remained a mystery for me. Until I stumbled upon this condensed read by Teleport. Good stuff for infra folks, go check it out!

​

What I Was Working On

As you might have noticed, I'm a visual learner. Otherwise, why there would be so many diagrams on my blog? 🙈 But I also believe that the hands-on experience matters. It's not enough to just read about a tool, or a trick, or a language, even if the text is well-sprinkled with drawings. You have to get your hands dirty and try it out - it'll deepen the understanding and make the knowledge more durable.

So, how to combine text, visuals, and interactivity in one place?

Right, build a Web-based Kubernetes dashboard! But fear not, I'm not trying to rewrite Tanzu's Octant 😇 And it's not a replacement for the explanatory text or static diagrams either. Instead, the dashboard is meant to be a complementary mechanism augmenting the existing and future blog materials.

Here is a (very early) example - an attempt to show what happens to a Deployment when one of the Pods gets deleted:

Having such a tool embedded in a blog page should facilitate the learning experience. Instead of just trusting me that a Kubernetes controller consists of a Deployment with a single leader Pod running on behalf of a Service Account and watching instances of a Custom Resource, the reader should be able to see it with their own eyes by just deploying the Kubebuilder's example project!

Here is another example - an attempt to visualize a Deployment receiving a Pod Template spec update:

For those of us dealing with and/or writing Kubernetes controllers, a solid understanding of their behavior is a must. The above example illustrates the behavior of some of the built-in controllers, but similarly, a custom controller can be visualized. Now, imagine how breathtakingly it should be to watch a chaos monkey controller harvesting Pods in the wild 🐒

Another important thing that can be studied with such a tool is how to work with the Kubernetes API. There are many ways to interact with Kubernetes resources - they can be updated, patched (using different strategies), or a change can be applied to them. The mighty kubectl tool often shields you from these details, and it's good and bad at the same time. One of my design goals while working on the Kubernetes UI is to keep all the interactions with the API server transparent. So, it should be possible to use it like a Postman's REST client but for Kubernetes API.

Another pet peeve of mine while developing applications (or controllers) for Kubernetes is the lack of the tools to quickly create a persistent view of the cluster (or multiple clusters) showing me updates to the relevant resources in real-time. For instance, when I'm working on a controller, I'd typically want to periodically check the states of the manager's deployment and pod(s), refer to the corresponding service account, and review roles and role bindings. So far, I've been doing it with ad-hoc kubectl queries or (if the project is long enough) by hacking together some nasty scripts harnessing kubectl with fancy selectors and the Linux watch tool. But having a page showing me (potentially interconnected and periodically refreshed) objects corresponding to a number of selectors would save me from a lot of pain. And probably for a multi-cluster/multi-version application leveraging a service mesh to glue things together it'd be even more useful.

The tool I've been working on is in a pre-alpha state, so many of the above things still exist only as rough ideas. But I can already use the dashboard to record some small demos. Hopefully, next month I'll be able to apply it for something more serious - like going through the Kubebuilder book and visualizing the progress.

The tool is written in Go (surprise surprise) with a Vue.js frontend, and I have all the plans to make it open-source one day. It's designed to work as a standalone desktop application and as a client-server deployment (with the server part having access to the Kubernetes clusters). And I'm really curious to hear your feedback!

​

What I Was Writing

Didn't have much time for writing this month, but I hope to catch up in the coming months - reinforced with the new tool 😊 Here is a short (but apparently well-received) article I published at the beginning of June - How To Start Programming In Go: Advice For Fellow DevOps Engineers. As always, based on personal experience, so YMMV.

​

What I Was Reading

• ​Notes on running containers with bubblewrap - Julia Evans on launching "containers" without Docker (or Podman). Do recommend, it might help close some containerization knowledge gaps!
• ​Getting started with ko: A fast container image builder for your Go applications - this article finally helped me to understand ko - oversimplifying, it's go build + COPY to a rootless image. Might be a boon for some Go developers, but I'd recommend learning (about) containers instead. It's a cool tech, don't try to hide it from people!
• ​Debugging containerd - good hands-on read for Kubernetes cluster admins facing some issues caused by the recent DockerShim deprecation.
• ​The End of Localhost - The prediction makes sense for some types of work (like frontend or simple backend), while for other types of work (like systems programming/infra development), it sounds too far-fetched. However, it brings up another very important point - development environments should be treated like cattle, not pets. And they also should be isolated well - installing dev packages on the host system is a security nightmare!
• ​Are Platform Teams Dead? - Platform teams are dead. Long live platform teams! The term "platform" is pretty vague. Indeed, probably platform teams will be moving higher up the stack. But they won't be extinct. And abstracting the cloud was always a bad idea. It's like writing wrappers for every new dependency - we don't do it often nowadays, but it used to be a popular (anti-)pattern when the number of packages a typical project would use was still sane.
• ​Every Sufficiently Advanced Configuration Language is Wrong - Can't agree more!
• ​Don't Use Kubernetes, Yet - Can't agree more again!
• ​How fast are Linux pipes anyway? - I enjoyed the insights into the Linux pipe internals more than the performance optimization itself. A nice deep read!
• ​kubeshop/testkube - How to e2e test applications running in a cluster the Cloud Native way. Particularly enjoyed the README file - it clearly explains the What and the How, with no marketing buzz.
• ​Learn Makefiles With the tastiest examples - Makefiles are everywhere in DevOps, so it's definitely worth learning about make. This resource starts from quite a one-sided introduction to make (as a build tool for C++ projects), but the rest is full of useful information. Also, looking at make from the angle of a build (i.e., file manipulation) tool could explain many of its quirks.
• ​nivenly.com - I'm a long-time Kris Nóva's follower, but somehow this site dodged my attention. This is one piece of art!
• ​Oh My Git! - An open source game to learn Git by visualizing the internal structures of Git repositories in realtime. I definitely share the mindset with the creators of this tool. That's how the tech education should look like ❤️

​

Stay Tuned

Well, this is it for June. Hope you enjoyed the month as much as I have :)

Oh, totally forgot! That Kubernetes UI I've been working on, I don't have a good name for it yet. If someone has an idea, I'd love to hear it!

Stay safe and healthy!

Cheers,

Ivan Velichko
​