After I started the migration from the cloud to my own infra , I had more things to monitor, like the VPS which hosted my sites. I set up alerts on grafana and that works well enough to alert me, but alerting has multiple moving parts that could go wrong, and I didn’t want to rely on that fully.

There were also a few things I kept an eye on every day that lived across multiple websites:

• Traffic to my site
• Notifications across some platforms (e.g. GitHub)
• Any form submissions on tally.so

I also wanted to eliminate some windows that constantly took up screen real-estate. My music player, chat apps like fluxer and discord — I didn’t need the apps themselves, just a “currently playing” widget and unread counts.

I dedicated one monitor out of my three to a full overview of everything I wanted to keep an eye on. It would replace the apps already living there anyway.

I looked around, and there were tools that could do some of these. Grafana can do hosts, alerting, etc. Conky , eww and its ilk could do the music player, unread stats and host based stuff. They could be extended to do some of the additional bits I needed.

But what I really wanted was something that went across those boundaries and provided across the board, in some ways providing grafana like functionality in a conky like interface.

The former is infrastructure-out giving you an overview of your estate, while the latter is machine-in, as a lens into your system. Neither cared about things like the traffic to my site or form submissions on tally - at least, not without extra fiddling.

What I wanted was a display surface across everything. I considered shoehorning the features into one of those tools, but getting mpris into grafana felt much like trying to fit a round peg into a square hole. I might have more luck building some extra widgets for eww or conky, but that would involve yuck and c++ respectively. While I’ve worked with c++, I had not worked with yuck before, and neither felt like it would be fun for me.

I did not want to spend a lot of time writing code that I didn’t particularly enjoy to add functionality to software that didn’t quite fit what I actually wanted. There was also the possibility that I’d get knee deep in it before I found that it didn’t quite do what I needed, or I needed to patch it, and have to jump through hoops to get it in, or maintain a fork.

Thus mantel was born.

I started with a quick prototype that Claude built in go and it worked well enough. I wasn’t interested in building it enough to spend a lot of effort on it.

Over a few weeks, I got Claude to add a few extra bits. The plan was to eventually bring in some graphs from grafana by linking to them directly, which should bring enough observability into one dashboard. Getting to know grafana a bit better, I realised that some of the observability I wanted - like being able to have a status widget with its dependencies in the various possible states would involve a lot of learning grafana and I could see it going into UI component fiddling. One look at the panel with all the customisation options was enough to make me feel sleepy. I’d rather write a bunch of code any day.

But it wasn’t quite working for me. Claude did a good enough job for a prototype, but as I wanted more features, the rough edges started to show. I added a cache volume and to monitor that, I would have to run a script as root - fine for now, but it could get ugly fast. There was a perceptible lag in response. I’d start timewarrior from the console and it would take a few seconds before the dashboard updated. Same when I stopped it. Same with the CPU usage - not a serious problem, but annoying.

I got claude to refactor some of the code to make it easier to extend, and it worked - well enough. It didn’t have tests and if I wanted to extend stuff, I’d really have to rely on Claude.

I am familiar with go, and like it well enough - I do a lot of work in it, but I don’t really program in go for pleasure anymore.

I considered getting to know the code - there wasn’t a lot, but it did not fill me with enthusiasm. Making the whole thing more responsive would involve as much effort as a ground up rewrite, partly because there wasn’t much, and what was there was built around polling. Since it was working well enough, I tried to put it off.

As the annoyances started mounting, and the desire for more things showing up on the dashboard increased, it finally tipped over into a decision to build it “properly.”

I seriously considered owning the codebase, and wiring in SSE, or rebuilding parts of it to support SSE, but I struggled to muster the enthusiasm to write another tool in go and then to maintain it.

The Stack

I decided to use zig for the new version, mainly because I love it. It’s 0.16 and likely a long way from a 1.0 release. The maintenance burden is immediately higher because it gets a lot of breaking changes. There are fewer libraries, and the ones that are there are less well tested.

There is a lot more risk associated with zig. However, I love working with it, and I’d rather spend far more time writing something in zig - in some ways it’s a feature.

Still, I checked to make sure that zig @ 0.16 has enough of the features to allow me to build it without too many obstacles. Zig doesn’t support SSE out of the box, but there is a library http.zig which supports it. Testing it is a bit tricky because zig doesn’t yet support non-blocking i/o or timeouts, but that’s another post.

The rest of what we need is mainly the ability to call other executables, network calls and async, all of which are available, sometimes just enough - but enough.

For the UI design, there are two main problems to address - I am not good at it, and I don’t enjoy working on it. To make it easier on myself, two things that help are live-reload and Claude.

I considered DVUI which I’d used before. The main problem with it is that there isn’t a tight build loop. I would have to build and run each time, slowing it down. DVUI is also niche enough that Claude is unlikely to be able to help me theme it.

Most UI toolkits tend to design for applications, and designing a dashboard that is information dense and with a specific synthwave/retro aesthetic I have in mind (to match the rest of my window manager theme) will involve a lot of tweaking or going straight to rendering as graphics.

I would have built using a TUI, which would still be a retro aesthetic I would enjoy, but poor image support discounts it even before the absence of live reload.

Web on the other hand, would do live reload on code change pretty easily. Flutter would too, but I’d used it before and felt like a level of unnecessary complexity.

For the frontend, web is the easiest interface to work with. Vanilla Javascript, and lit were considered, but I’d get annoyed with the html hiding inside quotes.

htmx would be cool, but I’d be building it with div tags, while I liked the neatness of component html tags. It also did not have live reload out of the box.

Svelte is relatively simple and doesn’t need html to be in quotes, and has live reload. I have also played with it a little bit in the past, and enjoyed the experience. It does however add a whole set of additional tooling requirements, not to mention (p)npm.

I decided to give svelte a try first, primarily for the tight live reload. If it ends up taking more time than is saved by the live-reload, I’ll switch to htmx as the cleaner option.

Once I have a decent handle on the svelte version, it might even make sense to rewrite it in htmx to streamline the maintenance burden.

Concurrency Model

The makeshift version of mantel polls, which is good enough, but also annoying, particularly with timewarrior and mpris, as mentioned earlier. Server Side Events would be a better candidate for quicker updates. We don’t need bidirectional communication, so WebSockets would be overkill.

At some point, we might want to send messages back to the server from the dashboard, but they are likely to be few and far between. As such specific endpoints would make more sense than full blown WebSockets.

In terms of collecting the data, the plan is to hook into interrupts wherever possible. For example, I’d want to use inotify to watch the timewarrior files. We’d then know exactly when the time starts or stops, and we can push an event to update.

For MPRIS, we could listen to the dbus message bus for signals.

Some things, we’d still need to poll, like the cpu, but we can do it more often and send updates.

On a side note, I don’t want to be running it as root for obvious reasons. However, we’ll still need data that is accessible only as root. In the prototype, I used sudo restricted to running specific scripts that output the data. With the new version, I’ll get prometheus to pick up the data and mantel can query that. It lets mantel run entirely as the current user. Running as the current user is also important for things like MPRIS and timewarrior.

I’ve used SSE before, with henge in go, and at a contract in 2024 in java.

The idea is that we’d have a few generic events:

• string: single string. e.g. hostname, label
• number: single numeric value. e.g. inbox count
• numbers: array of numbers (CPU cores, memory breakdown)
• state: enum of up | down | warn | err | unknown. e.g. host
• percentage: { value:number, max: number}. e.g. mpris track position, total cpu/mem usage

If a collector needs to send information that is tightly bundled together, we’d create event types for them, like for mpris and timewarrior

• mpris: including title, artist, album, progress, media link etc.
• timew: including timer status, tags, last timer tags, link to start it etc.

Keeping it simple would also make it easier to migrate to htmx (or something else) later.

Progress

I’ve already come across a rough edge of zig, writing a test for SSE when everything blocks, but I enjoyed working around it .

The more I think about it, the more it feels like htmx might be a better fit - mainly because it’s small and doesn’t have a ton of dependencies. Once I have a collector working, I’ll try both out before I decide the way forward.