henge
My most interesting and intense technical work happened a log time ago now. The closest recent work I did was building a Proof of Concept Notification System for 500k users
I wanted to build and demonstrate some recent production grade experience - specifically around kubernetes and go .
My original plan was to build a simple feature flagging system. I had spent some time thinking about it and I had a good idea ow what I wanted to build.
It didn’t really connect with me though. It had some challenge, but I didn’t really feel the technical ability stretch.
I was also tinkering with some embedded projects - os-adjacent tinkering with a raspberry pi pico, and a temperature sensor with an esp32. I was really enjoying writing C code and I wondered if I could bring both of these together.
After some research, I identified an interesting niche gap. Configuration management across a fleet of embedded devices. While there are enterprise level platforms to support this, there was nothing at the hobbyist level - presumably because everyone just duct-taped something together.
I could get my go on, get some kubernetes experience, even if it isn’t strictly necessary at the hobbyist level, and more importantly, I could write a C reference client (at some point)
Minimum Capabilities
- It should be possible to add a device,
- Which would then be able to connect.
- It should be possible to create a schema with basic validation
- Set a value against a field for a device
- Which would push the config to the device
I only had a few hours a week I could dedicate to it, and I didn’t want it to take six months.
Build Stages
Stage 1 — The Platform
- Control plane in Go
- Push mechanism
- Minimal REST API: register devices, read/write config, check device state
- Go reference client: connects, receives pushes, logs what it applied
- Deployed on k8s cluster with proper configuration:
- resource limits,
- liveness/readiness probes,
- rolling deployment,
- persistent storage
- Public repo with README explaining engineering decisions
- Correct reconnection with backoff
- Done when: curl the API, change a value, Go client receives it within a second, all running live on GKE
Stage 2 — The C Client (optional)
- C client targeting Raspberry Pi (Linux, constrained environment)
- Same protocol as Go client — control plane unchanged
- Atomic config application (write to temp, verify, rename)
- No memory leaks
- Something observable on the hardware when config changes
- Done when: Raspberry Pi connected to live GKE service, config pushed, Pi applies it, control plane records acknowledgement
Stage 3 — Product layer (ongoing, optional)
- Rollout control (percentage-based, device groups)
- Config history and audit trail
- Simple web UI or keep API-first
- Hosted version with auth, rate limiting, uptime guarantees
- Terraform module for self-hosters
- Only pursued if Stage 1 and 2 are solid and there’s interest
Data Storage
Where to store data is a key consideration. In the early stages, we can use an in-memory database.
Once persistence becomes relevant, there are many options. For a single node, we can store the data locally. bbolt is a good candidate here.
Once we need multi-node setups, we need a separate database. With my postgresql experience, that is a strong, if heavy candidate. Redis is potentially another option, as is rqlite (which I’ve not used before)
I will leave this option to be decided later.
I also considered etcd, but I didn’t want to integrate something and use it in a way it was not intended without being abl e stress test it enough to have the confidence that it’ll work as advertised.