- Easy and Hard Tech Startups Startup
Inkmi is Dream Jobs for CTOs and written as a decoupled monolith in Go, HTMX, Alpinejs, NATS.io and Postgres. I document my adventures and challenges in writing the application here on this blog, tune in again.
I’m building Inkmi - Dream Jobs for CTOsas a solo entrepreneur. So time is precious, and I don’t have devops people looking into things.
To make Inkmi less error-prone and less work to operate, I looked for ways to make it self-healing. There are many practices out there and I have adopted some:
- Systemd listeners
- Restart every 24h
- Check yourself if you are working, otherwise stop
- Send notifications to a monitor
- Check if your binary has changed
Inkmi is written in Go and deployed as one binary with embedded assists. I use systemd
to start and monitor the binary (reduce privileges, manage memory).
Systemd listeners
Systemd does TCP networking and keeps connections working. This queues connections outside the binary and helps with spikes. When the binary restarts, Systemd keeps the connections and hands them over to the newly started Inkmi. This way customers don’t see connections interrupted or ended.
With the systemd
API for Go and Echo this looks like
listeners, err := activation.Listeners()
// IP address we're listening to
_ = listeners[0].Addr().String()
// set listeners for Echo to use
e.Listener = listeners[0]
24h Restart
Something from my Windows NT days and that I laughed about back then, Systemd restarts the binary every 24h. Should there be a memory or other problem, it will go away. One needs to check graphs in e.g. Grafana though, otherwise you will miss a problem that needs to be fixed in code.
Check if you’re still working
Inkmi tries a /health
URL every minute to check if it still responds. If it doesn’t respond to itself, it exists. Systemd then will restart the binary. With Systemd keeping TCP connections, customers will not see an impact. As Inkmi is written in Go, restarts are very fast, e.g. compared to JVM language restarts.
Send notifications
Inkmi sends a notification to Systemd every few seconds. If Systemd stops receiving those, it assumes the binary hangs and restarts the binary.
Telling systemd
we’re ready to receive connections:
_, err := daemon.SdNotify(false, daemon.SdNotifyReady)
Telling systemd
we’re ok:
_, _ = daemon.SdNotify(false, daemon.SdNotifyWatchdog)
Check for a changed binary
The Go Inkmi binary watches itself for changes. When the binary changes, Inkmi exists. Systemd then will restart Inkmi with the new binary. This is how releases have no impact on customers.
watcher, err := fsnotify.NewWatcher()
...
_ = watcher.Add(os.Args[0])
<-watcher.Events
...
Example systemd
configuration
The systemd
service file looks something like this, then you need the socket file.
[Unit]
Description=MyService
Requires=myservice.socket
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/myservice
WatchdogSec=30s
Restart=always
RuntimeMaxSec=1d
ProtectSystem=strict
ProtectHome=true
PrivateUsers=true
PrivateTmp=true
DynamicUser=yes
[Install]
WantedBy=multi-user.target
Results
With this in place, the Inkmi binary has been very stable and I have less work tending the application of my startup. In my former startups I didn’t have a self healing setup and systems needed much more attention.
About Inkmi
Inkmi is a website with Dream Jobs for CTOs. We're on a mission to transform the industry to create more dream jobs for CTOs. If you're a seasoned CTO looking for a new job, or a senior developer ready for your first CTO calling, head over to https://www.inkmi.com
- Easy and Hard Tech Startups Startup