I’m posting this so I have something to link to the next time external monitoring comes up.
A bunch of us run a fully blown home lab in addition to openHAB. And I openHAB makes a relatively poor monitoring and alerting system. It’s made for home automation, not IT network monitoring.
There are a whole lot of potential options for monitoring the health and status of one’s network infrastructure. Graylog (log aggregation), ELK Stack, Prometheus with Graphana, Zabbix, etc. I personally used to run Zabbix, and it has helped me in the past recognize and solve problems.
However, recently I upgraded Zabbix and my database got corrupted … again. Zabbix is also super heavy weight and has way more features than I care about. I mainly just want to keep up with the usual stats: CPU, RAM, SWAP, network, IO, file system utilization, Docker container status, temperature.
A new self-hosted service came across my news feed the other day called Beszel. Compared to most of the rest of the monitoring tools mentioned above, Beszel is refreshingly simple to install and configure.
I’m not going to post a full tutorial here because their docs are good enough. But essentially, like these other services, one sets up a “hub” which is the central server which aggregates and displays the statistics and each machine you run it on has an agent which feeds the hub using websockets, Unix sockets (for the agent running on the same machine where the hub is running) and the whole architecture is secured using ssh (i.e. the agents ssh to the hub and are authenticated using ssh keys). The whole ssh key thing does make connecting the agents to the hub a little bit awkward. You need to pass a token and the SSH key on the command to install OH and the hub needs to be in a special mode to accept the connection from the new agent. This makes it hard to automate. But the agents self update which makes maintenance long term easier.
Installation details can be found at https://www.beszel.dev/. Installation went very smoothly for me.
Some lessons learned:
- the docker containers must run as root with host networking to access all the stats necessary
- if you get “fingerprint mismatch” in an agent’s logs and try to fix it by reinstalling, you must restart the service to pick up the new token and key
- to delete a host click on the user menu and systems. Deleting the fingerprint from the settings menu does not remove the host
I got the whole thing set up in under an hour. Some screen shots:
You can set up alerts through lots of different services. Unfortunately you cannot set up an alert for when a specific Docker container goes down. But given the amount of effort spent to set this up that’s a minor missing feature.