Yep. But it’s a project in an of itself and not at all light weight. But it’s been worth the effort for me. I discovered one of my RPis was silently rebooting itself every couple of hours without my knowledge. Turned out it needed a better power supply.
It also showed me that about every couple of hours one of my VMs was hitting a sustained load of 8-10 about twice a day. I was able to narrow that problem down to crappy default PostgreSQL settings. Of course this was a case where everything was fine until I installed Zabbix and PostgreSQL started to get pounded a lot more. That helped bring the load down to a sustained 2 with spikes to 4-5 so I got a new RPi 4 and moved a few services (vaultwarden, heimdall, and guacamole) off of the VM to the RPI 4. I also move my virtual desktop to the new RPi 4 and now everything is much happier with loads under 1 with spikes to 2.
It’s still telling me that my HAProxy on my pfSense firewall is restarting sporadically and for no apparent reason which I need to track down. I’ve also some weird inefficiencies on my network
I’m able to set triggers that generate an email (can be just about any kind of notification like Telegram or SMS) so I’m immediately informed when something goes wrong. I have it running on all my VMs and all my RPis. Pretty much everything that is always running.
If it helps you or others, here is the Ansible playbook I wrote to set this up.
---
# tasks file for roles/zabbix
- name: Create the zabbix user and group
include_role:
name: create-user
vars:
uid: "{{ zabbix_uid }}"
gid: "{{ zabbix_gid }}"
user_name: zabbix
create_home: False
service: zabbix
- name: Check if docker group exists
shell: /usr/bin/getent group | awk -F":" '{print $1}'
register: etc_groups
changed_when: False
- name: Add secondary Groups to zabbix user
user:
name: zabbix
groups: docker
append: yes
become: true
when: '"docker" in etc_groups.stdout_lines'
- block:
- name: Create the folders for logging and settings
file:
path: "{{ item }}"
state: directory
owner: zabbix
group: zabbix
mode: u=rwx,g=rwx,o=rx
recurse: yes
loop:
- "{{ zabbix_home }}"
- "{{ zabbix_home }}/alertscripts"
- "{{ zabbix_home }}/externalscripts"
- "{{ zabbix_home }}/modules"
- "{{ zabbix_home }}/enc"
- "{{ zabbix_home }}/ssh_keys"
- "{{ zabbix_home }}/ssl/certs"
- "{{ zabbix_home }}/ssl/keys"
- "{{ zabbix_home }}/ssl/ssl_ca"
- "{{ zabbix_home }}/snmptraps"
- "{{ zabbix_home }}/mibs"
- "{{ zabbix_home }}/export"
become: true
# Create database and user
- name: Install psycopg2
pip:
name: psycopg2-binary
become: True
- name: Create postgres database for Zabbix
postgresql_db:
login_host: "{{ postgresql_host }}"
login_password: "{{ postgresql_password }}"
login_user: "{{ postgresql_user }}"
name: "{{ zabbix_db_name }}"
- name: Create zabbix user for zabbix database
postgresql_user:
db: "{{ zabbix_db_name }}"
login_host: "{{ postgresql_host }}"
login_password: "{{ postgresql_password }}"
login_user: "{{ postgresql_user }}"
name: "{{ zabbix_db_user }}"
password: "{{ zabbix_db_password }}"
priv: ALL
# Server: config variables https://hub.docker.com/r/zabbix/zabbix-server-pgsql/
- name: Pull/update the zabbix server docker image
docker_container:
detach: True
env:
DB_SERVER_HOST: "{{ postgresql_ip }}"
POSTGRES_USER: "{{ zabbix_db_user }}"
POSTGRES_PASSWORD: "{{ zabbix_db_password }}"
POSTGRES_DB: "{{ zabbix_db_name }}"
ZBX_STARTVMWARECOLLECTORS: "5"
ZBX_STARTDISCOVERERS: "5"
ZBX_HISTORYCACHESIZE: "128M"
ZBX_HISTORYINDEXCACHESIZE: "4M"
ZBX_HOUSEKEEPINGFREQUENCY: "1"
ZBX_MAXHOUSEKEEPERDELETE: "5000"
# ZBX_LOADMODULE: dummy1.so,dummy2.so # modules are in /var/lib/zabbix/modules
# ZBX_DEBUGLEVEL: 3 # 0-5, 5 is TRACE
# ZBX_TIMEOUT: 4 # timeout for processing checks
# ZBX_JAVAGATEWAY_ENABLE: false
hostname: "{{ ansible_fqdn }}"
image: zabbix/zabbix-server-pgsql:latest
log_driver: "{{ docker_log_driver }}"
name: zabbix-server
published_ports:
- "10051:10051"
pull: True
restart: False
restart_policy: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- "{{ zabbix_home }}/alertscripts:/usr/lib/zabbix/alertscripts"
- "{{ zabbix_home }}/externalscripts:/usr/lib/zabbix/externalscripts"
- "{{ zabbix_home }}/modules:/var/lib/zabbix/modules"
- "{{ zabbix_home }}/enc:/var/lib/zabbix/enc"
- "{{ zabbix_home }}/ssh_keys:/var/lib/zabbix/ssh_keys"
- "{{ zabbix_home }}/ssl/certs:/var/lib/zabbix/ssl/certs"
- "{{ zabbix_home }}/ssl/keys:/var/lib/zabbix/ssl/keys"
- "{{ zabbix_home }}/ssl/ssl_ca:/var/lib/zabbix/ssl/ssl_ca"
- "{{ zabbix_home }}/snmptraps:/var/lib/zabbix/snmptraps"
- "{{ zabbix_home }}/mibs:/var/lib/zabbix/mibs"
- "{{ zabbix_home }}/export:/var/lib/zabbix/export"
# Web interface
- name: Pull/update the zabbix web docker image
docker_container:
detach: True
env:
ZBX_SERVER_HOST: "{{ zabbix_server_ip }}"
DB_SERVER_HOST: "{{ postgresql_ip }}"
POSTGRES_USER: "{{ zabbix_db_user }}"
POSTGRES_PASSWORD: "{{ zabbix_db_password }}"
POSTGRES_DB: "{{ zabbix_db_name }}"
# ZBX_HISTORYSTORAGEURL: elasticsearch config
# ZBX_HISTORYSTORAGETYPES: elasticsearch config
PHP_TZ: America/Denver
ZBX_SERVER_NAME: Zabbix
hostname: "{{ ansible_fqdn }}"
image: zabbix/zabbix-web-nginx-pgsql:latest
log_driver: "{{ docker_log_driver }}"
name: zabbix-web
published_ports:
- "9090:8080"
pull: True
restart: False
restart_policy: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
when: zabbix_server
# TODO Make this more dynamic, see if there is a way to look up the latest release number on GitHub or something
- name: Add zabbix repo ubuntu
apt:
deb: https://repo.zabbix.com/zabbix/5.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.4-1%2Bubuntu20.04_all.deb
when: (not 'pis' in group_names) and (not 'muninn' in inventory_hostname)
become: yes
- name: Add zabbix repo rpi
apt:
deb: https://repo.zabbix.com/zabbix/5.4/raspbian/pool/main/z/zabbix-release/zabbix-release_5.4-1%2Bdebian10_all.deb
when: ('pis' in group_names) or ('muninn' in inventory_hostname)
become: yes
- name : Install the zabbix-agent2
apt:
name: zabbix-agent2
update_cache: yes
become: yes
- name: Install the config
template:
dest: /etc/zabbix/zabbix_agent2.d/custom-zabbix.conf
mode: u=rw,g=r,o=r
src: custom-zabbix.j2
force: true
become: True
notify:
- Restart the zabbix agent to pick up config
# this task is safe because of the validate
- name: Allow zabbix to have passwordless sudo
lineinfile:
dest: /etc/sudoers
state: present
regexp: '^zabbix'
line: 'zabbix ALL=(ALL) NOPASSWD: ALL'
validate: 'visudo -cf %s'
become: True
- name: Add cron job to create log directory on reboot for RPis
block:
- name: Add the cron shell variable
cron:
name: SHELL
env: true
job: /bin/bash
- name: Add cron job to create the directory at boot
cron:
name: "Zabbix log directory"
job: mkdir /var/log/zabbix && chown zabbix:zabbix /var/log/zabbix
special_time: reboot
become: True
when: "'pis' in group_names"
This installs the server stuff if zabbix_server is true and the agent stuff if zabbix_server is false so this same role can be used on all the machines. It also needs to add a different apt repo for the RPis as it does for my Ubuntu VMs so zabbix-agent2 can be installed which is required for good Docker integration. Also, on my RPis I put my logs into a RAM disk so I need to create and set the permissions on the directory Zabbix logs to before Zabbix starts which I do with a cron job that runs at boot.
My custom-zabbix.j2 file is:
Hostname={{ ansible_hostname }}
LogFileSize=1024
Server={{ zabbix_conn_str }}
ServerActive={{ zabbix_conn_str }}
The handler is
---
# handlers file for roles/zabbix
- name: Restart the zabbix agent to pick up config
systemd:
name: zabbix-agent2
state: restarted
become: True
Everything in {{ }} is a variable that will need to be set (except for the ones that start with “ansible” which gets set for you. See Ansible Revisited for more on using Ansible.
Using the role to install the server:
---
# Prerequisites:
# - install ssh
# - set up ssh for certificate only logins
# - install python3
# - all hosts added to the inventory under [homeauto]
# TODO set up root with ssh certs for git interactions
- hosts: homeauto
vars:
- update: False
roles:
- common
- { role: vm, when: not update }
- firemotd
- mount-data
- nut-client
- msmtp
- fail2ban
- { role: multitail, openhab_logs: '/srv/openhab/userdata/logs/' }
- { role: docker, when: not update }
- role: zabbix
vars:
- zabbix_server: True
- openzwave
- mosquitto
- openhab
- docker-prune
- tripwire
Example showing using the role to install the agent.
- hosts: media
vars:
- update: False
roles:
- common
- { role: vm, when: not update }
- firemotd
- mount-data
- nut-client
- msmtp
- fail2ban
- { role: multitail, openhab_logs: '/srv/openhab2/userdata/logs/' }
- { role: docker, when not update }
- gitlab
- plex
- { role: calibre, calibre_type: 'docker' }
- postgresql
- redis
- elasticsearch
- nextcloud
- { role: zabbix, zabbix_server: False }
- docker-prune
- tripwire
Note the above shows two different ways to set variables when including a role into a playbook.