Memory usage (docker) 2.5.9 pushing 1.5-1.6gb

I’ve had to set a docker memory limit of 1.7gb for my openHAB instance (and about 1.2 for Java)

It seems to peg and get restarted when I add an item or sometimes a rule change.

Just wondering what other people’s memory usage is.

I have a fairly big system with a fair number of items, if someone has an easy way I’ll be able to tell you how many.

Edit: interestingly it seems to have dropped from 1.6 earlier today to 900mb. Still high but a fairly amount of head room now.

There have been a couple of threads about the apparent large size of Docker containers. Mine seems to sit at around 1.2 BG with 1.3 GB max over the past day. It’s pretty stable but I’m not doing much of anything with OH right now beyond letting it run these days. Looking over the past month it looks like It’s been around 2 GB at times.

I don’t know the causes for it growing and shrinking. I run the SNAPSHOTS and upgrade a couple times a week so there are caused by restarting OH dues to the upgrades or or something else. It’s running on a VM running on an old (now Intel i3 processor). The host has 32 GB ram but the VM OH is running on only has 4 GB.

As an aside, I’ve learned a whole lot about my system after having installed and configured Zabbix. It was well worth the effort to install and set up.

Anyway, what you describe sounds more like a bug or memory leak or the like. I don’t remember there being anything reported in the 2.5.9 timeframe so it’s most likely a binding but could be almost anything.

Hi thanks…is zabbix where you get those pretty stats from?
I want to do something like that…but I’m currently weary to install antyhing else on my host, it’s only an old gigabyte brix a6 with 4 gb of ram…and its pretty close to full.

But I’m running a container docker2mqtt that exposes running containers to mqtt for monitoring…and I’ve been altering it over this past 24 hours (because I don’t have a long running history of contianer cpu/memory) to pull from docker stats command into mqtt and therefore into influx…(of course I could just telegraf it and be done with it)

Mine seems to have retreated from original start up values…and longer term I will look at a new mini pc, but with 8gb ram.

Yep. But it’s a project in an of itself and not at all light weight. But it’s been worth the effort for me. I discovered one of my RPis was silently rebooting itself every couple of hours without my knowledge. Turned out it needed a better power supply.

It also showed me that about every couple of hours one of my VMs was hitting a sustained load of 8-10 about twice a day. I was able to narrow that problem down to crappy default PostgreSQL settings. Of course this was a case where everything was fine until I installed Zabbix and PostgreSQL started to get pounded a lot more. That helped bring the load down to a sustained 2 with spikes to 4-5 so I got a new RPi 4 and moved a few services (vaultwarden, heimdall, and guacamole) off of the VM to the RPI 4. I also move my virtual desktop to the new RPi 4 and now everything is much happier with loads under 1 with spikes to 2.

It’s still telling me that my HAProxy on my pfSense firewall is restarting sporadically and for no apparent reason which I need to track down. I’ve also some weird inefficiencies on my network

I’m able to set triggers that generate an email (can be just about any kind of notification like Telegram or SMS) so I’m immediately informed when something goes wrong. I have it running on all my VMs and all my RPis. Pretty much everything that is always running.

If it helps you or others, here is the Ansible playbook I wrote to set this up.

---
# tasks file for roles/zabbix

- name: Create the zabbix user and group
  include_role:
    name: create-user
  vars:
    uid: "{{ zabbix_uid }}"
    gid: "{{ zabbix_gid }}"
    user_name: zabbix
    create_home: False
    service: zabbix

- name: Check if docker group exists 
  shell: /usr/bin/getent group | awk -F":" '{print $1}'
  register: etc_groups
  changed_when: False

- name: Add secondary Groups to zabbix user
  user:
    name: zabbix
    groups: docker
    append: yes
  become: true
  when: '"docker" in etc_groups.stdout_lines'

- block:
  - name: Create the folders for logging and settings
    file:
      path: "{{ item }}"
      state: directory
      owner: zabbix
      group: zabbix
      mode: u=rwx,g=rwx,o=rx
      recurse: yes
    loop:
      - "{{ zabbix_home }}"
      - "{{ zabbix_home }}/alertscripts"
      - "{{ zabbix_home }}/externalscripts"
      - "{{ zabbix_home }}/modules"
      - "{{ zabbix_home }}/enc"
      - "{{ zabbix_home }}/ssh_keys"
      - "{{ zabbix_home }}/ssl/certs"
      - "{{ zabbix_home }}/ssl/keys"
      - "{{ zabbix_home }}/ssl/ssl_ca"
      - "{{ zabbix_home }}/snmptraps"
      - "{{ zabbix_home }}/mibs"
      - "{{ zabbix_home }}/export"
    become: true

  # Create database and user
  - name: Install psycopg2
    pip:
      name: psycopg2-binary
    become: True

  - name: Create postgres database for Zabbix
    postgresql_db:
      login_host: "{{ postgresql_host }}"
      login_password: "{{ postgresql_password }}"
      login_user: "{{ postgresql_user }}"
      name: "{{ zabbix_db_name }}"

  - name: Create zabbix user for zabbix database
    postgresql_user:
      db: "{{ zabbix_db_name }}"
      login_host: "{{ postgresql_host }}"
      login_password: "{{ postgresql_password }}"
      login_user: "{{ postgresql_user }}"
      name: "{{ zabbix_db_user }}"
      password: "{{ zabbix_db_password }}"
      priv: ALL

  # Server: config variables https://hub.docker.com/r/zabbix/zabbix-server-pgsql/
  - name: Pull/update the zabbix server docker image
    docker_container:
      detach: True
      env:
        DB_SERVER_HOST: "{{ postgresql_ip }}"
        POSTGRES_USER: "{{ zabbix_db_user }}"
        POSTGRES_PASSWORD: "{{ zabbix_db_password }}"
        POSTGRES_DB: "{{ zabbix_db_name }}"
        ZBX_STARTVMWARECOLLECTORS: "5"
        ZBX_STARTDISCOVERERS: "5"
        ZBX_HISTORYCACHESIZE: "128M"
        ZBX_HISTORYINDEXCACHESIZE: "4M"
        ZBX_HOUSEKEEPINGFREQUENCY: "1"
        ZBX_MAXHOUSEKEEPERDELETE: "5000"
  #      ZBX_LOADMODULE: dummy1.so,dummy2.so # modules are in /var/lib/zabbix/modules
  #      ZBX_DEBUGLEVEL: 3 # 0-5, 5 is TRACE
  #      ZBX_TIMEOUT: 4 # timeout for processing checks
  #      ZBX_JAVAGATEWAY_ENABLE: false
      hostname: "{{ ansible_fqdn }}"
      image: zabbix/zabbix-server-pgsql:latest
      log_driver: "{{ docker_log_driver }}"
      name: zabbix-server
      published_ports:
        - "10051:10051"
      pull: True
      restart: False
      restart_policy: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - /etc/timezone:/etc/timezone:ro
        - "{{ zabbix_home }}/alertscripts:/usr/lib/zabbix/alertscripts"
        - "{{ zabbix_home }}/externalscripts:/usr/lib/zabbix/externalscripts"
        - "{{ zabbix_home }}/modules:/var/lib/zabbix/modules"
        - "{{ zabbix_home }}/enc:/var/lib/zabbix/enc"
        - "{{ zabbix_home }}/ssh_keys:/var/lib/zabbix/ssh_keys"
        - "{{ zabbix_home }}/ssl/certs:/var/lib/zabbix/ssl/certs"
        - "{{ zabbix_home }}/ssl/keys:/var/lib/zabbix/ssl/keys"
        - "{{ zabbix_home }}/ssl/ssl_ca:/var/lib/zabbix/ssl/ssl_ca"
        - "{{ zabbix_home }}/snmptraps:/var/lib/zabbix/snmptraps"
        - "{{ zabbix_home }}/mibs:/var/lib/zabbix/mibs"
        - "{{ zabbix_home }}/export:/var/lib/zabbix/export"

  # Web interface
  - name: Pull/update the zabbix web docker image
    docker_container:
      detach: True
      env:
        ZBX_SERVER_HOST: "{{ zabbix_server_ip }}"
        DB_SERVER_HOST: "{{ postgresql_ip }}"
        POSTGRES_USER: "{{ zabbix_db_user }}"
        POSTGRES_PASSWORD: "{{ zabbix_db_password }}"
        POSTGRES_DB: "{{ zabbix_db_name }}"
  #      ZBX_HISTORYSTORAGEURL: elasticsearch config
  #      ZBX_HISTORYSTORAGETYPES: elasticsearch config
        PHP_TZ: America/Denver
        ZBX_SERVER_NAME: Zabbix
      hostname: "{{ ansible_fqdn }}"
      image: zabbix/zabbix-web-nginx-pgsql:latest
      log_driver: "{{ docker_log_driver }}"
      name: zabbix-web
      published_ports:
        - "9090:8080"
      pull: True
      restart: False
      restart_policy: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - /etc/timezone:/etc/timezone:ro
  when: zabbix_server

# TODO Make this more dynamic, see if there is a way to look up the latest release number on GitHub or something
- name: Add zabbix repo ubuntu
  apt:
    deb: https://repo.zabbix.com/zabbix/5.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.4-1%2Bubuntu20.04_all.deb
  when: (not 'pis' in group_names) and (not 'muninn' in inventory_hostname)
  become: yes

- name: Add zabbix repo rpi
  apt:
    deb: https://repo.zabbix.com/zabbix/5.4/raspbian/pool/main/z/zabbix-release/zabbix-release_5.4-1%2Bdebian10_all.deb
  when: ('pis' in group_names) or ('muninn' in inventory_hostname)
  become: yes

- name : Install the zabbix-agent2
  apt:
    name: zabbix-agent2
    update_cache: yes
  become: yes

- name: Install the config
  template:
    dest: /etc/zabbix/zabbix_agent2.d/custom-zabbix.conf
    mode: u=rw,g=r,o=r
    src: custom-zabbix.j2
    force: true
  become: True
  notify:
    - Restart the zabbix agent to pick up config

# this task is safe because of the validate
- name: Allow zabbix to have passwordless sudo
  lineinfile:
    dest: /etc/sudoers
    state: present
    regexp: '^zabbix'
    line: 'zabbix ALL=(ALL) NOPASSWD: ALL'
    validate: 'visudo -cf %s'
  become: True

- name: Add cron job to create log directory on reboot for RPis
  block:
    - name: Add the cron shell variable
      cron:
        name: SHELL
        env: true
        job: /bin/bash

    - name: Add cron job to create the directory at boot
      cron:
        name: "Zabbix log directory"
        job: mkdir /var/log/zabbix && chown zabbix:zabbix /var/log/zabbix
        special_time: reboot
  become: True
  when: "'pis' in group_names"

This installs the server stuff if zabbix_server is true and the agent stuff if zabbix_server is false so this same role can be used on all the machines. It also needs to add a different apt repo for the RPis as it does for my Ubuntu VMs so zabbix-agent2 can be installed which is required for good Docker integration. Also, on my RPis I put my logs into a RAM disk so I need to create and set the permissions on the directory Zabbix logs to before Zabbix starts which I do with a cron job that runs at boot.

My custom-zabbix.j2 file is:

Hostname={{ ansible_hostname }}
LogFileSize=1024
Server={{ zabbix_conn_str }}
ServerActive={{ zabbix_conn_str }}

The handler is

---
# handlers file for roles/zabbix

- name: Restart the zabbix agent to pick up config
  systemd:
    name: zabbix-agent2
    state: restarted
  become: True

Everything in {{ }} is a variable that will need to be set (except for the ones that start with “ansible” which gets set for you. See Ansible Revisited for more on using Ansible.

Using the role to install the server:

---

# Prerequisites:
# - install ssh
# - set up ssh for certificate only logins
# - install python3
# - all hosts added to the inventory under [homeauto]

# TODO set up root with ssh certs for git interactions

- hosts: homeauto
  vars:
    - update: False
  roles:
    - common
    - { role: vm, when: not update }
    - firemotd
    - mount-data
    - nut-client
    - msmtp
    - fail2ban
    - { role: multitail, openhab_logs: '/srv/openhab/userdata/logs/' }
    - { role: docker, when: not update }
    - role: zabbix
      vars:
        - zabbix_server: True
    - openzwave
    - mosquitto
    - openhab
    - docker-prune
    - tripwire

Example showing using the role to install the agent.

- hosts: media
  vars:
    - update: False
  roles:
    - common
    - { role: vm, when: not update }
    - firemotd
    - mount-data
    - nut-client
    - msmtp
    - fail2ban
    - { role: multitail, openhab_logs: '/srv/openhab2/userdata/logs/' }
    - { role: docker, when not update }
    - gitlab
    - plex
    - { role: calibre, calibre_type: 'docker' }
    - postgresql
    - redis
    - elasticsearch
    - nextcloud
    - { role: zabbix, zabbix_server: False }
    - docker-prune
    - tripwire

Note the above shows two different ways to set variables when including a role into a playbook.

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.