Openhab 4.0.x rules need a lot of memory

Don‘t know if this had been asked before, but did you update your rules to use GraalVM and not Nashorn?
Also check the config for 3.x leftovers…
This one might be helpfull

Yes, sure that is next point what i do after upgrade. I had to fix all rules. And the system crash all over the time and rules are sooooo slow if they are started in the first time. Also And I thing, hey if i must touch all rules, I can convert them directly to get them up to date with the new GraalVM. The log is now cleare without issues of any rule hope for sure. I never comes over 24h cause of fix or try something with restart. :slight_smile:

What are leftovers in config from 3.x? Did you have more details? Must I delete something?

Other Thread I will check. But all things are created by UI.

As Rich was already mentioning the swap file is quite huge. My total mem consumption is 740 MB. Your is 1.4GB.
Can you do a htop screenshot sorted by memory (without threads) and scroll down for a while until the next non-oh related command is shown, like this:

What is the output of

sudo service dphys-swapfile status.

could be that addons.config in userdata/config (not at home, therefore cannot validate path), might still refer to no longer existing addons.

If you don’t want to spend a lot of time on this I think your options are clear.

  1. Move stuff off of the RPi3 (e.g. InfluxDB and Grafana) to give OH more RAM to work with
  2. Abandon InfluxDB and Grafana for something that requires less memory (e.g. rrd4j and MainUI charts)
  3. Move everything to a machine with more RAM

Given the symptoms, any one of these should fix your problem.

Anything else is going to take a lot of time and effort and you still may end up with the same three options when you are done.

There may be some half measures you can use for the three options above. For example, maybe just close down Grafana and use MainUI charting. Research InfluxDB to see if you can configure it to use less RAM.

But as far as I can tell, the problem here is that everything you are running on this machine needs about 1.4-1.7 GB of RAM and you only have 1 GB of RAM. The only way to fix it is going to be reducing the amount of RAM required to 1 GB which is a huge reduction. I doubt you’ll ever be able to reach that through tweaks to configs and command line arguments.

Oh I’ve got that one memorized. ;-). $OH_USERDATA/config/org/openhab/addons.config

1 Like
● dphys-swapfile.service - dphys-swapfile - set up, mount/unmount, and delete a swap file
     Loaded: loaded (/lib/systemd/system/dphys-swapfile.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2023-08-16 15:17:12 CEST; 2 days ago
       Docs: man:dphys-swapfile(8)
   Main PID: 457 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 2176)
        CPU: 0
     CGroup: /system.slice/dphys-swapfile.service

Warning: journal has been rotated since unit was started, output may be incomplete.
openhab  27391 96.1 62.9 1164640 626336 ?      Ssl  16:00  26:13 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Djava.library.path=/var/lib/openhab/tmp/lib -Djetty.http.compliance=RFC2616 -Dorg.apache.cxf.osgi.http.transport.disable=true -Dorg.ops4j.pax.web.listening.addresses= -Dorg.osgi.service.http.port=8080 -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xms192m -Xmx768m -XX:+ExitOnOutOfMemoryError --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.locator-4.4.3.jar --patch-module java.xml=/usr/share/openhab/runtime/lib/endorsed/ --add-opens java.base/ --add-opens java.base/ --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/ --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.text=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.desktop/java.awt.font=ALL-UNNAMED --add-exports=java.base/ --add-exports=java.base/ --add-exports=java.base/ --add-exports=java.base/ --add-exports=java.base/ --add-exports=java.base/ --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED -Dkaraf.instances=/var/lib/openhab/tmp/instances -Dkaraf.home=/usr/share/openhab/runtime -Dkaraf.base=/var/lib/openhab -Dkaraf.etc=/var/lib/openhab/etc -Dkaraf.log=/var/log/openhab -Dkaraf.restart.jvm.supported=true -Djava.util.logging.config.file=/var/lib/openhab/etc/ -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /usr/share/openhab/runtime/lib/boot/org.apache.karaf.diagnostic.boot-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.jaas.boot-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.main-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.specs.activator-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/osgi.core-8.0.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/istack-commons-runtime-3.0.10.jar:/usr/share/openhab/runtime/lib/jdk9plus/jakarta.xml.bind-api-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.annotation-api-1.3.2.jar:/usr/share/openhab/runtime/lib/jdk9plus/jaxb-runtime-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/org.apache.servicemix.specs.activation-api-1.2.1-1.2.1_3.jar:/usr/share/openhab/runtime/lib/jdk9plus/txw2-2.3.3.jar org.apache.karaf.main.Main
pi        5061  7.8  6.9 229968 68832 ?        Sl   Aug16 213:20 node index.js
grafana    971  0.4  3.8 907516 38472 ?        Ssl  Aug16  14:13 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/ --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
influxdb   680  0.7  3.7 1073756 37236 ?       Sl   Aug16  22:19 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
root       139  1.1  1.6  92820 16284 ?        Ss   Aug16  33:16 /lib/systemd/systemd-journald
frontail 27398  0.3  1.2 122952 12796 ?        Ssl  16:00   0:05 node /usr/lib/node_modules/frontail/bin/frontail --disable-usage-stats --ui-highlight --ui-highlight-preset /usr/lib/node_modules/frontail/preset/openhab_AEM.json --theme openhab_AEM --lines 2000 --number 200 /var/log/openhab/openhab.log /var/log/openhab/events.log
root         1  0.0  0.4  34792  4764 ?        Ss   Aug16   0:37 /sbin/init
pi       27137  3.8  0.4  10500  4612 pts/0    S+   15:56   1:11 htop
openhab  32445  0.3  0.4   8568  4556 ?        S    16:27   0:00 arping -w 5 -C 1 -i enxb827eb1224b9
pi        1011  4.6  0.3  11496  3840 tty1     S+   Aug16 136:30 htop -u openhab
root       384  0.0  0.3  60728  3660 ?        Ssl  Aug16   1:41 /usr/sbin/NetworkManager --no-daemon
pi        5035  0.0  0.3 160288  3416 ?        Ssl  Aug16   0:03 npm start
pi       24939  0.0  0.3   8556  3024 pts/1    Ss   15:47   0:00 -bash
pi       32455  0.0  0.2  11000  2664 pts/1    R+   16:28   0:00 ps aux --sort=-%mem
root     24622  0.0  0.2  14436  2312 ?        Ss   15:44   0:00 sshd: pi [priv]
root     24931  0.0  0.2  14436  2292 ?        Ss   15:47   0:00 sshd: pi [priv]
root       414  0.0  0.2  21176  2196 ?        Ss   Aug16   0:01 /lib/systemd/systemd-logind
root       423  0.0  0.1  11956  1772 ?        Ss   Aug16   0:36 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
mosquit+  1047  0.3  0.1  13552  1768 ?        Ss   Aug16  10:26 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
root       678  0.0  0.1  70676  1716 ?        Ss   Aug16   0:45 /usr/sbin/nmbd --foreground --no-process-group
pi       24638  0.0  0.1   8556  1712 pts/0    Ss   15:44   0:00 -bash
pi       24637  0.2  0.1  14436  1668 ?        S    15:44   0:06 sshd: pi@pts/0
root       890  0.0  0.1  84512  1504 ?        S    Aug16   0:01 /usr/sbin/smbd --foreground --no-process-group
frontail 27519  0.0  0.1   6988  1488 ?        S    16:00   0:00 tail -n 200 -F /var/log/openhab/openhab.log /var/log/openhab/events.log
pi       24938  0.0  0.1  14436  1412 ?        S    15:47   0:00 sshd: pi@pts/1
root     22460  0.0  0.1  95620  1380 ?        Ssl  Aug16   0:01 /usr/libexec/packagekitd
message+   383  0.0  0.1   7828  1324 ?        Ss   Aug16   0:47 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root       847  0.0  0.1  84524  1248 ?        Ss   Aug16   0:01 /usr/sbin/smbd --foreground --no-process-group
systemd+   343  0.0  0.1  22208  1236 ?        Ssl  Aug16   0:01 /lib/systemd/systemd-timesyncd
avahi      381  0.0  0.1   6832  1220 ?        Ss   Aug16   0:52 avahi-daemon: running [openhabian.local]
root       397  0.0  0.1  39468  1180 ?        Ssl  Aug16   0:00 /usr/libexec/polkitd --no-debug
root       407  0.5  0.1  26460  1048 ?        Ssl  Aug16  14:59 /usr/sbin/rsyslogd -n -iNONE
root       490  0.0  0.0  48496   924 ?        Ssl  Aug16   0:00 /usr/sbin/ModemManager
pi         930  0.0  0.0  14324   860 ?        Ss   Aug16   0:00 /lib/systemd/systemd --user
root       876  0.0  0.0  82716   800 ?        S    Aug16   0:00 /usr/sbin/smbd --foreground --no-process-group
root       879  0.0  0.0  82720   764 ?        S    Aug16   0:00 /usr/sbin/smbd --foreground --no-process-group
root       562  0.0  0.0  12328   712 ?        Ss   Aug16   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root     26437  0.0  0.0 217220   704 ?        Ss   Aug16   0:14 /usr/sbin/apache2 -k start
root       382  0.0  0.0   8116   652 ?        Ss   Aug16   0:00 /usr/sbin/cron -f
root       163  0.0  0.0  20004   640 ?        Ss   Aug16   0:01 /lib/systemd/systemd-udevd
root       459  0.0  0.0  27612   628 ?        SLsl Aug16   0:00 /usr/sbin/rngd -r /dev/hwrng
openhab  32444  0.0  0.0   8428   628 ?        S    16:27   0:00 ping -w 5 -c 1
www-data 13780  0.0  0.0 218204   620 ?        S    00:00   0:00 /usr/sbin/apache2 -k start
www-data 31018  0.0  0.0 218196   608 ?        S    09:14   0:00 /usr/sbin/apache2 -k start
root       430  0.0  0.0  21368   576 ?        Ss   Aug16   0:00 /usr/libexec/bluetooth/bluetoothd
www-data 30675  0.0  0.0 217300   572 ?        S    09:12   0:00 /usr/sbin/apache2 -k start
www-data 30786  0.0  0.0 217300   572 ?        S    09:13   0:00 /usr/sbin/apache2 -k start
www-data 31059  0.0  0.0 217300   572 ?        S    09:14   0:00 /usr/sbin/apache2 -k start
www-data 31242  0.0  0.0 217300   564 ?        S    09:16   0:00 /usr/sbin/apache2 -k start
www-data 31300  0.0  0.0 217252   540 ?        S    09:16   0:00 /usr/sbin/apache2 -k start
nobody     418  0.0  0.0   5228   532 ?        Ss   Aug16   0:01 /usr/sbin/thd --triggers /etc/triggerhappy/triggers.d/ --socket /run/thd.socket --user nobody --deviceglob /dev/input/event*
pi         949  0.0  0.0   8564   524 tty1     S    Aug16   0:00 -bash
root       685  0.0  0.0   8180   472 tty1     Ss   Aug16   0:00 /bin/login -f
www-data 31084  0.0  0.0 217300   460 ?        S    09:14   0:00 /usr/sbin/apache2 -k start
www-data 31132  0.0  0.0 217300   460 ?        S    09:15   0:00 /usr/sbin/apache2 -k start
root       541  0.0  0.0  40568   448 ?        Ssl  Aug16   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
www-data 31262  0.0  0.0 217300   272 ?        S    09:16   0:00 /usr/sbin/apache2 -k start
avahi      393  0.0  0.0   6672     4 ?        S    Aug16   0:00 avahi-daemon: chroot helper
pi         931  0.0  0.0  36244     4 ?        S    Aug16   0:00 (sd-pam)

@rlkoshak I don’t understand you again. Sorry… But I see 750mb ram is in use and only 296mb swap is in use exactly in this moment. I don’t see that is something wrong here…
If openhab stops then there is 172MB in use and swap is 154MB… So tell me please whats wrong with it? I can’t find the issue at the moment. If swap will be filled totally than there is a problem with Openhab or Openhab needs 2GB of RAM. Then all users will have problems to run Openhab 4 on Pi 3. That don’t make seens to me sorry.

This all looks good. Last question, output of

swapon -s
Filename				Type		Size	Used	Priority
/var/swap                              	file    	1890300	312472	-2

BTW there is no bad entry in addons.config


According to this screenshot, 1.4 GB are in use

Yes sure something is eating the mem… But for me that is no problem. I we can discuss if swap is also full.

No, it is dedicated to OH. Persistence is rrd4j but with files stored on a mounted drive.

To run just OH4 and a standard setup (18 bindings installed in my case), a RPI 3 seems to be ok with no problem. Maybe just a little slow to start.

@Lolodomo thanks for your feedback.

So i will say, the Pi is ok with RAM and SWAP.

So nobody had explained me what VM Thread is and what that thing doing. That is on the second position if nothing works for 5-10min on htop list.

Remember to that picture:

If I roughly add up my CPU of all various commands I end up almost exactly with the 22% out of 4GB ~ 800MB which is inline with the memory bar of htop in the upper left corner (no surprise).

If I do the same with your system, I end up with ~84% out of 1GB = 840MB which is also fine, BUT swapon showed us that another 312MB are in use.
This is to me unusual.
Maybe you get some tools so that you can identify which processes are using swap file.

Yes then we end in openhab. But what i say… We talk about something what we don’t need really. We can talk about it, with the swap is full. What i say… I it doesn’t matter if swap is 400 or 800… Cause if the swap is so high openhab is running! We don’t need talk about… I don’t say Openhab isn’t running.

But there is a process depended with VM Thread what openhab blocked to working… It needs then 5-10mins and Openhab is working… And we must investigate this!

This does not calculate on top of your memory consumption. OH and all of its threads are provided with the same memory consumption rate

I’m looking at your screen shots for htop.

  • 762 MB in use, 836 MB swap
  • 777 MB in use, 817 MB swap
  • 815 MB in use, 796 MB swap
  • 748 MB in use, 685 MB swap
  • 888 MB in use, 224 MB swap

That’s where I come up with my 1.4 to 1.7 GB range.

What’s wrong here is there is any amount of swap in use. As soon as even 1 MB of swap starts being used your system load skyrockets, and process start to have to wait around for stuff that should be in RAM to be fetched from disk.

Once swap starts being used it takes a really long time for it to be flushed and it never really fully gets flushed.

I’m sure things have changed in the two decades since my last OS class, but in general swap works like this:

  1. a program requests some memory but there isn’t enough unused RAM to allocate that amount to the program
  2. the operating system looks through the table and finds the stuff in RAM that hasn’t been used for the longest amount of time, copies that to disk (i.e. swap) and then frees up that part of RAM.
  3. Repeat for subsequent requests for newly allocated RAM.
  4. When a program exits or frees up it’s RAM, the OS does not automatically restore stuff that has been swapped out. It remains in swap until it’s used.
  5. When memory that is in swap is requested, something else in RAM needs to be swapped out and then the requested data is swapped back into RAM for use.

So why does this cause problems. The speed of typical DRAM is around 2-20 GB/s (let’s say an RPi 3 is around 12 GB/s, it doesn’t really matter though we are just doing fast math here). An SSD is going to be 50-200 MB/s. But this is an RPi so we are actually limited by the speed of USB which maxes out at 52 MB/s, and that’s shared among all your USB devices and the ethernet. And IIRC it’s not symmetric but we’ll assume the max.

Some quick math… fetching data from swap is 236 % the speed of fetching the same data from RAM. And as shown above, it has to be done twice (write out something from RAM to swap, read in the needed data back into RAM) every time something from swap needs to be fetched!

So if it takes 1 msec to fetch something from RAM (which is stupid slow for RAM BTW), it’s going to take almost half a second to fetch the same data that has been swapped out.

That is the problem. You will notice a 472 % slowdown in your system. That’s why any amount of swap in use is a problem.

Remember how in 2 above I mentioned that the RAM that hasn’t been used in the longest is what gets chosen to become swapped out? Well, compared to most everything else you have running on your machine, your rules are probably running the least meaning their stuff in RAM is touched the least and they are the most likely to be swapped out. It’s no wonder you have the most problems when your rules run. What gets used even less often? The code that reads in and parses the rules so editing your rules is going to be awful.

You are welcome to continue tilting at windmills on this but the problem is blindingly clear to me, as is the solution. You can’t just ignore the swap in use here.

1 Like

Sorry i disagree again! I don’t have a problem with speed…

And i don’t have an issue with my system. What you say sound’s that the hole system is affected. No. It isn’t… only Openhab. So on the other side if Openhab eats so many memory and swap it over, somebody must explain me why… If Openhab start everything is fine and then something is eaten the memory then we have a problem… And what you explain is clear but more a side affect of a problem with Openhab itself.

I disagree.
I showed you the following:

  1. The application openhab is using as much memory than my system - for sure. no problem here.
  2. Your PI runs at 85% memory consumption plus additional 312MB which sums up to 1.2 GB which is by far too much even taking influx and Grafana into consideration.
    Your system requires almost double memory space than my system
  3. Yes it does matter how large your swap file is. Your swap file is a third of your overall memory required. If it swaps back and forth then I am not surprised why there are timeouts in OH and some components do not react within 5 seconds

I disagree again. Look at all your error messages in your log like these:

These are not the problems. These are the result of your system being occupied by swapping.

1 Like