Don‘t know if this had been asked before, but did you update your rules to use GraalVM and not Nashorn?
Also check the config for 3.x leftovers…
This one might be helpfull
Yes, sure that is next point what i do after upgrade. I had to fix all rules. And the system crash all over the time and rules are sooooo slow if they are started in the first time. Also And I thing, hey if i must touch all rules, I can convert them directly to get them up to date with the new GraalVM. The log is now cleare without issues of any rule hope for sure. I never comes over 24h cause of fix or try something with restart. 
What are leftovers in config from 3.x? Did you have more details? Must I delete something?
Other Thread I will check. But all things are created by UI.
As Rich was already mentioning the swap file is quite huge. My total mem consumption is 740 MB. Your is 1.4GB.
Can you do a htop screenshot sorted by memory (without threads) and scroll down for a while until the next non-oh related command is shown, like this:
What is the output of
sudo service dphys-swapfile status.
could be that addons.config in userdata/config (not at home, therefore cannot validate path), might still refer to no longer existing addons.
If you don’t want to spend a lot of time on this I think your options are clear.
- Move stuff off of the RPi3 (e.g. InfluxDB and Grafana) to give OH more RAM to work with
- Abandon InfluxDB and Grafana for something that requires less memory (e.g. rrd4j and MainUI charts)
- Move everything to a machine with more RAM
Given the symptoms, any one of these should fix your problem.
Anything else is going to take a lot of time and effort and you still may end up with the same three options when you are done.
There may be some half measures you can use for the three options above. For example, maybe just close down Grafana and use MainUI charting. Research InfluxDB to see if you can configure it to use less RAM.
But as far as I can tell, the problem here is that everything you are running on this machine needs about 1.4-1.7 GB of RAM and you only have 1 GB of RAM. The only way to fix it is going to be reducing the amount of RAM required to 1 GB which is a huge reduction. I doubt you’ll ever be able to reach that through tweaks to configs and command line arguments.
Oh I’ve got that one memorized. ;-). $OH_USERDATA/config/org/openhab/addons.config
1 Like
● dphys-swapfile.service - dphys-swapfile - set up, mount/unmount, and delete a swap file
Loaded: loaded (/lib/systemd/system/dphys-swapfile.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2023-08-16 15:17:12 CEST; 2 days ago
Docs: man:dphys-swapfile(8)
Main PID: 457 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 2176)
CPU: 0
CGroup: /system.slice/dphys-swapfile.service
Warning: journal has been rotated since unit was started, output may be incomplete.
openhab 27391 96.1 62.9 1164640 626336 ? Ssl 16:00 26:13 /usr/bin/java -XX:-UsePerfData -Dopenhab.home=/usr/share/openhab -Dopenhab.conf=/etc/openhab -Dopenhab.runtime=/usr/share/openhab/runtime -Dopenhab.userdata=/var/lib/openhab -Dopenhab.logdir=/var/log/openhab -Dfelix.cm.dir=/var/lib/openhab/config -Djava.library.path=/var/lib/openhab/tmp/lib -Djdk.util.zip.disableZip64ExtraFieldValidation=true -Djetty.host=0.0.0.0 -Djetty.http.compliance=RFC2616 -Dorg.apache.cxf.osgi.http.transport.disable=true -Dorg.ops4j.pax.web.listening.addresses=0.0.0.0 -Dorg.osgi.service.http.port=8080 -Dorg.osgi.service.http.port.secure=8443 -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xms192m -Xmx768m -XX:+ExitOnOutOfMemoryError --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.locator-4.4.3.jar --patch-module java.xml=/usr/share/openhab/runtime/lib/endorsed/org.apache.karaf.specs.java.xml-4.4.3.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.text=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.desktop/java.awt.font=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.file=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=java.base/sun.net.www.content.text=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.security.sasl/com.sun.security.sasl=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED -Dkaraf.instances=/var/lib/openhab/tmp/instances -Dkaraf.home=/usr/share/openhab/runtime -Dkaraf.base=/var/lib/openhab -Dkaraf.data=/var/lib/openhab -Dkaraf.etc=/var/lib/openhab/etc -Dkaraf.log=/var/log/openhab -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/var/lib/openhab/tmp -Djava.util.logging.config.file=/var/lib/openhab/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /usr/share/openhab/runtime/lib/boot/org.apache.karaf.diagnostic.boot-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.jaas.boot-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.main-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/org.apache.karaf.specs.activator-4.4.3.jar:/usr/share/openhab/runtime/lib/boot/osgi.core-8.0.0.jar:/usr/share/openhab/runtime/lib/jdk9plus/istack-commons-runtime-3.0.10.jar:/usr/share/openhab/runtime/lib/jdk9plus/jakarta.xml.bind-api-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/javax.annotation-api-1.3.2.jar:/usr/share/openhab/runtime/lib/jdk9plus/jaxb-runtime-2.3.3.jar:/usr/share/openhab/runtime/lib/jdk9plus/org.apache.servicemix.specs.activation-api-1.2.1-1.2.1_3.jar:/usr/share/openhab/runtime/lib/jdk9plus/txw2-2.3.3.jar org.apache.karaf.main.Main
pi 5061 7.8 6.9 229968 68832 ? Sl Aug16 213:20 node index.js
grafana 971 0.4 3.8 907516 38472 ? Ssl Aug16 14:13 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
influxdb 680 0.7 3.7 1073756 37236 ? Sl Aug16 22:19 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
root 139 1.1 1.6 92820 16284 ? Ss Aug16 33:16 /lib/systemd/systemd-journald
frontail 27398 0.3 1.2 122952 12796 ? Ssl 16:00 0:05 node /usr/lib/node_modules/frontail/bin/frontail --disable-usage-stats --ui-highlight --ui-highlight-preset /usr/lib/node_modules/frontail/preset/openhab_AEM.json --theme openhab_AEM --lines 2000 --number 200 /var/log/openhab/openhab.log /var/log/openhab/events.log
root 1 0.0 0.4 34792 4764 ? Ss Aug16 0:37 /sbin/init
pi 27137 3.8 0.4 10500 4612 pts/0 S+ 15:56 1:11 htop
openhab 32445 0.3 0.4 8568 4556 ? S 16:27 0:00 arping -w 5 -C 1 -i enxb827eb1224b9 192.168.179.102
pi 1011 4.6 0.3 11496 3840 tty1 S+ Aug16 136:30 htop -u openhab
root 384 0.0 0.3 60728 3660 ? Ssl Aug16 1:41 /usr/sbin/NetworkManager --no-daemon
pi 5035 0.0 0.3 160288 3416 ? Ssl Aug16 0:03 npm start
pi 24939 0.0 0.3 8556 3024 pts/1 Ss 15:47 0:00 -bash
pi 32455 0.0 0.2 11000 2664 pts/1 R+ 16:28 0:00 ps aux --sort=-%mem
root 24622 0.0 0.2 14436 2312 ? Ss 15:44 0:00 sshd: pi [priv]
root 24931 0.0 0.2 14436 2292 ? Ss 15:47 0:00 sshd: pi [priv]
root 414 0.0 0.2 21176 2196 ? Ss Aug16 0:01 /lib/systemd/systemd-logind
root 423 0.0 0.1 11956 1772 ? Ss Aug16 0:36 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
mosquit+ 1047 0.3 0.1 13552 1768 ? Ss Aug16 10:26 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
root 678 0.0 0.1 70676 1716 ? Ss Aug16 0:45 /usr/sbin/nmbd --foreground --no-process-group
pi 24638 0.0 0.1 8556 1712 pts/0 Ss 15:44 0:00 -bash
pi 24637 0.2 0.1 14436 1668 ? S 15:44 0:06 sshd: pi@pts/0
root 890 0.0 0.1 84512 1504 ? S Aug16 0:01 /usr/sbin/smbd --foreground --no-process-group
frontail 27519 0.0 0.1 6988 1488 ? S 16:00 0:00 tail -n 200 -F /var/log/openhab/openhab.log /var/log/openhab/events.log
pi 24938 0.0 0.1 14436 1412 ? S 15:47 0:00 sshd: pi@pts/1
root 22460 0.0 0.1 95620 1380 ? Ssl Aug16 0:01 /usr/libexec/packagekitd
message+ 383 0.0 0.1 7828 1324 ? Ss Aug16 0:47 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 847 0.0 0.1 84524 1248 ? Ss Aug16 0:01 /usr/sbin/smbd --foreground --no-process-group
systemd+ 343 0.0 0.1 22208 1236 ? Ssl Aug16 0:01 /lib/systemd/systemd-timesyncd
avahi 381 0.0 0.1 6832 1220 ? Ss Aug16 0:52 avahi-daemon: running [openhabian.local]
root 397 0.0 0.1 39468 1180 ? Ssl Aug16 0:00 /usr/libexec/polkitd --no-debug
root 407 0.5 0.1 26460 1048 ? Ssl Aug16 14:59 /usr/sbin/rsyslogd -n -iNONE
root 490 0.0 0.0 48496 924 ? Ssl Aug16 0:00 /usr/sbin/ModemManager
pi 930 0.0 0.0 14324 860 ? Ss Aug16 0:00 /lib/systemd/systemd --user
root 876 0.0 0.0 82716 800 ? S Aug16 0:00 /usr/sbin/smbd --foreground --no-process-group
root 879 0.0 0.0 82720 764 ? S Aug16 0:00 /usr/sbin/smbd --foreground --no-process-group
root 562 0.0 0.0 12328 712 ? Ss Aug16 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 26437 0.0 0.0 217220 704 ? Ss Aug16 0:14 /usr/sbin/apache2 -k start
root 382 0.0 0.0 8116 652 ? Ss Aug16 0:00 /usr/sbin/cron -f
root 163 0.0 0.0 20004 640 ? Ss Aug16 0:01 /lib/systemd/systemd-udevd
root 459 0.0 0.0 27612 628 ? SLsl Aug16 0:00 /usr/sbin/rngd -r /dev/hwrng
openhab 32444 0.0 0.0 8428 628 ? S 16:27 0:00 ping -w 5 -c 1 192.168.179.102
www-data 13780 0.0 0.0 218204 620 ? S 00:00 0:00 /usr/sbin/apache2 -k start
www-data 31018 0.0 0.0 218196 608 ? S 09:14 0:00 /usr/sbin/apache2 -k start
root 430 0.0 0.0 21368 576 ? Ss Aug16 0:00 /usr/libexec/bluetooth/bluetoothd
www-data 30675 0.0 0.0 217300 572 ? S 09:12 0:00 /usr/sbin/apache2 -k start
www-data 30786 0.0 0.0 217300 572 ? S 09:13 0:00 /usr/sbin/apache2 -k start
www-data 31059 0.0 0.0 217300 572 ? S 09:14 0:00 /usr/sbin/apache2 -k start
www-data 31242 0.0 0.0 217300 564 ? S 09:16 0:00 /usr/sbin/apache2 -k start
www-data 31300 0.0 0.0 217252 540 ? S 09:16 0:00 /usr/sbin/apache2 -k start
nobody 418 0.0 0.0 5228 532 ? Ss Aug16 0:01 /usr/sbin/thd --triggers /etc/triggerhappy/triggers.d/ --socket /run/thd.socket --user nobody --deviceglob /dev/input/event*
pi 949 0.0 0.0 8564 524 tty1 S Aug16 0:00 -bash
root 685 0.0 0.0 8180 472 tty1 Ss Aug16 0:00 /bin/login -f
www-data 31084 0.0 0.0 217300 460 ? S 09:14 0:00 /usr/sbin/apache2 -k start
www-data 31132 0.0 0.0 217300 460 ? S 09:15 0:00 /usr/sbin/apache2 -k start
root 541 0.0 0.0 40568 448 ? Ssl Aug16 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
www-data 31262 0.0 0.0 217300 272 ? S 09:16 0:00 /usr/sbin/apache2 -k start
avahi 393 0.0 0.0 6672 4 ? S Aug16 0:00 avahi-daemon: chroot helper
pi 931 0.0 0.0 36244 4 ? S Aug16 0:00 (sd-pam)
@rlkoshak I don’t understand you again. Sorry… But I see 750mb ram is in use and only 296mb swap is in use exactly in this moment. I don’t see that is something wrong here…
If openhab stops then there is 172MB in use and swap is 154MB… So tell me please whats wrong with it? I can’t find the issue at the moment. If swap will be filled totally than there is a problem with Openhab or Openhab needs 2GB of RAM. Then all users will have problems to run Openhab 4 on Pi 3. That don’t make seens to me sorry.
This all looks good. Last question, output of
swapon -s
Filename Type Size Used Priority
/var/swap file 1890300 312472 -2
BTW there is no bad entry in addons.config
:org.apache.felix.configadmin.revision:=L"141"
automation=",jsscripting"
binding="ipcamera,netatmo,dwdunwetter,tr064,pixometer,telegram,ntp,dwdpollenflug,tankerkoenig,harmonyhub,avmfritz,openweathermap,network,miio,astro,mqtt,icalendar,http,exec"
felix.fileinstall.filename="file:/var/lib/openhab/etc/org.openhab.addons.cfg"
legacy=B"false"
misc="openhabcloud"
package="standard"
persistence=",influxdb"
remote=B"true"
service.pid="org.openhab.addons"
transformation="exec,javascript,map,jsonpath,regex,jinja"
ui="basic"
voice="picotts"
According to this screenshot, 1.4 GB are in use
Yes sure something is eating the mem… But for me that is no problem. I we can discuss if swap is also full.
No, it is dedicated to OH. Persistence is rrd4j but with files stored on a mounted drive.
To run just OH4 and a standard setup (18 bindings installed in my case), a RPI 3 seems to be ok with no problem. Maybe just a little slow to start.
@Lolodomo thanks for your feedback.
So i will say, the Pi is ok with RAM and SWAP.
So nobody had explained me what VM Thread is and what that thing doing. That is on the second position if nothing works for 5-10min on htop list.
Remember to that picture:
If I roughly add up my CPU of all various commands I end up almost exactly with the 22% out of 4GB ~ 800MB which is inline with the memory bar of htop in the upper left corner (no surprise).
If I do the same with your system, I end up with ~84% out of 1GB = 840MB which is also fine, BUT swapon showed us that another 312MB are in use.
This is to me unusual.
Maybe you get some tools so that you can identify which processes are using swap file.
Yes then we end in openhab. But what i say… We talk about something what we don’t need really. We can talk about it, with the swap is full. What i say… I it doesn’t matter if swap is 400 or 800… Cause if the swap is so high openhab is running! We don’t need talk about… I don’t say Openhab isn’t running.
But there is a process depended with VM Thread what openhab blocked to working… It needs then 5-10mins and Openhab is working… And we must investigate this!
This does not calculate on top of your memory consumption. OH and all of its threads are provided with the same memory consumption rate
I’m looking at your screen shots for htop.
- 762 MB in use, 836 MB swap
- 777 MB in use, 817 MB swap
- 815 MB in use, 796 MB swap
- 748 MB in use, 685 MB swap
- 888 MB in use, 224 MB swap
That’s where I come up with my 1.4 to 1.7 GB range.
What’s wrong here is there is any amount of swap in use. As soon as even 1 MB of swap starts being used your system load skyrockets, and process start to have to wait around for stuff that should be in RAM to be fetched from disk.
Once swap starts being used it takes a really long time for it to be flushed and it never really fully gets flushed.
I’m sure things have changed in the two decades since my last OS class, but in general swap works like this:
- a program requests some memory but there isn’t enough unused RAM to allocate that amount to the program
- the operating system looks through the table and finds the stuff in RAM that hasn’t been used for the longest amount of time, copies that to disk (i.e. swap) and then frees up that part of RAM.
- Repeat for subsequent requests for newly allocated RAM.
- When a program exits or frees up it’s RAM, the OS does not automatically restore stuff that has been swapped out. It remains in swap until it’s used.
- When memory that is in swap is requested, something else in RAM needs to be swapped out and then the requested data is swapped back into RAM for use.
So why does this cause problems. The speed of typical DRAM is around 2-20 GB/s (let’s say an RPi 3 is around 12 GB/s, it doesn’t really matter though we are just doing fast math here). An SSD is going to be 50-200 MB/s. But this is an RPi so we are actually limited by the speed of USB which maxes out at 52 MB/s, and that’s shared among all your USB devices and the ethernet. And IIRC it’s not symmetric but we’ll assume the max.
Some quick math… fetching data from swap is 236 % the speed of fetching the same data from RAM. And as shown above, it has to be done twice (write out something from RAM to swap, read in the needed data back into RAM) every time something from swap needs to be fetched!
So if it takes 1 msec to fetch something from RAM (which is stupid slow for RAM BTW), it’s going to take almost half a second to fetch the same data that has been swapped out.
That is the problem. You will notice a 472 % slowdown in your system. That’s why any amount of swap in use is a problem.
Remember how in 2 above I mentioned that the RAM that hasn’t been used in the longest is what gets chosen to become swapped out? Well, compared to most everything else you have running on your machine, your rules are probably running the least meaning their stuff in RAM is touched the least and they are the most likely to be swapped out. It’s no wonder you have the most problems when your rules run. What gets used even less often? The code that reads in and parses the rules so editing your rules is going to be awful.
You are welcome to continue tilting at windmills on this but the problem is blindingly clear to me, as is the solution. You can’t just ignore the swap in use here.
1 Like
Sorry i disagree again! I don’t have a problem with speed…
And i don’t have an issue with my system. What you say sound’s that the hole system is affected. No. It isn’t… only Openhab. So on the other side if Openhab eats so many memory and swap it over, somebody must explain me why… If Openhab start everything is fine and then something is eaten the memory then we have a problem… And what you explain is clear but more a side affect of a problem with Openhab itself.
I disagree.
I showed you the following:
- The application openhab is using as much memory than my system - for sure. no problem here.
- Your PI runs at 85% memory consumption plus additional 312MB which sums up to 1.2 GB which is by far too much even taking influx and Grafana into consideration.
Your system requires almost double memory space than my system - Yes it does matter how large your swap file is. Your swap file is a third of your overall memory required. If it swaps back and forth then I am not surprised why there are timeouts in OH and some components do not react within 5 seconds
I disagree again. Look at all your error messages in your log like these:
These are not the problems. These are the result of your system being occupied by swapping.
1 Like