UNSUCCESS: openHAB updated from 4.0.2 to 4.1.0 on docker - openhab unresponsive

Hi, im using docker since a long time for my openhab. Again i had the situation that i wanted to to an upgrade. In logfile i found the line
SUCCESS: openHAB updated from 4.0.2 to 4.1.0
Again i had the Error-Handler on each Thing as on last migrations.
But it was not possible to download any add-ons. So i triet again the clean of the cache and tmp.

Then i tried again to download “knx” add-on, but im seeing only rotating circle and it makes no progress and also im seeing no entries in logfile which gives me a hint. I tried also other add-ons, but exactly the same. i waited also longer time, but also no change in behavior.
Does anyone has an idea what i can do to get add-ons in again?

How do you use it?
What host OS, and Docker version do you use?
How do you manage the containers? Using the command line, Docker compose or do you use something like Portainer?
Are the Docker volumes on the same host or stored somewhere else in your network e.g. using NFS?

Moin,

are you using the Openhab container directly on the host network or is it in the isolated bridge network of the docker?

I am using Synology Docker, with portainer and watchtower, both also in separate own Docker containers and Openhab in container but on host network and upgrade went smoothly from 4.0.4 to 4.1.0 with only small hick-ups

Solong
B

hi @wborn
container runs on

  • raspi4 8GB
  • Debian 11
  • Docker version 24.0.7
  • docker-compose Containermanagement, no Portaine…
  • data is stored locally, no nfs on an SSD (disk is about 30% so space is enough available)
  • container runs in host network
    As told the container runs fine and also the conversion from old version to new one was fine.
    But the container is not able to update/download any add-on. Not automaticly, not manually initiated. Restart of container made no difference and also not after cleaning tmp/cache.
    The behavoir with the addons is not new, it was also existing on former versions too but then it was possible to initiate the downloads manually (you can see my upper referenced old thred to this).

So from my point of view we are speaking not about problems from outside container, the downloads of add-ons are made from inside container

I also tested that internetconnection is working fine from inside container:

docker exec -it openhab-openhab-1 ping openhab.org
PING openhab.org (188.114.97.12) 56(84) bytes of data.
64 bytes from 188.114.97.12 (188.114.97.12): icmp_seq=1 ttl=56 time=19.0 ms
64 bytes from 188.114.97.12 (188.114.97.12): icmp_seq=2 ttl=56 time=22.2 ms

Update 13:00 it seems that some things now coming to green. lets wait, maybe it comes to an end from itself. I will let you know later the result

It doesn’t download add-ons from that host. That’s just the website.

Add-ons are downloaded using the JFrog Artifactory CDN using openhab.jfrog.io which is replicated across several data centers.

E.g.:

https://openhab.jfrog.io/artifactory/libs-release/org/openhab/addons/bundles/org.openhab.binding.astro/4.1.0/org.openhab.binding.astro-4.1.0.jar

For the JFrog status see:

https://status.jfrog.io/

This time i have things green but cpu is very high and gui very unresponsive

CONTAINER ID   NAME                             CPU %     MEM USAGE / LIMIT
6752985a8aaa   openhab-openhab-1                204%   0B / 0B

Just run top to see what is consuming your CPU resources.
If it is something in openHAB run top -H so it shows thread names.

Hi @wborn
this gives me something like that, hope this helps that we can find out more:

docker exec -it openhab-openhab-1 top -H
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND       
    127 openhab   20   0 6014320   1.2g  25392 R  78.9  15.1  13:58.53 GC Thread#3                                                                                                                                                            
    126 openhab   20   0 6014320   1.2g  25392 R  74.4  15.1  14:00.76 GC Thread#2                                                                                                                                                            
    107 openhab   20   0 6014320   1.2g  25392 R  67.9  15.1  13:55.41 GC Thread#0                                                                                                                                                            
    125 openhab   20   0 6014320   1.2g  25392 R  64.9  15.1  13:59.08 GC Thread#1

all other threads had less 3% cpu

GC is Garbage Collection. The JVM starts these when it is running low on memory. So either Java is not assigned enough memory (using the -Xmx parameter) or some kind of memory leak causes the required memory to grow indefinitely.

Hi @wborn
i changed the paramater from -Xmx600m to -Xmx800m

Now (about 5mins after start) im seeing the cpu/threads:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
    207 openhab   20   0 5889192 826048  24636 S 192.7  10.3  17:51.22 java                                                                                                                                                                   
      1 openhab   20   0    1944    452    388 S   0.0   0.0   0:00.12 tini
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
   1287 openhab   20   0 5819692 777872  24636 S   6.8   9.7   0:02.97 SocketListener(                                                                                                                                                        
   1317 openhab   20   0 5819692 777872  24636 R   6.8   9.7   0:02.92 SocketListener(                                                                                                                                                        
   1338 openhab   20   0 5819692 777872  24636 S   6.8   9.7   0:02.65 SocketListener(                                                                                                                                                        
   1401 openhab   20   0 5819692 777872  24636 S   6.8   9.7   0:02.55 SocketListener(                                                                                                                                                        
    662 openhab   20   0 5815612 777512  24636 R   6.1   9.7   0:03.01 SocketListener(                                                                                                                                                        
   1224 openhab   20   0 5819692 777872  24636 R   6.1   9.7   0:02.61 SocketListener(                                                                                                                                                        
   1349 openhab   20   0 5819692 777872  24636 R   6.1   9.7   0:03.03 SocketListener(                                                                                                                                                        
   1176 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.69 SocketListener(                                                                                                                                                        
   1203 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.82 SocketListener(                                                                                                                                                        
   1221 openhab   20   0 5819692 777872  24636 S   5.4   9.7   0:02.85 SocketListener(                                                                                                                                                        
   1230 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.82 SocketListener(                                                                                                                                                        
   1254 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.62 SocketListener(                                                                                                                                                        
   1278 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.64 SocketListener(                                                                                                                                                        
   1284 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.96 SocketListener(                                                                                                                                                        
   1332 openhab   20   0 5819692 777872  24636 R   5.4   9.7   0:02.60 SocketListener(                                                                                                                                                        
   1433 openhab   20   0 5819692 777872  24636 S   5.4   9.7   0:02.54 SocketListener(                                                                                                                                                        
    650 openhab   20   0 5815612 777512  24636 R   4.8   9.7   0:02.86 SocketListener(                                                                                                                                                        
   1182 openhab   20   0 5819692 777872  24636 R   4.8   9.7   0:02.89 SocketListener(                                                                                                                                                        
   1191 openhab   20   0 5819692 777872  24636 R   4.8   9.7   0:02.88 SocketListener(
.......

i did a second try with 1000 and seeing:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
    986 openhab   20   0 6035628 863724  24552 S  99.9  10.8   0:06.84 OH-thingHandler                                                                                                                                                        
   1270 openhab   20   0 6035628 863724  24552 R  88.1  10.8   0:03.33 SocketListener(                                                                                                                                                        
   1399 openhab   20   0 6035628 863724  24552 R  82.8  10.8   0:03.13 SocketListener(                                                                                                                                                        
   1205 openhab   20   0 6035628 863724  24552 R  82.5  10.8   0:03.12 SocketListener(                                                                                                                                                        
   1199 openhab   20   0 6035628 863724  24552 R  81.5  10.8   0:03.08 SocketListener(                                                                                                                                                        
   1383 openhab   20   0 6035628 863724  24552 R  81.5  10.8   0:03.08 SocketListener(                                                                                                                                                        
   1297 openhab   20   0 6035628 863724  24552 R  81.0  10.8   0:03.06 SocketListener(                                                                                                                                                        
   1181 openhab   20   0 6035628 863724  24552 R  80.7  10.8   0:03.05 SocketListener(                                                                                                                                                        
   1264 openhab   20   0 6035628 863724  24552 R  80.7  10.8   0:03.05 SocketListener(                                                                                                                                                        
   1148 openhab   20   0 6035628 863724  24552 R  80.4  10.8   0:03.04 SocketListener(                                                                                                                                                        
   1160 openhab   20   0 6035628 863724  24552 R  80.4  10.8   0:03.04 SocketListener(                                                                                                                                                        
   1169 openhab   20   0 6035628 863724  24552 R  80.4  10.8   0:03.04 SocketListener(                                                                                                                                                        
   1208 openhab   20   0 6035628 863724  24552 R  80.4  10.8   0:03.04 SocketListener(                                                                                                                                                        
   1369 openhab   20   0 6035628 863724  24552 R  80.4  10.8   0:03.04 SocketListener(                                                                                                                                                        
   1172 openhab   20   0 6035628 863724  24552 R  80.2  10.8   0:03.03 SocketListener(                                                                                                                                                        
   1250 openhab   20   0 6035628 863724  24552 R  80.2  10.8   0:03.03 SocketListener(                                                                                                                                                        
   1157 openhab   20   0 6035628 863724  24552 R  79.9  10.8   0:03.02 SocketListener(                                                                                                                                                        
   1187 openhab   20   0 6035628 863724  24552 R  79.4  10.8   0:03.00 SocketListener(                                                                                                                                                        
   1364 openhab   20   0 6035628 863724  24552 R  79.4  10.8   0:03.00 SocketListener(                                                                                                                                                        
   1163 openhab   20   0 6035628 863724  24552 R  79.1  10.8   0:02.99 SocketListener(                                                                                                                                                        
   1353 openhab   20   0 6035628 863724  24552 R  78.8  10.8   0:02.98 SocketListener(                                                                                                                                                        
   1423 openhab   20   0 6035628 863724  24552 S  78.8  10.8   0:02.98 SocketListener(                                                                                                                                                        
   1235 openhab   20   0 6035628 863724  24552 R  78.6  10.8   0:02.97 SocketListener(                                                                                                                                                        
   1190 openhab   20   0 6035628 863724  24552 R  78.0  10.8   0:02.95 SocketListener(                                                                                                                                                        
   1335 openhab   20   0 6035628 863724  24552 R  78.0  10.8   0:02.95 SocketListener(                                                                                                                                                        
   1166 openhab   20   0 6035628 863724  24552 R  77.8  10.8   0:02.94 SocketListener(                                                                                                                                                        
   1193 openhab   20   0 6035628 863724  24552 R  77.8  10.8   0:02.94 SocketListener(                                                                                                                                                        
   1253 openhab   20   0 6035628 863724  24552 R  77.8  10.8   0:02.94 SocketListener(                                                                                                                                                        
   1378 openhab   20   0 6035628 863724  24552 R  77.8  10.8   0:02.94 SocketListener(        

i tried a shell:threads on command line of karaf console and got out that for each existing docker-network which is existing such thread was started. So im not sure if for some add-ons it is ignored that openhab is running on host-network

"SocketListener(192-168-32-1.local.)" Id=315 in RUNNABLE (running in native)
    at java.base@17.0.9/sun.nio.ch.DatagramChannelImpl.receive0(Native Method)
    at java.base@17.0.9/sun.nio.ch.DatagramChannelImpl.receiveIntoNativeBuffer(DatagramChannelImpl.java:750)
    at java.base@17.0.9/sun.nio.ch.DatagramChannelImpl.receive(DatagramChannelImpl.java:728)
    at java.base@17.0.9/sun.nio.ch.DatagramChannelImpl.trustedBlockingReceive(DatagramChannelImpl.java:666)
    at java.base@17.0.9/sun.nio.ch.DatagramChannelImpl.blockingReceive(DatagramChannelImpl.java:635)
    at java.base@17.0.9/sun.nio.ch.DatagramSocketAdaptor.receive(DatagramSocketAdaptor.java:240)
    at java.base@17.0.9/java.net.DatagramSocket.receive(DatagramSocket.java:700)
    at javax.jmdns.impl.SocketListener.run(SocketListener.java:57)

If your container sees more than one network, you might se more than one in MainUi/Network settings as well.
Limit those settings to just the one IP you use.

Just a guess as I am no docker user/expert, but this applies to all installations where the machine has more than one interface (LAN+WiFi e.g.)

Perhaps you didn’t use any addons that use jmdns for discovery previously. It’s now used for the addon discovery:

Would be nice if there is a way to restrict this to certain interfaces.

hi @hmerk
i limited before update the network in ui, thats correct.
After the update - now - the gui is unvailable.
So i guess that configuration was not changed it should be still limited to my main IP

yeah, this could be possibly the reason. Is there a way to switch this off/back to former methode by karaf? A short time after a restart i have access to karaf before all is hanging

Also it would be a good thing to write something about this change if someone uses docker in Release-Notes There is already a headline for “Breaking Changes”

Maybe there is a real possibility like a variable EXTRA_JAVA_OPTS to set hard an network-interface which java listens on. This possibly would also do it so it is forbidden for java to use any other interface which exists…

To test if this is caused by the add-on suggestion service (mDNS and/or UPnP), you can add the following lines to the the cfg/services/addons.cfg file before starting OH:

suggestionFinderMdns = false
suggestionFinderUpnp = false

The suggestion finder service will not start and use these underlying services in that case. So if they were not used by any of your add-ons before, they will not be started at all.

The same options are available from the UI, but setting it in the addons.cfg file allows you to avoid starting it at all.

1 Like

hi @Mherwege
i did it in the way you told and after changing this restarted the container:

docker exec -it openhab-openhab-1 grep suggestion /openhab/conf/services/addons.cfg
suggestionFinderMdns = false
suggestionFinderUpnp = false

But it shows on “top -H” still the mass of SocketListener and JmDNS Threads. Im not sure if these 2 parameters are working in the expected way…

But a positive thing: The CPU-Usage is a lot lower now about 15 mins later and so the gui is usable now with this

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
     27 openhab   20   0 6034604   1.5g  25812 S  11.3  19.9  39:14.56 java                                                                                                                                                                   
   2678 root      20   0    9812   3280   2740 R   0.3   0.0   0:00.11 top                                                                                                                                                                    
      1 openhab   20   0    1944    436    372 S   0.0   0.0   0:00.16 tini

This time im seeing no more SocketListener-Threads, but still JmDNS Threads. So it could be that the mass of SocketListener-Threads which were gone now were causing the issue.

Update: SocketListener-Threads are still there again. CPU again high as before. Only from waiting, no change

This sounds like you have another issue. UI should be available after starting openHAB.

OK, im summarizing what i had with 4.1.0 together with docker:

docker exec -it openhab-openhab-1 ps -T -p 200|grep Socket|wc -l
90
docker exec -it openhab-openhab-1 ps -T -p 200|grep JmDNS|wc -l
213

what does the issue cause on my site?

  • unavailability of gui
  • uninstall/ install of addons impossible
  • a lot of fun on 24th december 2023 :slight_smile:

what was tried?

  • disable suggestionfinder → the threadcount does not come down. Seems an issue from usage together with docker and network in hostmode

Update: i downgraded to 4.0.3 and my smarthome works again fine. So anyone which has same problem - downgrade is no problem.
If someone has an idea how to disable the SocketListener-Threads/JmDNS-Threads i can run again the 4.1.0 and can try this…

On 4.0.3 we have these thread-counts to get a relation:

docker exec -it openhab-openhab-1 ps -T -p 560|grep Socket|wc -l
18
docker exec -it openhab-openhab-1 ps -T -p 560|grep JmDNS|wc -l
68

I still think there is another issue with your upgraded container, as you are reporting unavailability of the GUI (MainUI).