I’m seeing the same thing. Didn’t happen after the migration to 3 only in the latest sw version. I’m wondering if it can be the rules.
I had the Zulu Azul JDK installed, replaced it for OpenJDK 11 and now the issue seeems to be gone.
After update to the newest Version, it works by me without problems.
My current version is:
openjdk 11.0.10 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+27-CA (build 11.0.10+9-LTS)
OpenJDK Client VM Zulu11.45+27-CA (build 11.0.10+9-LTS, mixed mode)
It not working with update java. After some weeks it happens again.
It happes again. Here the result from Shell:threads --list:
Id │ Name │ State │ CPU time │ Usr time
────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┼───────────┼──────────
1 │ main │ WAITING │ 2820 │ 2510
2 │ Reference Handler │ RUNNABLE │ 13754 │ 9020
3 │ Finalizer │ WAITING │ 18119 │ 15160
4 │ Signal Dispatcher │ RUNNABLE │ 0 │ 0
8 │ Common-Cleaner │ TIMED_WAITING │ 1728 │ 1200
10 │ Active Thread: Equinox Container: 96ace3be-2a71-4f60-9e75-628f6ece24eb │ TIMED_WAITING │ 19809 │ 15190
12 │ Framework Event Dispatcher: org.eclipse.osgi.internal.framework.EquinoxEventPublisher@1cbe147 │ WAITING │ 68 │ 50
13 │ Bundle File Closer │ WAITING │ 177 │ 110
14 │ Start Level: Equinox Container: 96ace3be-2a71-4f60-9e75-628f6ece24eb │ WAITING │ 21538 │ 20490
15 │ Karaf Lock Monitor Thread │ TIMED_WAITING │ 25258 │ 13190
16 │ Karaf Shutdown Socket Thread │ RUNNABLE │ 3 │ 0
27 │ Coordination Timer │ WAITING │ 0 │ 0
28 │ CM Configuration Updater │ WAITING │ 1573 │ 1440
29 │ CM Event Dispatcher │ WAITING │ 0 │ 0
30 │ fileinstall-/var/lib/openhab/etc │ TIMED_WAITING │ 30275 │ 26280
31 │ FileSystemWatchService │ RUNNABLE │ 1 │ 0
33 │ Log4j2-TF-5-Scheduled-1 │ TIMED_WAITING │ 3789 │ 2570
35 │ EventAdminAsyncThread #11 │ WAITING │ 15223978 │ 11456810
36 │ EventAdminAsyncThread #12 │ WAITING │ 15133070 │ 11400180
40 │ fileinstall-/usr/share/openhab/addons │ TIMED_WAITING │ 13161 │ 9600
41 │ FileSystemWatchService │ RUNNABLE │ 0 │ 0
42 │ encryption-1-thread-1 │ TIMED_WAITING │ 11481 │ 7340
43 │ FileSystemWatchService │ RUNNABLE │ 0 │ 0
44 │ SCR Component Actor │ WAITING │ 0 │ 0
52 │ Thread-14 │ RUNNABLE │ 11 │ 10
53 │ sshd-SshServer[1310344]-timer-thread-1 │ TIMED_WAITING │ 15508 │ 12180
57 │ EMF Reference Cleaner │ WAITING │ 57 │ 50
59 │ ForkJoinPool.commonPool-worker-3 │ WAITING │ 475 │ 450
60 │ OH-OSGiEventManager │ TIMED_WAITING │ 58818889 │ 49328450
62 │ OH-startlevel-1 │ TIMED_WAITING │ 11029 │ 10030
71 │ FileSystemWatchService │ RUNNABLE │ 1 │ 0
72 │ Dir Watcher │ WAITING │ 0 │ 0
74 │ ServletEventDispatcher: 1 │ WAITING │ 7 │ 0
75 │ ServletEventDispatcher: 2 │ WAITING │ 6 │ 0
76 │ ServletEventDispatcher: 3 │ WAITING │ 1 │ 0
79 │ Session-HouseKeeper-1ea3346 │ TIMED_WAITING │ 126 │ 100
80 │ qtp4532796-80-acceptor-0@946b3f-0.0.0.0:8443@7550ec{SSL,[ssl, http/1.1]}{0.0.0.0:8443} │ RUNNABLE │ 93 │ 70
83 │ qtp4532796-83-acceptor-0@d20d21-default@1a4f92e{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} │ RUNNABLE │ 39 │ 20
93 │ OH-karaf-addons-1 │ TIMED_WAITING │ 1038 │ 980
94 │ com.google.inject.internal.util.$Finalizer │ WAITING │ 0 │ 0
96 │ pool-9-thread-1 │ WAITING │ 10 │ 10
97 │ Timer-1 │ WAITING │ 0 │ 0
100 │ OH-eventwatcher-1 │ TIMED_WAITING │ 31354349 │ 23751750
101 │ pool-5-thread-1 │ WAITING │ 10346508 │ 6867040
102 │ Timer-2 │ WAITING │ 0 │ 0
103 │ OH-ruleengine-1 │ WAITING │ 86 │ 80
104 │ Timer-3 │ WAITING │ 0 │ 0
105 │ Timer-4 │ WAITING │ 0 │ 0
106 │ Timer-5 │ WAITING │ 0 │ 0
107 │ SocketListener(sh-onflow-ch.local.) │ RUNNABLE │ 506596 │ 475530
108 │ JmDNS(sh-onflow-ch.local.).Timer │ TIMED_WAITING │ 49716 │ 43440
109 │ JmDNS(sh-onflow-ch.local.).State.Timer │ TIMED_WAITING │ 259 │ 220
112 │ Timer-6 │ WAITING │ 405 │ 280
114 │ lsp-1 │ RUNNABLE │ 2 │ 0
115 │ Timer-8 │ WAITING │ 0 │ 0
116 │ Timer-9 │ WAITING │ 0 │ 0
119 │ Timer-10 │ WAITING │ 0 │ 0
120 │ Timer-11 │ WAITING │ 0 │ 0
121 │ ServletModel-17-121 │ RUNNABLE │ 0 │ 0
122 │ ServletModel-17-122 │ RUNNABLE │ 0 │ 0
123 │ ServletModel-17-123 │ TIMED_WAITING │ 163 │ 140
124 │ ServletModel-17-124 │ TIMED_WAITING │ 156 │ 80
125 │ ServletModel-17-125 │ TIMED_WAITING │ 166 │ 120
126 │ ServletModel-17-126 │ TIMED_WAITING │ 157 │ 50
127 │ ServletModel-17-127 │ TIMED_WAITING │ 156 │ 60
128 │ ServletModel-17-128 │ TIMED_WAITING │ 159 │ 110
129 │ upnp-main-1 │ TIMED_WAITING │ 21328 │ 17050
130 │ upnp-main-2 │ RUNNABLE │ 5842 │ 5200
132 │ upnp-main-4 │ RUNNABLE │ 0 │ 0
135 │ jupnp-jetty-client-135 │ TIMED_WAITING │ 5858 │ 4510
136 │ jupnp-jetty-client-136 │ RUNNABLE │ 5861 │ 4520
141 │ OH-httpClient-common-141 │ RUNNABLE │ 264129 │ 193340
143 │ OH-httpClient-common-143 │ TIMED_WAITING │ 260963 │ 191340
144 │ OH-httpClient-common-144 │ RUNNABLE │ 1 │ 0
145 │ OH-httpClient-common-145 │ RUNNABLE │ 0 │ 0
146 │ OH-httpClient-common-146 │ TIMED_WAITING │ 269592 │ 197440
147 │ OH-httpClient-common-147 │ TIMED_WAITING │ 257259 │ 188950
148 │ OH-httpClient-common-148 │ TIMED_WAITING │ 276158 │ 203890
149 │ OH-httpClient-common-149 │ RUNNABLE │ 273795 │ 201200
150 │ OH-httpClient-common-150 │ TIMED_WAITING │ 262796 │ 193520
151 │ OH-httpClient-common-151 │ TIMED_WAITING │ 276326 │ 202590
152 │ OH-usb-serial-discovery-linux-sysfs-1 │ TIMED_WAITING │ 259471 │ 150570
158 │ OH-scriptwatcher-1 │ TIMED_WAITING │ 1230 │ 960
159 │ OH-rulesRefresher-1 │ WAITING │ 1 │ 0
161 │ FileSystemWatchService │ RUNNABLE │ 0 │ 0
162 │ OH-thingHandler-1 │ TIMED_WAITING │ 1156551 │ 1087950
166 │ OH-thingHandler-2 │ TIMED_WAITING │ 1173575 │ 1104650
167 │ HttpClient@1c9926c-scheduler │ TIMED_WAITING │ 77429 │ 62780
168 │ OH-thingHandler-3 │ TIMED_WAITING │ 1167700 │ 1098590
172 │ RxSchedulerPurge-1 │ TIMED_WAITING │ 26928 │ 22470
173 │ RxCachedWorkerPoolEvictor-1 │ TIMED_WAITING │ 287 │ 220
175 │ OH-thingHandler-4 │ TIMED_WAITING │ 1186064 │ 1116340
176 │ OH-thingHandler-5 │ WAITING │ 1169801 │ 1100530
177 │ nioEventLoopGroup-2-1 │ RUNNABLE │ 34863506 │ 20253790
179 │ RXTXPortMonitor(/dev/ttyUSB0) │ TIMED_WAITING │ 97 │ 30
180 │ OH-binding-rfxcom:bridge:a9cfc67d79 │ RUNNABLE │ 306025 │ 197310
181 │ RxComputationThreadPool-1 │ WAITING │ 11 │ 10
182 │ items-3 │ TIMED_WAITING │ 10814057 │ 6240780
183 │ OH-rule-LocalDiskSpace-1 │ WAITING │ 113383 │ 106910
184 │ pool-14-thread-1 │ TIMED_WAITING │ 339181 │ 275200
189 │ OH-rule-EGWohnzimmerLichtOFF-1 │ WAITING │ 2 │ 0
191 │ Thread-63 │ TIMED_WAITING │ 22 │ 20
192 │ Thread-64 │ TIMED_WAITING │ 25 │ 20
193 │ Thread-65 │ TIMED_WAITING │ 24 │ 0
194 │ Thread-66 │ TIMED_WAITING │ 28 │ 20
195 │ Thread-67 │ TIMED_WAITING │ 27 │ 0
196 │ Thread-68 │ TIMED_WAITING │ 29 │ 20
197 │ Thread-69 │ TIMED_WAITING │ 25 │ 20
198 │ Thread-70 │ TIMED_WAITING │ 24 │ 20
199 │ Thread-71 │ TIMED_WAITING │ 29 │ 10
200 │ Thread-72 │ TIMED_WAITING │ 21 │ 20
201 │ Thread-73 │ TIMED_WAITING │ 20 │ 10
202 │ Thread-74 │ TIMED_WAITING │ 26 │ 10
203 │ JmDNS pool-10-thread-1 │ WAITING │ 41119 │ 40360
204 │ RxComputationThreadPool-2 │ WAITING │ 28 │ 20
205 │ RxComputationThreadPool-3 │ RUNNABLE │ 155465094 │ 147351820
206 │ RxComputationThreadPool-4 │ WAITING │ 8 │ 0
207 │ :CoapEndpoint-UDP-0.0.0.0:5683#1 │ TIMED_WAITING │ 2206719 │ 2149540
208 │ UDP-Receiver-0.0.0.0/0.0.0.0:5683[0] │ RUNNABLE │ 126893 │ 95550
209 │ UDP-Sender-0.0.0.0/0.0.0.0:5683[0] │ WAITING │ 0 │ 0
210 │ :CoapEndpoint-UDP-0.0.0.0:0#1 │ TIMED_WAITING │ 2167 │ 2040
211 │ UDP-Receiver-0.0.0.0/0.0.0.0:0[0] │ RUNNABLE │ 21 │ 10
212 │ UDP-Sender-0.0.0.0/0.0.0.0:0[0] │ WAITING │ 74 │ 60
215 │ OH-rule-EGBueroLichtOFF-1 │ WAITING │ 4 │ 0
216 │ pool-1-thread-1 │ WAITING │ 65 │ 60
258 │ OH-rule-HeizungBueroAUS-1 │ WAITING │ 89 │ 80
259 │ OH-rule-HeizungBueroEIN-1 │ WAITING │ 95 │ 80
326 │ OH-httpClient-common-326 │ TIMED_WAITING │ 251609 │ 184420
342 │ OH-httpClient-common-342 │ TIMED_WAITING │ 282055 │ 206940
450 │ OH-rule-EGBueroLichtON-1 │ WAITING │ 2 │ 0
1960 │ Connector-Scheduler-7550ec │ TIMED_WAITING │ 3802 │ 3360
2045 │ Connector-Scheduler-1a4f92e │ WAITING │ 8 │ 0
2514 │ OH-httpClient-common-2514 │ TIMED_WAITING │ 258439 │ 188970
6497 │ OH-rule-EGWohnzimmerLichtOn-1 │ WAITING │ 3 │ 0
6825 │ OH-rule-OGGangButtonTopShort-1 │ WAITING │ 2576 │ 2520
7430 │ HttpClient@59f2b8-scheduler │ WAITING │ 726 │ 340
9806 │ OH-rule-EGKellerWasserpumpeOFF-1 │ WAITING │ 1 │ 0
14859 │ OH-rule-EGKellerWasserpumpeOn-1 │ WAITING │ 1 │ 0
15835 │ OH-rule-OGGangButtonBottomShort-1 │ WAITING │ 1493 │ 1490
15840 │ OH-rule-OGGangBottomTopLong-1 │ WAITING │ 13 │ 10
18218 │ OH-rule-OGGangButtomBottomLong-1 │ WAITING │ 13 │ 10
27954 │ OH-httpClient-common-27954 │ TIMED_WAITING │ 112999 │ 82420
27955 │ OH-httpClient-common-27955 │ TIMED_WAITING │ 99239 │ 72470
27956 │ OH-httpClient-common-27956 │ TIMED_WAITING │ 110284 │ 80890
40983 │ OH-rule-KlingelLastUpdate-1 │ WAITING │ 2719 │ 2660
42202 │ qtp4532796-42202 │ RUNNABLE │ 1045 │ 1020
42891 │ jupnp-jetty-client-42891 │ RUNNABLE │ 1392 │ 970
45492 │ jupnp-jetty-client-45492 │ TIMED_WAITING │ 54 │ 30
45494 │ jupnp-jetty-client-45494 │ TIMED_WAITING │ 53 │ 40
49729 │ OH-scheduler-16 │ TIMED_WAITING │ 24 │ 0
50193 │ qtp4532796-50193 │ TIMED_WAITING │ 426 │ 420
51738 │ qtp4532796-51738 │ RUNNABLE │ 970 │ 900
3768302 │ qtp4532796-3768302 │ RUNNABLE │ 31 │ 0
6616487 │ items-35242 │ TIMED_WAITING │ 44 │ 10
6626375 │ OH-common-8928 │ TIMED_WAITING │ 49 │ 40
6727545 │ OH-discovery-7596 │ TIMED_WAITING │ 2 │ 0
6744015 │ qtp4532796-6744015 │ RUNNABLE │ 1 │ 0
6744017 │ qtp4532796-6744017 │ TIMED_WAITING │ 16 │ 10
6744019 │ qtp4532796-6744019 │ TIMED_WAITING │ 179 │ 170
6744021 │ qtp4532796-6744021 │ TIMED_WAITING │ 38 │ 20
6744022 │ qtp4532796-6744022 │ TIMED_WAITING │ 36 │ 30
6744023 │ qtp4532796-6744023 │ TIMED_WAITING │ 99 │ 90
6744993 │ qtp4532796-6744993 │ TIMED_WAITING │ 22 │ 10
6757195 │ qtp4532796-6757195 │ TIMED_WAITING │ 3 │ 0
6827312 │ qtp4532796-6827312 │ TIMED_WAITING │ 0 │ 0
6827313 │ qtp4532796-6827313 │ TIMED_WAITING │ 0 │ 0
6832413 │ OH-discovery-7601 │ TIMED_WAITING │ 0 │ 0
6841789 │ OH-common-8942 │ TIMED_WAITING │ 12 │ 10
6874727 │ OH-common-8943 │ TIMED_WAITING │ 2 │ 0
6878189 │ OH-common-8944 │ TIMED_WAITING │ 6 │ 0
6891904 │ OH-discovery-7603 │ TIMED_WAITING │ 0 │ 0
6908218 │ OH-discovery-7604 │ TIMED_WAITING │ 0 │ 0
6911422 │ OH-discovery-7605 │ TIMED_WAITING │ 0 │ 0
6917465 │ OH-common-8945 │ TIMED_WAITING │ 0 │ 0
6926663 │ sshd-SshServer[1310344]-nio2-thread-1 │ WAITING │ 339 │ 300
6926827 │ Thread-421 │ TIMED_WAITING │ 0 │ 0
6930193 │ sshd-SshServer[1310344]-nio2-thread-2 │ WAITING │ 724 │ 650
6930360 │ org.apache.karaf.shell.ssh.SshTerminal@1402f8e input pump thread │ WAITING │ 1 │ 0
6930516 │ Karaf ssh console user openhab │ WAITING │ 380 │ 370
6930543 │ pipe-Shell:threads --list │ RUNNABLE │ 146 │ 110
6930545 │ job controller 1 │ WAITING │ 18 │ 10
6930584 │ pool-16-thread-3 │ TIMED_WAITING │ 32 │ 10
6930586 │ pool-16-thread-4 │ TIMED_WAITING │ 483 │ 440
6932206 │ OH-eventexecutor-6878823 │ RUNNABLE │ 0 │ 0
6932207 │ OH-eventexecutor-6878824 │ RUNNABLE │ 0 │ 0
I don’t know what I can do at next? Please help me?
Have the same issue and as a workaround check syslog every 3 minutes by a cronjob. The script looks like this:
#!/bin/bash
exec >>/var/log/openhab/restart.log
exec 2>&1
tail -20 /var/log/syslog | grep -q LinkedBlockingQueue
[ $? -ne 0 ] && exit 0
echo `date`: Restarting openhab
/usr/sbin/service openhab restart
echo `date`: openhab restarted
tail -2 /var/log/openhab/restart.log | mailx -s 'Openhab restarted' your@address.here
exit 0
Crontab entry:
*/3 * * * * /root/bin/checkopenhab
There is a second thread for the same topic here:
The common part seems to be ARM architecture and azul java. I try switching to openjdk as @RhavoX suggested.
To reiterate my statement above, why don’t you switch to openHABian ? It takes care of all these issues. Sure it’s no guarantee there’s no bugs but for sure it’s way more intelligent and less of a hassle to everybody to maintain a single setup rather than to debug everybody’s own specific setup.
“Anything else ARM based such as ODroids, OrangePis and the like may work or not.”
My instance runs on a odroidhc1 …
I’ve switched now to Openhabian with same Configuration. Now we will see it is this the problem or something inside Openhab.
It happens again. Now with openhabian.
What can I do as next to find the problem?
And it happens again. What can I do to find the problem or deliver more informations?
Attached the openhab.log with error.openhab.log (81.9 KB)
It looks like an error with influxDB. But openhab should not stop work when it happes?
I changed on /etc/influxdb/influxdb.conf the following values:
[data]
query-log-enabled = false
[http]
flux-log-enabled = false
log-enabled = false
suppress-write-log = true
Thats all what I modify on influx. Installation of Influx was over Openhabian.
The error in the log mainly looks to me as if influxdb is not running at that moment or using a different port than 8086:
...
Caused by: java.net.ConnectException: Failed to connect to /127.0.0.1:8086
...
Caused by: java.net.ConnectException: Connection refused (Connection refused)
...
I don’t think this is related to the syslog errors problem.
At least it looks like it doesn’t happen that often when using plain openjdk instead of Zulu Java, so it might be related to some strange behaviour of the JRE itself. openHAB 3 runs for one week now for me without showing the problem again.
Can you confirm when your syslog starts going mad, does the timestamp tally with the openhab.log?
And it happens again yesterday.
Openhab-Logs:
- openhab.log begins with first entry after restart
- openhab.log.5 ends some days before (Mrz 26, 17:49)
- openhab.log.6 is missing (not exists)
- openhab.log.7 is arround 5MB big and begings with:
2021-03-28 18:00:16.802 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
2021-03-28 18:00:16.804 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
2021-03-28 18:00:16.804 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
2021-03-28 18:00:16.805 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
2021-03-28 18:00:16.805 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
2021-03-28 18:00:16.806 [WARN ] [.incoming.MqttIncomingPublishService] - QoS 0 publish message dropped.
and ends with:
2021-03-28 18:00:56.981 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler HomieThingHandler of thing mqtt:homie300:6afa898ba3:egentreedoorbell tried updating channel doorbell#ring although the handler was already disposed.
2021-03-28 18:00:56.981 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler HomieThingHandler of thing mqtt:homie300:6afa898ba3:egentreedoorbell tried updating channel doorbell#ring although the handler was already disposed.
2021-03-28 18:00:56.981 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler HomieThingHandler of thing mqtt:homie300:6afa898ba3:egentreedoorbell tried updating channel doorbell#ring although the handler was already disposed.
2021-03-28 18:00:56.981 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler HomieThingHandler of thing mqtt:homie300:6afa898ba3:egentreedoorbell tried updating channel doorbell#ring although the handler was already disposed.
2021-03-28 18:00:56.982 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler HomieThingHandler of thing mqtt:homie300:6afa898ba3:egentreedoorbell tried updating channel doorbell#ring although the handler was already disposed.
Between are all the same kind of messages (many of them).
In the syslog is also nothing between Mar 26, 17:50:49 and MAR 28, 18:01.01. Also the same gap as on openhab logs. And also the same gap on events.log from openhab.
I think (not 100% sure) but my restart (after dedecting that openhab items not response) was around MAR, 28, 18:00.
On /opt/zram/zram2/upper/openhab I see the missing openhab.log.6 with 0 bytes:
What can I do as next? I have no clue where I can search as next to solve the problem.
Thank you very mutch for your help.
And it happens again. Now I was able to take a look into syslog before restart. Same java exceptions.
Please help me.
I don’t know what I can do now.
It happens again and again.
I tryed the following setting with no luck:
org.eclipse.smarthome.threadpool:ruleEngine=100
The openhab.log.6 entry in one of your previous posts is not a normal file;
it seems to be a character device …; besides that it is not owned by openhab:openhab
Do the repeated occurences of your problem also show that one of the files seem to be a character device ?
What happens when the log tries to roll over, and shuffle existing historic logs down one, I wonder?
I think the problem of missing openhab.log.6 is, that the /var/log is full (because of syslog errors) and then i reboot the system. My guess is, that zram is then not able to write down the changes.
I don’t know.