Embedded mqtt broker going offline every day

Add-ons Other
1

Hi everyone!

I have been using OpenHAB with embedded MQTT broker for almost 2 years now already, with no major issues until recently.
But now I have MQTT broker crashing and driving me nuts. I have read many posts with no solution for me yet.
And yes i did connect another tasmota device to MQTT two weeks ago, but issues apeared lately in last 4 days.

I still have OpenHAB 2.4.0 on windows 10 pc (64 bit).
CPU Celeron dual core (20 – 70 % load), 8gb ram (40% used)
Embedded MQTT broker 2.4
MQTT binding 2.4.
PC is Running only OpenHAB and Influx DB.

So, when I restart the win 10 computer OpenHAB loads fine and all MQTT devices connect without issues. Also devices remain connected for minimally 1 to 2 hours before reconnecting, nothing abnormal as MQTT devices all use Wi-Fi. Devices reconnect immediately after their reconnect timeout fires.

But sometimes for unknown reason the MQTT broker drops the connection, it is weird that it is likely to happen between 2 and 3 AM, but it happened also during the day, actually it happened 2 times today.
And the broker never starts again until I restart the OpenHAB / pc.

So, I did restore entire OpenHAB directory from one that I backed up in July 2020. From the latest version I only copied conf folder and things.json to the restored OpenHAB folder. Then the embedded broker ran for 2 days straight and now it started crashing again?

I was also working on getting OH 2.5 with same configuration running on Linux peppermint but I had trouble migrating things config from paper UI and mosquito broker kept dropping the connection to some MQTT devices every few seconds. So that may mean something?

Back to the windows machine…

Is the issue with client IDs, probably not, as all devices communicate for hours just fine?

I have about 20 devices connected.
Is it the broker overloaded with too many messages per second? up to 10 messages per second, max message size 350 Bytes. Average message size 50 Bytes.

Is one MQTT device communicating badly and invoking errors?

It seems that OpenHAB kills embedded broker service as it may take to much time to process and in Karaf console in ttop it said timed waiting for MQTT broker when it stopped working.

I had a look at openhab.log and I can’t see any exact reason why broker crashes.

2021-02-04 20:00:02.463 [INFO ] [ansport.mqtt.internal.ClientCallback] - MQTT connection to ‘127.0.0.1’ was lost
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:181) [234:org.eclipse.paho.client.mqttv3:1.2.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[?:?]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92) ~[?:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:133) ~[?:?]
… 7 more
2021-02-04 20:00:02.479 [WARN ] [r.internal.EmbeddedBrokerServiceImpl] - Embedded broker offline
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:181) [234:org.eclipse.paho.client.mqttv3:1.2.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[?:?]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92) ~[?:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:133) ~[?:?]
… 7 more
2021-02-04 20:00:02.479 [WARN ] [.moquette.server.netty.NettyAcceptor] - An InterruptedException was caught while waiting for event loops to terminate…
2021-02-04 20:00:02.479 [WARN ] [.moquette.server.netty.NettyAcceptor] - Forcing shutdown of worker event loop…
2021-02-04 20:00:02.479 [WARN ] [.moquette.server.netty.NettyAcceptor] - Forcing shutdown of boss event loop…
2021-02-04 20:00:02.495 [ERROR] [io.moquette.server.Server ] - Moquette is not started, MQTT message interceptor cannot be removed. InterceptorId=collectmetrics

These logs still don’t tell why it crashed, maybe they tell how it crashed but that only the pro undertands.

But I can’t find any MQTT broker.log file as many posts write about where is it in Linux.

Any ideas why broker is crashing, and maybe possible solutions will be appreciated.
Thanks a lot in advance.
And thanks for your time.
Cheers.
~ Matej

2

Sorry but you are going to get no help what so ever on the embedded MQTT server from OpenHAB 2.4 The embedded MQTT server in 2.4 in Moquette and is considered abandoned. The standard advice is to switch to Mosquitto MQTT server. If you can define an exact problem, the most recent release is a fork Jan was maintaining and by the way is what I’m am currently still running in production but oh well

3

Thanks Andrew_Rowe for your advice, it was super helpful as i finally stopped bothering with embeded broker and i uninstalled it completely. That helped me to decide to use mosquitto.

And than i remembered, i used mosquitto on windows some time ago on my personal machine for testing.

So i installed mosquitto on openhab / windows pc from Download | Eclipse Mosquitto and for now it is all up and running, huray!
Obviously i had to configure new broker in paper UI and reconfigure all devices to use new broker.

I am not shure how stable it will be, but for last 30 minutes it seems to be working fine.

It is actually more stable with less disconections than mosquitto on linux, not shure why, does it have different default config?

But only one mqtt device, that was sending some messages with qos 1 on previous broker and did stay connected to broker for few hours before reconnecting, is now disconecting briefly every few seconds.
It is working thoo but “on the edge of reliability”.

Every few seconds that mqtt device sends some messages, they come through and than it reconects imediately, also all messages apear to be qos 0. Does the broker not accept qos 1 by default and than reconects?

Every idea for why device is posiblly disconecting highly appriciated.

Thanks again!

Cheers
~ Matej

4

Have you confirmed this? That all your devices are identifying themselves to Mosquitto with unique ID’s? Unpredictable things happen if two devices share the same ID…

5

So the solution for failing embedded mqtt broker, was indeed mosquitto broker, installed as service on the pc.

All clients have uniqe IDs, the latest isuue was only overloaded ESP8266 that kept dropping the connection as looptime was 500 - 1200 ms.
When it went over 1000 ms, then a brief disconection occurred.

As i wanted to send too much traffic to broker and display every second.

So i made an optimisation by pausing unecessary serial traffic to Nextion dispaly (connected to ESP) when it is not in use.

Now ESP looptime is 50 - 200 ms and it keeps connection for about an hour, and than it reconnects without noticable interuption.

Thanks to Andrew i finally installed more reliable mqtt broker
And thanks to hafniumzinc i made a spreadsheet for clients with custom client IDs.

Broker still seems to be working fine and devices work without noticable interuptions!

Thanks,

Matej