MQTT socket Error when Wifi reconnects, then disconnection and "dead" reconnection of broker

  • Platform information:
    • Hardware: Rpi4b
    • Raspian 10 (buster), linux 5.10.11
    • Java 11.0.9.1
    • openHAB version 3:
  • mqtt server/broker: mosquitto on different computer/sbc, connected via LAN to openhab
  • several homie or generic mqtt clients (switches etc.) connect to the mosquitto broker via WLAN, ESP8266 based, afaik
  • all clients use the openhab user credentials to connect to the broker, but have different IDs

Issue: I have my wifi turned off when I am not at home or at night. When I return and the Wifi is turned on again, the esp8266 mqtt clients (switches and stuff) connect to the broker again, which is what they are supposed to do.
However, for some reason the connection between the broker and openhab is compromised and openhab disconnects with a lot of errors thrown. Then it attempts to reconnect, but this connection is “dead”. Nothing can be controlled afterwards.

Workaround: I sense when the Wifi connection is established again and run a DSL rule with an “executeCommandLine” action that sends bundle:restart org.openhab.binding.mqtt to the Openhab CLI.
However, this does not feel right :wink:

moquitto log on the computer that on which the broker is running:

1613760960: New connection from 192.168.1.45 on port 1883.
1613760960: New client connected from 192.168.1.45 as shelly1-500291F0DABF (c1, k60, u'openhab').
....
1613760962: New connection from 192.168.1.48 on port 1883.
1613760962: New client connected from 192.168.1.48 as Homie-esp-living (c1, k15, u'openhab').
1613760962: Socket error on client b11e476e-ec84-4b73-bcf9-1edbf34e5515, disconnecting.
1613760972: New connection from 192.168.1.30 on port 1883.
1613760972: New client connected from 192.168.1.30 as b11e476e-ec84-4b73-bcf9-1edbf34e5515 (c1, k60, u'openhab').

openhab log:

 2021-02-19 19:56:02.568 [ERROR] [ler.disconnect.MqttDisconnectHandler] - Exception while disconnecting.
 java.lang.IllegalStateException: null
      at com.hivemq.client.internal.util.collections.ChunkedArrayQueue$ChunkedArrayQueueIterator.remove(ChunkedArrayQueue.java:210) ~[?:?]
      at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingPublishService.drain(MqttIncomingPublishService.java:131) ~[?:?]
      at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingPublishService.onPublish(MqttIncomingPublishService.java:92) ~[?:?]
     at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingPublishService.onPublishQos0(MqttIncomingPublishService.java:65) ~[?:?]
     at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingQosHandler.readPublishQos0(MqttIncomingQosHandler.java:115) ~[?:?]
     at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingQosHandler.readPublish(MqttIncomingQosHandler.java:103) ~[?:?]
     at com.hivemq.client.internal.mqtt.handler.publish.incoming.MqttIncomingQosHandler.channelRead(MqttIncomingQosHandler.java:92) ~[?:?]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [bundleFile:4.1.42.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [bundleFile:4.1.42.Final]
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [bundleFile:4.1.42.Final]
     at com.hivemq.client.internal.mqtt.handler.publish.outgoing.MqttOutgoingQosHandler.channelRead(MqttOutgoingQosHandler.java:317) [bundleFile:?]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [bundleFile:4.1.42.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [bundleFile:4.1.42.Final]
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [bundleFile:4.1.42.Final]
     at com.hivemq.client.internal.mqtt.handler.ping.MqttPingHandler.channelRead(MqttPingHandler.java:82) [bundleFile:?]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [bundleFile:4.1.42.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [bundleFile:4.1.42.Final]
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [bundleFile:4.1.42.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) [bundleFile:4.1.42.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315) [bundleFile:4.1.42.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429) [bundleFile:4.1.42.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) [bundleFile:4.1.42.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [bundleFile:4.1.42.Final]
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [bundleFile:4.1.42.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [bundleFile:4.1.42.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [bundleFile:4.1.42.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [bundleFile:4.1.42.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [bundleFile:4.1.42.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [bundleFile:4.1.42.Final]
     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [bundleFile:4.1.42.Final]
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [bundleFile:4.1.42.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [bundleFile:4.1.42.Final]
   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [bundleFile:4.1.42.Final]
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [bundleFile:4.1.42.Final]
   at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [bundleFile:4.1.42.Final]
   at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [bundleFile:4.1.42.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [bundleFile:4.1.42.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?] 

<same java error repeats about 15 times>

2021-02-19 19:56:12.553 [INFO ] [.reconnect.PeriodicReconnectStrategy] - Try to restore connection to '192.168.1.31'. Next attempt in 60000ms
2021-02-19 19:56:12.607 [INFO ] [.transport.mqtt.MqttBrokerConnection] - Starting MQTT broker connection to '192.168.1.31' with clientid b11e476e-ec84-4b73-bcf9-1edbf34e5515

After this connection, the Things in the UI show as online and there is nothing suspicious, however, they are dead and cannot be controlled. Either the mqtt bundle on the openhab needs to be restarted, or the mosquitto service on the other computer (this works as well).
Anybody has an idea what’s going on?
Again the openhab server and the mqtt broker are connected over LAN (i.e. Cat5 cables and router), not Wlan.