I rely heavily to input via MQTT to my openHAB instance (v3.3-release on RaspberryPI 4 / 4GB).
last night without any information in the logs openHAB stopped using incoming MQTT-payloads. Not one of the mqtt-Things was offline - neither was the MQTT bridge. What didn’t stop was sending out item states via MQTT. After restarting the binding via console (bundle:restart org.openhab.binding.mqtt) it worked again.
So, two questions:
happened this to anyone else? I’ve got about >400 items relying on MQTT either way, but this seems nevertheless a non-perfomance issue…
is there a way to tell in a running OH3, thnat MQTT-payloads don’t get processed?
as said: all things were green ok, I could send out a warning, if the intervals of one (or all) MQTT-based items won’t update, …
Got any other Items that updated as expected during the problem period? (You might be looking at a generally stuffed openHAB event queue, not MQTT specific)
Only if you know you were expecting at least one message i.e. that some of your targets send periodic reports.
Then it becomes trivial and this is what ‘expire’ feature was intended for. There are a number of ways to exploit that and other similar strategies in the forum.
Side note - when someone says “I’ve got 500 OH3 Items and some performance problem” I generally start probing about if they still have persistence service settings at default (“everything”). Have we done that already?
Why the duplication?
“not a problem” does not mean it is not thrashing your host. It’s only a Pi.
Consider that as your Maria is external, that’s an amount of HTTP traffic … competing with the HTTP traffic required for MQTT. I wouldn’t dismiss this so readily.
A next useful step might be to see if you have Mosquitto logs for that period - did the messages get to the broker to begin with? Has the broker anything to say about unsubscriptions? etc.
Canb you tell us more about that, “elsewhere”? You have two openHABs perhaps? Make sure each has a unique ID when connecting to broker (else the broker cannot forward messages to both)
Not to quibble, but neither of these are HTTP. Both are TCP/IP but the protocol for MariaDB is JDBC and the protocol for MQTT is, well, MQTT.
I just want to head off the chance that someone comes along and reads this and thinks they can use the HTTP binding for these.
Your point is still valid though.
@binderth, I suspect we’ll need some trace level logs from the MQTT binding when this behavior occurs. I’ve never experienced this nor have I see reports of similar behavior reported.
Elsewhere as in Node-Red, my Wallbox and my evcc could use the incoming MQTT-topics/payloads meanwhile.
hmm… got some, but they just show no incoming messages at all - until the binding restart
But what I just saw, there’s some double and triple entries for “Providing state description for channel…” - which aren’t even linked to an MQTT-item?
like here:
2022-12-13 09:25:49.444 [TRACE] [.MqttChannelStateDescriptionProvider] - Providing state description for channel modbus:data:kostalInverterInfo:kostalBatteryChargeDischarge:number
2022-12-13 09:25:49.447 [TRACE] [.MqttChannelStateDescriptionProvider] - Providing state description for channel modbus:data:kostalInverterInfo:kostalBatteryChargeDischarge:number
2022-12-13 09:25:49.451 [TRACE] [.MqttChannelStateDescriptionProvider] - Providing state description for channel modbus:data:kostalInverterInfo:kostalBatteryChargeDischarge:number
So, let’s see:
the modbus:data:kostalInverterInfo:kostalBatteryChargeDischarge’s “Value as Number”-channel is linked to
the item KOS_BatteryChargeDischarge only
the item KOS_BatteryChargeDischarge is also only linked to that channel - no MQTT involved at all?
and also
the bridge for modbus:data:kostalInverterInfo:kostalBatteryChargeDischarge is modbus:poller:kostal:kostalInverterInfo:
How exactly did you put the binding into trace level logging? I think the “provided state description” stuff is a distraction. Since that appears to be the only thing logged at the trace level, I’d bump the level up to debug. I can’t imagine anything to deal with state description would prevent the binding from receiving and processing messages.
Beyond that indeed I don’t see anything in the logs, but there’s a lot of noise so something may have been missed. But I also don’t see any logs from the receipt of a message.
log:set TRACE org.openhab.binding.mqtt it was.
And yes, very noisy indeed.
But strangely enough, some MQTT-items (BB_EltakoImpulse e.g.) did get updated… Which I didn’t get at first. Sheeesh.
Off-topic - for the State Description sideshow, try purging dead links initially.
Also possibly off-topic but performance related; non-trivial Modbus configurations with default settings can provide a throughput challenge for persistence services, because of the Item update rate.
I think I may experience the same thing as you, but it’s only with two MQTT devices and it doesn’t happen suddenly. I have:
Three Tasmota-flashed power bars
An octoPi server
The SleepAsAndroid app on my Android phone
My Tasmota power bars always work, but I’ve noticed sometimes that OH stops getting updates from octoPi and SleepAsAndroid, even though messages are still received by the broker running on my OH server (confirmed with MQTT Explorer). My quick fix is to remove and then re-add the things (by commenting them out in my things file, saving, and uncommenting).
Anecdotally, I think this only happens after I’ve restarted OH (which is infrequent enough that I haven’t worried about it). The only significant difference between the Tasmota and octoPi/SleepAsAndroid things is that I don’t have any command channels in the latter group. I just receive state updates for them.
I initially thought that this might be due to not using Last Will and Testament, but all of the devices can still send messages to the broker–it’s only openHAB that stops updating…and only for some devices. Saying that, I don’t have a good grasp of LWT (which kind of makes my head spin).
I usually stick to snapshots, have hundreds of mqtt items, and they are usually very chatty (e.g. motion sensors). I haven’t noticed something like this on my system, but since I tinker a lot, my openhab gets restarted at least once a week, and often many times a day.
Did this issue happen after a prolonged uptime? Can you reproduce it?
Sorry, didn’t quite get this. What do you mean with “default settings”?
Didn’t know that dead links tipp, but only a bunch of orphaned amazonechocontrols found - but purged now, thanks.
My openhab is running for 5 days straight now. Had to reboot for some OS update. But… my mqtt-Things that stopped had only receiving channels. So this could be an indicator. The LWT stuff is active, though
I’ve never had this issue before. My Pi usually stays on like forever and only gets rebooted if needed. Openhab gets restarted of there’s some error, but last week I didn’t change a thing, and Pi/openHAB were up since 5 days now. Today I only restarted the MQTT-binding.
If you want to follow up on performance tuning Modbus (which can impact other sub-systems, it’s very capable of generating buckets of Item updates) see
For the MQTT stuff, you’ve got something of a job to pin it down further.
There’s no suggestion that the link between openHAB and your broker is in trouble, the clues say it remains alive.
(Where is your broker living? Which one is it?)
It looks like either
Broker has ‘forgotten’ OH subscriptions, or lost pathway to push them.
or
openHAB is forgetting to process them, maybe because it has forgotten channels with matching topics.
Neither are known problems.
I think you might need to be debug logging io.mqtt.transport or something to see incoming topic messages .
My bet is, OH had some issues with the subscribed topics and didn’t mean to share that in the logs somehow. As I said, the topics were sending, the broker “Mosquitto 2.0.15” dockerized in my Synology.
No issues in the logs at that time - all topics (including the ones missing in OH3) were sent as expected.
ok - I’ll remember that for next time - hopefully it was a one-timer!
It’s not really complicated. When an MQTT client registers on the Broker, it can provide a LWT topic and message. When this happens, the broker will set up a heart beat between itself and the client. If the client fails to respond to the heartbeat, the broker will publish the LWT message to the LWT topic, thereby alerting other clients that this client has gone offline.
Put in less technical terms, the client tells the broker “if you don’t hear from me, publish this so everyone knows I’m gone.”
The LWT is almost certainly not related to this behavior.
The thing is, we haven’t seen that yet. The broker could have pushed some messages to elswhere, but forgotten where openHAB lives or that OH was subscribed for the same topics.
(Consider something like your DNS messing up could give this effect, and still allow OH to publish to MQTT)
Hence the interest on whether the missing messages truly turn up at openHAB, to guide which service to even look into.
“My mail hasn’t come, the neighbours did” - maybe the dog ate it, maybe the postman lost your mail. If you can find out if the letterbox opened, you’ll get a better idea who to blame.
If it happens again restart your MQTT server see if that helps. Then you know at least where the error is happening. For me restarting mqtt server helped which made me realize that I misconfigured mosquito.
