[SOLVED] Properly fetch MQTT states

rkrisi · November 28, 2018, 6:27am

Hi all,

I have a problem in my setup. When openhab restarts, the states of MQTT states are not fetched. So for example the light is on, but the switch in OH is OFF.
I have tried MQTT retain, it is not a good option, because it turns back to the old state (when mosqitoo was running). I have also tried mapdb, but that also sends a command(?!), not just updates the state.
How could I do it properly? So after restart it should just check the current state of MQTT devices.

Thanks!

Dim · November 28, 2018, 6:40am

Are you restarting your MQTT Broker as well at the same time ? (most likely running on the same host as OH2). If yes, then neither OH2 not the Broker will pick up the latest true state from your remote MQTT endpoints.

You don’t need to reboot the entire host when you make changes to OH2 that require a restart. Just restart the OH2 service (if really needed).

The proper way is to configure your MQTT Bound Item to subscribe to the correct state topic, keep your Broker alive and retain enabled (optional).

The next time OH2 service will startup, it will receive the retained message from the topic.

If you don’t use retain, you will need to wait for the endpoint to publish its state at a later time (or push a change via a Command from OH2).

(if I remember well) MapDB restoreOnStartup does not issue Commands, it updates the State of the Items to the last stored value.

rkrisi · November 28, 2018, 7:53am

I’m not the one who is restarting the broker
If I have to, I just restart the service. However if I have to restart the whole RPi, that also works because it restarts quickly, so hopefully no states are changed in that 2-3 min.
The problem is the error which I commented in the M6 post. Openhab freezes, nothing works then (it looks like it blocks main threads…), so the broker also don’t work. And it stays in that state for a longer period of time, so the next time when OH is restarted by the service, it pulls wrong states from the broker, changing the state of the light.

Dim · November 28, 2018, 7:54am

you need to address the root-cause then (OH2 service freezing up)… let’s continue the discussion over here

rkrisi · November 28, 2018, 7:55am

Yes! Thanks! Until then I just disabled retain, so might lights won’t change state suddenly

rkrisi · November 28, 2018, 9:32am

So it seems that was just a temporary problem…
So about retain… How does it properly work? Because after disabling Retain on the devices, it still retained them, because it was defined in the mqtt cfg to retain… so how does it work? I just need to enable retain in mqtt.cfg? Why the devices has a Retain ON/OFF?

Dim · November 28, 2018, 9:35am

retain can be set: globally on the Broker config or on the publisher message

read more here: https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages

rkrisi · November 28, 2018, 9:41am

Thanks! So if I want to use it on all MQTT devices then I just need to define it in the config and every(?) message sent on MQTT is retained…
However what happens if the broker lose connection to a device and then after a little time reconnects (and the current state has changed since its disconnected) it will revert the state. There is no solution for ‘pulling’ state from a device? I mean retain is good, but maybe best for devices which are just listening to different topics and not publishing info (Like in the example, so it will get the temperature immediately and it can display the temp - for example). But for light switches, it should prioritize the device current state, because it is (might be) more relevant than the last state stored by MQTT or any other database…

vzorglub · November 28, 2018, 9:50am

I my own opinion, retain is a pain in the ***
I use restoreOnStartUp with MapDB for OH restart
If the broker need a reboot (Different pi…) then when I reboot it OH reconnects within 30s or so.
The devices states will be updated the next time they publish.

Dim · November 28, 2018, 9:50am

Yes, in your mosquitto.cfg (if using Eclipse Mosquitto as the MQTT Broker)

It also depends on the endpoint device.

Some endpoint devices (mqtt clients) will publish their status when they will re-establish a broker connection (well… actually… most won’t do this).

I think not… I maybe wrong.

rkrisi · November 28, 2018, 9:53am

Yes I thought the same, but I didn’t want to write this, because I thought I’m missing something on this Retain.
This is what I want to use now. So if I use MapDB it will just ‘postUpdate’ the states? So it will not change state right?
So for example if the OH goes down and the last state stored in mapdb is OFF and during the downtime it becomes ON, after booting up OH it will just update the switch to OFF (however it is ON), but won’t send an OFF command, right?
I think this is the best way for MQTT switches, because your lights won’t change state unpredictically and the state will become 100% right after a state change.

rkrisi · November 28, 2018, 9:55am

Thanks for the clearing this for me!

vzorglub · November 28, 2018, 9:59am

Correct

Now if you want to use retain for some devices you just need to define 2 brokers in the mqtt.cfg.
Both connect to the same broker but one with retain enabled.

For example:

broker.url=tcp://192.168.0.34:1883
brokerretain.url=tcp://192.168.0.34:1883

# Optional. True or false. Defines if the broker should retain the messages sent to
# it. Defaults to false.
broker.retain=false
brokerretain.retain=true

And in the item’s binding definition use brokerretain instead of broker

But I am still trying to think of a use for retain in an Home Automation environment.

Dim · November 28, 2018, 10:01am

one more additional clarification here:

the sequence is: OH2 starts up → Item state = NULL → restoreOnStartup → Item state = OFF → MQTT subscription from Item binding config → Item state = whatever the broker reports (if message was retained)

The next time the client (endpoint device) will publish its state, everybody will be “synchronized” in the chain (OH2<->Broker<->Device)

edit:

Note: This only applies for the outbound (publish) messages originating from OH2 to the Broker.
It sets the retain flag to true for all MQTT messages that OH2 (client) sends to this Broker alias.
It doesn’t affect the publishing of the remote client (the endpoint device, e.g. a SOnOff)

I also do NOT use retain anywhere… it’s a PITA like Vincent correctly wrote

Dim · November 28, 2018, 10:16am

look how big of a PITA is retain:

I have an ESP controlling my garage gate
I had set retain on OH2 MQTT binding config (as Vincent wrote above) and my command was retained in the MQTT Broker.

The next time the ESP client lost connection (due to a power failure) it would reconnect and receive the retained command… resulting to opening the freaking gate

So… if someone wanted to open my gate… he could easily toggle the mains power supply outside my house (the power meter is not secured and has an on/off switch)

rkrisi · November 28, 2018, 10:23am

Yes! Thats the main problem I found. I dont know one use-case where this can be useful. First I wanted to use for ‘syncing’ states but as it seems this is not for that (thougth the same after enabling but I give it a try).
I dont know what is the alternative use of MQTT (as far as I know it is mainly for home assistant systems), but there might be useful this behaviour.

Anyway thanks for both of you to helping me understand MQTT better…

vzorglub · November 28, 2018, 10:41am

Now if you want to clear the retained topics in mosquitto there are 2 solutions:

Clear the mosquitto.db:

sudo service mosquitto stop
sudo rm /var/lib/mosquitto/mosquitto.db
sudo service mosquitto start
This will clear ALL the retained messages

Send an empty payload to the retained topic
This will clear only the topics selected

Dim · November 28, 2018, 11:27am

by the way: I think that I have been wrong here…
there is no parameter to set globally “retain=true” flag for MQTT messages in mosquitto.conf man page | Eclipse Mosquitto
that’s normal since this is controlled by the client (not the server)

rkrisi · November 28, 2018, 3:08pm

That’s weird because I had a flag set to true there! And it seems that it worked, because after removing it won’t change on retainted messages…

rlkoshak · November 28, 2018, 4:25pm

That is correct.

One concept that is important to realize is that MQTT is more of a push than a pull. When your client subscribes to a topic with a retained message, the broker will push that retained message. When the client subscribes to a topic without a retained message then there is no message to pull.

In none of these cases is there anything pulling any data from the devices or the broker.

This is also not unique to MQTT. Many/most technologies OH integrates with works similarly.

My question for you, how are the topics set up? Are there separate topics for commanding the lights and the lights reporting state updates? If so, what you may be seeing is that when you use the retain messages on the command topic the lights will get that retained message again when the broker comes back online, causing them to turn on or off.

So to do this correctly you need to configure the devices that are publishing the state updates to use retained messages (the client sending the message controls whether the message is retained or not) and not mqtt.cfg.

Another thing to realize about retained messages is that they are retained forever, even if you publish non-retained messages after the retained message. You need to explicitly clear out retained messages by sending an empty message with retain turned on to clear out those old retained messages. Perhaps this is why it appeared that MapDB was sending commands with restoreOnStartup.

If you have control over how the device works, you can implement something where you publish a message when OH starts up to a topic that all your devices subscribes to. When they receive that message the devices publish their current state. As I said above, MQTT is strictly push, not pull.

I couldn’t agree more. I avoid it.

Correct. But depending on how your MQTT binding configs are on the Items, a postUpdate might also be published. Make sure the type filed is command and not state or *. Otherwise postUpdate will cause the update to be published as well.

But remember it is the sending client that controls the retain so this will apply to messages published by OH only. In this case, the messages that really need to be retained are those that are sent by the devices.

Ha! I had the exact same problem with retain on my garage door openers. It took me weeks to figure out what was going on. I couldn’t figure out why the garage doors would open every time the controller restarted. I though there was a pulse or something on the GPIO. But no, it was working as designed, it was that retained message.

I can think of a couple.

It would indeed work if your devices were using the retain bit on the messages they send for their state updates. Then when OH comes up it will receive the latest state. MQTT was designed with a spotty network connectivity in mind and using the retain bit like this helps address this.
Send one retained message to set/reset a device to a default state when it loses connectivity. As I said above, that one retained message will persist even though each subsequent message to the topic isn’t retained.

Remember, MQTT was created for use in the industrial oil industry with remote devices and sensors on oil pipelines and stuff like that. It is super light weight, easy to implement, super energy efficient, and very reliable, all things that are useful in the home automation context as well. MQTT is exceptionally powerful and there is a reason it is heavily used in home automation.

This would make a good tutorial post, maybe with how to send the empty message using mosquitto_pub. All of us at one point or another run into retained messages problems and clearing them is not intuitive.