Retained huh, what is it good for? Absolutely

Tutorials & Examples Solutions
1

a lot actually.

Most of us when we get started with MQTT probably encounter the same problem. We read up on retained messages and QOS and think that it sounds like a good idea so we set up all of our topics with the retained flag. Then we can’t figure out why the garage door opener keeps opening every time the NodeMCU attached to it reboots.

Then we go to the other extreme and turn the retained flag off on all the topics. But that means that we have to go through all sorts of hoops with OH System started Rules to trigger the devices to report their current states, or set the reporting on the device to be every so often even if the state didn’t change.

But when properly deployed, the retained flag is actually super powerful and can save us a whole lot of work.

A Retained flag primer

What is the retained flag and how does it work?

The retained flag is a flag that can be set on an MQTT message that tells the broker to hold on to that message. Any new client that subscribes to that topic will receive the most recent retained message published to that topic.

  • The message publisher sets the retained flag, not the subscriber.

  • The retained flag can be set on a message by message basis. However, the original MQTT 1.x binding only allowed one to set it on a broker connection by broker connection basis. The MQTT 2.x binding allows one to set it on a Channel by Channel basis. But remember the bullet point above, setting the retained flag in OH will only effect messages published by openHAB.

  • Only the most recent message on any given topic with the retained flag get’s saved.

  • If subscribing to a wild card topic, the client will receive all the retained messages on all the topics that match upon subscription.

When should I use a retained message?

Sensors and status topics are excellent candidates for retained messages. Consider the following example. Let’s say we have some devices that report the open/closed status of a door. We all know that systems are not 100% reliable so let’s assume that openHAB was rebooted. While openHAB is coming back online, the sensor reported a new state for the door.

Without using the retained flag, when openHAB comes back online, the Item will remain in the NULL state until the sensor reports it’s new state. If the sensor only reports changes then we have to wait for the door to open or close before we get a valid state. If the sensor wastes energy and cycles by reporting periodically whether there is a change or not, then we need to wait for the polling period. If the sensor supports requesting a report, one needs to code a System started Rule in openHAB to publish a message that causes the sensor to report it’s current state. Again, if this is a battery powered device it means that the device needs to be constantly connected so it can receive the message wasting power.

But, if the sensor uses the retained flag then openHAB will receive the most recently reported state immediately upon connecting to the broker and subscribing to the topic. All you have to do is set that one little flag and you don’t have to worry about any of the complications in the previous paragraph.

To put this more generically, I’ll quote from the HiveMQ page on the subject:

A retained message makes sense when you want newly-connected subscribers to receive messages immediately (without waiting until a publishing client sends the next message).

So any status or sensor reading that has a meaning beyond the mere instant when it was reported could benefit from using the retained flag.

When should you not use a retained message?

When the topic is a command topic. You do not want to retain commands to devices because every time the device restarts or loses it’s connection to the broker, it will receive the command again when it reestablishes the connection to the broker. You don’t want your garage door to open every time you reboot your WiFi router.

But this isn’t a hard and fast rule. For example, maybe one has a device that controls a light and the commands received indicate the state we want the light to be in. And let’s assume that the light doesn’t remember it’s previous state when it restarts. In that case, a retained message for the command makes sense and is useful as this will cause the light to receive the most recent command again when it comes online and enter its previous state.

Help, I sent a retained message and want to get rid of it!

This is actually pretty easy to manage, though a little counter intuitive. You need to send an empty message to the topic with the retain flag set. This will clear out the retained message. You can do this with any of the many MQTT clients.

A specific use case

Is my device online? How do I tell? If you don’t use retained messages and the LWT you will have to implement some sort of heart beat on the device to always know when it is online.

But if you do the following, you will always know when the device is online without the need for a heart beat.

  1. Configure the device to publish “ONLINE” when it successfully connects to the broker to the LWT topic. Be sure to set the retained flag to true for this message.

  2. Configure the LWT to publish “OFFLINE” to the LWT topic with the retained flag set to true so the broker changes the retained message on the LWT topic to “OFFLINE” when the device disconnects.

Now when ever OH or any other client comes online and subscribes to the LWT topic, it will immediately know whether the device is online or not because it will immediately receive the “ONLINE” or “OFFLINE” message.

I can’t claim credit for this use case. I believe I first saw it in use by the Shelly 1, though I could be missremembering. I think the Homie standard works like this too.

The nice thing about this approach is you have pretty much guaranteed that you can know when the device is online or offline without the wasted energy of setting up a heart beat.

16 Likes
2

Well, knock me over with a feather! Rick is an evangelist for retained MQTT messages?

That’s not the same Rick in a certain discussion last August. The old Rick considered retained messages to be the devil incarnate.

A way to handle this. Not always a better way. Not always even a desirable way.
We can go back and forth on this for many posts. But I already know you will not convinced me. I sent months using retain=true for everything. It caused far more problems then it solved in my use cases.

I started with retain=true. I have no retain=true left in my configuration. You will never convince me that it is the correct or only good choice in all, or even in most cases. I’ve been burned far too many times already.

I agree with the new Rick. There are many good applications for retained messages. Definitely not for everything but a state topic is a good candidate (i.e. sensors and other devices reporting their state). Whereas for command topics, not so much … unless you don’t mind the possibility of devices coming alive whenever they disconnect/reconnect to WiFi.

I recall saying this and old Rick immediately countered with an edge-case example of a dead sensor. Better dead (NULL) than retained!

Hello new Rick! I like your more accepting view of retained messages. I’ve been using a mix (state topics with retained, command topics without retained) for well over a year and it works well (three different home automation systems staying in sync via MQTT).

Keep the faith, new Rick! :wink:

3

A guy can change his mind, can’t he? :wink:

There opening of the paragraph explains where I and I think a lot of newish users of MQTT fall. First it’s “retain all the things!” Until you get bit. Then it’s “don’t retain all the things!”

I just needed to see and really understand a really good example. And it needs to be relevant. I actually don’t have any use cases in my home system where retained makes sense, except for the LWT example.

Well, I still stand by that statement. In my mind is better for sensor Items to reflect that we don’t know what state it is in rather than some steak value from who knows when. Especially if you know the sensor is offline if much rather see my Items set to UNDEF or remain NULL.

And that is why I still don’t have any use cases for retained in my set up.

1 Like
4

Of course! Yet it seems odd that you’re not actually practicing what you preach.

Yesterday it was:

Sensors and status topics are excellent candidates for retained messages.

today it’s:

better for sensor Items to reflect that we don’t know what state it is in rather than some steak value from who knows when … that is why I still don’t have any use cases for retained in my set up

Yesterday’s advice isn’t good enough for your own sensors yet you promoted it to others. :man_shrugging:

Guess there’s no new Rick.

5

Seems like a good combo to use retained messages and LWT.

Let’s imagine a sensor that reports the outside temperature every 15 minutes. Good enough resolution for outdoor temps. Now- your openhab restarted 30 seconds after the sensor reported, you have to wait 14.5 minutes to have the temperature. If that message was retained- you’d be set.

BUT now imagine your sensor goes down! Now you’re stuck with a retained message from 2 or 3 hours ago. This is where a lwt message comes in handy to change that item from a temperature to “null” or whatever non reporting broken status you like…

I’m not defending Rick’s flip-flop. Just an example of how the different mqtt tools can be used!

6

I agree with you.

In fact, I made the same case when old Rick and I circled this same tree last August.

Me:

A better way to handle disconnected devices is to use another MQTT feature, designed for this very purpose. Peter describes it in his post below and it’s the Last Will and Testament . When the Broker detects the temperature sensor has disconnected, it can publishes the sensor’s Last Will which can be something descriptive like “UNAVAILABLE” or “SENSOR_LOSS”. This is the feature’s purpose, to inform subscribers that a publisher has disconnected.

Old Rick:

I’m fully aware of LWT and use it regularly. I’m not talking about detecting a device has failed, though LWT implies that the only failure that occurs is that the device loses connection with the broker.

1 Like
7

the individual that you are attacking is trying to help the community with his experience. He is teaching all of us and we value his help. You on the other hand don’t seem to be contributing anything worth while

1 Like
8

That’s a very low bar for what qualifies as an ‘attack’ but you’re free to characterize it as you wish.

He is teaching all of us

Indeed he does and his contributions are appreciated. However, in this particular thread he described something we had discussed many months ago and he was adamantly against the use of retained messages for almost all purposes, including the ones he mentions in his post above.

I was surprised to read, what appeared to be, a complete reversal of his original position on the subject of retained messages. However, within short order I learned he still holds the same position.

That’s the long and short of it.

Now if I shared your low bar for ‘attack’, I’d call that statement a personal attack.

9

My needs do not match everyone’s needs. I propose a lot of things that I don’t have a use for in my own set up. More than half of the DPs wouldn’t exist if I limited myself to only my own set up. Similarly, I’m open minded enough to recognize that my own requirements and constraints are not universal.

I don’t find it at all contradictory and I did say “in my mind”. For example, I have a bunch of temp and humidity sensors around the house which controls the heater, fan, and humidifier. I’d rather have those fall back to a default behavior than turn things on or off things based on hours old sensor readings. So I don’t use retained for these sensor readings and my rules treat NULL and UNDEF meaningfully.

I do use the LWT and have Rules to set my Items to UNDEF when they are offline. I could use retained messages but I don’t actually get any advantage from doing that. If I did then I would have the Item change to some old value which will trigger all the rules and turn on/off the HVAC stuff based on old readings. Then almost immediately thereafter they change to UNDEF, only now I’ll have to wait for the HVAC to enter the appropriate state because I throttle the commands to avoid cycling the furnace too rapidly and potentially damaging the unit. But by not using retained messages, the Items come up as NULL and then get set to UNDEF when OH sees the sensor is offline. But in either case the sensor gets ignored so the HVAC won’t enter a potentially incorrect state.

I just don’t see it as a flip flop. If there is a flip flop it’s from “retained never” to “retained sometimes”. No where do I mean to imply in the OP that retained is appropriate for all sensor or status topics in all situations. I personally do not have cases beyond LWT where it makes sense. That doesn’t mean I can’t advocate for it to those who do have use cases where it does makes sense.

If you think I need to add to the OP to make that more clear I will.

I do not see this discussion as an attack. At most it’s a misunderstanding of what I mean and meant in the OP and past discussions on the topic. I am not now and never have advocated for the use of retained for all sensor readings. Just because I personally still do not have a use for it in my sensor readings does not mean that I see no use for it. Nor does it mean that I have to have a use for it in my personal configuration.

No I do not. My position has changed from no retained to sometimes retained. In my personal setup, as described above, using retained will cause me more work and introduce undesirable behavior. Should I be forced to use retained in order to maintain some short of MQTT purity? Or can I be free to acknowledge that there are cases where retained doesn’t make sense, even for sensor values, and in my personal setup that is the only case? And can I be trusted enough to understand my personal configuration to know when it makes sense and when it doesn’t?

I purposefully didn’t bring up this discussion in the OP because I assumed the words “candidates for” and “when you want newly-connected” was enough to imply that I’m not advocating to use retained for all sensors and statuses and I trust the readers to know enough about their own configuration to know when it makes sense for them.

If you think this isn’t clear from the OP I’ll gladly change it or add to it.

10

I wasn’t meaning to call anyone out on anything. I like the OP and need to think about how to use retained flags to keep things running nicely.

The discussion and ideas are always helpful to me. I get thinking about new things to add and how to better use the tools available. Mqtt is a powerful system and I’m sure many of us are only using the basics. Adding qos, retained, lwt, etc. in the right usage will make systems more robust and failure safe.

That’s the beauty of all this. Between openhab, arduino, node red, and the myriad of hardware available- each of us gets to build out the smart home of our dreams. Discussion of all choices is always great to me!