My MQTT 1.x to 2.x Migration Experiences

mqtt2
Tags: #<Tag:0x00007f51e4576380>

(Rich Koshak) #1

Edit 1: I may have uncovered a bug

I know others have posted similar and very useful tutorials along these same lines. But the new MQTT 2 binding is very new and has lots of options so I wanted to documented my personal experience migrating with a couple of lessons learned and my recommended best practices along the way.

I am by no means an expert in MQTT nor either of the bindings. If I write something wrong here please let me know.

Goals

My initial goal was to take what I have device and Item wise with my MQTT1 setup and move to MQTT 2 with a minimum amount of changes. Clearly, in the process I have discovered that my overall approach to using MQTT is not the best and my topic hierarchy was not well thought out. Changing that will be a future exercise.

I have a mix of some custom devices (sensorReporter), ESP Easy, and Sonoffs running Tasmota.

Tools

Just in time for this exercise @thomasnordquist published an MQTT tool that listens to and publishes the topic hierarchy of your existing setup. It also shows the message traffic in real time. This tool was a real time saver. Normally when I’d do a migration like this I’d use a combination of the REST API, custom sitemaps, watching the logs and an MQTT client. With this tool I only needed it and to watch the logs. Go check it out, you won’t regret it.

Logging Config

If you are like me you will inevitably run into some strange issues and will want to look at more detailed logging from the binding. Here is my log4j2 config to put the MQTT binding into debug and put the logs in its own file. This proved very important to find one of the problems I ran into.

# MQTT
log4j2.logger.mqtt.name = org.eclipse.smarthome.binding.mqtt
log4j2.logger.mqtt.level = DEBUG
log4j2.logger.mqtt.additivity = false
log4j2.logger.mqtt.appenderRefs = MQTT
log4j2.logger.mqtt.appenderRef.mqtt.ref = MQTT

# MQTT
log4j2.appender.mqtt.name = MQTT
log4j2.appender.mqtt.type = RollingRandomAccessFile
log4j2.appender.mqtt.fileName = ${openhab.logdir}/mqtt.log
log4j2.appender.mqtt.filePattern = ${openhab.logdir}/mqtt.log.%i
log4j2.appender.mqtt.immediateFlush = true
log4j2.appender.mqtt.append = true
log4j2.appender.mqtt.layout.type = PatternLayout
log4j2.appender.mqtt.layout.pattern = %d{dd-MMM-yyyy HH:mm:ss.SSS} [%-5.5p] [%-50.50c] - %m%n
log4j2.appender.mqtt.policies.type = Policies
log4j2.appender.mqtt.policies.size.type = SizeBasedTriggeringPolicy
log4j2.appender.mqtt.policies.size.size = 10MB
log4j2.appender.mqtt.strategy.type = DefaultRolloverStrategy
log4j2.appender.mqtt.strategy.max = 10

This will create an mqtt.log file in the same folder as your openhab.log and events.log. It also sets the logging level for the binding to DEBUG.

ESP Easy

I mainly have a series of sensors attached to these devices: temp, humidity, and photoresistor along with some monitoring publishes in the LWT and uptime.

LWT

It appears that ESP Easy publishes retained messages to the LWT topic which is actually really clever. When the device connects, it publishes “Connected” to the LWT topic and when it falls offline the LWT mechanism will publish “Not Connected” to the same topic. Thank you MQTT Explorer for revealing this fact to me (it’s probably in the docs but I missed it)! I am definitely going to be changing how I do this in my own code, though I think Homie does this approach too and that is my ultimate goal.

Uptime

ESP Easy can also be configured to publish it’s uptime in minutes.

Sensors

The sensors data gets published to a separate topic under the device name’s root.

image
The above is a screenshot of the topic hierarchy for one of these devices, mbr-sensors.

Thing

We will create one Thing to represent each ESP Easy device with channels for each of the topics in the screenshot above.

The sensors are all pretty straight forward. They are Number values so we use a Number type Channel. Humidity is a percent so we could theoretically use a Percent type Channel but I chose to just use a Number and set the min value to 0 and max value to 100.

Uptime is a little different. A number of minutes since it came up gets published to this topic but I don’t actually do any math with this number anywhere so instead I convert it to DD:HH:MM:SS format using a JS transform. Therefore I made this a Text type Channel.

Finally, the most interesting channel is the LWT. Because the device uses retained messages this topic will always represent the online status of the device. So we can use an On Off type Channel and the alternative ON and OFF values and ultimately link this to a Switch to represent the online status.

The Items these Channels are linked to are as follows:

Channel Item Type
Temperature Number:Temperature
Humidity Number:Dimensionless
Light Number
Uptime String
Last Will and Testament Switch

I’ve read in the release thread that the UoM is not supposed to work but I wanted to see what would happen. It appears to work for me, though the values published are already in my default units so I might just be getting lucky. Or there have been changes published to make them work.

Sonoff Tasmotas

To be completed later

sensorReporter (my stuff)

I have a bunch of stuff running off of RPis scattered through the house and exposed to OH using sensorReporter and MQTT. These are sensors, actuators, and status (uptime, heartbeat, LWT). The topic hierarchy for these are all over the place and need to be reworked. Ultimately I plan on implementing Homie in sensorReporter but doing so is going to take a significant refactoring of the code so it will be awhile.

I’m only going to focus on one of the jobs I have running off of sensorReporter, my garage door openers. I have two garage doors and each has an actuator to open it and a sensor to tell if it is open or closed.

image

Each garage door gets its own Thing with two Channels each. The door senor is an Open Closed type and the actuator is a Switch. The messages published to the topics are already openHAB friendly for these types. So it’s just a matter of putting the right topic in the right part of each Channel.

[Edit 1]
While trying to get my LWT Items to work for sensorReporter I ran into a bit of trouble. One issue is that for some reason my script isn’t actually generating the LWT message. I think it is getting stuck in a zombie state and keeping the connection to the broker alive but otherwise unresponsive.

So I used to have an Item that would get set to OFF when the LWT message is received, set to ON every time the uptime message is published (every five minutes) and I use the Expire binding to set it to OFF if the uptime isn’t published for too long. I could not get this to work using MQTT 2 as I wanted.

I created a Text Channel to receive the Uptime string from sensorReporter. Then I created an On Off Channel also subscribed to this same topic. But I created a MAP transform that always returns ON (thanks to the new default capability added to the Map transformation). Then I set the Switch Item to link to this On Off Channel with an Expire binding to time it out. Theoretically the Item should be updated to ON every time there is a new uptime message published.

Unfortunately this didn’t work. The Item does indeed get set to ON but it immediately gets set to UNDEF. The Item I have linked to the Text Channel subscribed to the same topic does not exhibit this behavior. So as a work around I implemented this behavior in a Rule and I’ve filed an issue on the binding.

[Edit 1]

Lessons Learned

  • Create a separate Thing for each logical device and resist the urge to create one Generic MQTT Thing with channels for each and every one of your dozens of current MQTT topics. Stick with the documented concept of Things. One Thing represent one device.

  • Double check that the Item you link to corresponds with the Channel type. I discovered, for example, that linking an On Off Type channel to a String Item just silently fails. No warnings or errors in the logs but also no messages get published. It was just a fluke that I noticed that my Item was a String and the Channel was a Switch. @David_Graeff, is there anything that can be done to get some sort of feedback when making a mistake like this. I don’t like when things just fail silently.


[SOLVED] Tasmota - I2C and OH2
(Hakan Tandogan) #2

This. This helps a lot in properly compartmentalizing the setup. By now, I have two dozen "Generic MQTT Thing"s in my system, one for each (tasmotified) Sonoff switch, one for each Miflora plant sensor, one for my OctoPrint instance and a few things here and there for other MQTT-able device.


(john) #4

Hi Rick. @rlkoshak
Great post! I am reading more as I have time, in the forums. One of my ESP8266 modules went south the other day so I doubled up a relay and BME280 into a Wemos. Once I got them set up, I had to get them back into OH2. Try as I may, I could not get them to display in OH. I went back and watched BK Hobby’s videos about setting things up. There was just something missing when I was using Paper UI to set this up. Then I remembered a post you replied to when I was first starting out. I want back and found that post, followed your directions and everything is working. I’m starting to document these kinds of tips when I get them working. Easier to reference my notes than searching the forums for … what was that called again? lol

The above post really helped me to understand how to expand the sensors and their applications. I can read something several times and I may get it but if I can see it in action or screenshots once I usually get it.

Even though I’m using Tasmota, I like the idea of checking if the sensor is online or not. I think I saw in one of BK’s videos where the data on the sitemap would change color if the sensor data was fresh or stale. That sounds like a rule I will have to learn how to write.

I just want to say thanks again for all your help and these great posts to help further my knowledge.

John Frankforther


(Rich Koshak) #5

You could publish your notes as a Github gist or something like that and then not only will you have your notes protected out in the cloud, but others can find and learn from you. :wink:

Not a Rule, but the visibility tag in the sitemap.


(john) #6

Now that I have a solid OH2 running and all the sensors reporting and the switch working, I’m planning to do a write up how I setup the ESP8266 with a relay and BME280 sensor using Tasmota. Then show how I got it to connect with OH2 MQTT.

If you don’t mind checking in on this, once I get going, to make sure I’m not giving out the wrong info.

I have rebooted the RPi several times and reloaded the 8266’s just to make sure everything is still working. One thing I did do is to make all the ESP’s and Rpi set at a static IP address. That way I can go directly to that address and make changes or see if that node is still working.

The next two projects are to replace the ON/OFF switch with a pushbutton or wire the relay in as if it is a second light switch. (like a room with 1 light and 2 switches) The other is to make a motorized drapery Opener/Closer with light and temperature sensors. This will help to reduce heat/cooling costs. Too hot or cold outside, close drapes. Temp OK and sunny, open drapes or something like that.

John