Auto configuration of MQTT ESP32 devices in OpenHAB

[Matej_Kotnik]

1. Overwiew
I build ESP32 and ESP8266 devices that talk over MQTT to OpenHAB.

My ESP devices usually have items like measurements, intelligence diagnostics, and settings that get saved in EEPROM.
Readings are combined in cluster messages in a few channels to conserve CPU on ESP while sending.

They do not meet the JSON standard, but REGEX appears to be fine for messages like this from mqtt explorer:

My simple standardized device:

Roleta_Matej // rollsershutters

// readings
Status = Gor: 0; Dol: 0; Stop: 0; Premik: 30; Korak: 20; LPF_Rolete: 0; LPF_Luc1: 0; LPF_Luc2: 0; Rssi:-81; T: nan; V: nan; Cmd_Luc1: 0; Cmd_Luc2: 0;

Status_Dol = 0 // runs down

Status_Gor = 0 // runs up

Status_Stop = 0

Ukaz_Korak_Dol = 20 // time command down

Ukaz_Korak_Gor = 20 // time command up

2.Autodiscovery and data parsing
I know of home assistant autodiscovery in OH, others are too complicated.

When I started I did parsing with rules, the way I knew, not a clean way but it still works.
Now I want to move over to REGEX or JSON for new devices as Rich recommended.
I avoided JSON as it encapsulates text in " " that interferes with " " of the string syntax in Arduino, you need additional characters to fix it, and JSON libraries waste lots of ram on Arduinos,on ESP32 is fine.

3. MQTT to OH Builder
My uncle writen OH Builder in visual basic to build things, items, rules and sitemap file to integrate new ESP device based on MQTT messages the device published to MQTT.
It creates basic dacboard on sitemap.

Now it time to upgrade Builder to REGEX and JSON, it could also be brought over to other script like bash.

If anyone else has similar requirements for integrating ESP devices in openhab, i appticiate recommendations and examples on how to build it properly.

4. Arduino MQTT to HA autodiscovery
Esentially there is probably interest for arduino library for OpenHab over MQTT, that also supports HA autodiscovery so we don’t reinvent the wheel.
I even cant find arduino library for HomeAssistant over MQTT, how is that possible? Like other is JSON library, but the autodiscovery part is missing for HA, that OH seems to support, hopefully a custom device.

5. Discovered items into Sitemaps
Then there could be addition of a script that is run besides OH or a new Addon to automatically make new configs from MQTT for items and sitemaps, possibly someday also rules for typical automation.

6.Plans
I will add more bellow, like OH Builder scripts and example Arduino Code to rewiew and give insight, so others can see and we get inspired on what the solution may look like.

For now that’s it, i will continue when time allows.

Cheers
Matej

[rlkoshak (Rich Koshak)]

@Matej_Kotnik

I avoided JSON as it encapsulates text in " " that interferes with " " of the string syntax in Arduino

It’s been awhile since I’ve coded directly for Arduinos but can’t you use ' ' for the outer quotes? Then you can use the " " inside without escaping.

'My "favorite" food is blue cheese'

@Matej_Kotnik

and JSON libraries waste lots of ram on Arduinos

If it helps you get the job done and you don’t run out of RAM totally to do what you want, who cares? If you indeed are running out of RAM and cannot do what you need to otherwise, . TANSTAAFL.

@Matej_Kotnik

I even cant find arduino library for HomeAssistant over MQTT, how is that possible?

The HA MQTT standard isn’t really. As far as I can tell they offer no reference implementation and those third party firmwares for ESP (e.g. ESPHome, Tasmota, etc.) implement the HA standard themselves based on examples. And I believe most of them are dropping the HA support anyway. openHAB also implemented support based on examples.

But given the standard uses JSON, if there were a library you’d have to import both a JSON library and the HA library so you’ll be stuck in an even worse situation in terms of RAM.

At this point it might be worth pointing out that openHAB also supports autodiscovery of the Homie standard and there are librarys at least for ESP for that. GitHub - homieiot/homie-esp8266: 💡 ESP8266 framework for Homie, a lightweight MQTT convention for the IoT

[JimH]

FWIW I just ran a quick test and was not able to use apostrophes as the “outside” quotes. I have to escape the “internal” quotes in the JSON string with a backspace.

I use this to publish a valid JSON string:

sprintf(buf, "{ \"Attic\": { "
          "\"humidity\": \"%i\""  
          ", \"temperature\": \"%i\"" 
          ", \"restarting\": \"%s\"} }"
          ,
          PreviousHumidity,
          PreviousTemperature,
          AtticRestarting.c_str());
    MQTT_client.publish(MqttTopicState, buf, false);

The JSON string is:

{ "Attic": { "humidity": "56", "temperature": "98", "restarting": "No"} }

openHAB Thing definition:

    Thing topic AtticThing          "Attic "            [availabilityTopic="Attic/tele/LWT", payloadAvailable="Online", payloadNotAvailable="Offline"] {
    Channels:
        Type string : LWT           "state"             [stateTopic="Attic/tele/LWT"]
        Type number : humidity      "Humidity"          [stateTopic="Attic/state",transformationPattern="JSONPATH:$.Attic.humidity"]
        Type number : temperature   "Temperature"       [stateTopic="Attic/state",transformationPattern="JSONPATH:$.Attic.temperature"]
    }

openHAB Item definition:

Number AtticHumidity                    "Attic Humidity [%d]"                               {channel="mqtt:topic:MqttBroker:AtticThing:humidity"} 
Number:Temperature AtticTemp            "Attic Temperature [%.0f %unit%]"                   {channel="mqtt:topic:MqttBroker:AtticThing:temperature"}

[Matej_Kotnik]

That JSON formatting is quite a neat alternative to JSON lib.

When I use ESP8266 and make JSON there isn’t an issue. I tried with the library and no mem issues. But there are projects, where Arduino is the core for reliability like greenhouse controllers, where wifi or mqtt reconnects shouldn’t interfere with processes.

Arduino sends data over ESP to MQTT and I use ESP as a transparent bridge so I want to conserve RAM and flash.

I was aware of the Homie project, a while ago it looked intimidating to set up. But it evolved to be quite friendly for custom applications, I will try it out to compare with a bare JSON solution.

I assume it connects to Items over JSON config in the background so no config files are needed?
I know OH can auto-bind channels to Items, is it a wise solution?

Can the Homie device also be auto-integrated into the sitemap?
How could it be automated to build a sitemap, and insert a Text menu with frames and entries?

I was planning to write a script that auto-writes config files, based on MQTT messages, or from OH Items for the Homie option, is this a good plan?

I can do some of it with VS Code to insert data manually into the sitemap, but I like to automate configuration where possible.

Cherrs
Matej

[Moxified]

Have you considered using some of the esp breakout solutions like the Adafruit Huzzah? I replaced an Arduino Mega with ethernet shield years ago with a Huzzah. Been a great solution. Has 2 DHT22’s, IR receiver, 4 relays, and 2 binary inputs (motion sensor and reed switch circuit off of my doorbell).

[rlkoshak (Rich Koshak)]

@Matej_Kotnik

I assume it connects to Items over JSON config in the background so no config files are needed?

I’m not sure I understand the question. Are you referring to the JSONDB? Then yes,
if the devices follow the Homie standard OH will automatically discover the Things and once accepted out of the inbox they will be added to OH as managed Things, no separate .things files are required.

@Matej_Kotnik

I know OH can auto-bind channels to Items, is it a wise solution?

I’m not sure what you are referring to here. OH does not autobind Items to Channels. You have the option to “create equipment from thing” which will create Items linked to Channels but that’s more of a semi-auto process, not a fully automatic process. There is still a bit of work the end user needs to do to make sure only those Channels desired have Items created and to customize the Items as desired prior to their creation.

Fully automatic creation of Items hasn’t existed as part of OH itself since OH 2.5.

@Matej_Kotnik

Can the Homie device also be auto-integrated into the sitemap?

Once you have Items it’s all the same whether they are linked to Homie MQTT Things or Zwave. That’s kind of the whole point of Items. Sitemaps work with Items.

@Matej_Kotnik

How could it be automated to build a sitemap, and insert a Text menu with frames and entries?

Something external to OH will need to be used for sitemaps.

If you use MainUI on-the-other-hand, if you structure your Items using the Semantic Model, then the Locations, Equipment, and Properties tabs of the Overview page get automatically generated.

@Matej_Kotnik

I was planning to write a script that auto-writes config files, based on MQTT messages, or from OH Items for the Homie option, is this a good plan?

It’s not a path I would follow unless and until I fully explored everything that OH has to offer and only go down that path if what OH provides is not sufficient. Your questions indicate there is a lot about OH you do not yet understand or know about and that lack of knowledge will very likely lead you down a bad path.

As general philosophy, I find code that writes code (and lets be clear, OH configs are basically code) to be brittle and inflexible. There has to be a really good reason to do it. There also has to be a balance between how much it takes to write the script compared to how much it takes to do it manually. The savings by doing it through a script needs to be significant and ongoing.

@Matej_Kotnik

I can do some of it with VS Code to insert data manually into the sitemap, but I like to automate configuration where possible.

This is where I completely agree. Configuration is indeed something that should be automated. However, openHAB is a platform upon which you build your bespoke home automation system. “Configuring” openHAB is a developement activity. It’s way more involved compared to setting a few parameters in a .ini or .yaml file somewhere. You can definitely do it, but in the end all you’ve done is moved your development outside of OH to something custom and really saved little or perhaps spent significantly more effort than if you just developed using OH native tools.

[Matej_Kotnik]

Hi Rich, thanks for indepth anwser.

For Things I meant JSONDB, back then it discovered MAX and Tasmota devices that are in there already.

@rlkoshak

Fully automatic creation of Items hasn’t existed as part of OH itself since OH 2.5.

I didn’t know that, cause i always used config files and i use sitemaps.

I write arduino code, publish to mqtt, lots of code already
It always felt a bit much to write all… things, then in VS Code import things to items, than set up items, than import to sitemap, than edit sitemap.
For example Athom smart plugs, they’ll have same items, the modularity is what i am missing, to just add one similar device in without too much work.

I have already made the tool OH_Builder that helps me to write all the configs for measurements that all follow the sCSV json like style i presented above.

As my ESP devices i make all follow that type of interface i need, strings of measurements and some mqtt channels for settings, you see the pattern, what i mean.
For basic use ESP device needs dashboard for measurements and settings.

Then, i add some rules that serve the analytics or automation, that is actually fun part of OH.
But manual integration of devices isn’t really. It is bunch of copy paste nonsense, where i make typo mistakes and wonder what is wrong for an hour, right?

Other friends tell me that their business control systems make dashboards from data that devices publish than admin just confirms the dasbord module creation, that’s what i target.

So i plan to imrove OH_Builder, i am obsessed with how simple it is to add in zigbee devices in newest HA but i have no clue about HA and custom MQTT stuff, and i have quite all integrated in OH.

I am quite familiar with OH configs like my sitemap is 1000s of lines and quite organized and functional with lots of layer 2 menus, 76 rules and such… but i really don’t know much about new OH Main UI and semantic model, i did try it out already tho, but i didn’t explore much, it is too slow in the android app it isn’t instant like sitemaps.

I may actually play with Homie to see how simple it is, it may simplify the mqtt arduino side.

I realised My OH Builder Auto cofigs need to be simplified with no rules to parse measuremts strings, and accept settings ass well to build switches and number Setpoints.

I may to spring script over to python or something to work on linux also and be able to connect to mqtt directly.

I will load OH builder here and some pictures to illustrate what is already working and better describe my goal. OH_Builder_16.5.23.zip - Google Drive

It is VB script that is built in a macro that runs in Word, that was my uncle project, as he knows VB really well, and i understand it also, it was easy to make nice GUI as well.

Created config files : OH_Build_Results.zip - Google Drive
Parsing rules, they will be replaced in new version with JSONPath in items .