MQTT 2.4 documentation

mqtt-binding
Tags: #<Tag:0x00007fe053fc5860>

(Garrett Porter) #21

PS: Looking at the official things documentation there is no mention of the bridge{ thing{ channels }} syntax. Only the “typical” way to do it and then a link to go look at each of the individual bindings.


(Jürgen Baginski) #22

When creating a thing via file (Option 1)

And when creating it via PaperUI (also connecting to the broker “opusMQTT”):

Note that in the second example the brokername is not part of the channel-syntax.


(Garrett Porter) #23

Awesome.

Though this does not support what I’m coming to believe is incorrect syntax for channels that I’ve seen around. This is still mqtt:topic:... where the incorrect version goes mqtt:<broker-name>:topic...

Thanks for the screenshots! I’m headed out to work but I’ll probably fiddle with the paper UI tonight. Nothing else interesting to do in a hotel room…


(David Graeff) #24

Ok I can understand the fear of using Paper UI when reading this: https://www.openhab.org/docs/configuration/things.html#defining-things-using-files

  • “internal database”
  • “Please check each individual binding’s documentation for details”
  • “Benefits of defining Things, Items and other aspects of openHAB in configuration text files”

This text is highly biased and sounds like Paper UI defined Things could magically disappear at any time (internal “database” OMG!).

Could someone please fix that article. Maybe we just migrate all the MQTT examples over there (minus the mqtt1->mqtt2 migration section of course) and just add a link in the MQTT binding doc to that article.

I would do it myself, but then the article would contain sentences like “Please refrain from using .thing files whenever you can. The syntax is highly unusual and cannot be found anywhere outside of OH” and such.

Cheers, David


(Vincent Regaud) #25

I would remoce the whole thing and leave:

Defining Things Using Files

Things can also be defined manually by creating .things configuration text files. These files are stored in $OPENHAB_CONF/things .

Please see the individual binding documentation for the text files configurations


(David Graeff) #26

But that sounds like the binding developer is responsible for the crappy .thing file syntax. And the binding developer is only the victim in this game.


(Jürgen Baginski) #27

What about the existing documentation of eclipse smarthome? Is this going to be in cooperated into the openhab documentation? At least the part we are talking about in here has more detail on the Eclipse site! See Here
IMHO putting the examples for things AND bridges into the openhab docs would solve most of the problems.


(Garrett Porter) #28

Thank you for this link! This is great information that I had no idea where to find. This actually helps explain one of my original questions with the structure of the keywords and such. I don’t understand the relationship between OH and ESH - I just use OH so that’s where I read things.

And just like that we are back on the dispute between Paper UI and textual config. All I wanted to do was fix the incorrect examples and learn how the MQTT config works… this all gets complicated so quickly. Nothing new, just observing.


(Jürgen Baginski) #29

I’m going to file a change on the openHAB documentation, in order to show the examples for Things and Bridges (like in the eclipse documentation) now!

Edit: Done!

…and this time Github didn’t complain! :rofl:


(Jürgen Baginski) #30

Are you sure on that number? I do find several of those (hue, AVM Fritz!, Tankerkoenig) however MQTT is definitely the most widespreadly used one.


(Garrett Porter) #31

@opus - you’ve killing me with this new ESH documentation. I found this:

https://www.eclipse.org/smarthome/documentation/features/dsl.html#referencing-existing-channel-types

Is this a way to define formats for thing types? Like say I have 10 tasmota switches that I want to control each with a different name of course but could I then do one-line channels as a “Type” definition?

This is too much. I need to pick one and go with it.


(Hakan Tandogan) #32

That’s why you need either auto-discovery (which would work best if Tasmota would support Homie or openhab-mqtt would support the Tasmota topic scheme)…

Or te** fil** (removed possibly offensive text here) :speak_no_evil:


(Garrett Porter) #33

It’s the future. Problem is the future isn’t ready - and I’m not ready for the future. Tasmota supposedly supports home assistant format discovery - but there was an issue with the 2.4 binding that made it not work? I understand it’s been fixed (2.5 snapshot I believe) but haven’t tried any updates to check that for myself…

Just another reason I don’t really trust discovery. None of my things have ever been discovered properly with any binding…


(Rich Koshak) #34

It is wholly based on the Channel ID and the Channel ID is based on how the Thing is defined.

Were I to tackle this, I would first exercise the binding thoroughly using PaperUI to see what is possible. Then I would figure out how to do that same thing in a .things file. Don’t attack the problem from the .things file first because I’m all but positive the standard is loose enough that it will allow you to do things that are confusing at best and unexpected and not supported at worse.

The GUIs in PaperUI represent the way the binding is expected to be used. The documentation should reflect the what the binding is expected to be used. I have concerns about thoroughly covering each and every N! combinations of Thing definitions that the .things file syntax make possible. It will only generate a huge maintenance work load and .things files as they exist today are not going to exist in OH 3.

Assuming that the standard syntax are adequately documented somewhere, I’d rather see a table for all of the parameters and what they mean to define an MQTT Thing, some text explaining the Bridge/Thing relationships, and maybe one or two examples. I agree with David, it shouldn’t be the MQTT binding’s job to correct deficient documentation elsewhere. But given the nearing end of life for .things files in their current syntax (1 year ish is what I’m guessing), how much effort is it worth to address this? It’s a tough call.

I did the copy and paste and edit of my MQTT things like this by first creating the one in PaperUI then editing the JSONDB manually. Worked like a champ. I just had to stop OH while I made the edits.

Just to repeat what I said above, I agree with David that 30 pages of examples and documentation covering how to create MQTT Things in .things files I’d rather see that effort spent beefing up the .things syntax documentation (where ALL bindings will benefit) and then the MQTT binding can and should just have to document the parts that are specific to it (i.e. Bridge/Thing relationship and the specific parameters).

Which is part of the problem that David was referring to. The .things file documentation is not complete nor up to date. So let’s spend the effort there instead of putting it all in the MQTT binding’s docs.

At the time that was written, that was probably true. I well remember the early days of OH 2 pre-release where everything in the “internal database” got wiped out routinely. And when it was MapDB, there was nothing that could be done to restore or recover except rebuild everything.

A lot of the attitudes that were formed back then are still around.

But this is part of the problem. Most of the format and structure of .things files are standard. The bindings shouldn’t be responsible for documenting the standard parts. They should only be responsible for documenting the binding unique parts, i.e. the parameters.

From the release of OH 2 and until about two weeks ago the “core” parts of OH were moved out to a separate project under the Eclipse Foundation in the hope that we could get some commercial investment of time and code. That commercial investment didn’t pan out so now there is a great effort to reencorporate the core into OH.

So the tl;dr is ESH was the core of OH and it is where most of the stuff like .things file syntax was implemented. But now that is being moved back into the OH baseline.

Just to provide a counter point, I’ve never had a binding fail to correctly auto-discover a Thing.


(Garrett Porter) #35

Rich, I always appreciate the thoroughness and thoughtfulness of your replies.

I definitely understand the approach a lot of you are taking in recommending to go back and figure out the core functionality of openhab when trying to figure out how bindings and things work. That’s a best-case situation in the world. But as I expect most users do, I usually just want to learn how to configure my hardware in the bindings I use then get on to the rest of life and actually use the system I’ve set up. I’m now about a week into my quest of “figure out how mqtt binding works” and I feel like I’m only slightly closer from when I started.

How do I do this? Where does JSONDB live? How do I edit JSONDB (text file!!)? I started using OH around April last year. Initially everything but the astro binding was entered as a text file in .items and .cfg files - then I moved on to .rules files. Files everywhere, no JSON, no database (that now I’m scared might destroy itself?), and no recommendations to go use Paper UI for everything.

Things sure have changed a lot with the recommendations I see since then. And you know what - I’m scared of the progress and I’m scared of the writing on the wall that text files are going away.

My main cash investment in automation is a large Insteon installation (every light switch and many plugs in my house). There is no apparent development to bring that up to a 2.x binding. There isn’t even a maintainer for that binding around anymore. Best case for me looking at that future is I lock myself into a pi with OH 2.4 and hope things like myopenhab.org aren’t “turned off” in the future.

If the MQTT 2.4 binding is meant to be set up in text files and if I have a good assurance that the black-box Paper UI is going to work properly, I will go ahead and embrace that.

A lot of the recommendations on the forums assume that someone really knows what they are doing above and beyond what I bet the vast majority of OH users are actually using and capable of. (edit the JSONDB manually, read the ESH documentation to learn how to configure a binding, Amanda backup, etc.)

I’ve just been bouncing all over the place with my practice 2.4 system trying to decide if I should embrace paper ui in its current form, if I could dump my .rules files, if I should learn how to define 100% of my system in text files and just never touch paper ui…

Again, I appreciate all of the help and support from all of you. I wish I could help more. I wish I had time to learn java programming and write a 2.x binding for Insteon.

I’m off to keep learning and using my pieced together system until it blows up in my face.

XKCD comes to mind.


(Rich Koshak) #36

Everything you do in PaperUI, the REST API, or the Karaf console gets saved to an “internal database”. This database is called “JASONDB” and as the name implies, it’s just a big JSON formatted text file.

You don’t want to edit this file while OH is running because OH only reads this file when it first starts but it periodically writes back to it which will wipe out your edits.

The file is located in /var/lib/openhab2/jsondb.

Note that editing this file by hand is not a long term solution. There has been a lot of discussion and a lot of work going on, largely lead by David, to figure out what the next step is. There are a number of technical reasons why we need to drop .things files, .items files, and .rules files in their current form including:

  • the Xtend libraries are one of the main reason that we are still stuck on Java 8
  • these files are causing huge problems for many users during OH startup including vastly exaggerated startup times on RPis and Rules starting to trigger even before that .rules file has been fully parsed and loaded resulting in errors
  • there are two sources of truth when it comes to the OH config resulting to a bifurcated configuration which causes huge amounts of confusion and problems like the ones listed above
  • there is no way for UI users to migrate to text configs and back because there is no writer that can write out configs in these formats

The current situation with these config files is unsustainable.

No no no! Text files are NOT going away. No one is saying that text files are going away. But because of the issues listed above, the text files will have to become radically different from their current form.

The current thinking (after several threads many hundreds of postings long) is that text configs will work as follows:

  • the “truth” will be in the JSONDB (or whatever replaces JSONDB)
  • there will be an import/export function to export the config to text configs an import text configs into the system on command (no file watchers)
  • the format of these files will be in a format for which there are good third party libraries to both read and write (YAML is the leading contender but JSON and XML are also options)
  • PaperUI will be rewritten to make editing of everything possible in a more text file like manner

There isn’t a whole lot that can be done about that. At some point (probably OH 3) support for OH 1.x bindings will be lost. I estimate we are at least two major released of OH 2 before that will happen.

But if there isn’t even a maintainer for the binding, at some point something is going to break anyway and not get fixed so it’s hard to really blame progress for the potential demise of this binding.

All I can recommend is to set up a bug bounty and hope that someone takes up the effort to rewrite it as a 2.x binding, or take up the job yourself.

Other options include keeping around an old OH server to drive your Insteon and moving everything else to the new one and use an integration between the two instances. Right now this can be manually set up using MQTT though there is talk of a bus integration service that might facilitate this.

David feels very strongly against text configs in general and the current format of them in particular. He’s the developer of the binding so you will get the best experience from this binding through the UIs.

But like I mentioned above, you have text edit support now and will continue to have this into the future.

PaperUI is only a black box because you haven’t opened the lid to peak inside. It’s really not that opaque. Everything that PaperUI does goes through the REST API. Install the REST API Docs and browse around through the REST API calls. You will see pretty much everything PaperUI does and how it does it. Then look in the JSONDB and you can see how it’s saved.

Oh, and the JSONDB gets automatically backed up (/var/lib/openhab2/jsondb/backups) so you can always compare and recover from a problem.

And yes, making changes and edits in PaperUI is painful. In it’s current form it is a really really bad tool. But it doesn’t have to be that way and if you look at the PaperUI NG design study thread, you will see what PaperUI could be. And if it were live today I would have no compunction about recommending using it for everything (the NGRE is going to be transformational in helping non-developers create Rules).

But it’s not so my recommendation remains the same as it’s been since OH 2.1 was released. Use PaperUI to manage ALL Things and text configs for everything else.

Once the PaperUI NG gets implemented and goes live, I’ll amend my recommendation to include using it for everything but Rules and Sitemaps (assuming everything else is feature complete). Once the NGRE matures a bit more, that will become my recommendation for Rules.

All of these are stop gap solutions until a better solution becomes available. ALL of the OH 2.x versions have been OH transitioning from the all text based you must almost be a developer to use OH 1.x to something more approachable to the average user. That transition is still ongoing. Some parts are really mature (Things in PaperUI), some are started or only half baked (PaperUI itself), and some are not even started or thought about (Persistence, Sitemaps).

But, if you stick to PaperUI, you will not be asked to do anything beyond what you are capable of. It is only when you refuse to use PaperUI, or run into something that PaperUI doesn’t yet support that you have to do any of the things you mention. Would you rather the answer be “sorry, can’t do that” or would you rather the answer be “here is a work around you can use until PaperUI matures and I’ll help you through it”?

See my recommendations above. Use PaperUI for Things. Use text files for everything else. When the PaperUI NG gets released, reassess.

If you are adventurous, wait a couple of weeks and you can try out the NGRE for Rules. But only if you want to get ahead of the curve (and maybe help me document it :wink: ).


(David Graeff) #37

openHAB 2 is only 2 years old. A new release happens every half a year (therefore .4). With every release one of the subsystems of the old OH1 got replaced or a major binding found its way in.

There are still OH1 subsystems running though:

  • Persistence
  • OH1-binding support subsystem
  • Thing/Item/DSL-Rule files
  • Sitemaps

So it will take another 2 years for openHAB if development happens at the same pace to get rid of the old cruft and gain consistency in everything. Unfortunately we are a little under time pressure. The eclipse foundation wants a OH3 release to happen this year where we replace all “eclipse” namespace parts within openHAB (which breaks the Java API for not adapted bindings).

Fortunately all mentioned parts above have an open pull request already and just somebody need to continue working on them. My personal contribution will be a new UI, so that the New Rule Engine can fully replace DSL-Rule files.

Cheers, David


(Rich Koshak) #38

:frowning: So shooting for making all breaking changed to OH 3 may not be realistic. I’d rather wait for OH 4 or OH 3.2 than rush through a bunch of breaking changes this year. Chaos would result.

Does the removal of “eclipse” from the name spaces have to be a full version update?

:fireworks:


(Garrett Porter) #39

Awesome summary. I’ve gone through a bit of those hundreds long threads and gotten lost in the back and forth between everyone.

There was a bounty on this one - it’s been canceled/taken back (I don’t know the right term). I’ve seriously considered putting some cash into it but I don’t know how much it will take to get things done. I’ve also considered writing a new one myself but again, I don’t know where to start. It’s really two huge projects - learning to write a binding and learning the massive world of insteon commands. I was about to message Bernd the other day (he’s around on github) to see what it would take / if he would be willing. Maybe I’ll do that.

Yep - I’ve done this. I didn’t realize Paper UI worked through the REST calls though. Interesting. But again - I’m not actually a programmer so my hours reading through the REST API docs ended up in me saying “oh that’s interesting” and not actually doing anything productive with it.

Yes I’m excited for this.

I’ll watch for it and see how I can help - I generally keep my “practice pi” 2 or 3 steps ahead of my production system.

Thanks again Rich!

Between getting Insteon updated and/or replacing all of the stuff in my house I’m sure I’ll be around. If one or the other of those happened now I’d likely be right alongside you with paper ui, New Rule Engine, etc. Honestly, I’m probably about to go all-in with paper ui anyway where I can (all but my 1.x stragglers).

Seriously though David - keep up the good work. I’m sure whatever you come up with will be great.

Seems to be a recurring theme around here…


(David Graeff) #40

If you go with semantic tagging (API breakage -> new major version) then yes. There are some ideas floating to not have a OH3 by dynamically adjusting namespaces while reading configurations etc.

Still, all the mentioned topics should be tackled. Unfortunately not by me.
If you look at what is required for my vision of Paper UI NG (or should I say OH3), that is quite a list: