On some occasions, such as when things are created using HTTP or MQTT bindings, users have to create channels manually. On such occasions there are three buttons in the Channels tab:
Would it be possible to add another button, for example Add Channels from Template, allowing users to import channel definitions from a centralized repo ? I can volunteer to create some templates if developers create such functionality.
Sounds like a good idea but it will require a lot of infrastructure to be put in place.
In the mean time, now that the code tab supports both file based YAML and DSL syntax you could post example Things here to the forum and users can copy/paste/edit the Channels to the code tab. It’s not as nice as a centralized repo and the ability to define properties, but it’s a start.
If you open your Thing and click “code” there is now a “YAML|DSL” button at the bottom. You can select what syntax you prefer. To post the Channels you can copy from here. To use Channels copied from elsewhere you can paste them into this code tab.
The netbox community has a similar repo here → GitHub - netbox-community/devicetype-library: A collection of community-sourced DeviceType definitions for import to NetBox · GitHub
Maybe the Add Channels from Template button can open a github repo for openhab device types so that users can copy what they need. This is what netbox does. It simply opens the repo, users have to browse through it until they find what they want, then copy from github and paste to netbox.
Doing this is openhab should not be difficult, I think. Using github’s pull request mechanism would allow advanced users to submit templates to such repo.
That would be a completely different appraoch for all the other repos that OH currently does support: UI Widgets, Block Libraries, Rule Templates, Marketplace Bindings and Transformations. It would be odd to use something completely different for this.
But if you think it’s not difficult to achieve a PR would certainly be considered. There will need to be support in openhab-core to query the repo and make it availble in a new REST API endpoint and MainUI needs changes to take advantage of it.
Are there many “standard” stuff/devices out there that would warrant having a database to be shared around?
Let me give some examples of devices I use:
MQTT and tasmota: 1-relay, 2-relay, 4-relay (with our without temperature sensors). lamps with dimming. Frequent use of pulsetime
MQTT and zigbee2mqtt: various types of sensors (movement, contact, luminosity, temperature, flood, humidity, etc), smart plugs, control buttons (multi touch, dimming, etc). Z2M supports thousands of different devices
HTTP: in this category I have devices not so “universal”, such as a pellet stove from Ecoforest and a wallbox from V2C, and one perhaps more general: metereology (from IPMA/Portugal)
I would love to have copy/paste capability in the MODBUS binding as well, but this is another story… I see that this binding supports some types of inverters, but unfortunately not huawey, that I’m using. Probably it would be easier to have a database with channel definitions instead of waiting for some kind soul to modify this binding to support more device types.
Ideally it’s not just a “channel-list template” but a full “Thing template” which includes the Thing Config as well as a list of channels.
This is currently already possible using the YAML Composer add-on, the only thing needed would be a place to share such “templates”.
But of course, YAML Composer is limited to the YAML format, and to file-based config.
What you’re proposing is a UI-based solution.
Such a template would require some form of pattern / replacement mechanism and a form of “configuration parameter” to fill in those patterns / templates. Similar to widget props/parameters.
In the case of UI implementation, you’d first select which template to apply, then you’d need to fill in the required values, e.g. “topic prefix” (with default values, e.g “zigbee2mqtt” - in my case I changed it to just “zigbee”), “device id”, key, etc.
Having such a thing would make life easier for UI based users, but it requires a decently stocked up source(s) of ready made templates to be useful and an easy way for people to contribute.
Instead of reinventing the wheel, one could make some form of “mass import” for Things from YAML (just like it is with Items AFAIU), and then make the add-on have the option of not outputting the files to the conf/yaml folder. With some tweaking around this, YAML templates could be used to generate “UI” Things as well.
I was not aware of this. Seems too complex for what I have in mind. Also I’m trying to get rid of file-based config, and probably most users also prefer UI.
Yes.
Not necessarily. Templates could have comments (in MQTT it would just modify the topic name in vereal lines) and users could copy from the template, paste into vsc (or another editor), make a mass update, then copy / paste the result directly to openHAB.
Of course it would be preferrable to have oopenHAB’s UI making this update but I do not see this as a requirement.
That’s why I’m proposing a github repo for template database. Users could contribute via pull requests.
This is a good suggestion. The template db could have one folder containning yaml’s to be used by the yaml composer.
Get rid of..? We’ve just made file based configs much more flexible/powerful. You can now “design” Things, Items, Rules etc. in the UI and simply export them to YAML files - or the other way around, take a file based element and duplicate it in the UI to “import” it there. I’d say that file based configs is better/stronger now than it ever was. That doesn’t mean that you’re not free to use JSONDB based storage, of course, but it shouldn’t be seen as if “moving away from files” is somehow “progress”.
If you want the “convenience” of being able to do everything from the UI without ever touching an editor or a file, you must also expect to do without more powerful things like templating. Not because it can’t be made in a UI, but because designing UIs for “generic” features is often a huge task, and one that is difficult to get “simple enough” that people will actually understand it anyway. Not to mention that making a UI using a web page has lots of restrictions making things more difficult, and making it work on tiny devices with touch screens is even harder. Add to that the many quirks of Framework7 (used by MainUI), and you have a huge number of restrictions to what you can actually make the UI do, making it even harder to make something that is logical and intuitive to use.
I’m not saying that things like this can’t be done in the UI, but I see it as highly unlikely that somebody will volunteer to do it, given all the challenges and how much work it is. OH has had rule templates since.. 2018-19?, and they still can’t be created or edited in the UI - and I’ve seen no plans for anybody to make this - despite the fact that these are very similar to rules, and by just “copying” the rule editor, you would probably be 70% there already.
Which makes me sad. And I agree, it should be a pretty easy addition. On can take the parameters code from the Widget builder combined with the rules editor and be about 90% there.
But since I seem to be just about the only person producing rule templates before now I can’t get to angry that no one has taken this on. I still have great hopes for rule templates but they’ve been so hard to create before the new YAML file format I’m not surprised few others have bothered to go through the pain to create and maintain one. Now that we can create local rule templates and do not have to use the marketplace to even test one mayab more will try their hand at creating them. And then maybe more demand will prompt someone to volunteer.
Maybe I should rename this topic to Templates only. All of o sudden it looks like UI, yaml composer, line mode things, Rich’s rule templates, and who knows what else, can benefit from the github repo I’m suggesting.
Note, there already is a way to distribute and install the followng in OH through MainUI:
This is a capability that already exists today and has for some time. It has it’s problems for sure which is most of the discussion above.
I’m all for adding Thing Templates to the list of stuff that users can post somewhere and install from the add-on store. However, I have significant reservations of introducing something completely new and different that only works for Things. Inconsistencies like that make a system harder to use, not easier.
Also, way back when the marketplace was introduced for add-ons and widgets and block libraries back in OH 3.0, GitHub was considered as the repository but rejected because it would put off those less technical users. Submitting a PR was deemed too technical compared to opening a new forum thread. It was also to full of gatekeepers (someone needs to review and approve the PRs).
Note, I’m not arguing for nor against that approach over all, just that it was considered in the past and rejected for valid reasons.
I am arguing against any change that would introduce a completely new concept/approach that only applies to Things. Not everything discussed above fits that though. For example, @Nadahar’s bulk import mirrors Items bulk import. That wouldn’t be something new from a consistency perspective.
My problem here is that I don’t use any of these “dynamic” bindings, so I don’t quite grasp what it’s about.
This is already something called ThingType that defines itself as:
ThingTypedescribes a concrete type ofThing.This description is used as template definition for the creation of the according concrete
Thingobject.
I already have an add-on on the marketplace that allows supplying ThingTypes from local files: Thing Type File Provider [4.1.0.0;6.0.0.0). It parses XML files only, but it could easily be changed to support other file formats (it uses XML because that’s what’s used in the binding definitions, so you can copy and tweak the existing ones).
I still have no idea if what’s being discussed here is related to ThingTypes or not, or if ThingTypes could be used to help solve the same challenge. Most bindings already provide ThingTypes, and e.g. the Z-Wave database provides just that - new devices need new ThingTypes to be recognized.
If in fact this is about the same thing, or could be made into the same thing, I think the option of supplying ThingTypes via e.g. the marketplace would be great, and I could help make parsers and whatnot needed to do it. As it is today, when a new e.g. Z-Wave or ZigBee device is acquired, it’s an often a months long process of getting them to actually work with OH, something my add-on was supposed to address, but it seems that it’s too hard to use XML files for most users. Making this available in YAML and/or JSON on the marketplace, with somebody potentially making a “converter” for e.g. HA device definitions, would make many things better.
But, I still don’t know if this is what we’re talking about here. I assume that these bindings with “dynamic” Things have done that simply because it’s overwhelming to deal with the huge amount of variations you would otherwise find out there. In that sense, it sounds like, at least in principle, we could be touching on “the same concept”. Most bindings provide pre-defined ThingTypes, and new devices won’t work until a new version with updated ThingType definitions has been released. Others leave this whole problem to the users, by “being dynamic”..? Or, isn’t it that “simple”?
It’s not. What’s being discussed here is creating a template for Things where, for example, one creates a generic MQTT Thing from a template similar to a custom widget or a rule template. Most of the thing is predefined but there are some parameters the end user can use to provide the details that are always going to be specific (e.g. IP addresses, topic names, bridge Thing, etc). The Channels are predefined with any transformations necessary but the root MQTT topic might be something that the user supplies through a parameter.
The main audience for these would be Things that cannot be discovered and have to be hand coded. For example, there is no AdGuard Home add-on. I could create an HTTP Thing template that lets openHAB interact with AGH and the user would supply the main part of the URL to their* AGH instance but all the REST API parts of the URLs and JSONPATH transformations and such would remain the same.
So we are talking about bindings like MQTT, HTTP, modbus, KNX, etc. Bindgins where Things are discovered and created would not be candidates for this. The binding does all that work for you already.
It probably doesn’t need to be any more complicated than the find and replace type thing that is done for rule templates.
Maybe. I don’t know enough about that. But I suspect that might not be the same sort of thing so may not be a good choice. I can’t say for sure. The zwave binding discovers and populates all the Channels for you, so that wouldn’t be a good candidate for this. I mean there’s no reason it couldn’t be used for a binding like this, but there really isn’t much of a point to do so. The binding already does all the work that using a template would save.
This is a use case that I had not thought of but it’s a slightly adjacent use case I think from the original proposal. I really like the idea of being able to figure out a Thing easily while waiting for a new device being added to the database for Zwave (my understanding was for Zigbee to support “kinda sortaa” zigbee devices requires changes to the code, Zigbee doesn’t have a database as far as I know).
Predefining Thing types isn’t really feasible for all technologies. How would athe HTTP binding “discover” and configure a webhook running on an ESP32, for example? Thing templates would mainly address that use case. IF they could also address the use case you are talking about, that would be all klinds of awesome!
I understand the principle, I just can’t translate it into practice. Is there a “generic structure” to these “dynamic Things”, or is that something that varies from binding to binding? I haven’t used any of them, which is why I don’t have a clue about how this is actually done. I see this for MQTT for example. Does that mean that the binding “builds” channels dynamically from these specific “codes”, or are they in reality mapped to standard ChannelTypes? ThingTypes allows defining Channels using ChannelTypes, so I’m just wondering how much of an overlap there really is here. There are some standard ChannelTypes in core, but it doesn’t look like they map well to the “supported channel types” in MQTT.
After looking a bit in the MQTT binding code, it doesn’t look like it relies very heavily on the “core concepts” of ThingType and ChannelType, but I also find things like this, where it turns out to be a “mapping” between the two:
So, I’m somewhat confused, but it seems to be a “considerable distance” between the structure of a “generic MQTT Thing” definition and ThingType/ChannelType.
I understand the conceptual difference, bindings where devices are “predefined” and have certain properties, supply their own ThingType/ChannelType definitions and the user doesn’t have to deal with that as soon as the correct ThingType is identified. But, I would think that in theory, the same “structure” could be used to define devices that aren’t “predefined” - and my question here was in a way “how far away” these things are from each other.
You could in principle imagine a binding where some predefined types were supplied with the binding, but where you could also add custom type definitions for devices that didn’t “conform” to any of the “standard setups”. If such a case existed, it would be natural to try to use the system that is already there, under the hood, to also define the “custom devices”.
But, I’m starting to suspect that this train has already left the station, and that various bindings have found different “proprietary ways” to deal with this. The question is then: Is there enough in common between these that you could actually find a template system that could cover them? How similar is the definition of a dynamic/generic MQTT Thing and an HTTP Thing for example? Are they at all possible to handle with “one system”?
The Z-Wave binding does this in a quite generic way, so you don’t need to actually change the code itself unless you need to support new “types”. As long as you can do with what is already defined of different channel types, all that’s needed to support a new device is a new ThingType definitions (ThingType definitions also contain ChannelType definitions). This is exactly what the database is, a mapping between device capabilities and OH Thing/Channel type definitions.
But, making a binding that generic can be a big task, so I suspect that perhaps there aren’t that many other bindings that can handle a new device simply by getting a new definition.
The fact that the ZWave binding discovers and populates for you, isn’t that “relevant”, that’s only a consequence of the fact that the definition gives it the information it needs to do so. Discovery is done using the manufacturer/model/revision or whatever “code” that all devices report, and which is mapped to specific ThingType definitions. Only when the binding finds a matching definition, can it do discovery and population - and the binding comes with a lot of definitions embedded in the JAR - typically everything that is defined in the database. But, as I’ve demonstrated, by using my “ThingType provider”, the ZWave binding can do the same for devices it doesn’t know.
In that sense, sharing ThingType definitions would make a lot of sense, but I now suspect that perhaps the ZWave binding is the only one that is generic enough for this to actually work.
The devil is in the details, but I would guess that the binding would need to have coded supports for let’s say a pre-defined list of ChannelTypes. As long as you could do with only those, a new definition would be all it took, but as soon as you need a new ChannelType implemented, code changes would be required. So, this might come down to how “standardized” the devices themselves are. Z-Wave has a system of “device classes” (if I remember correctly), which kind of forces a structure that allow some reuse. If you make e.g. a temperature sensor, you must choose between version 1, 2 and 3 of the “temperature sensor class” in what it communicates. This means that writing a binding that is generic is fairly “easy” since once you support the three existing classes for temperature sensors, you can handle all Z-Wave temperature sensors (until they add a new version of the class).
If ZigBee is “less structured” and doesn’t have such a system, it’s more likely that every manufacturer manages to make some twist in how you communicate with the sensor, how units are specified, how it refreshes and whatnot. If that’s the case, it’s more difficult to make the binding “generic” as a consequence, but I’d still bet that there is the potential for some reuse at least within the devices of each manufacturer. But, sitting down and “mapping” all these things, to make it generalizable, can also be a big task - maybe big enough that nobody has tried.
It wasn’t criticism, I understand the difference in the nature of the devices. Discovery is one thing, that’s only possible when there’s some absolute “code” that can be mapped to the actual structure, but even without discovery, it could be possible to let users build e.g. ThingTypes themselves. But, it’s fairly pointless if there are no bindings that are generic enough to utilize those definitions.
Can anybody provide some “example Thing configurations” for the dynamic bindings in question, so that it’s possible to see how much they have in common and how they would have to be “templated”. I mean, if it’s just about using simple substitutions, it’s hard to think that it woud make sense to reimplement what is already in the YAML Composer.
I can see that it would still be nice to have it all through the UI without touching YAML Composer / files.
If we were to add it to the marketplace, where would the Thing Templates go in the subcategories?
We might need to create a new category, Thing Templates?
Then people would first curate and select the templates they want to “install” (download).
The Thing details page would only offer the list of templates that have already been downloaded, much like how when adding a Thing, you’d only get a choice of bindings already installed.
Going on a different direction, the template database can just be a text on the forum that people can copy paste into the Code tab. This is more or less similar to Tasmota templates or ESPHome device templates
If we were to do it this way, we can already just start uploading them here. All that’s needed is perhaps a special forum category for them.
It’s a bit manual but nevertheless we’ll have a central place to browse / share such templates. No need for any changes in the mainui itself.
Had the YAML Composer been integrated in core, we could’ve had dynamic variable / replacement system directly in the yaml code without the need for special replacement routines. But that is now not a reality in our timeline.
Even if there is such a facility to install Thing templates, it would still be very tedious to be applying the template to multiple Things. Say you have 20 of the same coloured lights via Zigbee, you’d have to apply the template to 20 Things one by one changing each Thing ID. If there’s a magical input box to ask you for the thing id and it then applies that id for you into the template - it’d still be asking you 20 times. If not, you’d have to perform text search/replace on each Thing. To me it’s super tedious.
A nicer way would be to display multiple things in a table and letting you edit the cells and apply with one click. The closest thing I’ve seen is the bulk Items import (from file-based).
Using the file based config + yaml composer is muuuuch easier and quicker even for hundreds of Things that use it. And if you need to tweak the “template” it would automatically apply to all the instances of that template by just editing the template.
Defining a syntax for them, hosting them on the marketplace in a separate category and all that isn’t really the challenge IMO, to me, the challenge is still exactly how it would work, logically, not to mention how it would look in the UI should it be integrated. I just don’t see this yet. How do you build dynamic Things in the UI today? Is there a kind of +/- for adding channels? How would the template system integrate with this?
edit: Just so that’s said, there is a “generic” template system in core, so somebody had the idea that it could become more than just rule templates the whole time. RuleTemplate is an interface that extends Template<Rule>. But, I can’t see how that would be relevant for something like this, I just wanted to mention it.
