Next generation design : Ideas & Discussions


(Juelicher) #41

Blockquote
You are a true openHAB 1 user ^^. Nothing bad about it, but I wonder how you handle a ZigBee or zwave network with a hundred devices without auto discovery.

As I wrote, “from my perspective” and I certainly understand that others will have other requirements. As I do not use ZigBee and zwave (mostly KNX) this is not a requirement for me.

Blockquote
Regarding your requirements: You do not even try to find a solution that only involves one source of truth.

Because it is not I requirements for me, and I stated my requirements for the way I use openHAB. Again, others will have other requirements and maybe some day openHAB will not fullfill my requirements any more, as I am probably not part of the majority. I surly hope not, but I would have to accept that. By the way, I could also hand back your remark with “You see only the developer perspective without trying to find s solution, that will fit all use cases”. I am sure, there are possibilities, but probably no easy ones.

I think, wie should discuss this topic, which certainly is an important one, without any subtle animosity. From my experience it is important to first collect all requirements and prioritize and sort them out later, otherwise important input will get lost. I would not drop requirements right from the beginning, just because currently no resolution can be seen.

Maybe we should move this discussion to an other thread, so that it can be seen more easily by others?


(YvesHanoulle) #42

I think, wie should discuss this topic, which certainly is an important one, without any subtle animosity. From my experience it is important to first collect all requirements and prioritize and sort them out later, otherwise important input will get lost. I would not drop requirements right from the beginning, just because currently no resolution can be seen.

I don’t believe everything can be seen.
I prefer a system that makes it easy to change later when we discover new things, as we will (and the world will change)

that does not mean we should not have the discussion now.

And yes on the new thread. I know some people can move parts of a discusion to a new forum. Please do that.


(Vincent Regaud) #43

From which post would you like it split?
And what title would you like for the new thread?


(YvesHanoulle) #44

from @David_Graeff post

title: next generation design : idea’s & discussions


(YvesHanoulle) #45

thanks


(David Graeff) #46

I see the developer perspective, of course. It is about maintainability, basically.

I have this suggestion, two folded:

Thing files as input

  • Use Thing/Item files for a “first-start” input only, by for example copying those files into an “input/” directory.
  • Works on start as well as while OH is running.
  • OH consumes those input files and removes them on success (every single line could be imported and didn’t cause a conflict) and let them sit there on failure (maybe creating a thing-file.thing.failed log file).

Sync back to Thing files

  • Via a command (openHAB console, REST API [Paper UI]) openHAB can store its current state back to Thing/Item files. Via parameters openHAB can be instructed to split Things to multiple files. Comments for Things are restored, global file comments are restored (if a generated file matches an input/ filename).

Would this approach be viable?

The important part is, that Thing/Item files are not viewed as constant definition files like they are now. Every input should go through the openHAB API and stored internally in however openHAB like it do to for performance and conflict-avoidance reasons.


(Markus Storm) #47

Wasn’t there a mentioning that the current implementation is based on libs that do allow for reading but not for writing back to .things ? It’s an idea I like but it would probably be a pretty large change[addition] possibly noone is willing to take on.


(David Graeff) #48

For the maintenance standpoint that would be a huge advantage though as Thing/Item file handling is fully decoupled from openHAB.
If at some point it is dropped from the official distribution (and probably made available as external bundle) it can be maintained by all those who see this very valuable for their setup. Could be made available as addon under the “misc” section, and people could even provide similar bundles that write Yaml, XML, MS word documents for Thing/Item inputs/outputs.


(Juelicher) #49

@ David_Graeff
A would have to think a bit longer about your suggestion, but at a first glance I see the benefits of this approach. It would solve a lot of the use cases, where I would currently just edit the text file with a few extra step. I would not mind those steps, and they could probably be automated.

If the “exported” files are a 1:1 representation of the internal openHAB configuration and if they maintain the order, those files could be stored in a source code management system. That would allow openHAB to use an internal format, that is best suitable, without the need for making it editable by humans. For example, without knowing if its fits the internal needs of openHAB, I could imagine some sort of embedded SQL database (e.g. SQLite) with all the power these would bring

It would also solve the problem of a migration path from the old to the new configuration, even if the current files would have to meet certain formatting standards.

Blockquote
I don’t believe everything can be seen.

Certainly not, and I agree that any approch should provide enough flexibility. But I am apposed to drop requirements that someone states at an early point, just because they seem to cause problems.

BTW: Thanks for creating the new thread!


(Scott Karns) #50

I’m late to join this great discussion, but I would like to start by thanking @David_Graeff for starting this, as it is much needed.

In my opinion, this would in general be a good thing, although I would still use my editor of choice to edit configuration files. The most significant added capability for me would be the ability to configure things/channels/items using the UI for a new binding that might not yet be well documented, then have the textual form of its configuration going forward.

Why would I insist on using my editor(s) of choice you ask. It’s simply because I am most productive with the editor I have used for the past 40 years. I didn’t start in the computer world with anything close to a GUI, I have always been most comfortable working from the command line. I realize that is not the case for the majority of people, but there is a sizable minority out there who feel the same way.

My biggest gripe about using a GUI for configuration is that it is easy to forget some checkbox, radio button or other configuration item, particularly when configuration for something is complex and I haven’t used the GUI configuration tool in a relatively long time. I can currently place comments in textual configuration files as reminders to my future self, which I find to be a very high value capability.

I have to agree that the current syntax of textual configuration files is quite arcane – in the extreme – and, as a result, error prone. A more structured textual configuration syntax, e.g., JSON, YAML, XML (blech!), … would be a major step forward, but, IMO, it would have to support comments, both in-line and block. Also, when the source of a change or a new configuration item results from my edits, order would have to be maintained, otherwise maintaining my configuration in a source control system would be damn near useless.

The above is nothing more than my opinion, don’t take my opinion as my view that there is no other way to do this, only that these are the aspects of configuration that are most important to me.

Again, thank you for initiating this discussion, David!


(Garrett Porter) #51

In response to David’s original question - sign me up for the “one UI” system that can edit my items, things, bindings, sitemaps, etc. I’m at a point where most of my stuff is openhab 1.x and I’m trying to get past my mental roadblock to understand 2.x. If there was just one way to do things and I could see/edit that file then I think I would get understanding sooner.

I tried VSCode last year and failed miserably. I never got it to link between VSCode and my OpenHABIAN system. I should try that again.

Mine is Insteon. Also Expire - but as David said that’s coming in 2.x soon-ish.

I like all the talk lately about “what do the users want” and “how can we make things better” so I summarized my experience here if anyone would like to read way too much about my progress.


(YvesHanoulle) #52

in this whole discussion I feel there are two idea’s mixed up

A) We need a better format for the input files: the current format is not so computer friendly
B We need to save the configuration

This has also created a third ida:

C) we need to have the same place for input and output

And we had most discussions around that last © part.

I’m a big fan of going forward in small steps.

I think that if we have already A a lot of problems with the current format could be gone.
B could be worked on seperate and would not give much upgrade problems.

Then later we can see if we still need C.

If we do, going from A +B to see, will be a lot easier for upgrade (as they are both computer readibel and testable files)

yes that would be more work in the end
yet work that is much safer and easier for upgrades.

and at this moment I think it’s too early to know if we really need C.


(YF) #53

For completeness, we need to have well-defined documentation on how to reset the system and re-load everything from the .things and .items file generated in your “Sync back to Things files” scenario above.


(Rich Koshak) #54

No where am I saying I want to get rid of the ability to manually create Things. And even today you can manually create Things that go into Jsondb. You can create them. manually through:

  • PaperUI
  • Habmin
  • curl and a properly formatted json
  • manually editing the JSONDB

None of these are really optimal though so we are talking about what we need to put in place to make getting these manually created Things into the Jsondb better.

I was guessing the problem was caused by the xtend libraries. That issue makes it seem like the whole text config loading is basically FUBAR, which we kind of already knew.

The reason there are two ways to do things is because of the backwards compatibility.

It’s a good list.

Since of it is already supported by the db, others are not. but it’s at heart something to take into consideration.

But I also agree with David. In order to have a transition path for beginners who have out grown the uis to a more text based config, both have to interact. This is going to mean some compromises somewhere

@vzorglub has kindly done so. It never occurred to me to do it before now. Thanks Vincent!

It addressed my biggest concern which is having a migration path from one approach to the other.

The NGRE Rules might be some trouble but I think we can work it out. (it’s a can of worms we don’t need to open here).

That’s a standing requirement for everything we do. Sometimes we miss the mark but it’s anyways a requirement.


(Sebastian) #55

Hi erveryone,

just want to add my 2ct from a user perspective (although most points have already been mentioned)

So my point of view is a let’s call it “advanced user view”. I have no idea about openHAB internal software architecture and its development but I’m using OH since 1.6 I think and still love today to do everything possible with text based config files.

Version control: yes
Human readable: yes
Human friendly: no

I get it that the currently used text files (things, items, …) are limited and can’t be adapted to support all the new features that are already there or will come up in the future.

As long as there still would be a way to do a text based config that is more human friendly than the current JSONDB implementation I would absolutely be happy with this, because …

:+1: 100% agree

So, this would be my requirements for more human-friendliness
(I know some of them might be quite big wishes but you asked for them :wink:)

As an advanced user I would like to have the control on where things and items are stored to structure my system according to my needs

E.g. maybe I want to group my things by technology (hue, zwave, …) and my items by room

As an advanced user I would like to use a text editor with syntax and semantics checking to reduce the risk of nok config files

For JSON(DB) this is already possible with e.g. VSC, but I would also like to see an addon like the current openhab extension to check the semantics.

:+1: and I add:

As an advanced user I would like to have config files that are not changed by the system when there is no actual config change to get proper version control diffs

What does that mean: I’m just starting with gaining some experience with JSONDB but what I don’t like is that it seems there are config settings and other things (let’s call runtime information) mixed in this database. E.g for the Zwave binding there is a property “zwave_lastheal” for each thing that gets updated every night after the heal. So every day the version control system reports the JSONDB file as changed although the config hasn’t changed really. I don’t know if there are more examples that use the JSONDB like that.

This surely is intended as the architecture is today but when talking about an official way for text based config that should be separated in two different files/databases/whatever

As an advanced user I would like to have a config text file format that is “human friendly” to be able to edit it with a text editor easily

Although I know there is more information in the JSONDB than in the *.things file it feels much more complex to edit or even write something like this

"network:pingdevice:192_168_178_50": {
    "class": "org.eclipse.smarthome.core.thing.internal.ThingImpl",
    "value": {
      "label": "Handy",
      "channels": [
        {
          "acceptedItemType": "Switch",
          "kind": "STATE",
          "uid": {
            "segments": [
              "network",
              "pingdevice",
              "192_168_178_50",
              "online"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "network",
              "online"
            ]
          },
          "label": "Online",
          "description": "Gibt an ob das Gerät aktuell online oder offline ist.",
          "configuration": {
            "properties": {}
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Number",
          "kind": "STATE",
          "uid": {
            "segments": [
              "network",
              "pingdevice",
              "192_168_178_50",
              "latency"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "network",
              "latency"
            ]
          },
          "label": "Pingzeit",
          "description": "Gibt an wie lange ein Ping in Millisekunden an das Gerät dauert.",
          "configuration": {
            "properties": {}
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "DateTime",
          "kind": "STATE",
          "uid": {
            "segments": [
              "network",
              "pingdevice",
              "192_168_178_50",
              "lastseen"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "network",
              "lastseen"
            ]
          },
          "label": "Zuletzt gesehen",
          "description": "Gibt Zeit/Datum an wann das Gerät zuletzt gesehen wurde.",
          "configuration": {
            "properties": {}
          },
          "properties": {},
          "defaultTags": []
        }
      ],
      "configuration": {
        "properties": {
          "hostname": "192.168.178.50",
          "refresh_interval": 60000,
          "refreshInterval": 60000,
          "retry": 1,
          "timeout": 5000
        }
      },
      "properties": {
        "uses_dhcp_listen": "no",
        "dhcp_state": "No access right for port 67. Bound to port 6767 instead. Port forwarding necessary!",
        "presence_detection_type": "",
        "arp_state": "Disabled",
        "uses_ios_wakeup": "Yes",
        "icmp_state": "System ping feature test failed. Using Java ping",
        "uses_arp_pings": "no"
      },
      "uid": {
        "segments": [
          "network",
          "pingdevice",
          "192_168_178_50"
        ]
      },
      "thingTypeUID": {
        "segments": [
          "network",
          "pingdevice"
        ]
      }
    }
  }

compared to this

Thing network:pingdevice:handyS "Handy Sebastian" [ hostname="192.168.178.50", port=0, retry=2, timeout=5000, refresh_interval=60000 ]

So again the question: Which information belongs really in a config database/file and which information should be stored in a different database/file?

As an advanced user I would like to add persistent comments to my config file to document my setup

Cheers
Sebastian


(David Graeff) #56

Actually… Think like you are an openHAB newbie. In Json I assign “label” to “Handy Sebastian”. But there is no self explanatory ordering in the Thing file line and we keep adding stuff (tags, semantic tags, groups, location, properties, configuration) with more and more strange syntax ("@", square brackets, round brackets, quotation characters).

I think I like the idea of using yaml as a successor file format, that follows my idea from above (reading it as soon as it hits an input/ directory and outputting it, when asked). Like in:

# My awesome comment for the next Thing
network:pingdevice:192_168_178_50:
  label: Handy
  configuration:
    hostname: 192.168.178.50
    refresh_interval: 60000
    refreshInterval: 60000
    retry: 1
    timeout: 5000

The following still holds true for my idea:

  • Files are used as batch input and configuration means. (They are not watched permanently)
  • Files are one way to perform and restore a backup. Restoring the jsondb would restore also the runtime information though and might still be the preferred way (because it also contains sensitive information like pairing ids etc).

(Sebastian) #57

Absolutely. I’m not stuck with the current config files and agree they are neither self explanatory nor will we have enough different kinds of brackets with new features coming.

I like your example and also prefer yaml to json syntax. Also the reduced information compared to the current JSONDB format.

Here I’m not sure what to think about this idea.

I think I don’t like the part of removing successfully parsed files. Doesn’t this mean you have to copy your files between different locations each time you want to perform a config update?

But I like the idea of considering the config files just another kind of UI (did I get this right?)

So jumping on the bandwagon I can image the following setup:

  • JSONDB will be the single source of truth for the config (and also the runtime data like mentioned above)
  • JSONDB can as is be edited via GUI, REST, KARAF
  • JSONDB shouldn’t be edited by hand (-> “real” database could be used if beneficial)
  • textual configuration is possible via yaml configuration files in a well defined directory structure (this is the replacement for current DSL config files)
  • this yaml files contain only config information, no runtime information
  • by some trigger signal (file changed or rest call e.g.) this files get loaded into openHAB
  • if the parser finds a difference between yaml file and JSONDB the DB is updated
  • another trigger will do the opposite: compare DB with yaml file. If there is a difference detected the user is faced with a message like “config_whatever.yaml was changed via GUI. Do you wan’t to update your yaml configuration? -> YES or NO”. If yes download config from DB back to yaml file

Just some brainstorming, no idea if this in-line with other users UseCases or even feasible with the openHAB design


(YF) #58

I really like that yaml format. But let’s dive it a bit further. Effectively we still support two formats: the internal storage (JSON DB) and the input/backup format which could be yaml. Why don’t we then provide an option to support reading and watching those yaml files, since the core kind of support it already. This would satisfies the needs of advance user.

The restriction still holds that what the original yaml file the user created is not static; OH is free to write additional info to that file. So essentially, the same as what we have to day with the .things file. However they are in yaml format and they can be changed by OH.


(Hilbrand Bouwkamp) #59

Here are some thoughts (Some parts have been mentioned by others).

The current jsondb files don’t seem to be designed to be edited by humans. (The fact it contains db in the name should have been a hint). This doesn’t directly mean it isn’t usable, but might need some changes if it would be the replacement for openHAB 1 config files. Here are the points:

  • The json files contain redundant data. For a database this is not uncommon as sometimes it’s more efficient. But for human’s this only adds problems, because if you forget to change all you will get errors or worse it will not give any feedback. For example see uid segments in channels it also contains references to thing id.
  • The files currently contain state data specific for the system, that is not specifically useful for the user. For example it contains all channels of a thing. You can leave them out of course, but then you can not add them in PaperUI without adding them in the json files. (This is also a known problem when a binding is updated and gets new channels, you than need to read the thing).
  • It’s a format that can be easily read by computers, but there is no defined order. Therefor the system can store them in any order it likes. This is a problem if you want to manually compare changes, because if the order changes for whatever reason it’s difficult to compare version.
  • Files that can be changed by users need a lot of additional checks to give feedback to the user in case something is wrong. I don’t think that is currently the case with the json files.

So I would not say current jsondb files are ready to be a replacement of the openHAB 1 configuration files…

In general having configuration files that are editable by users and can be written by a system are very hard to develop. It’s like having generated code that can be edited by a user. Because when the system reads these files it needs to keep track to the structure in the file in order to keep the structure. This is also very likely why openHAB 1 configuration files can only be read and not saved. If the configuration files can contain comments complexity increases. Compilers become much more complex when they need to keep comments. I’m not saying it should not be done, but just to remind that it’s really not trivial to make it happen. It helps to have a format that has this out-of-the-box with libraries supporting it.

There is another complexity. If users can store configuration in their own named files the system needs to track which configuration is in which file. Because if something is changed by the system it must be stored again the same file. Also new configurations made in a ui need to be stored in a file and probably the user wants to select what file. Therefor the system must get knowledge about these files. Currently that is not in the code. It would also mean this information needs to become available in the user interface.

Yes yaml files might look a like the new nice shying thing, but I’ll just quote balloob (about yaml configuration in HA): ... configuration.yaml will not go away, but just solely relying on it is preventing new features of being built...

So I think we should really think about how configuration files should look.


(David Graeff) #60

But HA has exactly the same problem. They have an internal state and a bunch of yaml files, created by the user, that the system cannot write back to.

I agree. But they are suitable for git backups, because only changes appear in diffs.

That’s why I suggested a “storage” assignment for Things and Items.

The comment can be stored in the Thing, just another field. I wouldn’t allow free floating comments. They cannot be restored easily.


That is our current problem that we want to solve! We have two sources of truth (“storage” that is read on start-up) then. We want exactly one, the internal one that is most efficient for the core.

Other inputs are really just temporary inputs.

I like the summary, except this part. A user will be confused if the file is processed on a change, but not on boot-up. A REST api call / openhab console command to read from the files is possible though. We wouldn’t need a separate “input/” directory then.