Add an include option to the jsondb files for file-based configurations

CrazyElectron · December 31, 2023, 8:29am

Before filing a request on Github, I wanted to discus the idea here in the forum.

The use case:
Fully automated setup and configuration of OpenHAB and supporting systems (a Kubernetes cluster using Ansible and encrypted variables in a Github repo in my case) with configuration files instead of UI.

I am finally moving from Sitemap to mainUI and noticed that it is very much Web UI oriented and not so much config files oriented. The struggle I have with the current jsondb setup is that all entries of a specific category end up in one big file.

Looking at userdata/jsondb/*.json files I came up with the idea that an include folder mechanism, much like a common method used in Linux, would be a great addition to simplify handling changes in configurations as well as automation of config deployments/updates. It would be an addition to the current mechanism without impacting the existing UX.

Let’s take the UI widgets as an example. I installed 3 widgets from with the UI and added a dummy custom widget. The resulting uicomponents_ui_widget.json looks like this (most details collapsed):

  "garbage_list_v1": {
    "class": "org.openhab.core.ui.components.RootUIComponent",
    "value": {
      "uid": "garbage_list_v1",
      "tags": [
        "marketplace:114871"
      ],
      "props": {
      },
      "timestamp": "Feb 3, 2021, 7:17:40 PM",
      "component": "f7-card",
      "config": {
        "title": "\u003dprops.title"
      },
      "slots": {
      }
    }
  },
  "rlk_battery_status": {
    "class": "org.openhab.core.ui.components.RootUIComponent",
    "value": {
      "uid": "rlk_battery_status",
      "tags": [
        "marketplace:128043"
      ],
      "props": {
      },
      "timestamp": "Nov 28, 2022, 8:17:12 AM",
      "component": "oh-list-card",
      "config": {
      }
    }
  },
  "washing_machine_v4": {
    "class": "org.openhab.core.ui.components.RootUIComponent",
    "value": {
      "uid": "washing_machine_v4",
      "tags": [
        "marketplace:116621"
      ],
      "props": {
      },
      "timestamp": "Oct 2, 2021, 9:24:59 PM",
      "component": "f7-card",
      "config": {
        "title": "\u003d(props.title)"
      },
      "slots": {
    }
  },
  "widget_53505866a9": {
    "class": "org.openhab.core.ui.components.RootUIComponent",
    "value": {
      "uid": "widget_53505866a9",
      "tags": [],
      "props": {
      },
      "timestamp": "Dec 31, 2023, 8:31:08 AM",
      "component": "f7-card",
      "config": {
      }
    }
  }
}

This can grow to a large collection when adding many widgets with everything in the same file, both your own custom widgets and installed marketplace widgets. This makes it harder to use a file-based configuration management, like (many?) OH users still do and prefer. And it also makes it harder to do more structural/larger changes to an existing setup because a simple backup/restore of the json file will not be enough, and you might vey well end up manually editing the file and copy/paste your custom stuff.

Similar situations could arise with the other jsonsb files but I haven’t looked into it yet…

My proposal is to implement an include folder mechanism. The UI widget example would result in a folder called userdata/jsondb/uicomponents_ui_widget.d/ where individual widget definition files are stored. They will be loaded after the base file uicomponents_ui_widget.json to make it behave predictably. This way, it is easy to maintain your own widget collection and have them updated, applied or removed as needed.

Thoughts anyone (besides telling me that file based configurations are a thing of the past… )?

splatch · December 31, 2023, 11:27am

I like idea, especially that there is long awaited separation of marketplace entries from user entries within UI components. This would make whole setup also a bit more similar to linux distros and their way to handle configuration of various daemons.

The jsondb is internal storage format which is managed internally by openHAB JsondbStorageService, which is then goes up to managed providers (for things, items) and UIComponentProvider and corresponding ManagedUIComponentProvider.
I speak about all this because I been looking into this problem a while ago with OH 3.0.x (before managed ui component was made) and brought own XML providers which allowed import/export definitions of things, links and items from multiple files (I could export it as one but import as multiple). I repeated same exercise with pages and widgets, however I found that XML representation of JSON was over bloated. Management of widgets and pages didn’t fit the model, so JSON seemed still better than my imaginary format. Still with all limitations of main ui rendering (i.e looping over items/groups/i18n) I started to dig and come to a point where I made a widget/page generator which used input files in hbs format (handlebars template) and produce valid page/widget entry in memory-only provider.
You can check example template: Example teamplate of overview page for openHAB made with handlebars. · GitHub.

Main point of above effort is to work in a bit smarter way and be able to construct specific widgets or pages from fragments, simply to avoid duplication of these all over the place.

rlkoshak · January 1, 2024, 6:45pm

There are multiple ways one could achieve the same result.

Personally, I’d like to see the extension of the YAML file work started to add support for custom semantic tags. I like the idea that what you see in the code tabs matches what you can put in config files. There was also talk about making it so you can mix Things, Items, metadata, widgets, et al into the same file.

With that you can create complete modules of config.

Given how the JSONDB is written and read from, I’m not sure adding a feature like this to it would give a satisfactory experience. For example, these are only read at OH start up so if you modified one of these subdirectory files, you’ll have to restart OH to pick them up.

I see no reason why includes couldn’t be added as a feature in this YAML approach. I worry that mucking with the JSONDB is going to be lots of work, potentially disruptive, and still end up with a less than satisfying result.

See RFC: YAML configuration · Issue #3666 · openhab/openhab-core · GitHub for details and further discussion.

CrazyElectron · January 2, 2024, 6:50am

That would indeed be great as it results in modules that are self-contained and provide everything needed.

That is indeed a somewhat limiting factor, but still separation between ‘distribution’ and ‘custom’ would be a nice start.

I’ll check what is happening over there and see if I can add anything to the discussion.

UPDATE: Very interesting discussion and I like the “move-to-YAML” line of thinking (coming from stuff like Kubernetes manifests and Ansible). Personally, I’m not a fan of grouping elements, as proposed by some, on the highest level and leaning more towards the kind: Item and kind: Thing approach. But either way, I think a template generator would be a great tool to support this YAML stuff. However, I will not add to the discussion as moving to YAML - one way or the other - will be a great step forwards.

rlkoshak · January 2, 2024, 3:30pm

Note, in the JSONDB there really isn’t any ‘distribution’ though. So that’s kind of a distinction without meaning. The ‘distribution’ version of these files is an empty file or lack of presence of the file in the first place.

So be careful not to carry your simile too far.

CrazyElectron · January 2, 2024, 6:42pm

I used the term ‘distribution’ loosely (too loose I suppose…). I meant to refer to stuff coming with OH and installed from marketplace (although that could also benefit from having its own folder).

rlkoshak · January 2, 2024, 6:56pm

Keep in mind that the marketplace is for stuff that explicitly not a part of the OH distribution at all. It’s all 3rd party individually maintained (or not) stuff published by members of the community. Even the add-ons on the Marketplace are not part of the distribution.

It also gets complicated because, at least for the block libraries and UI widgets (hopefully someday rule templates too) once installed they can be modified/customized by you. So if you install a UI widget and modify it, ? Where does that go?

CrazyElectron · January 3, 2024, 7:31am

I understand there are some complexities, but they can easily be dealt with. Having the option to separate what is coming from OH, from marketplace and what you create or modify yourself (which then becomes a custom part) would still be beneficial IMHO.
And like with Linux, having a folder to add custom config doesn’t mean you have to use it (for instance, I never put anything in /etc/interfaces/interfaces.d in Debian, but the option is there, just in case…)

rlkoshak · January 3, 2024, 8:45pm

Since nothing comes from OH in the first place, it’s really just distinguishing between marketplace and everything else.

And that’s kind of my point. In the Linux simile, software usually comes with a default config. OH comes with no config out of the box. A newly installed OH will have no files at all in the JSONDB folder. Even after going through the first run wizard, I don’t think there will be anything in JSONDB.

CrazyElectron · January 4, 2024, 9:06am

I’m not stating it is the same, just that it is similar. In Linux you can add 3rd party repositories - which are similar in nature marketplace in OH context - and install packages from there that can generally also use the aforementioned structure. But that’s not my point. The argument I’m making is that there should be a distinction between what you create/add/modify yourself and what is or can be installed and used as-is from OH (currently none) and other sources (what I would call the OH ecosystem).

Not entirely true , but nothing related to this discussion topic.
After initial setup wizard there is:

users.json
uicomponents_ui_page.json
backup/...

But, enough about this. I am looking forward to progress (OH 5? ) as discussed in issue #3666. Meanwhile I will use some Ansible and Ruby magic to get the desired results.

rlkoshak · January 4, 2024, 4:42pm

I should note you can have that distinction now.

Only two files in JSONDB comes from openHAB itself without user provided information, and with only two exceptions, the stuff from the marketplace gets stored in separate JSONDB files from user supplied configs.

If you use all text based configs in /etc/openhab instead of managed configs, all your stuff will exist there and only marketplace stuff will exist in JSONDB. Even if you use managed configs, with one exception (see below) user defined stuff and openHAB/Marketplace provided stuff are already segregated from each other.

Perhaps the current state of OH isn’t as bad as it first appears on this front. Except for marketplace UI widgets and blockly libraries, there already is separation between user supplied stuff and marketplace/OH supplied stuff in the JSONDB.

File	Marketplace, User, openHAB	Notes
amazonechocontrol%3Account%3Aalexa_account.json	User	Populated with user configuration of the Amazon Echo Control binding
automation_rules_disabled.json	User	Disabled rules
automation_rules.json	User	UI created rules
habot_cards.json	User	Saved cards created by users in HABot
marketplace_ruletemplates.json	Marketplace	Populated with rule templates installed from a marketplace
org.openhab.core.config.discovery.DiscoveryResult.json	User or openHAB	Populated with Things discovered based on user installed add-ons and configurations
org.openhab.core.items.Item.json	User	User created Items
org.openhab.core.items.Metadata.json	User	User created Item metadata
org.openhab.core.persistence.PersistenceServiceConfiguration.json	User	User create persistence configs
org.openhab.core.thing.link.ItemChannelLink.json	User	User create links between Items and Channels
org.openhab.core.thing.Thing.json	User	User created or discovered and user accepted Things
org.openhab.core.tools.UpgradeTool	openHAB	Populated with a record when the upgradeTool has run during an OH upgrade
org.openhab.core.transform.Transformation.json	User	User created transformations
org.openhab.marketplace.json	Marketplace	I think this is a legacy file not used any more
thing_status_storage.json	openHAB	I’m not sure what this file is for but it looks to have stuff in it stored by installed add-ons relating to their online status. It’s not storing user defined configs.
uicomponents_system_sitemap.json	User	User defined sitemaps
uicomponents_ui_blocks.json	Both User and Marketplace	User defined MainUI blocks
uicomponents_ui_page.json	User	User defined MainUI Pages
uicomponents_ui_widget.json	Both User and Marketplace	User defined and marketplace installed UI widgets.
users.json	User	User defined usernames, roles, and passwords.

CrazyElectron · January 6, 2024, 7:34am

Thanks. Useful overview.