Lots of people manage multiple OH instances for multiple clients. Having the ability to create “modules” of configuration which can be deployed as units to some but maybe not all these instances would be useful. With Blockly support that also means that the end users of these OH instances have a configuration that can more easily read, understand, and modify.
Does this mean we can have individual files created by the UI?
That would be OK. The JSON file is a bit of pain to work with (but I can live with it).
Individual files controlled by UI would be the best of both worlds?
No. The UI can only modify the JSONDB. Any config loaded from anywhere else will be treated as read only in the UI.
Hello everyone,
Interesting discussion and interesting activities on-going !
I’m trying to understand if there is a way to enjoy the benefits of versioning .items, .rules etc… files (e.g. with GitHub) while being in the Main UI world?
Jsondb deltas do not seem to be very easy to read…
So what would be the solution? Generate the YAML version of jsondb, and use that to snapshot the config in GitHub?
I’m pretty sure many of you are already doing this (versioning of config, in a readable way, with deltas between x and y versions etc…) one way or the other and I am interested in understanding what you are doing…
I like what @andre showed in this topic:
But a key point there is that he is using file based config.
Even if I would let AI use REST API to update the config (and so possibly would not be able to use the approval solution he has put in place), I would like to be able to a posteriori track what AI did, in a user friendly way…
Am I making sense?
There are many aspects to this, and I don’t mean to comment on all of them, but just need to emphasize one thing: Git and GitHub are two completely different things. Git is an open-source tool to handle versioning of files, GitHub is a corporate website that offers hosting of Git repositories. You can very much use Git without ever touching GitHub, and whatever you do, do not put your OH configuration on GitHub. It will contain API keys, passwords etc., which would then be publicly available for everyone, unless you pay for a private GitHub repository of course.
I use Git to store my OH configuration myself, both for file based configuration and JSON DB, the way I do it works reasonably but have some quirks. But, crucially, I don’t host it on GitHub. I use my own “Git server” which is basically just a local web server that is only reachable on my local network, and that uses encryption keys for authentication - so that it won’t serve you anything if you don’t have a valid key.
You could run something like this that would split the json file into individual files that you could also keep version control with and backup?
#!/bin/bash
#This splits the json file into indivual files so you can keep them incase you change something
#and you can just replace that part rather than restoring the whole json file
cd /var/lib/openhab/jsondb/
mkdir -p split
for file in *.json; do
echo "Processing $file..."
# strip .json extension for folder name
base=$(basename "$file" .json)
outdir="split/$base"
mkdir -p "$outdir"
# detect if JSON is an object or array
if jq -e 'type == "object"' "$file" > /dev/null; then
jq -c 'to_entries[]' "$file" | while IFS= read -r entry; do
key=$(jq -r '.key' <<< "$entry")
value=$(jq '.value' <<< "$entry")
[ "$key" = "null" ] && continue
echo "$value" > "$outdir/$key.json"
done
elif jq -e 'type == "array"' "$file" > /dev/null; then
jq -c '.[]' "$file" | while IFS= read -r obj; do
uid=$(jq -r '.uid // .name // .id // empty' <<< "$obj")
[ -z "$uid" ] && continue
jq . <<< "$obj" > "$outdir/$uid.json"
done
else
echo "Skipping $file (not array/object)"
fi
done
Probably the next step is to use file-based YAML config, but make it editable via mainui.
Yes there are many challenges to this:
What would be the point of that? There’s nothing wrong with JSON except that it’s awkward to write/edit scripts without multiline strings. But, when you do the editing in the UI, that doesn’t matter.
If JSONDB was made into YAMLDB, nothing would be better as a result. JSON isn’t “worse” to use with Git than YAML.
You “can’t”/shouldn’t have files that are both managed automatically and maintained manually. That’s just an endless mess of small formatting changes that messes everything up. It’s extremely frustrating to try to “work by hand” with files that are generated automatically.
If anything were to be done, I’d say move JSONDB to JSON5. It fixes the shortcomings of JSON by having multiline strings, comments etc., which would make it superior to YAML in every way. But, since humans aren’t supposed to manage the JSON files manually, the lack of multiline strings isn’t really a problem. It would be nice if comments were preserved for things like pages and widgets though.
I wasn’t suggesting moving away from jsondb to “yamldb”.
Everything jsondb related stays as is (managed entity etc).
Only make the file-based yaml config that we have now → editable.
Many systems do this. Example: I can edit my Frigate config using their UI (e.g. toggle a feature, add mask zone visually by dragging over the video) and using their web based yaml editor (similar to our code editor), as well as edit the yaml directly. Their visual editor updates the same YAML file just fine.
That could apply to openhab: we can visually modify page/widget layout using mainui’s “wysiwyg” (yes it could use some improvements) and save it straight to our file-based yaml.
That’s a totally different thing to my suggestion. It has no bearing whatsoever to the yaml idea, and if that’s a good idea, which it sounds like it is, then sure! It can only make things better.
I somehow think I’d still prefer dealing with yaml than json when manual editing.
I don’t think OH core can handle that. The assumption that you only have one managed provider is “built in” in the entire logic, the whole resolution of “where to save something” doesn’t exist. If something is to be saved, it is sent to the one managed provider.
I know, and I hate it. But, it depends a little bit. If the structure is “simple enough”, it might work reasonably well. But, having something formatted, only to have some automation overwrite it all, is just so extremely frustrating.
That’s a matter of taste. I can’t stand whitespace-based formats and hate the whole idea that a space character extra or missing will invalidate the whole document. I very, very much prefer using brackets or similar for structure, and let whitespace be irrelevant. Also, YAML mandates a 2 character indentation and space only, which is also a real pain.
That is an “implementation detail issue”.
Simple: You can simply not use that feature. We can add a system option: “allow UI edit” (or whatever)
Indeed. I love how openhab supports multiple ways so everyone is happy.
Thank you Nadar for pointing this out. Yes, I probably should be careful with having my secrets stored on github.
One thing though : github does allow as many private repos as needed with the free tier. It does not mean it is a good idea to store secrets there, but I thought I’d mention it.
Of course it can, just like you can edit YAML or DSL source files today in operation:
the running OH instance file config provider picks up your edits.
For every active component, OH core knows the source (UI or a file).
I re-tried to version control the raw jsondb as my last attempt was a decent while ago… And I did setup my own git server @Nadahar, thanks for the note.
I found out also that I can filter some changes like wakeup info for zwave things from the versioning to avoid pollution (on top of filtering out some files which are not relevant for what I’m trying to do) and the end result is better than I remembered.
I’ll see how it goes and if it helps monitoring the changes AI makes on the config…
I took note of the various options like split the files into pieces before versionning and / or generating yaml versions of it before versionning, thank you all.
Yes, add whatever you don’t need and that is making noise into .gitignore. They don’t need to be part of the backup, and don’t need to show up in what’s changed. I don’t mean to say that this is “the right definition” in any way, but this is the current content of my OH .gitignore:
Thumbs.db
readme.txt
/conf/services/addons.cfg
/conf/html/index.html
/conf/automation/jrule/items/org/openhab/automation/jrule/items/generated
/conf/automation/jrule/jar
*.dpkg-dist
/userdata/.karaf
/userdata/cache
/userdata/jsondb/backup
/userdata/jsondb/users.json
/userdata/persistence
/userdata/secrets
/userdata/tmp
/userdata/config/jmx/acl/org/apache/karaf
/userdata/config/org/apache/felix
/userdata/config/org/apache/karaf
/userdata/marketplace
/userdata/config/org/openhab/core/service/WatchService
/userdata/etc/host.key
/userdata/etc/host.key.pub
The exact paths depends on how you’ve mapped this to Git, I’ve created a new “srv” called openHAB-git, which contains both openHAB-userdata and openHAB-conf, which allows me to store data from both in one Git repo. Thus the paths in the above file are what they are.
I never said that it couldn’t react to changes, it can. I’m saying that the current logic in core can’t handle more than one writable/managed Provider. It can have an unlimited number of Providers, but all “writes” goes to the one that is managed. Therefore, if you want to edit YAML from the UI, you must either replace JSONDB with YAMLDB, or you must rewrite the logic that governs how editable resources are handled.
I would also be careful with having AI (or any external system in general) access secrets.
Many systems support using env vars in config properties, I think that would be a nice feature in openHAB as well. Just as a proposal: you may have a $ENV:MY_SECRET$ syntax or something like that to load secrets from environment instead of putting them into config directly.
Absolutely true.
But, I would also be careful with making non-secrets available publicly or to one of the “spying cloud providers”. Because, even without the secrets, the configuration might reveal a lot of details about your home that you don’t necessarily want to be public knowledge.
A few years back I requested that order be preserved in the JSONDB for just this purpose and the maintainers agreed. Therefore adding or deleting an Item (for example) doesn’t cause the whole file show up as a diff as all the entries are reordered. This makes git and the JSONDB behave quite well together. What you see in the diff is just what actually changed.
I run in Docker so both my config and userdata folders are under the same root folder and I check them both in to the same repo. For another example, here’s my .gitignore. I’ve not reviewed it in a long time so there may be stuff that should be added to it.
userdata/*
!userdata/secrets
!userdata/etc
!userdata/jsondb
!userdata/config
!userdata/habot
!userdata/openhabcloud
!userdata/uuid
!userdata/zigbee
!userdata/zwave
userdata/jsondb/backup
userdata/backup
.Trash-1000
conf/html/hum.jpg
conf/html/light.jpg
conf/html/power.jpg
conf/html/temp.jpg
conf/automation/lib/javascript/personal/node_modules
conf/automation/js/node_modules/*
!conf/automation/js/node_modules/rlk_personal
*py.class
*.swp
*~
*.old
Note the syntax is a little weird. The first part excludes all of userdata from git but then adds exceptions where I pick and choose which folders to add. So new folders that show up in userdata do not automatically get checked in.
Thanks a lot @rlkoshak and @Nadahar for sharing your detailed configs… I’ll review in details.
One thing I discovered this morning is the possibility to filter out some changes in a file that I otherwise want to track i.e. the things file, by adding a .gitattributes file with this content (e.g.):
org.openhab.core.thing.Thing.json filter=zwave-clean
This automatically removed from the diffs the changes in the things file which contained “zwave-clean”…
Same works for some heartbeat info that changes often in the things file and that is not really what I’m trying to track here.
Filtering it out from the diffs is one thing, but what will it do when you make a commit? Will it then commit the “filtered content” in the state it was before you enabled the filtering, or as it currently is?