Config: text file vs jsondb

Setup, Configuration and Use Items & Sitemaps
1

There has been a lot of discussion in the past few months about text based item/thing configuration and whether it will still be around for OH3 (the answer is yes, but that isn’t my question). I find myself liking said text files less and less though, sure it’s necessary for me to keep them in a git repo because my setup is too big to change things by hand, but the syntax is bloated and a little limited. I’ve just built some rules that will automate the discovery of any new Espurna devices and generate items and things for them, which is a problem because my current MQTT bridge is sitting in a text file, making it un-editable.

So where do we go from here?

  • New file format we could change the format of the files, there has already been discussion about using yaml but I don’t think a decision was reached. But with this option we still have the problem of un-editable items and things.
  • jsondb has been suggested as an alternate place to edit items and things, but is it safe to put this file in a git repo? When OH writes it out do the contents change order? OH wouldn’t care but git will. If the order is guaranteed, then this is an option and perhaps I will write a tool to import/export from yaml so that it is more human-readable.
  • external tool I was thinking about maybe making an external program (that could be run by OH in an unattended mode) that would import items/things from yaml files and add/update them via the REST API. It would be necessary to export them from OH and handle conflicts as well, since the items and things added by it would be editable.

What are your thoughts on these options?
Are you having the same issues with the limitations of the items/things file syntax?
Are you also having issues with mixed dynamic/fixed thing definitions?
How are you getting around these issues?

2 Likes
2

My first question that I am not understanding why are thing. Files uneditable?

3

The files are, the things added from files are not. The auto discovery is running in a rule and I don’t want it to be modifying the item/thing files.

In the MQTT2 binding the bridge is a thing and you define the MQTT device things below it in the thing file. Because of that relationship, all things linked to the bridge must appear in the file and new ones cannot be added at runtime and linked to the bridge.

That gets me thinking though, this is the only major issue I’ve had with file generated items/things. If we say that files trump what’s in the jsondb when files are read by my proposed tool it wouldn’t need to export back into the files. That would make it quite a bit simpler.

4

What makes you think that? It is not recommended to mix file -based and PaperUI-based setups, however it is possible to do that. IMHO is is not recommended because the user will VERY EASY mix up what is possible and what not! In the case of a file-based MQTT-bridge and a PaperUI created MQT-thing, the thing can be manipulated only via PaperUI, the bridge only via the file!

2 Likes
5

interesting discussion
All my things are created in PaperUI and all my items are in text files
nothing to add but following, hope this can be discussed with out getting ugly

6

Same here… I like the discovery part of paperUI, hence let OH deal with that part, whereas the items and sitemaps I want to keep more stable. I don’t like the UI for linking those

7

Comment: I make extensive use of Modbus binding. This is a low-level protocol, with no discovery abilities. It requires extensive manual configuring in openHAB - devices can have hundreds of datapoints, no two designs are even vaguely alike.

For that purpose, file based config is more usable, particularly as it is easy to see hierarchy and compare Things and channels at a glance.

It’s the exact opposite of, say, z-wave, where everything is discoverable and no-one would want to configure all that detail manually.

openHABs flexibility suggests there are likely to continue to be valid reasons for both methods.

2 Likes
8

I tried it. I am creating things and channels dynamically.

I agree. The chances that I will need to make changes via the UI to items/things from files are slim. I’m also not looking to manage things on the bridge via the UI really, what I really need is the ability to add things dynamically to the bridge which comes to the same as using a UI.

9

I agree. All of my non-dynamic items reside in files and I like it there because I can add and remove with git checkouts and I want to keep that ability.

I agree, I’m not suggesting we add or remove either one. I’m saying that my needs have outgrown the abilities of the text files and I’m looking to see if anyone else has had this problem, how you dealt with it, and what you think about my thoughts on a way around it.

1 Like
10

I do and have experienced no problems.

I don’t know if the order is guaranteed, but in practice I’ve never had it save in a different order. New stuff gets appended to the end.

If you go down this path, it might be worth filing a feature request issue to add in a guarantee of order when the DB gets saved.

Wait until OH 3.0 Beta comes out before spending a lot of time on anything. Lots of stuff is changing with OH 3 and I fear any effort you spend now will have to reworked significantly when OH 3 comes out. One of the biggest changes is that PaperUI is being completely replaced with a brand new UI which I’m certain will be much more user friendly.

Items don’t bother me too much, but when OH 3 comes out I plan on migrating them all to JSONDB. Yannick has written already an importer that will take .items file configs and import them into JSONDB. On another project I used a very early prototype of that feature to import around 40 Items from .items files to JSONDB. It took all of two minutes. I believe there is a similar tool done or at least in work to do the same for .things files.

.things files bother me a lot, mostly because even though I refuse to do more than the barest minimum to provide support for them on this forum, syntax errors and ignorance over what the fields are consumes far too much of my overall support time on this forum. I’m not saying using PaperUI is perfect, but at least all the fields are apparent and it’s almost impossible to create a Thing with syntax errors.

No because all of my Things are in JSONDB and all of my Items are in .items files. I might start to experience similar problem if I decided to migrate my Rules to JSONDB before OH 3 comes out as not everything I need (mainly triggers) are supported right now with JSONDB Rules.

All Things go in JSONDB. All my Items will go into JSONDB when OH 3 is released (there are a couple of things to do with metadata that makes it more convenient for now to keep Items in .items files. When OH 3 comes out, all of my Rules will go into JSONDB along with anything else that is made available (sitemaps, persistence, etc).

As part of some work I did on the HestiaPi, which is a thermostat running openHAB on an RPi W 0, I migrated the config from a 100% text based config to a 100% JSONDB based config. The improvements in performance, flexibility, and usability were huge. For one metric, moving the configs to JSONDB took it from a 25-30 minute boot time to around 10 minutes. That’s still a long time but it’s an RPi 0, not exactly supported hardware.

It’s worth mention that whether or not the JSONDB files get saved with in the same order, this is still possible. What you lose if they get saved in different order is the ability to easily see what actually changed when a file gets checked in, which is a big deal.

Anyway, my conclusion is that I would try to get the JSONDB configured so it guarantees order when it saves, if possible and move everything to JSONDB by filing an issue. If that isn’t possible because it’s too much work or the devs don’t want to do it, I’ll still use JSONDB for everything. The stability and speed it provides over text based configs are well worth losing the ability to easily see what changed in the JSON files in git (and in my experience I’ve not actually seen it save in different orders in practice). I don’t have anything like Modbus with the concerns that rossko57 mentions but my hope is that the new UIs will make managing those much easier.

I’ll also mention that I came to this conclusion slowly. I started out as “all things much be in text files!” and gradually adopted to “all Things much be in JSONDB, everything else in text files!” But after moving Rules and all from text files to JSONDB for HestiaPi I am a full up JSONDB convert. Granted, the benefits are probably exaggerated on an RPi 0, but the benefits were really large and surprising.

2 Likes
11

Thank you for the first-hand account!

Good idea! I can’t seem to get my head around Java to be any help, but maybe a small modification like this I could work on and get some help getting an IDE setup and be able to compile.

I know, I’m looking forward to many of these changes like the UI (which looks great!). I share the hesitance to spend much time on something, but if such a tool were to use the REST API I think it would be fairly safe as I don’t expect that will change much.

I read all about this last fall when he introduced it, and reread it over the weekend. It’s not quite what I want though. If all I wanted to do is bring everything into the JSONDB and no longer manage via git then that would be perfect, but that’s not going to work because I have several rules generating items and things dynamically that I don’t want to be considered by git.

My Espurna auto-discovery rules add a thing and basic device statistic channels when a new device is discovered. As channel topics appear it then creates more channels on the thing and creates items for them which it connects to Selector, which in turn generates more items to allow command/update endpoint linking. None of these things or items should be considered by git.

I’m sure! This is a major driving force in my desire to move away from the current XText files.

Neither do I, but I want to keep my structured and separate files for my static items and things that I can keep in git. This is why my thoughts are pointing towards a tool that will import from say yaml and update the JSONDB to match those files.

Final thoughts

I think an external tool is still going to be the best bet to get what I want, but I think feeding directly into the JSONDB files while OH is offline will be the way to do it. Such a tool would need to keep track of the last items/things/etc it put into the JSONDB and add/remove/modify where differences exist, but I had already considered this when I came up with the idea. I’m not seeing any other way to have both static items/things in version control and dynamic items/things excluded from said vc.

I would also move my items into this tool because of the speed gains from moving away from XText, but also to keep my git control and ability to have them in separate files.

Thanks for your detailed and helpful input as always, Rich.