Maybe I am not so technical minded, but what problems do you guys see for a migration as long as this issue isn’t resolved? Can you give me an example? A textual configuration of zwave is already possible atm, isn’t it? Ok, the things are discovered automatically, but the rest can be done through text files.
@sihui, @mike_mccaffery, @tobbesjufem To play devil’s advocate and channel chris for a second, why specifically does it HAVE to be text files? If you can just as easily recreate your environment by backing up and restoring a MapDB file why would that be unacceptable?
I’m not saying that OH 2 is there yet and would probably continue to recommend using 1.9 if it worries you, but I’m concerned people are too wrapped up in “it MUST be text files!” without even considering there might be other perfectly acceptable alternatives. I’d prefer text files too but at least I’m open enough to consider alternatives, particularly if the alternative is looking the automatically generated documentation for over 314 devices to find the proper syntax for my specific devices.
Besides, I’d rather he spent his time merging the SECURITY command class stuff and/or working to release the Zigbee binding.
Whilst more often that not, restoring the mapDB files seems to work, I have had a few occasions where I had to abandon the previous mapDB files and start fresh. This always happened after upgrades to OH following which something or other must have got broken. As an example, just today I had to go through a similar exercise when following upgrade to the latest nightly, my logs were full of errors writing to mapDB persistence. I tried all sorts of things to resolve including uninstalling/reinstalling persistence bindings etc but the problem just would not go away. Reverting back to a backup of the previous build that I was running got rid of the errors. In the end, I tried a fresh install without restoring the mapDB database (not persistence one - the main one used by OH for things/items etc) and all is working fine.
To be fair to Chris, certainly in my case, a new install does not take that long to do from scratch as all I had to do to do was (a) install my z-stick in Habmin, with the same device name as before and then (b) add all the discovered z-wave devices with their default settings. As all my items are defined in the items file and linked to the appropriate channels, the recovery process is only a matter of few minutes.
My question was more philosophical than addressing the state of OH 2 today. I am fully aware that there are problems with corruption and being unable to restore an old MapDB database file and I absolutely would not recommend that approach today because of all the problems you mentioned plus more.
But if it could be made to work reliably would it still be unacceptable? That’s my main question.
As often happens on this forum and elsewhere, people tend to focus too much on their perceived solution (e.g. how do I configure everything using text files) than on the real problem (e.g. how do I reliably backup, restore, and/or migrate my full OH configuration) which closes off the opportunity for alternative solutions to the problem.
Fair enough and yes, as you rightly point out, a reliable backup/restore/migration would probably address most users’ concerns. From a philosophical perspective though, the idea of text based things files is built in into OH2 and so if any binding - intentionally as opposed to being a work-in-progress - does not work fully with such files, it would seem to be breaking the model.
NOW we come to the point where I can see what is in the database and how to edit that stuff and have a little bit more control over it … maybe I’ll give it a try with zwave (don’t want to wait any longer …)
I perfectly agree with @smar .
In my case just for the zwave binding I have to keep the autoapprove option enabled (inside runtime.cfg), and because of this I have hundreds of Things coming up that I have no use for.
I understand that there are other priorities, but the text file configuration is part of the model and in my opinion should be made available for those who use it.
@chris: I make myself available to help you to modify the source code in order to add the nodeid parameter in the definition files of each zwave device xml file. You just ned to ask.
I will look at this in the near future, but there are a few things to consider which may not be so obvious (it’s not just that I’m being lazy or difficult in not implementing this ;)) -:
This is a breaking change so everyone will need to delete their configuration after this change!
I am working on some major changes to the binding so I really prefer not to make other changes to the master as merging is difficult
To help solve these issues I produced an alternative to the MapDB database that produces text files that are easily editable. I would really like to see this incorporated as I believe it provides a good solution (ie text files, automatic backups, and editable via either text editor OR the UI).
Thanks @chris and fully understand. I wasn’t pushing in any way - just giving an opinion. I’m sure I speak for all when I say how much we appreciate the tremendous amount of work you have put in into the z-wave binding. For me, this is the most important binding for my openHAB use.
PS Again from a personal perspective, I really do like the idea of your mapDB replacement as json files. I think this would address the robust backup/restore concerns that users have. Can’t wait to see it get merged!
Don’t worry - I didn’t mean to suggest you did (hence the smiley).
Thanks - as above, it’s not a question of time - the change is quite small but I’m trying to avoid changing the master branch at the moment. I would prefer to try and wrap up all the breaking changes into a single update if I can.
No - I will also update the binding to support the .items files. I agree that this should also work, but as above, this change will break everyones system so I’m not currently jumping into it… When I do it I’ll look at ways of limiting the impact.
Im not sure I was clear in my last msg so… im grateful to Chris for all his work so far, of course I am, but whilst we are in this development phase of OH2 it’s prudent to be able to go back to a well-known state, mapdb just doesn’t do that, chris himself has had issues with this in the past and he proposed a different storage mechanism – whatever happened to that ? Text files are the most basic mechanism to define that well known state as it stands, and I believe this is defined in the base ESH standard design, although it doesn’t specifically state it, im assuming this isn’t an optional mechanism. So until I can revert to a well-known state as OH2 progresses and things change I’m am happy to continue to run production on OH1 rather than migrate prod to OH2 and still have other OH2 test systems. I have even been down the route of trying to replay the XML via the REST api, which actually works reasonably well but after 30 or so devices just becomes too cumbersome to maintain. so unless you have a better option than MapDB, text files are the only viable option, it works well for all the option things I need to define.
I’ll just point out that this statement still focuses primarily on the proposed solution rather than the problem. My original question was to mainly just figure out if someone managed to make some binary database format reliable and support roll back to a previous state, even rolling back just one part of the DB back to a prior state, would you still find that solution unacceptable?
I agree, right now MapDB does not do this. MapDB may never ever be able to do this. But to make the leap from “MapDB can’t do it so text files are the only solution” is, IMHO excluding a whole host of potential solutions prematurely. And I guess that is my main point.
I’m more than willing to accept that text files may indeed be the best solution.
While I might agree, I think the perceived advantage (rightly so!) of text files is that if they get screwed up, you can edit them and fix the corrupted values. On the other hand, the downside of text files is they are probably less tolerant to errors.
As probably mentioned elsewhere and eluded to above by @mike_mccaffery I implemented the Json storage after loosing my complete system configuration, so I’m completely sympathetic to the concern and I think it addresses some/many/most of the thoughts here…
It automatically keeps backups of each file.
If lots of writes are made to the database over a short time, the system will only write to disk periodically to improve performance, and reduce the number of writes that may impact flash drives used on the likes of RPi.
It puts each type of information (items, things, links) into a separate file, and stores 5 previous versions in a backup subfolder in case it’s required.
If it can’t open a file, it will automatically try and open the newest backup file
I completely agree. I actually do believe that text files is probably the right answer and I look forward to trying out your JSON data store. If it works out I’d like to see it become the standard. My biggest concern though is fragmentation, but I’m sure everyone is concerned about that.
I think JSON strikes the right balance between human readable and easily machine parsable.
Have you given any thought to adding this as not only a data store for OH config but as a persistence option? I’m not sure there would be much demand for it but I must admit I’m thinking it might be a nice way to bootstrap an Item’s initial state (e.g. manually set the value in the text file and touch the .items file to restoreOnStartup to this value) or overriding values that have been saved in the past that might be wrong or causing problems (e.g. I have an Item that got populated once but the device is currently dead and I’ve been too lazy to create a System started rule to reset the value until I can get around to rebuilding the device).
No - it’s not something I’d considered. In a previous life (ie before OH) I wrote a persistence system for the Vera called DataMine - that did you text files and the performance wasn’t as bad as I initially expected. However, as a general persistence store, I think a database is probably more suitable…
For a simple ‘restore on startup’ persistence service though, it could be ok. However, I would probably argue that the configuration database is a special case - the consequences of a failure a pretty major - even if it only happens once a year. On the other hand, a once a year failure of the startup persistence might not even be noticed…
I was mainly thinking of using it in place of MapDB for restoreOnStartup, not something that would be used for historic data. And it really only just scratches a couple of edge cases I have which I’ve already found work arounds for.
I completely agree, the configuration DB is a special case.