Recommended way to backup/restore OH2 configurations and things?

Hi,
What is the recommended way to backup and restore an openhab2 installation?
My usecase is that I would like to save and restore everything for an upgrade.

The easy stuff is (om my ubuntu sever):

  • conf folder
  • runtime/karaf/etc/openHAB-wrapper.conf for saving custom java options
  • runtime/karaf/bin/openHAB-service mostly for the “RUN_AS_USER” value
  • userdata/persistence
  • userdata/myopenhab/secret and userdata/uuid

What about configurations created by the paper ui? For example the hue bindings stuff (bridge and lamps).
Right now I need to authorize my bridge again, and add the lamps/link the channels manually.

BR,
Hans

2 Likes

OpenHAB2 looks more complex than openHAB1 on that particular point, especially due to the new used mapdb and the fact things have to be saved from different locations (conf, userdata, …).
Currently, when I upgrade, I only save/restore my conf folder. But I have to redo discovery and link channels to items. That is OK because I am only using 2 bindings 2.0.

Maybe a wiki page could help everybody.

Maybe a tool to export/import the mapdb to a readable file could help too.

i agree a tool like this would be ideal…or perhaps an option to specify the path to the previous installation’s directory and OH2 will look at it and import the data into the new conf locations.
This will allow for rolling upgrades, and the abilty to restart the previous version quickly without it being changed.

@greg: OH2 will load all your old files (items, sitemaps, rules, …) without any changes. The only change you will have to do is to replace your old openhab.cfg file by several files, one per binding.
The new database is in fact only for new OH2 stuff (things, channels, …).

The overall idea of the distro structure is described here.
The static user files should reside in conf, the dynamically generated data in userdata, the read-only “binary” in runtime and optionally, bundles can be put in the addons folder.
Packaging systems such as apt, synology, snappy, etc. should simply only provision the runtime folder and also only update this while keeping the other folders in place.

So far the theory towards we strive, but currently there are a few issues:

  • In openHAB 1, we started the OSGi framework always with the -clean option, which discarded any information and hence made the static configuration files the only source for settings. In the new distro, we do not clean the OSGi framework upon startup, so that we can also change configuration through the Paper UI, the OSGi console or any other mechanism. The problem is that it is not obvious for the user what is installed and configured anymore (removing a bundle from addons does not uninstall it). The information is well-hidden somewhere in the OSGi cache in userdata.
  • Replacing the runtime folder does not automatically upgrade the installation if the userdata folder is kept. We should find a good mechanism to update for online/offline distro likewise.
  • The new ESH concepts store stuff (e.g. discovered things) in a database (in userdata/mapdb), which is not human-readable - this isn’t nice for backups as it is either all or nothing and cannot be manually modified. An export to a human-readable format would be welcome.
  • Since items/things/sitemaps etc. in the conf folder are considered to be read-only for the runtime, the UIs cannot be used to modify them - only information in the mapdb is modifiable. This leads to either/or configuration options, which make a smooth transition to the new concepts difficult. There is a discussion about possible solutions in the ESH forum.

In general, I see two sides in our user base that need to be served:

  • The openHAB 1 users, who want to continue using what they have and know as smoothly as possible. Most stuff should be configured textually in the conf folder and wiping the userdata folder should not really harm them.
  • New non-tech users, who want ease of setup and use, so what the new concepts of openHAB 2 promises. Possibly there is no textual configuration at all and all is done through UIs and configuration is stored in the mapdb and the OSGi framework.

Our distro should be able to support both and also a middle path (which will probably be the majority of users when the discovery feature is used more and more by the first group).

I would be interested in your ideas how we could tackle the above problems. Do you always want to have full control through textual files? Or would you also trust a setup where all your settings are soaked into a binary database file? What would you expect the UI to do if you try to modify a file that you have defined yourself in the conf folder?

I think we still need to clarify these point before it makes sense to write excessive documentation about it - so I count on your constructive ideas!

6 Likes

Very interesting message @Kai

In my opinion, what is defined in text files should have priority. That is, if a thing is setup in a textual file, it should be taken as it is defined and Paper UI should not discover a new copy of this thing. Currently, if I am right, you will have two instances for the same thing in this case ?

And we just need a backup tool exporting the mapdb into thing files and cfg files in a backup folder. Combining that with my previous point, to migrate to a new OH version, the user will first export his mapdb to textual and readable files, upgrate and then put exported textual files into his conf folder.

Exporting will even help an advanced user to understand how to work with textual files rather than UI.

For a full backup/export, I probably missed few things like how to backup inbox and links between channels and items. Maybe specific textual files should be built for that during backup lke an item file containing all items linked to channels.

And I think Paper UI should not update conf folder.

Thank you for very informative and interesting answers.
I don’t have any personal preferences of how to store the settings. As long is it possible to save/restore them.
An advantage with text files is that you can easily share your configurations with other users. I also think that the textfiles should have priority (I don’t have any special reason for that, but it feels more explicit created configuration). If the ui tries to modify a file, maybe it should notify the user “File changed on file system, do you want to reload the configuration?”.

@Kai: what do you think about our suggestions ?

I am not sure, if I understand them correctly or if they are missing some important points.

what is defined in text files should have priority.

So if you have a classicui.cfg defining your default sitemap name, what should happen in the Paper UI, if the user edits this setting? Shall it not be possible at all? Shall it be possible, but overwritten with the next restart again (because the classicui.cfg is re-applied then)?

Yes, as I said that are many other things in userdata besides the mapdb. It is all the configuration of the OSGi instance with bundles/features being installed/started, service configurations, etc. I doubt that it will be easy to export this all to some text format and specifically there is not an equivalent in the conf folder where things can be put.

So in general, the backup should always include conf+userdata folder as a whole. What we imho should make sure is that this works this way, i.e. that you can replace the runtime folder and all continues to work.

@Kai
to your question:
" Do you always want to have full control through textual files? "

A clear yes, definitely!

I see no way to get rid of text files.
This is strength of Openhab and the openhab Designer is a strong tool supporting this nicely.
All the approaches using a web Ui I have seen work miserably.
(e.g. Homematic, the poor user experience by this web ui was one of the main drivers that brought me to openhab, another example is QiViCon where the integration of “Apps” is working badly).

Too my opinion it will take ages until the industry will be able to have a click experience getting rid of the user’s control logic (rules files). Maybe deep learning and a full exposure of everything to the cloud (not desired by myself) will bring this ability in some distant future.

And since I see no immediate way to get rid of rule text files, a mixture of web ui and text files is not desirable because it is great to have one tool (designer) to do it all.
Also renaming and refactorying accordingly is very inconvienent when having to use a web ui.

Additionally I regard it as very inconvienent having to click all through this configuration UI in the paper UI, everytime I have to install a new build.

coming back to the original question what to backup in case of a installation of new build.
(windows).
This is what I would be thinking. Do not know whether this is complete.

  1. the conf folder
  2. the secret under userdata/myopenhab and uidd file under userdata
  3. the userdata/logs/. Unfortunately the logging persistance store user logs here and not in the persistence folder (?) or does it.
  4. the userdata/persistence folder
  5. the addons folder in case the user has installed addons here.
  6. anything else ??

Would it be possible to backup the userdata folder as a whole and overwrite the userdata folder that comes with the distribution?

@Lolodomo
"That is, if a thing is setup in a textual file, it should be taken as it
is defined and Paper UI should not discover a new copy of this thing.
Currently, if I am right, you will have two instances for the same thing
in this case ?"

If this is true then I would consider it as a bug!

In my opinion, what is defined in text files should have priority. That is, if a thing is setup in a textual file, it should be taken as it is defined and Paper UI should not discover a new copy of this thing. Currently, if I am right, you will have two instances for the same thing in this case ?

Whats the future (and current) wished behaviour here would interest me?

Hi there, my first post :slightly_smiling:

As @martin_klimke said, Ui’s often mean more work then simple editing text files.
Also I dint’t like a mix of settings in DB and text files.
Best for me: All config in text files (as in OH1) and let the Ui edit those files. So you can make changes within the Ui and in the text files. Not possible with a little regex magic?

4 Likes

@Kai can you enlight us a little … See #13

I would also agree that this is a good approach. In HABmin, for the ‘HABmin specific configuration’ (like chart configuration, floorplans etc) I write everything out using XML. This provides a fully flexible format that’s srtuctured and easily extensible, but still editable in a text file. It would however be slower than mapdb, but as always, there’s a compromise and it’s probably not an issue…

I would guess that it’s probably possible to replace the mapdb implementation with something else (I believe the architecture supports this although I’ve not looked at it myself). If so, something structured like XML might work quite well.

(just my 2c worth :wink:)

… For things added from habmin2 is it possible to have a switch… “Add to XML” … “Add to mapdb” as a user choice? :slight_smile:

No - that won’t work.

I think it will be possible though to replace mapdb with an XML version. I would guess it’s probably not too hard, but I might be completely wrong as I’ve not looked at it!

(Ok, I just looked at the code, and I’m pretty sure this should be possible - it’s a service, so changing to a different system shouldn’t be an issue, and making this read/write XML should be (relatively!) ‘simple’ :wink: )

1 Like

Would it be possible to backup the userdata folder as a whole and overwrite the userdata folder that comes with the distribution?

Well, in general a full backup comprises the conf and the userdata folder, that’s the idea. Note that the distribution does not come with any userdata folder at all, so this does not get overwritten either.

That is, if a thing is setup in a textual file, it should be taken as it is defined and Paper UI should not discover a new copy of this thing.

But should it also not be possible for the Paper UI to edit the Thing? Or have a binding add dynamically properties to it, which it read from the device (e.g. the Tesla binding adds the info that you have a sunroof)?

Currently, if I am right, you will have two instances for the same thing in this case ?

No, it a thing with the same UID already exists (being defined in a things-file), you will not see it in the inbox.

let the Ui edit those files. So you can make changes within the Ui and in the text files.

There are a few reasons against this:

  • the conf folder is meant to be read-only and only the system admin should have write-access to it. This is something I would want to keep, because if you open it up for the runtime to access it, the user has no idea of what stuff is changed in there without his knowledge. Well, we could say that the runtime copies this stuff to userdata and edits it there, so it would be a kind of import mechanism for static files.
  • The current files are parsed through Xtext. While I agree that the Designer is meant to be a nice tool, I also see that most people are actually using vi or emacs instead. And having Xtext as a part of the runtime has the problem that it is quite heavy purely for handling configuration files and that there do not seem to be people in the openHAB community, who can deal with it, so that simple bugs stay open forever. For these reasons, I would only want to keep the existing files and their infrastructure for OH1 compatibility, but not really build on top of it to also allow editing them through Xtext (what Xtext in general supports).
  • Not using Xtext for writing files would probably mean that a lot of formatting, comments, etc. get lost within the files.
  • The Xtext files are not meant to be a “db replacement”. Doing edit operations through the REST API can cause quite some heavy db activity as it is not done as bulk, but in many small transactions. Even the MapDB implementation seems to be sometimes too slow to handle it properly on a RasPi.
  • By far not all configuration is handled through openHAB/Xtext itself. If you install a new Karaf feature on the console, there is no way to have that serialized into the “addons.cfg”. Likewise all changes about logging configuration or other services. You can do a hell of a lot in Karaf and all such settings are stored somewhere in the OSGi internals according to good OSGi manners…

For these reasons, I think the best approach is to go with the possibility of having the “old” files as a kind of import and treating the database as THE point of reference.

I think it will be possible though to replace mapdb with an XML version.

Yes, ESH allows plugging in different storage implementations. We are actually serializing all stuff to JSON, so this would be my preferred format.
But you need to be aware that you will end up implementing a complete database - with consistency checks, recovery, internal caches, indexes, query optimizations, etc. I am not too sure that this is the way to go.
An alternative might be to accept the binary (and thus efficient) format of MapDB and rather provide an “export to text” mechanism. This could be e.g. triggered automatically once in a while or upon a “backup” request.

FTR, you might also want to read this discussion here: Eclipse Community Forums: Eclipse SmartHome » Dynamically created Channels during initialize()

3 Likes

Wouldn’t it be possible to have a simple “Show config as plain text” option in the PaperUI, so the configuration can be copied and pasted to the files in the conf dir? This way you can tinker in the UI and persist the configuration when you think it’s finished.