Replacement of mapdb by jsondb

Setup, Configuration and Use News & Important Changes
1

All,

I feel sorry to tell you that the distro build #624 brings another breaking change, but as in the past, I am certain that it is for the better!

There has been a long discussion around the problem that the Things that are created through the Paper UI are stored in a binary database (the mapdb, data to be found under userdata/mapdb), which is not readable or manually editable (e.g. for fixing or cleaning).
@chris came up with a nice new implementation of a storage service (luckily, ESH is designed so modular that you can easily switch implementations), which stores all data as plain JSON in the file system (to be found under userdata/jsondb). It keeps backups and a write delay can be configured (might e.g. be useful to limit writes on your SD cards). Starting with distro #624, you will find those options configurable as a system service:

Screen Shot 2016-11-29 at 20.36.39.png2332Ă—742 41.3 KB

Many people have asked for this, so I hope you like it! The downside is that we do not have a migration process from mapdb to jsondb, meaning that you will have to manually re-create the contents of your database (hopefully for a last time!).

Regards,
Kai

15 Likes
2

Hi,

does this mean that the mabdb is not used anymore and I have to migrate, or is it still the default implementation and I can switch to json optionally?

3

No options, json now for everyone. We didn’t see any reason to keep mapdb as an option, since thejsondb behaves much better regarding performance as well - see @chris’ test results.

1 Like
4

If one has everything configured through files in the config folder I assume there is no impact, is that a correct assumption ?

5

Absolutely correct!

6

I think is it for better after loosing all data while testing the snapshots in the binary mapdb without the option to fix or cleaning it in the past (and i know that oH2 is still not for production :wink:)

Even if it’s not the last time (cause oH2 is not for production) - it’s now the first time with the option of advanced oH2-users to fix or clean the database :smile:

7

Absolute Beauty !!!
Great change

Within minutes, build # 624 was up and running and my Aeotec Z-Stick Gen5 was added easily.

root@host:/var/lib/openhab2/jsondb# ls -al
drwxr-xr-x 2 openhab openhab 4096 Nov 29 23:11 backup
-rw-r--r-- 1 openhab openhab 2262 Nov 29 23:11 org.eclipse.smarthome.config.discovery.DiscoveryResult.json
-rw-r--r-- 1 openhab openhab 3690 Nov 29 23:11 org.eclipse.smarthome.core.thing.Thing.json

root@host:/var/lib/openhab2/jsondb# more org.eclipse.smarthome.core.thing.Thing.json 
{
  "zwave:serial_zstick:ZW090C": {
    "class": "org.eclipse.smarthome.core.thing.internal.BridgeImpl",
    "value": {
      "label": "Z-Wave Serial Controller",
      "channels": [
        {
          "acceptedItemType": "Number",
          "kind": "STATE",
          "uid": {
            "segments": [
              "zwave",
              "serial_zstick",
              "ZW090C",
              "serial_sof"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "zwave",
              "serial_sof"

[...]

I love this change ! :grinning:

8

Just to keep to be right with that. What exactly ist stored in the jsonDB (by the way i like that idea). The option the edit the data is in my point of view a great step toward to get rid off the different items etc. files.

Things?
Items?
Rules?

Thanks Thomas

9

Exactly the same thing that was stored in the MapDB - all the configuration that you do through the user interface. This currently includes things, items and links from memory… The new rule engine will I assume also go in here.

10

@chris

I was not sure about the things :wink: stored in the mapdb. So for me this is the moment to move from .somethng files to the jsonDB and to fully use the paperui for configuration

Thanks
Thomas

11

Upgrading to latest snapshot and changing from mapdb to jsondb worked without any problems.

For Zwave: note that the id of the stick changes while adding it as a new thing (at least for my zwave.me stick).
So if you have any textual items configuration one need to change that to match the new id.

Great improvement, thanks :+1:

12

Any new thing that is created through PaperUI or HABmin will by default be generated with a new UID, but you can of course name your thing whatever you like (at least in HABmin it allows you to specify the UID when you create the controller).

13

Never came up with the idea to try that :innocent:
But adapting the items files was not a problem … :sunglasses:

14

Even in PaperUI, you can override the newly created UID.

15

Hi all,
upgrading to build 624, I have issue described here:

I don’t know if could be related to mandb or not, but I cannot upgrade.

Any help would be appreciated.

Marco

16

I think this change also causes integer configuration parameters to be persisted as floats. E.g. with the LIFX binding I see the following:

18:52:28.940 [WARN ] [inding.lifx.handler.LifxLightHandler] - Invalid value '300.0' for transition time, using default instead.

The definition of this parameter is:

            <parameter name="fadetime" type="integer" required="false">
                <label>Fade time</label>
                <description>The time to fade to the new color value (in ms).</description>
                <default>300</default>
            </parameter>

The code is expecting a long and not a float:

        try {
            return Long.parseLong(fadeCfg.toString());
        } catch (NumberFormatException e) {
            logger.warn("Invalid value '{}' for transition time, using default instead.", fadeCfg.toString());
            return FADE_TIME_DEFAULT;
        }

I’ve tested this with a completely new installation of build #624.

17

While I know this change is excellent and I fully support it, the migration for some of us is going to be a bit more painful than others. I have a rather large install and tried to do as much as possible in mapdb when using 2.0 bindings.

I am using the REST API to extract as much data as I can before burning my mapdb to the ground and populating the jsondb. The caveat is this only works if you are still running pre build 624.

These commands may be handy for others as well. Substitute your hostname and port as appropriate.

export Host=your.host.here
export Port=your.port.here

curl -X GET --header "Accept: application/json" "$Host:$Port/rest/items?recursive=true" -k > OHitems.json
curl -X GET --header "Accept: application/json" "$Host:$Port/rest/links" -k > OHlinks.json
curl -X GET --header "Accept: application/json" "$Host:$Port/rest/services" -k > OHservices.json
curl -X GET --header "Accept: application/json" "$Host:$Port/rest/things" -k > OHthings.json

Perhaps I will try posting a thing or two back through the API as a form of migration and see how it goes, but for now, I at least have the names saved off that match my rule set.

edit: If anyone is curious, it looks like I have 797 items currently :slight_smile:

1 Like
18

AFAIK mapdb has stored only the last value of each item and therefore was mainly used for “RestoreOnStartup”. Correct?

If so, has jsondb the same limitation? Or can I also use it to produce historical data (charting)?

19

No - no values are stored here. This storage is only used to store your configuration - it’s not a persistence service, and I think you are thinking about the mapdb persistence service.

1 Like
20

Oops…yes, I did. Ok, forget my question then. :wink: