Design Pattern: Group Based Persistence

Edit: Updated for OH 4

See Design Pattern: What is a Design Pattern and How Do I Use Them for an explanation of what a DP is and how to use them.

Problem Statement

There are several reasons why a home automation enthusiast would want to use openHAB persistence: charting, using historical Item states in rules logic, restore previous values on openHAB restart, detailed analysis, access to the data with external tools, etc.

By default, a persistence add-on will implement a default strategy which will usually be something like “all Items on every change with restoreOnStartup”. This may be just fine for most use cases. However, many times one may not want to save all Items to a given database, or they may want to apply filters that apply to some Items, or have some Items save to one database and other Items saved to a different one.

Concept

Use Groups to identify those Items that should be saved to a specific database under a specific strategy and set up the persistence configuration using the “member of Group” list of Items.

For example, one can set up MapDB for restoreOnStartup, rrd4j for charting recent data and historical data, and InfluxDB for a small set of Items for detailed external analysis.

For each strategy, create a Group and only use these groups in the .persist files.

Then allocate your Items to whatever Group(s) based on how that specific Item will be used.

An advantage is that one applies the strategy at the Item level. However, this means your persistence configuration gets spread out across your Items instead of centralized in your persistence configs.

Example

Let’s say we have a small set of Items we have a set of Items we want to do restoreOnStartup, another set we want to save on every change for basic charting, and a few we want to save to a different database on every change to detailed analysis. We’ll create three Group Items:

  • RestoreOnStartup
  • DefaultEveryChange
  • AnalysisEveryChange

Set the Items whose persistence fits into one or more of these strategies as members of the strategy to apply. Let’s say we’ll use MapDB for RestoreOnStartup, rrd4j for DefaultEveryChange, and InfluxDB for AnalysisEveryChange. Note that this does not mean that two or all three of these strategies are used in the same database.

Managed Configs

Navigate to Settings → Persistence and select the desired add-on to configure under “Configure Persistence Policies”.

MapDB Config:

  1. Click “add configuration”, click “Select groups” and select “RestoreOnStartup”
  2. Under “strategies” make sure “everyChange” and “restoreOnStartup” are selected

Note, if you want to apply filters, create the filter first and then select the filter(s) you want to apply for this Group under “Select filters”.

Repeat the steps above for RRD4J and InfluxDB, selecting the Group name and strategies and filters (if applicable) as appropriate.

File Based Config

mapdb.persist file:

Strategies {
        default = everyChange
}

Items {
        RestoreOnStartup* : strategy = everyChange, restoreOnStartup
}

NOTE: To make restoreOnStartup work we need to persist every change. The * after the Group name means “all members of this Group”.

rrd4j.persist file:

Strategies {
        everyMinute : "0 * * * * ?"
        default = everyChange
}

Items {
        // additionally persist weather info every minute
        DefaultEveryChange* : strategy = everyUpdate, everyMinute
}

NOTE: Because of the way rrd4j works, an everyMinute strategy is required.

**influxdb.persist file:

// persistence strategies have a name and a definition and are referred to in the "Items" section
Strategies {
        default = everyChange
}

Items {
        AnalysisEveryChange* : strategy = everyUpdate
}

Advantages

The advantages of this approach include:

  • The persistence behavior for each Item is documented with the Item (i.e. the Group membership) keeping as much information about the behavior of the Item in one place
  • Adding new Items or changing how they are persisted simply involves changing that Item’s Group membership.
  • Once set up, the .persist files/config need never be changed unless you decide move from one persistence engine to another
  • Uses the best persist engine for the desired behavior. For example MapDB can save all types of Items but only keeps the last value do it is great for restoreOnStartup but not great for historic data whereas rrd4j is great for charting recent data but can’t save anything that isn’t a numerical value.

Disadvantages:

  • If you have a Group with an aggregation function that you also want to persist, this DP cannot be used on that Group Item. The Group Item will need to be listed individually in the .persist files.

A note about usage:

openHAB can have only one default persistence engine. Any calls you make in Rules where you do not specify a persistence engine will use the default. When using this approach for Persistence be careful when and if you change the default persistence engine as that may break some of your Rules.

To avoid problems, I recommend always specifying the persistence engine you want to use in all of your persistence calls in rules. For example, in JS Scripting:

items.MyItem.history.previousState(true, 'rrd4j');
items.MyItem.history.lastUpdate('mapdb');
44 Likes

HI @rlkoshak

I found you only use every change on mapsdb else all in every update,
may I know it was due to database requirement or just personal favorite?

P.S. I found in my RPI3B, influxDB miss store few of items from time to time until I change to every minute, I haven’t try every update yet.

I use everyChange because I only use it for restoreOnStartup. Given that I only need to save a new value when an Item’s state changes. If I receive an update that doesn’t change the Item’s state, I don’t need to overwrite the value already stored in the DB.

However, if I were using MapDB to also figure out when the last time the Item received an update, whether or not the update caused the Item to change, then I’d use everyUpdate instead.

1 Like

Thank you @rlkoshak for this post.

Here is my experience:

  • I was looking at 14 different Persistence add-ons that are available to me, with no guidance on what to chose or why
  • I then looked at the user manual under configuration -> persistence - which to this date is a stub http://docs.openhab.org/configuration/persistence.html
  • I then found your post by pure luck … and feel extremely grateful for this contribution.

I’m wonderig if @ThomDietrich (contributor on docs page) would want to add this great content directly into the docs … at least the rrd4j and MapDB items … these 2 side by side (and why) - this is perfect for 95% of new adopters … and for the rest it will be a perfect jumping board into more advanced persistence options (aka the other 11 or 12 …)

Greetings, Kai

2 Likes

There is an issue already opened for this and in all likelihood I’m supposed to be working on it. Thom will concur, I’m pretty good on the forum, not so good on github.

Of course, Thom and I disagree as to whether this DP is even a good idea (he would rather define everything in the .persist file and not mix concerns in the .items files.

Hey guys,
as always I’m trying to provide clarity about the possibilities of openHAB in the docs without mixing my personal preferences in as the one true solution. :angel:

@kai-bcn you are absolutely right, the stub looks kind of sad. I felt the same way about the Transformation stub and started working on it just last week (check it out, click the View button to see the result of the source code).

Next could be the Persistence article but quite frankly I don’t know when I’ll find the time to get to that. It would actually be great if somebody like you @kai-bcn could go through the old wiki article and do some spell checking and grammar improvements (Either in the wiki on GitHub or if that is not your turf, you could also do that here in a private message). Next we could simply move it over to documentation and replace nothing with something. https://github.com/openhab/openhab1-addons/wiki/Persistence

Best! Thomas <-- A mystery solved

2 Likes

I just started using this approach to persistence (I find it much easier to maintain as my number of items starts increasing) but I have noticed a small issue. I have a group item Group:Switch:AND(OFF,ON) Presence_Group <present> (Persistence_EveryChange) that contains presence items for me and my wife, which I want persisted (the state of the group as well as the state of the individual items).

However, it only works if I explicitly write the group item name in the .persist file and not as part of the persistence group. I guess this behavior is intentional, to avoid persisting a lot of groups, which most of the time isn’t very useful, but thought I should mention it here in case someone else have the same issue.

2 Likes

@rlkoshak, thanks for yet another useful Design Pattern. As a non-engineer who is looking to learn as well as improve my openHAB setup, I appreciate your mix of examples and explanation. I was recently setting up a timeline in HABpanel. When it didn’t display properly, I figured out I had my rrd4j configured incorrectly, and your topic helped me get it straight.

2 Likes

This is an extremely helpful design pattern, shortened my learning curve dramatically, kudos Rich!

4 Likes

quick question. if i have a group of lights, can i just add the group to the gHistory, or do i also need to include each item individually in the group gHistory?

That is a good question. I’m at an airport right now so can’t test it out, but I suspect Persistence does not recurse though subgroups so you would have to add the Items to gHistory individually. But it is worth a test.

I use the nested group approach for persistence in my setup and it works for me.

3 Likes

Thanks @rlkoshak and @cweitkamp I will give it a go. I’m hoping it does work, else my items file is going to explode with additional group memberships i’ll have to add.

Hi Rich (@rlkoshak)

First of all a big thanks for the helpful DP.

I use mapDB for restoring the state of some items and rrd4j for charting. Now I am struggle with the default persistence engine for OH. If I use mapDB as default engine generating of the charts will fail at Basic UI. If I set rrd4j as default engine restoring and charting seems to work.

Is there a possibility to define the persistence engine within the sidemap?

Best regards
Raphael

https://www.openhab.org/docs/configuration/sitemaps.html#element-type-chart

service sets the persistence service to use. If no service is specified, openHAB will use the first queryable persistence service it finds. Therefore, for an installation with only a single persistence service, this is not required.

Hi Rich (@rlkoshak)

Thanks for the feedback. I overlooked this part of the documentation. Sorry!

Best regards

Raphael

No problem. There are lots and lots of details. It’s easy to miss.

Dear @rlkoshak ,

Many thanks for this great tutorial.
I hope you can help me with the following issue:

I did the setup as described in the first post of this thread. I want to use MapDB for restore on startup and RRD4J for charting. After switching from my approach - which was to include specific items into rrd4j.persist - to using groups, it doesn’t render the charts anymore.

My rrd4j.persist file:

Strategies {
    everyMinute:    "0 * * * * ?"
    every10Minute:  "10 * * * * ?"
    everyHour:      "0 0 * * * ?"
    everyDay:       "0 0 0 * * ?"
    default = everyChange
}

Items {
    gR* : strategy = everyUpdate, everyMinute
}
// .items File
    Group gM            // mapdb - persist all items on every change and restore them from the db at startup
    Group gR            // rrdj4 - for Items to preserve their history + charting

    String SonoffS201_wifirssi      "Sonoff S20 #1: Wifi RSSI [%s]"     (gM,gR,Gsonoff) { channel="mqtt:topic:mosquitto:sonoff_0xxx:Sonoff_0xxx_rssi" }

test.sitemap

   Frame label="Charts Sonoff S20" {
        Text item=Gcharts {
            Chart item=SonoffS201_wifirssi refresh=60000 period=h service="rrd4j" legend=true
            }
    }

I always get this message - for all items I’ve tried so far:

[WARN ] [thome.ui.internal.chart.ChartServlet] - Chart generation failed: null

With no charts:

http://openhabianpi:8080/rest/persistence

delivers:

[{"id":"mapdb","label":"mapdb","type":"Queryable"},{"id":"rrd4j","label":"rrd4j","type":"Queryable"}]

Default persistence is set to rrd4j:
image

Unfortunately I don’t use rrd4j any more and I don’t use OH generated charts any more so I don’t really have anything to offer. What you have posted looks like it should work and the error is not informative.

Ok, thanks for letting me know.

Should I open a new thread for this issue? I want to avoid cross posting …