Design Pattern: Group Based Persistence

See Design Pattern: What is a Design Pattern and How Do I Use Them for an explanation of what a DP is and how to use them.

Problem Statement

There are several reasons why a home automation enthusiast would want to use openHAB persistence: charting, using historical Item states in rules logic, restore previous values on openHAB restart, detailed analysis, access to the data with external tools, etc. However, different databases are more or less well suited to each of these use cases.

Concept

Since no one persistence engine is best for all use cases, configure more than one and use Groups to allocate which Items get saved to which persistence engine and how.

For example, one can set up MapDB for restoreOnStartup, rrd4j for charting recent data and historical data, and MySQL for Items for detailed external analysis.

For each use case, create a Group and only use these groups in the .persist files.

Then allocate your Items to whatever group(s) based on how that specific Item will be used.

Example

In my current setup I restoreOnStartup on all my Items, I use rrd4j for charting and historical state, I have a couple of Items I expose to IFTTT through my.openhab, and no Items that I analyze or expose the DB outside of OH.

Items:

// Persistence Groups
Group gMyOpenhab // for Items that are exposed to IFTTT
Group gChart     // for Items to persist for charting
Group gHistory   // for Items to preserve their history
//Group gRestore // for Items to restore on startup (currently everything so commented out)
...
// Example Item that I Chart
Number          Weather_Temp_Min        "Minimum Outside Temp [%.0f] °F"        <temperature> (gChart, gWeather_Temp_Chart)

// Example Item that I use historical data
Contact       N_D_Front                        "Front Door [MAP(en.map):%s]"             <frontdoor>  (gHistory, gDoorSensors, gRemindDoorSensors, gAlarmSensors, gHydraSensors)    { mqtt="<[mosquitto:entry_sensors/main/front_door:state:default]" }

// Example Item that I expose to my.openhab NOTE: it is also a member of gHistory
Switch  T_D_Garage1     "Garage Door 1"         <garagedoor> (gHistory, gMyOpenhab, gGarageOpeners)

mapdb.persist file: I only use MapDB for restoreOnStartup and restoreOnStartup applies to all Items

Strategies {
        default = everyUpdate
}

Items {
        // persist all items on every change and restore them from the db at startup
        * : strategy = everyChange, restoreOnStartup
}

NOTE: I don’t know if setting a default strategy is required, I included it anyway, but it is basically meaningless because it is never used in the Items section.

To make restoreOnStartup work we need to persist every change. The ‘*’ resolves to all Items.

rrd4j.persist file: I use rrd4j for historic data. Because of limitations of rrd4j data must be saved at least every minute for charting or else you end up with blank charts and previousState and historicState do not appear to work.

Strategies {
        // for rrd charts, we need a cron strategy
        everyMinute : "0 * * * * ?"

        default = everyChange
}

Items {
        // additionally persist weather info every minute
        gHistory* : strategy = everyUpdate, everyMinute
}

**influxdb.persist file: I use InfluxDB along with Grafana for generating complex and attractive charts. See the InfluxDB+Grafana tutorial for details.

// persistence strategies have a name and a definition and are referred to in the "Items" section
Strategies {
        default = everyChange
}

Items {
        gChart* : strategy = everyUpdate
}

myopenhab.persist file

Strategies {
        default = everyUpdate
}

Items {
        gMyOpenhab* : strategy = everyUpdate
}
Advantages

Advantages

The advantages of this approach include:

  • The persistence behavior for each Item is documented with the Item (i.e. the group membership) keeping as much information about the behavior of the Item in one place
  • Adding new Items or changing how they are persisted simply involves changing that Item’s group membership.
  • Once set up, the .persist files need never be changed unless you decide move from one persistence engine to another
  • Uses the best persist engine for the desired behavior. For example MapDB can save all types of Items but only keeps the last value do it is great for restoreOnStartup but not great for historic data whereas rrd4j is great for charting recent data but can’t save anything that isn’t a numerical value so isn’t so great for restoreOnStartup

Disadvantages:

  • If you have a Group with an aggregation function that you also want to persist, this DP cannot be used on that Group Item. The Group Item will need to be listed individually in the .persist files.

A note about usage:

openHAB can have only one default persistence engine. Any calls you make in Rules where you do not specify a persistence engine will use the default. When using this approach for Persistence be careful when and if you change the default persistence engine as that may break some of your Rules.

I avoid problems, I recommend always specifying the persistence engine you want to use in all of your persistence calls. For example:

MyItem.previousState("rrd4j")
MyItem.lastUpdate("mapdb")

In Scripted Automation Rules, you can define configuration constants in the configuration file (e.g. in Python that’s automation/lib/python/configuration.py) and import that variable into your Rules. For example

prev_db = "rrd4j"
lastup_db = "mapdb"
from configuration import prev_db, lastup_db
from core.actions import PersistenceExtensions
    
...
    PersistenceExtensions.previousState(ir.getItem("MyItem"), prev_db)
    PersistenceExtensions.lastUpdate(ir.getItem("MyItem"), lastup_db)

This way should you change your mind on which DB the values should be pulled from you only need update your configuration file.

44 Likes

HI @rlkoshak

I found you only use every change on mapsdb else all in every update,
may I know it was due to database requirement or just personal favorite?

P.S. I found in my RPI3B, influxDB miss store few of items from time to time until I change to every minute, I haven’t try every update yet.

I use everyChange because I only use it for restoreOnStartup. Given that I only need to save a new value when an Item’s state changes. If I receive an update that doesn’t change the Item’s state, I don’t need to overwrite the value already stored in the DB.

However, if I were using MapDB to also figure out when the last time the Item received an update, whether or not the update caused the Item to change, then I’d use everyUpdate instead.

1 Like

Thank you @rlkoshak for this post.

Here is my experience:

  • I was looking at 14 different Persistence add-ons that are available to me, with no guidance on what to chose or why
  • I then looked at the user manual under configuration -> persistence - which to this date is a stub http://docs.openhab.org/configuration/persistence.html
  • I then found your post by pure luck … and feel extremely grateful for this contribution.

I’m wonderig if @ThomDietrich (contributor on docs page) would want to add this great content directly into the docs … at least the rrd4j and MapDB items … these 2 side by side (and why) - this is perfect for 95% of new adopters … and for the rest it will be a perfect jumping board into more advanced persistence options (aka the other 11 or 12 …)

Greetings, Kai

2 Likes

There is an issue already opened for this and in all likelihood I’m supposed to be working on it. Thom will concur, I’m pretty good on the forum, not so good on github.

Of course, Thom and I disagree as to whether this DP is even a good idea (he would rather define everything in the .persist file and not mix concerns in the .items files.

Hey guys,
as always I’m trying to provide clarity about the possibilities of openHAB in the docs without mixing my personal preferences in as the one true solution. :angel:

@kai-bcn you are absolutely right, the stub looks kind of sad. I felt the same way about the Transformation stub and started working on it just last week (check it out, click the View button to see the result of the source code).

Next could be the Persistence article but quite frankly I don’t know when I’ll find the time to get to that. It would actually be great if somebody like you @kai-bcn could go through the old wiki article and do some spell checking and grammar improvements (Either in the wiki on GitHub or if that is not your turf, you could also do that here in a private message). Next we could simply move it over to documentation and replace nothing with something. https://github.com/openhab/openhab1-addons/wiki/Persistence

Best! Thomas <-- A mystery solved

2 Likes

I just started using this approach to persistence (I find it much easier to maintain as my number of items starts increasing) but I have noticed a small issue. I have a group item Group:Switch:AND(OFF,ON) Presence_Group <present> (Persistence_EveryChange) that contains presence items for me and my wife, which I want persisted (the state of the group as well as the state of the individual items).

However, it only works if I explicitly write the group item name in the .persist file and not as part of the persistence group. I guess this behavior is intentional, to avoid persisting a lot of groups, which most of the time isn’t very useful, but thought I should mention it here in case someone else have the same issue.

2 Likes

@rlkoshak, thanks for yet another useful Design Pattern. As a non-engineer who is looking to learn as well as improve my openHAB setup, I appreciate your mix of examples and explanation. I was recently setting up a timeline in HABpanel. When it didn’t display properly, I figured out I had my rrd4j configured incorrectly, and your topic helped me get it straight.

2 Likes

This is an extremely helpful design pattern, shortened my learning curve dramatically, kudos Rich!

4 Likes

quick question. if i have a group of lights, can i just add the group to the gHistory, or do i also need to include each item individually in the group gHistory?

That is a good question. I’m at an airport right now so can’t test it out, but I suspect Persistence does not recurse though subgroups so you would have to add the Items to gHistory individually. But it is worth a test.

I use the nested group approach for persistence in my setup and it works for me.

3 Likes

Thanks @rlkoshak and @cweitkamp I will give it a go. I’m hoping it does work, else my items file is going to explode with additional group memberships i’ll have to add.

Hi Rich (@rlkoshak)

First of all a big thanks for the helpful DP.

I use mapDB for restoring the state of some items and rrd4j for charting. Now I am struggle with the default persistence engine for OH. If I use mapDB as default engine generating of the charts will fail at Basic UI. If I set rrd4j as default engine restoring and charting seems to work.

Is there a possibility to define the persistence engine within the sidemap?

Best regards
Raphael

service sets the persistence service to use. If no service is specified, openHAB will use the first queryable persistence service it finds. Therefore, for an installation with only a single persistence service, this is not required.

Hi Rich (@rlkoshak)

Thanks for the feedback. I overlooked this part of the documentation. Sorry!

Best regards

Raphael

No problem. There are lots and lots of details. It’s easy to miss.

Dear @rlkoshak ,

Many thanks for this great tutorial.
I hope you can help me with the following issue:

I did the setup as described in the first post of this thread. I want to use MapDB for restore on startup and RRD4J for charting. After switching from my approach - which was to include specific items into rrd4j.persist - to using groups, it doesn’t render the charts anymore.

My rrd4j.persist file:

Strategies {
    everyMinute:    "0 * * * * ?"
    every10Minute:  "10 * * * * ?"
    everyHour:      "0 0 * * * ?"
    everyDay:       "0 0 0 * * ?"
    default = everyChange
}

Items {
    gR* : strategy = everyUpdate, everyMinute
}
// .items File
    Group gM            // mapdb - persist all items on every change and restore them from the db at startup
    Group gR            // rrdj4 - for Items to preserve their history + charting

    String SonoffS201_wifirssi      "Sonoff S20 #1: Wifi RSSI [%s]"     (gM,gR,Gsonoff) { channel="mqtt:topic:mosquitto:sonoff_0xxx:Sonoff_0xxx_rssi" }

test.sitemap

   Frame label="Charts Sonoff S20" {
        Text item=Gcharts {
            Chart item=SonoffS201_wifirssi refresh=60000 period=h service="rrd4j" legend=true
            }
    }

I always get this message - for all items I’ve tried so far:

[WARN ] [thome.ui.internal.chart.ChartServlet] - Chart generation failed: null

With no charts:

http://openhabianpi:8080/rest/persistence

delivers:

[{"id":"mapdb","label":"mapdb","type":"Queryable"},{"id":"rrd4j","label":"rrd4j","type":"Queryable"}]

Default persistence is set to rrd4j:
image

Unfortunately I don’t use rrd4j any more and I don’t use OH generated charts any more so I don’t really have anything to offer. What you have posted looks like it should work and the error is not informative.

Ok, thanks for letting me know.

Should I open a new thread for this issue? I want to avoid cross posting …