Design Pattern: Group Based Persistence

Edit: Updated for OH 4

See Design Pattern: What is a Design Pattern and How Do I Use Them for an explanation of what a DP is and how to use them.

Problem Statement

There are several reasons why a home automation enthusiast would want to use openHAB persistence: charting, using historical Item states in rules logic, restore previous values on openHAB restart, detailed analysis, access to the data with external tools, etc.

By default, a persistence add-on will implement a default strategy which will usually be something like ā€œall Items on every change with restoreOnStartupā€. This may be just fine for most use cases. However, many times one may not want to save all Items to a given database, or they may want to apply filters that apply to some Items, or have some Items save to one database and other Items saved to a different one.

Concept

Use Groups to identify those Items that should be saved to a specific database under a specific strategy and set up the persistence configuration using the ā€œmember of Groupā€ list of Items.

For example, one can set up MapDB for restoreOnStartup, rrd4j for charting recent data and historical data, and InfluxDB for a small set of Items for detailed external analysis.

For each strategy, create a Group and only use these groups in the .persist files.

Then allocate your Items to whatever Group(s) based on how that specific Item will be used.

An advantage is that one applies the strategy at the Item level. However, this means your persistence configuration gets spread out across your Items instead of centralized in your persistence configs.

Example

Let’s say we have a small set of Items we have a set of Items we want to do restoreOnStartup, another set we want to save on every change for basic charting, and a few we want to save to a different database on every change to detailed analysis. We’ll create three Group Items:

  • RestoreOnStartup
  • DefaultEveryChange
  • AnalysisEveryChange

Set the Items whose persistence fits into one or more of these strategies as members of the strategy to apply. Let’s say we’ll use MapDB for RestoreOnStartup, rrd4j for DefaultEveryChange, and InfluxDB for AnalysisEveryChange. Note that this does not mean that two or all three of these strategies are used in the same database.

Managed Configs

Navigate to Settings → Persistence and select the desired add-on to configure under ā€œConfigure Persistence Policiesā€.

MapDB Config:

  1. Click ā€œadd configurationā€, click ā€œSelect groupsā€ and select ā€œRestoreOnStartupā€
  2. Under ā€œstrategiesā€ make sure ā€œeveryChangeā€ and ā€œrestoreOnStartupā€ are selected

Note, if you want to apply filters, create the filter first and then select the filter(s) you want to apply for this Group under ā€œSelect filtersā€.

Repeat the steps above for RRD4J and InfluxDB, selecting the Group name and strategies and filters (if applicable) as appropriate.

File Based Config

mapdb.persist file:

Strategies {
        default = everyChange
}

Items {
        RestoreOnStartup* : strategy = everyChange, restoreOnStartup
}

NOTE: To make restoreOnStartup work we need to persist every change. The * after the Group name means ā€œall members of this Groupā€.

rrd4j.persist file:

Strategies {
        everyMinute : "0 * * * * ?"
        default = everyChange
}

Items {
        // additionally persist weather info every minute
        DefaultEveryChange* : strategy = everyUpdate, everyMinute
}

NOTE: Because of the way rrd4j works, an everyMinute strategy is required.

**influxdb.persist file:

// persistence strategies have a name and a definition and are referred to in the "Items" section
Strategies {
        default = everyChange
}

Items {
        AnalysisEveryChange* : strategy = everyUpdate
}

Advantages

The advantages of this approach include:

  • The persistence behavior for each Item is documented with the Item (i.e. the Group membership) keeping as much information about the behavior of the Item in one place
  • Adding new Items or changing how they are persisted simply involves changing that Item’s Group membership.
  • Once set up, the .persist files/config need never be changed unless you decide move from one persistence engine to another
  • Uses the best persist engine for the desired behavior. For example MapDB can save all types of Items but only keeps the last value do it is great for restoreOnStartup but not great for historic data whereas rrd4j is great for charting recent data but can’t save anything that isn’t a numerical value.

Disadvantages:

  • If you have a Group with an aggregation function that you also want to persist, this DP cannot be used on that Group Item. The Group Item will need to be listed individually in the .persist files.

A note about usage:

openHAB can have only one default persistence engine. Any calls you make in Rules where you do not specify a persistence engine will use the default. When using this approach for Persistence be careful when and if you change the default persistence engine as that may break some of your Rules.

To avoid problems, I recommend always specifying the persistence engine you want to use in all of your persistence calls in rules. For example, in JS Scripting:

items.MyItem.history.previousState(true, 'rrd4j');
items.MyItem.history.lastUpdate('mapdb');
44 Likes
Jdbc persistence not working Error in SQL query!; (conn=1789) Data too long for column 'value' at row 1 Query: INSERT INTO item0024
Odd behavior when upgrading Synology DSM running OH
Ignoring item for persistance (influxdb2)
MariaDB: double entries with EveryUpdate strategy
Error that I don't understand
Persistence: "don't persist" alternative to override default strategy?
Help with Persistence DB config
Help with Persistence DB config
Persistence recommendations
Setting OH2 for KNX, a Windows install
[JS UI rule] How to check if item state changed with "ItemStateUpdatedEvent"
Postgresql huge space used by openhab
openHAB 4.0 wishlist
RestoreOnStartup when updating items file
Persistence of nested groups
Keep the item settings during reboot
Openhab2 RPI System Temperature and DS18B20 OneWire Chart with persistence
Last Stage After reboot
System started rule not executing on build #1034
Create txt File
YAGSA - Yet Another Group Structure Approach
[SOLVED] InfluxDB openHAB stores no Data
Push Notifications with subject and variables?
Comprehensive Wunderground using HTTP Binding Example
Thank you all!
Unable to restore state for item
MQTT persistence worked in 2.0.0 snapshot but not 2.1.0 release
One things / items file or many?
Persitance of items in group
Keep 'when Item received update' rules from running during RestoreOnStartup
Automatic persistence: detect which types cannot be properly persisted and skip them
Disable Persistance (rrdj) for some Items?
CHart displays one item correctly, another one not
OH3.1/3.2 out of memory error
Influxdb persistence exclusion
Excluding a certain items from being persisted into database
Persisting groups (MariaDB)
Problem min and max since?
Rules - Heating
Netatmo temperature and chart
Best persistence for new install?
Sitemap and Persistence Not Working As Expected
Design Pattern: Working with Groups in Rules
Persistence & Graphs - why so complicated?
InfluxDB + Grafana vs. MySQL + rrd4j
Low-power Z-Wave switch with two outputs needed
OH 3 Tips and Tricks
Wake Up Light
Group Based Persistence in OH3
Number value help please
OH3: how to persist data to an external relational database
InfluxDB+Grafana persistence and graphing
Blank items in UI after restart/reload of OH1
Interested in hardware
Some newb questions I could not find answer for
Items not appearing in myopenhab.org/items despite being correctly listed in my.openhab.org/rest/items
Presence detection via iPhone's mDNS Entry. How to integrate into OH2?
Storing item states for later recall
OH2: restoreOnStartup not working properly
Change rwe switch from "ON" to "1"
Group/(Sub)Group/items in persistence and rules
restoreOnStartup randomly skips Items
How can I persist serveral items in another way then the "default"-behavoir?
Design Pattern: Manual Trigger Detection
[SOLVED] Multiple persistence in influxDB
2.5 Milestone 5 Issues
Smart Virtual Thermostat (beta version)
Item status and configuration change
Qubino Smart Meters (ZMNHXD, ZMNHTD) endless problems
Struggling with the concept of "Groups" - limitations or misunderstanding?
Next generation design: A Paper UI replacement proposal
Finally home to OH - now how to organize filechanges / deployments etc?
Influxdb persistence adds decimal to the end of number items
Zwave Scene Number - Command or Update?
Rules stop executing after a while
[SOLVED] Persistence not working at all (persistence bundle/service not started?)
Combining 3 hue lamps in PaperUI and Items/sitemap does not work for me
OpenHab adding values into a table to be visualized by Grafana
Paper UI - How to add item to persistence?
Does HABpanel make persistence create an item table?
InfluxDB+Grafana persistence and graphing
Last Stage After reboot
No working trigger functions openHAB 2.1
How to store / query Influxdb from OH?
Problems with Items/Things in Habmin/PaperUI
Graph a group item in grafana?
What are your top 3 automations
Design Pattern Persistance Language Problem
Does restoreOnStartup require an everyChange/everyUpdate strategy
Developing a persistence addon (not a binding)
Remote access to image via app / cloud using url
My openHab Setup

HI @rlkoshak

I found you only use every change on mapsdb else all in every update,
may I know it was due to database requirement or just personal favorite?

P.S. I found in my RPI3B, influxDB miss store few of items from time to time until I change to every minute, I haven’t try every update yet.

I use everyChange because I only use it for restoreOnStartup. Given that I only need to save a new value when an Item’s state changes. If I receive an update that doesn’t change the Item’s state, I don’t need to overwrite the value already stored in the DB.

However, if I were using MapDB to also figure out when the last time the Item received an update, whether or not the update caused the Item to change, then I’d use everyUpdate instead.

1 Like

Thank you @rlkoshak for this post.

Here is my experience:

  • I was looking at 14 different Persistence add-ons that are available to me, with no guidance on what to chose or why
  • I then looked at the user manual under configuration → persistence - which to this date is a stub http://docs.openhab.org/configuration/persistence.html
  • I then found your post by pure luck … and feel extremely grateful for this contribution.

I’m wonderig if @ThomDietrich (contributor on docs page) would want to add this great content directly into the docs … at least the rrd4j and MapDB items … these 2 side by side (and why) - this is perfect for 95% of new adopters … and for the rest it will be a perfect jumping board into more advanced persistence options (aka the other 11 or 12 …)

Greetings, Kai

2 Likes

There is an issue already opened for this and in all likelihood I’m supposed to be working on it. Thom will concur, I’m pretty good on the forum, not so good on github.

Of course, Thom and I disagree as to whether this DP is even a good idea (he would rather define everything in the .persist file and not mix concerns in the .items files.

Hey guys,
as always I’m trying to provide clarity about the possibilities of openHAB in the docs without mixing my personal preferences in as the one true solution. :angel:

@kai-bcn you are absolutely right, the stub looks kind of sad. I felt the same way about the Transformation stub and started working on it just last week (check it out, click the View button to see the result of the source code).

Next could be the Persistence article but quite frankly I don’t know when I’ll find the time to get to that. It would actually be great if somebody like you @kai-bcn could go through the old wiki article and do some spell checking and grammar improvements (Either in the wiki on GitHub or if that is not your turf, you could also do that here in a private message). Next we could simply move it over to documentation and replace nothing with something. Persistence Ā· openhab/openhab1-addons Wiki Ā· GitHub

Best! Thomas ← A mystery solved

2 Likes

I just started using this approach to persistence (I find it much easier to maintain as my number of items starts increasing) but I have noticed a small issue. I have a group item Group:Switch:AND(OFF,ON) Presence_Group <present> (Persistence_EveryChange) that contains presence items for me and my wife, which I want persisted (the state of the group as well as the state of the individual items).

However, it only works if I explicitly write the group item name in the .persist file and not as part of the persistence group. I guess this behavior is intentional, to avoid persisting a lot of groups, which most of the time isn’t very useful, but thought I should mention it here in case someone else have the same issue.

2 Likes

@rlkoshak, thanks for yet another useful Design Pattern. As a non-engineer who is looking to learn as well as improve my openHAB setup, I appreciate your mix of examples and explanation. I was recently setting up a timeline in HABpanel. When it didn’t display properly, I figured out I had my rrd4j configured incorrectly, and your topic helped me get it straight.

2 Likes

This is an extremely helpful design pattern, shortened my learning curve dramatically, kudos Rich!

4 Likes

quick question. if i have a group of lights, can i just add the group to the gHistory, or do i also need to include each item individually in the group gHistory?

That is a good question. I’m at an airport right now so can’t test it out, but I suspect Persistence does not recurse though subgroups so you would have to add the Items to gHistory individually. But it is worth a test.

I use the nested group approach for persistence in my setup and it works for me.

3 Likes

Thanks @rlkoshak and @cweitkamp I will give it a go. I’m hoping it does work, else my items file is going to explode with additional group memberships i’ll have to add.

Hi Rich (@rlkoshak)

First of all a big thanks for the helpful DP.

I use mapDB for restoring the state of some items and rrd4j for charting. Now I am struggle with the default persistence engine for OH. If I use mapDB as default engine generating of the charts will fail at Basic UI. If I set rrd4j as default engine restoring and charting seems to work.

Is there a possibility to define the persistence engine within the sidemap?

Best regards
Raphael

https://www.openhab.org/docs/configuration/sitemaps.html#element-type-chart

service sets the persistence service to use. If no service is specified, openHAB will use the first queryable persistence service it finds. Therefore, for an installation with only a single persistence service, this is not required.

Hi Rich (@rlkoshak)

Thanks for the feedback. I overlooked this part of the documentation. Sorry!

Best regards

Raphael

No problem. There are lots and lots of details. It’s easy to miss.

Dear @rlkoshak ,

Many thanks for this great tutorial.
I hope you can help me with the following issue:

I did the setup as described in the first post of this thread. I want to use MapDB for restore on startup and RRD4J for charting. After switching from my approach - which was to include specific items into rrd4j.persist - to using groups, it doesn’t render the charts anymore.

My rrd4j.persist file:

Strategies {
    everyMinute:    "0 * * * * ?"
    every10Minute:  "10 * * * * ?"
    everyHour:      "0 0 * * * ?"
    everyDay:       "0 0 0 * * ?"
    default = everyChange
}

Items {
    gR* : strategy = everyUpdate, everyMinute
}
// .items File
    Group gM            // mapdb - persist all items on every change and restore them from the db at startup
    Group gR            // rrdj4 - for Items to preserve their history + charting

    String SonoffS201_wifirssi      "Sonoff S20 #1: Wifi RSSI [%s]"     (gM,gR,Gsonoff) { channel="mqtt:topic:mosquitto:sonoff_0xxx:Sonoff_0xxx_rssi" }

test.sitemap

   Frame label="Charts Sonoff S20" {
        Text item=Gcharts {
            Chart item=SonoffS201_wifirssi refresh=60000 period=h service="rrd4j" legend=true
            }
    }

I always get this message - for all items I’ve tried so far:

[WARN ] [thome.ui.internal.chart.ChartServlet] - Chart generation failed: null

With no charts:

http://openhabianpi:8080/rest/persistence

delivers:

[{"id":"mapdb","label":"mapdb","type":"Queryable"},{"id":"rrd4j","label":"rrd4j","type":"Queryable"}]

Default persistence is set to rrd4j:

Unfortunately I don’t use rrd4j any more and I don’t use OH generated charts any more so I don’t really have anything to offer. What you have posted looks like it should work and the error is not informative.

Ok, thanks for letting me know.

Should I open a new thread for this issue? I want to avoid cross posting …